amadan
WIP: walk lease and backend fence, quantity send lease and key fence, attestation path A parked
A checkpoint of the work since the round six brief. Not gated, not reviewed. Billing stays off, the quantity worker stays off (CONSOLE_BILLING_PRORATION unset), and nothing here was sent to a live provider except read-only AWS calls. Object store: - One walk at a time per bucket name: a lease object taken by compare-and-swap (walklock.go), held for the backend's signature window plus a skew allowance. - A backend fence (fence.go): every request of a walk is signed with the lease's instant, so AWS refuses a request sent after the window whenever a paused process sends it. Checked read-only against S3 and IAM (fence_aws_test.go, skipped unless CARLOS_FENCE_AWS_BUCKET is set): a request stamped 13 minutes old is accepted, one stamped 16 minutes old is refused. - Teardown is a write-ahead record on the store record and the ledger; a refused grant undoes its block; a fresh mint refuses an existing bucket and a name another record holds; DestroyedAt lets a retained teardown finish. - A randomized invariant simulation (walk_sim_test.go). Quantity worker: - A send lease: sending an operation and retiring it exclude each other. - A key fence: an operation nobody can show ran is acknowledged only after its idempotency key is consumed at the provider with a metadata-only request, and a retirement is decided on a provider read taken after that. - A key is never reused; an unverified addition always carries its hold. - A randomized invariant simulation with a held request and a moving clock. Replication attestation path A (scope, replicator probe, agent switch) is removed from this snapshot and kept on feat/carloku-scoped-attestation-20261001. It cannot prove an exclusive writer. Replication-write completeness remains a launch blocker. It is not waived and not scoped out. Known failures and open points at this commit: - internal/objstore: TestBucketNameInvariantsUnderRandomWalks fails at seed 778 with CARLOS_SIM_SEEDS=3000 (the Mint invariant). It passes at the default 400 seeds. Under investigation; not yet known whether the fault is in the walk or in the simulation's bookkeeping. - The quantity key fence rests on Stripe's idempotency semantics as documented. The tests use a fake provider that models them. That is not proof. It has not been run against Stripe (test mode is not authorized), a consumed key is not a permanent fencing token (the provider forgets keys), and the worker must stay off until it is verified in a Stripe sandbox. - The invoice engine's lease has no provider-side fence beyond identical replays under the same key; a request first arriving after the provider has forgotten the key is detected and held, not prevented. - The full gate has not been run on this commit. Run here: go build, go vet and tests of internal/objstore, internal/billingstripe, internal/ops and the billing tests of internal/console, all passing at default seeds. internal/host was not run. - Round six dispositions, handoffs and the requirements audit still describe 058ea506.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
24 files changed,
+2754
−1365
internal/billingstripe/client.go+52 −0internal/billingstripe/client_test.go+53 −0internal/console/billingquantity.go+298 −33internal/console/billingquantity_sim_test.go+345 −0internal/console/billingquantity_test.go+576 −33internal/console/billingservice.go+10 −6internal/host/agent.go+0 −6internal/host/durability.go+0 −9internal/host/durabilityreplicator.go+0 −255internal/host/durabilityreplicator_test.go+0 −214internal/host/durabilityscope.go+0 −269internal/host/durabilityscope_test.go+0 −365internal/objstore/aws.go+12 −2internal/objstore/fence.go+76 −0internal/objstore/fence_aws_test.go+91 −0internal/objstore/memprovisioner.go+34 −0internal/objstore/objstore.go+8 −3internal/objstore/ownership.go+30 −12internal/objstore/ownership_test.go+393 −13internal/objstore/walk.go+201 −33internal/objstore/walk_sim_test.go+366 −0internal/objstore/walklock.go+209 −0internal/ops/litestream.go+0 −60internal/ops/litestream_generated_test.go+0 −52
diff --git a/internal/billingstripe/client.go b/internal/billingstripe/client.go| index eed378538..c239834ef 100644 |
| --- a/internal/billingstripe/client.go |
| +++ b/internal/billingstripe/client.go |
| @@ -666,6 +666,58 @@ func (c *Client) UpdateQuantityWith(ctx context.Context, itemID string, quantity |
| return &item, nil |
| } |
| +// FenceQuantityKey makes the provider refuse any later quantity update sent |
| +// under idempotencyKey, without changing anything a customer is billed for. |
| +// |
| +// The provider remembers an idempotency key with the parameters it was |
| +// first used with, and refuses the key with any other parameters. This |
| +// sends a metadata-only update of the item under the key. One of three |
| +// things is then true, and each means a quantity update still on its way |
| +// under that key can no longer take effect: |
| +// |
| +// - the update ran (2xx): the key now belongs to this harmless request, |
| +// and the quantity update will be refused as a parameter mismatch; |
| +// - the provider refused it as a parameter mismatch (idempotency_error): |
| +// the key already belongs to the quantity update, which has run, and a |
| +// second copy of it is answered from memory and runs nothing; |
| +// - the item no longer exists (resource_missing): nothing can update it. |
| +// |
| +// fenced is false with no error when a request under the key is executing |
| +// at this moment (idempotency_key_in_use); the caller asks again later. |
| +// |
| +// The provider's memory of a key is not permanent (Stripe documents at |
| +// least 24 hours). The fence holds for that long from this call. |
| +func (c *Client) FenceQuantityKey(ctx context.Context, itemID string, generation int64, idempotencyKey string) (fenced bool, err error) { |
| + if !validID(itemID) { |
| + return false, &Error{Code: CodeInvalidID} |
| + } |
| + if generation < 0 { |
| + return false, &Error{Code: CodeInvalidParams} |
| + } |
| + form := url.Values{} |
| + form.Set("metadata[carloku_quantity_fence]", strconv.FormatInt(generation, 10)) |
| + var item SubscriptionItem |
| + err = c.do(ctx, http.MethodPost, "/v1/subscription_items/"+itemID, form, idempotencyKey, &item) |
| + if err == nil { |
| + if item.ID != itemID { |
| + return false, &Error{Code: CodeDecodeError} |
| + } |
| + return true, nil |
| + } |
| + var refused *Error |
| + if errors.As(err, &refused) && refused.Code == CodeAPIError { |
| + switch { |
| + case refused.HTTPStatus == http.StatusBadRequest && refused.StripeType == "idempotency_error": |
| + return true, nil |
| + case refused.HTTPStatus == http.StatusNotFound && refused.StripeCode == "resource_missing": |
| + return true, nil |
| + case refused.HTTPStatus == http.StatusConflict: |
| + return false, nil |
| + } |
| + } |
| + return false, err |
| +} |
| + |
| type subscriptionListResponse struct { |
| Data []Subscription `json:"data"` |
| HasMore bool `json:"has_more"` |
diff --git a/internal/billingstripe/client_test.go b/internal/billingstripe/client_test.go| index 02cca1619..c414e13ca 100644 |
| --- a/internal/billingstripe/client_test.go |
| +++ b/internal/billingstripe/client_test.go |
| @@ -879,3 +879,56 @@ func TestDo_PinnedAPIVersionOnEveryRequest(t *testing.T) { |
| } |
| } |
| } |
| + |
| +// The fence request carries the operation's key and nothing that is billed: |
| +// no quantity, no proration. What the provider answers decides whether the |
| +// key is held. |
| +func TestFenceQuantityKey(t *testing.T) { |
| + cases := []struct { |
| + name string |
| + status int |
| + body string |
| + wantFenced bool |
| + wantErr bool |
| + }{ |
| + {"the update ran: the key is now this request's", 200, `{"id":"si_1","quantity":10,"price":{"id":"price_1"}}`, true, false}, |
| + {"refused as a parameter mismatch: the key is the quantity update's", 400, `{"error":{"type":"idempotency_error","message":"Keys for idempotent requests can only be used with the same parameters they were first used with."}}`, true, false}, |
| + {"the item is gone: nothing can update it", 404, `{"error":{"type":"invalid_request_error","code":"resource_missing","message":"No such subscription item"}}`, true, false}, |
| + {"a request under the key is executing: ask again", 409, `{"error":{"type":"idempotency_error","code":"idempotency_key_in_use"}}`, false, false}, |
| + {"a server error holds nothing", 500, `{"error":{"type":"api_error"}}`, false, true}, |
| + {"another refusal holds nothing", 400, `{"error":{"type":"invalid_request_error","code":"parameter_missing"}}`, false, true}, |
| + {"a 404 that is not a missing item holds nothing", 404, `{"error":{"type":"invalid_request_error"}}`, false, true}, |
| + {"rate limited holds nothing", 429, `{"error":{"type":"rate_limit_error"}}`, false, true}, |
| + {"an answer for another item holds nothing", 200, `{"id":"si_other","quantity":10,"price":{"id":"price_1"}}`, false, true}, |
| + } |
| + for _, tc := range cases { |
| + t.Run(tc.name, func(t *testing.T) { |
| + var form map[string][]string |
| + var key, path string |
| + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { |
| + _ = r.ParseForm() |
| + form, key, path = r.PostForm, r.Header.Get("Idempotency-Key"), r.URL.Path |
| + w.WriteHeader(tc.status) |
| + _, _ = w.Write([]byte(tc.body)) |
| + })) |
| + defer srv.Close() |
| + fenced, err := testClient(t, srv.URL).FenceQuantityKey(context.Background(), "si_1", 7, "op-key") |
| + if fenced != tc.wantFenced || (err != nil) != tc.wantErr { |
| + t.Fatalf("fenced=%v err=%v, want fenced=%v err=%v", fenced, err, tc.wantFenced, tc.wantErr) |
| + } |
| + if path != "/v1/subscription_items/si_1" || key != "op-key" { |
| + t.Fatalf("sent to %q under key %q", path, key) |
| + } |
| + if len(form) != 1 || len(form["metadata[carloku_quantity_fence]"]) != 1 || form["metadata[carloku_quantity_fence]"][0] != "7" { |
| + t.Fatalf("the fence request carries more than its marker: %v", form) |
| + } |
| + }) |
| + } |
| + c := testClient(t, "http://127.0.0.1:1") |
| + if _, err := c.FenceQuantityKey(context.Background(), "../x", 1, "k"); err == nil { |
| + t.Fatal("an invalid item id was accepted") |
| + } |
| + if _, err := c.FenceQuantityKey(context.Background(), "si_1", 1, ""); err == nil { |
| + t.Fatal("a fence with no key was accepted") |
| + } |
| +} |
diff --git a/internal/console/billingquantity.go b/internal/console/billingquantity.go| index e8cf1c806..c6ecce2e6 100644 |
| --- a/internal/console/billingquantity.go |
| +++ b/internal/console/billingquantity.go |
| @@ -5,6 +5,7 @@ import ( |
| "database/sql" |
| "errors" |
| "fmt" |
| + "net/http" |
| "strconv" |
| "time" |
| @@ -81,6 +82,10 @@ const ( |
| // Filed when additions from a period that has since renewed are closed |
| // without a charge: a person should know seats went unbilled. |
| billingQuantityUnchargedReason = "billing_quantity_uncharged_period" |
| + // Filed when the provider refuses the request that fences an |
| + // operation's key. The operation cannot be acknowledged without the |
| + // fence, so the account's quantity work waits on a person. |
| + billingQuantityFenceReason = "billing_quantity_fence_refused" |
| ) |
| // prorationBehavior maps the configured policy to the provider's behaviour |
| @@ -203,7 +208,11 @@ func (s *Store) closeEndedQuantityAdditions(accountID int64, now time.Time) (ove |
| return false, err |
| } |
| if over { |
| - if _, err := tx.Exec(`UPDATE billing_quantity_operations SET acked_at=? WHERE account_id=? AND acked_at=0`, max(now.Unix(), 1), accountID); err != nil { |
| + // Not while a sender holds its lease: it is acknowledged on a later |
| + // pass, once no request for it can still be on its way. |
| + // Nor before its key is fenced, unless the provider confirmed it: |
| + // processQuantityAdditions fences it, and the next pass lands here. |
| + if _, err := tx.Exec(`UPDATE billing_quantity_operations SET acked_at=? WHERE account_id=? AND acked_at=0 AND send_lease_until<=? AND `+quantityOperationSettled, max(now.Unix(), 1), accountID, now.Unix()); err != nil { |
| return false, err |
| } |
| } |
| @@ -241,8 +250,20 @@ func (s *Store) markQuantityAdditionsProcessed(accountID int64, subscription str |
| if ackKey == "" { |
| return nil |
| } |
| - _, err := tx.Exec(`UPDATE billing_quantity_operations SET acked_at=? WHERE account_id=? AND idempotency_key=? AND acked_at=0`, max(now.Unix(), 1), accountID, ackKey) |
| - return err |
| + if _, err := tx.Exec(`UPDATE billing_quantity_operations SET acked_at=? WHERE account_id=? AND idempotency_key=? AND acked_at=0 AND `+quantityOperationSettled, max(now.Unix(), 1), accountID, ackKey); err != nil { |
| + return err |
| + } |
| + // Still outstanding: it is neither confirmed nor fenced, and is not |
| + // acknowledged. Nothing is marked either; the transaction is |
| + // abandoned and the next pass fences it first. |
| + var outstanding bool |
| + if err := tx.QueryRow(`SELECT EXISTS(SELECT 1 FROM billing_quantity_operations WHERE account_id=? AND idempotency_key=? AND acked_at=0)`, accountID, ackKey).Scan(&outstanding); err != nil { |
| + return err |
| + } |
| + if outstanding { |
| + return errBillingPending |
| + } |
| + return nil |
| } |
| if maxGeneration <= 0 { |
| // Nothing was pending when the step was decided. |
| @@ -280,6 +301,23 @@ func (s *Store) markQuantityAdditionsProcessed(accountID int64, subscription str |
| return err |
| } |
| } |
| + // An addition recorded against an operation nobody can show ran is |
| + // never cleared without the hold that says so. The hold is filed here, |
| + // in the transaction that clears the obligation, and not left to |
| + // whichever reconcile happened to see the quantity change: the quantity |
| + // can reach the target by a route no reconcile ties to this operation. |
| + if outcome == quantityOutcomeUnverified && ackKey != "" { |
| + var created int64 |
| + switch err := tx.QueryRow(`SELECT created_at FROM billing_quantity_operations WHERE account_id=? AND idempotency_key=?`, accountID, ackKey).Scan(&created); { |
| + case errors.Is(err, sql.ErrNoRows): |
| + case err != nil: |
| + return err |
| + default: |
| + if err := fileQuantityDateReviewTx(tx, accountID, subscription, created, now); err != nil { |
| + return err |
| + } |
| + } |
| + } |
| if err := ack(); err != nil { |
| return err |
| } |
| @@ -312,6 +350,15 @@ type quantityOperation struct { |
| // the provider carries was put there by this operation, at CreatedAt. |
| // Without it a matching quantity is only a matching quantity. |
| ConfirmedAt int64 |
| + // SendLeaseUntil is the unix second until which some process holds the |
| + // right to send this operation's request (claimQuantitySend). While it |
| + // has not passed, nobody may retire or replace the operation. |
| + SendLeaseUntil int64 |
| + // FencedAt is set once the provider holds this operation's idempotency |
| + // key in a way that stops any request under it from changing the |
| + // quantity from now on (fenceQuantityOperation). An operation nobody |
| + // can show ran is acknowledged only when it is set. |
| + FencedAt int64 |
| // Revision is billing_accounts.revision as read when the target was |
| // computed: any change to the billing row since (plan, band, terms, |
| // status, period, quantity) is a different world. |
| @@ -323,6 +370,10 @@ type quantityOperation struct { |
| // may be gone. A variable so a test can shorten it. |
| var quantityOperationSendWindow = 2 * time.Minute |
| +// quantityClock is the worker's clock. A variable so a test can move a |
| +// lease past its end without waiting for it. |
| +var quantityClock = time.Now |
| + |
| // Past this age an unacknowledged operation is never re-sent: the provider |
| // has forgotten its idempotency key, so a send would be a new request, not |
| // a replay. |
| @@ -343,14 +394,20 @@ func (s *Store) migrateBillingQuantityOperations() error { |
| if err := s.addColumnIfMissing("billing_quantity_operations", "revision", `INTEGER NOT NULL DEFAULT 0`); err != nil { |
| return err |
| } |
| - return s.addColumnIfMissing("billing_quantity_operations", "confirmed_at", `INTEGER NOT NULL DEFAULT 0`) |
| + if err := s.addColumnIfMissing("billing_quantity_operations", "confirmed_at", `INTEGER NOT NULL DEFAULT 0`); err != nil { |
| + return err |
| + } |
| + if err := s.addColumnIfMissing("billing_quantity_operations", "send_lease_until", `INTEGER NOT NULL DEFAULT 0`); err != nil { |
| + return err |
| + } |
| + return s.addColumnIfMissing("billing_quantity_operations", "fenced_at", `INTEGER NOT NULL DEFAULT 0`) |
| } |
| func (s *Store) quantityOperationFor(accountID int64) (quantityOperation, bool, error) { |
| var op quantityOperation |
| - err := s.db.QueryRow(`SELECT account_id,subscription,period_start,period_end,item,from_quantity,target,generation,behavior,idempotency_key,created_at,acked_at,revision,confirmed_at |
| + err := s.db.QueryRow(`SELECT account_id,subscription,period_start,period_end,item,from_quantity,target,generation,behavior,idempotency_key,created_at,acked_at,revision,confirmed_at,send_lease_until,fenced_at |
| FROM billing_quantity_operations WHERE account_id=?`, accountID). |
| - Scan(&op.AccountID, &op.Subscription, &op.PeriodStart, &op.PeriodEnd, &op.Item, &op.FromQuantity, &op.Target, &op.Generation, &op.Behavior, &op.Key, &op.CreatedAt, &op.AckedAt, &op.Revision, &op.ConfirmedAt) |
| + Scan(&op.AccountID, &op.Subscription, &op.PeriodStart, &op.PeriodEnd, &op.Item, &op.FromQuantity, &op.Target, &op.Generation, &op.Behavior, &op.Key, &op.CreatedAt, &op.AckedAt, &op.Revision, &op.ConfirmedAt, &op.SendLeaseUntil, &op.FencedAt) |
| if errors.Is(err, sql.ErrNoRows) { |
| return op, false, nil |
| } |
| @@ -361,7 +418,8 @@ func (s *Store) quantityOperationFor(accountID int64) (quantityOperation, bool, |
| // if and only if everything op was computed from is still true. In one |
| // transaction, under the membership write lock, it requires that: |
| // |
| -// - no operation is outstanding for the account; |
| +// - no operation is outstanding for the account, and op's key is not |
| +// the key of the account's previous operation; |
| // - the billing row is the very revision op was read from (which covers |
| // every column at once: plan, band, terms, customer), and is still the |
| // subscription, period, billed quantity and active status op names; |
| @@ -391,6 +449,18 @@ func (s *Store) beginQuantityOperation(op quantityOperation) (started bool, err |
| if outstanding { |
| return false, nil |
| } |
| + // A key is never used for a second operation. The account's previous |
| + // operation may have been fenced, and then the provider holds its key |
| + // for a request that is not this one. A key repeats only when the same |
| + // target is recomputed in the same second; the next pass is a later |
| + // second and a different key. |
| + var reused bool |
| + if err := tx.QueryRow(`SELECT EXISTS(SELECT 1 FROM billing_quantity_operations WHERE idempotency_key=?)`, op.Key).Scan(&reused); err != nil { |
| + return false, err |
| + } |
| + if reused { |
| + return false, nil |
| + } |
| state, err := scanBilling(tx.QueryRow(`SELECT `+billingColumns+` FROM billing_accounts WHERE account_id=?`, op.AccountID), op.AccountID) |
| if err != nil { |
| return false, err |
| @@ -427,7 +497,7 @@ func (s *Store) beginQuantityOperation(op quantityOperation) (started bool, err |
| VALUES(?,?,?,?,?,?,?,?,?,?,?,0,?) |
| ON CONFLICT(account_id) DO UPDATE SET subscription=excluded.subscription,period_start=excluded.period_start,period_end=excluded.period_end,item=excluded.item, |
| from_quantity=excluded.from_quantity,target=excluded.target,generation=excluded.generation,behavior=excluded.behavior, |
| - idempotency_key=excluded.idempotency_key,created_at=excluded.created_at,acked_at=0,revision=excluded.revision,confirmed_at=0`, |
| + idempotency_key=excluded.idempotency_key,created_at=excluded.created_at,acked_at=0,revision=excluded.revision,confirmed_at=0,send_lease_until=0,fenced_at=0`, |
| op.AccountID, op.Subscription, op.PeriodStart, op.PeriodEnd, op.Item, op.FromQuantity, op.Target, op.Generation, op.Behavior, op.Key, op.CreatedAt, op.Revision); err != nil { |
| return false, err |
| } |
| @@ -441,11 +511,112 @@ func (s *Store) confirmQuantityOperation(accountID int64, key string, now time.T |
| return err |
| } |
| -func (s *Store) ackQuantityOperation(accountID int64, key string, now time.Time) error { |
| - _, err := s.db.Exec(`UPDATE billing_quantity_operations SET acked_at=? WHERE account_id=? AND idempotency_key=? AND acked_at=0`, max(now.Unix(), 1), accountID, key) |
| +// The send lease: sending an operation's request and retiring the |
| +// operation are mutually exclusive across processes. |
| +// |
| +// Why it exists. An operation is retired when it is known not to have run |
| +// and is no longer wanted as it stands, and once it is retired a |
| +// replacement may be recorded and completed. A process that had already |
| +// decided to send the old operation, and was between that decision and the |
| +// request, would then send it for the first time over the replacement: a |
| +// lower target, under a key the provider had never seen. Re-reading the |
| +// row before sending only narrows that window. Each review round found |
| +// another way into it. |
| +// |
| +// The invariant. No request for an operation is sent outside a send lease |
| +// on it, and an operation is retired only when no lease on it is live. |
| +// Both are single conditional writes on the operation's row. The sender's |
| +// request carries a deadline that falls billingClockTolerance before the |
| +// lease ends, so two processes whose clocks differ by less than that |
| +// cannot disagree about whether the lease has run out. |
| +const quantitySendLease = 2 * billingClockTolerance |
| + |
| +// claimQuantitySend takes (or extends) the send lease on the account's |
| +// outstanding operation key. ok is false when that operation is no longer |
| +// outstanding (it was retired or completed), or its key has been fenced at |
| +// the provider: it must not be sent. |
| +func (s *Store) claimQuantitySend(accountID int64, key string, now time.Time) (until time.Time, ok bool, err error) { |
| + until = now.Add(quantitySendLease) |
| + res, err := s.db.Exec(`UPDATE billing_quantity_operations SET send_lease_until=? WHERE account_id=? AND idempotency_key=? AND acked_at=0 AND fenced_at=0`, until.Unix(), accountID, key) |
| + if err != nil { |
| + return time.Time{}, false, err |
| + } |
| + n, err := res.RowsAffected() |
| + return until, n == 1, err |
| +} |
| + |
| +// releaseQuantitySend gives the lease up once the request has returned, |
| +// whatever it returned. |
| +func (s *Store) releaseQuantitySend(accountID int64, key string) error { |
| + _, err := s.db.Exec(`UPDATE billing_quantity_operations SET send_lease_until=0 WHERE account_id=? AND idempotency_key=?`, accountID, key) |
| + return err |
| +} |
| + |
| +// retireQuantityOperation acknowledges an operation that did not run and is |
| +// no longer wanted, unless a send lease on it is live. retired false means |
| +// some process may be sending it: it is left outstanding and looked at |
| +// again next pass. |
| +func (s *Store) retireQuantityOperation(accountID int64, key string, now time.Time) (retired bool, err error) { |
| + res, err := s.db.Exec(`UPDATE billing_quantity_operations SET acked_at=? WHERE account_id=? AND idempotency_key=? AND acked_at=0 AND send_lease_until<=? AND `+quantityOperationSettled, max(now.Unix(), 1), accountID, key, now.Unix()) |
| + if err != nil { |
| + return false, err |
| + } |
| + n, err := res.RowsAffected() |
| + return n == 1, err |
| +} |
| + |
| +// The fence: a timed lease does not stop a request that is already on its |
| +// way. |
| +// |
| +// The send lease orders processes that are running. It cannot stop a |
| +// request a sender issued inside its lease and that arrives after the lease |
| +// has run out: the sender was paused between its last check and the wire, |
| +// or the request sat in a proxy. If the operation was retired and replaced |
| +// in the meantime, that request would run for the first time over the |
| +// replacement, under a key the provider had never seen, and lower the |
| +// quantity. |
| +// |
| +// What stops it is the provider. Before an operation that no one can show |
| +// ran is acknowledged in any way, its idempotency key is consumed at the |
| +// provider (billingstripe.FenceQuantityKey). From then on the provider |
| +// itself refuses the late request, or answers it from memory without |
| +// running it. |
| +// |
| +// The invariant, enforced by every statement that acknowledges an |
| +// operation (quantityOperationSettled): an operation is acknowledged only |
| +// when the provider answered its own request with success (confirmed_at), |
| +// or its key is fenced (fenced_at). A decision to retire is made on a |
| +// provider read taken after the fence, never before it, so a request that |
| +// ran just before the fence is seen and recorded, not retired as unsent. |
| +// |
| +// Its limit: the provider remembers a key for a bounded time (Stripe |
| +// documents at least 24 hours). A request delayed longer than that after |
| +// the fence is outside what a key can stop. What remains then is the |
| +// request's own date, which the provider refuses outside the period it was |
| +// computed for, and the mismatch review, which holds the account when the |
| +// provider's quantity is neither what the console bills nor what it wants. |
| +const quantityOperationSettled = `(confirmed_at<>0 OR fenced_at<>0)` |
| + |
| +// markQuantityFenced records that the provider holds the operation's key. |
| +func (s *Store) markQuantityFenced(accountID int64, key string, now time.Time) error { |
| + _, err := s.db.Exec(`UPDATE billing_quantity_operations SET fenced_at=? WHERE account_id=? AND idempotency_key=? AND acked_at=0 AND fenced_at=0`, max(now.Unix(), 1), accountID, key) |
| return err |
| } |
| +// fileQuantityDateReviewTx files the dating hold for an operation, unless |
| +// it is already open. |
| +func fileQuantityDateReviewTx(tx *sql.Tx, accountID int64, subscription string, created int64, now time.Time) error { |
| + reason := billingQuantityDateReviewReason(created) |
| + var open bool |
| + if err := tx.QueryRow(`SELECT EXISTS(SELECT 1 FROM billing_reviews WHERE account_id=? AND subscription=? AND reason=? AND resolved_at=0)`, accountID, subscription, reason).Scan(&open); err != nil { |
| + return err |
| + } |
| + if open { |
| + return nil |
| + } |
| + return fileBillingReviewTx(tx, accountID, subscription, reason, now) |
| +} |
| + |
| // markOperationLanded records an operation whose target the provider |
| // carries in a period that has since renewed: the additions it covered, in |
| // the period it was for, are marked with outcome and the operation's own |
| @@ -459,14 +630,26 @@ func (s *Store) markOperationLanded(op quantityOperation, outcome, reference str |
| if _, err := lockMembershipWritesTx(tx); err != nil { |
| return err |
| } |
| + // Not while a sender holds its lease (it may be about to confirm). |
| + acked, err := tx.Exec(`UPDATE billing_quantity_operations SET acked_at=? WHERE account_id=? AND idempotency_key=? AND acked_at=0 AND send_lease_until<=? AND `+quantityOperationSettled, max(now.Unix(), 1), op.AccountID, op.Key, now.Unix()) |
| + if err != nil { |
| + return err |
| + } |
| + if n, err := acked.RowsAffected(); err != nil { |
| + return err |
| + } else if n == 0 { |
| + return errBillingPending |
| + } |
| res, err := tx.Exec(`UPDATE billing_quantity_events SET processed_at=max(?,occurred_at_ns/1000000000+1), processing_outcome=?, processing_reference=? |
| WHERE account_id=? AND change=? AND processed_at=0 AND subscription=? AND period_start=? AND generation<=?`, |
| now.Unix(), outcome, reference, op.AccountID, quantityEventAddition, op.Subscription, op.PeriodStart, op.Generation) |
| if err != nil { |
| return err |
| } |
| - if _, err := tx.Exec(`UPDATE billing_quantity_operations SET acked_at=? WHERE account_id=? AND idempotency_key=? AND acked_at=0`, max(now.Unix(), 1), op.AccountID, op.Key); err != nil { |
| - return err |
| + if outcome == quantityOutcomeUnverified { |
| + if err := fileQuantityDateReviewTx(tx, op.AccountID, op.Subscription, op.CreatedAt, now); err != nil { |
| + return err |
| + } |
| } |
| if n, err := res.RowsAffected(); err != nil { |
| return err |
| @@ -533,9 +716,18 @@ func (s *Server) processQuantityAdditions(ctx context.Context, accountID int64, |
| // Additions nothing can charge any more are closed first, with no |
| // provider call; so is an operation of a subscription that is over. |
| over, err := s.store.closeEndedQuantityAdditions(accountID, now) |
| - if err != nil || over { |
| + if err != nil { |
| return err |
| } |
| + if over { |
| + // An operation it could not acknowledge is one nobody can show |
| + // ran. Its key is fenced now; the next pass acknowledges it. |
| + op, ok, err := s.store.quantityOperationFor(accountID) |
| + if err != nil || !ok || op.AckedAt != 0 { |
| + return err |
| + } |
| + return s.fenceQuantityOperation(ctx, op, now) |
| + } |
| step, err := s.raiseQuantity(ctx, acct, behavior, now) |
| if err != nil || step.done { |
| return err |
| @@ -607,7 +799,7 @@ func (s *Server) raiseQuantity(ctx context.Context, acct *Account, behavior stri |
| if state.Status != "active" { |
| return quantityStep{done: true}, nil |
| } |
| - read := time.Now() |
| + read := quantityClock() |
| item, ok, err := s.quantityProviderItem(ctx, acct, state) |
| if err != nil || !ok { |
| return quantityStep{done: !ok}, err |
| @@ -636,7 +828,7 @@ func (s *Server) raiseQuantity(ctx context.Context, acct *Account, behavior stri |
| // refuse on every replay. At or past the period end nothing is |
| // recorded at all. |
| // And never before the newest addition it charges for. |
| - created := max(time.Now().Unix(), state.PeriodStart, pending.NewestAt) |
| + created := max(quantityClock().Unix(), state.PeriodStart, pending.NewestAt) |
| if created >= state.PeriodEnd { |
| return quantityStep{done: true}, nil |
| } |
| @@ -689,30 +881,41 @@ func (s *Server) sendQuantityOperation(ctx context.Context, state billingState, |
| if hook := b.quantityBeforeSend; hook != nil { |
| hook() |
| } |
| - if time.Since(read) >= quantityOperationSendWindow || (op.PeriodEnd > 0 && time.Now().Unix() >= op.PeriodEnd) { |
| + if quantityClock().Sub(read) >= quantityOperationSendWindow || (op.PeriodEnd > 0 && quantityClock().Unix() >= op.PeriodEnd) { |
| return quantityStep{done: true}, nil |
| } |
| - // The operation must still be the account's outstanding one. Another |
| - // process may have retired it while this one was between its read and |
| - // here, and a retired operation's request, never having executed, is |
| - // not protected by its idempotency key: sent now it would run for the |
| - // first time, over whatever has been done since. |
| - if cur, ok, err := s.store.quantityOperationFor(op.AccountID); err != nil { |
| + // The send lease (claimQuantitySend). It is taken by one conditional |
| + // write that succeeds only while the operation is still outstanding, and |
| + // while it is live nobody can retire or replace the operation. The |
| + // request's deadline falls before the lease ends. |
| + claimed := quantityClock() |
| + until, ok, err := s.store.claimQuantitySend(op.AccountID, op.Key, claimed) |
| + if err != nil { |
| return quantityStep{}, err |
| - } else if !ok || cur.Key != op.Key || cur.AckedAt != 0 { |
| + } |
| + if !ok { |
| return quantityStep{done: true}, nil |
| } |
| - updated, err := b.client.UpdateQuantityWith(ctx, op.Item, op.Target, op.Behavior, time.Unix(op.CreatedAt, 0), op.Key) |
| + sendCtx, cancel := context.WithTimeout(ctx, until.Add(-billingClockTolerance).Sub(claimed)) |
| + updated, err := b.client.UpdateQuantityWith(sendCtx, op.Item, op.Target, op.Behavior, time.Unix(op.CreatedAt, 0), op.Key) |
| + cancel() |
| if err != nil { |
| + // The request has returned: this process is no longer sending. |
| + if rerr := s.store.releaseQuantitySend(op.AccountID, op.Key); rerr != nil { |
| + return quantityStep{}, errors.Join(err, rerr) |
| + } |
| var refused *billingstripe.Error |
| if errors.As(err, &refused) && refused.RefusedParam("proration_date") { |
| // The provider refused the request outright over its date: it |
| // did not execute, and the same date will be refused on every |
| // replay (the period moved, or a clock disagreed). The |
| - // operation is retired now, not replayed until it expires. The |
| - // additions stay pending and the next pass computes afresh. |
| - if aerr := s.store.ackQuantityOperation(op.AccountID, op.Key, time.Now()); aerr != nil { |
| - return quantityStep{}, errors.Join(err, aerr) |
| + // operation is not replayed until it expires: its key is |
| + // fenced now, and the next pass, reading the provider after |
| + // the fence, retires it. The additions stay pending and are |
| + // computed afresh after that. |
| + op.SendLeaseUntil = 0 |
| + if ferr := s.fenceQuantityOperation(ctx, op, quantityClock()); ferr != nil { |
| + return quantityStep{}, errors.Join(err, ferr) |
| } |
| } |
| return quantityStep{}, err |
| @@ -724,7 +927,10 @@ func (s *Server) sendQuantityOperation(ctx context.Context, state billingState, |
| // what lets the reconcile date the assignment at the operation's |
| // instant, so it is recorded before anything else can observe the new |
| // quantity through this process. |
| - if err := s.store.confirmQuantityOperation(op.AccountID, op.Key, time.Now()); err != nil { |
| + if err := s.store.confirmQuantityOperation(op.AccountID, op.Key, quantityClock()); err != nil { |
| + return quantityStep{}, err |
| + } |
| + if err := s.store.releaseQuantitySend(op.AccountID, op.Key); err != nil { |
| return quantityStep{}, err |
| } |
| if hook := b.quantityAfterSend; hook != nil { |
| @@ -740,14 +946,24 @@ func (s *Server) sendQuantityOperation(ctx context.Context, state billingState, |
| // re-sends one the provider has forgotten the key of. |
| func (s *Server) resolveQuantityOperation(ctx context.Context, acct *Account, state billingState, op quantityOperation, now time.Time) (quantityStep, error) { |
| abandon := func() (quantityStep, error) { |
| - // Provably not executed, and no longer wanted as it stands: the |
| - // additions stay pending and the next pass starts afresh. |
| - return quantityStep{done: true}, s.store.ackQuantityOperation(acct.ID, op.Key, now) |
| + // Not executed as far as the provider shows, and no longer wanted |
| + // as it stands. It is retired only on a provider read taken after |
| + // its key was fenced (or its request confirmed): until then a |
| + // request for it may still be on its way, and what this pass read |
| + // is not the last word. So an unfenced operation is fenced here |
| + // and decided next pass, from a fresh read. While some process |
| + // holds the right to send it, neither happens. The additions stay |
| + // pending and are computed afresh once it is retired. |
| + if !op.settled() { |
| + return quantityStep{done: true}, s.fenceQuantityOperation(ctx, op, now) |
| + } |
| + _, err := s.store.retireQuantityOperation(acct.ID, op.Key, quantityClock()) |
| + return quantityStep{done: true}, err |
| } |
| if op.Subscription != state.Subscription { |
| return abandon() |
| } |
| - read := time.Now() |
| + read := quantityClock() |
| item, ok, err := s.quantityProviderItem(ctx, acct, state) |
| if err != nil { |
| return quantityStep{}, err |
| @@ -771,6 +987,9 @@ func (s *Server) resolveQuantityOperation(ctx context.Context, acct *Account, st |
| // outcome says exactly that: landed, unverified. Otherwise it never |
| // ran and is closed. |
| if item.Quantity == op.Target { |
| + if !op.settled() { |
| + return quantityStep{done: true}, s.fenceQuantityOperation(ctx, op, now) |
| + } |
| return quantityStep{done: true}, s.store.markOperationLanded(op, quantityOutcomeUnverified, quantityReference(op.Item, op.Target, op.Behavior), now) |
| } |
| return abandon() |
| @@ -787,6 +1006,16 @@ func (s *Server) resolveQuantityOperation(ctx context.Context, acct *Account, st |
| outcome := quantityOutcomeUnverified |
| if op.ConfirmedAt != 0 { |
| outcome = quantityOutcomeFor(op.Behavior) |
| + } else if op.SendLeaseUntil > read.Unix() { |
| + // A sender holds the lease: its request may have just been |
| + // answered and the answer not yet written down. Nothing is |
| + // concluded until it has finished. |
| + return quantityStep{done: true}, nil |
| + } else if op.FencedAt == 0 { |
| + // Nobody can show this operation ran, and a request for it |
| + // may still be on its way. Its key is fenced first; the next |
| + // pass records it from a read taken after the fence. |
| + return quantityStep{done: true}, s.fenceQuantityOperation(ctx, op, now) |
| } |
| return quantityStep{state: state, generation: op.Generation, behavior: op.Behavior, key: op.Key, outcome: outcome, |
| reference: quantityReference(op.Item, op.Target, op.Behavior)}, nil |
| @@ -805,10 +1034,46 @@ func (s *Server) resolveQuantityOperation(ctx context.Context, acct *Account, st |
| // read above shows it did not land, and no sender can still be |
| // inside its send window, so it is closed and never re-sent. |
| return abandon() |
| + case op.FencedAt != 0: |
| + // Fenced, and the provider (read after the fence) shows it did not |
| + // run. It can never be sent again; it is retired. |
| + return abandon() |
| } |
| return s.sendQuantityOperation(ctx, state, op, read) |
| } |
| +// settled reports whether the operation may be acknowledged: the provider |
| +// confirmed its request, or holds its key fenced. |
| +func (op quantityOperation) settled() bool { return op.ConfirmedAt != 0 || op.FencedAt != 0 } |
| + |
| +// fenceQuantityOperation consumes op's idempotency key at the provider (see |
| +// quantityOperationSettled) and records that it did. It does nothing while |
| +// a sender holds the lease: that sender is running, and will finish or let |
| +// the lease run out. A refusal the provider will repeat files a review, |
| +// once, because the operation cannot be acknowledged without the fence. |
| +func (s *Server) fenceQuantityOperation(ctx context.Context, op quantityOperation, now time.Time) error { |
| + if op.settled() || op.SendLeaseUntil > quantityClock().Unix() { |
| + return nil |
| + } |
| + if hook := s.billing.quantityBeforeFence; hook != nil { |
| + hook() |
| + } |
| + fenced, err := s.billing.client.FenceQuantityKey(ctx, op.Item, op.Generation, op.Key) |
| + if err != nil { |
| + var refused *billingstripe.Error |
| + if errors.As(err, &refused) && refused.Code == billingstripe.CodeAPIError && refused.HTTPStatus >= 400 && refused.HTTPStatus < 500 && refused.HTTPStatus != http.StatusTooManyRequests { |
| + if rerr := s.store.fileBillingReviewOnce(op.AccountID, op.Subscription, billingQuantityFenceReason, now); rerr != nil { |
| + return errors.Join(err, rerr) |
| + } |
| + } |
| + return err |
| + } |
| + if !fenced { |
| + return nil |
| + } |
| + return s.store.markQuantityFenced(op.AccountID, op.Key, quantityClock()) |
| +} |
| + |
| // quantityMismatch files the review for a provider quantity that is |
| // neither what the console bills nor what it wants. It is filed once and |
| // left alone while open, so its occurrence count does not move under the |
diff --git a/internal/console/billingquantity_sim_test.go b/internal/console/billingquantity_sim_test.go| new file mode 100644 |
| index 000000000..71aa5738c |
| --- /dev/null |
| +++ b/internal/console/billingquantity_sim_test.go |
| @@ -0,0 +1,345 @@ |
| +package console |
| + |
| +import ( |
| + "context" |
| + "fmt" |
| + "math/rand" |
| + "os" |
| + "strconv" |
| + "strings" |
| + "testing" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/plans" |
| +) |
| + |
| +// The invariants of the team quantity worker, checked under random |
| +// interleavings of two console processes, webhook reconciles, member |
| +// changes and provider failures. The other tests each pin one schedule a |
| +// reviewer found. This states what must hold on every schedule. |
| +// |
| +// Monotone. No request the worker sends lowers the provider's quantity. |
| +// Leased. Every request is issued while its operation is outstanding |
| +// and a send lease on it is live. |
| +// Fenced. A request that arrives after its sender's lease has run out |
| +// (the sender was paused, or the request was held on its way) |
| +// and after its operation was acknowledged without ever having |
| +// run, does not run. The console cannot stop such a request: |
| +// the provider must, and the fake provider here refuses only |
| +// what the real one documents it refuses (a key replayed with |
| +// other parameters). |
| +// Held. An addition recorded as landed and unverified has an open |
| +// dating review. |
| +// Readable. The assignment history always prices: its instants are in |
| +// revision order. |
| +// Settles. With no more failures and no lease live, a few passes leave |
| +// no addition pending, the billed quantity equal to the |
| +// provider's, and at least the member count. |
| +func TestQuantityInvariantsUnderRandomSchedules(t *testing.T) { |
| + seeds := 60 |
| + if testing.Short() { |
| + seeds = 12 |
| + } |
| + if n, err := strconv.Atoi(os.Getenv("CARLOS_SIM_SEEDS")); err == nil && n > 0 { |
| + seeds = n |
| + } |
| + for seed := 0; seed < seeds; seed++ { |
| + if trace, problem := simulateQuantity(t, int64(seed)); problem != "" { |
| + t.Fatalf("seed %d: %s\n%s", seed, problem, strings.Join(trace, "\n")) |
| + } |
| + } |
| +} |
| + |
| +func simulateQuantity(t *testing.T, seed int64) (trace []string, problem string) { |
| + t.Helper() |
| + ctx := context.Background() |
| + rng := rand.New(rand.NewSource(seed)) |
| + // The fixture's period began thirty hours ago, so an operation can age |
| + // past the replay limit inside it. |
| + prevAge := quantityFixtureAge |
| + quantityFixtureAge = 30 * time.Hour |
| + fx, q, _ := newQuantityFixture(t, billingProrationImmediate) |
| + quantityFixtureAge = prevAge |
| + // One clock for everything, which a pause moves forward. |
| + var elapsed time.Duration |
| + prevClock := quantityClock |
| + quantityClock = func() time.Time { return time.Now().Add(elapsed) } |
| + defer func() { quantityClock = prevClock }() |
| + clock := func() time.Time { return quantityClock().UTC() } |
| + procB := &Server{cfg: fx.srv.cfg, store: fx.srv.store, billing: &billingRuntime{locks: newStoreWalkLocks(), client: fx.srv.billing.client, cfg: fx.srv.billing.cfg}} |
| + logf := func(format string, args ...any) { trace = append(trace, fmt.Sprintf(format, args...)) } |
| + fail := func(format string, args ...any) { |
| + if problem == "" { |
| + problem = fmt.Sprintf(format, args...) |
| + } |
| + } |
| + members := 10 |
| + // publish stands in for the entitlement publisher, which a member |
| + // change waits on after the billed quantity moves. |
| + publish := func() { |
| + row, err := scanEntitlementOutbox(fx.st.db.QueryRow(`SELECT `+entitlementOutboxColumns+` FROM entitlement_outbox WHERE account_id=?`, OpsAccountID)) |
| + if err == nil { |
| + _ = fx.st.markEntitlementPublished(OpsAccountID, row.Revision, row.Digest, clock()) |
| + } |
| + } |
| + addMember := func() { |
| + publish() |
| + members++ |
| + // Recorded two seconds ago, so an operation dated no earlier than |
| + // its newest addition is not dated ahead of this test's clock (an |
| + // operation dated ahead is left alone until that instant, which |
| + // TestQuantityFutureDatedOperationIsNotRetiredUnderItsSender pins). |
| + if err := fx.st.AddMember(OpsAccountID, fmt.Sprintf("sim%d-%d@example.test", seed, members), RoleMember, clock().Add(-2*time.Second)); err != nil { |
| + // Held by the membership gate (the entitlement has not caught |
| + // up): not added, and not this test's subject. |
| + members-- |
| + logf(" member not added: %v", err) |
| + return |
| + } |
| + logf(" member added (%d)", members) |
| + } |
| + depth := 0 |
| + // running is the process whose pass is in progress. Whatever happens |
| + // in the middle of it happens in the OTHER process: a process does not |
| + // interleave with itself, it waits on its own account lock. |
| + running := fx.srv |
| + other := func() *Server { |
| + if running == fx.srv { |
| + return procB |
| + } |
| + return fx.srv |
| + } |
| + // between runs something else in the middle of the worker: another |
| + // process's pass, a webhook, a new member. One level deep. |
| + between := func(where string) { |
| + if depth > 0 || rng.Intn(3) != 0 { |
| + return |
| + } |
| + depth++ |
| + defer func() { depth-- }() |
| + switch rng.Intn(3) { |
| + case 0: |
| + logf(" -- at %s: the other process runs a pass", where) |
| + if err := other().billingQuantityPass(ctx, clock()); err != nil { |
| + logf(" %v", err) |
| + } |
| + case 1: |
| + logf(" -- at %s: webhook in the other process", where) |
| + if err := other().reconcileBillingSubscription(ctx, "sub_team", fmt.Sprintf("evt_sim_%d_%d", seed, len(trace)), clock()); err != nil { |
| + logf(" %v", err) |
| + } |
| + case 2: |
| + logf(" -- at %s:", where) |
| + addMember() |
| + } |
| + } |
| + for _, rt := range []*billingRuntime{fx.srv.billing, procB.billing} { |
| + rt.quantityBeforeBegin = func() { between("before begin") } |
| + rt.quantityBeforeSend = func() { between("before send") } |
| + rt.quantityAfterSend = func() { between("after send") } |
| + } |
| + // Leased is checked where the request is issued. Then, sometimes, the |
| + // request is held on its way for longer than its lease (once in a run |
| + // for most of a day, which takes its operation past the replay limit) |
| + // while the other process works. A request whose operation was |
| + // acknowledged in that time without having run must never run. |
| + mustNeverRun := map[string]bool{} |
| + longPause := false |
| + ran := func(key string) bool { |
| + q.mu.Lock() |
| + defer q.mu.Unlock() |
| + for _, p := range q.posts { |
| + if p["key"] == key { |
| + return true |
| + } |
| + } |
| + return false |
| + } |
| + q.duringPost = func() { |
| + q.mu.Lock() |
| + key := q.arriving |
| + q.mu.Unlock() |
| + op, ok, err := fx.st.quantityOperationFor(OpsAccountID) |
| + if err != nil || !ok || op.AckedAt != 0 || op.Key != key { |
| + fail("Leased: a request was issued for %q with no such outstanding operation (%+v %v %v)", key, op, ok, err) |
| + return |
| + } |
| + if op.SendLeaseUntil <= quantityClock().Unix() { |
| + fail("Leased: a request was issued with no live send lease (%+v)", op) |
| + } |
| + if depth > 0 || rng.Intn(3) != 0 { |
| + between("request in flight") |
| + return |
| + } |
| + depth++ |
| + hold := quantitySendLease + time.Minute |
| + if !longPause && rng.Intn(2) == 0 { |
| + longPause, hold = true, 23*time.Hour |
| + } |
| + elapsed += hold |
| + logf(" -- the request is held on its way for %v; the other process works", hold) |
| + for i, n := 0, 1+rng.Intn(4); i < n; i++ { |
| + switch rng.Intn(4) { |
| + case 0: |
| + if members < 16 { |
| + addMember() |
| + } |
| + case 1: |
| + if err := other().reconcileBillingSubscription(ctx, "sub_team", fmt.Sprintf("evt_sim_%d_%d", seed, len(trace)), clock()); err != nil { |
| + logf(" webhook: %v", err) |
| + } |
| + default: |
| + publish() |
| + if err := other().billingQuantityPass(ctx, clock()); err != nil { |
| + logf(" pass: %v", err) |
| + } |
| + } |
| + } |
| + depth-- |
| + if now, ok, _ := fx.st.quantityOperationFor(OpsAccountID); ok && (now.Key != key || (now.AckedAt != 0 && now.ConfirmedAt == 0)) && !ran(key) { |
| + logf(" its operation was acknowledged unsent; the request must not run now") |
| + mustNeverRun[key] = true |
| + } |
| + } |
| + check := func() { |
| + for key := range mustNeverRun { |
| + if ran(key) { |
| + fail("Fenced: the request for %q ran after its operation was acknowledged as never having run: %+v", key, q.posts) |
| + } |
| + } |
| + if q.lowered { |
| + fail("Monotone: a request lowered the provider's quantity: %+v", q.posts) |
| + } |
| + events, err := fx.st.billingQuantityEvents(OpsAccountID) |
| + if err != nil { |
| + fail("events: %v", err) |
| + return |
| + } |
| + for _, e := range events { |
| + if e.ProcessingOutcome != quantityOutcomeUnverified { |
| + continue |
| + } |
| + var open int |
| + if err := fx.st.db.QueryRow(`SELECT COUNT(*) FROM billing_reviews WHERE account_id=? AND reason LIKE 'billing_quantity_date_unproven:%' AND resolved_at=0`, OpsAccountID).Scan(&open); err != nil || open == 0 { |
| + fail("Held: an addition is recorded as landed_unverified with no open dating review (%v)", err) |
| + } |
| + } |
| + history, err := fx.st.billingInvoiceHistory(OpsAccountID) |
| + if err != nil { |
| + fail("history: %v", err) |
| + return |
| + } |
| + var hist []plans.Assignment |
| + for _, h := range history { |
| + hist = append(hist, h.Assignment) |
| + } |
| + for i := 1; i < len(hist); i++ { |
| + if hist[i].EffectiveAt.Before(hist[i-1].EffectiveAt) { |
| + fail("Readable: revision %d takes effect before revision %d", hist[i].Revision, hist[i-1].Revision) |
| + } |
| + } |
| + } |
| + for step := 0; step < 14 && problem == ""; step++ { |
| + // Passes are periodic: each step is a later second. |
| + elapsed += time.Second |
| + q.mu.Lock() |
| + q.failPost, q.failAfter = false, false |
| + switch rng.Intn(6) { |
| + case 0: |
| + q.failPost = true |
| + logf(" next request fails before it runs") |
| + case 1: |
| + q.failAfter = true |
| + logf(" next request runs and its answer is lost") |
| + } |
| + q.mu.Unlock() |
| + switch rng.Intn(4) { |
| + case 0: |
| + if members < 16 { |
| + addMember() |
| + } |
| + case 1: |
| + logf("webhook") |
| + if err := fx.srv.reconcileBillingSubscription(ctx, "sub_team", fmt.Sprintf("evt_sim_%d_%d", seed, len(trace)), clock()); err != nil { |
| + logf(" %v", err) |
| + } |
| + default: |
| + running = fx.srv |
| + if rng.Intn(2) == 0 { |
| + running = procB |
| + } |
| + logf("pass (%s)", map[bool]string{true: "A", false: "B"}[running == fx.srv]) |
| + if err := running.billingQuantityPass(ctx, clock()); err != nil { |
| + logf(" %v", err) |
| + } |
| + } |
| + check() |
| + } |
| + if problem != "" { |
| + return trace, problem |
| + } |
| + // Settle: no failures, nothing interleaved. |
| + q.mu.Lock() |
| + q.failPost, q.failAfter = false, false |
| + q.mu.Unlock() |
| + for _, rt := range []*billingRuntime{fx.srv.billing, procB.billing} { |
| + rt.quantityBeforeBegin, rt.quantityBeforeSend, rt.quantityAfterSend = nil, nil, nil |
| + } |
| + q.duringPost = func() { |
| + if op, ok, _ := fx.st.quantityOperationFor(OpsAccountID); !ok || op.AckedAt != 0 || op.SendLeaseUntil <= quantityClock().Unix() { |
| + fail("Leased: a settling request arrived without its lease (%+v)", op) |
| + } |
| + } |
| + logf("-- settle --") |
| + running = fx.srv |
| + for i := 0; i < 7; i++ { |
| + elapsed += time.Second |
| + publish() |
| + if err := fx.srv.billingQuantityPass(ctx, clock()); err != nil { |
| + logf(" %v", err) |
| + } |
| + } |
| + check() |
| + if problem != "" { |
| + return trace, problem |
| + } |
| + if pending := pendingAdditions(t, fx); len(pending) != 0 { |
| + state, _ := fx.st.billingAccount(OpsAccountID) |
| + op, _, _ := fx.st.quantityOperationFor(OpsAccountID) |
| + var reviews []string |
| + rows, _ := fx.st.db.Query(`SELECT reason||' open='||(resolved_at=0) FROM billing_reviews WHERE account_id=?`, OpsAccountID) |
| + for rows != nil && rows.Next() { |
| + var r string |
| + _ = rows.Scan(&r) |
| + reviews = append(reviews, r) |
| + } |
| + if rows != nil { |
| + rows.Close() |
| + } |
| + q.mu.Lock() |
| + atProvider := q.quantity |
| + q.mu.Unlock() |
| + // Held by a review a person must resolve is a settled state: the |
| + // additions are owed, and the account says why they wait. |
| + for _, r := range reviews { |
| + if strings.HasSuffix(r, "open=1") { |
| + return trace, "" |
| + } |
| + } |
| + return trace, fmt.Sprintf("Settles: %d addition(s) still pending with no review; billed %d (status %s, period %d), provider %d, members %d, operation %+v, reviews %v", len(pending), state.Quantity, state.Status, state.PeriodStart, atProvider, members, op, reviews) |
| + } |
| + state, err := fx.st.billingAccount(OpsAccountID) |
| + if err != nil { |
| + return trace, err.Error() |
| + } |
| + q.mu.Lock() |
| + atProvider := q.quantity |
| + q.mu.Unlock() |
| + if state.Quantity != atProvider || state.Quantity < int64(members) { |
| + return trace, fmt.Sprintf("Settles: billed %d, provider %d, members %d", state.Quantity, atProvider, members) |
| + } |
| + if op, ok, _ := fx.st.quantityOperationFor(OpsAccountID); ok && op.AckedAt == 0 { |
| + return trace, fmt.Sprintf("Settles: an operation is still outstanding: %+v", op) |
| + } |
| + return trace, "" |
| +} |
diff --git a/internal/console/billingquantity_test.go b/internal/console/billingquantity_test.go| index 8edd687f2..4de18695c 100644 |
| --- a/internal/console/billingquantity_test.go |
| +++ b/internal/console/billingquantity_test.go |
| @@ -32,14 +32,37 @@ type quantityStripe struct { |
| failAfter bool // accept the POST, then answer 500: a lost response |
| // byKey is the provider's idempotency memory: a key it has seen is |
| // answered with what it answered then, and changes nothing. |
| - byKey map[string]int64 |
| - params map[string]string // key -> the parameters it was first sent with |
| - replays int |
| - lowered bool // the quantity ever went down |
| - refused int // requests refused for a proration date outside the period |
| + byKey map[string]int64 |
| + params map[string]string // key -> the parameters it was first sent with |
| + replays int |
| + lowered bool // the quantity ever went down |
| + refused int // requests refused for a proration date outside the period |
| + fences []string // keys taken by a metadata-only update |
| + mismatched []string // keys refused for arriving with other parameters |
| + // duringPost runs when an update has arrived at the provider and has |
| + // not yet taken effect: the request is in flight. |
| + duringPost func() |
| + // refuseFence makes the provider refuse a fence request as invalid. |
| + refuseFence bool |
| + // arriving is the key of the request the hooks are being run for. |
| + arriving string |
| + // duringFence is the same for a fence request. |
| + duringFence func() |
| } |
| func (q *quantityStripe) serve(w http.ResponseWriter, r *http.Request) { |
| + if r.Method == http.MethodPost { |
| + _ = r.ParseForm() |
| + q.mu.Lock() |
| + q.arriving = r.Header.Get("Idempotency-Key") |
| + q.mu.Unlock() |
| + if hook := q.duringPost; hook != nil && r.PostForm.Get("quantity") != "" { |
| + hook() |
| + } |
| + if hook := q.duringFence; hook != nil && r.PostForm.Get("quantity") == "" { |
| + hook() |
| + } |
| + } |
| q.mu.Lock() |
| defer q.mu.Unlock() |
| w.Header().Set("Content-Type", "application/json") |
| @@ -74,6 +97,7 @@ func (q *quantityStripe) serve(w http.ResponseWriter, r *http.Request) { |
| if prev, seen := q.byKey[key]; seen { |
| // A key replayed with different parameters is refused. |
| if q.params[key] != r.PostForm.Encode() { |
| + q.mismatched = append(q.mismatched, key) |
| w.WriteHeader(http.StatusBadRequest) |
| _, _ = w.Write([]byte(`{"error":{"type":"idempotency_error","message":"Keys for idempotent requests can only be used with the same parameters"}}`)) |
| return |
| @@ -85,6 +109,19 @@ func (q *quantityStripe) serve(w http.ResponseWriter, r *http.Request) { |
| if q.byKey == nil { |
| q.byKey, q.params = map[string]int64{}, map[string]string{} |
| } |
| + if r.PostForm.Get("quantity") == "" && q.refuseFence { |
| + w.WriteHeader(http.StatusBadRequest) |
| + _, _ = w.Write([]byte(`{"error":{"type":"invalid_request_error","message":"refused"}}`)) |
| + return |
| + } |
| + if r.PostForm.Get("quantity") == "" { |
| + // A metadata-only update: the fence. It takes the key and |
| + // changes nothing that is billed. |
| + q.byKey[key], q.params[key] = q.quantity, r.PostForm.Encode() |
| + q.fences = append(q.fences, key) |
| + _ = json.NewEncoder(w).Encode(item()) |
| + return |
| + } |
| q.byKey[key], q.params[key] = n, r.PostForm.Encode() |
| q.posts = append(q.posts, map[string]string{"quantity": r.PostForm.Get("quantity"), "proration_behavior": r.PostForm.Get("proration_behavior"), "proration_date": r.PostForm.Get("proration_date"), "key": key}) |
| if n < q.quantity { |
| @@ -102,6 +139,26 @@ func (q *quantityStripe) serve(w http.ResponseWriter, r *http.Request) { |
| } |
| } |
| +// fencePass runs the pass that fences an operation nobody can show ran, and |
| +// checks that it did exactly that: the key is fenced, and the operation is |
| +// still outstanding for the next pass to settle from a fresh read. |
| +func fencePass(t *testing.T, fx *consoleFixture, q *quantityStripe) { |
| + t.Helper() |
| + before, _, _ := fx.st.quantityOperationFor(OpsAccountID) |
| + if err := fx.srv.billingQuantityPass(context.Background(), time.Now().UTC()); err != nil { |
| + t.Fatal(err) |
| + } |
| + after, ok, err := fx.st.quantityOperationFor(OpsAccountID) |
| + if err != nil || !ok || after.Key != before.Key || after.AckedAt != 0 || after.FencedAt == 0 { |
| + t.Fatalf("the pass did not fence the operation and leave it for the next: %+v %v %v", after, ok, err) |
| + } |
| + q.mu.Lock() |
| + defer q.mu.Unlock() |
| + if _, held := q.byKey[after.Key]; !held { |
| + t.Fatalf("fenced in the store, but the provider does not hold the key %q", after.Key) |
| + } |
| +} |
| + |
| // quantityFixtureAge is how long before the test the fixture's subscription |
| // period began. A test that needs an older period sets it and restores it. |
| var quantityFixtureAge = time.Hour |
| @@ -285,11 +342,10 @@ func TestQuantityWorkerChargesTheTransientSeat(t *testing.T) { |
| } |
| } |
| -// The update landed but its response was lost. While the operation is |
| -// young another process could still be mid-flight with it, so nothing is |
| -// concluded. After that, the next pass reads the provider, sees the target |
| -// there, sends nothing and finishes. It cannot know whether its lost |
| -// request or someone's hand set that quantity, so the additions are |
| +// The update landed but its response was lost. The sender has returned, so |
| +// nobody holds the right to send. The next pass reads the provider, sees |
| +// the target there, sends nothing and finishes. It cannot know whether its |
| +// lost request or someone's hand set that quantity, so the additions are |
| // recorded as landed and unverified, the assignment is dated when it was |
| // observed, and a review holds the account until a person has looked. |
| func TestQuantityWorkerRecoversALostResponse(t *testing.T) { |
| @@ -305,14 +361,17 @@ func TestQuantityWorkerRecoversALostResponse(t *testing.T) { |
| if len(pendingAdditions(t, fx)) != 1 || !quantityHeld(t, fx) { |
| t.Fatal("an unconfirmed update was marked processed") |
| } |
| - // At once: the operation is young and unconfirmed. Nothing is recorded. |
| - if err := fx.srv.billingQuantityPass(ctx, time.Now().UTC()); !errors.Is(err, errBillingPending) { |
| - t.Fatalf("a pass over a young unconfirmed operation: %v, want pending", err) |
| - } |
| - if state, _ := fx.st.billingAccount(OpsAccountID); state.Quantity != 10 || len(pendingAdditions(t, fx)) != 1 { |
| - t.Fatalf("a young unconfirmed operation was settled: quantity %d", state.Quantity) |
| + if op, _, _ := fx.st.quantityOperationFor(OpsAccountID); op.SendLeaseUntil != 0 || op.ConfirmedAt != 0 { |
| + t.Fatalf("after a failed request the operation still carries a lease or a confirmation: %+v", op) |
| } |
| + // The next pass comes a while later. It fences the key (the provider |
| + // already holds it, for the request that ran), and the one after |
| + // records what the provider shows. |
| op := ageQuantityOperation(t, fx, 3*time.Minute) |
| + fencePass(t, fx, q) |
| + if len(q.fences) != 0 || len(q.mismatched) != 1 { |
| + t.Fatalf("the key the request ran under was taken again: fences %v mismatched %v", q.fences, q.mismatched) |
| + } |
| if err := fx.srv.billingQuantityPass(ctx, time.Now().UTC()); err != nil { |
| t.Fatal(err) |
| } |
| @@ -792,9 +851,11 @@ func TestQuantityWorkerCannotChargeAcrossARenewal(t *testing.T) { |
| t.Fatalf("refused=%d posts=%+v quantity=%d: an old period's seat was charged into the new one", q.refused, q.posts, q.quantity) |
| } |
| // A request refused over its date did not execute and never will with |
| - // that date: the operation is retired at once, not replayed for a day. |
| - if op, ok, _ := fx.st.quantityOperationFor(OpsAccountID); !ok || op.AckedAt == 0 { |
| - t.Fatalf("the refused operation is still outstanding: %+v", op) |
| + // that date. Its key is fenced at once, so it is never replayed and no |
| + // copy of it still on its way can run; it is retired once the console |
| + // has reconciled the renewal and read the provider again. |
| + if op, ok, _ := fx.st.quantityOperationFor(OpsAccountID); !ok || op.AckedAt != 0 || op.FencedAt == 0 || len(q.fences) != 1 || q.fences[0] != op.Key { |
| + t.Fatalf("the refused operation was not fenced at once: %+v fences %v", op, q.fences) |
| } |
| fx.srv.billing.quantityBeforeSend = nil |
| // Until the console has reconciled the renewal, nothing more is sent. |
| @@ -1049,6 +1110,7 @@ func TestQuantityCoincidentalChangeIsNotBackdated(t *testing.T) { |
| if err != nil || rev.ResolvedAt != 0 || rev.Occurrences != 1 { |
| t.Fatalf("no open review for the unproven charge date: %+v %v", rev, err) |
| } |
| + fencePass(t, fx, q) |
| if err := fx.srv.billingQuantityPass(context.Background(), time.Now().UTC()); err != nil { |
| t.Fatal(err) |
| } |
| @@ -1058,23 +1120,199 @@ func TestQuantityCoincidentalChangeIsNotBackdated(t *testing.T) { |
| assertLastAddition(t, fx, quantityOutcomeUnverified, "si_team:quantity=11:always_invoice") |
| } |
| -// A webhook that observes the target while the operation is young and |
| -// unconfirmed records nothing: the sender may be between the provider's |
| -// answer and writing it down. |
| -func TestQuantityWebhookOverAYoungUnconfirmedOperationWaits(t *testing.T) { |
| +// While a process holds an operation's send lease, nobody concludes |
| +// anything about it. A webhook that observes the target records nothing; a |
| +// worker in another process neither retires it nor marks it. Once the |
| +// lease is gone, the same observations settle it. |
| +func TestQuantityNothingIsConcludedWhileASenderHoldsTheLease(t *testing.T) { |
| fx, q, now := newQuantityFixture(t, billingProrationImmediate) |
| + ctx := context.Background() |
| if err := fx.st.AddMember(OpsAccountID, "eleventh@example.test", RoleMember, now.Add(time.Minute)); err != nil { |
| t.Fatal(err) |
| } |
| - q.failAfter = true |
| - if err := fx.srv.billingQuantityPass(context.Background(), time.Now().UTC()); err == nil { |
| - t.Fatal("a lost response must surface as an error") |
| + procB := &Server{cfg: fx.srv.cfg, store: fx.srv.store, billing: &billingRuntime{locks: newStoreWalkLocks(), client: fx.srv.billing.client, cfg: fx.srv.billing.cfg}} |
| + seen := false |
| + q.duringPost = func() { |
| + if seen { |
| + return |
| + } |
| + seen = true |
| + // The request is at the provider and has not taken effect. The |
| + // sender holds the lease. |
| + op, ok, _ := fx.st.quantityOperationFor(OpsAccountID) |
| + if !ok || op.SendLeaseUntil <= time.Now().Unix() { |
| + t.Errorf("no live send lease while the request is in flight: %+v", op) |
| + } |
| + // Another process tries to retire it, as the replay limit or a |
| + // moved period would have it do. |
| + if retired, err := fx.st.retireQuantityOperation(OpsAccountID, op.Key, time.Now()); err != nil || retired { |
| + t.Errorf("an operation was retired under its sender: retired=%v err=%v", retired, err) |
| + } |
| + // A whole pass of another process may finish this same operation |
| + // under its own key (the provider runs a key once). It may not |
| + // retire it unrun or put another in its place. |
| + if err := procB.billingQuantityPass(ctx, time.Now().UTC()); err != nil && !errors.Is(err, errBillingPending) { |
| + t.Errorf("process B: %v", err) |
| + } |
| + after, _, _ := fx.st.quantityOperationFor(OpsAccountID) |
| + q.mu.Lock() |
| + atProvider := q.quantity |
| + q.mu.Unlock() |
| + if after.Key != op.Key || (after.AckedAt != 0 && atProvider != op.Target) { |
| + t.Errorf("the operation was retired unrun or replaced while its request was in flight: %+v (provider at %d)", after, atProvider) |
| + } |
| } |
| - if err := fx.srv.reconcileBillingSubscription(context.Background(), "sub_team", "evt_quantity_young", time.Now().UTC()); !errors.Is(err, errBillingPending) { |
| - t.Fatalf("webhook over a young unconfirmed operation: %v, want pending", err) |
| + if err := fx.srv.billingQuantityPass(ctx, time.Now().UTC()); err != nil { |
| + t.Fatal(err) |
| } |
| - if state, _ := fx.st.billingAccount(OpsAccountID); state.Quantity != 10 { |
| - t.Fatalf("billed quantity %d: the change was recorded", state.Quantity) |
| + if !seen || len(q.posts) != 1 || q.lowered { |
| + t.Fatalf("seen=%v posts=%+v lowered=%v: the one operation ran once", seen, q.posts, q.lowered) |
| + } |
| + assertLastAddition(t, fx, quantityOutcomeCharged, "si_team:quantity=11:always_invoice") |
| + if op, _, _ := fx.st.quantityOperationFor(OpsAccountID); op.AckedAt == 0 || op.ConfirmedAt == 0 { |
| + t.Fatalf("the operation is not finished and confirmed: %+v", op) |
| + } |
| +} |
| + |
| +// A webhook that observes the target while a sender holds the lease |
| +// records nothing, and the worker in another process concludes nothing. |
| +// When the lease is gone the same observation settles. |
| +func TestQuantityReconcileWaitsWhileASenderHoldsTheLease(t *testing.T) { |
| + fx, q, op := agedQuantityFixture(t) |
| + ctx := context.Background() |
| + q.mu.Lock() |
| + q.quantity = 11 |
| + q.mu.Unlock() |
| + if _, ok, err := fx.st.claimQuantitySend(OpsAccountID, op.Key, time.Now()); err != nil || !ok { |
| + t.Fatalf("claim: %v %v", ok, err) |
| + } |
| + if err := fx.srv.reconcileBillingSubscription(ctx, "sub_team", "evt_quantity_in_flight", time.Now().UTC()); !errors.Is(err, errBillingPending) { |
| + t.Fatalf("a reconcile while a sender holds the lease: %v, want pending", err) |
| + } |
| + if err := fx.srv.billingQuantityPass(ctx, time.Now().UTC()); err != nil { |
| + t.Fatal(err) |
| + } |
| + if state, _ := fx.st.billingAccount(OpsAccountID); state.Quantity != 10 || len(pendingAdditions(t, fx)) != 1 { |
| + t.Fatalf("something was concluded under a live lease: quantity %d", state.Quantity) |
| + } |
| + if cur, _, _ := fx.st.quantityOperationFor(OpsAccountID); cur.AckedAt != 0 { |
| + t.Fatalf("the operation was acknowledged under a live lease: %+v", cur) |
| + } |
| + if len(q.fences)+len(q.mismatched) != 0 { |
| + t.Fatalf("the key was fenced under a live lease: %v %v", q.fences, q.mismatched) |
| + } |
| + if err := fx.st.releaseQuantitySend(OpsAccountID, op.Key); err != nil { |
| + t.Fatal(err) |
| + } |
| + fencePass(t, fx, q) |
| + if err := fx.srv.billingQuantityPass(ctx, time.Now().UTC()); err != nil { |
| + t.Fatal(err) |
| + } |
| + assertLastAddition(t, fx, quantityOutcomeUnverified, "si_team:quantity=11:always_invoice") |
| +} |
| + |
| +// The schedule of review round six. A recovery worker reads an operation |
| +// just inside the replay limit and goes to send it. Before its request |
| +// takes effect the operation crosses the limit, and another worker, seeing |
| +// the provider still at the old quantity, would retire it. It must not: |
| +// the first worker holds the lease. Without the lease the operation is |
| +// retired, a replacement for a higher target completes, and the first |
| +// worker's request then lowers the quantity. |
| +func TestQuantityOperationIsNotRetiredAcrossTheReplayLimitUnderItsSender(t *testing.T) { |
| + fx, q, op := agedQuantityFixture(t) |
| + ctx := context.Background() |
| + // Just inside the limit when the recovery worker reads it. |
| + inside := time.Now().Add(-quantityOperationReplayLimit + 30*time.Second).Unix() |
| + if _, err := fx.st.db.Exec(`UPDATE billing_quantity_operations SET created_at=? WHERE account_id=?`, inside, OpsAccountID); err != nil { |
| + t.Fatal(err) |
| + } |
| + procB := &Server{cfg: fx.srv.cfg, store: fx.srv.store, billing: &billingRuntime{locks: newStoreWalkLocks(), client: fx.srv.billing.client, cfg: fx.srv.billing.cfg}} |
| + seen := false |
| + q.duringPost = func() { |
| + if seen { |
| + return |
| + } |
| + seen = true |
| + // The operation is now past the limit, and another member joins. |
| + if _, err := fx.st.db.Exec(`UPDATE billing_quantity_operations SET created_at=? WHERE account_id=?`, time.Now().Add(-quantityOperationReplayLimit-time.Minute).Unix(), OpsAccountID); err != nil { |
| + t.Error(err) |
| + } |
| + if err := fx.st.AddMember(OpsAccountID, "twelfth@example.test", RoleMember, time.Now().UTC()); err != nil { |
| + t.Error(err) |
| + } |
| + for i := 0; i < 2; i++ { |
| + if err := procB.billingQuantityPass(ctx, time.Now().UTC()); err != nil && !errors.Is(err, errBillingPending) { |
| + t.Errorf("process B pass %d: %v", i, err) |
| + } |
| + } |
| + after, _, _ := fx.st.quantityOperationFor(OpsAccountID) |
| + if after.Key != op.Key || after.AckedAt != 0 { |
| + t.Errorf("the operation was retired or replaced under its sender: %+v", after) |
| + } |
| + // Nor is its key taken at the provider while its sender holds the |
| + // lease: that would refuse the sender's own request. |
| + q.mu.Lock() |
| + if len(q.fences)+len(q.mismatched) != 0 || after.FencedAt != 0 { |
| + t.Errorf("the operation was fenced under its sender: fences %v mismatched %v %+v", q.fences, q.mismatched, after) |
| + } |
| + q.mu.Unlock() |
| + } |
| + // The recovery worker's own request carries the original date, which |
| + // the test's provider accepts. |
| + if _, err := fx.st.db.Exec(`UPDATE billing_quantity_operations SET created_at=? WHERE account_id=?`, inside, OpsAccountID); err != nil { |
| + t.Fatal(err) |
| + } |
| + _ = fx.srv.billingQuantityPass(ctx, time.Now().UTC()) |
| + q.duringPost = nil |
| + if !seen { |
| + t.Fatal("the request never reached the provider") |
| + } |
| + for i := 0; i < 3; i++ { |
| + _ = fx.srv.billingQuantityPass(ctx, time.Now().UTC()) |
| + } |
| + if q.lowered { |
| + t.Fatalf("the provider's quantity was lowered: posts %+v", q.posts) |
| + } |
| + for i := 1; i < len(q.posts); i++ { |
| + prev, _ := strconv.Atoi(q.posts[i-1]["quantity"]) |
| + cur, _ := strconv.Atoi(q.posts[i]["quantity"]) |
| + if cur < prev { |
| + t.Fatalf("requests went 11 after 12: %+v", q.posts) |
| + } |
| + } |
| +} |
| + |
| +// The two-webhook schedule of review round six. An unconfirmed operation |
| +// (10 to 11) never executes. Someone sets the provider to 12, then to 11. |
| +// Neither webhook is the operation's own transition, so neither reconcile |
| +// files a dating hold. The worker then finds the target and records the |
| +// additions as landed and unverified. That record must carry its hold with |
| +// it: an addition is never cleared as unverified without a review. |
| +func TestQuantityUnverifiedAdditionIsNeverClearedWithoutAHold(t *testing.T) { |
| + fx, q, op := agedQuantityFixture(t) |
| + ctx := context.Background() |
| + for i, quantity := range []int64{12, 11} { |
| + q.mu.Lock() |
| + q.quantity = quantity |
| + q.mu.Unlock() |
| + if err := fx.srv.reconcileBillingSubscription(ctx, "sub_team", fmt.Sprintf("evt_by_hand_%d", i), time.Now().UTC()); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + if rev, err := fx.st.billingReview(OpsAccountID, "sub_team", billingQuantityDateReviewReason(op.CreatedAt)); err == nil && rev.Occurrences > 0 { |
| + t.Fatalf("fixture: a reconcile already filed the hold: %+v", rev) |
| + } |
| + fencePass(t, fx, q) |
| + if err := fx.srv.billingQuantityPass(ctx, time.Now().UTC()); err != nil { |
| + t.Fatal(err) |
| + } |
| + if len(q.posts) != 0 { |
| + t.Fatalf("the worker sent: %+v", q.posts) |
| + } |
| + assertLastAddition(t, fx, quantityOutcomeUnverified, "si_team:quantity=11:always_invoice") |
| + rev, err := fx.st.billingReview(OpsAccountID, "sub_team", billingQuantityDateReviewReason(op.CreatedAt)) |
| + if err != nil || rev.ResolvedAt != 0 || rev.Occurrences != 1 { |
| + t.Fatalf("the addition was cleared as unverified with no open hold: %+v %v", rev, err) |
| } |
| } |
| @@ -1352,11 +1590,14 @@ func TestQuantityWorkerClosesAnOperationOfAnEndedSubscription(t *testing.T) { |
| t.Fatal(err) |
| } |
| gets := q.gets |
| + // The only thing the provider is asked is to fence the key of the |
| + // operation nobody can show ran. Then it is acknowledged. |
| + fencePass(t, fx, q) |
| if err := fx.srv.billingQuantityPass(ctx, time.Now().UTC()); err != nil { |
| t.Fatal(err) |
| } |
| - if q.gets != gets || len(q.posts) != 0 { |
| - t.Fatalf("an ended subscription reached the provider: gets %d -> %d posts=%v", gets, q.gets, q.posts) |
| + if q.gets != gets || len(q.posts) != 0 || len(q.fences) != 1 { |
| + t.Fatalf("an ended subscription reached the provider: gets %d -> %d posts=%v fences=%v", gets, q.gets, q.posts, q.fences) |
| } |
| if op, ok, _ := fx.st.quantityOperationFor(OpsAccountID); !ok || op.AckedAt == 0 { |
| t.Fatalf("operation left outstanding: %+v", op) |
| @@ -1386,6 +1627,12 @@ func TestQuantityWorkerNeverReplaysAForgottenKey(t *testing.T) { |
| if _, err := fx.st.db.Exec(`UPDATE billing_quantity_operations SET created_at=?, idempotency_key='carloku-qty-aged' WHERE account_id=?`, aged, OpsAccountID); err != nil { |
| t.Fatal(err) |
| } |
| + // Fenced first: the provider has forgotten the key, so without the |
| + // fence a copy of the old request still on its way would run as new. |
| + fencePass(t, fx, q) |
| + if len(q.fences) != 1 || q.fences[0] != "carloku-qty-aged" { |
| + t.Fatalf("the expired operation's key was not taken at the provider: %v", q.fences) |
| + } |
| if err := fx.srv.billingQuantityPass(ctx, time.Now().UTC()); err != nil { |
| t.Fatal(err) |
| } |
| @@ -1453,7 +1700,7 @@ func TestQuantityWorkerFinishesAnOutstandingOperation(t *testing.T) { |
| if op, ok, _ := fx.st.quantityOperationFor(OpsAccountID); !ok || op.AckedAt != 0 || op.Target != 11 || op.ConfirmedAt != 0 { |
| t.Fatalf("operation not left outstanding and unconfirmed: %+v %v", op, ok) |
| } |
| - ageQuantityOperation(t, fx, 3*time.Minute) |
| + fencePass(t, fx, q) |
| if err := fx.srv.billingQuantityPass(ctx, time.Now().UTC()); err != nil { |
| t.Fatal(err) |
| } |
| @@ -1515,3 +1762,299 @@ func TestQuantityProcessingGuardIsReplacedWhenItsBodyChanges(t *testing.T) { |
| } |
| st.Close() |
| } |
| + |
| +// A timed lease alone does not fence a request that is already on its way. |
| +// |
| +// A sender takes the send lease and issues its request. The request does |
| +// not arrive: the sender was paused between its last check and the wire, or |
| +// the request sat in a proxy. Three hours pass. The lease has long run out |
| +// and the operation has crossed the replay limit, so another process |
| +// retires it, records a replacement for a higher quantity, and that lands. |
| +// Then the old request arrives, under a key the provider never saw before |
| +// the retirement. |
| +// |
| +// Nothing in the console can stop it now. The provider does: the retiring |
| +// process took the key at the provider before it retired the operation, so |
| +// the late request is refused and the quantity is not lowered. |
| +func TestQuantityPausedSenderIsRefusedByTheProviderAfterItsOperationIsRetired(t *testing.T) { |
| + fx, q, op := agedQuantityFixture(t) |
| + ctx := context.Background() |
| + var pause time.Duration |
| + prev := quantityClock |
| + quantityClock = func() time.Time { return time.Now().Add(pause) } |
| + t.Cleanup(func() { quantityClock = prev }) |
| + procB := &Server{cfg: fx.srv.cfg, store: fx.srv.store, billing: &billingRuntime{locks: newStoreWalkLocks(), client: fx.srv.billing.client, cfg: fx.srv.billing.cfg}} |
| + arrivedLate := false |
| + q.duringPost = func() { |
| + if arrivedLate { |
| + return |
| + } |
| + arrivedLate = true |
| + leased, _, _ := fx.st.quantityOperationFor(OpsAccountID) |
| + if leased.Key != op.Key || leased.SendLeaseUntil <= quantityClock().Unix() { |
| + t.Errorf("fixture: the sender did not hold the lease when it sent: %+v", leased) |
| + } |
| + // The request is on its way, and stays there for three hours. |
| + pause = 3 * time.Hour |
| + if leased.SendLeaseUntil > quantityClock().Unix() { |
| + t.Errorf("fixture: the lease outlived the pause: %+v", leased) |
| + } |
| + if err := fx.st.AddMember(OpsAccountID, "twelfth@example.test", RoleMember, quantityClock().UTC().Add(-time.Second)); err != nil { |
| + t.Error(err) |
| + } |
| + for i := 0; i < 4; i++ { |
| + if err := procB.billingQuantityPass(ctx, quantityClock().UTC()); err != nil && !errors.Is(err, errBillingPending) { |
| + t.Errorf("process B pass %d: %v", i, err) |
| + } |
| + } |
| + after, _, _ := fx.st.quantityOperationFor(OpsAccountID) |
| + q.mu.Lock() |
| + defer q.mu.Unlock() |
| + if after.Key == op.Key || after.Target != 12 || q.quantity != 12 { |
| + t.Errorf("fixture: process B did not retire the old operation and land its replacement: %+v, provider at %d", after, q.quantity) |
| + } |
| + if len(q.fences) != 1 || q.fences[0] != op.Key { |
| + t.Errorf("the old operation was retired without its key being taken at the provider: fences %v", q.fences) |
| + } |
| + } |
| + // The sender: the operation is twenty hours old, inside the replay |
| + // limit, and the provider shows it has not run. It sends. |
| + err := fx.srv.billingQuantityPass(ctx, quantityClock().UTC()) |
| + q.duringPost = nil |
| + if !arrivedLate { |
| + t.Fatal("the request never reached the provider") |
| + } |
| + if err == nil { |
| + t.Fatal("the late request's refusal must surface in its sender") |
| + } |
| + q.mu.Lock() |
| + quantity, lowered, posts, mismatched := q.quantity, q.lowered, append([]map[string]string(nil), q.posts...), append([]string(nil), q.mismatched...) |
| + q.mu.Unlock() |
| + if lowered || quantity != 12 { |
| + t.Fatalf("the late request ran over the replacement: provider at %d, posts %+v", quantity, posts) |
| + } |
| + for _, p := range posts { |
| + if p["key"] == op.Key { |
| + t.Fatalf("the retired operation's request executed: %+v", posts) |
| + } |
| + } |
| + if len(mismatched) != 1 || mismatched[0] != op.Key { |
| + t.Fatalf("the provider did not refuse the late request by its key: %v", mismatched) |
| + } |
| + for i := 0; i < 3; i++ { |
| + if err := fx.srv.billingQuantityPass(ctx, quantityClock().UTC()); err != nil && !errors.Is(err, errBillingPending) { |
| + t.Fatal(err) |
| + } |
| + } |
| + if state, _ := fx.st.billingAccount(OpsAccountID); state.Quantity != 12 { |
| + t.Fatalf("billed quantity %d, want 12", state.Quantity) |
| + } |
| + if cur, _, _ := fx.st.quantityOperationFor(OpsAccountID); cur.AckedAt == 0 || cur.Key == op.Key { |
| + t.Fatalf("the replacement was not completed: %+v", cur) |
| + } |
| +} |
| + |
| +// Every statement that acknowledges an operation refuses one that is |
| +// neither confirmed by the provider nor fenced there. This is the store's |
| +// half of the fence: no code path can retire, land or close an operation |
| +// nobody can show ran while a request for it could still take effect. |
| +func TestQuantityOperationIsNeverAcknowledgedUnfenced(t *testing.T) { |
| + for _, how := range []string{"retire", "landed", "mark", "ended"} { |
| + t.Run(how, func(t *testing.T) { |
| + fx, _, op := agedQuantityFixture(t) |
| + now := time.Now().UTC() |
| + state, _ := fx.st.billingAccount(OpsAccountID) |
| + acknowledge := func() { |
| + switch how { |
| + case "retire": |
| + if retired, err := fx.st.retireQuantityOperation(OpsAccountID, op.Key, now); err != nil { |
| + t.Fatal(err) |
| + } else if retired != (op.FencedAt != 0) { |
| + t.Fatalf("retired=%v with fenced_at=%d", retired, op.FencedAt) |
| + } |
| + case "landed": |
| + err := fx.st.markOperationLanded(op, quantityOutcomeUnverified, "ref", now) |
| + if (err == nil) != (op.FencedAt != 0) { |
| + t.Fatalf("markOperationLanded err=%v with fenced_at=%d", err, op.FencedAt) |
| + } |
| + case "mark": |
| + err := fx.st.markQuantityAdditionsProcessed(OpsAccountID, state.Subscription, state.PeriodStart, 0, quantityOutcomeIncluded, "ref", op.Key, now) |
| + if (err == nil) != (op.FencedAt != 0) { |
| + t.Fatalf("markQuantityAdditionsProcessed err=%v with fenced_at=%d", err, op.FencedAt) |
| + } |
| + case "ended": |
| + if _, err := fx.st.db.Exec(`UPDATE billing_accounts SET status='canceled' WHERE account_id=?`, OpsAccountID); err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, err := fx.st.closeEndedQuantityAdditions(OpsAccountID, now); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + } |
| + acknowledge() |
| + if cur, _, _ := fx.st.quantityOperationFor(OpsAccountID); cur.AckedAt != 0 { |
| + t.Fatalf("acknowledged with neither a confirmation nor a fence: %+v", cur) |
| + } |
| + if _, ok, err := fx.st.claimQuantitySend(OpsAccountID, op.Key, now.Add(-time.Hour)); err != nil || !ok { |
| + t.Fatalf("an unfenced operation could not be claimed: %v %v", ok, err) |
| + } |
| + if err := fx.st.markQuantityFenced(OpsAccountID, op.Key, now); err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, ok, err := fx.st.claimQuantitySend(OpsAccountID, op.Key, now); err != nil || ok { |
| + t.Fatalf("a fenced operation was claimed for sending: %v %v", ok, err) |
| + } |
| + op, _, _ = fx.st.quantityOperationFor(OpsAccountID) |
| + acknowledge() |
| + if cur, _, _ := fx.st.quantityOperationFor(OpsAccountID); cur.AckedAt == 0 { |
| + t.Fatalf("a fenced operation could not be acknowledged: %+v", cur) |
| + } |
| + }) |
| + } |
| +} |
| + |
| +// A decision to retire is made on a provider read taken after the fence, |
| +// never on one taken before it. Here the worker reads the provider, finds |
| +// an aged operation has not run, and goes to retire it. Between that read |
| +// and the fence, the operation's own request (held on its way until now) |
| +// arrives and runs. The fence then finds the key already taken. The |
| +// operation ran: it must be recorded as landed, with the hold that says |
| +// its date is unproven, and not retired as never sent. |
| +func TestQuantityRequestThatRanJustBeforeTheFenceIsRecordedNotRetired(t *testing.T) { |
| + fx, q, op := agedQuantityFixture(t) |
| + ctx := context.Background() |
| + if _, err := fx.st.db.Exec(`UPDATE billing_quantity_operations SET created_at=? WHERE account_id=?`, time.Now().Add(-quantityOperationReplayLimit-time.Minute).Unix(), OpsAccountID); err != nil { |
| + t.Fatal(err) |
| + } |
| + op, _, _ = fx.st.quantityOperationFor(OpsAccountID) |
| + ranLate := false |
| + fx.srv.billing.quantityBeforeFence = func() { |
| + if ranLate { |
| + return |
| + } |
| + ranLate = true |
| + if _, err := fx.srv.billing.client.UpdateQuantityWith(ctx, op.Item, op.Target, op.Behavior, time.Unix(op.CreatedAt, 0), op.Key); err != nil { |
| + t.Errorf("the held request: %v", err) |
| + } |
| + } |
| + if err := fx.srv.billingQuantityPass(ctx, time.Now().UTC()); err != nil { |
| + t.Fatal(err) |
| + } |
| + if !ranLate { |
| + t.Fatal("the worker never went to fence the aged operation") |
| + } |
| + cur, _, _ := fx.st.quantityOperationFor(OpsAccountID) |
| + if cur.AckedAt != 0 || cur.FencedAt == 0 { |
| + t.Fatalf("the operation was retired on the read taken before the fence: %+v", cur) |
| + } |
| + if len(q.fences) != 0 || len(q.mismatched) != 1 { |
| + t.Fatalf("fences %v mismatched %v: the key was the request's, the fence must have found it taken", q.fences, q.mismatched) |
| + } |
| + for i := 0; i < 2; i++ { |
| + if err := fx.srv.billingQuantityPass(ctx, time.Now().UTC()); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + if len(q.posts) != 1 || q.posts[0]["key"] != op.Key { |
| + t.Fatalf("posts %+v: want only the operation's own request", q.posts) |
| + } |
| + assertLastAddition(t, fx, quantityOutcomeUnverified, "si_team:quantity=11:always_invoice") |
| + rev, err := fx.st.billingReview(OpsAccountID, "sub_team", billingQuantityDateReviewReason(op.CreatedAt)) |
| + if err != nil || rev.ResolvedAt != 0 { |
| + t.Fatalf("no open hold for the charge whose date is unproven: %+v %v", rev, err) |
| + } |
| + if state, _ := fx.st.billingAccount(OpsAccountID); state.Quantity != 11 { |
| + t.Fatalf("billed quantity %d, want 11", state.Quantity) |
| + } |
| +} |
| + |
| +// A fence the provider refuses in a way it will repeat cannot be waited |
| +// out. The operation stays outstanding (it is never acknowledged unfenced) |
| +// and a review is filed, once, so a person sees why the account's quantity |
| +// work has stopped. |
| +func TestQuantityRefusedFenceFilesAReviewAndAcknowledgesNothing(t *testing.T) { |
| + fx, q, op := agedQuantityFixture(t) |
| + ctx := context.Background() |
| + if _, err := fx.st.db.Exec(`UPDATE billing_quantity_operations SET created_at=? WHERE account_id=?`, time.Now().Add(-quantityOperationReplayLimit-time.Minute).Unix(), OpsAccountID); err != nil { |
| + t.Fatal(err) |
| + } |
| + q.mu.Lock() |
| + q.refuseFence = true |
| + q.mu.Unlock() |
| + for i := 0; i < 3; i++ { |
| + if err := fx.srv.billingQuantityPass(ctx, time.Now().UTC()); err == nil { |
| + t.Fatal("a refused fence must surface") |
| + } |
| + } |
| + if cur, _, _ := fx.st.quantityOperationFor(OpsAccountID); cur.Key != op.Key || cur.AckedAt != 0 || cur.FencedAt != 0 { |
| + t.Fatalf("acknowledged or marked fenced over a refused fence: %+v", cur) |
| + } |
| + rev, err := fx.st.billingReview(OpsAccountID, "sub_team", billingQuantityFenceReason) |
| + if err != nil || rev.ResolvedAt != 0 || rev.Occurrences != 1 { |
| + t.Fatalf("review for the refused fence: %+v %v", rev, err) |
| + } |
| + if len(q.posts) != 0 { |
| + t.Fatalf("sent: %+v", q.posts) |
| + } |
| +} |
| + |
| +// A fenced operation the provider shows did not run is retired by the next |
| +// pass, whatever its age, and is never sent: its key is no longer its own. |
| +// The addition it covered is still owed, and a fresh operation charges it. |
| +func TestQuantityFencedOperationIsRetiredAndReplacedNeverSent(t *testing.T) { |
| + fx, q, op := agedQuantityFixture(t) |
| + ctx := context.Background() |
| + // The fixture ages the operation's date and not its key, which would |
| + // otherwise be the key a fresh operation computes in this same second. |
| + if _, err := fx.st.db.Exec(`UPDATE billing_quantity_operations SET idempotency_key='carloku-qty-aged' WHERE account_id=?`, OpsAccountID); err != nil { |
| + t.Fatal(err) |
| + } |
| + op.Key = "carloku-qty-aged" |
| + if err := fx.st.markQuantityFenced(OpsAccountID, op.Key, time.Now()); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := fx.srv.billingQuantityPass(ctx, time.Now().UTC()); err != nil { |
| + t.Fatal(err) |
| + } |
| + if cur, _, _ := fx.st.quantityOperationFor(OpsAccountID); cur.Key != op.Key || cur.AckedAt == 0 { |
| + t.Fatalf("the fenced operation was not retired: %+v", cur) |
| + } |
| + if len(q.posts) != 0 { |
| + t.Fatalf("the fenced operation was sent: %+v", q.posts) |
| + } |
| + if len(pendingAdditions(t, fx)) != 1 { |
| + t.Fatal("the addition the retired operation covered is no longer owed") |
| + } |
| + if err := fx.srv.billingQuantityPass(ctx, time.Now().UTC()); err != nil { |
| + t.Fatal(err) |
| + } |
| + if len(q.posts) != 1 || q.posts[0]["key"] == op.Key || q.posts[0]["quantity"] != "11" { |
| + t.Fatalf("posts %+v: want one fresh request for 11 under a fresh key", q.posts) |
| + } |
| + if state, _ := fx.st.billingAccount(OpsAccountID); state.Quantity != 11 || len(pendingAdditions(t, fx)) != 0 { |
| + t.Fatalf("not charged afresh: quantity %d", state.Quantity) |
| + } |
| +} |
| + |
| +// A key is never used for a second operation: the previous one may have |
| +// been fenced, and the provider then holds that key for another request. |
| +func TestBeginQuantityOperationNeverReusesAKey(t *testing.T) { |
| + fx, _, op := agedQuantityFixture(t) |
| + if err := fx.st.markQuantityFenced(OpsAccountID, op.Key, time.Now()); err != nil { |
| + t.Fatal(err) |
| + } |
| + if retired, err := fx.st.retireQuantityOperation(OpsAccountID, op.Key, time.Now()); err != nil || !retired { |
| + t.Fatalf("retire: %v %v", retired, err) |
| + } |
| + state, _ := fx.st.billingAccount(OpsAccountID) |
| + again := op |
| + again.AckedAt, again.FencedAt, again.Revision = 0, 0, state.Revision |
| + again.CreatedAt = time.Now().Unix() |
| + if started, err := fx.st.beginQuantityOperation(again); err != nil || started { |
| + t.Fatalf("an operation was recorded under its predecessor's key: started=%v err=%v", started, err) |
| + } |
| + again.Key = op.Key + "-next" |
| + if started, err := fx.st.beginQuantityOperation(again); err != nil || !started { |
| + t.Fatalf("a fresh key was refused: started=%v err=%v", started, err) |
| + } |
| +} |
diff --git a/internal/console/billingservice.go b/internal/console/billingservice.go| index 6b5770a81..14bc6a5f7 100644 |
| --- a/internal/console/billingservice.go |
| +++ b/internal/console/billingservice.go |
| @@ -50,6 +50,9 @@ type billingRuntime struct { |
| // quantityAfterSend runs after the provider accepted an update and the |
| // operation was confirmed, before the worker's own reconcile. |
| quantityAfterSend func() |
| + // quantityBeforeFence runs before an operation's key is fenced at the |
| + // provider. Test seam only. |
| + quantityBeforeFence func() |
| } |
| var ( |
| @@ -504,14 +507,15 @@ func (s *Server) reconcileBillingSubscription(ctx context.Context, id, eventID s |
| // success answer to that operation (ConfirmedAt) shows the |
| // operation is what changed it. Without that answer: |
| // |
| - // - a young operation may be mid-flight in another process, |
| - // its answer not yet recorded. Nothing is recorded now; the |
| - // event is retried and the worker's next pass settles it; |
| - // - an older one is ambiguous for good (a lost response, or |
| - // someone set the same number by hand). The change is |
| + // - while a process holds the operation's send lease, its |
| + // request may be in flight or its answer not yet recorded. |
| + // Nothing is recorded now; the event is retried and the |
| + // worker's next pass settles it; |
| + // - with no lease live it is ambiguous for good (a lost |
| + // response, or someone set the same number by hand). The change is |
| // recorded when observed, and the store files a review, |
| // because the instant the provider charged from is unknown. |
| - if op.ConfirmedAt == 0 && time.Now().Unix()-op.CreatedAt < int64(quantityOperationSendWindow.Seconds()) { |
| + if op.ConfirmedAt == 0 && op.SendLeaseUntil > quantityClock().Unix() { |
| return errBillingPending |
| } |
| quantityStart, quantityKey, quantityConfirmed = time.Unix(op.CreatedAt, 0).UTC(), op.Key, op.ConfirmedAt != 0 |
diff --git a/internal/host/agent.go b/internal/host/agent.go| index cad1267f8..6b1bebf83 100644 |
| --- a/internal/host/agent.go |
| +++ b/internal/host/agent.go |
| @@ -253,12 +253,6 @@ func RunAgent(args []string) { |
| if durabilityErr != nil { |
| log.Printf("agent: durability: %v", durabilityErr) |
| } |
| - // The scoped replication attestation is off unless this box's |
| - // environment asks for it by name. Unset, the ledger attests nothing. |
| - if scopedAttestationEnabled(os.Getenv) { |
| - durabilityPub.enableScopedAttestation(newReplicatorProbe(), reg.List, dd) |
| - log.Printf("agent: durability: scoped attestation enabled (%s=scoped)", durabilityAttestEnv) |
| - } |
| go durabilityPub.loop() |
| // Fleet box identity (customer-fleets plan, Task 6 fix round — controller |
diff --git a/internal/host/durability.go b/internal/host/durability.go| index 90355bb53..8d3ca3657 100644 |
| --- a/internal/host/durability.go |
| +++ b/internal/host/durability.go |
| @@ -42,11 +42,6 @@ type durabilityPublisher struct { |
| stopped atomic.Bool |
| bindingFailed atomic.Bool |
| registered bool |
| - // attest, when set (enableScopedAttestation), observes the box and |
| - // applies the scoped attestation. It runs before every seal. Nil is |
| - // the default and the production state: nothing is attested. |
| - attest func(context.Context) |
| - scopeVerifiedAt time.Time |
| } |
| func newDurabilityPublisher(dd, box string, now func() time.Time) (*durabilityPublisher, error) { |
| @@ -156,10 +151,6 @@ func (p *durabilityPublisher) publish(ctx context.Context, store release.Store) |
| if p.stopped.Load() { |
| return errors.New("durability_publisher_stopped") |
| } |
| - // The claim is checked before anything is sealed under it. |
| - if p.attest != nil { |
| - p.attest(ctx) |
| - } |
| if err := p.ledger.Seal(); err != nil { |
| return err |
| } |
diff --git a/internal/host/durabilityreplicator.go b/internal/host/durabilityreplicator.go| deleted file mode 100644 |
| index 456079619..000000000 |
| --- a/internal/host/durabilityreplicator.go |
| +++ /dev/null |
| @@ -1,255 +0,0 @@ |
| -package host |
| - |
| -import ( |
| - "context" |
| - "errors" |
| - "fmt" |
| - "github.com/carlosframework/platform/internal/registry" |
| - "github.com/carlosframework/platform/internal/usage" |
| - "os" |
| - "os/exec" |
| - "path/filepath" |
| - "strconv" |
| - "strings" |
| - "time" |
| - |
| - "github.com/carlosframework/platform/internal/ops" |
| -) |
| - |
| -// The reader of the external replicator's state, for durabilityScope. |
| -// Like the scope, NOTHING IN THE AGENT CALLS THIS YET. |
| - |
| -const ( |
| - externalReplicatorUnit = "carlos-litestream.service" |
| - externalReplicatorConfig = "/etc/litestream.yml" |
| -) |
| - |
| -// replicatorProbe reads what the box can observe about the box-wide |
| -// litestream unit. Its three functions are seams; newReplicatorProbe wires |
| -// the real ones. |
| -type replicatorProbe struct { |
| - configPath string |
| - unit string |
| - readFile func(string) ([]byte, error) |
| - modTime func(string) (time.Time, error) |
| - // unitState reports whether the unit is active and since when, and |
| - // the command it runs. |
| - unitState func(ctx context.Context, unit string) (active bool, since time.Time, argv []string, err error) |
| - // resolve follows symlinks. |
| - resolve func(string) (string, error) |
| -} |
| - |
| -func newReplicatorProbe() replicatorProbe { |
| - return replicatorProbe{ |
| - configPath: externalReplicatorConfig, |
| - unit: externalReplicatorUnit, |
| - readFile: os.ReadFile, |
| - modTime: func(path string) (time.Time, error) { |
| - fi, err := os.Stat(path) |
| - if err != nil { |
| - return time.Time{}, err |
| - } |
| - return fi.ModTime(), nil |
| - }, |
| - unitState: systemdUnitState, |
| - resolve: filepath.EvalSymlinks, |
| - } |
| -} |
| - |
| -// read observes the replicator. Anything it cannot establish is an Err, and |
| -// an Err attests nothing (durabilityScope). |
| -// |
| -// The unit and the config's modification time are read before and after |
| -// the config itself. If either moved (a restart or a rewrite landed while |
| -// this was reading), what was read may belong to neither state, so nothing |
| -// is concluded this time. between, when not nil, runs inside that bracket: |
| -// the caller reads the registry there, so the routes it pairs with this |
| -// state are from the same moment (readDurabilityScope). |
| -// |
| -// A config that is not, line for line, what the generator writes is an |
| -// Err: this code will not speak for a hand-kept config, because it cannot |
| -// know it has seen every database in it. A box whose config is still kept |
| -// by hand must be cut over to the generated one before it can attest. |
| -// |
| -// A running unit must be running the command this code expects, the |
| -// replicator reading that one config file and nothing else. A unit given |
| -// databases on its command line, or another config, replicates what this |
| -// code cannot see. |
| -// |
| -// Every configured path is resolved through symlinks, and both the path as |
| -// written and its target are reported. A path that cannot be resolved is |
| -// an Err. |
| -func (p replicatorProbe) read(ctx context.Context, between func() error) replicatorState { |
| - active, since, argv, err := p.unitState(ctx, p.unit) |
| - if err != nil { |
| - return replicatorState{Err: fmt.Errorf("unit %s: %w", p.unit, err)} |
| - } |
| - if active && !expectedReplicatorCommand(argv, p.configPath) { |
| - return replicatorState{Err: fmt.Errorf("unit %s runs %q, not the replicator on %s alone", p.unit, strings.Join(argv, " "), p.configPath)} |
| - } |
| - out := replicatorState{UnitActive: active, UnitStarted: since} |
| - raw, err := p.readFile(p.configPath) |
| - switch { |
| - case errors.Is(err, os.ErrNotExist): |
| - // No config: the external replicator has nothing to replicate. A |
| - // unit that is nevertheless running is replicating from something |
| - // this code cannot see. |
| - if active { |
| - return replicatorState{Err: fmt.Errorf("unit %s is running and %s does not exist", p.unit, p.configPath)} |
| - } |
| - if between != nil { |
| - if err := between(); err != nil { |
| - return replicatorState{Err: err} |
| - } |
| - } |
| - active2, _, _, err := p.unitState(ctx, p.unit) |
| - if err != nil || active2 { |
| - return replicatorState{Err: fmt.Errorf("unit %s changed state while it was being read", p.unit)} |
| - } |
| - if _, err := p.readFile(p.configPath); !errors.Is(err, os.ErrNotExist) { |
| - return replicatorState{Err: fmt.Errorf("%s appeared while it was being read", p.configPath)} |
| - } |
| - return out |
| - case err != nil: |
| - return replicatorState{Err: fmt.Errorf("read %s: %w", p.configPath, err)} |
| - } |
| - modified, err := p.modTime(p.configPath) |
| - if err != nil { |
| - return replicatorState{Err: fmt.Errorf("stat %s: %w", p.configPath, err)} |
| - } |
| - written, err := ops.GeneratedLitestreamDBs(string(raw)) |
| - if err != nil { |
| - return replicatorState{Err: fmt.Errorf("%s: %w", p.configPath, err)} |
| - } |
| - paths := make([]string, 0, len(written)) |
| - for _, path := range written { |
| - paths = append(paths, path) |
| - real, err := p.resolve(path) |
| - switch { |
| - case errors.Is(err, os.ErrNotExist): |
| - // Named and not there: nothing is streamed from it now, and |
| - // the path as written still names its owner. |
| - case err != nil: |
| - return replicatorState{Err: fmt.Errorf("resolve %s: %w", path, err)} |
| - case filepath.Clean(real) != filepath.Clean(path): |
| - paths = append(paths, filepath.Clean(real)) |
| - } |
| - } |
| - if between != nil { |
| - if err := between(); err != nil { |
| - return replicatorState{Err: err} |
| - } |
| - } |
| - // The config, its modification time and the unit again: all must be |
| - // what they were. |
| - again, err := p.readFile(p.configPath) |
| - if err != nil || string(again) != string(raw) { |
| - return replicatorState{Err: fmt.Errorf("%s changed while it was being read", p.configPath)} |
| - } |
| - modified2, err := p.modTime(p.configPath) |
| - if err != nil || !modified2.Equal(modified) { |
| - return replicatorState{Err: fmt.Errorf("%s was rewritten while it was being read", p.configPath)} |
| - } |
| - active2, since2, argv2, err := p.unitState(ctx, p.unit) |
| - if err != nil { |
| - return replicatorState{Err: fmt.Errorf("unit %s: %w", p.unit, err)} |
| - } |
| - if active2 != active || !since2.Equal(since) || strings.Join(argv2, "\x00") != strings.Join(argv, "\x00") { |
| - return replicatorState{Err: fmt.Errorf("unit %s changed state while its config was being read", p.unit)} |
| - } |
| - out.ConfigPresent, out.Paths, out.ConfigModified = true, paths, modified |
| - return out |
| -} |
| - |
| -// expectedReplicatorCommand reports whether argv is the box-wide |
| -// replicator as bootstrap installs it: the litestream binary, replicate, |
| -// and this config file, with no database or replica arguments after it. |
| -func expectedReplicatorCommand(argv []string, config string) bool { |
| - return len(argv) == 4 && filepath.Base(argv[0]) == "litestream" && argv[1] == "replicate" && argv[2] == "-config" && argv[3] == config |
| -} |
| - |
| -// readDurabilityScope reads the routes and the replicator's state as one |
| -// observation and computes the scope from them. The registry is read |
| -// inside the probe's bracket: if the unit restarted or the config was |
| -// rewritten around that read, nothing is attested. |
| -func readDurabilityScope(ctx context.Context, p replicatorProbe, listRoutes func() ([]registry.Route, error), dataDir string) (usage.DurabilityAttestation, replicatorState, string) { |
| - var routes []registry.Route |
| - rs := p.read(ctx, func() error { |
| - var err error |
| - routes, err = listRoutes() |
| - if err != nil { |
| - return fmt.Errorf("list routes: %w", err) |
| - } |
| - return nil |
| - }) |
| - scope, reason := durabilityScope(routes, dataDir, rs, p.resolve) |
| - return scope, rs, reason |
| -} |
| - |
| -// systemdUnitState asks systemd whether unit is active, when it became so, |
| -// and what it runs. The timestamp is requested in unix form: the default |
| -// rendering carries a zone abbreviation, which cannot be parsed back |
| -// reliably. |
| -func systemdUnitState(ctx context.Context, unit string) (bool, time.Time, []string, error) { |
| - out, err := exec.CommandContext(ctx, "systemctl", "show", unit, "--timestamp=unix", "-p", "LoadState", "-p", "ActiveState", "-p", "ActiveEnterTimestamp", "-p", "ExecStart").Output() |
| - if err != nil { |
| - return false, time.Time{}, nil, err |
| - } |
| - return parseSystemdUnitState(string(out)) |
| -} |
| - |
| -// parseSystemdUnitState reads `systemctl show --timestamp=unix` output for |
| -// LoadState, ActiveState, ActiveEnterTimestamp and ExecStart. Only two |
| -// answers are accepted: a unit that is active with a start time and one |
| -// command, and a unit that is plainly not running (inactive or failed, or |
| -// not installed at all). Any state in between (activating, deactivating, |
| -// reloading), a missing property, a timestamp in another form, or a unit |
| -// with no command or several is an error. |
| -func parseSystemdUnitState(out string) (bool, time.Time, []string, error) { |
| - props := map[string]string{} |
| - execStarts := 0 |
| - for _, line := range strings.Split(strings.TrimSpace(out), "\n") { |
| - if k, v, ok := strings.Cut(line, "="); ok { |
| - if k == "ExecStart" { |
| - execStarts++ |
| - } |
| - props[k] = strings.TrimSpace(v) |
| - } |
| - } |
| - load, state := props["LoadState"], props["ActiveState"] |
| - switch { |
| - case load == "not-found" && state == "inactive": |
| - return false, time.Time{}, nil, nil |
| - case load != "loaded": |
| - return false, time.Time{}, nil, fmt.Errorf("load state %q", load) |
| - case state == "inactive", state == "failed": |
| - return false, time.Time{}, nil, nil |
| - case state != "active": |
| - return false, time.Time{}, nil, fmt.Errorf("active state %q", state) |
| - } |
| - stamp, ok := strings.CutPrefix(props["ActiveEnterTimestamp"], "@") |
| - if !ok { |
| - return false, time.Time{}, nil, fmt.Errorf("start time %q is not in unix form", props["ActiveEnterTimestamp"]) |
| - } |
| - secs, err := strconv.ParseInt(stamp, 10, 64) |
| - if err != nil || secs <= 0 { |
| - return false, time.Time{}, nil, fmt.Errorf("start time %q", props["ActiveEnterTimestamp"]) |
| - } |
| - // ExecStart={ path=/usr/local/bin/litestream ; argv[]=/usr/local/bin/litestream replicate -config /etc/litestream.yml ; ignore_errors=no ; ... } |
| - if execStarts != 1 { |
| - return false, time.Time{}, nil, fmt.Errorf("%d ExecStart commands", execStarts) |
| - } |
| - _, rest, ok := strings.Cut(props["ExecStart"], "argv[]=") |
| - if !ok { |
| - return false, time.Time{}, nil, fmt.Errorf("ExecStart %q has no argv", props["ExecStart"]) |
| - } |
| - args, _, ok := strings.Cut(rest, " ;") |
| - if !ok { |
| - return false, time.Time{}, nil, fmt.Errorf("ExecStart %q is not in the expected form", props["ExecStart"]) |
| - } |
| - argv := strings.Fields(args) |
| - if len(argv) == 0 { |
| - return false, time.Time{}, nil, fmt.Errorf("ExecStart %q has an empty argv", props["ExecStart"]) |
| - } |
| - return true, time.Unix(secs, 0).UTC(), argv, nil |
| -} |
diff --git a/internal/host/durabilityreplicator_test.go b/internal/host/durabilityreplicator_test.go| deleted file mode 100644 |
| index 81fd3c0c6..000000000 |
| --- a/internal/host/durabilityreplicator_test.go |
| +++ /dev/null |
| @@ -1,214 +0,0 @@ |
| -package host |
| - |
| -import ( |
| - "context" |
| - "errors" |
| - "os" |
| - "reflect" |
| - "testing" |
| - "time" |
| - |
| - "github.com/carlosframework/platform/internal/ops" |
| - "github.com/carlosframework/platform/internal/registry" |
| -) |
| - |
| -const unitExec = "ExecStart={ path=/usr/local/bin/litestream ; argv[]=/usr/local/bin/litestream replicate -config /etc/litestream.yml ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }\n" |
| - |
| -func TestParseSystemdUnitState(t *testing.T) { |
| - cases := []struct { |
| - name string |
| - out string |
| - active bool |
| - since int64 |
| - fails bool |
| - }{ |
| - {"running", "LoadState=loaded\nActiveState=active\nActiveEnterTimestamp=@1790817621\n" + unitExec, true, 1790817621, false}, |
| - {"stopped", "LoadState=loaded\nActiveState=inactive\nActiveEnterTimestamp=\n" + unitExec, false, 0, false}, |
| - {"failed", "LoadState=loaded\nActiveState=failed\nActiveEnterTimestamp=@1790817621\n" + unitExec, false, 0, false}, |
| - {"not installed", "LoadState=not-found\nActiveState=inactive\nActiveEnterTimestamp=\n", false, 0, false}, |
| - {"starting: neither running nor stopped", "LoadState=loaded\nActiveState=activating\nActiveEnterTimestamp=\n" + unitExec, false, 0, true}, |
| - {"stopping", "LoadState=loaded\nActiveState=deactivating\nActiveEnterTimestamp=@1790817621\n" + unitExec, false, 0, true}, |
| - {"a start time with a zone abbreviation", "LoadState=loaded\nActiveState=active\nActiveEnterTimestamp=Thu 2026-10-01 01:40:21 IST\n" + unitExec, false, 0, true}, |
| - {"running with no start time", "LoadState=loaded\nActiveState=active\nActiveEnterTimestamp=\n" + unitExec, false, 0, true}, |
| - {"running with no command", "LoadState=loaded\nActiveState=active\nActiveEnterTimestamp=@1790817621\n", false, 0, true}, |
| - {"running with two commands", "LoadState=loaded\nActiveState=active\nActiveEnterTimestamp=@1790817621\n" + unitExec + unitExec, false, 0, true}, |
| - {"masked", "LoadState=masked\nActiveState=inactive\nActiveEnterTimestamp=\n", false, 0, true}, |
| - {"nothing at all", "", false, 0, true}, |
| - } |
| - for _, c := range cases { |
| - active, since, argv, err := parseSystemdUnitState(c.out) |
| - if (err != nil) != c.fails || active != c.active || (c.since != 0 && since.Unix() != c.since) { |
| - t.Errorf("%s: active=%v since=%v err=%v", c.name, active, since, err) |
| - } |
| - if active && !expectedReplicatorCommand(argv, "/etc/litestream.yml") { |
| - t.Errorf("%s: argv %q is not recognised as the replicator", c.name, argv) |
| - } |
| - } |
| - for name, argv := range map[string][]string{ |
| - "a database on the command line": {"/usr/local/bin/litestream", "replicate", "/data/x.db", "s3://bucket/x"}, |
| - "another config": {"/usr/local/bin/litestream", "replicate", "-config", "/etc/other.yml"}, |
| - "an extra argument": {"/usr/local/bin/litestream", "replicate", "-config", "/etc/litestream.yml", "-exec", "x"}, |
| - "another binary": {"/usr/local/bin/rsync", "replicate", "-config", "/etc/litestream.yml"}, |
| - "nothing": nil, |
| - } { |
| - if expectedReplicatorCommand(argv, "/etc/litestream.yml") { |
| - t.Errorf("%s: accepted as the replicator", name) |
| - } |
| - } |
| -} |
| - |
| -func TestReplicatorProbe(t *testing.T) { |
| - ctx := context.Background() |
| - written := time.Date(2026, 9, 17, 8, 0, 0, 0, time.UTC) |
| - argv := []string{"/usr/local/bin/litestream", "replicate", "-config", "/etc/litestream.yml"} |
| - generated := ops.LitestreamConfig([]string{"/data/registry.db", "/data/api.example.com.db", "/var/lib/carlos-app/kass.example.com/instances/y.db"}, "bucket", "prefix", "eu-west-1") |
| - probe := func(cfg *string, active bool, since time.Time) replicatorProbe { |
| - return replicatorProbe{configPath: "/etc/litestream.yml", unit: externalReplicatorUnit, |
| - readFile: func(string) ([]byte, error) { |
| - if cfg == nil { |
| - return nil, os.ErrNotExist |
| - } |
| - return []byte(*cfg), nil |
| - }, |
| - modTime: func(string) (time.Time, error) { return written, nil }, |
| - unitState: func(context.Context, string) (bool, time.Time, []string, error) { |
| - if !active { |
| - return false, time.Time{}, nil, nil |
| - } |
| - return active, since, argv, nil |
| - }, |
| - // One link: a file in the unit's tree that points at another |
| - // tenant's database. |
| - resolve: func(path string) (string, error) { |
| - if path == "/var/lib/carlos-app/kass.example.com/instances/y.db" { |
| - return "/data/y.example.com.db", nil |
| - } |
| - return path, nil |
| - }, |
| - } |
| - } |
| - // The ordinary box: a generated config, the unit restarted after it. |
| - // The link is reported as written and as resolved. |
| - got := probe(&generated, true, written.Add(time.Minute)).read(ctx, nil) |
| - if got.Err != nil || !got.ConfigPresent || !got.UnitActive || !got.ConfigModified.Equal(written) || |
| - !reflect.DeepEqual(got.Paths, []string{"/data/api.example.com.db", "/data/registry.db", "/var/lib/carlos-app/kass.example.com/instances/y.db", "/data/y.example.com.db"}) { |
| - t.Fatalf("generated config: %+v", got) |
| - } |
| - // No config and no unit: nothing external replicates. |
| - if got := probe(nil, false, time.Time{}).read(ctx, nil); got.Err != nil || got.ConfigPresent { |
| - t.Fatalf("no config, no unit: %+v", got) |
| - } |
| - // A running unit with no config is replicating from somewhere unseen. |
| - if got := probe(nil, true, written).read(ctx, nil); got.Err == nil { |
| - t.Fatal("a running unit with no config was accepted") |
| - } |
| - // A hand-kept config is refused whole. |
| - hand := "dbs:\n - path: /data/registry.db\n replicas:\n - url: s3://bucket/x\n" |
| - if got := probe(&hand, true, written.Add(time.Minute)).read(ctx, nil); !errors.Is(got.Err, ops.ErrConfigNotGenerated) || len(got.Paths) != 0 { |
| - t.Fatalf("a hand-kept config: %+v", got) |
| - } |
| - // A unit given a database on its command line. |
| - args := probe(&generated, true, written.Add(time.Minute)) |
| - args.unitState = func(context.Context, string) (bool, time.Time, []string, error) { |
| - return true, written.Add(time.Minute), []string{"/usr/local/bin/litestream", "replicate", "/data/x.db", "s3://bucket/x"}, nil |
| - } |
| - if got := args.read(ctx, nil); got.Err == nil { |
| - t.Fatal("a unit replicating from its command line was accepted") |
| - } |
| - // A path that cannot be resolved. |
| - blind := probe(&generated, true, written.Add(time.Minute)) |
| - blind.resolve = func(string) (string, error) { return "", errors.New("permission denied") } |
| - if got := blind.read(ctx, nil); got.Err == nil { |
| - t.Fatal("a config path that could not be resolved was accepted") |
| - } |
| - // The unit restarts while the config is being read. |
| - calls := 0 |
| - racing := probe(&generated, true, written.Add(time.Minute)) |
| - racing.unitState = func(context.Context, string) (bool, time.Time, []string, error) { |
| - calls++ |
| - return true, written.Add(time.Duration(calls) * time.Minute), argv, nil |
| - } |
| - if got := racing.read(ctx, nil); got.Err == nil { |
| - t.Fatal("a unit that restarted mid-read was accepted") |
| - } |
| - // The config is rewritten while it is being read: its bytes, or only |
| - // its modification time. |
| - reads := 0 |
| - rewriting := probe(&generated, true, written.Add(time.Minute)) |
| - rewriting.readFile = func(string) ([]byte, error) { |
| - reads++ |
| - if reads > 1 { |
| - return []byte(generated + " - path: /data/new.db\n"), nil |
| - } |
| - return []byte(generated), nil |
| - } |
| - if got := rewriting.read(ctx, nil); got.Err == nil { |
| - t.Fatal("a config rewritten mid-read was accepted") |
| - } |
| - stats := 0 |
| - touched := probe(&generated, true, written.Add(time.Minute)) |
| - touched.modTime = func(string) (time.Time, error) { |
| - stats++ |
| - return written.Add(time.Duration(stats) * time.Second), nil |
| - } |
| - if got := touched.read(ctx, nil); got.Err == nil { |
| - t.Fatal("a config whose modification time moved mid-read was accepted") |
| - } |
| - // systemd cannot be asked. |
| - deaf := probe(&generated, true, written) |
| - deaf.unitState = func(context.Context, string) (bool, time.Time, []string, error) { |
| - return false, time.Time{}, nil, errors.New("systemctl: not found") |
| - } |
| - if got := deaf.read(ctx, nil); got.Err == nil { |
| - t.Fatal("an unreadable unit state was accepted") |
| - } |
| -} |
| - |
| -// The routes and the replicator's state are one observation. A route flips |
| -// from hibernating to always-on, the config is regenerated and the unit |
| -// restarts, all while the scope is being read: the registry read falls |
| -// inside the bracket, the unit is seen to have moved, and nothing is |
| -// attested. And the link in the unit's tree excludes the owner of the |
| -// database it points at. |
| -func TestReadDurabilityScopeIsOneObservation(t *testing.T) { |
| - ctx := context.Background() |
| - written := time.Date(2026, 9, 17, 8, 0, 0, 0, time.UTC) |
| - argv := []string{"/usr/local/bin/litestream", "replicate", "-config", "/etc/litestream.yml"} |
| - routes := []registry.Route{ |
| - {Host: "kass.example.com", Account: "bae", Backing: registry.BackingUnit, DBPath: "/var/lib/carlos-app/kass.example.com"}, |
| - {Host: "y.example.com", Account: "bac", Hibernate: true, DBPath: "/data/y.example.com.db"}, |
| - {Host: "shop.example.com", Account: "bad", Hibernate: true, DBPath: "/data/shop.example.com.db"}, |
| - } |
| - cfg := ops.LitestreamConfig([]string{"/data/registry.db", "/var/lib/carlos-app/kass.example.com/instances/y.db"}, "bucket", "prefix", "eu-west-1") |
| - restarts := 0 |
| - p := replicatorProbe{configPath: "/etc/litestream.yml", unit: externalReplicatorUnit, |
| - readFile: func(string) ([]byte, error) { return []byte(cfg), nil }, |
| - modTime: func(string) (time.Time, error) { return written, nil }, |
| - unitState: func(context.Context, string) (bool, time.Time, []string, error) { |
| - return true, written.Add(time.Duration(1+restarts) * time.Minute), argv, nil |
| - }, |
| - resolve: func(path string) (string, error) { |
| - if path == "/var/lib/carlos-app/kass.example.com/instances/y.db" { |
| - return "/data/y.example.com.db", nil |
| - } |
| - return path, nil |
| - }, |
| - } |
| - scope, _, reason := readDurabilityScope(ctx, p, func() ([]registry.Route, error) { return routes, nil }, "/data") |
| - if !scope.Attested || !reflect.DeepEqual(scope.Unmetered, []string{"bac", "bae"}) { |
| - t.Fatalf("scope %+v (%s): want the unit's owner and the owner of the database its link points at excluded, and bad vouched for", scope, reason) |
| - } |
| - // The unit restarts while the registry is being read. |
| - scope, _, reason = readDurabilityScope(ctx, p, func() ([]registry.Route, error) { |
| - restarts++ |
| - return routes, nil |
| - }, "/data") |
| - if scope.Attested || reason == "" { |
| - t.Fatalf("a restart around the registry read attested: %+v", scope) |
| - } |
| - // The registry cannot be read. |
| - scope, _, _ = readDurabilityScope(ctx, p, func() ([]registry.Route, error) { return nil, errors.New("registry locked") }, "/data") |
| - if scope.Attested { |
| - t.Fatal("an unreadable registry attested") |
| - } |
| -} |
diff --git a/internal/host/durabilityscope.go b/internal/host/durabilityscope.go| deleted file mode 100644 |
| index b2902eb2a..000000000 |
| --- a/internal/host/durabilityscope.go |
| +++ /dev/null |
| @@ -1,269 +0,0 @@ |
| -package host |
| - |
| -import ( |
| - "context" |
| - "fmt" |
| - "log" |
| - "path/filepath" |
| - "sort" |
| - "strings" |
| - "time" |
| - |
| - "github.com/carlosframework/platform/internal/ops" |
| - "github.com/carlosframework/platform/internal/registry" |
| - "github.com/carlosframework/platform/internal/release" |
| - "github.com/carlosframework/platform/internal/usage" |
| -) |
| - |
| -// The scoped attestation of the replication ledger: what this box may |
| -// honestly claim about who replicates through the ledger and who does not |
| -// (docs/superpowers/plans/2026-10-01-durability-attestation-path.md, path |
| -// A). The agent calls this only when CARLOS_DURABILITY_ATTEST=scoped is |
| -// set in its environment (scopedAttestationEnabled). It is unset |
| -// everywhere, and setting it on a production box needs a scoped approval. |
| -// |
| -// A box has two replicators. The agent's own (the activator's embedded |
| -// Litestream, the move worker) records every request in the ledger. The |
| -// box-wide litestream unit, an external binary run as root from |
| -// /etc/litestream.yml, records nothing: it replicates every database that |
| -// is not a hibernating exec route's (ops.FleetDBs). An account can be |
| -// vouched for only when the external replicator touches none of its |
| -// databases, so the attestation names the accounts it does touch and |
| -// excludes them. |
| - |
| -// replicatorState is what the box can observe about the external |
| -// replicator. |
| -type replicatorState struct { |
| - // ConfigPresent is false when there is no config file at all: the |
| - // external replicator has nothing to replicate. |
| - ConfigPresent bool |
| - // Paths are the database paths the config names, each as written and, |
| - // where that differs, as resolved through symlinks. |
| - Paths []string |
| - // ConfigModified is the config file's modification time. |
| - ConfigModified time.Time |
| - // UnitActive and UnitStarted describe the unit that runs the external |
| - // replicator: whether it is running, and since when. |
| - UnitActive bool |
| - UnitStarted time.Time |
| - // Err is set when any of the above could not be read. |
| - Err error |
| -} |
| - |
| -// durabilityScope computes the attestation for this box from its routes |
| -// and the external replicator's observed state. reason says why nothing is |
| -// attested, when nothing is. |
| -// |
| -// An account is excluded when either says the external replicator may |
| -// touch its databases: |
| -// |
| -// - the registry: it owns a unit-backed route (whose database only the |
| -// box-wide replicator can reach, and which may sit in a hand-kept |
| -// config even when the route records no path), or an exec route that |
| -// records a database and does not hibernate (which the config |
| -// generator hands to the box-wide replicator). A route with no |
| -// database and no unit, a static site for one, excludes nobody; |
| -// - the config: it owns a route whose database path the live config |
| -// names, whatever kind of route the registry says it is. EVERY route |
| -// whose path equals or encloses a configured path is an owner, not the |
| -// first one found: a hibernating route's database can sit inside a |
| -// unit's tree, and two routes can record one file. |
| -// |
| -// Paths are compared as written and as resolved. resolve follows symlinks |
| -// (filepath.EvalSymlinks on a real box). A unit tenant controls its own |
| -// state directory and can link a file in it to another tenant's database; |
| -// the generator, which follows links, then names the link, and only the |
| -// resolved path shows whose database the replicator is streaming. The |
| -// caller resolves the config's paths (replicatorState.Paths); this |
| -// function resolves the routes'. A route path that does not resolve (the |
| -// database does not exist yet) is matched as written. |
| -// |
| -// Nothing is attested, for anyone, when the box cannot tell what the |
| -// external replicator is doing: |
| -// |
| -// - its state could not be read; |
| -// - the unit is running and there is no config file: it is replicating |
| -// from a config that has been removed since it started, or from |
| -// arguments, and neither can be seen; |
| -// - the config names a database no route owns and that is not the box's |
| -// own (the registry or the console database): somebody's data is being |
| -// replicated and nobody can be named as its owner; |
| -// - the unit has been running since before the config was last written: |
| -// it is replicating from an older config this code cannot see, which |
| -// may name databases the current one dropped. |
| -// |
| -// The box's own databases are platform data on the platform's prefix, not |
| -// tenant replication, and exclude nobody. |
| -// |
| -// What the caller owes: Paths must be every database the config names |
| -// (replicatorProbe refuses a config it cannot fully account for), the |
| -// routes and the replicator's state must be one coherent observation |
| -// (readDurabilityScope brackets them). The console database's path is |
| -// resolved the way the generator resolves it (ops.ConsoleDBPath). |
| -func durabilityScope(routes []registry.Route, dataDir string, rs replicatorState, resolve func(string) (string, error)) (usage.DurabilityAttestation, string) { |
| - if rs.Err != nil { |
| - return usage.DurabilityAttestation{}, fmt.Sprintf("the external replicator's state could not be read: %v", rs.Err) |
| - } |
| - if rs.UnitActive && !rs.ConfigPresent { |
| - return usage.DurabilityAttestation{}, "the external replicator is running and has no config file; what it replicates cannot be seen" |
| - } |
| - if rs.ConfigPresent && rs.UnitActive && rs.UnitStarted.Before(rs.ConfigModified) { |
| - return usage.DurabilityAttestation{}, "the external replicator has been running since before its config was last written; what it replicates is not what the config says" |
| - } |
| - excluded := map[string]bool{} |
| - type owned struct { |
| - path string |
| - tree bool |
| - account string |
| - } |
| - var owners []owned |
| - for _, rt := range routes { |
| - acct := release.RouteAccount(rt) |
| - if rt.Backing == registry.BackingUnit || (rt.DBPath != "" && !rt.Hibernate) { |
| - excluded[acct] = true |
| - } |
| - if rt.DBPath == "" { |
| - continue |
| - } |
| - tree := rt.Backing == registry.BackingUnit |
| - written := filepath.Clean(rt.DBPath) |
| - owners = append(owners, owned{path: written, tree: tree, account: acct}) |
| - if resolve != nil { |
| - if real, err := resolve(written); err == nil && filepath.Clean(real) != written { |
| - owners = append(owners, owned{path: filepath.Clean(real), tree: tree, account: acct}) |
| - } |
| - } |
| - } |
| - own := map[string]bool{ |
| - filepath.Clean(filepath.Join(dataDir, "registry.db")): true, |
| - filepath.Clean(ops.ConsoleDBPath(dataDir)): true, |
| - } |
| - if rs.ConfigPresent { |
| - for _, raw := range rs.Paths { |
| - path := filepath.Clean(raw) |
| - if !filepath.IsAbs(path) { |
| - return usage.DurabilityAttestation{}, fmt.Sprintf("the external replicator's config names %s, which is not an absolute path", raw) |
| - } |
| - if own[path] { |
| - continue |
| - } |
| - matched := false |
| - for _, o := range owners { |
| - // A unit route's path may be a directory the generator |
| - // walks for databases (ops.treeDBs). |
| - if path == o.path || (o.tree && strings.HasPrefix(path, o.path+string(filepath.Separator))) { |
| - excluded[o.account] = true |
| - matched = true |
| - } |
| - } |
| - if !matched { |
| - return usage.DurabilityAttestation{}, fmt.Sprintf("the external replicator replicates %s, which no route owns", path) |
| - } |
| - } |
| - } |
| - out := usage.DurabilityAttestation{Attested: true} |
| - for acct := range excluded { |
| - out.Unmetered = append(out.Unmetered, acct) |
| - } |
| - sort.Strings(out.Unmetered) |
| - return out, "" |
| -} |
| - |
| -// durabilityAttestEnv is the agent's switch for the scoped attestation. |
| -// Unset, or set to anything but "scoped", the ledger attests nothing, as |
| -// it always has. |
| -const durabilityAttestEnv = "CARLOS_DURABILITY_ATTEST" |
| - |
| -func scopedAttestationEnabled(getenv func(string) string) bool { |
| - return strings.TrimSpace(getenv(durabilityAttestEnv)) == "scoped" |
| -} |
| - |
| -// enableScopedAttestation makes every publish tick observe the box and |
| -// apply the scope BEFORE it seals anything, so a day is never closed on a |
| -// claim that was not checked in the same tick. |
| -func (p *durabilityPublisher) enableScopedAttestation(probe replicatorProbe, listRoutes func() ([]registry.Route, error), dataDir string) { |
| - if p == nil { |
| - return |
| - } |
| - lastReason := "\x00" |
| - p.attest = func(ctx context.Context) { |
| - scope, rs, reason := readDurabilityScope(ctx, probe, listRoutes, dataDir) |
| - if err := p.applyScope(scope, rs, p.now()); err != nil { |
| - log.Printf("agent: durability: attestation: %v", err) |
| - } |
| - if reason != lastReason { |
| - if reason != "" { |
| - log.Printf("agent: durability: attesting nothing: %s", reason) |
| - } else { |
| - log.Printf("agent: durability: attesting, with %d account(s) excluded", len(scope.Unmetered)) |
| - } |
| - lastReason = reason |
| - } |
| - } |
| -} |
| - |
| -// applyScope moves the ledger's claim to scope, dating every change at the |
| -// earliest instant the previous claim could have stopped being true. |
| -// |
| -// - Nothing was attested and scope attests: the claim starts now. It |
| -// says nothing about the time before this observation. |
| -// - Scope excludes an account the ledger was vouching for: the exclusion |
| -// starts no later than the last tick that verified the old claim, and |
| -// no later than the instant the replicator's config was last written |
| -// (the replicator cannot touch a database before a config naming it |
| -// existed). That keeps a day from closing complete for an account the |
| -// external replicator began touching just before midnight. |
| -// - Scope attests nothing and the ledger was attesting: the claim ends |
| -// at the last tick that verified it. |
| -// |
| -// An exclusion never lapses within a run: the ledger keeps the union. |
| -// |
| -// When the ledger refuses the instant (it precedes the current coverage |
| -// session), the change is filed as an escape instead, which the ledger |
| -// clamps itself and which also makes the day of the escape incomplete for |
| -// everyone on the box: blunter, and never more generous. |
| -func (p *durabilityPublisher) applyScope(scope usage.DurabilityAttestation, rs replicatorState, now time.Time) error { |
| - cur := p.ledger.Attestation() |
| - verified := p.scopeVerifiedAt |
| - if verified.IsZero() || verified.After(now) { |
| - verified = now |
| - } |
| - defer func() { p.scopeVerifiedAt = now }() |
| - if !scope.Attested { |
| - if !cur.Attested { |
| - return nil |
| - } |
| - if err := p.ledger.SetAttestation(usage.DurabilityAttestation{}, verified); err != nil { |
| - // An owner that cannot be named withdraws the claim for |
| - // everyone, from the instant given. |
| - p.ledger.Escaped("", verified) |
| - } |
| - return nil |
| - } |
| - if !cur.Attested { |
| - return p.ledger.SetAttestation(scope, now) |
| - } |
| - had := map[string]bool{} |
| - for _, acct := range cur.Unmetered { |
| - had[acct] = true |
| - } |
| - var added []string |
| - for _, acct := range scope.Unmetered { |
| - if !had[acct] { |
| - added = append(added, acct) |
| - } |
| - } |
| - if len(added) == 0 { |
| - return nil |
| - } |
| - from := verified |
| - if rs.ConfigPresent && !rs.ConfigModified.IsZero() && rs.ConfigModified.Before(from) { |
| - from = rs.ConfigModified |
| - } |
| - if err := p.ledger.SetAttestation(scope, from); err != nil { |
| - for _, acct := range added { |
| - p.ledger.Escaped(acct, from) |
| - } |
| - } |
| - return nil |
| -} |
diff --git a/internal/host/durabilityscope_test.go b/internal/host/durabilityscope_test.go| deleted file mode 100644 |
| index d7070eb66..000000000 |
| --- a/internal/host/durabilityscope_test.go |
| +++ /dev/null |
| @@ -1,365 +0,0 @@ |
| -package host |
| - |
| -import ( |
| - "context" |
| - "errors" |
| - "reflect" |
| - "testing" |
| - "time" |
| - |
| - "github.com/carlosframework/platform/internal/ops" |
| - "github.com/carlosframework/platform/internal/registry" |
| - "github.com/carlosframework/platform/internal/release" |
| - "github.com/carlosframework/platform/internal/usage" |
| -) |
| - |
| -func scopeRoutes() []registry.Route { |
| - return []registry.Route{ |
| - // A hibernating exec route: the activator replicates it, metered. |
| - {Host: "shop.example.com", Account: "bac", App: "shop", Hibernate: true, DBPath: "/data/shop.example.com.db"}, |
| - // An always-on exec route with a database: the box-wide replicator's. |
| - {Host: "api.example.com", Account: "bad", App: "api", DBPath: "/data/api.example.com.db"}, |
| - // A unit-backed route whose path is a tree. |
| - {Host: "kass.example.com", Account: "bae", App: "kass", Backing: registry.BackingUnit, Hibernate: true, DBPath: "/var/lib/carlos-app/kass.example.com"}, |
| - // A static site: no process, no database. |
| - {Host: "docs.example.com", Account: "baf", App: "docs"}, |
| - } |
| -} |
| - |
| -func scopeState(paths ...string) replicatorState { |
| - written := time.Date(2026, 9, 17, 8, 0, 0, 0, time.UTC) |
| - return replicatorState{ConfigPresent: true, Paths: paths, ConfigModified: written, UnitActive: true, UnitStarted: written.Add(time.Minute)} |
| -} |
| - |
| -// testResolve stands in for the filesystem: one link, everything else as |
| -// written. |
| -func testResolve(path string) (string, error) { |
| - if path == "/data/link.db" { |
| - return "/data/real.db", nil |
| - } |
| - return path, nil |
| -} |
| - |
| -func TestDurabilityScope(t *testing.T) { |
| - generated := []string{"/data/registry.db", "/data/console/console.db", "/data/api.example.com.db", "/var/lib/carlos-app/kass.example.com/registry.db", "/var/lib/carlos-app/kass.example.com/instances/a.db"} |
| - cases := []struct { |
| - name string |
| - routes []registry.Route |
| - state replicatorState |
| - attested bool |
| - unmetered []string |
| - }{ |
| - {"the generated config: always-on and unit owners are excluded, hibernating and static owners are vouched for", |
| - scopeRoutes(), scopeState(generated...), true, []string{"bad", "bae"}}, |
| - {"no config and no running unit: the registry alone decides", |
| - scopeRoutes(), replicatorState{}, true, []string{"bad", "bae"}}, |
| - {"a running unit with no config file: nothing is attested", |
| - scopeRoutes()[:1], replicatorState{UnitActive: true, UnitStarted: time.Date(2026, 9, 17, 8, 0, 0, 0, time.UTC)}, false, nil}, |
| - {"a hibernating route's database inside a unit's tree: both owners are excluded, in either route order", |
| - []registry.Route{ |
| - {Host: "kass.example.com", Account: "bae", Backing: registry.BackingUnit, DBPath: "/srv/unit"}, |
| - {Host: "a.example.com", Account: "bac", Hibernate: true, DBPath: "/srv/unit/a.db"}, |
| - }, scopeState("/srv/unit/a.db"), true, []string{"bac", "bae"}}, |
| - {"the same, routes reversed", |
| - []registry.Route{ |
| - {Host: "a.example.com", Account: "bac", Hibernate: true, DBPath: "/srv/unit/a.db"}, |
| - {Host: "kass.example.com", Account: "bae", Backing: registry.BackingUnit, DBPath: "/srv/unit"}, |
| - }, scopeState("/srv/unit/a.db"), true, []string{"bac", "bae"}}, |
| - {"two routes record one database: both owners are excluded", |
| - []registry.Route{ |
| - {Host: "x.example.com", Account: "bac", Hibernate: true, DBPath: "/data/x.db"}, |
| - {Host: "y.example.com", Account: "bad", Hibernate: true, DBPath: "/data/x.db"}, |
| - }, scopeState("/data/x.db"), true, []string{"bac", "bad"}}, |
| - {"a link in a unit's tree to a hibernating route's database, reported as written and as resolved: both owners are excluded", |
| - []registry.Route{ |
| - {Host: "kass.example.com", Account: "bae", Backing: registry.BackingUnit, DBPath: "/srv/unit"}, |
| - {Host: "y.example.com", Account: "bac", Hibernate: true, DBPath: "/data/y.example.com.db"}, |
| - }, scopeState("/srv/unit/instances/y.db", "/data/y.example.com.db"), true, []string{"bac", "bae"}}, |
| - {"a route whose own recorded path is a link: the config's resolved path still finds it", |
| - []registry.Route{{Host: "z.example.com", Account: "bac", Hibernate: true, DBPath: "/data/link.db"}}, |
| - scopeState("/data/real.db"), true, []string{"bac"}}, |
| - {"a relative path in the config: nothing is attested", |
| - scopeRoutes(), scopeState("data/x.db"), false, nil}, |
| - {"only hibernating exec routes: nobody is excluded", |
| - scopeRoutes()[:1], scopeState("/data/registry.db"), true, nil}, |
| - {"the config names a hibernating route's database: its owner is excluded on that evidence", |
| - scopeRoutes(), scopeState(append([]string{"/data/shop.example.com.db"}, generated...)...), true, []string{"bac", "bad", "bae"}}, |
| - {"the config names a database no route owns: nothing is attested", |
| - scopeRoutes(), scopeState(append([]string{"/data/leftover.db"}, generated...)...), false, nil}, |
| - {"a path that only shares a prefix with a unit's tree is not in the tree", |
| - scopeRoutes(), scopeState("/var/lib/carlos-app/kass.example.com.bak/x.db"), false, nil}, |
| - {"the unit has been running since before the config was written: nothing is attested", |
| - scopeRoutes(), func() replicatorState { |
| - s := scopeState(generated...) |
| - s.UnitStarted = s.ConfigModified.Add(-time.Hour) |
| - return s |
| - }(), false, nil}, |
| - {"the replicator's state could not be read: nothing is attested", |
| - scopeRoutes(), replicatorState{Err: errors.New("permission denied")}, false, nil}, |
| - {"a route with no valid account is the platform's: the platform account is excluded", |
| - []registry.Route{{Host: "console.example.com", DBPath: "/data/other.db"}}, scopeState("/data/other.db"), true, []string{release.OpsAccount}}, |
| - {"a unit route with no recorded path still excludes its owner", |
| - []registry.Route{{Host: "kass.example.com", Account: "bae", Backing: registry.BackingUnit}}, replicatorState{}, true, []string{"bae"}}, |
| - } |
| - for _, c := range cases { |
| - t.Run(c.name, func(t *testing.T) { |
| - got, reason := durabilityScope(c.routes, "/data", c.state, testResolve) |
| - if got.Attested != c.attested || !reflect.DeepEqual(got.Unmetered, c.unmetered) { |
| - t.Fatalf("attested=%v unmetered=%v (reason %q), want attested=%v unmetered=%v", got.Attested, got.Unmetered, reason, c.attested, c.unmetered) |
| - } |
| - if !got.Attested && (reason == "" || len(got.Unmetered) != 0) { |
| - t.Fatalf("an unattested scope must say why and exclude nothing: %+v %q", got, reason) |
| - } |
| - }) |
| - } |
| -} |
| - |
| -// The test the attestation plan requires before any box may attest: a box |
| -// with a unit-backed route and the external replicator present publishes a |
| -// day that is NOT complete for that route's owner, and is complete for an |
| -// account whose only route hibernates under the activator. |
| -func TestScopedAttestationPublishesADayIncompleteForTheUnitOwner(t *testing.T) { |
| - // The box came up the evening before, so the 17th is wholly inside |
| - // this run. |
| - at := time.Date(2026, 9, 16, 23, 0, 0, 0, time.UTC) |
| - clock := at |
| - p, err := newDurabilityPublisher(t.TempDir(), "box1", func() time.Time { return clock }) |
| - if err != nil { |
| - t.Fatal(err) |
| - } |
| - t.Cleanup(func() { |
| - ctx, cancel := context.WithTimeout(context.Background(), time.Second) |
| - defer cancel() |
| - if !p.stopped.Load() { |
| - _ = p.stop(ctx, nil) |
| - } |
| - _ = p.ledger.WaitLate(ctx) |
| - }) |
| - // As opened in production the ledger attests nothing. |
| - if p.ledger.Attestation().Attested { |
| - t.Fatal("a freshly opened production ledger attests") |
| - } |
| - scope, reason := durabilityScope(scopeRoutes(), "/data", scopeState("/data/registry.db", "/data/api.example.com.db", "/var/lib/carlos-app/kass.example.com/registry.db"), testResolve) |
| - if !scope.Attested { |
| - t.Fatalf("fixture scope: %s", reason) |
| - } |
| - if err := p.ledger.SetAttestation(scope, at); err != nil { |
| - t.Fatal(err) |
| - } |
| - clock = time.Date(2026, 9, 17, 10, 0, 0, 0, time.UTC) |
| - rec := p.recorder(registry.Route{Account: "bac", App: "shop", Host: "shop.example.com"}, usage.PurposeReplicate) |
| - if rec == nil { |
| - t.Fatal("no recorder for the hibernating route") |
| - } |
| - clock = time.Date(2026, 9, 18, 0, 1, 0, 0, time.UTC) // past midnight: the day can close |
| - store := release.NewDirStore(t.TempDir()) |
| - if err := p.publish(context.Background(), store); err != nil { |
| - t.Fatal(err) |
| - } |
| - raw, _, err := store.Get(context.Background(), usage.DurabilityDayKey("box1", "2026-09-17")) |
| - if err != nil { |
| - t.Fatal(err) |
| - } |
| - day, err := usage.ParseDurabilityDay(raw, "box1", "2026-09-17") |
| - if err != nil { |
| - t.Fatal(err) |
| - } |
| - from, to := time.Date(2026, 9, 17, 0, 0, 0, 0, time.UTC), time.Date(2026, 9, 18, 0, 0, 0, 0, time.UTC) |
| - ok, unmetered, why := day.Scope(from, to) |
| - if !ok { |
| - t.Fatalf("the day proves nothing for anyone: %s", why) |
| - } |
| - if !reflect.DeepEqual(unmetered, []string{"bad", "bae"}) { |
| - t.Fatalf("excluded %v, want the always-on owner and the unit owner", unmetered) |
| - } |
| - for _, snap := range day.Epochs { |
| - c, err := snap.Decode() |
| - if err != nil { |
| - t.Fatal(err) |
| - } |
| - if complete, _ := c.CompletenessFor("bae", from, to); complete { |
| - t.Fatal("the day reads complete for the unit-backed route's owner") |
| - } |
| - if complete, why := c.CompletenessFor("bac", from, to); !complete { |
| - t.Fatalf("the day does not read complete for the hibernating route's owner: %s", why) |
| - } |
| - } |
| -} |
| - |
| -// The switch. Only the exact value turns the attestation on. |
| -func TestScopedAttestationIsOffUnlessAskedForByName(t *testing.T) { |
| - for value, want := range map[string]bool{"": false, "scoped": true, " scoped ": true, "true": false, "1": false, "all": false, "SCOPED": false} { |
| - if got := scopedAttestationEnabled(func(string) string { return value }); got != want { |
| - t.Errorf("%s=%q: enabled=%v, want %v", durabilityAttestEnv, value, got, want) |
| - } |
| - } |
| - // A publisher nobody enabled attests nothing, tick after tick. |
| - p, _, _ := newTestDurabilityPublisher(t) |
| - store := release.NewDirStore(t.TempDir()) |
| - for i := 0; i < 2; i++ { |
| - if err := p.publish(context.Background(), store); err != nil { |
| - t.Fatal(err) |
| - } |
| - } |
| - if p.attest != nil || p.ledger.Attestation().Attested { |
| - t.Fatal("a publisher with the switch off attests") |
| - } |
| -} |
| - |
| -// attestFixture is a publisher with the scoped attestation enabled over a |
| -// box whose routes, replicator config and unit the test controls. |
| -type attestFixture struct { |
| - p *durabilityPublisher |
| - clock time.Time |
| - routes []registry.Route |
| - paths []string |
| - written time.Time |
| - started time.Time |
| - stateErr error |
| - store release.Store |
| -} |
| - |
| -func newAttestFixture(t *testing.T) *attestFixture { |
| - t.Helper() |
| - f := &attestFixture{clock: time.Date(2026, 9, 16, 23, 0, 0, 0, time.UTC), store: release.NewDirStore(t.TempDir())} |
| - f.written = f.clock.Add(-time.Hour) |
| - f.started = f.written.Add(time.Minute) |
| - f.routes = []registry.Route{ |
| - {Host: "shop.example.com", Account: "bac", App: "shop", Hibernate: true, DBPath: "/data/shop.example.com.db"}, |
| - {Host: "api.example.com", Account: "bad", App: "api", DBPath: "/data/api.example.com.db"}, |
| - } |
| - f.paths = []string{"/data/registry.db", "/data/api.example.com.db"} |
| - p, err := newDurabilityPublisher(t.TempDir(), "box1", func() time.Time { return f.clock }) |
| - if err != nil { |
| - t.Fatal(err) |
| - } |
| - t.Cleanup(func() { |
| - ctx, cancel := context.WithTimeout(context.Background(), time.Second) |
| - defer cancel() |
| - if !p.stopped.Load() { |
| - _ = p.stop(ctx, nil) |
| - } |
| - _ = p.ledger.WaitLate(ctx) |
| - }) |
| - argv := []string{"/usr/local/bin/litestream", "replicate", "-config", "/etc/litestream.yml"} |
| - probe := replicatorProbe{configPath: "/etc/litestream.yml", unit: externalReplicatorUnit, |
| - readFile: func(string) ([]byte, error) { |
| - return []byte(ops.LitestreamConfig(f.paths, "bucket", "prefix", "eu-west-1")), nil |
| - }, |
| - modTime: func(string) (time.Time, error) { return f.written, nil }, |
| - unitState: func(context.Context, string) (bool, time.Time, []string, error) { |
| - if f.stateErr != nil { |
| - return false, time.Time{}, nil, f.stateErr |
| - } |
| - return true, f.started, argv, nil |
| - }, |
| - resolve: func(path string) (string, error) { return path, nil }, |
| - } |
| - p.enableScopedAttestation(probe, func() ([]registry.Route, error) { return f.routes, nil }, "/data") |
| - f.p = p |
| - return f |
| -} |
| - |
| -func (f *attestFixture) tick(t *testing.T, at time.Time) { |
| - t.Helper() |
| - f.clock = at |
| - if err := f.p.publish(context.Background(), f.store); err != nil { |
| - t.Fatal(err) |
| - } |
| -} |
| - |
| -// completeFor reads the published 17th and says whether it is complete for |
| -// acct over the whole day. |
| -func (f *attestFixture) completeFor(t *testing.T, acct string) bool { |
| - t.Helper() |
| - raw, _, err := f.store.Get(context.Background(), usage.DurabilityDayKey("box1", "2026-09-17")) |
| - if err != nil { |
| - t.Fatal(err) |
| - } |
| - day, err := usage.ParseDurabilityDay(raw, "box1", "2026-09-17") |
| - if err != nil { |
| - t.Fatal(err) |
| - } |
| - from, to := time.Date(2026, 9, 17, 0, 0, 0, 0, time.UTC), time.Date(2026, 9, 18, 0, 0, 0, 0, time.UTC) |
| - for _, snap := range day.Epochs { |
| - c, err := snap.Decode() |
| - if err != nil { |
| - t.Fatal(err) |
| - } |
| - ok, _ := c.CompletenessFor(acct, from, to) |
| - return ok |
| - } |
| - return false |
| -} |
| - |
| -// With the switch on, each tick observes the box before it seals. The |
| -// hibernating route's owner is vouched for and the always-on route's is |
| -// not. |
| -func TestEnabledScopedAttestationVouchesPerAccount(t *testing.T) { |
| - f := newAttestFixture(t) |
| - f.tick(t, f.clock) |
| - if got := f.p.ledger.Attestation(); !got.Attested || !reflect.DeepEqual(got.Unmetered, []string{"bad"}) { |
| - t.Fatalf("after the first tick: %+v", got) |
| - } |
| - f.tick(t, time.Date(2026, 9, 17, 12, 0, 0, 0, time.UTC)) |
| - f.tick(t, time.Date(2026, 9, 18, 0, 5, 0, 0, time.UTC)) |
| - if !f.completeFor(t, "bac") { |
| - t.Fatal("the day is not complete for the hibernating route's owner") |
| - } |
| - if f.completeFor(t, "bad") { |
| - t.Fatal("the day reads complete for the always-on route's owner") |
| - } |
| -} |
| - |
| -// At 23:58 a route stops hibernating, the config is regenerated to name |
| -// its database and the unit restarts. The next tick is at 00:05. The day |
| -// that is about to close must NOT close complete for that account: the |
| -// exclusion is dated at the config's writing, before the seal. |
| -func TestScopedAttestationExcludesFromWhenTheConfigWasWritten(t *testing.T) { |
| - f := newAttestFixture(t) |
| - f.tick(t, f.clock) |
| - f.tick(t, time.Date(2026, 9, 17, 23, 0, 0, 0, time.UTC)) |
| - if !reflect.DeepEqual(f.p.ledger.Attestation().Unmetered, []string{"bad"}) { |
| - t.Fatalf("fixture: %+v", f.p.ledger.Attestation()) |
| - } |
| - f.routes[0].Hibernate = false |
| - f.paths = append(f.paths, "/data/shop.example.com.db") |
| - f.written = time.Date(2026, 9, 17, 23, 58, 0, 0, time.UTC) |
| - f.started = f.written.Add(10 * time.Second) |
| - f.tick(t, time.Date(2026, 9, 18, 0, 5, 0, 0, time.UTC)) |
| - if got := f.p.ledger.Attestation(); !got.Attested || !reflect.DeepEqual(got.Unmetered, []string{"bac", "bad"}) { |
| - t.Fatalf("after the route changed: %+v", got) |
| - } |
| - if f.completeFor(t, "bac") { |
| - t.Fatal("the 17th closed complete for an account the external replicator began touching at 23:58") |
| - } |
| -} |
| - |
| -// When the box can no longer tell what the replicator is doing, the claim |
| -// ends, and it ends at the last tick that verified it, not at the tick |
| -// that noticed. |
| -func TestScopedAttestationEndsWhenThePremiseFails(t *testing.T) { |
| - for name, breakIt := range map[string]func(f *attestFixture){ |
| - "the unit's state cannot be read": func(f *attestFixture) { f.stateErr = errors.New("systemctl: not found") }, |
| - "the config names a database no route owns": func(f *attestFixture) { f.paths = append(f.paths, "/data/leftover.db") }, |
| - "the config is newer than the running unit": func(f *attestFixture) { f.written = f.started.Add(time.Hour) }, |
| - } { |
| - t.Run(name, func(t *testing.T) { |
| - f := newAttestFixture(t) |
| - f.tick(t, f.clock) |
| - f.tick(t, time.Date(2026, 9, 17, 6, 0, 0, 0, time.UTC)) |
| - if !f.p.ledger.Attestation().Attested { |
| - t.Fatal("fixture: should be attesting") |
| - } |
| - breakIt(f) |
| - f.tick(t, time.Date(2026, 9, 17, 18, 0, 0, 0, time.UTC)) |
| - if f.p.ledger.Attestation().Attested { |
| - t.Fatal("still attesting after the premise failed") |
| - } |
| - f.tick(t, time.Date(2026, 9, 18, 0, 5, 0, 0, time.UTC)) |
| - if f.completeFor(t, "bac") { |
| - t.Fatal("the day reads complete for an account although the claim ended during it") |
| - } |
| - }) |
| - } |
| -} |
diff --git a/internal/objstore/aws.go b/internal/objstore/aws.go| index 91fa09894..4e25bdccf 100644 |
| --- a/internal/objstore/aws.go |
| +++ b/internal/objstore/aws.go |
| @@ -78,7 +78,7 @@ var _ Provisioner = (*awsProvisioner)(nil) |
| // twin, so a non-AWS backend (endpoint set, "aws" backend) still means |
| // IAM-against-real-AWS in v1 — there is nothing else for it to point at. |
| func NewAWSProvisioner(ctx context.Context, region, endpoint, boundary string) (Provisioner, error) { |
| - s3Client, err := S3ClientFor(ctx, region, endpoint) |
| + s3Client, err := s3ClientFor(ctx, region, endpoint, func(o *awss3.Options) { o.HTTPSignerV4 = newFencedSigner() }) |
| if err != nil { |
| return nil, err |
| } |
| @@ -86,7 +86,10 @@ func NewAWSProvisioner(ctx context.Context, region, endpoint, boundary string) ( |
| if err != nil { |
| return nil, fmt.Errorf("objstore: load aws config: %w", err) |
| } |
| - iamClient := iam.NewFromConfig(cfg) |
| + // Both clients sign through the fence (fence.go): a request made inside |
| + // a walk carries the walk's lease instant as its signing time, so AWS |
| + // itself refuses it once that lease is too old. |
| + iamClient := iam.NewFromConfig(cfg, func(o *iam.Options) { o.HTTPSignerV4 = newFencedSigner() }) |
| return &awsProvisioner{s3: s3Client, iam: iamClient, region: region, boundary: boundary}, nil |
| } |
| @@ -97,6 +100,10 @@ func NewAWSProvisioner(ctx context.Context, region, endpoint, boundary string) ( |
| // The economics storage sweep (console.sizeProvisionedStore) is the first: |
| // it sizes a tenant's provisioned bucket directly, with no IAM involved. |
| func S3ClientFor(ctx context.Context, region, endpoint string) (*awss3.Client, error) { |
| + return s3ClientFor(ctx, region, endpoint) |
| +} |
| + |
| +func s3ClientFor(ctx context.Context, region, endpoint string, extra ...func(*awss3.Options)) (*awss3.Client, error) { |
| cfg, err := awsconfig.LoadDefaultConfig(ctx, awsconfig.WithRegion(region)) |
| if err != nil { |
| return nil, fmt.Errorf("objstore: load aws config: %w", err) |
| @@ -108,6 +115,9 @@ func S3ClientFor(ctx context.Context, region, endpoint string) (*awss3.Client, e |
| o.RequestChecksumCalculation = aws.RequestChecksumCalculationWhenRequired |
| o.ResponseChecksumValidation = aws.ResponseChecksumValidationWhenRequired |
| } |
| + for _, fn := range extra { |
| + fn(o) |
| + } |
| }), nil |
| } |
diff --git a/internal/objstore/fence.go b/internal/objstore/fence.go| new file mode 100644 |
| index 000000000..3edb48641 |
| --- /dev/null |
| +++ b/internal/objstore/fence.go |
| @@ -0,0 +1,76 @@ |
| +package objstore |
| + |
| +import ( |
| + "context" |
| + "errors" |
| + "net/http" |
| + "time" |
| + |
| + "github.com/aws/aws-sdk-go-v2/aws" |
| + v4 "github.com/aws/aws-sdk-go-v2/aws/signer/v4" |
| +) |
| + |
| +// The fence: what makes a walk's lease mean something at the BACKEND. |
| +// |
| +// A lease the walk checks before it calls the provider is not enough. A |
| +// process can be paused after the check and before the request leaves |
| +// (a stopped VM, a long collection, a debugger), and wake after its lease |
| +// has run out and another walk has taken the bucket. Nothing the client |
| +// checks beforehand can stop the request it then sends. Only the backend |
| +// can refuse it. |
| +// |
| +// So every request a walk makes is stamped with the instant its lease was |
| +// taken, and the backend refuses a request whose stamp is too old: |
| +// |
| +// - AWS signs each request with a timestamp and rejects one whose |
| +// timestamp is more than fifteen minutes from its own clock. The |
| +// signer normally stamps "now"; under a fence it stamps the lease's |
| +// instant instead (fencedSigner). A request sent after that window is |
| +// rejected by AWS whenever the process happens to send it. |
| +// - The in-memory provisioner refuses a call whose fence is older than |
| +// the same window. |
| +// |
| +// The lease on a bucket name lasts longer than the window (fenceWindow |
| +// plus fenceSkew), and a lease that was not given back cleanly cannot be |
| +// taken over until it has run out. So by the time another walk may take |
| +// the name, every request the earlier walk could ever send is one the |
| +// backend refuses. fenceSkew is the allowance for this host's clock |
| +// running ahead of the backend's. |
| +const ( |
| + fenceWindow = 15 * time.Minute |
| + fenceSkew = 5 * time.Minute |
| +) |
| + |
| +// ErrFenced: the backend refused a request whose walk's lease is too old. |
| +var ErrFenced = errors.New("objstore: request refused: its walk's lease has run out") |
| + |
| +type fenceKey struct{} |
| + |
| +// withFence marks every provider call made with ctx as part of a walk |
| +// whose lease was taken at `at`. |
| +func withFence(ctx context.Context, at time.Time) context.Context { |
| + return context.WithValue(ctx, fenceKey{}, at.UTC()) |
| +} |
| + |
| +func fenceFrom(ctx context.Context) (time.Time, bool) { |
| + at, ok := ctx.Value(fenceKey{}).(time.Time) |
| + return at, ok |
| +} |
| + |
| +// fencedSigner signs with the walk's lease instant when the request's |
| +// context carries one, and with the time it is given otherwise. It is |
| +// installed on the provisioner's S3 and IAM clients. |
| +type fencedSigner struct { |
| + inner interface { |
| + SignHTTP(ctx context.Context, credentials aws.Credentials, r *http.Request, payloadHash string, service string, region string, signingTime time.Time, optFns ...func(*v4.SignerOptions)) error |
| + } |
| +} |
| + |
| +func newFencedSigner() fencedSigner { return fencedSigner{inner: v4.NewSigner()} } |
| + |
| +func (f fencedSigner) SignHTTP(ctx context.Context, credentials aws.Credentials, r *http.Request, payloadHash string, service string, region string, signingTime time.Time, optFns ...func(*v4.SignerOptions)) error { |
| + if at, ok := fenceFrom(ctx); ok { |
| + signingTime = at |
| + } |
| + return f.inner.SignHTTP(ctx, credentials, r, payloadHash, service, region, signingTime, optFns...) |
| +} |
diff --git a/internal/objstore/fence_aws_test.go b/internal/objstore/fence_aws_test.go| new file mode 100644 |
| index 000000000..022cbb2da |
| --- /dev/null |
| +++ b/internal/objstore/fence_aws_test.go |
| @@ -0,0 +1,91 @@ |
| +package objstore |
| + |
| +import ( |
| + "context" |
| + "errors" |
| + "os" |
| + "strings" |
| + "testing" |
| + "time" |
| + |
| + "github.com/aws/aws-sdk-go-v2/aws" |
| + awsconfig "github.com/aws/aws-sdk-go-v2/config" |
| + "github.com/aws/aws-sdk-go-v2/service/iam" |
| + awss3 "github.com/aws/aws-sdk-go-v2/service/s3" |
| + "github.com/aws/smithy-go" |
| +) |
| + |
| +// TestFenceAgainstAWS checks the fence's premise against the real backend, |
| +// with read-only calls: AWS refuses a request whose signing time is older |
| +// than its window, whenever it arrives, and accepts one inside it. |
| +// |
| +// It runs only when CARLOS_FENCE_AWS_BUCKET names a bucket the caller's |
| +// credentials may list. It lists one key of that bucket and asks IAM for a |
| +// user that does not exist. It changes nothing. |
| +func TestFenceAgainstAWS(t *testing.T) { |
| + bucket := os.Getenv("CARLOS_FENCE_AWS_BUCKET") |
| + if bucket == "" { |
| + t.Skip("set CARLOS_FENCE_AWS_BUCKET (and AWS credentials) to check the fence against AWS; read-only") |
| + } |
| + region := os.Getenv("CARLOS_FENCE_AWS_REGION") |
| + if region == "" { |
| + region = "eu-west-1" |
| + } |
| + ctx := context.Background() |
| + s3c, err := s3ClientFor(ctx, region, "", func(o *awss3.Options) { |
| + o.HTTPSignerV4 = newFencedSigner() |
| + o.RetryMaxAttempts = 1 |
| + }) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + cfg, err := awsconfig.LoadDefaultConfig(ctx, awsconfig.WithRegion(region)) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + iamc := iam.NewFromConfig(cfg, func(o *iam.Options) { |
| + o.HTTPSignerV4 = newFencedSigner() |
| + o.RetryMaxAttempts = 1 |
| + }) |
| + code := func(err error) string { |
| + var api smithy.APIError |
| + if errors.As(err, &api) { |
| + return api.ErrorCode() + ": " + api.ErrorMessage() |
| + } |
| + if err != nil { |
| + return err.Error() |
| + } |
| + return "ok" |
| + } |
| + list := func(age time.Duration) string { |
| + _, err := s3c.ListObjectsV2(withFence(ctx, time.Now().Add(-age)), &awss3.ListObjectsV2Input{Bucket: aws.String(bucket), MaxKeys: aws.Int32(1)}) |
| + return code(err) |
| + } |
| + getUser := func(age time.Duration) string { |
| + _, err := iamc.GetUser(withFence(ctx, time.Now().Add(-age)), &iam.GetUserInput{UserName: aws.String("carlos-fence-probe-does-not-exist")}) |
| + return code(err) |
| + } |
| + for _, c := range []struct { |
| + age time.Duration |
| + refused bool |
| + }{ |
| + {time.Minute, false}, |
| + {fenceWindow - 2*time.Minute, false}, |
| + {fenceWindow + time.Minute, true}, |
| + {fenceWindow + fenceSkew, true}, |
| + } { |
| + s3Answer, iamAnswer := list(c.age), getUser(c.age) |
| + t.Logf("signed %v ago: S3 says %q; IAM says %q", c.age, s3Answer, iamAnswer) |
| + s3Refused := strings.Contains(s3Answer, "RequestTimeTooSkewed") |
| + iamRefused := strings.Contains(iamAnswer, "Signature expired") || strings.Contains(iamAnswer, "SignatureDoesNotMatch") || strings.Contains(iamAnswer, "RequestExpired") |
| + if s3Refused != c.refused { |
| + t.Errorf("S3, signed %v ago: refused=%v, want %v (%s)", c.age, s3Refused, c.refused, s3Answer) |
| + } |
| + if c.refused && !iamRefused { |
| + t.Errorf("IAM, signed %v ago: accepted (%s); the fence does not hold for IAM", c.age, iamAnswer) |
| + } |
| + if !c.refused && iamRefused { |
| + t.Logf("IAM refuses a request signed only %v ago: its window is shorter than S3's, which is the safe direction", c.age) |
| + } |
| + } |
| +} |
diff --git a/internal/objstore/memprovisioner.go b/internal/objstore/memprovisioner.go| index 4da037136..2f6d98b21 100644 |
| --- a/internal/objstore/memprovisioner.go |
| +++ b/internal/objstore/memprovisioner.go |
| @@ -67,6 +67,9 @@ func (p *MemProvisioner) takeFailure(method string) error { |
| } |
| func (p *MemProvisioner) EnsureBucket(ctx context.Context, name string) (string, error) { |
| + if err := memFenced(ctx); err != nil { |
| + return "", err |
| + } |
| p.mu.Lock() |
| defer p.mu.Unlock() |
| if err := p.takeFailure("EnsureBucket"); err != nil { |
| @@ -79,6 +82,9 @@ func (p *MemProvisioner) EnsureBucket(ctx context.Context, name string) (string, |
| } |
| func (p *MemProvisioner) EnsureUser(ctx context.Context, name, path, bucketARN string, deny *DenyTag) (string, error) { |
| + if err := memFenced(ctx); err != nil { |
| + return "", err |
| + } |
| p.mu.Lock() |
| defer p.mu.Unlock() |
| if err := p.takeFailure("EnsureUser"); err != nil { |
| @@ -110,6 +116,9 @@ func (p *MemProvisioner) PolicyFor(user string) string { |
| } |
| func (p *MemProvisioner) MintKey(ctx context.Context, user string) (string, string, error) { |
| + if err := memFenced(ctx); err != nil { |
| + return "", "", err |
| + } |
| p.mu.Lock() |
| defer p.mu.Unlock() |
| if err := p.takeFailure("MintKey"); err != nil { |
| @@ -130,6 +139,9 @@ func (p *MemProvisioner) MintKey(ctx context.Context, user string) (string, stri |
| } |
| func (p *MemProvisioner) DeleteKey(ctx context.Context, user, id string) error { |
| + if err := memFenced(ctx); err != nil { |
| + return err |
| + } |
| p.mu.Lock() |
| defer p.mu.Unlock() |
| if err := p.takeFailure("DeleteKey"); err != nil { |
| @@ -142,6 +154,9 @@ func (p *MemProvisioner) DeleteKey(ctx context.Context, user, id string) error { |
| } |
| func (p *MemProvisioner) ReapKeys(ctx context.Context, user string, keep []string) error { |
| + if err := memFenced(ctx); err != nil { |
| + return err |
| + } |
| p.mu.Lock() |
| defer p.mu.Unlock() |
| if err := p.takeFailure("ReapKeys"); err != nil { |
| @@ -164,6 +179,9 @@ func (p *MemProvisioner) ReapKeys(ctx context.Context, user string, keep []strin |
| } |
| func (p *MemProvisioner) EmptyBucket(ctx context.Context, name string) error { |
| + if err := memFenced(ctx); err != nil { |
| + return err |
| + } |
| p.mu.Lock() |
| defer p.mu.Unlock() |
| if err := p.takeFailure("EmptyBucket"); err != nil { |
| @@ -178,6 +196,9 @@ func (p *MemProvisioner) EmptyBucket(ctx context.Context, name string) error { |
| } |
| func (p *MemProvisioner) DeleteBucket(ctx context.Context, name string) error { |
| + if err := memFenced(ctx); err != nil { |
| + return err |
| + } |
| p.mu.Lock() |
| defer p.mu.Unlock() |
| if err := p.takeFailure("DeleteBucket"); err != nil { |
| @@ -188,6 +209,9 @@ func (p *MemProvisioner) DeleteBucket(ctx context.Context, name string) error { |
| } |
| func (p *MemProvisioner) DeleteUser(ctx context.Context, user string) error { |
| + if err := memFenced(ctx); err != nil { |
| + return err |
| + } |
| p.mu.Lock() |
| defer p.mu.Unlock() |
| if err := p.takeFailure("DeleteUser"); err != nil { |
| @@ -228,3 +252,13 @@ func (m *MemProvisioner) BucketGrantees(_ context.Context, name string) ([]strin |
| sort.Strings(out) |
| return out, nil |
| } |
| + |
| +// memFenced is the in-memory backend's half of the fence (fence.go): a call |
| +// whose walk's lease instant is older than the window is refused, as AWS |
| +// refuses a request whose signing time is too old. |
| +func memFenced(ctx context.Context) error { |
| + if at, ok := fenceFrom(ctx); ok && walkClock().After(at.Add(fenceWindow)) { |
| + return ErrFenced |
| + } |
| + return nil |
| +} |
diff --git a/internal/objstore/objstore.go b/internal/objstore/objstore.go| index b8a1a7063..63a8656ef 100644 |
| --- a/internal/objstore/objstore.go |
| +++ b/internal/objstore/objstore.go |
| @@ -145,13 +145,18 @@ type Record struct { |
| Teardown *TeardownBlock `json:"teardown,omitempty"` |
| } |
| -// TeardownBlock records the irreversible steps of a teardown that have |
| -// succeeded. |
| +// TeardownBlock is a teardown's write-ahead record on the store it is |
| +// taking apart. It is written before the teardown revokes or destroys |
| +// anything, so from that write on the record never again reads as a |
| +// healthy, provisioned store: a grant over it runs the walk in full |
| +// instead of trusting a Provisioned block whose user or bucket may be gone. |
| type TeardownBlock struct { |
| + // StartedAt is when a teardown first began on this record. |
| + StartedAt time.Time `json:"started_at"` |
| // BucketDeletedAt: this store's bucket was emptied and deleted. A |
| // re-run must never empty or delete a bucket of that name again: if |
| // one exists now, somebody else created it. |
| - BucketDeletedAt time.Time `json:"bucket_deleted_at"` |
| + BucketDeletedAt time.Time `json:"bucket_deleted_at,omitzero"` |
| } |
| func Key(acct, app string) string { return release.AppPrefix(acct, app) + "store.json" } |
diff --git a/internal/objstore/ownership.go b/internal/objstore/ownership.go| index 00b866a68..b30a2c4b8 100644 |
| --- a/internal/objstore/ownership.go |
| +++ b/internal/objstore/ownership.go |
| @@ -132,6 +132,11 @@ type BucketOwnership struct { |
| // enabled on it and did not disable it, because another app still held |
| // the bucket. Whoever destroys the bucket owes the scanner's removal. |
| ScannerOwed bool `json:"scanner_owed,omitempty"` |
| + // DestroyedAt is when a teardown last finished destroying the bucket: |
| + // whatever any store had put in it before that instant is gone. A store |
| + // that had already left the bucket by then has nothing left in it, and |
| + // its own unfinished teardown has nothing left to wait for. |
| + DestroyedAt *time.Time `json:"destroyed_at,omitempty"` |
| } |
| func validOwnershipTime(t time.Time) error { |
| @@ -396,7 +401,8 @@ func CloseBucketOwner(ctx context.Context, s release.Store, bucket, acct, app st |
| // marker goes with its interval, and the scanner it owed is gone |
| // with the bucket. |
| if d := b.Destroying; d != nil && d.Account == acct && d.App == app { |
| - b.Destroying, b.ScannerOwed, changed = nil, false, true |
| + at := until |
| + b.Destroying, b.ScannerOwed, b.DestroyedAt, changed = nil, false, &at, true |
| } |
| if !changed { |
| return nil, errOwnershipUnchanged |
| @@ -422,18 +428,21 @@ func CloseBucketOwner(ctx context.Context, s release.Store, bucket, acct, app st |
| // same time cannot each defer to the other and leave it with nobody: the |
| // second to write finds the first Leaving and destroys. If this app had |
| // scanning enabled, the ledger records that the scanner is owed. other |
| -// names one holder. |
| +// names one holder. foreign reports whether ANY app of another account |
| +// still has an open interval on the bucket, leaving or not: an app that is |
| +// leaving has not had its credentials revoked yet, and may never, so |
| +// objects left in the bucket are still within another account's reach. |
| // |
| // A store with no interval of its own (provisioned before ledgers) gets |
| // one, from its grant, so the decision is recorded. An app whose interval |
| // is already closed (a re-run after its close) is not reopened. |
| -func beginBucketTeardown(ctx context.Context, s release.Store, bucket, acct, app string, from time.Time, holders []OwnerRef, scanning bool, now time.Time) (destroy, scannerOwed bool, other OwnerRef, err error) { |
| +func beginBucketTeardown(ctx context.Context, s release.Store, bucket, acct, app string, from time.Time, holders []OwnerRef, scanning bool, now time.Time) (destroy, scannerOwed bool, other OwnerRef, foreign bool, err error) { |
| from, now = from.UTC(), now.UTC() |
| if err := validOwnershipTime(from); err != nil { |
| - return false, false, OwnerRef{}, fmt.Errorf("objstore: teardown of bucket %s: %v", bucket, err) |
| + return false, false, OwnerRef{}, false, fmt.Errorf("objstore: teardown of bucket %s: %v", bucket, err) |
| } |
| err = writeBucketOwnership(ctx, s, bucket, func(b *BucketOwnership) (*BucketOwnership, error) { |
| - destroy, scannerOwed, other = false, false, OwnerRef{} |
| + destroy, scannerOwed, other, foreign = false, false, OwnerRef{}, false |
| if b == nil { |
| b = &BucketOwnership{V: 1, Bucket: bucket} |
| } |
| @@ -446,21 +455,30 @@ func beginBucketTeardown(ctx context.Context, s release.Store, bucket, acct, app |
| } |
| continue |
| } |
| + if iv.open() && iv.Account != acct { |
| + foreign = true |
| + } |
| if iv.open() && !iv.Leaving && other.Account == "" { |
| other = OwnerRef{Account: iv.Account, App: iv.App, Since: iv.From} |
| } |
| } |
| - if other.Account == "" { |
| - for _, h := range holders { |
| - if !known[h.Account+"/"+h.App] { |
| - other = h |
| - break |
| - } |
| + for _, h := range holders { |
| + if known[h.Account+"/"+h.App] { |
| + continue |
| + } |
| + if h.Account != acct { |
| + foreign = true |
| + } |
| + if other.Account == "" { |
| + other = h |
| } |
| } |
| if d := b.Destroying; other.Account == "" && d != nil && (d.Account != acct || d.App != app) { |
| // Another teardown already decided it is the last holder. |
| other = *d |
| + if d.Account != acct { |
| + foreign = true |
| + } |
| } |
| if mine < 0 && !known[acct+"/"+app] { |
| b.Intervals = append(b.Intervals, OwnerInterval{Account: acct, App: app, From: from}) |
| @@ -487,7 +505,7 @@ func beginBucketTeardown(ctx context.Context, s release.Store, bucket, acct, app |
| } |
| return b, nil |
| }) |
| - return destroy, scannerOwed, other, err |
| + return destroy, scannerOwed, other, foreign, err |
| } |
| // stillDestroying re-reads the ledger and reports whether acct/app's |
diff --git a/internal/objstore/ownership_test.go b/internal/objstore/ownership_test.go| index 88514ae1d..dea2ef67a 100644 |
| --- a/internal/objstore/ownership_test.go |
| +++ b/internal/objstore/ownership_test.go |
| @@ -4,6 +4,8 @@ import ( |
| "context" |
| "encoding/json" |
| "errors" |
| + "github.com/aws/aws-sdk-go-v2/aws" |
| + "net/http" |
| "testing" |
| "time" |
| @@ -503,11 +505,12 @@ func TestGrantStopsWhenLedgerWriteFails(t *testing.T) { |
| mustLedger(t, s, ledgerBucket) |
| } |
| -// A bucket that already exists under the name this grant would mint may be |
| -// reachable by someone the ledger never saw (an operator adopted it for |
| -// another app before ledgers existed). The first grant must not claim the |
| -// ledger is complete. |
| -func TestGrantClaimsNothingForABucketThatAlreadyExists(t *testing.T) { |
| +// A grant that mints its bucket finds a bucket of that name already there, |
| +// and the ledger has never heard of this store. Somebody's objects are in |
| +// it. The grant is refused and leaves no grant block behind; an operator |
| +// who knows whose bucket it is adopts it by name, and that adoption claims |
| +// nothing about its past. |
| +func TestMintingGrantIsRefusedForABucketThatAlreadyExists(t *testing.T) { |
| ctx := context.Background() |
| s := release.NewDirStore(t.TempDir()) |
| _, pub := testSealKeypair(t) |
| @@ -515,9 +518,23 @@ func TestGrantClaimsNothingForABucketThatAlreadyExists(t *testing.T) { |
| if _, err := p.EnsureBucket(ctx, ledgerBucket); err != nil { |
| t.Fatal(err) |
| } |
| + p.Buckets[ledgerBucket]["left-behind"] = []byte("somebody's") |
| declare(t, s, "bab", "titogo", "TITO_S3") |
| - if _, err := Grant(ctx, s, p, testDeployment(), pub, "bab", "titogo", "op", GrantSourceOperator, "", t0); err != nil { |
| - t.Fatal(err) |
| + if _, err := Grant(ctx, s, p, testDeployment(), pub, "bab", "titogo", "op", GrantSourceOperator, "", t0); !errors.Is(err, ErrBucketHeld) { |
| + t.Fatalf("a minting grant onto an existing bucket: %v, want ErrBucketHeld", err) |
| + } |
| + if rec, _, _ := Get(ctx, s, "bab", "titogo"); rec == nil || rec.Grant != nil || rec.Provisioned != nil { |
| + t.Fatalf("the refused grant left its block on the record: %+v", rec) |
| + } |
| + if _, ok := p.Users[UserName("bab", "titogo")]; ok { |
| + t.Fatal("the refused grant created a user") |
| + } |
| + if b, _, _ := ReadBucketOwnership(ctx, s, ledgerBucket); b != nil { |
| + t.Fatalf("the refused grant wrote a ledger: %+v", b) |
| + } |
| + // Adopted by name, on an operator's say-so. |
| + if _, err := Grant(ctx, s, p, testDeployment(), pub, "bab", "titogo", "op", GrantSourceOperator, ledgerBucket, t0); err != nil { |
| + t.Fatalf("adopting the existing bucket: %v", err) |
| } |
| b := mustLedger(t, s, ledgerBucket) |
| if b.CompleteSince != nil { |
| @@ -614,6 +631,7 @@ func TestPartialBackfillOfASharedBucketResolvesNoOwner(t *testing.T) { |
| // bucket. When A is granted, the bucket does not exist, and A's ledger must |
| // still not claim to name everyone: B would reach the new bucket at once. |
| func TestGrantClaimsNothingWhileAnotherPrincipalNamesTheBucket(t *testing.T) { |
| + advance := leaseClock(t) |
| ctx := context.Background() |
| s := release.NewDirStore(t.TempDir()) |
| _, pub := testSealKeypair(t) |
| @@ -630,6 +648,8 @@ func TestGrantClaimsNothingWhileAnotherPrincipalNamesTheBucket(t *testing.T) { |
| if err := Teardown(ctx, s, p, nil, "bac", "other"); err == nil { |
| t.Fatal("the injected DeleteUser failure did not stop the teardown") |
| } |
| + // The failed walk keeps its lease until it runs out. |
| + advance(walkLease + time.Second) |
| p.FailNext = nil |
| if _, exists := p.Buckets[ledgerBucket]; exists { |
| t.Fatal("fixture: the bucket should be gone") |
| @@ -762,11 +782,23 @@ func TestObservationHourDoesNotResolveToTheNewOwner(t *testing.T) { |
| // Probes answer at once in these tests unless one says otherwise. |
| func init() { grantElapsed = func(time.Time) time.Duration { return 0 } } |
| +// leaseClock puts the walk leases on a clock the test moves. The returned |
| +// function advances it. |
| +func leaseClock(t *testing.T) func(time.Duration) { |
| + t.Helper() |
| + clock := time.Now() |
| + prev := walkClock |
| + walkClock = func() time.Time { return clock } |
| + t.Cleanup(func() { walkClock = prev }) |
| + return func(d time.Duration) { clock = clock.Add(d) } |
| +} |
| + |
| // A teardown closes its ownership interval only after the credentials are |
| // revoked, and while it is unfinished nothing can be granted the name: the |
| // ledger says the bucket is being destroyed, and a grant is refused by the |
| // same compare-and-swap that would have opened its interval. |
| func TestTeardownHoldsTheNameUntilItHasFinished(t *testing.T) { |
| + advance := leaseClock(t) |
| ctx := context.Background() |
| s := release.NewDirStore(t.TempDir()) |
| _, pub := testSealKeypair(t) |
| @@ -779,6 +811,8 @@ func TestTeardownHoldsTheNameUntilItHasFinished(t *testing.T) { |
| if err := Teardown(ctx, s, p, nil, "bab", "titogo"); err == nil { |
| t.Fatal("the injected DeleteUser failure did not stop the teardown") |
| } |
| + // The failed walk keeps its lease until it runs out. |
| + advance(walkLease + time.Second) |
| if _, exists := p.Buckets[ledgerBucket]; exists { |
| t.Fatal("fixture: the bucket should be gone by the time DeleteUser fails") |
| } |
| @@ -830,6 +864,7 @@ func TestTeardownHoldsTheNameUntilItHasFinished(t *testing.T) { |
| // back: the marker is lifted, the record forgets the deleted bucket, and a |
| // later teardown destroys the new one. |
| func TestGrantOverItsOwnUnfinishedTeardownTakesTheStoreBack(t *testing.T) { |
| + advance := leaseClock(t) |
| ctx := context.Background() |
| s := release.NewDirStore(t.TempDir()) |
| _, pub := testSealKeypair(t) |
| @@ -842,6 +877,8 @@ func TestGrantOverItsOwnUnfinishedTeardownTakesTheStoreBack(t *testing.T) { |
| if err := Teardown(ctx, s, p, nil, "bab", "titogo"); err == nil { |
| t.Fatal("the injected DeleteUser failure did not stop the teardown") |
| } |
| + // The failed walk keeps its lease until it runs out. |
| + advance(walkLease + time.Second) |
| p.FailNext = nil |
| if _, err := Grant(ctx, s, p, testDeployment(), pub, "bab", "titogo", "op", GrantSourceOperator, "", t0.Add(time.Hour)); err != nil { |
| t.Fatalf("re-grant: %v", err) |
| @@ -903,6 +940,7 @@ func TestMintingGrantIsRefusedWhileAnotherAppHoldsTheName(t *testing.T) { |
| // the teardown does not report itself done: B's record stands until a |
| // person, or A's own teardown, settles the bucket. |
| func TestRetriedLegacyTeardownLeavesABucketAnotherAppNowHolds(t *testing.T) { |
| + advance := leaseClock(t) |
| ctx := context.Background() |
| s := release.NewDirStore(t.TempDir()) |
| _, pub := testSealKeypair(t) |
| @@ -915,6 +953,8 @@ func TestRetriedLegacyTeardownLeavesABucketAnotherAppNowHolds(t *testing.T) { |
| if err := Teardown(ctx, s, p, nil, "bac", "other"); err == nil { |
| t.Fatal("the injected DeleteUser failure did not stop the teardown") |
| } |
| + // The failed walk keeps its lease until it runs out. |
| + advance(walkLease + time.Second) |
| p.FailNext = nil |
| // As the older build left it. |
| if err := s.Delete(ctx, BucketOwnershipKey(ledgerBucket)); err != nil { |
| @@ -929,9 +969,26 @@ func TestRetriedLegacyTeardownLeavesABucketAnotherAppNowHolds(t *testing.T) { |
| t.Fatal(err) |
| } |
| declare(t, s, "bab", "titogo", "TITO_S3") |
| + // This build would not grant A the name at all while B's record names |
| + // the bucket. |
| + if _, err := Grant(ctx, s, p, testDeployment(), pub, "bab", "titogo", "op", GrantSourceOperator, "", t0.Add(72*time.Hour)); !errors.Is(err, ErrBucketHeld) { |
| + t.Fatalf("a minting grant while a legacy store's record names the bucket: %v, want ErrBucketHeld", err) |
| + } |
| + // The older build did. B's record is put out of sight for the length |
| + // of A's grant, which is what that build saw, and put back. |
| + raw, _, err := s.Get(ctx, Key("bac", "other")) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := s.Delete(ctx, Key("bac", "other")); err != nil { |
| + t.Fatal(err) |
| + } |
| if _, err := Grant(ctx, s, p, testDeployment(), pub, "bab", "titogo", "op", GrantSourceOperator, "", t0.Add(72*time.Hour)); err != nil { |
| t.Fatal(err) |
| } |
| + if _, err := s.Put(ctx, Key("bac", "other"), raw, ""); err != nil { |
| + t.Fatal(err) |
| + } |
| p.Buckets[ledgerBucket]["invoice.pdf"] = []byte("the new owner's data") |
| sc := NewMemScanner() |
| if err := sc.EnsureEnabled(ctx, ledgerBucket); err != nil { |
| @@ -1090,11 +1147,11 @@ func TestConcurrentTeardownsOfASharedBucketLeaveExactlyOneDestroyer(t *testing.T |
| } |
| // Both teardowns have read the ledger with both intervals open. Then |
| // each makes its decision. |
| - destroyOne, _, _, err := beginBucketTeardown(ctx, s, ledgerBucket, "bab", "one", t0, nil, true, now) |
| + destroyOne, _, _, _, err := beginBucketTeardown(ctx, s, ledgerBucket, "bab", "one", t0, nil, true, now) |
| if err != nil { |
| t.Fatal(err) |
| } |
| - destroyTwo, owed, _, err := beginBucketTeardown(ctx, s, ledgerBucket, "bab", "two", t0, nil, false, now) |
| + destroyTwo, owed, _, _, err := beginBucketTeardown(ctx, s, ledgerBucket, "bab", "two", t0, nil, false, now) |
| if err != nil { |
| t.Fatal(err) |
| } |
| @@ -1109,10 +1166,10 @@ func TestConcurrentTeardownsOfASharedBucketLeaveExactlyOneDestroyer(t *testing.T |
| t.Fatalf("destroying marker: %+v", b.Destroying) |
| } |
| // A decision repeated (a re-run) is the same decision. |
| - if again, _, _, _ := beginBucketTeardown(ctx, s, ledgerBucket, "bab", "one", t0, nil, true, now); again { |
| + if again, _, _, _, _ := beginBucketTeardown(ctx, s, ledgerBucket, "bab", "one", t0, nil, true, now); again { |
| t.Fatal("the leaving app's re-run decided to destroy a bucket another app is destroying") |
| } |
| - if again, _, _, _ := beginBucketTeardown(ctx, s, ledgerBucket, "bab", "two", t0, nil, false, now); !again { |
| + if again, _, _, _, _ := beginBucketTeardown(ctx, s, ledgerBucket, "bab", "two", t0, nil, false, now); !again { |
| t.Fatal("the destroyer's re-run changed its mind") |
| } |
| } |
| @@ -1132,7 +1189,7 @@ func TestGrantAndTeardownCannotBothWin(t *testing.T) { |
| if err := OpenBucketOwner(ctx, s, ledgerBucket, "bab", "two", now, now, false); err != nil { |
| t.Fatal(err) |
| } |
| - if destroy, _, other, err := beginBucketTeardown(ctx, s, ledgerBucket, "bab", "one", t0, nil, false, now); err != nil || destroy || other.App != "two" { |
| + if destroy, _, other, _, err := beginBucketTeardown(ctx, s, ledgerBucket, "bab", "one", t0, nil, false, now); err != nil || destroy || other.App != "two" { |
| t.Fatalf("teardown after a grant landed: destroy=%v other=%+v err=%v", destroy, other, err) |
| } |
| // The teardown first. |
| @@ -1140,7 +1197,7 @@ func TestGrantAndTeardownCannotBothWin(t *testing.T) { |
| if err := OpenBucketOwner(ctx, s, ledgerBucket, "bab", "one", t0, t0, false); err != nil { |
| t.Fatal(err) |
| } |
| - if destroy, _, _, err := beginBucketTeardown(ctx, s, ledgerBucket, "bab", "one", t0, nil, false, now); err != nil || !destroy { |
| + if destroy, _, _, _, err := beginBucketTeardown(ctx, s, ledgerBucket, "bab", "one", t0, nil, false, now); err != nil || !destroy { |
| t.Fatalf("teardown with no other holder: destroy=%v err=%v", destroy, err) |
| } |
| if err := OpenBucketOwner(ctx, s, ledgerBucket, "bab", "two", now, now, false); !errors.Is(err, ErrBucketBeingDestroyed) { |
| @@ -1189,3 +1246,326 @@ func TestCompletenessIsDatedAfterTheProbesAnswer(t *testing.T) { |
| t.Fatal("the 11:00 hour did not resolve") |
| } |
| } |
| + |
| +// One walk at a time on a bucket name, across processes, and the authority |
| +// is the invocation's, not the app's. |
| +func TestWalkLeaseIsOneInvocationAtATime(t *testing.T) { |
| + ctx := context.Background() |
| + s := release.NewDirStore(t.TempDir()) |
| + _, pub := testSealKeypair(t) |
| + p := NewMemProvisioner("eu-west-1") |
| + declare(t, s, "bab", "titogo", "TITO_S3") |
| + if _, err := Grant(ctx, s, p, testDeployment(), pub, "bab", "titogo", "op", GrantSourceOperator, "", t0); err != nil { |
| + t.Fatal(err) |
| + } |
| + p.Buckets[ledgerBucket]["kept"] = []byte("data") |
| + // Another process is in the middle of a teardown of this very app. |
| + other, err := acquireBucketWalk(ctx, s, ledgerBucket, "bab", "titogo", "teardown") |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + // A second teardown of the same app does not inherit its authority. |
| + if err := Teardown(ctx, s, p, nil, "bab", "titogo"); !errors.Is(err, ErrBucketBusy) { |
| + t.Fatalf("a second teardown while one holds the lease: %v, want ErrBucketBusy", err) |
| + } |
| + if string(p.Buckets[ledgerBucket]["kept"]) != "data" { |
| + t.Fatal("the refused teardown touched the bucket") |
| + } |
| + // Nor does a grant, of this app or another. |
| + if _, err := Grant(ctx, s, p, testDeployment(), pub, "bab", "titogo", "op", GrantSourceOperator, "", t0.Add(time.Hour)); !errors.Is(err, ErrBucketBusy) { |
| + t.Fatalf("a grant while a teardown holds the lease: %v, want ErrBucketBusy", err) |
| + } |
| + declare(t, s, "bac", "other", "OTHER_S3") |
| + if _, err := Grant(ctx, s, p, testDeployment(), pub, "bac", "other", "op", GrantSourceOperator, ledgerBucket, t0.Add(time.Hour)); !errors.Is(err, ErrBucketBusy) { |
| + t.Fatalf("another app's grant while a teardown holds the lease: %v, want ErrBucketBusy", err) |
| + } |
| + // The holder's calls carry its fence, and a deadline that falls before |
| + // the backend's window for that fence closes. |
| + bounded, cancel := other.bound(ctx) |
| + deadline, ok := bounded.Deadline() |
| + at, fenced := fenceFrom(bounded) |
| + cancel() |
| + if !ok || !fenced || !at.Equal(other.lock.SignedAt) || !deadline.Before(time.Now().Add(fenceWindow)) || !at.Add(fenceWindow).Before(other.lock.Until) { |
| + t.Fatalf("the walk's calls are not fenced inside its lease: fence %v deadline %v, lease %v to %v", at, deadline, other.lock.SignedAt, other.lock.Until) |
| + } |
| + if held, err := other.held(ctx); err != nil || !held { |
| + t.Fatalf("the holder does not hold its own lease: %v %v", held, err) |
| + } |
| + // Given back cleanly, the name is free again; and a release cannot be |
| + // repeated into someone else's lease. |
| + other.release(true) |
| + next, err := acquireBucketWalk(ctx, s, ledgerBucket, "bac", "other", "grant") |
| + if err != nil { |
| + t.Fatalf("after a clean release: %v", err) |
| + } |
| + other.closed = false |
| + other.release(true) // a stale release, after another invocation took the lease |
| + if held, err := next.held(ctx); err != nil || !held { |
| + t.Fatalf("a stale release took another invocation's lease away: %v %v", held, err) |
| + } |
| + next.release(true) |
| + if err := Teardown(ctx, s, p, nil, "bab", "titogo"); err != nil { |
| + t.Fatalf("teardown once the lease is free: %v", err) |
| + } |
| +} |
| + |
| +// A walk that ended with an error may have a request still on its way. Its |
| +// lease is not freed: until it runs out, only the same app running the |
| +// same kind of walk again may take the name. And a clean finish of that |
| +// retry still does not free the name before the first attempt has run out. |
| +func TestWalkLeaseStandsAfterAWalkThatFailed(t *testing.T) { |
| + ctx := context.Background() |
| + s := release.NewDirStore(t.TempDir()) |
| + advance := leaseClock(t) |
| + failed, err := acquireBucketWalk(ctx, s, ledgerBucket, "bab", "titogo", "teardown") |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + failed.release(false) |
| + if _, err := acquireBucketWalk(ctx, s, ledgerBucket, "bac", "other", "grant"); !errors.Is(err, ErrBucketBusy) { |
| + t.Fatalf("another app after a failed walk: %v, want ErrBucketBusy", err) |
| + } |
| + if _, err := acquireBucketWalk(ctx, s, ledgerBucket, "bab", "titogo", "grant"); !errors.Is(err, ErrBucketBusy) { |
| + t.Fatalf("the same app, another kind of walk, after a failed walk: %v, want ErrBucketBusy", err) |
| + } |
| + advance(time.Minute) |
| + retry, err := acquireBucketWalk(ctx, s, ledgerBucket, "bab", "titogo", "teardown") |
| + if err != nil { |
| + t.Fatalf("the same app re-running the same walk: %v", err) |
| + } |
| + retry.release(true) |
| + // The retry was clean, but the first attempt's requests could still be |
| + // accepted by the backend until its window closes. |
| + if _, err := acquireBucketWalk(ctx, s, ledgerBucket, "bac", "other", "grant"); !errors.Is(err, ErrBucketBusy) { |
| + t.Fatalf("another app before the failed attempt has run out: %v, want ErrBucketBusy", err) |
| + } |
| + advance(walkLease) |
| + free, err := acquireBucketWalk(ctx, s, ledgerBucket, "bac", "other", "grant") |
| + if err != nil { |
| + t.Fatalf("after the failed attempt ran out: %v", err) |
| + } |
| + free.release(true) |
| +} |
| + |
| +// The proof that a timed lease is not all there is. A teardown takes its |
| +// lease and is then paused, for longer than the lease, at the worst |
| +// moment: after every check it makes and before its request leaves. While |
| +// it sleeps the lease runs out, another walk takes the name, finishes, and |
| +// a different app is granted the bucket and fills it. The first teardown |
| +// wakes and sends its request to empty the bucket. |
| +// |
| +// No check in the client can stop that request: the checks were all made |
| +// before the pause. The backend stops it. The request carries the lease's |
| +// instant, and the backend refuses a request stamped that long ago. |
| +// |
| +// At every instant, one of two things is true: the backend would still |
| +// accept the old walk's requests and nobody can take its lease, or |
| +// somebody has taken the lease and the backend refuses the old walk. |
| +func TestPausedWalkCannotActAfterItsLeaseIsTaken(t *testing.T) { |
| + ctx := context.Background() |
| + s := release.NewDirStore(t.TempDir()) |
| + _, pub := testSealKeypair(t) |
| + p := NewMemProvisioner("eu-west-1") |
| + advance := leaseClock(t) |
| + declare(t, s, "bab", "titogo", "TITO_S3") |
| + if _, err := Grant(ctx, s, p, testDeployment(), pub, "bab", "titogo", "op", GrantSourceOperator, "", t0); err != nil { |
| + t.Fatal(err) |
| + } |
| + // The teardown that will be paused: it holds the lease, and this is |
| + // the context every one of its provider calls is made with. The test |
| + // keeps the context without its client-side deadline, which is exactly |
| + // what a pause after the deadline check amounts to. |
| + paused, err := acquireBucketWalk(ctx, s, ledgerBucket, "bab", "titogo", "teardown") |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + stale := withFence(ctx, paused.lock.SignedAt) |
| + |
| + // While the backend would still accept its requests, nobody can take |
| + // the lease: not at the start, and not at the very end of the window. |
| + for _, d := range []time.Duration{0, fenceWindow - time.Second} { |
| + advance(d) |
| + if err := memFenced(stale); err != nil { |
| + t.Fatalf("%v into the lease the backend already refuses the holder: %v", d, err) |
| + } |
| + if _, err := acquireBucketWalk(ctx, s, ledgerBucket, "bac", "other", "grant"); !errors.Is(err, ErrBucketBusy) { |
| + t.Fatalf("%v into the lease it was taken over while the backend still accepts the holder: %v", d, err) |
| + } |
| + } |
| + // Past the window and before the lease runs out: the backend refuses |
| + // the holder, and still nobody can take the lease. The two never overlap. |
| + advance(2 * time.Second) |
| + if err := memFenced(stale); !errors.Is(err, ErrFenced) { |
| + t.Fatalf("past the window the backend accepts the holder: %v", err) |
| + } |
| + if _, err := acquireBucketWalk(ctx, s, ledgerBucket, "bac", "other", "grant"); !errors.Is(err, ErrBucketBusy) { |
| + t.Fatalf("taken over before the lease ran out: %v", err) |
| + } |
| + // The lease runs out. Another process finishes the teardown, and |
| + // another app is granted the name and fills the bucket. |
| + advance(fenceSkew) |
| + if err := Teardown(ctx, s, p, nil, "bab", "titogo"); err != nil { |
| + t.Fatalf("the teardown that took over: %v", err) |
| + } |
| + declare(t, s, "bac", "other", "OTHER_S3") |
| + if _, err := Grant(ctx, s, p, testDeployment(), pub, "bac", "other", "op", GrantSourceOperator, ledgerBucket, t0.Add(time.Hour)); err != nil { |
| + t.Fatalf("the grant after it: %v", err) |
| + } |
| + p.Buckets[ledgerBucket]["invoice.pdf"] = []byte("the new holder's data") |
| + // The paused teardown wakes and sends what it was about to send. |
| + for name, call := range map[string]func() error{ |
| + "EmptyBucket": func() error { return p.EmptyBucket(stale, ledgerBucket) }, |
| + "DeleteBucket": func() error { return p.DeleteBucket(stale, ledgerBucket) }, |
| + "DeleteUser": func() error { return p.DeleteUser(stale, UserName("bac", "other")) }, |
| + } { |
| + if err := call(); !errors.Is(err, ErrFenced) { |
| + t.Fatalf("%s from the paused walk after its lease was taken: %v, want the backend's refusal", name, err) |
| + } |
| + } |
| + if got := string(p.Buckets[ledgerBucket]["invoice.pdf"]); got != "the new holder's data" { |
| + t.Fatalf("the paused walk destroyed the new holder's bucket (object now %q)", got) |
| + } |
| + if _, ok := p.Users[UserName("bac", "other")]; !ok { |
| + t.Fatal("the paused walk removed the new holder's user") |
| + } |
| + // And its release, when it finally returns, touches nobody's lease. |
| + paused.release(false) |
| + if _, err := Grant(ctx, s, p, testDeployment(), pub, "bac", "other", "op", GrantSourceOperator, ledgerBucket, t0.Add(2*time.Hour)); err != nil { |
| + t.Fatalf("the new holder's next walk after the stale release: %v", err) |
| + } |
| +} |
| + |
| +// The AWS half of the fence: a request made inside a walk is signed with |
| +// the walk's lease instant, not with the time it happens to be sent, so |
| +// AWS's own check on the age of a signature is a check on the lease. |
| +func TestFencedSignerStampsTheLeaseInstant(t *testing.T) { |
| + signer := newFencedSigner() |
| + creds := aws.Credentials{AccessKeyID: "AKIDEXAMPLE", SecretAccessKey: "secret"} |
| + lease := time.Date(2026, 9, 3, 10, 0, 0, 0, time.UTC) |
| + sentAt := lease.Add(40 * time.Minute) // the process woke much later |
| + sign := func(ctx context.Context) string { |
| + req, err := http.NewRequest(http.MethodDelete, "https://s3.eu-west-1.amazonaws.com/"+ledgerBucket, nil) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := signer.SignHTTP(ctx, creds, req, "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "s3", "eu-west-1", sentAt); err != nil { |
| + t.Fatal(err) |
| + } |
| + return req.Header.Get("X-Amz-Date") |
| + } |
| + if got := sign(withFence(context.Background(), lease)); got != "20260903T100000Z" { |
| + t.Fatalf("a fenced request is dated %s, want the lease instant 20260903T100000Z", got) |
| + } |
| + if got := sign(context.Background()); got != "20260903T104000Z" { |
| + t.Fatalf("an unfenced request is dated %s, want the time it was sent", got) |
| + } |
| +} |
| + |
| +// A store that left a bucket to another account does not wait for ever. |
| +// Once the last holder has destroyed that bucket, the first store's objects |
| +// are gone with it, and its teardown finishes, even if a bucket of the same |
| +// name has since been granted to somebody new. |
| +func TestRetainedTeardownFinishesOnceTheBucketItLeftIsDestroyed(t *testing.T) { |
| + ctx := context.Background() |
| + s := release.NewDirStore(t.TempDir()) |
| + _, pub := testSealKeypair(t) |
| + p := NewMemProvisioner("eu-west-1") |
| + const shared = "carlos-store-shared-archive" |
| + at := t0 |
| + clock := ownershipNow |
| + ownershipNow = func() time.Time { return at } |
| + defer func() { ownershipNow = clock }() |
| + for _, acct := range []string{"bac", "bad"} { |
| + declare(t, s, acct, "one", "ARCHIVE_S3") |
| + if _, err := Grant(ctx, s, p, testDeployment(), pub, acct, "one", "op", GrantSourceOperator, shared, at); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + at = at.Add(time.Hour) |
| + if err := Teardown(ctx, s, p, nil, "bac", "one"); !errors.Is(err, ErrBucketRetained) { |
| + t.Fatalf("first teardown: %v, want ErrBucketRetained", err) |
| + } |
| + // Re-granted while retained: the walk runs in full and mints a new key. |
| + // It does not report the store healthy on a key that was deleted. |
| + at = at.Add(time.Hour) |
| + rec, err := Grant(ctx, s, p, testDeployment(), pub, "bac", "one", "op", GrantSourceOperator, shared, at) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if u, ok := p.Users[UserName("bac", "one")]; !ok || len(u.Keys) == 0 || rec.Provisioned == nil || u.Keys[rec.Provisioned.KeyID] == "" { |
| + t.Fatalf("the re-grant left the store on a key that no longer exists: record %+v", rec.Provisioned) |
| + } |
| + at = at.Add(time.Hour) |
| + if err := Teardown(ctx, s, p, nil, "bac", "one"); !errors.Is(err, ErrBucketRetained) { |
| + t.Fatalf("second teardown: %v, want ErrBucketRetained", err) |
| + } |
| + // The other holder goes, and takes the bucket with it. |
| + at = at.Add(time.Hour) |
| + if err := Teardown(ctx, s, p, nil, "bad", "one"); err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, exists := p.Buckets[shared]; exists { |
| + t.Fatal("the last holder's teardown left the bucket") |
| + } |
| + // Somebody new is granted the name before the first store re-runs. |
| + at = at.Add(time.Hour) |
| + declare(t, s, "bae", "new", "ARCHIVE_S3") |
| + if _, err := Grant(ctx, s, p, testDeployment(), pub, "bae", "new", "op", GrantSourceOperator, shared, at); err != nil { |
| + t.Fatal(err) |
| + } |
| + p.Buckets[shared]["fresh"] = []byte("the newcomer's data") |
| + at = at.Add(time.Hour) |
| + if err := Teardown(ctx, s, p, nil, "bac", "one"); err != nil { |
| + t.Fatalf("the first store's teardown after its bucket was destroyed: %v, want it to finish", err) |
| + } |
| + if rec, _, _ := Get(ctx, s, "bac", "one"); rec != nil { |
| + t.Fatalf("the first store's record remains: %+v", rec) |
| + } |
| + if string(p.Buckets[shared]["fresh"]) != "the newcomer's data" { |
| + t.Fatal("finishing the old teardown touched the newcomer's bucket") |
| + } |
| +} |
| + |
| +// The decision to destroy is read again from the ledger immediately before |
| +// each destructive call. The lease keeps other walks out, so a change in |
| +// between is a hand on the ledger (an operator's, or a build that does not |
| +// take the lease). Whichever it is, the teardown stops: nothing more is |
| +// destroyed on a decision the ledger no longer carries. |
| +func TestTeardownStopsWhenTheLedgerNoLongerCarriesItsDecision(t *testing.T) { |
| + ctx := context.Background() |
| + s := release.NewDirStore(t.TempDir()) |
| + _, pub := testSealKeypair(t) |
| + mem := NewMemProvisioner("eu-west-1") |
| + leaseClock(t) |
| + declare(t, s, "bab", "titogo", "TITO_S3") |
| + if _, err := Grant(ctx, s, mem, testDeployment(), pub, "bab", "titogo", "op", GrantSourceOperator, "", t0); err != nil { |
| + t.Fatal(err) |
| + } |
| + // The marker is withdrawn while the bucket is being emptied: after the |
| + // walk confirmed that step, before it would delete the bucket. |
| + var calls []string |
| + p := &interleavingProvisioner{MemProvisioner: mem, between: func(method string) { |
| + calls = append(calls, method) |
| + if method != "EmptyBucket" { |
| + return |
| + } |
| + if err := writeBucketOwnership(ctx, s, ledgerBucket, func(b *BucketOwnership) (*BucketOwnership, error) { |
| + b.Destroying = nil |
| + return b, nil |
| + }); err != nil { |
| + t.Error(err) |
| + } |
| + }} |
| + err := Teardown(ctx, s, p, nil, "bab", "titogo") |
| + if err == nil { |
| + t.Fatalf("the teardown finished on a decision the ledger no longer carried (calls %v)", calls) |
| + } |
| + if _, ok := mem.Buckets[ledgerBucket]; !ok { |
| + t.Fatalf("the bucket was deleted on a decision the ledger no longer carried (calls %v)", calls) |
| + } |
| + for _, c := range calls { |
| + if c == "DeleteBucket" { |
| + t.Fatalf("DeleteBucket was called: %v", calls) |
| + } |
| + } |
| +} |
diff --git a/internal/objstore/walk.go b/internal/objstore/walk.go| index d6a1a6c9d..f764d857b 100644 |
| --- a/internal/objstore/walk.go |
| +++ b/internal/objstore/walk.go |
| @@ -139,7 +139,7 @@ func ValidBucketName(name string) bool { return bucketNameRe.MatchString(name) } |
| // whose authority was already established and must not restate it, exactly as |
| // it must not re-choose the bucket. |
| func Grant(ctx context.Context, s release.Store, p Provisioner, d Deployment, |
| - sealPub *ecdh.PublicKey, acct, app, actor, source, adoptBucket string, now time.Time) (*Record, error) { |
| + sealPub *ecdh.PublicKey, acct, app, actor, source, adoptBucket string, now time.Time) (granted *Record, walkErr error) { |
| // The caller's clock was read before this walk began. What the walk |
| // observes, it observes later: the completeness claim below is dated |
| // by the caller's clock plus the time this walk has run, taken AFTER |
| @@ -154,18 +154,14 @@ func Grant(ctx context.Context, s release.Store, p Provisioner, d Deployment, |
| if rec == nil || rec.Declared == nil { |
| return nil, ErrNotDeclared |
| } |
| - // A grant over an unfinished teardown takes the store back. Whatever |
| - // that teardown had already destroyed is created again below, so its |
| - // record of having destroyed it must not outlive this. And the store is |
| - // not healthy whatever its record and delivered variables say: the |
| - // walk runs in full (rebuilt, at the short-circuit below). |
| + // A grant over an unfinished teardown takes the store back. The record |
| + // says a teardown began (TeardownBlock), so whatever its Provisioned |
| + // block and delivered variables say, the user or the bucket may be |
| + // gone: the walk runs in full (rebuilt, at the short-circuit below). |
| + // The block is cleared only by the write that records the rebuilt |
| + // store. A grant that is refused or fails part-way leaves it standing, |
| + // so the record never reads as healthy in between. |
| rebuilt := rec.Teardown != nil |
| - if rebuilt { |
| - rec.Teardown = nil |
| - if etag, err = Put(ctx, s, acct, app, rec, etag); err != nil { |
| - return nil, err |
| - } |
| - } |
| // The key this walk started from, for the step-6 retry's guard below. "" |
| // is the ordinary case: a first grant, or a re-run of one that died before |
| // it recorded anything. |
| @@ -186,12 +182,51 @@ func Grant(ctx context.Context, s release.Store, p Provisioner, d Deployment, |
| if !ValidBucketName(bucket) { |
| return nil, fmt.Errorf("objstore: invalid bucket name %q", bucket) |
| } |
| - if rec.Grant == nil { |
| + // One walk at a time on this bucket name, across every process |
| + // (walklock.go). Everything below runs inside the lease and stops |
| + // before it ends. |
| + walk, err := acquireBucketWalk(ctx, s, bucket, acct, app, "grant") |
| + if err != nil { |
| + return nil, err |
| + } |
| + defer func() { walk.release(walkEndedClean(walkErr)) }() |
| + ctx, cancel := walk.bound(ctx) |
| + defer cancel() |
| + // The record again, now that no other walk can be changing it. |
| + if rec, etag, err = Get(ctx, s, acct, app); err != nil { |
| + return nil, err |
| + } |
| + if rec == nil || rec.Declared == nil { |
| + return nil, ErrNotDeclared |
| + } |
| + if held := rec.Grant; held != nil && held.Bucket != "" && held.Bucket != bucket { |
| + return nil, fmt.Errorf("objstore: store record for %s/%s changed while its walk was starting — re-run: %w", acct, app, release.ErrCASConflict) |
| + } |
| + rebuilt = rec.Teardown != nil |
| + startKeyID = "" |
| + if rec.Provisioned != nil { |
| + startKeyID = rec.Provisioned.KeyID |
| + } |
| + wroteGrant := rec.Grant == nil |
| + if wroteGrant { |
| rec.Grant = &GrantBlock{Backend: d.Backend, Bucket: bucket, Source: source, At: now, By: actor, Minted: adoptBucket == ""} |
| if etag, err = Put(ctx, s, acct, app, rec, etag); err != nil { |
| return nil, err |
| } |
| } |
| + // refused undoes the grant block this call wrote when the name turns |
| + // out not to be this store's to take. A store whose grant was refused |
| + // holds nothing, and its record must not say otherwise: another |
| + // store's teardown reads these records to find who holds a bucket. |
| + refused := func(cause error) error { |
| + if wroteGrant { |
| + rec.Grant = nil |
| + if _, perr := Put(ctx, s, acct, app, rec, etag); perr != nil { |
| + return errors.Join(cause, fmt.Errorf("objstore: undo the grant block of %s/%s: %w", acct, app, perr)) |
| + } |
| + } |
| + return cause |
| + } |
| // 1b. bucket ownership history (ownership.go). Opened here, before the |
| // idempotence short-circuit and before any credential exists, so that |
| // no tenant can move a byte out of this bucket while the ledger names |
| @@ -226,8 +261,44 @@ func Grant(ctx context.Context, s release.Store, p Provisioner, d Deployment, |
| if rec.Grant.Minted && rec.Provisioned == nil { |
| prober, canProbe := p.(BucketProber) |
| inventory, canList := p.(GranteeProber) |
| + exists, perr := false, error(nil) |
| + if canProbe { |
| + exists, perr = prober.BucketExists(ctx, bucket) |
| + } |
| + // A grant that mints its bucket and has never held the name must |
| + // not be handed a bucket that is already there: it holds somebody's |
| + // objects (a store that left it behind, or one the platform does |
| + // not know). The ledger says whether this store held the name |
| + // before; a re-run of its own crashed grant did, and goes on. |
| + ledger, _, lerr := ReadBucketOwnership(ctx, s, bucket) |
| + if lerr != nil { |
| + return nil, fmt.Errorf("objstore: read ownership of %s: %w", bucket, lerr) |
| + } |
| + held := false |
| + if ledger != nil { |
| + for _, iv := range ledger.Intervals { |
| + if iv.Account == acct && iv.App == app { |
| + held = true |
| + } |
| + } |
| + } |
| + if !held { |
| + if canProbe && perr == nil && exists { |
| + return nil, refused(fmt.Errorf("%w: %s already exists and %s/%s has never held it", ErrBucketHeld, bucket, acct, app)) |
| + } |
| + // And never while another store's RECORD names the bucket. A |
| + // store provisioned before ledgers existed holds its bucket |
| + // with no interval to show for it; the ledger's own check |
| + // below cannot see it, and its record can. |
| + holders, herr := otherStoreHolders(ctx, s, bucket, acct, app) |
| + if herr != nil { |
| + return nil, fmt.Errorf("objstore: list the stores holding %s: %w", bucket, herr) |
| + } |
| + if len(holders) > 0 { |
| + return nil, refused(fmt.Errorf("%w: %s by %s/%s", ErrBucketHeld, bucket, holders[0].Account, holders[0].App)) |
| + } |
| + } |
| if canProbe && canList { |
| - exists, perr := prober.BucketExists(ctx, bucket) |
| grantees, gerr := inventory.BucketGrantees(ctx, bucket) |
| minted = perr == nil && gerr == nil && !exists && len(grantees) == 0 |
| observed = now.Add(grantElapsed(started)) |
| @@ -238,7 +309,11 @@ func Grant(ctx context.Context, s release.Store, p Provisioner, d Deployment, |
| // grant is refused while another app's teardown is destroying the |
| // bucket. Both are decided by the ledger write itself. |
| if err := openBucketOwner(ctx, s, bucket, acct, app, grantOwnershipFrom(rec.Grant, now), observed, minted, rec.Grant.Minted); err != nil { |
| - return nil, fmt.Errorf("objstore: record ownership of %s: %w", bucket, err) |
| + err = fmt.Errorf("objstore: record ownership of %s: %w", bucket, err) |
| + if errors.Is(err, ErrBucketHeld) || errors.Is(err, ErrBucketBeingDestroyed) { |
| + return nil, refused(err) |
| + } |
| + return nil, err |
| } |
| // 2. the principal and its policy. EnsureUser runs UNCONDITIONALLY, before |
| // the idempotence short-circuit below — the same fix internal/mail's walk |
| @@ -317,6 +392,7 @@ func Grant(ctx context.Context, s release.Store, p Provisioner, d Deployment, |
| } |
| rec.Provisioned = &Provisioned{Bucket: bucket, Region: region, Endpoint: d.Endpoint, |
| IAMUser: user, UserARN: arn, KeyID: id, PrevKeyID: prev, At: now} |
| + rec.Teardown = nil // the store is whole again |
| if _, err := Put(ctx, s, acct, app, rec, etag); err != nil { |
| if !errors.Is(err, release.ErrCASConflict) { |
| return nil, err |
| @@ -351,6 +427,7 @@ func Grant(ctx context.Context, s release.Store, p Provisioner, d Deployment, |
| return nil, err |
| } |
| fresh.Provisioned = rec.Provisioned |
| + fresh.Teardown = nil |
| if _, perr := Put(ctx, s, acct, app, fresh, freshEtag); perr != nil { |
| return nil, perr |
| } |
| @@ -525,8 +602,8 @@ func RotateFinish(ctx context.Context, s release.Store, p Provisioner, acct, app |
| // it as trivially deletable would erase the only record naming that estate, |
| // which is precisely the failure §9's ordering rule exists to prevent. Only a |
| // record with NO grant at all — declared, never acted on — dies trivially. |
| -func Teardown(ctx context.Context, s release.Store, p Provisioner, sc Scanner, acct, app string) error { |
| - rec, _, err := Get(ctx, s, acct, app) |
| +func Teardown(ctx context.Context, s release.Store, p Provisioner, sc Scanner, acct, app string) (walkErr error) { |
| + rec, etag, err := Get(ctx, s, acct, app) |
| if err != nil { |
| return err |
| } |
| @@ -555,6 +632,53 @@ func Teardown(ctx context.Context, s release.Store, p Provisioner, sc Scanner, a |
| if p == nil { |
| return fmt.Errorf("objstore: %s/%s has backend estate and this deployment has no storage backend configured", acct, app) |
| } |
| + // One walk at a time on this bucket name, across every process |
| + // (walklock.go). The lease is this invocation's alone: a second |
| + // teardown of the same app waits, and cannot inherit this one's |
| + // authority or keep it after this one has finished. |
| + leased := "" |
| + switch { |
| + case rec.Provisioned != nil: |
| + leased = rec.Provisioned.Bucket |
| + case rec.Grant != nil: |
| + leased = rec.Grant.Bucket |
| + } |
| + if leased != "" && ValidBucketName(leased) { |
| + walk, werr := acquireBucketWalk(ctx, s, leased, acct, app, "teardown") |
| + if werr != nil { |
| + return werr |
| + } |
| + defer func() { walk.release(walkEndedClean(walkErr)) }() |
| + var cancel context.CancelFunc |
| + ctx, cancel = walk.bound(ctx) |
| + defer cancel() |
| + // The record again, now that no other walk can be changing it. |
| + if rec, etag, err = Get(ctx, s, acct, app); err != nil { |
| + return err |
| + } |
| + if rec == nil { |
| + return nil |
| + } |
| + startKeyID, startBucket = "", "" |
| + if rec.Provisioned != nil { |
| + startKeyID = rec.Provisioned.KeyID |
| + } |
| + if rec.Grant != nil { |
| + startBucket = rec.Grant.Bucket |
| + } |
| + if rec.Grant == nil && rec.Provisioned == nil { |
| + return deleteRecordIfUnchanged(ctx, s, acct, app, startKeyID, startBucket) |
| + } |
| + } |
| + // Written ahead of everything else: from here on the record says a |
| + // teardown has begun, whatever it gets through before it stops. A grant |
| + // that finds this block does not take the Provisioned block at its word. |
| + if rec.Teardown == nil { |
| + rec.Teardown = &TeardownBlock{StartedAt: ownershipNow()} |
| + if etag, err = Put(ctx, s, acct, app, rec, etag); err != nil { |
| + return fmt.Errorf("objstore: record the teardown of %s/%s: %w", acct, app, err) |
| + } |
| + } |
| // Coordinates: the provisioned block when there is one, since the bucket |
| // an app's data is in is whatever was recorded at provision time. Falling |
| // back to the grant covers the crashed-grant shape above — and the user |
| @@ -591,7 +715,7 @@ func Teardown(ctx context.Context, s release.Store, p Provisioner, sc Scanner, a |
| // |
| // A teardown that already deleted its bucket says so on its own record |
| // (TeardownBlock) and never touches a bucket of that name again. |
| - destroy, scannerOwed := false, false |
| + destroy, scannerOwed, foreign := false, false, false |
| bucketDone := rec.Teardown != nil && !rec.Teardown.BucketDeletedAt.IsZero() |
| var holder OwnerRef |
| if bucket != "" && ValidBucketName(bucket) && !bucketDone { |
| @@ -602,7 +726,7 @@ func Teardown(ctx context.Context, s release.Store, p Provisioner, sc Scanner, a |
| now := ownershipNow() |
| scanning := rec.Scanning != nil && rec.Scanning.Enabled != nil |
| var derr error |
| - destroy, scannerOwed, holder, derr = beginBucketTeardown(ctx, s, bucket, acct, app, grantOwnershipFrom(rec.Grant, now), holders, scanning, now) |
| + destroy, scannerOwed, holder, foreign, derr = beginBucketTeardown(ctx, s, bucket, acct, app, grantOwnershipFrom(rec.Grant, now), holders, scanning, now) |
| if derr != nil { |
| return fmt.Errorf("objstore: decide the teardown of %s: %w", bucket, derr) |
| } |
| @@ -646,6 +770,11 @@ func Teardown(ctx context.Context, s release.Store, p Provisioner, sc Scanner, a |
| // the marker standing no grant can have landed, so a change here is |
| // an operator's hand on the ledger; the teardown stops and is re-run. |
| confirm := func(step string) error { |
| + // The lease first: a walk whose time has run out does not |
| + // start another destructive call, whatever the ledger says. |
| + if err := ctx.Err(); err != nil { |
| + return fmt.Errorf("objstore: the teardown of %s/%s ran out of its lease before %s; run it again: %w", acct, app, step, err) |
| + } |
| if !ValidBucketName(bucket) { |
| return nil |
| } |
| @@ -674,16 +803,14 @@ func Teardown(ctx context.Context, s release.Store, p Provisioner, sc Scanner, a |
| return fmt.Errorf("objstore: delete bucket %s: %w", bucket, err) |
| } |
| // Recorded at once: a re-run after a failure below must not empty or |
| - // delete a bucket of this name again. |
| - fresh, fetag, gerr := Get(ctx, s, acct, app) |
| - if gerr != nil { |
| - return fmt.Errorf("objstore: re-read store record for %s/%s: %w", acct, app, gerr) |
| - } |
| - if fresh != nil { |
| - fresh.Teardown = &TeardownBlock{BucketDeletedAt: ownershipNow()} |
| - if _, perr := Put(ctx, s, acct, app, fresh, fetag); perr != nil { |
| - return fmt.Errorf("objstore: record the deleted bucket for %s/%s: %w", acct, app, perr) |
| - } |
| + // delete a bucket of this name again. The mark is written against |
| + // the record exactly as this teardown last wrote it (its etag), never |
| + // against a fresh read: a record that has changed since describes |
| + // some other estate, and saying ITS bucket was deleted would be |
| + // false. If it has changed the write loses, and the walk stops. |
| + rec.Teardown.BucketDeletedAt = ownershipNow() |
| + if etag, err = Put(ctx, s, acct, app, rec, etag); err != nil { |
| + return fmt.Errorf("objstore: record the deleted bucket for %s/%s: %w", acct, app, err) |
| } |
| } |
| // DeleteUser reaps the keys and the inline policy, so the credential dies |
| @@ -707,16 +834,47 @@ func Teardown(ctx context.Context, s release.Store, p Provisioner, sc Scanner, a |
| return fmt.Errorf("objstore: close ownership of %s: %w", bucket, err) |
| } |
| } |
| - // Left in place for an app of ANOTHER account: the bucket may still hold |
| - // this app's objects, and that account can read them. This app's access |
| + // Left in place while an app of ANOTHER account can still reach it (it |
| + // holds the bucket, or it is leaving and its credentials are not yet |
| + // revoked): the bucket may still hold this app's objects, and that |
| + // account can read them. This app's access |
| // is revoked and its interval closed, and the walk stops here with the |
| // record standing, so the teardown is not reported as done and a person |
| // decides what happens to the data. Run again once the other store is |
| // gone, it destroys the bucket and finishes. An app of the SAME account |
| // holding the bucket is the operator's own sharing, and the teardown |
| // finishes quietly. |
| - if !destroy && !bucketDone && holder.Account != "" && holder.Account != acct { |
| - return fmt.Errorf("%w: %s is held by %s/%s; %s/%s's access is revoked and its record kept", ErrBucketRetained, bucket, holder.Account, holder.App, acct, app) |
| + // The decision was made before this app's credentials were revoked, and |
| + // an operator may have adopted the bucket for another account since. |
| + // The ledger is read once more, so the answer given here is about now. |
| + // |
| + // And one thing lifts the hold: the bucket this store left has since |
| + // been destroyed by its last holder (DestroyedAt is later than this |
| + // store's own leaving). Its objects went with it. Whoever holds a |
| + // bucket of that name now was granted a new one, and this teardown has |
| + // nothing left to wait for. |
| + if !destroy && !bucketDone && bucket != "" && ValidBucketName(bucket) { |
| + latest, _, lerr := ReadBucketOwnership(ctx, s, bucket) |
| + if lerr != nil { |
| + return fmt.Errorf("objstore: re-read ownership of %s: %w", bucket, lerr) |
| + } |
| + if latest != nil { |
| + var left *time.Time |
| + for _, iv := range latest.Intervals { |
| + if iv.open() && iv.Account != acct { |
| + foreign, holder = true, OwnerRef{Account: iv.Account, App: iv.App, Since: iv.From} |
| + } |
| + if iv.Account == acct && iv.App == app && iv.Until != nil && (left == nil || iv.Until.After(*left)) { |
| + left = iv.Until |
| + } |
| + } |
| + if d := latest.DestroyedAt; foreign && d != nil && left != nil && left.Before(*d) { |
| + foreign = false |
| + } |
| + } |
| + } |
| + if !destroy && !bucketDone && foreign { |
| + return fmt.Errorf("%w: %s is held by %s/%s, and a store of another account can still reach it; %s/%s's access is revoked and its record kept", ErrBucketRetained, bucket, holder.Account, holder.App, acct, app) |
| } |
| // The delivered set goes even though the credentials behind it are already |
| // dead: leaving them would hand the app's next render a bucket name and a |
| @@ -771,6 +929,16 @@ func Teardown(ctx context.Context, s release.Store, p Provisioner, sc Scanner, a |
| return deleteRecordIfUnchanged(ctx, s, acct, app, startKeyID, startBucket) |
| } |
| +// walkEndedClean reports whether a walk that returned err left nothing on |
| +// its way to the backend: it succeeded, or it was refused before it called |
| +// the backend, or it stopped on a decision (a retained bucket) after every |
| +// call it made had been answered. Any other error may be a call that timed |
| +// out here and can still run there, so the lease is not freed. |
| +func walkEndedClean(err error) bool { |
| + return err == nil || errors.Is(err, ErrBucketRetained) || errors.Is(err, ErrBucketHeld) || |
| + errors.Is(err, ErrBucketBeingDestroyed) || errors.Is(err, ErrNotDeclared) |
| +} |
| + |
| // grantElapsed is how long a grant has been running. A seam, so a test can |
| // make a probe slow. |
| var grantElapsed = func(started time.Time) time.Duration { return time.Since(started) } |
diff --git a/internal/objstore/walk_sim_test.go b/internal/objstore/walk_sim_test.go| new file mode 100644 |
| index 000000000..a897db940 |
| --- /dev/null |
| +++ b/internal/objstore/walk_sim_test.go |
| @@ -0,0 +1,366 @@ |
| +package objstore |
| + |
| +import ( |
| + "context" |
| + "errors" |
| + "fmt" |
| + "math/rand" |
| + "os" |
| + "strconv" |
| + "strings" |
| + "testing" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/release" |
| +) |
| + |
| +// The invariants of a bucket name shared by several stores, checked under |
| +// random interleavings of grants and teardowns with a failure injected at |
| +// a random provider step. The earlier tests each pin one interleaving a |
| +// reviewer found; this one states what must hold after EVERY step of ANY |
| +// interleaving, and what must be reachable at the end. |
| +// |
| +// Data. A store that was granted and has not been asked to go still |
| +// has its bucket and the object it wrote. |
| +// Marker. The ledger names an app as destroying the bucket only while |
| +// that app's teardown is unfinished. |
| +// Scanner. A scanner plan never outlives its bucket. |
| +// Gone. A store whose teardown finished has no user and no record. |
| +// Owner. The ledger resolves an hour to an app only if, at every |
| +// instant sampled in that hour, no other app had credentials. |
| +// Mint. A grant that mints its bucket is never handed a bucket that |
| +// holds the objects of another account's store that has gone. |
| +// Quiet. A teardown that reports success has not left its store's |
| +// objects where another account's credentials can read them. |
| +// |
| +// The walks are not only run one after another. With some probability a |
| +// second walk (another app's grant or teardown) runs in the MIDDLE of the |
| +// first, at the moment the first calls the provider to create, empty or |
| +// delete a bucket or a user: the interleaving of two console processes. |
| +// |
| +// Settles. With no more failures, re-running every unfinished teardown |
| +// reaches: no bucket, no user, no record, no scanner plan, no |
| +// open interval, no marker. Nothing is stuck for good. |
| +type simApp struct { |
| + acct, app, adopt string |
| + // "none": no store. "live": granted, owed its data. "broken": a grant |
| + // or teardown stopped part-way. "retained": its teardown revoked its |
| + // access and reported that the bucket was left to another account. |
| + state string |
| +} |
| + |
| +func (a *simApp) name() string { return a.acct + "/" + a.app } |
| + |
| +// interleavingProvisioner runs between() at the start of each destructive |
| +// or creating provider call, before the call takes effect. |
| +type interleavingProvisioner struct { |
| + *MemProvisioner |
| + between func(method string) |
| +} |
| + |
| +func (p *interleavingProvisioner) EnsureBucket(ctx context.Context, name string) (string, error) { |
| + p.between("EnsureBucket") |
| + return p.MemProvisioner.EnsureBucket(ctx, name) |
| +} |
| + |
| +func (p *interleavingProvisioner) EmptyBucket(ctx context.Context, name string) error { |
| + p.between("EmptyBucket") |
| + return p.MemProvisioner.EmptyBucket(ctx, name) |
| +} |
| + |
| +func (p *interleavingProvisioner) DeleteBucket(ctx context.Context, name string) error { |
| + p.between("DeleteBucket") |
| + return p.MemProvisioner.DeleteBucket(ctx, name) |
| +} |
| + |
| +func (p *interleavingProvisioner) DeleteUser(ctx context.Context, user string) error { |
| + p.between("DeleteUser") |
| + return p.MemProvisioner.DeleteUser(ctx, user) |
| +} |
| + |
| +func TestBucketNameInvariantsUnderRandomWalks(t *testing.T) { |
| + seeds := 400 |
| + if n, err := strconv.Atoi(os.Getenv("CARLOS_SIM_SEEDS")); err == nil && n > 0 { |
| + seeds = n |
| + } |
| + if testing.Short() { |
| + seeds = 60 |
| + } |
| + for seed := 0; seed < seeds; seed++ { |
| + if trace, problem := simulateBucketName(t, int64(seed), 40); problem != "" { |
| + t.Fatalf("seed %d: %s\n%s", seed, problem, strings.Join(trace, "\n")) |
| + } |
| + } |
| +} |
| + |
| +func simulateBucketName(t *testing.T, seed int64, steps int) (trace []string, problem string) { |
| + t.Helper() |
| + ctx := context.Background() |
| + rng := rand.New(rand.NewSource(seed)) |
| + s := release.NewDirStore(t.TempDir()) |
| + _, pub := testSealKeypair(t) |
| + mem := NewMemProvisioner("eu-west-1") |
| + ip := &interleavingProvisioner{MemProvisioner: mem, between: func(string) {}} |
| + var p Provisioner = ip |
| + sc := NewMemScanner() |
| + apps := []*simApp{ |
| + {acct: "bab", app: "titogo", adopt: "", state: "none"}, // mints the name |
| + {acct: "bac", app: "other", adopt: ledgerBucket, state: "none"}, // another account adopts it |
| + {acct: "bab", app: "second", adopt: ledgerBucket, state: "none"}, // the same account adopts it |
| + } |
| + faults := []string{"EnsureBucket", "EnsureUser", "MintKey", "EmptyBucket", "DeleteBucket", "DeleteUser", "BucketExists", "BucketGrantees"} |
| + logf := func(format string, args ...any) { trace = append(trace, fmt.Sprintf(format, args...)) } |
| + object := func(a *simApp) string { return "obj-" + a.acct + "-" + a.app } |
| + |
| + mintLeak := "" |
| + grant := func(a *simApp, at time.Time) { |
| + if rec, _, _ := Get(ctx, s, a.acct, a.app); rec == nil { |
| + declare(t, s, a.acct, a.app, "S3") |
| + } |
| + was := a.state |
| + _, err := Grant(ctx, s, p, testDeployment(), pub, a.acct, a.app, "op", GrantSourceOperator, a.adopt, at) |
| + logf("grant %s: %v", a.name(), err) |
| + if err != nil { |
| + // A refused or failed grant does not un-report a retained |
| + // bucket, and does not take a live store's data away. |
| + if a.state != "live" && a.state != "retained" { |
| + a.state = "broken" |
| + } |
| + return |
| + } |
| + a.state = "live" |
| + // An app an operator adopted the name for while it is held (and |
| + // which is live now) is sharing by the operator's choice. Objects |
| + // of a store that is not holding the bucket are a leak. |
| + if a.adopt == "" && was != "live" { |
| + for key := range mem.Buckets[ledgerBucket] { |
| + for _, o := range apps { |
| + // A sibling app of the same account sharing the |
| + // bucket is the operator's own arrangement. |
| + if key == object(o) && o != a && o.acct != a.acct && o.state != "live" && o.state != "retained" { |
| + mintLeak = fmt.Sprintf("Mint: %s minted its bucket and found %s in it, left by a store that no longer holds it", a.name(), key) |
| + } |
| + } |
| + } |
| + } |
| + if b, ok := mem.Buckets[ledgerBucket]; ok { |
| + b[object(a)] = []byte(a.name()) |
| + } |
| + if rng.Intn(3) == 0 { |
| + if err := sc.EnsureEnabled(ctx, ledgerBucket); err != nil { |
| + t.Fatal(err) |
| + } |
| + enableScanning(t, s, a.acct, a.app) |
| + logf(" scanning enabled by %s", a.name()) |
| + } |
| + } |
| + teardown := func(a *simApp) error { |
| + rec, _, _ := Get(ctx, s, a.acct, a.app) |
| + if rec == nil { |
| + return nil |
| + } |
| + // Asked to go: from this instant the store is not owed its data. |
| + a.state = "broken" |
| + estate := rec.Grant != nil || rec.Provisioned != nil |
| + err := Teardown(ctx, s, p, sc, a.acct, a.app) |
| + logf("teardown %s: %v", a.name(), err) |
| + if errors.Is(err, ErrBucketRetained) { |
| + // Reported, not hidden: the bucket was left to another |
| + // account and may hold this store's objects. A person decides. |
| + a.state = "retained" |
| + return err |
| + } |
| + if err != nil { |
| + return err |
| + } |
| + a.state = "none" |
| + // A teardown that reports success has not left this store's |
| + // objects where another ACCOUNT can read them. |
| + // (Only a teardown that had something to take apart: a later no-op |
| + // on a bare declaration is not what left anything behind.) |
| + if b, ok := mem.Buckets[ledgerBucket]; ok && estate && b[object(a)] != nil { |
| + for _, o := range apps { |
| + if _, has := mem.Users[UserName(o.acct, o.app)]; has && o.acct != a.acct { |
| + mintLeak = fmt.Sprintf("Quiet: %s's teardown reported success and left its object readable by %s", a.name(), o.name()) |
| + } |
| + } |
| + } |
| + return nil |
| + } |
| + check := func() string { |
| + if mintLeak != "" { |
| + return mintLeak |
| + } |
| + bucket, exists := mem.Buckets[ledgerBucket] |
| + for _, a := range apps { |
| + switch a.state { |
| + case "live": |
| + if !exists { |
| + return fmt.Sprintf("Data: %s is live and its bucket is gone", a.name()) |
| + } |
| + if string(bucket[object(a)]) != a.name() { |
| + return fmt.Sprintf("Data: %s is live and its object is gone", a.name()) |
| + } |
| + case "none": |
| + if _, ok := mem.Users[UserName(a.acct, a.app)]; ok { |
| + return fmt.Sprintf("Gone: %s is torn down and still has a user", a.name()) |
| + } |
| + if rec, _, _ := Get(ctx, s, a.acct, a.app); rec != nil && (rec.Grant != nil || rec.Provisioned != nil) { |
| + return fmt.Sprintf("Gone: %s is torn down and still has a granted record", a.name()) |
| + } |
| + } |
| + } |
| + if !exists && sc.IsEnabled(ledgerBucket) { |
| + return "Scanner: the plan outlived the bucket" |
| + } |
| + ledger, _, err := ReadBucketOwnership(ctx, s, ledgerBucket) |
| + if err != nil { |
| + return "ledger unreadable: " + err.Error() |
| + } |
| + if ledger != nil && ledger.Destroying != nil { |
| + for _, a := range apps { |
| + if a.acct == ledger.Destroying.Account && a.app == ledger.Destroying.App && a.state != "broken" && a.state != "retained" { |
| + return fmt.Sprintf("Marker: the ledger says %s is destroying the bucket and its state is %s", a.name(), a.state) |
| + } |
| + } |
| + } |
| + return "" |
| + } |
| + |
| + at := t0 |
| + prevNow := ownershipNow |
| + ownershipNow = func() time.Time { return at } |
| + defer func() { ownershipNow = prevNow }() |
| + // The walk leases run on the same simulated clock. Each step is an |
| + // hour after the last, so a walk that failed has run out of its lease |
| + // by the next step; a walk that runs in the middle of another meets |
| + // the other's lease, as two processes would. |
| + prevWalk := walkClock |
| + walkClock = func() time.Time { return at } |
| + defer func() { walkClock = prevWalk }() |
| + // Who had credentials at each sampled instant. |
| + type sample struct { |
| + at time.Time |
| + users map[string]bool |
| + } |
| + var samples []sample |
| + snapshot := func() { |
| + users := map[string]bool{} |
| + for _, a := range apps { |
| + if _, ok := mem.Users[UserName(a.acct, a.app)]; ok { |
| + users[a.name()] = true |
| + } |
| + } |
| + samples = append(samples, sample{at, users}) |
| + } |
| + owner := func() string { |
| + ledger, _, err := ReadBucketOwnership(ctx, s, ledgerBucket) |
| + if err != nil || ledger == nil { |
| + return "" |
| + } |
| + for _, sm := range samples { |
| + acct, app, ok := ledger.OwnerAt(sm.at) |
| + if !ok { |
| + continue |
| + } |
| + for name := range sm.users { |
| + if name != acct+"/"+app { |
| + return fmt.Sprintf("Owner: the hour of %s resolves to %s/%s while %s had credentials", sm.at.Format("01-02 15:04"), acct, app, name) |
| + } |
| + } |
| + } |
| + return "" |
| + } |
| + for step := 0; step < steps; step++ { |
| + // Two samples an hour: one just after the step, one mid-hour. |
| + at = at.Add(30 * time.Minute) |
| + snapshot() |
| + at = at.Add(30 * time.Minute) |
| + a := apps[rng.Intn(len(apps))] |
| + mem.FailNext = nil |
| + if rng.Intn(3) == 0 { |
| + f := faults[rng.Intn(len(faults))] |
| + mem.FailNext = map[string]error{f: errors.New("injected " + f)} |
| + logf(" next %s fails", f) |
| + } |
| + // Sometimes another app's walk runs in the middle of this one, at |
| + // one of its provider calls. Only one level deep, and the data |
| + // invariant is checked the moment the inner walk returns: a store |
| + // granted inside another's teardown is owed its data from then on. |
| + inner := false |
| + midProblem := "" |
| + ip.between = func(method string) { |
| + if inner || rng.Intn(4) != 0 { |
| + return |
| + } |
| + inner = true |
| + b := apps[rng.Intn(len(apps))] |
| + if b == a { |
| + inner = false |
| + return |
| + } |
| + logf(" -- during %s of %s:", method, a.name()) |
| + if rng.Intn(2) == 0 { |
| + grant(b, at) |
| + } else { |
| + _ = teardown(b) |
| + } |
| + logf(" -- back to %s", a.name()) |
| + inner = false |
| + } |
| + if rng.Intn(2) == 0 { |
| + grant(a, at) |
| + } else { |
| + _ = teardown(a) |
| + } |
| + ip.between = func(string) {} |
| + if midProblem != "" { |
| + return trace, midProblem |
| + } |
| + snapshot() |
| + if problem := check(); problem != "" { |
| + return trace, problem |
| + } |
| + if problem := owner(); problem != "" { |
| + return trace, problem |
| + } |
| + } |
| + // No more failures. Everything unfinished is re-run until nothing moves. |
| + mem.FailNext = nil |
| + ip.between = func(string) {} |
| + logf("-- settle --") |
| + for round := 0; round < 8; round++ { |
| + for _, a := range apps { |
| + at = at.Add(time.Hour) |
| + _ = teardown(a) |
| + } |
| + if problem := check(); problem != "" { |
| + return trace, "while settling: " + problem |
| + } |
| + } |
| + for _, a := range apps { |
| + if rec, _, _ := Get(ctx, s, a.acct, a.app); rec != nil && (rec.Grant != nil || rec.Provisioned != nil) { |
| + return trace, fmt.Sprintf("Settles: %s still has a granted record", a.name()) |
| + } |
| + if _, ok := mem.Users[UserName(a.acct, a.app)]; ok { |
| + return trace, fmt.Sprintf("Settles: %s still has a user", a.name()) |
| + } |
| + } |
| + if _, exists := mem.Buckets[ledgerBucket]; exists { |
| + return trace, "Settles: the bucket is still there with nobody holding it" |
| + } |
| + if sc.IsEnabled(ledgerBucket) { |
| + return trace, "Settles: the scanner plan is still there" |
| + } |
| + if ledger, _, _ := ReadBucketOwnership(ctx, s, ledgerBucket); ledger != nil { |
| + if ledger.Destroying != nil { |
| + return trace, fmt.Sprintf("Settles: the ledger still says %s/%s is destroying the bucket", ledger.Destroying.Account, ledger.Destroying.App) |
| + } |
| + for _, iv := range ledger.Intervals { |
| + if iv.Until == nil { |
| + return trace, fmt.Sprintf("Settles: an interval is still open for %s/%s", iv.Account, iv.App) |
| + } |
| + } |
| + } |
| + return trace, "" |
| +} |
diff --git a/internal/objstore/walklock.go b/internal/objstore/walklock.go| new file mode 100644 |
| index 000000000..0eea6e2fe |
| --- /dev/null |
| +++ b/internal/objstore/walklock.go |
| @@ -0,0 +1,209 @@ |
| +package objstore |
| + |
| +import ( |
| + "context" |
| + "crypto/rand" |
| + "encoding/hex" |
| + "encoding/json" |
| + "errors" |
| + "fmt" |
| + "os" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/release" |
| +) |
| + |
| +// The walk lease: one walk at a time on a bucket name, across every |
| +// console process, with authority bound to the invocation that holds it. |
| +// |
| +// Why it exists. A grant and a teardown each make several provider calls, |
| +// and what they are entitled to do is decided from the ownership ledger. |
| +// The console's own lock is per process. Two processes could therefore run |
| +// two walks on one bucket at once, and every protection written into the |
| +// ledger named an APP, not an invocation: a second teardown of the same |
| +// app inherited the first one's authority to destroy, and kept it after |
| +// the first had finished and the name had been granted to someone else. |
| +// Each review round found another interleaving of that one fault. |
| +// |
| +// The invariant. No provider call of a walk executes outside that walk's |
| +// lease, and leases on one bucket name do not overlap. It is kept by two |
| +// things together: |
| +// |
| +// - the lease is one object, taken and extended by compare-and-swap, so |
| +// two invocations cannot both hold it; |
| +// - every call the walk makes carries a context deadline that falls |
| +// before the lease ends. A process that is paused past its lease wakes |
| +// to a deadline that has passed, and its next call does not start. |
| +// |
| +// The deadline only stops a process that is running. A process paused |
| +// between its last check and the request leaving it is stopped by the |
| +// backend, not by this: see fence.go. The lease is as long as the |
| +// backend's window for a request stamped at the lease's start, plus an |
| +// allowance for clock skew, so a lease cannot be taken over while any |
| +// request of the walk that held it could still be accepted. |
| +// |
| +// A walk that ends with an error may have a request still on its way (a |
| +// call that timed out here can run there). Its lease is therefore not |
| +// freed: it is marked ended, and until it runs out only the same app |
| +// running the same kind of walk again may take it. A walk that ends |
| +// cleanly had every call answered, and frees the name at once, unless an |
| +// earlier attempt it took over from had not run out yet. |
| +const ( |
| + // walkLease is how long a bucket name stays reserved for one walk. |
| + walkLease = fenceWindow + fenceSkew |
| + // walkMargin is how long before the backend's window closes the walk |
| + // stops calling the provider. Emptying a large bucket is paginated and |
| + // resumable: a walk that runs out of time stops, and running it again |
| + // takes a new lease and carries on. |
| + walkMargin = 30 * time.Second |
| +) |
| + |
| +// ErrBucketBusy: another walk holds the bucket's lease. |
| +var ErrBucketBusy = errors.New("objstore: another walk is in progress on this bucket") |
| + |
| +// walkClock is the wall clock leases are measured on. A seam for tests. |
| +var walkClock = time.Now |
| + |
| +// walkLockKey is the lease object for a bucket name. It is deliberately |
| +// not under the ownership ledger's prefix: it is coordination, not |
| +// evidence, and nothing that reads ledgers should ever meet it. |
| +func walkLockKey(bucket string) string { return "objstore/walks/" + bucket + ".json" } |
| + |
| +type walkLock struct { |
| + V int `json:"v"` |
| + Bucket string `json:"bucket"` |
| + Account string `json:"account_sqid"` |
| + App string `json:"app"` |
| + Kind string `json:"kind"` // "grant" or "teardown" |
| + Token string `json:"token"` |
| + // SignedAt is the instant the lease was taken: the stamp every request |
| + // of the walk carries, and what the backend's window is measured from. |
| + SignedAt time.Time `json:"signed_at"` |
| + // Until is when the name stops being reserved. |
| + Until time.Time `json:"until"` |
| + // Ended: the walk has returned, with an error. The reservation stands |
| + // until Until, for anyone but the same app re-running the same kind. |
| + Ended bool `json:"ended,omitempty"` |
| + // Carry is the latest Until of the ended attempts this lease took over |
| + // from: the name is not freed before it. |
| + Carry time.Time `json:"carry,omitzero"` |
| +} |
| + |
| +// bucketWalk is a held lease. |
| +type bucketWalk struct { |
| + s release.Store |
| + key string |
| + lock walkLock |
| + etag string |
| + closed bool |
| +} |
| + |
| +// acquireBucketWalk takes the bucket's lease for one invocation, or |
| +// refuses with ErrBucketBusy naming who holds it. A lease whose time has |
| +// passed is taken over; one that was released is free. |
| +func acquireBucketWalk(ctx context.Context, s release.Store, bucket, acct, app, kind string) (*bucketWalk, error) { |
| + var raw [12]byte |
| + if _, err := rand.Read(raw[:]); err != nil { |
| + return nil, err |
| + } |
| + key := walkLockKey(bucket) |
| + for attempt := 0; attempt < 3; attempt++ { |
| + now := walkClock() |
| + var carry time.Time |
| + prev, etag, err := s.Get(ctx, key) |
| + switch { |
| + case errors.Is(err, os.ErrNotExist): |
| + etag = "" |
| + case err != nil: |
| + return nil, fmt.Errorf("objstore: read the walk lease of %s: %w", bucket, err) |
| + default: |
| + var held walkLock |
| + if jerr := json.Unmarshal(prev, &held); jerr != nil { |
| + return nil, fmt.Errorf("objstore: walk lease of %s: %w", bucket, jerr) |
| + } |
| + if held.Until.After(now) { |
| + retry := held.Ended && held.Account == acct && held.App == app && held.Kind == kind |
| + if !retry { |
| + return nil, fmt.Errorf("%w: %s, by a %s of %s/%s until %s", ErrBucketBusy, bucket, held.Kind, held.Account, held.App, held.Until.UTC().Format(time.RFC3339)) |
| + } |
| + carry = held.Until |
| + if held.Carry.After(carry) { |
| + carry = held.Carry |
| + } |
| + } |
| + } |
| + lock := walkLock{V: 1, Bucket: bucket, Account: acct, App: app, Kind: kind, Token: hex.EncodeToString(raw[:]), SignedAt: now.UTC(), Until: now.Add(walkLease).UTC(), Carry: carry} |
| + body, err := json.Marshal(lock) |
| + if err != nil { |
| + return nil, err |
| + } |
| + newEtag, err := s.Put(ctx, key, body, etag) |
| + if err == nil { |
| + return &bucketWalk{s: s, key: key, lock: lock, etag: newEtag}, nil |
| + } |
| + if !errors.Is(err, release.ErrCASConflict) { |
| + return nil, fmt.Errorf("objstore: take the walk lease of %s: %w", bucket, err) |
| + } |
| + } |
| + return nil, fmt.Errorf("%w: %s (lost the race for the lease three times)", ErrBucketBusy, bucket) |
| +} |
| + |
| +// bound returns ctx carrying the walk's fence and cut off before the |
| +// backend's window for it closes. Every call the walk makes goes through |
| +// it. |
| +func (w *bucketWalk) bound(ctx context.Context) (context.Context, context.CancelFunc) { |
| + // The time left is measured on the lease's clock and counted down on |
| + // the context's own. |
| + left := w.lock.SignedAt.Add(fenceWindow - walkMargin).Sub(walkClock()) |
| + return context.WithTimeout(withFence(ctx, w.lock.SignedAt), left) |
| +} |
| + |
| +// held reports whether the lease is still this invocation's: the object |
| +// carries its token and its time has not passed. |
| +func (w *bucketWalk) held(ctx context.Context) (bool, error) { |
| + if !walkClock().Before(w.lock.SignedAt.Add(fenceWindow - walkMargin)) { |
| + return false, nil |
| + } |
| + raw, _, err := w.s.Get(ctx, w.key) |
| + if err != nil { |
| + return false, err |
| + } |
| + var cur walkLock |
| + if err := json.Unmarshal(raw, &cur); err != nil { |
| + return false, err |
| + } |
| + return cur.Token == w.lock.Token, nil |
| +} |
| + |
| +// release gives the lease up. clean says the walk returned without error: |
| +// every call it made was answered, so nothing of it can still be on its |
| +// way, and the name is freed (unless an attempt it took over from has not |
| +// run out). Otherwise the lease is marked ended and stands until it runs |
| +// out. |
| +// |
| +// It writes against the etag this invocation last wrote, so it can only |
| +// ever release its own lease: if another invocation has taken over, the |
| +// write loses and changes nothing. |
| +func (w *bucketWalk) release(clean bool) { |
| + if w == nil || w.closed { |
| + return |
| + } |
| + w.closed = true |
| + done := w.lock |
| + done.Ended = true |
| + if clean { |
| + done.Until = time.Time{} |
| + if done.Carry.After(walkClock()) { |
| + done.Until = done.Carry |
| + } |
| + } else if done.Carry.After(done.Until) { |
| + done.Until = done.Carry |
| + } |
| + body, err := json.Marshal(done) |
| + if err != nil { |
| + return |
| + } |
| + ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) |
| + defer cancel() |
| + _, _ = w.s.Put(ctx, w.key, body, w.etag) |
| +} |
diff --git a/internal/ops/litestream.go b/internal/ops/litestream.go| index c75eeb0c5..21396ef01 100644 |
| --- a/internal/ops/litestream.go |
| +++ b/internal/ops/litestream.go |
| @@ -22,12 +22,10 @@ package ops |
| import ( |
| "bytes" |
| - "errors" |
| "fmt" |
| "os" |
| "os/exec" |
| "path/filepath" |
| - "regexp" |
| "sort" |
| "strconv" |
| "strings" |
| @@ -186,60 +184,6 @@ func ParseLitestreamDBs(cfg string) []string { |
| return dbs |
| } |
| -// ErrConfigNotGenerated means a config is not, line for line, what |
| -// LitestreamConfig writes. |
| -var ErrConfigNotGenerated = errors.New("litestream config is not the generated shape") |
| - |
| -var generatedConfigLines = []*regexp.Regexp{ |
| - regexp.MustCompile(`^#.*$`), |
| - regexp.MustCompile(`^snapshot:$`), |
| - regexp.MustCompile(`^ interval: \S+$`), |
| - regexp.MustCompile(`^ retention: \S+$`), |
| - regexp.MustCompile(`^dbs:$`), |
| - regexp.MustCompile(`^ replicas:$`), |
| - regexp.MustCompile(`^ - url: s3://\S+$`), |
| - regexp.MustCompile(`^ region: \S+$`), |
| -} |
| - |
| -var generatedConfigPath = regexp.MustCompile(`^ - path: (/\S+)$`) |
| - |
| -// GeneratedLitestreamDBs is ParseLitestreamDBs for a caller that must know |
| -// it has seen EVERY database the config names, not only the ones written |
| -// the way the generator writes them. It accounts for every line of the |
| -// file: each must be one LitestreamConfig produces, and a file with any |
| -// other line (a hand-kept config, a second replica, a directory entry, an |
| -// include, a tab) is refused with ErrConfigNotGenerated. It is not a YAML |
| -// parser and does not try to be; a config it cannot fully account for is |
| -// one it will not speak for. |
| -func GeneratedLitestreamDBs(cfg string) ([]string, error) { |
| - if !strings.HasPrefix(cfg, "# GENERATED by `carlos ops litestream`") { |
| - return nil, ErrConfigNotGenerated |
| - } |
| - var dbs []string |
| - seen := map[string]bool{} |
| - for n, line := range strings.Split(strings.TrimSuffix(cfg, "\n"), "\n") { |
| - if m := generatedConfigPath.FindStringSubmatch(line); m != nil { |
| - if seen[m[1]] { |
| - return nil, fmt.Errorf("%w: line %d names %s twice", ErrConfigNotGenerated, n+1, m[1]) |
| - } |
| - seen[m[1]] = true |
| - dbs = append(dbs, m[1]) |
| - continue |
| - } |
| - known := false |
| - for _, re := range generatedConfigLines { |
| - if re.MatchString(line) { |
| - known = true |
| - break |
| - } |
| - } |
| - if !known { |
| - return nil, fmt.Errorf("%w: line %d", ErrConfigNotGenerated, n+1) |
| - } |
| - } |
| - return dbs, nil |
| -} |
| - |
| // CoverageShortfall names every database the EXISTING config replicates that |
| // the proposed set does not — the databases a write would stop replicating. |
| // |
| @@ -369,10 +313,6 @@ func treeDBs(root string) []string { |
| // layout <data>/console/console.db — expressed against dataDir rather than |
| // hardcoded to /data so a test box's console is found under its own root and |
| // never the real one. |
| -// ConsoleDBPath is where this box's console keeps its database, as the |
| -// config generator resolves it. |
| -func ConsoleDBPath(dataDir string) string { return consoleDBPath(dataDir) } |
| - |
| func consoleDBPath(dataDir string) string { |
| if p := strings.TrimSpace(os.Getenv("CONSOLE_DB")); p != "" { |
| return filepath.Clean(p) |
diff --git a/internal/ops/litestream_generated_test.go b/internal/ops/litestream_generated_test.go| deleted file mode 100644 |
| index 5ff612dda..000000000 |
| --- a/internal/ops/litestream_generated_test.go |
| +++ /dev/null |
| @@ -1,52 +0,0 @@ |
| -package ops |
| - |
| -import ( |
| - "errors" |
| - "reflect" |
| - "strings" |
| - "testing" |
| -) |
| - |
| -// The strict reader accounts for every line the generator writes, with and |
| -// without a region, and returns exactly the generator's databases. |
| -func TestGeneratedLitestreamDBsRoundTrip(t *testing.T) { |
| - paths := []string{"/data/registry.db", "/data/console/console.db", "/var/lib/carlos-app/kass.example.com/instances/a.db"} |
| - for _, region := range []string{"eu-west-1", ""} { |
| - cfg := LitestreamConfig(paths, "bucket", "prefix", region) |
| - got, err := GeneratedLitestreamDBs(cfg) |
| - if err != nil { |
| - t.Fatalf("region %q: %v\n%s", region, err, cfg) |
| - } |
| - want := ParseLitestreamDBs(cfg) |
| - if !reflect.DeepEqual(got, want) || len(got) != len(paths) { |
| - t.Fatalf("region %q: got %v, the loose parser reads %v", region, got, want) |
| - } |
| - } |
| - if got, err := GeneratedLitestreamDBs(LitestreamConfig(nil, "bucket", "prefix", "eu-west-1")); err != nil || len(got) != 0 { |
| - t.Fatalf("an empty generated config: %v %v", got, err) |
| - } |
| -} |
| - |
| -// Anything the generator does not write is refused: the loose parser would |
| -// read such a file and silently miss what it does not recognise. |
| -func TestGeneratedLitestreamDBsRefusesAnythingElse(t *testing.T) { |
| - base := LitestreamConfig([]string{"/data/registry.db"}, "bucket", "prefix", "eu-west-1") |
| - cases := map[string]string{ |
| - "a hand-kept config with no marker": strings.SplitN(base, "\n", 2)[1], |
| - "a database written in flow style": base + " - {path: /data/hidden.db}\n", |
| - "a database indented differently": base + " - path: /data/hidden.db\n", |
| - "a directory entry": base + " - dir: /var/lib/tenants\n", |
| - "a second replica": base + " - type: sftp\n", |
| - "a relative path": base + " - path: data/relative.db\n", |
| - "a path with a trailing comment": base + " - path: /data/x.db # and more\n", |
| - "the same database twice": base + " - path: /data/registry.db\n", |
| - "a tab": strings.Replace(base, " - path:", "\t- path:", 1), |
| - "a key the generator never writes": base + "addr: :9090\n", |
| - "an empty file": "", |
| - } |
| - for name, cfg := range cases { |
| - if got, err := GeneratedLitestreamDBs(cfg); !errors.Is(err, ErrConfigNotGenerated) { |
| - t.Errorf("%s: got %v %v, want ErrConfigNotGenerated", name, got, err) |
| - } |
| - } |
| -} |