amadan
The storage pass sizes a store only when it is the sole holder of its bucket
OUTSTANDING 19i (round sixteen, Opus N2): the sizer lists a whole bucket, so two stores an operator adopted onto one bucket were each recorded with all of its bytes, and two accounts would each carry them. - objstore.OtherHolders: the stores that hold a bucket now, by the store records and the ledger's open intervals, with a store that left (a closed interval) not counted and a record no walk can follow always counted. The grant's take-back check uses it; its behaviour is unchanged. - storagePass holds a store, for its own account, when another store holds its bucket or the holders cannot be read. Reproductions for a shared bucket, a sole holder, a holder that left and a holder known only to the ledger; each check was mutation-tested in a scratch copy. A branch of its own, off the launch branch at c68ecddd. Not reviewed. Billing is off.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
4 files changed,
+213
−23
implementation/carloku-paid-launch/OUTSTANDING.md+1 −1internal/console/economicspass.go+25 −0internal/console/economicspass_test.go+139 −0internal/objstore/walk.go+48 −22
CI passed — run details
diff --git a/implementation/carloku-paid-launch/OUTSTANDING.md b/implementation/carloku-paid-launch/OUTSTANDING.md| index 5aac89c6e..7d2c3679b 100644 |
| --- a/implementation/carloku-paid-launch/OUTSTANDING.md |
| +++ b/implementation/carloku-paid-launch/OUTSTANDING.md |
| @@ -65,7 +65,7 @@ Each is concrete. None is waived. |
| 19. **No review of the whole branch has passed.** Astra rejected rounds one to sixteen; Opus accepted each with fixes. Round seventeen read only the diff since round sixteen: Opus accepted, Astra accepted with fixes (one P2, fixed since and unconfirmed). That is not a pass of the branch: the last whole-branch reading was round sixteen's, and it was rejected. Every round has closed the shapes the round before reported and found neighbouring ones, in the store walks since round seven. Some concern records no walk of this build writes; others need only this build's ordinary walks (a failed rotation, a retained teardown, a scanning door racing a teardown), and two were faults in the previous round's own fixes. Round eleven found one P1 and two P2 in the store walks (a grant could move an older record to another bucket name; a waiting rotation ignored a grant that ran; the user and its policy were written before the grant was counted). Its fixes are on the launch branch (`1395f7bf`). Round twelve read them: every round-eleven finding is closed for its reported shape, Astra rejected again (one P1, two P2) and Opus accepted with fixes (one P2, three P3). The new findings were about records no walk of this build writes (a teardown note for another bucket, a record naming two buckets) being acted on by teardown, rotation and the scanning doors; a rotation started after a grant that failed at its last write; and a take-back that returned access to a bucket another account holds, through the member's own door. Round thirteen read those fixes: closed for their reported shapes, Astra rejected again (two P1, four P2) and Opus accepted with fixes (one P2, five P3). The new findings were paths that read such a record and acted without the check (the holder inventory, the storage pass, the declare door), a rotation's own failed last write, and a retained record blocking another account's take-back. The check now sits in the record's reader and writer. Round fourteen read those: closed for their reported shapes; Astra rejected (two P1, two P2), Opus accepted with fixes (two P2, two P3). Round fifteen read those: all closed; Astra rejected (one P1, one P2), Opus accepted with fixes (one P2, two P3), all about readers of a bucket that a retained teardown left to another account (a sweep in flight, the scan status, the storage pass). Round sixteen read those: all closed, and one of them had made a single store under teardown hold storage for every account. Astra rejected (one P1, two P2), Opus accepted with fixes (one P1, one P2, two P3). The storage hold is now per account; two findings are left open as 19f and 19i. Those fixes are on the launch branch and unconfirmed until the next round reads them. Each round's fixes are on the launch branch, gated locally, and unconfirmed until the next round reads them. |
| 19g. **Refusals whose doors say the wrong thing, and records only a person can repair.** A record that names two buckets (`ErrTwoBuckets`) or whose teardown note is about another bucket (`ErrUnfinishedTeardown`) is refused by the record's one reader and one writer, so nothing acts on it and nothing writes it; the settings page shows such a store as unreadable. No walk of this build writes either; whether an earlier build left such records in production has not been checked. There is no operator verb to repair one. The doors that meet these refusals answer with sentences written for other failures: the grant and provision doors say to run the grant again, the destroy door and the purge say to try again, the scanning doors say to try again or to re-run the grant. The same goes for a rotation refused because a grant must be run again first (`ErrGrantUnfinished`) and for a member's take-back refused because another account holds the bucket (`ErrBucketHeld`). Correct copy for each needs approval (A). |
| -19i. **A live bucket shared by two stores is sized in full for each of them.** The daily storage pass lists the whole bucket for every store record that names it, so two accounts on one operator-adopted bucket would each carry all of its bytes. It predates this branch. Storage on a shared bucket must hold, not bill, until the pass uses the holder inventory and sizes only a sole holder. Not built. |
| +19i. **A live bucket shared by two stores was sized in full for each of them.** Fixed on this branch (`feat/carloku-storage-sole-holder-20261001`), not on the launch branch: the daily storage pass sizes a store only when it is the sole holder of its bucket, by the store records and the ownership ledger (`objstore.OtherHolders`, the same inventory the grant's take-back check now uses). A store on a bucket another store holds is held for its own account, every day, until one of them leaves: a person has to decide how such a bucket is billed, and there is no door for that. A store that left (closed ledger interval) does not count. It reads every store record once per store per day. Not reviewed. |
| 19h. **A rotation whose delivery fails is repaired by a grant, which swaps the app's key with no overlap.** A failed delivery may still land, so the record stays marked, the next rotation is refused, and the grant run again replaces whichever key the app holds. Hosts hold a dead key until they adopt the new config. Before this branch such a rotation was resumed by running it again, which kept the overlap when the delivery had not landed and broke the app when it had. A repair that waits out the signing window and then reads which key was delivered would keep the overlap where the window is enforced (observed on AWS only, 19d); it is not built. |
| 19f. **A store that leaves a bucket with an unsupported name does not hand on its scanner debt.** There is no ledger for such a name. The last holder would delete the bucket and leave a scanning plan behind. Today's scanner (ClamAV) has no plan to leave, so this is latent. Also latent for the same reason: the scan grant door checks that no teardown has begun and then enables scanning as two steps, so a retained teardown landing between them would leave scanning enabled on the other account's bucket with nothing recording that it is owed. And the scan queue's drain checks permission once per batch, so a drain admitted before a retained teardown could return the other account's object metadata; nothing calls the drain or enqueues today, and it must be fixed before the queue is wired. |
| 19e. **A purge can end with a store that was granted during it.** A declare and a grant admitted while the app was live can land between the purge's last check and the removal of its marker. The purge now reports that as an error and never deletes the record, but it does not prevent it: the doors check liveness when they admit a request, not when they write. And after that error the estate is named only by its record and a log line. The app's console row is still trashed, so the settings page and the store-destroy door answer 404 and a second purge answers "not trashed", until the console restarts and restores the row from the record. The flash the member sees says "Nothing was deleted", which is false in this case; correct copy needs approval (A). |
diff --git a/internal/console/economicspass.go b/internal/console/economicspass.go| index 263e84e21..e048798a8 100644 |
| --- a/internal/console/economicspass.go |
| +++ b/internal/console/economicspass.go |
| @@ -226,6 +226,22 @@ func (s *Server) storagePass(ctx context.Context, date string) error { |
| // (and only this account), not measured. |
| st.StoreProvisioned = true |
| st.StoreUnswept = true |
| + } else if others, herr := storeSharedWith(ctx, s.cfg.Releases, rec, acct, app); herr != nil || len(others) > 0 { |
| + // The sizer lists the whole bucket. A bucket another store |
| + // also holds (an operator adopted it for both) has no line |
| + // between this store's bytes and the other's, so its size |
| + // is not this app's storage: recorded in full for each of |
| + // them, two accounts would each carry all of it. Whose |
| + // bytes are whose cannot be told from here, so the day is |
| + // held for this account, not measured. The same when the |
| + // holders cannot be read. |
| + if herr != nil { |
| + log.Printf("console: storage pass: holders of %s/%s's bucket: %v", acct, app, herr) |
| + } else { |
| + log.Printf("console: storage pass: %s/%s shares its bucket with %d other store(s); held", acct, app, len(others)) |
| + } |
| + st.StoreProvisioned = true |
| + st.StoreUnswept = true |
| } else if rec != nil && rec.Provisioned != nil { |
| st.StoreProvisioned = true |
| n, serr := s.sizeStore(ctx, rec.Provisioned) |
| @@ -2409,3 +2425,12 @@ func (s *Server) handleAPIEconomicsBackhaul(w http.ResponseWriter, r *http.Reque |
| } |
| writeJSON(w, http.StatusOK, buildBackhaulReport(v)) |
| } |
| + |
| +// storeSharedWith names the other stores that hold a provisioned store's |
| +// bucket. A record with nothing provisioned holds no bucket to size. |
| +func storeSharedWith(ctx context.Context, store release.Store, rec *objstore.Record, acct, app string) ([]objstore.OwnerRef, error) { |
| + if rec == nil || rec.Provisioned == nil || rec.Provisioned.Bucket == "" { |
| + return nil, nil |
| + } |
| + return objstore.OtherHolders(ctx, store, rec.Provisioned.Bucket, acct, app) |
| +} |
diff --git a/internal/console/economicspass_test.go b/internal/console/economicspass_test.go| index a55698b0a..6269f6800 100644 |
| --- a/internal/console/economicspass_test.go |
| +++ b/internal/console/economicspass_test.go |
| @@ -1714,3 +1714,142 @@ func TestStoragePassDiscardsASizeTakenAcrossAReplacementOfTheStore(t *testing.T) |
| } |
| } |
| } |
| + |
| +// The sizer lists a whole bucket. A bucket two stores hold has no line |
| +// between one store's bytes and the other's, so neither is sized: recorded |
| +// in full for each, both would carry all of it. Each is held for its own |
| +// account. A store that is the only holder is sized, and a store that left |
| +// the bucket (its ledger interval is closed, its record is kept until its |
| +// retained teardown is run again) does not count as holding it. |
| +func TestStoragePassSizesOnlyASoleHolderOfItsBucket(t *testing.T) { |
| + at := time.Date(2026, 8, 10, 9, 0, 0, 0, time.UTC) |
| + const shared, other = "carlos-store-shared-x", "bcd" |
| + provisioned := func(bucket string) *objstore.Record { |
| + return &objstore.Record{ |
| + Grant: &objstore.GrantBlock{Backend: "aws", Bucket: bucket, Source: objstore.GrantSourceOperator, At: at, By: "op"}, |
| + Provisioned: &objstore.Provisioned{Bucket: bucket, Region: "eu-west-1", IAMUser: "u", KeyID: "K", At: at}, |
| + } |
| + } |
| + run := func(t *testing.T, fx *consoleFixture) (map[string]usage.AppStorage, []string) { |
| + t.Helper() |
| + ctx := context.Background() |
| + var sized []string |
| + fx.srv.storeSizer = func(_ context.Context, p *objstore.Provisioned) (int64, error) { |
| + sized = append(sized, p.Bucket) |
| + return 7 << 30, nil |
| + } |
| + if err := fx.srv.storagePass(ctx, "2026-08-12"); err != nil { |
| + t.Fatal(err) |
| + } |
| + raw, _, err := fx.rel.Get(ctx, usage.StorageDayKey("2026-08-12")) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + var day usage.StorageDay |
| + if err := json.Unmarshal(raw, &day); err != nil { |
| + t.Fatal(err) |
| + } |
| + return day.Apps, sized |
| + } |
| + mine, theirs := fixtureAcct+"/"+fixtureApp, other+"/shop" |
| + seedOther := func(t *testing.T, fx *consoleFixture) { |
| + t.Helper() |
| + // The other account's app has to exist in the deployment bucket |
| + // for the pass to visit it. |
| + if _, err := fx.rel.Put(context.Background(), release.AppPrefix(other, "shop")+"marker", []byte("x"), ""); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + |
| + t.Run("two stores on one bucket", func(t *testing.T) { |
| + fx := newConsoleFixture(t) |
| + ctx := context.Background() |
| + seedOther(t, fx) |
| + if _, err := objstore.Put(ctx, fx.rel, fixtureAcct, fixtureApp, provisioned(shared), ""); err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, err := objstore.Put(ctx, fx.rel, other, "shop", provisioned(shared), ""); err != nil { |
| + t.Fatal(err) |
| + } |
| + apps, sized := run(t, fx) |
| + if len(sized) != 0 { |
| + t.Fatalf("a shared bucket was sized: %v", sized) |
| + } |
| + for _, key := range []string{mine, theirs} { |
| + if app := apps[key]; !app.StoreUnswept || app.StoreBytes != 0 || !app.StoreProvisioned { |
| + t.Errorf("%s: %+v; want held, not measured", key, app) |
| + } |
| + } |
| + }) |
| + |
| + t.Run("a sole holder", func(t *testing.T) { |
| + fx := newConsoleFixture(t) |
| + ctx := context.Background() |
| + seedOther(t, fx) |
| + if _, err := objstore.Put(ctx, fx.rel, fixtureAcct, fixtureApp, provisioned(shared), ""); err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, err := objstore.Put(ctx, fx.rel, other, "shop", provisioned("carlos-store-their-own"), ""); err != nil { |
| + t.Fatal(err) |
| + } |
| + apps, sized := run(t, fx) |
| + if len(sized) != 2 { |
| + t.Fatalf("sized %v; want both stores, each alone on its bucket", sized) |
| + } |
| + for _, key := range []string{mine, theirs} { |
| + if app := apps[key]; app.StoreUnswept || app.StoreBytes != 7<<30 { |
| + t.Errorf("%s: %+v; want measured", key, app) |
| + } |
| + } |
| + }) |
| + |
| + t.Run("the other store has left", func(t *testing.T) { |
| + fx := newConsoleFixture(t) |
| + ctx := context.Background() |
| + seedOther(t, fx) |
| + if _, err := objstore.Put(ctx, fx.rel, fixtureAcct, fixtureApp, provisioned(shared), ""); err != nil { |
| + t.Fatal(err) |
| + } |
| + left := provisioned(shared) |
| + left.Teardown = &objstore.TeardownBlock{StartedAt: at.Add(time.Hour)} |
| + if _, err := objstore.Put(ctx, fx.rel, other, "shop", left, ""); err != nil { |
| + t.Fatal(err) |
| + } |
| + for _, o := range []struct{ acct, app string }{{fixtureAcct, fixtureApp}, {other, "shop"}} { |
| + if err := objstore.OpenBucketOwner(ctx, fx.rel, shared, o.acct, o.app, at, at, false); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + if err := objstore.CloseBucketOwner(ctx, fx.rel, shared, other, "shop", at.Add(time.Hour)); err != nil { |
| + t.Fatal(err) |
| + } |
| + apps, sized := run(t, fx) |
| + if len(sized) != 1 || sized[0] != shared { |
| + t.Fatalf("sized %v; want the remaining store's bucket once", sized) |
| + } |
| + if app := apps[mine]; app.StoreUnswept || app.StoreBytes != 7<<30 { |
| + t.Errorf("the remaining store: %+v; want measured", app) |
| + } |
| + if app := apps[theirs]; !app.StoreUnswept || app.StoreBytes != 0 { |
| + t.Errorf("the store that left: %+v; want held", app) |
| + } |
| + }) |
| + |
| + t.Run("another holder on the ledger only", func(t *testing.T) { |
| + fx := newConsoleFixture(t) |
| + ctx := context.Background() |
| + if _, err := objstore.Put(ctx, fx.rel, fixtureAcct, fixtureApp, provisioned(shared), ""); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := objstore.OpenBucketOwner(ctx, fx.rel, shared, other, "shop", at, at, false); err != nil { |
| + t.Fatal(err) |
| + } |
| + apps, sized := run(t, fx) |
| + if len(sized) != 0 { |
| + t.Fatalf("a bucket with another open interval was sized: %v", sized) |
| + } |
| + if app := apps[mine]; !app.StoreUnswept || app.StoreBytes != 0 { |
| + t.Errorf("%+v; want held", app) |
| + } |
| + }) |
| +} |
diff --git a/internal/objstore/walk.go b/internal/objstore/walk.go| index 7866a968b..47e3dab4a 100644 |
| --- a/internal/objstore/walk.go |
| +++ b/internal/objstore/walk.go |
| @@ -300,34 +300,14 @@ func grant(ctx context.Context, s release.Store, p Provisioner, d Deployment, |
| // written, from the store records and the ledger, as a minting grant |
| // checks them. |
| if rec.Teardown != nil && source != GrantSourceOperator { |
| - holders, herr := otherStoreHolders(ctx, s, bucket, acct, app) |
| + holders, herr := OtherHolders(ctx, s, bucket, acct, app) |
| if herr != nil { |
| - return nil, fmt.Errorf("objstore: list the stores holding %s: %w", bucket, herr) |
| - } |
| - ledger, _, lerr := ReadBucketOwnership(ctx, s, bucket) |
| - if lerr != nil { |
| - return nil, fmt.Errorf("objstore: read ownership of %s: %w", bucket, lerr) |
| - } |
| - if ledger != nil { |
| - for _, iv := range ledger.Intervals { |
| - if iv.open() && iv.Account != acct { |
| - holders = append(holders, OwnerRef{Account: iv.Account, App: iv.App, Since: iv.From}) |
| - } |
| - } |
| + return nil, herr |
| } |
| for _, h := range holders { |
| if h.Account == acct { |
| continue |
| } |
| - // A record that outlived its store's access is not a hold. A |
| - // store that left the bucket has a closed interval on the |
| - // ledger, written after its credentials were revoked, and |
| - // keeps its record only until its retained teardown is run |
| - // again. Only what the ledger shows counts for that: a store |
| - // with no interval at all has an unknown history, and holds. |
| - if !h.refused && ledger != nil && ledger.left(h.Account, h.App) { |
| - continue |
| - } |
| return nil, fmt.Errorf("%w: %s by %s/%s of another account; only an operator's grant gives %s/%s access to it again", ErrBucketHeld, bucket, h.Account, h.App, acct, app) |
| } |
| } |
| @@ -1625,3 +1605,49 @@ func Coherent(rec *Record, acct, app string) error { |
| func SameStore(a, b *Record) bool { |
| return a != nil && b != nil && estateOf(a) == estateOf(b) |
| } |
| + |
| +// OtherHolders names the stores, other than acct/app, that hold bucket now, |
| +// by the two things that can show it: a store record that names the bucket |
| +// (a record exists from its grant until its teardown has finished, ledger |
| +// or no ledger), and an open interval on the bucket's ownership ledger. |
| +// |
| +// A record that outlived its store's access is not a hold: a store that |
| +// left the bucket has a closed interval on the ledger, written after its |
| +// credentials were revoked, and keeps its record only until its retained |
| +// teardown is run again. Only the ledger can show that; a store with no |
| +// interval at all has an unknown history, and holds. A record no walk can |
| +// follow (Coherent) holds every bucket it names whatever the ledger shows. |
| +// |
| +// It reads every store record in the deployment. For a bucket name this |
| +// build would refuse there is no ledger, and the records alone answer. |
| +func OtherHolders(ctx context.Context, s release.Store, bucket, acct, app string) ([]OwnerRef, error) { |
| + holders, err := otherStoreHolders(ctx, s, bucket, acct, app) |
| + if err != nil { |
| + return nil, fmt.Errorf("objstore: list the stores holding %s: %w", bucket, err) |
| + } |
| + var ledger *BucketOwnership |
| + if ValidBucketName(bucket) { |
| + if ledger, _, err = ReadBucketOwnership(ctx, s, bucket); err != nil { |
| + return nil, fmt.Errorf("objstore: read ownership of %s: %w", bucket, err) |
| + } |
| + } |
| + seen := map[string]bool{} |
| + var out []OwnerRef |
| + for _, h := range holders { |
| + if !h.refused && ledger != nil && ledger.left(h.Account, h.App) { |
| + continue |
| + } |
| + seen[h.Account+"/"+h.App] = true |
| + out = append(out, h) |
| + } |
| + if ledger != nil { |
| + for _, iv := range ledger.Intervals { |
| + if !iv.open() || (iv.Account == acct && iv.App == app) || seen[iv.Account+"/"+iv.App] { |
| + continue |
| + } |
| + seen[iv.Account+"/"+iv.App] = true |
| + out = append(out, OwnerRef{Account: iv.Account, App: iv.App, Since: iv.From}) |
| + } |
| + } |
| + return out, nil |
| +} |