amadan
Close round fifteen: nothing reads a retained store's bucket as that store's
Round fifteen (Astra reject: one P1, one P2; Opus accept with fixes: one P2, two P3) found every round fourteen finding closed. Its findings are about a store whose teardown left its bucket to another account's store and kept its record. - A sweep already running stands down when a teardown block appears on the record, lists no further page and hands back nothing. - The scan status shows no coverage for a store under teardown. - The daily storage pass does not size such a store; it records it as unswept, which holds the day. - Regrant refuses a record with nothing provisioned. - Put moves the earlier place of the key-change mark to the record's. Each fix has a reproduction and was mutation-checked in a scratch copy. The round's reports, dispositions and the round sixteen brief are added. OUTSTANDING.md carries the side branches' current tips and gate results. No review round has passed. Billing is off; nothing attests.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
15 files changed,
+446
−21
docs/superpowers/specs/CODEX-DISPOSITIONS-carloku-paid-launch-overnight.md+17 −0docs/superpowers/specs/CODEX-FINDINGS-carloku-paid-launch-overnight-round15.md+83 −0docs/superpowers/specs/CODEX-REVIEW-BRIEF-carloku-paid-launch-overnight-round16.md+27 −0docs/superpowers/specs/OPUS-REVIEW-carloku-paid-launch-overnight-round15.md+103 −0implementation/carloku-paid-launch/AGENT_HANDOFF.md+1 −1implementation/carloku-paid-launch/OUTSTANDING.md+10 −10internal/console/apiscan.go+7 −0internal/console/apiscan_test.go+39 −2internal/console/economicspass.go+9 −0internal/console/economicspass_test.go+37 −0internal/objstore/objstore.go+11 −5internal/objstore/ownership_test.go+26 −0internal/objstore/sweep.go+18 −0internal/objstore/sweep_test.go+49 −0internal/objstore/walk.go+9 −3
CI passed — run details
diff --git a/docs/superpowers/specs/CODEX-DISPOSITIONS-carloku-paid-launch-overnight.md b/docs/superpowers/specs/CODEX-DISPOSITIONS-carloku-paid-launch-overnight.md| index 89b0eb15a..26ab3a91f 100644 |
| --- a/docs/superpowers/specs/CODEX-DISPOSITIONS-carloku-paid-launch-overnight.md |
| +++ b/docs/superpowers/specs/CODEX-DISPOSITIONS-carloku-paid-launch-overnight.md |
| @@ -453,3 +453,20 @@ Both found every round thirteen finding closed for its reported shape, and Opus |
| | Both, claims: the rotation's doc, the reader's doc, "since round eleven the findings concern records no walk of this build writes" | Corrected. The last was false: several findings (a failed rotation, a retained teardown, the scanning re-grant) need only this build's ordinary walks. | `walk.go`, `objstore.go`, `OUTSTANDING.md` 19 | none | |
| Mutation checks in a scratch copy, each failing its test: the refused mark ignored in the teardown's decision, and in the take-back; the re-grant's refusal off; the scanning door using the ordinary grant; the earlier mark not read; the rotation's mark moved back before its first step; the sweep's and the queue's teardown checks off; the empty provisioned name accepted; the revoke door's guard off. Storage simulation: 3,000 seeds on this code. |
| + |
| +## Round fifteen (reviewed `973a9f15`; Astra rejected, Opus accepted with fixes) |
| + |
| +Reports: `CODEX-FINDINGS-carloku-paid-launch-overnight-round15.md`, `OPUS-REVIEW-carloku-paid-launch-overnight-round15.md`. |
| + |
| +Both found every round fourteen finding closed. Astra: one P1, one P2. Opus: one P2, two P3. Astra's F2 and Opus's N1 are the same fault. All of them are about a store whose teardown left its bucket to another account's store and kept its record: readers that still treat that bucket as the store's. They predate this branch. Nothing that attests, reaches Stripe with a switch unset, or makes an open late-request defect reachable. |
| + |
| +| Finding | Disposition | Where | Reproduction | |
| +| --- | --- | --- | --- | |
| +| Astra F1: a sweep already past its checks lists the bucket after a retained teardown and publishes the other account's objects as this store's coverage | Accepted. The sidecar's writer, which reads the record again before every write, stands down for a record with a teardown block; the sweep lists no further page and hands back nothing. The scan status shows no coverage for a store whose teardown has begun. | `sweep.go` `writeSidecar`, `RunSweep`; `console/apiscan.go` `scanReply` | `TestSweepInFlightDoesNotPublishAfterATeardownBegins`, `TestScanStatusShowsNoCoverageForAStoreUnderTeardown` | |
| +| Astra F2, Opus N1: the daily storage pass sizes a retained bucket and records the other account's bytes on the store that left | Accepted. A record with a teardown block is not sized; the store is recorded as unswept, which holds that day's storage for the account. It stays held for as long as the retained record stands (19a). | `console/economicspass.go` `storagePass` | `TestStoragePassDoesNotSizeAStoreUnderTeardown` | |
| +| Opus N2: `Regrant` can make a first store if the record was destroyed and declared again between the door's look and the walk | Accepted. Refused when nothing is provisioned on the record read under the lease. | `walk.go` `grant` | `TestRegrantNeverMakesAFirstStore` | |
| +| Opus N3 (latent): the scan grant door checks for a teardown and enables scanning as separate steps | Accepted, not fixed. Today's scanner does nothing on enable. Added to 19f. | `OUTSTANDING.md` 19f | none | |
| +| Astra, noted: the earlier place of the mark is an exported field and `Put` would marshal it | Accepted. `Put` moves it to the record's field before it writes. | `objstore.go` `Put` | `TestTheMarkIsReadFromWhereAnEarlierBuildWroteIt` (the repaired record carries the word in neither place) | |
| +| Both, claims: the mark field's doc, "an earlier build carries no mark", "resumed by running it again" stated absolutely, `Inspect` "for display", a test comment, 19h's reliance on the signing window | Corrected. | `objstore.go`, `walk.go`, `AGENT_HANDOFF.md`, `apiscan_test.go`, `OUTSTANDING.md` | none | |
| + |
| +Mutation checks in a scratch copy, each failing its test: the sidecar writer's teardown check off; the storage pass sizing a store under teardown; the status showing coverage; the re-grant accepting a record with nothing provisioned. |
diff --git a/docs/superpowers/specs/CODEX-FINDINGS-carloku-paid-launch-overnight-round15.md b/docs/superpowers/specs/CODEX-FINDINGS-carloku-paid-launch-overnight-round15.md| new file mode 100644 |
| index 000000000..aa3f81ab9 |
| --- /dev/null |
| +++ b/docs/superpowers/specs/CODEX-FINDINGS-carloku-paid-launch-overnight-round15.md |
| @@ -0,0 +1,83 @@ |
| +# Round-fifteen review: `973a9f15` |
| + |
| +Reviewed against base `aed42815`, concentrating on `d5705a63..973a9f15`. No files changed, tests run, network commands, or Stripe/AWS CLI calls. Test counterfactuals below are based on inspection. |
| + |
| +All locations refer to `973a9f15`. **O** = `internal/objstore`; **C** = `internal/console`; **D** = `implementation/carloku-paid-launch`; **S** = `docs/superpowers/specs`. |
| + |
| +## 1. Round-fourteen dispositions |
| + |
| +| Finding or claim | Disposition | Fix and regression | Would the test pass without the fix? | |
| +|---|---|---|---| |
| +| **Astra F1; refused-holder claim:** historical ledger intervals erase an incoherent record’s hold | **Closed for the reported teardown and take-back decisions.** The inventory preserves `refused`, and both decisions respect it. | `O/walk.go:1417`, `:321`; `O/ownership.go:516`. `TestARefusedRecordsHoldIsNotReconciledAwayByAnOldClosedInterval`, `O/ownership_test.go:3263`. | **No.** Removing either decision’s protection fails its retained-bucket or refused-take-back assertion. | |
| +| **Astra F2; scanning take-back claim:** teardown between the scanning write and re-grant restores access | **Closed.** `Regrant` refuses teardown on the under-lease read. The other production `Grant` callers are explicit grant/provision doors. | `O/walk.go:159`, `:252`; `C/apiscan.go:583`. Tests: `O/ownership_test.go:3307`; `C/apiscan_test.go:716`. | **No.** Removing the refusal or reverting the console caller to `Grant` fails. Neither test schedules teardown after lease acquisition; exclusion there follows from the walk lease and guarded writes. | |
| +| **Astra F3; config-comparison claim:** a failed delivery can land after an unchanged-config observation | **Closed.** Every delivery error retains the mark; the comparison is gone. The repair’s loss of overlap is **accepted and left open as 19h**. | `O/walk.go:748`, `:770`, `:776`. `TestRotateAfterAResealFailureIsRepairedByTheGrant`, `O/walk_test.go:710`. | **No** if the old clearing branch is restored. The fixture fails the config write before its effect; it does not simulate a delayed commit. | |
| +| **Astra F4:** the previous nested delivery mark is discarded | **Closed in the reader and ordinary rewrite paths.** | `O/objstore.go:75`, `:257`. `TestTheMarkIsReadFromWhereAnEarlierBuildWroteIt`, `O/ownership_test.go:3352`. | **No.** Without migration, the first marked-read assertion fails. The test also checks both rotation refusals and successful repair. | |
| +| **Opus N1; rotation documentation:** failures before delivery unnecessarily force repair without overlap | **Closed for the reported reap/mint failures.** Rotation marks after those operations. | `O/walk.go:731`, `:735`, `:748`; revised explanation at `:679`. `TestRotationThatFailsBeforeDeliveryIsResumedWithItsOverlap`, `O/ownership_test.go:3545`. | **No.** Moving the mark back before the failing operation fails the unmarked-record assertion. | |
| +| **Opus N2; scanning claim:** a retained store continues publishing another account’s bucket coverage | **Partly closed.** A sweep begun after teardown stops. An already-running sweep can still publish: new F1 below. | `O/sweep.go:143`; `O/queue.go:486`. `TestAStoreUnderTeardownIsNotSweptOrQueued`, `O/ownership_test.go:3592`. | **No**, but indirectly: without either guard the fixture reaches `ErrScanningDisabled`. Antivirus is unset and the recorded engine is empty, so the no-listing assertion does not establish protection with scanning operational. | |
| +| **Opus N3:** an empty provisioned bucket beside a named grant is accepted and orphaned by teardown | **Closed.** | `O/walk.go:1589`, through checked `Get`. `TestAProvisionedBlockWithNoBucketBesideAGrantThatNamesOneIsRefused`, `O/ownership_test.go:3616`. | **No.** The explicit reader-refusal assertion fails. | |
| +| **Opus N4:** the revoke guard is untested | **Closed.** The fixture now enables the scanner before invoking revoke and checks it remains enabled. | Guard: `C/apiscan.go:421`. Test: `C/apiscan_test.go:642`, setup at `:666`, assertion at `:679`. | **No.** Removing the teardown guard allows `EnsureDisabled` to clear the enabled bucket. | |
| +| **Opus claim:** `leaseForRotation` says repair preserves the delivered key | **Closed as documentation.** It now says repair replaces the key. | `O/walk.go:876`; repair behavior exercised at `O/ownership_test.go:3172` and `O/walk_test.go:710`. | **Yes** without the comment correction; these are behavioral tests. | |
| +| **Opus claim:** `Inspect` exposes refused records to someone repairing them | **Closed in `Inspect`’s documentation.** It explicitly says no such door exists. | `O/objstore.go:231`. | No wording test. The separate `Coherent` comment at `O/walk.go:1562` still mentions display. | |
| +| **Astra claim:** earlier-build records uniformly contain no delivery evidence | **Partly closed.** Nested evidence is recovered, but the handoff still makes the blanket statement. Genuinely pre-mark records remain **accepted and left open**. | Migration: `O/objstore.go:257`; regression: `O/ownership_test.go:3352`. Stale wording: `D/AGENT_HANDOFF.md:35`. | Migration test: **no** without migration; **yes** without correcting the handoff. | |
| +| **Astra claim:** findings since round eleven concern only records this build cannot write | **Closed as documentation.** Ordinary failed rotations, retained teardowns and scanning races are now acknowledged. | `D/OUTSTANDING.md:66`; dispositions `S/CODEX-DISPOSITIONS-carloku-paid-launch-overnight.md:453`. | No wording test. | |
| + |
| +The console re-grant test’s comment says “user gone” (`C/apiscan_test.go:734`), but its fixture only adds a teardown block and leaves the user intact. The package-level regression at `O/ownership_test.go:3307` does perform the retained teardown and checks that no user is recreated. |
| + |
| +## 2. New findings |
| + |
| +### F1. P1 — An in-flight sweep publishes another account’s objects after retained teardown |
| + |
| +**Where:** `O/sweep.go:143`, `:226`, `:286`, `:396`; `C/apiscan.go:85`, `:209`. |
| + |
| +**Scenario:** |
| + |
| +1. Accounts A and B have operator-adopted stores on bucket X. A has scanning enabled, with deployment antivirus enabled and matching engine provenance. |
| +2. A’s sweep passes the new teardown check. Pause it before its listing request executes. |
| +3. A’s teardown revokes its credentials, closes its ownership interval and returns `ErrBucketRetained`, leaving its record and scanning block standing. |
| +4. B uploads objects that did not exist while A had access. |
| +5. Resume A’s sweep. The deployment’s lister and scanner read those objects. |
| +6. `writeSidecar` re-reads A’s record, but checks only record existence and scanning enablement. It ignores the teardown block and writes the new coverage. |
| +7. A’s member calls `store scan status`. `scanReply` returns that sidecar without a teardown restriction. |
| + |
| +Object counts, bytes and verdict counts leak. With a truncated page and an exhausted sweep budget, `Current.After` also exposes an object key created after A’s access ended. |
| + |
| +This uses ordinary current-build operations. The new entry check does not protect the publication decision, even though that decision already re-reads the record. A retained teardown leaves the sidecar untouched, so its CAS does not prevent the write. |
| + |
| +**Smallest acceptable fix:** Make `writeSidecar` stand down for teardown records, including its retry path; stop further pages when authorization has ended; and prevent scan-status responses from exposing coverage for a retained store. Add an interleaving regression with scanning fully enabled: pause before listing, complete retained teardown, add B’s objects, resume, and assert that neither persisted coverage nor A’s status response exposes them. |
| + |
| +The test at `O/ownership_test.go:3592` covers only a teardown already present before the sweep starts. |
| + |
| +### F2. P2 — Daily storage sizing attributes a retained bucket’s subsequent contents to the account that left |
| + |
| +**Where:** `C/economicspass.go:211`, `:218`, `:850`, `:1021`, `:1634`. |
| + |
| +**Scenario:** |
| + |
| +1. A and B share an operator-adopted bucket X. |
| +2. A tears down its store. Its credentials are revoked and its interval closes; X remains for B and A’s record remains under `ErrBucketRetained`. |
| +3. B subsequently adds 31 GB. |
| +4. The next daily `storagePass` reads A’s retained record. Because `Provisioned` remains present, it lists X using deployment credentials and records those bytes under A’s app with `StoreUnswept=false`. |
| +5. The same pass sizes B’s store. `foldStorageAndCensus` attributes the bytes to both accounts. |
| + |
| +In a 31-day month, that single snapshot adds an erroneous 1 GB-month to A’s storage meter. Subsequent snapshots repeat the attribution for as long as the retained record stands. The ownership interval is never consulted, and the snapshot is create-only, so rerunning the pass does not correct it. |
| + |
| +Billing remains disabled and the other meter holds remain effective; this is incorrect persisted usage, not a demonstrated current charge. I found no direct member-facing display of these byte totals in this snapshot. |
| + |
| +**Smallest acceptable fix:** Do not size a teardown record’s bucket as that app’s current storage. Where the remaining attribution cannot be established, mark it unswept and hold instead of recording another account’s bytes as measured usage. Add a regression that completes retained teardown, changes B’s bucket contents, runs storage collection and folding, and checks that A receives no measured attribution for those contents. |
| + |
| +## 3. Claims the code contradicts |
| + |
| +| Location | Claim | What the code supports | |
| +|---|---|---| |
| +| `D/AGENT_HANDOFF.md:35`; round-fourteen dispositions at `S/CODEX-DISPOSITIONS-carloku-paid-launch-overnight.md:450` | A store whose teardown has begun is not swept and does not publish new coverage. | The initial sweep check stops new sweeps. An admitted sweep still publishes after teardown through `writeSidecar`: F1. | |
| +| `D/AGENT_HANDOFF.md:35` | “A record from an earlier build carries no mark.” | Some earlier records carry the nested mark, which `Inspect` now recovers at `O/objstore.go:257`. This statement needs to identify builds predating either mark. | |
| +| `O/objstore.go:158` | Rotation sets `Delivering` before reaping and minting. | Rotation now reaps and mints at `O/walk.go:731` and `:735`, then marks at `:748`. Grant still marks before changing keys. | |
| +| `O/walk.go:1562` | `Inspect` is also used “for display.” | Its corrected documentation at `O/objstore.go:231` says no display door exists; Settings reads through checked `Get`. | |
| + |
| +**Legacy mark write-back:** Normal `Get`/`Inspect` → `Put` paths migrate the nested mark and clear `LegacyDelivering`, so they do not write the old representation back. However, the exported field still has `json:"delivering,omitempty"` (`O/objstore.go:75`), and `Put` directly marshals it (`:276`). An explicitly constructed record with `LegacyDelivering=true` can therefore emit the old representation. No current production caller found sets that field; “never written again” describes the current callers, not an enforced serialization rule. |
| + |
| +`OUTSTANDING.md` 19 now acknowledges ordinary generated states. Items 19g and 19h accurately retain the documented repair and overlap limitations; those are not new findings. |
| + |
| +## 4. Verdict |
| + |
| +**Reject.** |
| \ No newline at end of file |
diff --git a/docs/superpowers/specs/CODEX-REVIEW-BRIEF-carloku-paid-launch-overnight-round16.md b/docs/superpowers/specs/CODEX-REVIEW-BRIEF-carloku-paid-launch-overnight-round16.md| new file mode 100644 |
| index 000000000..22018c3a9 |
| --- /dev/null |
| +++ b/docs/superpowers/specs/CODEX-REVIEW-BRIEF-carloku-paid-launch-overnight-round16.md |
| @@ -0,0 +1,27 @@ |
| +# Review brief, round 16: Carloku paid launch, overnight changes of 2026-10-01 |
| + |
| +Read the round 1 brief (`CODEX-REVIEW-BRIEF-carloku-paid-launch-overnight.md`) for the fixed rulings and the output rules, the round 3 to 15 briefs for rulings 6 to 12, then `CODEX-DISPOSITIONS-carloku-paid-launch-overnight.md` (its last section covers round 15) and `implementation/carloku-paid-launch/OUTSTANDING.md`. |
| + |
| +Review the commit named in the prompt. Read source only with `git show <commit>:<path>`. The changes since round 15 are `git diff 973a9f15 <commit>`. The code changes are in `internal/objstore` (`sweep.go`, `walk.go`, `objstore.go`, tests) and `internal/console` (`apiscan.go`, `economicspass.go`, tests). The rest is documents. |
| + |
| +Still the fixed snapshot: round 15's findings, nothing else. Nothing may attest. The quantity worker has two switches, both unset. Billing is off. |
| + |
| +Do not report again, as findings, what `OUTSTANDING.md` lists as open: items 6 and 6a, 7 and 9, 19a to 19h. Do report anything that makes one of them worse than the document says, and any claim about them the code contradicts. |
| + |
| +## First: a disposition table |
| + |
| +For every round 15 finding (Astra F1, F2 and the claims; Opus N1 to N3 and the claims): closed, partly closed, not closed, or accepted and left open as the dispositions say. File and line of the fix and of the test, and whether the test would pass without the fix. |
| + |
| +## Then: new findings only |
| + |
| +- **Readers of a retained store's bucket.** A teardown that leaves its bucket to another account's store keeps its record, with `Provisioned` still naming the bucket. The claim: from the moment the teardown block is on the record, nothing reads that bucket as this store's and shows or bills the result to this store's account. Fixed this round: a sweep in flight (`writeSidecar`), the scan status (`scanReply`), the daily storage pass (`storagePass`). Find another: a queue drain, a status or settings view, a usage fold, an economics view, an export, a metric. |
| +- **`Regrant`** now refuses a record with nothing provisioned. Find a scanning door that this leaves unable to converge a store it should. |
| +- **The storage hold.** A store under teardown is recorded as unswept. Say what that holds, for whom and for how long, and whether an ordinary (not retained) teardown that takes more than a day changes anything it should not. |
| +- **Claims** in `AGENT_HANDOFF.md` item 5, `OUTSTANDING.md` 19, 19a, 19f, 19g and 19h, and the round 15 dispositions. |
| + |
| +## What to return |
| + |
| +1. The disposition table. |
| +2. New findings, most severe first, in the round 1 format. |
| +3. Claims the code contradicts. |
| +4. A one-line verdict: accept, accept with fixes, or reject. |
diff --git a/docs/superpowers/specs/OPUS-REVIEW-carloku-paid-launch-overnight-round15.md b/docs/superpowers/specs/OPUS-REVIEW-carloku-paid-launch-overnight-round15.md| new file mode 100644 |
| index 000000000..281d02c45 |
| --- /dev/null |
| +++ b/docs/superpowers/specs/OPUS-REVIEW-carloku-paid-launch-overnight-round15.md |
| @@ -0,0 +1,103 @@ |
| +# Opus production and authority review, round 15: Carloku paid launch, `aed42815..973a9f15` |
| + |
| +**Verdict: accept with fixes.** Every round 14 finding is closed for its reported shape. One new P2: the daily storage pass still sizes a bucket that a retained teardown left to another account, and puts those bytes on the departed store's billed storage meter. It also has two P3 findings. |
| + |
| +I read the code with `git show` only. I changed nothing, ran no tests and made no network calls. Each "fails without the fix?" answer comes from reading the test against the code. The code changes since round 14 (`d5705a63..973a9f15`) are in `internal/objstore` (`objstore.go`, `walk.go`, `ownership.go`, `sweep.go`, `queue.go`, tests) and in `internal/console/apiscan.go` and its test. |
| + |
| +Abbreviations: **O** = `internal/objstore`, **C** = `internal/console`, **D** = `implementation/carloku-paid-launch`. |
| + |
| +**Production authority** |
| + |
| +- **Nothing attests.** `SetAttestation` and `SetAllClientsInstrumented` (`usage/durabilityledger.go:1455`, `:1475`) still have no non-test caller. |
| +- **The quantity worker cannot reach Stripe with either switch unset.** `quantityWorker()` (`C/billingquantity.go:131-137`) needs a proration behaviour, and it also needs `QuantityFence == "stripe-sandbox-verified"`. `cmd/console/billing.go:40` rejects any other fence value. |
| +- **These files did not change this round:** `billingquantity.go`, `billingservice.go`, `internal/billingstripe`, `cmd/console`, `walklock.go`, `fence.go`, `aws.go`, `internal/sigfence`, `internal/usage`. |
| + - My round 9 to 14 reading of the send lease, the key fence, the second switch, the walk lease and the signing fence stands. |
| + - AWS is still the only backend on which the fence has been observed (19d). |
| + - `TestSandboxQuantityFence` has still not been run, and no document says it has. |
| +- **Billing.** Billing is off. N1 below is a path that would charge a customer wrongly once storage is billed. |
| +- **Data.** I found no path that destroys a customer's data. |
| +- **Leaks.** I found no leak of an identifier or a secret. N1 publishes another account's aggregate byte count, nothing more. |
| + |
| +## 1. Disposition table (round 14) |
| + |
| +| Finding | Status | Fix | Test; fails without the fix? | |
| +|---|---|---|---| |
| +| **Astra F1**: the ledger reconciles away a refused record's hold | **Closed.** `storeHolders` sets `refused` (`O/walk.go:1417`). `beginBucketTeardown` no longer excuses a refused holder by a ledger entry (`O/ownership.go:516`). The take-back check no longer excuses one by `left` (`O/walk.go:321`). I followed every holder list to its decision and none loses the mark: the minting check counts any holder (`:451`), and the no-ledger branch counts `len(holders)` (`:1087-1095`). Ledger-derived holders are open intervals, so they need no mark. | `ownership.go`, `walk.go` | `TestARefusedRecordsHoldIsNotReconciledAwayByAnOldClosedInterval` (`O/ownership_test.go:3263`): **fails** both ways. Without `!h.refused` in teardown, Y is destroyed. Without it in Grant, the take-back gets through. | |
| +| **Astra F2**: a scanning re-grant takes a store back after a teardown in between | **Closed.** `Regrant` → `grant(noTakeBack)` refuses `rec.Teardown != nil` on the record read under the lease, after the estate comparison and before any write (`O/walk.go:252`). `regrantForScanning` uses it (`C/apiscan.go:583`). The only other `Grant` callers are the operator grant door and the member provision door, and both are meant to make a grant decision. A teardown that starts after the lease is taken needs the same lease name, which is excluded until the lease expires. On AWS the fence covers the time after that. | `walk.go`, `apiscan.go` | `TestRegrantNeverTakesAStoreBack` (`:3307`) and `TestScanRegrantDoesNotTakeBackAStoreTornDownInBetween` (`C/apiscan_test.go:716`): **both fail** without the check, through the error and the record bytes. The console test's key-count check would not catch a take-back on its own, because reap plus mint leaves one key either way; the record-bytes check does. | |
| +| **Astra F3**: an unchanged config does not prove the delivery will not land | **Closed as accepted.** `configGeneration` is gone, and the mark stays on every failure from the delivery on (`O/walk.go:764-772`). The cost of this is open as 19h. | `walk.go` | `TestRotateAfterAResealFailureIsRepairedByTheGrant` (`O/walk_test.go:710`): **fails** if the mark is cleared on a failed delivery. | |
| +| **Astra F4**: the earlier nested mark is dropped | **Closed.** `Inspect` carries `grant.delivering` into `Record.Delivering` and clears it (`O/objstore.go:257-260`). `Get`, the holder inventory and every walk read through `Inspect`. **This build cannot write the earlier field back out:** `omitempty` is false after `Inspect`, and no literal sets it. `release/trash.go` parses the record raw, but it refuses any record with a grant block. | `objstore.go` | `TestTheMarkIsReadFromWhereAnEarlierBuildWroteIt` (`:3352`): **fails** at the `Get` assertion. | |
| +| **Astra claims** (the F1 to F4 claims, and "since round eleven …") | **Corrected**, apart from the residue listed in §3. `OUTSTANDING.md` 19 now says that some findings need only ordinary walks. | — | — | |
| +| **Opus N1**: the mark was written before step one | **Closed.** The mark is written after the mint and before `writeEnv` (`O/walk.go:739-762`). A failure at steps one to three leaves no mark, and the re-run keeps `KeyID`. | `walk.go` | `TestRotationThatFailsBeforeDeliveryIsResumedWithItsOverlap` (`:3545`): **fails** with the mark moved back, both for the reap case and the mint case. | |
| +| **Opus N2**: the scan pass sweeps a retained store | **Closed for the sweep and the queue** (`O/sweep.go:140`, `O/queue.go:487`). `drainOne` checks `queuePermitted` again before each item. The storage pass is another reader (new N1). | `sweep.go`, `queue.go` | `TestAStoreUnderTeardownIsNotSweptOrQueued` (`:3592`): **fails** through `neverLister` and through `ok`. | |
| +| **Opus N3**: an empty provisioned name beside a grant bucket | **Closed.** `Coherent` refuses it (`O/walk.go:1590`), and `storeHolders`' refused branch still counts the grant bucket. Nothing in this build writes an empty provisioned name: the only constructor is `O/walk.go:599`. | `walk.go` | `TestAProvisionedBlockWithNoBucketBesideAGrantThatNamesOneIsRefused` (`:3616`): **fails**. | |
| +| **Opus N4**: the revoke door's guard was untested | **Closed.** Scanning is enabled first, then the test asserts it is still enabled after the enforce and revoke doors. | `C/apiscan_test.go:655-683` | `TestScanDoorsDoNotReachTheScannerForARefusedRecord` (`:642`): **fails** in the "teardown begun" case without `rec.Teardown == nil` at `C/apiscan.go:421`. The refused-record cases do not reach the guard, because `Get` errors first. | |
| +| **Opus claims** (the Rotate doc, `leaseForRotation`, the `Inspect` doc, handoff item 5) | **Corrected**, apart from the `Record.Delivering` doc (§3). | — | — | |
| + |
| +## 2. New findings, most severe first |
| + |
| +### N1. P2: the storage pass sizes a bucket that a retained teardown left to another account, and bills it to the store that left |
| + |
| +**Where:** `C/economicspass.go:213-219` sizes `rec.Provisioned.Bucket` for any record with a `Provisioned` block. `:1021-1023` folds `StoreBytes` into `storage_gb_month`, which is a billed meter (`C/billingcalculation.go:23`, `economics/compute.go:104`). |
| + |
| +**Scenario:** |
| +1. An operator adopted bucket X for A (account 1) and C (account 2). |
| +2. A's member destroys the store. Teardown revokes A's user, closes A's interval and returns `ErrBucketRetained`. A's record keeps `Provisioned{X}` and a teardown block. Nothing ever runs that teardown again. |
| +3. Every day, `storagePass` sizes X with the deployment's credentials and writes the full size as A's `StoreBytes`. |
| +4. A's statement carries `storage_gb_month` for data that is C's, written after A's access ended, for as long as X lives. Once storage is billed, A is charged for it, and the statement shows A the size of C's data. |
| + |
| +This is the same class as round 14 N2, which fixed the sweep but not this reader. The bug predates this branch. |
| + |
| +**Smallest fix:** in `storagePass`, do not size a record with `rec.Teardown != nil`. Record it as unprovisioned, or hold it, but never at C's size. Add a test: a retained teardown, one storage pass, and an assertion that the sizer was not called for that record. |
| + |
| +### N2. P3: `Regrant` can mint a first store, which is not a convergence |
| + |
| +**Where:** `regrantForScanning` checks `Provisioned != nil` on its own read (`C/apiscan.go:567`). `grant(noTakeBack)` refuses only on a teardown block (`O/walk.go:252`). The `Regrant` doc says it "converges an already granted store". |
| + |
| +**Scenario:** |
| +1. Between `regrantForScanning`'s read and `Regrant`'s first read, a destroying teardown finishes and deletes the record. |
| +2. The member declares again. |
| +3. `Regrant` reads a declaration-only record and runs a full first grant. It uses the old record's source (operator, or account approval) and is attributed to the operator, without the provision door's approval check. |
| + |
| +The window is two consecutive reads, so this is hard to reach. Nothing crosses an account boundary. |
| + |
| +**Smallest fix:** under `noTakeBack`, also refuse a record read under the lease that has no `Provisioned` block. |
| + |
| +### N3. P3 (latent): the scan grant door checks for a teardown, then enables scanning, as separate steps |
| + |
| +**Where:** `C/apiscan.go:360-364`. The door reads the record, checks `Teardown == nil`, then calls `EnsureEnabled(bucket)`. |
| + |
| +**Scenario:** a retained teardown runs between the check and the call. Scanning is then enabled on X, which C holds, and no record or ledger entry says the scanner is owed. |
| + |
| +**Why it is latent:** today's only scanner (ClamAV, `O/clamav.go:92`) does nothing in `EnsureEnabled`. A plan-based scanner would make this the 19f shape. |
| + |
| +**Smallest fix:** call `EnsureEnabled` only after the record write, and only when the record read under that write has no teardown block. Alternatively, add it to 19f. |
| + |
| +### Asked and not found |
| + |
| +- **A bucket destroyed, or access given, past a refused record:** none. The teardown decision, the no-ledger decision, the minting check and the take-back check all count a refused holder whatever the ledger shows. `stillDestroying` reads only the ledger, but this build cannot write a refused record, because `Put` refuses one. |
| +- **A scanning door taking a store back:** none through `Regrant`. |
| + - A teardown block that a member's own take-back cleared is a legitimate state, and converging it is correct. |
| + - The door's failure copy still says "re-run `carlos store grant`", which on a torn-down store means an operator take-back. That copy is 19g's, so I do not report it again. |
| +- **A key change the app could receive with no mark on the record:** none. |
| + - Grant marks in its counting write, before `EnsureUser`. |
| + - Rotate marks before `writeEnv`. If the mark write fails, nothing is delivered. If it fails with a lost response, the result is a conservative mark. |
| + - Step five's retry clears the mark only together with the key it records. |
| + - `RotateFinish` is refused while the mark stands. |
| +- **A store that can be neither rotated nor repaired:** none. Every marked record can be repaired by an operator's grant, and by a member's grant while the account's approval stands. The repair swaps the key without overlap (19h). |
| +- **The quantity worker and the walk fence:** unchanged, and nothing in this diff changes how they work. |
| + |
| +## 3. Claims the code contradicts |
| + |
| +| Location | Claim | What the code supports | |
| +|---|---|---| |
| +| `O/objstore.go:158-160` (`Record.Delivering` doc) | It "is set by a grant or a rotation before it reaps the user's keys and mints" | A rotation now sets it after the reap and the mint, just before delivery (`O/walk.go:748`). The handoff states this correctly; the field doc does not. | |
| +| `D/AGENT_HANDOFF.md` item 5 | "A rotation that fails before delivery leaves no mark and is resumed by running it again" | This holds except when the mark's own write fails with a lost response: the mark may have landed, and then the repair is a grant. That is conservative, but the sentence is absolute. | |
| +| `D/AGENT_HANDOFF.md` item 5; dispositions, round 14 N2 row | The scan pass does not read a store whose teardown has begun, which reads as "no reader publishes a retained bucket" | The storage pass sizes it and bills it (N1). | |
| +| `C/apiscan_test.go:733-735` (comment) | The fixture is "as its record stands afterwards … block written, user gone" | The fixture keeps the user and its key. The assertions still hold, but the comment describes a state the test does not build. | |
| +| `D/OUTSTANDING.md` 19h | Waiting out the signing window "would keep the overlap" | That rests on the signing fence, which has been observed only on AWS (19d). On another backend the window does not fence a late config write. | |
| + |
| +Nothing in `walklock.go`, `fence.go`, `aws.go`, the idempotency evidence document, the quantity worker's fence or second switch, or `OUTSTANDING.md` items 6, 6a, 7, 9 and 19a to 19h is made worse by this round. |
| + |
| +## 4. Verdict |
| + |
| +**Accept with fixes.** Nothing attests, the worker cannot reach Stripe with either switch unset, and nothing destroys data or leaks a secret. Every round 14 finding is closed. N1 (the storage pass over a retained bucket) should land before storage is billed. N2, N3 and the `Delivering` doc are small. |
diff --git a/implementation/carloku-paid-launch/AGENT_HANDOFF.md b/implementation/carloku-paid-launch/AGENT_HANDOFF.md| index d0f24638b..04a612888 100644 |
| --- a/implementation/carloku-paid-launch/AGENT_HANDOFF.md |
| +++ b/implementation/carloku-paid-launch/AGENT_HANDOFF.md |
| @@ -32,7 +32,7 @@ When a review is approved, apply each string byte for byte: the site by replacin |
| 2. Console GET `/a/{sqid}/billing` (`billingpage.go`, `templates/billing.html`): current plan, owner-only chooser posting to the existing checkout door, `?country=` price preview computed from the plans terms, error and pending banners, portal button once a customer exists, read-only for members, byte-identical 404 for non-members, closed state when billing is off. `plans.Countries()` exposes the embedded country list. |
| 3. Durability ledger: the epoch read race is fixed (epoch and created-at are immutable after recover) with a `-race` regression. An unmetered replication fallback on a box whose ledger opened now files an escape (`DurabilityLedger.Escaped`): the box-day is incomplete and the owner is excluded for the run. A box whose ledger failed to open has no publisher and files nothing. The console fold asks each box's published day to prove the union of that box's durability, metrics and edge windows; a box with no ledger directory, no day file, or a ledger whose coverage starts after the box began serving holds the day for everyone. There is no tolerance. Durability days are counted per account: a box-day attested with an exclusion set is complete for every account the box did not exclude. The move worker's checkpoint is recorded against the route's owner. Nothing attests: attestation is false everywhere, in this code and in production. The scoped attestation (path A) was built, reviewed in round six, found unable to prove an exclusive writer, and removed from this branch. It is kept on `feat/carloku-scoped-attestation-20261001` with its findings open. Replication-write completeness is a launch blocker, not a waived requirement. |
| 4. `internal/awsbilling`: Cost and Usage Report reader (versioned deliveries only), newest-delivery selection, digests, charged S3 transfer classifier. It holds on a charged usage line it does not count and on any non-usage line (fee, refund, credit, discount) with a cost on a transfer usage type. Every transfer usage type the flagship account wrote from July to September 2026 is pinned in a test. |
| -5. `internal/objstore/ownership.go`: append-only bucket ownership ledger, opened at grant, closed at teardown. It resolves an owner only for a ledger that claims completeness, and only for hours that begin at or after the instant it began vouching, so a store's partial first hour never resolves (a ledger that began vouching exactly on the hour answers for that hour). Completeness needs a grant block recorded as minting, nothing delivered to the app yet, a probe showing the bucket does not exist, and an inventory showing no principal's policy names the bucket (`GranteeProber`), and is dated when both probes had answered. The AWS provisioner cannot list principals yet, so on AWS no ledger claims completeness, new stores included, and store transfer holds. A minting grant is refused while another app holds the name by ledger interval or by store record, and for a bucket that already exists and it never held. Grant, teardown and both rotation verbs are walks. A walk that spends backend authority takes a lease (`walklock.go`) on the bucket name, or on a hash of it when the name is one this build would refuse, or on a hash of the app when the record names no bucket: an object taken by compare-and-swap, bound to the invocation, thirty minutes long, standing until it runs out if the walk failed or any of its requests needed a second attempt. The record is read again under the lease and must still be the same estate, not only the same name: teardown and grant compare the grant instant, the key, how many grants have reached the provider, whether a teardown had begun, whether it noted the bucket deleted and how many take-backs it has seen; rotation compares the same estate and requires that no teardown has begun. Every grant is counted on the record before any call that changes anything at the provider, the user and its policy included (two read-only probes, of the bucket and of who can reach it, come first), so a healthy store granted again writes its record once. A grant marks the record before it changes the user's keys, and a rotation just before it delivers the new key (`Record.Delivering`); each takes the mark off when it records the result. A rotation that fails before delivery leaves no mark and is resumed by running it again, with the app's key kept. One whose delivery fails keeps the mark, because a failed write may still land. While the mark stands a grant run again walks in full and a rotation is refused (`ErrGrantUnfinished`). The mark does not save the key the app was holding: the repairing grant reaps and replaces it. A record from an earlier build carries no mark, which means not recorded, not finished. An older grant block that names no bucket is made to name it in the write that counts the walk. A record's bucket is the one it is provisioned on, else the one its grant block names: the arguments choose a bucket only for a store that has none, so a grant never moves a store to another name. A record whose grant block and provisioned block name different buckets (an earlier build could write one), or whose teardown note is about a bucket that is not its own, is refused by the record's one reader and one writer (`objstore.Get` and `objstore.Put`; the check is `objstore.Coherent`; `ErrTwoBuckets`, `ErrUnfinishedTeardown`), so no door, pass or walk can act on one or write one. The holder inventory reads such a record anyway (`objstore.Inspect`) and counts it as a holder of every bucket it names, whatever the ledger shows for that app. An operator repairs such a record by hand; there is no verb for it. A take-back over a bucket that a store of another account holds is refused unless an operator grants it; a store whose ledger interval is closed does not count as holding it; a scanning door never changes a store whose teardown has begun, its re-grant (`objstore.Regrant`) refuses one under the walk's own lease, and the scan pass does not sweep or queue one. A failed walk's lease stands until it runs out, with one exception: the same app re-running the same kind of walk may take it at once. A declaration-only teardown and the app purge take no lease; they delete the store record only as the generation they read. The user belongs to the app and the lease to the name, so a late request of a failed teardown can still remove the policy of a store granted afterwards under another name (round eight, Opus N2, open). Every request of a walk, to S3, to IAM and to the deployment bucket that holds the records and ledgers, is signed with the instant its lease was taken (`internal/sigfence`), so AWS refuses a request a paused process sends after its lease could be taken over. The record and its scan sidecar are deleted as the generation that was read. The signing-time rule was checked read-only against S3 and IAM with a list and two gets (`TestFenceAgainstAWS`); no delete was sent. A declaration-only teardown takes no lease (there is no bucket name) and relies on the conditional delete alone. A teardown writes a block on its own record before it touches anything and decides by compare-and-swap on the ledger (for a bucket name this build would refuse there is no ledger, and it decides from the store records alone): if nobody else holds the bucket it marks the bucket as being destroyed; otherwise it leaves the bucket and its scanner, revokes its own credentials, and records that the scanner is owed. A grant by the same app over its own unfinished teardown takes the store back, runs in full, and voids the teardown's note that the bucket was deleted. A teardown that leaves a bucket to another account keeps its record and returns `ErrBucketRetained` (discussion 191). It can finish once that bucket has been destroyed, when someone runs it again; nothing runs it again by itself. For a bucket name this build would refuse there is no ledger: a store whose teardown has begun is not counted as a holder there, so the last of two sharers destroys the bucket. `ErrBucketRetained` has no member copy yet; it needs Paul's review. A randomized simulation (`walk_sim_test.go`) checks the invariants over sequential walks with provider and store faults, including failures after the effect, and walks frozen past their lease while another takes the name. It does not generate a second bucket name, rotations, stores with no ledger interval or disagreeing clocks; those have direct tests. |
| +5. `internal/objstore/ownership.go`: append-only bucket ownership ledger, opened at grant, closed at teardown. It resolves an owner only for a ledger that claims completeness, and only for hours that begin at or after the instant it began vouching, so a store's partial first hour never resolves (a ledger that began vouching exactly on the hour answers for that hour). Completeness needs a grant block recorded as minting, nothing delivered to the app yet, a probe showing the bucket does not exist, and an inventory showing no principal's policy names the bucket (`GranteeProber`), and is dated when both probes had answered. The AWS provisioner cannot list principals yet, so on AWS no ledger claims completeness, new stores included, and store transfer holds. A minting grant is refused while another app holds the name by ledger interval or by store record, and for a bucket that already exists and it never held. Grant, teardown and both rotation verbs are walks. A walk that spends backend authority takes a lease (`walklock.go`) on the bucket name, or on a hash of it when the name is one this build would refuse, or on a hash of the app when the record names no bucket: an object taken by compare-and-swap, bound to the invocation, thirty minutes long, standing until it runs out if the walk failed or any of its requests needed a second attempt. The record is read again under the lease and must still be the same estate, not only the same name: teardown and grant compare the grant instant, the key, how many grants have reached the provider, whether a teardown had begun, whether it noted the bucket deleted and how many take-backs it has seen; rotation compares the same estate and requires that no teardown has begun. Every grant is counted on the record before any call that changes anything at the provider, the user and its policy included (two read-only probes, of the bucket and of who can reach it, come first), so a healthy store granted again writes its record once. A grant marks the record before it changes the user's keys, and a rotation just before it delivers the new key (`Record.Delivering`); each takes the mark off when it records the result. A rotation that fails before delivery leaves no mark and is resumed by running it again, with the app's key kept (unless the write of the mark itself was lost in transit and landed, in which case the mark stands, which errs toward the repair). One whose delivery fails keeps the mark, because a failed write may still land. While the mark stands a grant run again walks in full and a rotation is refused (`ErrGrantUnfinished`). The mark does not save the key the app was holding: the repairing grant reaps and replaces it. A record from a build before the mark existed carries none, which means not recorded, not finished; the one earlier build of this branch that kept the mark inside the grant block is read. An older grant block that names no bucket is made to name it in the write that counts the walk. A record's bucket is the one it is provisioned on, else the one its grant block names: the arguments choose a bucket only for a store that has none, so a grant never moves a store to another name. A record whose grant block and provisioned block name different buckets (an earlier build could write one), or whose teardown note is about a bucket that is not its own, is refused by the record's one reader and one writer (`objstore.Get` and `objstore.Put`; the check is `objstore.Coherent`; `ErrTwoBuckets`, `ErrUnfinishedTeardown`), so no door, pass or walk can act on one or write one. The holder inventory reads such a record anyway (`objstore.Inspect`) and counts it as a holder of every bucket it names, whatever the ledger shows for that app. An operator repairs such a record by hand; there is no verb for it. A take-back over a bucket that a store of another account holds is refused unless an operator grants it; a store whose ledger interval is closed does not count as holding it; a scanning door never changes a store whose teardown has begun, its re-grant (`objstore.Regrant`) refuses one under the walk's own lease and never makes a first store, the scan pass does not sweep or queue one, a sweep already running does not publish after one begins, the scan status shows no coverage for one, and the daily storage pass holds it instead of sizing its bucket. A failed walk's lease stands until it runs out, with one exception: the same app re-running the same kind of walk may take it at once. A declaration-only teardown and the app purge take no lease; they delete the store record only as the generation they read. The user belongs to the app and the lease to the name, so a late request of a failed teardown can still remove the policy of a store granted afterwards under another name (round eight, Opus N2, open). Every request of a walk, to S3, to IAM and to the deployment bucket that holds the records and ledgers, is signed with the instant its lease was taken (`internal/sigfence`), so AWS refuses a request a paused process sends after its lease could be taken over. The record and its scan sidecar are deleted as the generation that was read. The signing-time rule was checked read-only against S3 and IAM with a list and two gets (`TestFenceAgainstAWS`); no delete was sent. A declaration-only teardown takes no lease (there is no bucket name) and relies on the conditional delete alone. A teardown writes a block on its own record before it touches anything and decides by compare-and-swap on the ledger (for a bucket name this build would refuse there is no ledger, and it decides from the store records alone): if nobody else holds the bucket it marks the bucket as being destroyed; otherwise it leaves the bucket and its scanner, revokes its own credentials, and records that the scanner is owed. A grant by the same app over its own unfinished teardown takes the store back, runs in full, and voids the teardown's note that the bucket was deleted. A teardown that leaves a bucket to another account keeps its record and returns `ErrBucketRetained` (discussion 191). It can finish once that bucket has been destroyed, when someone runs it again; nothing runs it again by itself. For a bucket name this build would refuse there is no ledger: a store whose teardown has begun is not counted as a holder there, so the last of two sharers destroys the bucket. `ErrBucketRetained` has no member copy yet; it needs Paul's review. A randomized simulation (`walk_sim_test.go`) checks the invariants over sequential walks with provider and store faults, including failures after the effect, and walks frozen past their lease while another takes the name. It does not generate a second bucket name, rotations, stores with no ledger interval or disagreeing clocks; those have direct tests. |
| 6. Statement folds: `console/durabilityfold.go` (`durability_writes_1k`, policy in `usage.BillableReplicationWrites`) and `console/storeegressfold.go` (`store_egress_gb`). `usage.MeterSource` names a source for both meters. Month evidence pins ownership ledgers by their month projection and knows the durability box directory. Statements carry the five billed meters by UTC day (`Statement.Days`), summing to the monthly totals; the other meters are not recorded by day. With a transfer report configured, the statements catch-up walks a 36-month window and sweeps every month that has an open statement, is missing a statement an account is owed (the platform's included), or is a gap between metered months. An account's statement that will not parse, or that cannot be read three passes running, is reported once per process and the month is classified without it. The platform's own statement is different: while it cannot be read the month stays unknown, for as long as that lasts. A month seen with no statement is remembered for six hours, a closed statement for the life of the process. |
| 7. Operator API doors (`billingoperator.go`): list and resolve billing review holds, list and set the house marker. No CLI verb yet. |
| 8. Team quantity worker (`billingquantity.go`) behind two switches, both unset: `CONSOLE_BILLING_PRORATION` (`immediate` or `next_invoice`, Paul's decision) and `CONSOLE_BILLING_QUANTITY_FENCE=stripe-sandbox-verified` (set only after `TestSandboxQuantityFence` has passed against a Stripe test-mode account; it has not been run). With either unset it makes no provider call. An operation is recorded by a compare-and-set on the billing row's revision, subscription, period, quantity and status, on the target and newest pending addition recomputed from the database, and on its own instant lying inside the period and not before the addition it charges for. The request dates its proration at that instant (`proration_date`). Sending and retiring exclude each other through a send lease on the operation's row, owned by one sender's token; an operation dated ahead of the observing clock is left alone. Every key ends in the account's operation serial. An operation nobody can show ran is acknowledged only after its idempotency key has been tried at the provider with a metadata-only request (the fence), or after a person has resolved the review it was held with: a resolved review stands in for the fence, on that person's word and with no provider check. If the fence request took the key, the operation never ran and may be retired, on a provider read taken after the fence when the subscription is current. If the key is held by the operation's own request, that request began and its outcome is not known: the operation is recorded as landed if the provider shows its target, and otherwise held with a review (`billing_quantity_operation_unsettled`) and never replaced until a person resolves it. A fence the provider refuses is held the same way. The fence rests on Stripe's documented key rules, is modelled by a fake in the tests, is not verified against Stripe, and lasts only as long as Stripe remembers a key (documented as at least 24 hours): `docs/superpowers/specs/2026-10-01-stripe-idempotency-evidence.md`. A quantity that goes down inside a period files `billing_quantity_lowered`. An operation is confirmed only by the provider's success answer to its own request. A confirmed landing takes effect at the operation's instant, so a seat's charge and its allowance start together; a quantity that merely matches an unconfirmed operation's target is recorded as `landed_unverified`, dated when observed, with a review. If the provider refuses a request and names `proration_date`, the operation's key is fenced at once and it is retired on the next read; if it does not name it, the operation waits out the 22-hour replay limit. None of the provider's side is confirmed: test mode is not authorized. Only additions of the subscription's current period can raise it; older ones are closed uncharged with a review; additions and operations of an ended subscription are closed. It does not yet lower the quantity at renewal. |
diff --git a/implementation/carloku-paid-launch/OUTSTANDING.md b/implementation/carloku-paid-launch/OUTSTANDING.md| index 46bf9db85..ed4576b39 100644 |
| --- a/implementation/carloku-paid-launch/OUTSTANDING.md |
| +++ b/implementation/carloku-paid-launch/OUTSTANDING.md |
| @@ -14,8 +14,8 @@ Work is at one of these stages, and each item says which: |
| - **Absent**: no code. |
| - **Designed**: a written plan, no code. |
| -- **Partial, unwired**: code and tests on a side branch that nothing calls. Not gated, not reviewed. |
| -- **On the launch branch**: gated locally on the exact commit. No review round has passed and CI has not run. |
| +- **Partial, unwired**: code and tests on a side branch that nothing on the launch branch calls. Not reviewed, no CI run. Each item says whether the local gate has passed on it. |
| +- **On the launch branch**: gated locally on the exact commit, and CI has passed on it. No review round has passed. |
| - **Proven**: as above. |
| ## Reviewable now |
| @@ -32,9 +32,9 @@ Each is concrete. None is waived. |
| ### Meters (the complete metered model cannot invoice until both are proven) |
| -1. **Replication writes (`durability_writes_1k`) cannot be proven on any box.** The external replicator writes outside the ledger. Path A (scoped attestation) was built, reviewed in round six, and cannot prove an exclusive writer; it is off this branch and parked on `feat/carloku-scoped-attestation-20261001`. Path B (the platform's own replicator as the only writer) is **partial, unwired**, on `feat/carloku-metered-replicator-20261001` (`2140eb7b`, on the forge): a litestream store whose every replica client is metered with no unmetered fallback, an owner for each database the box replicates, and a fleet store that refuses anything it cannot meter, each with tests. **Absent**: the agent wiring that binds those meters to the box's ledger, a restore path that does not shell out to the external binary, the unit change, moving the replica credentials so nothing else on the box holds them (the unit runs as the same user as the agent today), and any attestation. Until all of that exists and is approved for a box, this meter is incomplete every month and nothing is invoiced. |
| +1. **Replication writes (`durability_writes_1k`) cannot be proven on any box.** The external replicator writes outside the ledger. Path A (scoped attestation) was built, reviewed in round six, and cannot prove an exclusive writer; it is off this branch and parked on `feat/carloku-scoped-attestation-20261001`. Path B (the platform's own replicator as the only writer) is **partial, unwired**, on `feat/carloku-metered-replicator-20261001` (`ebab1a37`, on the forge; the full local gate passes on it): a litestream store whose every replica client is metered with no unmetered fallback, an owner for each database the box replicates, and a fleet store that refuses anything it cannot meter, each with tests. **Absent**: the agent wiring that binds those meters to the box's ledger, a restore path that does not shell out to the external binary, the unit change, moving the replica credentials so nothing else on the box holds them (the unit runs as the same user as the agent today), and any attestation. Until all of that exists and is approved for a box, this meter is incomplete every month and nothing is invoiced. |
| 2. **Store transfer (`store_egress_gb`) has no source.** No Cost and Usage Report export exists on the account. It needs the scoped infrastructure approval, a first real delivery, an IAM principal inventory in the AWS provisioner (not built), and the bucket ownership attestation for stores that already exist (designed, not built). |
| -3. **A store's partial first hour** with a charged line holds the month, and nothing releases it. It holds by default and nobody is asked to waive it. What can release it is evidence: an operator door that records, for one bucket-hour, who is shown to have owned it and on what evidence, which the fold then honours. **Partial, unwired into this branch**, on `feat/carloku-bucket-hour-evidence-20261001` (`457c8da3`): the record, the fold change and three operator API doors, with tests and mutation checks. It accepts only the hour a complete ledger began vouching in and only the owner that ledger names. The full local gate passes on it. Not reviewed, no CI run, no CLI verb, and its operator-facing sentences are unreviewed. |
| +3. **A store's partial first hour** with a charged line holds the month, and nothing releases it. It holds by default and nobody is asked to waive it. What can release it is evidence: an operator door that records, for one bucket-hour, who is shown to have owned it and on what evidence, which the fold then honours. **Partial, unwired into this branch**, on `feat/carloku-bucket-hour-evidence-20261001` (`58c1870c`): the record, the fold change and three operator API doors, with tests and mutation checks. It accepts only the hour a complete ledger began vouching in and only the owner that ledger names. The full local gate passes on it. Not reviewed, no CI run, no CLI verb, and its operator-facing sentences are unreviewed. |
| 4. **A box's first minutes** before its ledger opened hold that day for everyone. Same as item 3: it holds by default, and the release is an operator evidence door, not a waiver. **Absent**. |
| ### Stripe |
| @@ -42,7 +42,7 @@ Each is concrete. None is waived. |
| 5. **No catalog, portal configuration, webhook endpoint or runtime key exists.** The plan and a print-only script are written. Needs scoped approval, then a test-mode rehearsal, then a real purchase. |
| 6. **The quantity worker's fence is unverified against Stripe.** It rests on documented idempotency rules, modelled by a fake. `TestSandboxQuantityFence` must pass in a Stripe sandbox, under both proration policies, before `CONSOLE_BILLING_QUANTITY_FENCE` is set. Test mode is not authorized, so it has not been run. A pass is not enough by itself: check 5b logs what the worker's own update does with a declining card, and that log has to be read and decided on, because the worker records an addition as charged on any success and sends no `payment_behavior`. Two things it cannot settle and that stay open after it: what Stripe does with a key across an API key rotation, and how Stripe later reconciles a request it answered with a 500. |
| 6a. **An ended subscription's operation may need a person every time.** If Stripe refuses the fence request on a canceled subscription's item (the sandbox run logs what it answers), every canceled team subscription with an unconfirmed operation waits on a resolved review, and until then the account's quantity work is held, under a later subscription too. That needs a code decision from the sandbox log before the worker is switched on. |
| -7. **Defect, open: an old quantity request can run after Stripe has forgotten its key.** Stripe forgets keys after 24 hours. A sender frozen longer than that can still land a request for an operation that was retired and replaced. On the launch branch that is only detected after the write (`billing_quantity_lowered`), which is not a fix. The fix is **partial, unwired into this branch**, on `feat/carloku-late-request-fencing-20261001` (`68f77faf`, on the forge; design in `docs/superpowers/plans/2026-10-01-late-request-fencing.md`): a transport in which a request cannot outlive its connection, with tests. Its one premise, the provider's idle limit, is unmeasured. The path that could cause the defect, the quantity worker, stays disabled until that is gated, reviewed and measured in a Stripe sandbox. Nobody is asked to accept this risk. |
| +7. **Defect, open: an old quantity request can run after Stripe has forgotten its key.** Stripe forgets keys after 24 hours. A sender frozen longer than that can still land a request for an operation that was retired and replaced. On the launch branch that is only detected after the write (`billing_quantity_lowered`), which is not a fix. The fix is **partial, unwired into this branch**, on `feat/carloku-late-request-fencing-20261001` (`1772e840`, on the forge; the full local gate passes on it, after the gate caught and a commit fixed a timeout that the new transport reported as a network fault; design in `docs/superpowers/plans/2026-10-01-late-request-fencing.md`): a transport in which a request cannot outlive its connection, with tests. Its one premise, the provider's idle limit, is unmeasured. The path that could cause the defect, the quantity worker, stays disabled until that is reviewed, merged and measured in a Stripe sandbox. Nobody is asked to accept this risk. |
| 8. **Stripe's treatment of a proration date outside the period** is assumed from documentation. Same sandbox run. |
| 9. **Defect, open: a late invoice-line request can leave a finalized invoice with a wrong total.** Past Stripe's key memory a stale line request runs as new. If it lands between the engine's line check and the finalize, the invoice is finalized with a duplicate line. The engine flags it and will not charge it, but a finalized invoice is open at Stripe, and an open invoice can be paid by the customer from Stripe's hosted page or the portal. That is customer-payable exposure and it is treated as such. Prevention is the same transport as item 7, at the same stage (**partial, on the fencing branch**). That branch also voids such an invoice when it finds one; that is remediation after the fact, its limits are pinned by tests there (the invoice is payable until the void lands, and a customer who pays first is owed a refund), and it does not close this item. Billing stays OFF, so invoice writes are disabled until the prevention is proven. |
| @@ -63,16 +63,16 @@ Each is concrete. None is waived. |
| ### Process |
| -19. **No review round has passed.** Astra rejected rounds one to fourteen; Opus accepted each with fixes. Every round has closed the shapes the round before reported and found neighbouring ones, in the store walks since round seven. Some concern records no walk of this build writes; others need only this build's ordinary walks (a failed rotation, a retained teardown, a scanning door racing a teardown), and two were faults in the previous round's own fixes. Round eleven found one P1 and two P2 in the store walks (a grant could move an older record to another bucket name; a waiting rotation ignored a grant that ran; the user and its policy were written before the grant was counted). Its fixes are on the launch branch (`1395f7bf`). Round twelve read them: every round-eleven finding is closed for its reported shape, Astra rejected again (one P1, two P2) and Opus accepted with fixes (one P2, three P3). The new findings were about records no walk of this build writes (a teardown note for another bucket, a record naming two buckets) being acted on by teardown, rotation and the scanning doors; a rotation started after a grant that failed at its last write; and a take-back that returned access to a bucket another account holds, through the member's own door. Round thirteen read those fixes: closed for their reported shapes, Astra rejected again (two P1, four P2) and Opus accepted with fixes (one P2, five P3). The new findings were paths that read such a record and acted without the check (the holder inventory, the storage pass, the declare door), a rotation's own failed last write, and a retained record blocking another account's take-back. The check now sits in the record's reader and writer. Round fourteen read those: closed for their reported shapes; Astra rejected (two P1, two P2), Opus accepted with fixes (two P2, two P3). Its fixes are on the launch branch and unconfirmed until the next round reads them. Each round's fixes are on the launch branch, gated locally, and unconfirmed until the next round reads them. |
| +19. **No review round has passed.** Astra rejected rounds one to fifteen; Opus accepted each with fixes. Every round has closed the shapes the round before reported and found neighbouring ones, in the store walks since round seven. Some concern records no walk of this build writes; others need only this build's ordinary walks (a failed rotation, a retained teardown, a scanning door racing a teardown), and two were faults in the previous round's own fixes. Round eleven found one P1 and two P2 in the store walks (a grant could move an older record to another bucket name; a waiting rotation ignored a grant that ran; the user and its policy were written before the grant was counted). Its fixes are on the launch branch (`1395f7bf`). Round twelve read them: every round-eleven finding is closed for its reported shape, Astra rejected again (one P1, two P2) and Opus accepted with fixes (one P2, three P3). The new findings were about records no walk of this build writes (a teardown note for another bucket, a record naming two buckets) being acted on by teardown, rotation and the scanning doors; a rotation started after a grant that failed at its last write; and a take-back that returned access to a bucket another account holds, through the member's own door. Round thirteen read those fixes: closed for their reported shapes, Astra rejected again (two P1, four P2) and Opus accepted with fixes (one P2, five P3). The new findings were paths that read such a record and acted without the check (the holder inventory, the storage pass, the declare door), a rotation's own failed last write, and a retained record blocking another account's take-back. The check now sits in the record's reader and writer. Round fourteen read those: closed for their reported shapes; Astra rejected (two P1, two P2), Opus accepted with fixes (two P2, two P3). Round fifteen read those: all closed; Astra rejected (one P1, one P2), Opus accepted with fixes (one P2, two P3), all about readers of a bucket that a retained teardown left to another account (a sweep in flight, the scan status, the storage pass). Those fixes are on the launch branch and unconfirmed until the next round reads them. Each round's fixes are on the launch branch, gated locally, and unconfirmed until the next round reads them. |
| 19g. **Refusals whose doors say the wrong thing, and records only a person can repair.** A record that names two buckets (`ErrTwoBuckets`) or whose teardown note is about another bucket (`ErrUnfinishedTeardown`) is refused by the record's one reader and one writer, so nothing acts on it and nothing writes it; the settings page shows such a store as unreadable. No walk of this build writes either; whether an earlier build left such records in production has not been checked. There is no operator verb to repair one. The doors that meet these refusals answer with sentences written for other failures: the grant and provision doors say to run the grant again, the destroy door and the purge say to try again, the scanning doors say to try again or to re-run the grant. The same goes for a rotation refused because a grant must be run again first (`ErrGrantUnfinished`) and for a member's take-back refused because another account holds the bucket (`ErrBucketHeld`). Correct copy for each needs approval (A). |
| -19h. **A rotation whose delivery fails is repaired by a grant, which swaps the app's key with no overlap.** A failed delivery may still land, so the record stays marked, the next rotation is refused, and the grant run again replaces whichever key the app holds. Hosts hold a dead key until they adopt the new config. Before this branch such a rotation was resumed by running it again, which kept the overlap when the delivery had not landed and broke the app when it had. A repair that waits out the signing window and then reads which key was delivered would keep the overlap; it is not built. |
| -19f. **A store that leaves a bucket with an unsupported name does not hand on its scanner debt.** There is no ledger for such a name. The last holder would delete the bucket and leave a scanning plan behind. Today's scanner (ClamAV) has no plan to leave, so this is latent. |
| +19h. **A rotation whose delivery fails is repaired by a grant, which swaps the app's key with no overlap.** A failed delivery may still land, so the record stays marked, the next rotation is refused, and the grant run again replaces whichever key the app holds. Hosts hold a dead key until they adopt the new config. Before this branch such a rotation was resumed by running it again, which kept the overlap when the delivery had not landed and broke the app when it had. A repair that waits out the signing window and then reads which key was delivered would keep the overlap where the window is enforced (observed on AWS only, 19d); it is not built. |
| +19f. **A store that leaves a bucket with an unsupported name does not hand on its scanner debt.** There is no ledger for such a name. The last holder would delete the bucket and leave a scanning plan behind. Today's scanner (ClamAV) has no plan to leave, so this is latent. Also latent for the same reason: the scan grant door checks that no teardown has begun and then enables scanning as two steps, so a retained teardown landing between them would leave scanning enabled on the other account's bucket with nothing recording that it is owed. |
| 19e. **A purge can end with a store that was granted during it.** A declare and a grant admitted while the app was live can land between the purge's last check and the removal of its marker. The purge now reports that as an error and never deletes the record, but it does not prevent it: the doors check liveness when they admit a request, not when they write. And after that error the estate is named only by its record and a log line. The app's console row is still trashed, so the settings page and the store-destroy door answer 404 and a second purge answers "not trashed", until the console restarts and restores the row from the record. The flash the member sees says "Nothing was deleted", which is false in this case; correct copy needs approval (A). |
| 19b. **Two round-eight findings are accepted and not fixed.** A late request of a failed teardown can remove the policy of a store the same app is granted afterwards under another name (the store goes dark; nothing is lost or leaked). A console whose clock is far ahead reserves a bucket name until that time, with no operator verb to clear it. |
| 19c. **S3's conditional delete is unverified on a real backend.** The store walks and the app purge delete a record only as the generation they read. That has run against the directory store and an HTTP stub. No `If-Match` delete has been sent to AWS or to an S3-compatible store, and a backend that ignored the header would delete unconditionally. It needs one write to a scratch object, which is a production-adjacent write nobody has approved. |
| 19d. **The signing-time fence has been observed on AWS only.** An S3-compatible deployment or store endpoint has not been probed for its window. |
| -19a. **A retained teardown finishes only when someone runs it again.** Nothing runs it. A store that left a bucket to another account keeps its record until then. |
| -20. **Amadan CI has passed on `1395f7bf` and on `d0a21aa1`.** It sat queued on every tip all night and first ran at 07:34 Dublin on 2026-10-01. Any later code tip needs its own run; the local gate (build, vet, gofmt, `make test`, cross builds) is not CI and is not offered in its place. An agent account can neither see nor manage the runners. The hub's `carlos/platform` repository also answered 500 on and off through the night; bundles of every tip are in the run directory. |
| +19a. **A retained teardown finishes only when someone runs it again.** Nothing runs it. A store that left a bucket to another account keeps its record until then. Until then its scanning shows no coverage and its storage is held each day, not measured: the bucket's bytes are the other account's. |
| +20. **Amadan CI has passed on `1395f7bf`, `d0a21aa1`, `d5705a63` and `973a9f15`.** It sat queued on every tip all night and first ran at 07:34 Dublin on 2026-10-01. Any later code tip needs its own run; the local gate (build, vet, gofmt, `make test`, cross builds) is not CI and is not offered in its place. An agent account can neither see nor manage the runners. The hub's `carlos/platform` repository also answered 500 on and off through the night; bundles of every tip are in the run directory. |
| 21. **A scanner backend that calls a provider must be fenced like the walks.** Today's scanner (ClamAV) makes no provider call at teardown. A native one would, and its client would need the fenced signer. |
| ## Approvals to ask for, each with its exact scope |
diff --git a/internal/console/apiscan.go b/internal/console/apiscan.go| index 015fe258d..3025f0683 100644 |
| --- a/internal/console/apiscan.go |
| +++ b/internal/console/apiscan.go |
| @@ -90,6 +90,13 @@ func (s *Server) scanReply(ctx context.Context, acct *Account, app string, rec * |
| // fail the whole door. |
| log.Printf("console: api scan status %s/%s: sidecar: %v", acct.Sqid, app, err) |
| } |
| + // A store whose teardown has begun shows no coverage. Its access to the |
| + // bucket has ended or is ending, and a teardown that left the bucket to |
| + // another account's store keeps this record: a sidecar written around |
| + // that moment can count, or name, objects that are not this member's. |
| + if rec.Teardown != nil { |
| + side = nil |
| + } |
| out := map[string]any{ |
| "app": app, |
| "backend": s.cfg.StoreDeployment.Backend, |
diff --git a/internal/console/apiscan_test.go b/internal/console/apiscan_test.go| index 9aeb5209a..5744275bb 100644 |
| --- a/internal/console/apiscan_test.go |
| +++ b/internal/console/apiscan_test.go |
| @@ -730,8 +730,10 @@ func TestScanRegrantDoesNotTakeBackAStoreTornDownInBetween(t *testing.T) { |
| if err := fx.srv.regrantForScanning(ctx, pub, acct, "titogo", "alice@keymail.dev"); err != nil { |
| t.Fatalf("re-grant of a healthy store: %v", err) |
| } |
| - // The teardown that lands in between, as its record stands afterwards |
| - // when it left the bucket to another store: block written, user gone. |
| + // The teardown that lands in between, reduced to what the re-grant |
| + // decides on: its block on the record. (A real one would also have |
| + // removed the user; here the user and its key are left, so the |
| + // assertions below can see that the re-grant did not touch them.) |
| rec, _, err := objstore.Get(ctx, fx.rel, fx.sqid, "titogo") |
| if err != nil || rec == nil || rec.Provisioned == nil { |
| t.Fatalf("fixture: %+v %v", rec, err) |
| @@ -752,3 +754,38 @@ func TestScanRegrantDoesNotTakeBackAStoreTornDownInBetween(t *testing.T) { |
| t.Fatalf("the re-grant changed the record:\n%s\n%s", before, after) |
| } |
| } |
| + |
| +// The scan status of a store whose teardown has begun shows no coverage: a |
| +// sidecar written around that moment can count objects that are another |
| +// account's. |
| +func TestScanStatusShowsNoCoverageForAStoreUnderTeardown(t *testing.T) { |
| + fx, _, _ := scanArmed(t, "code-scan-status-teardown") |
| + ctx := context.Background() |
| + acct, err := fx.st.AccountBySqid(fx.sqid) |
| + if err != nil || acct == nil { |
| + t.Fatal(err) |
| + } |
| + if _, err := objstore.PutSidecar(ctx, fx.rel, fx.sqid, "titogo", &objstore.ScanSidecar{ |
| + LastComplete: &objstore.Sweep{Objects: 7, Scanned: 7}}, ""); err != nil { |
| + t.Fatal(err) |
| + } |
| + rec, _, err := objstore.Get(ctx, fx.rel, fx.sqid, "titogo") |
| + if err != nil || rec == nil { |
| + t.Fatalf("fixture: %+v %v", rec, err) |
| + } |
| + out, err := fx.srv.scanReply(ctx, acct, "titogo", rec) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if side, _ := out["coverage"].(*objstore.ScanSidecar); side == nil || side.LastComplete == nil { |
| + t.Fatalf("fixture: a live store's status shows no coverage: %v", out["coverage"]) |
| + } |
| + rec.Teardown = &objstore.TeardownBlock{StartedAt: time.Now().UTC()} |
| + out, err = fx.srv.scanReply(ctx, acct, "titogo", rec) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if side, _ := out["coverage"].(*objstore.ScanSidecar); side != nil { |
| + t.Fatalf("the status of a store under teardown shows coverage: %+v", side) |
| + } |
| +} |
diff --git a/internal/console/economicspass.go b/internal/console/economicspass.go| index 10e5ff9f4..1efed6578 100644 |
| --- a/internal/console/economicspass.go |
| +++ b/internal/console/economicspass.go |
| @@ -208,6 +208,15 @@ func (s *Server) storagePass(ctx context.Context, date string) error { |
| // that never had a store and never correctable afterward. |
| log.Printf("console: storage pass: store record %s/%s: %v", acct, app, gerr) |
| st.StoreUnswept = true |
| + } else if rec != nil && rec.Provisioned != nil && rec.Teardown != nil { |
| + // A store whose teardown has begun. If that teardown left |
| + // the bucket to another account's store, the record stays |
| + // and the bucket's bytes are that account's now: sizing it |
| + // here would put them on this app's storage meter. What |
| + // part, if any, is still this app's cannot be told, so the |
| + // day is held for it, not measured. |
| + st.StoreProvisioned = true |
| + st.StoreUnswept = true |
| } else if rec != nil && rec.Provisioned != nil { |
| st.StoreProvisioned = true |
| n, serr := s.sizeStore(ctx, rec.Provisioned) |
diff --git a/internal/console/economicspass_test.go b/internal/console/economicspass_test.go| index c18b8e1c3..debb0fa2b 100644 |
| --- a/internal/console/economicspass_test.go |
| +++ b/internal/console/economicspass_test.go |
| @@ -1528,3 +1528,40 @@ func TestStoragePassDoesNotMeasureARefusedRecord(t *testing.T) { |
| } |
| } |
| } |
| + |
| +// A store whose teardown has begun, and may have left its bucket to another |
| +// account's store, is not sized: the bucket's bytes are not this app's to |
| +// be metered for. The day is held for it. |
| +func TestStoragePassDoesNotSizeAStoreUnderTeardown(t *testing.T) { |
| + fx := newConsoleFixture(t) |
| + ctx := context.Background() |
| + at := time.Date(2026, 8, 10, 9, 0, 0, 0, time.UTC) |
| + rec := &objstore.Record{ |
| + Grant: &objstore.GrantBlock{Backend: "aws", Bucket: "carlos-store-shared-x", Source: objstore.GrantSourceOperator, At: at, By: "op"}, |
| + Provisioned: &objstore.Provisioned{Bucket: "carlos-store-shared-x", Region: "eu-west-1", IAMUser: "u", KeyID: "K", At: at}, |
| + Teardown: &objstore.TeardownBlock{StartedAt: at}, |
| + } |
| + if _, err := objstore.Put(ctx, fx.rel, fixtureAcct, fixtureApp, rec, ""); err != nil { |
| + t.Fatal(err) |
| + } |
| + sized := 0 |
| + fx.srv.storeSizer = func(context.Context, *objstore.Provisioned) (int64, error) { |
| + sized++ |
| + return 31 << 30, nil |
| + } |
| + if err := fx.srv.storagePass(ctx, "2026-08-12"); err != nil { |
| + t.Fatal(err) |
| + } |
| + raw, _, err := fx.rel.Get(ctx, usage.StorageDayKey("2026-08-12")) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + var day usage.StorageDay |
| + if err := json.Unmarshal(raw, &day); err != nil { |
| + t.Fatal(err) |
| + } |
| + app := day.Apps[fixtureAcct+"/"+fixtureApp] |
| + if sized != 0 || !app.StoreUnswept || app.StoreBytes != 0 { |
| + t.Fatalf("sized %d time(s), day entry %+v; want no sizing and an unswept store", sized, app) |
| + } |
| +} |
diff --git a/internal/objstore/objstore.go b/internal/objstore/objstore.go| index a23575f93..197baab67 100644 |
| --- a/internal/objstore/objstore.go |
| +++ b/internal/objstore/objstore.go |
| @@ -155,16 +155,18 @@ type Record struct { |
| // Teardown is what an unfinished teardown has already done, so a |
| // re-run does not do it again to something that has changed since. |
| Teardown *TeardownBlock `json:"teardown,omitempty"` |
| - // Delivering is set by a grant or a rotation before it reaps the |
| - // user's keys and mints and delivers a new one, and cleared by the |
| - // write that records the result. While it stands the record may name a |
| + // Delivering is set by a grant before it changes the user's keys at |
| + // all, and by a rotation after it has minted the new key and just |
| + // before it delivers it, and cleared by the write that records the |
| + // result. While it stands the record may name a |
| // key that is gone and the app may hold one the record does not name. |
| // The mark does not spare that key: it makes the state visible, so a |
| // rotation is refused (ErrGrantUnfinished) instead of quietly reaping |
| // on top of it, and a grant run again does not take the healthy |
| // short-circuit but walks in full and leaves the record naming a key |
| - // the user has. A record from a build before the mark reads false, |
| - // which means "not recorded", not "finished". |
| + // the user has. A record from a build before either form of the mark |
| + // (see GrantBlock.LegacyDelivering) reads false, which means "not |
| + // recorded", not "finished". |
| Delivering bool `json:"delivering,omitempty"` |
| } |
| @@ -271,6 +273,10 @@ func Put(ctx context.Context, s release.Store, acct, app string, rec *Record, et |
| if err := Coherent(rec, acct, app); err != nil { |
| return "", err |
| } |
| + // The mark has one place. Its earlier place is never written. |
| + if rec.Grant != nil && rec.Grant.LegacyDelivering { |
| + rec.Delivering, rec.Grant.LegacyDelivering = true, false |
| + } |
| rec.Schema = Schema |
| rec.App = app |
| body, err := json.MarshalIndent(rec, "", " ") |
diff --git a/internal/objstore/ownership_test.go b/internal/objstore/ownership_test.go| index 8d1c41abb..d1249af7f 100644 |
| --- a/internal/objstore/ownership_test.go |
| +++ b/internal/objstore/ownership_test.go |
| @@ -3632,3 +3632,29 @@ func TestAProvisionedBlockWithNoBucketBesideAGrantThatNamesOneIsRefused(t *testi |
| t.Fatal("the refused teardown removed the record or the object") |
| } |
| } |
| + |
| +// A convergence does not make a first store either. The store was |
| +// destroyed and declared again between the scanning door's look and its |
| +// re-grant: the re-grant finds nothing provisioned and refuses. |
| +func TestRegrantNeverMakesAFirstStore(t *testing.T) { |
| + ctx := context.Background() |
| + s := release.NewDirStore(t.TempDir()) |
| + _, pub := testSealKeypair(t) |
| + p := NewMemProvisioner("eu-west-1") |
| + leaseClock(t) |
| + declare(t, s, "bab", "titogo", "TITO_S3") |
| + if _, err := Regrant(ctx, s, p, testDeployment(), pub, "bab", "titogo", "op", GrantSourceOperator, t0); !errors.Is(err, ErrNotProvisioned) { |
| + t.Fatalf("a re-grant of a declaration: %v, want ErrNotProvisioned", err) |
| + } |
| + if len(p.Users) != 0 || len(p.Buckets) != 0 { |
| + t.Fatalf("the refused re-grant reached the provider: users %d, buckets %d", len(p.Users), len(p.Buckets)) |
| + } |
| + if rec := inspect(t, s, "bab", "titogo"); rec == nil || rec.Grant != nil { |
| + t.Fatalf("the refused re-grant wrote the record: %+v", rec) |
| + } |
| + if w, err := acquireBucketWalk(ctx, s, ledgerBucket, "bab", "second", "grant"); err != nil { |
| + t.Fatalf("the refused re-grant left the name reserved: %v", err) |
| + } else { |
| + w.release(true) |
| + } |
| +} |
diff --git a/internal/objstore/sweep.go b/internal/objstore/sweep.go| index 01466a46e..126ff77d7 100644 |
| --- a/internal/objstore/sweep.go |
| +++ b/internal/objstore/sweep.go |
| @@ -100,6 +100,11 @@ func (l *s3Lister) ListPage(ctx context.Context, bucket, startAfter string, limi |
| // to stop writing. RunSweep converts it to a nil error. |
| var errStandDown = errors.New("objstore: sweep standing down") |
| +// errSweepTornDown is the stand-down for a store whose teardown began while |
| +// the sweep was running. What the sweep holds in memory by then may count |
| +// another account's objects, so RunSweep hands back nothing at all. |
| +var errSweepTornDown = fmt.Errorf("%w: the store is being torn down", errStandDown) |
| + |
| // The two sweep failures an operator can actually act on, named so a door can |
| // tell them apart from a transport error and say which it was. Everything else |
| // stays a logged detail: bucket names, ARNs and request ids do not belong in a |
| @@ -297,6 +302,9 @@ func RunSweep(ctx context.Context, s release.Store, lister ObjectLister, sc Scan |
| } |
| newETag, err := writeSidecar(ctx, s, acct, app, side, etag, pageStart) |
| + if errors.Is(err, errSweepTornDown) { |
| + return nil, nil |
| + } |
| if errors.Is(err, errStandDown) { |
| return side, nil |
| } |
| @@ -364,6 +372,9 @@ func markNotApplicable(ctx context.Context, s release.Store, acct, app string) ( |
| side.NotApplicable = true |
| side.Current = nil |
| if _, err := writeSidecar(ctx, s, acct, app, side, etag, ""); err != nil { |
| + if errors.Is(err, errSweepTornDown) { |
| + return nil, nil |
| + } |
| if errors.Is(err, errStandDown) { |
| return side, nil |
| } |
| @@ -393,6 +404,13 @@ func writeSidecar(ctx context.Context, s release.Store, acct, app string, |
| if err != nil { |
| return "", err |
| } |
| + // The same for a store whose teardown began while this sweep was |
| + // running: what it listed after that may be another account's objects |
| + // (a retained teardown leaves the bucket to it), and coverage this |
| + // app's member can read is not written from them. |
| + if rec != nil && rec.Teardown != nil { |
| + return "", errSweepTornDown |
| + } |
| if rec == nil || rec.Scanning == nil || rec.Scanning.Enabled == nil { |
| return "", errStandDown |
| } |
diff --git a/internal/objstore/sweep_test.go b/internal/objstore/sweep_test.go| index 627f6c0cb..5355de2ad 100644 |
| --- a/internal/objstore/sweep_test.go |
| +++ b/internal/objstore/sweep_test.go |
| @@ -566,3 +566,52 @@ func TestSweepRefusesWhenNoAttestationIsOnRecord(t *testing.T) { |
| t.Fatal("a store with no attestation on record was swept anyway") |
| } |
| } |
| + |
| +// A sweep passes its checks and is about to list. The store's teardown then |
| +// begins and leaves the bucket to another account's store, whose objects |
| +// arrive. The sweep lists them. It does not publish them: the write of the |
| +// sidecar reads the record again, finds the teardown, and stands down, and |
| +// no later page is listed. |
| +func TestSweepInFlightDoesNotPublishAfterATeardownBegins(t *testing.T) { |
| + s, p, sc, bucket := sweepFixture(t, 2) |
| + ctx := context.Background() |
| + l := &memLister{p: p, pageSize: 2} |
| + l.before = func() { |
| + if l.calls != 1 { |
| + return |
| + } |
| + rec, etag, err := Get(ctx, s, "bab", "titogo") |
| + if err != nil || rec == nil { |
| + t.Errorf("fixture: %+v %v", rec, err) |
| + return |
| + } |
| + rec.Teardown = &TeardownBlock{StartedAt: time.Now().UTC()} |
| + if _, err := Put(ctx, s, "bab", "titogo", rec, etag); err != nil { |
| + t.Error(err) |
| + } |
| + for i := 2; i < 6; i++ { |
| + p.Buckets[bucket][objectName(i)] = []byte("another account's object") |
| + } |
| + } |
| + side, err := RunSweep(ctx, s, l, sc, "bab", "titogo", time.Minute) |
| + if err != nil { |
| + t.Fatalf("RunSweep: %v", err) |
| + } |
| + if l.calls != 1 { |
| + t.Fatalf("the sweep listed %d pages; want it to stop after the one in flight", l.calls) |
| + } |
| + stored, _, err := GetSidecar(ctx, s, "bab", "titogo") |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + for name, sd := range map[string]*ScanSidecar{"returned": side, "stored": stored} { |
| + if sd == nil { |
| + continue |
| + } |
| + for which, sw := range map[string]*Sweep{"current": sd.Current, "last complete": sd.LastComplete} { |
| + if sw != nil && (sw.Objects != 0 || sw.After != "") { |
| + t.Fatalf("%s sidecar publishes what was listed after the teardown began (%s): %+v", name, which, sw) |
| + } |
| + } |
| + } |
| +} |
diff --git a/internal/objstore/walk.go b/internal/objstore/walk.go| index 29233b6bf..0624448e2 100644 |
| --- a/internal/objstore/walk.go |
| +++ b/internal/objstore/walk.go |
| @@ -252,6 +252,13 @@ func grant(ctx context.Context, s release.Store, p Provisioner, d Deployment, |
| if noTakeBack && rec.Teardown != nil { |
| return nil, fmt.Errorf("%w: %s/%s", ErrStoreBeingTornDown, acct, app) |
| } |
| + // Nor does a convergence make a first store: a record with nothing |
| + // provisioned (the store was destroyed and declared again between the |
| + // caller's look and this walk) is a grant's to act on, with the |
| + // approval checks its doors make. |
| + if noTakeBack && rec.Provisioned == nil { |
| + return nil, ErrNotProvisioned |
| + } |
| rebuilt = rec.Teardown != nil |
| startKeyID = "" |
| if rec.Provisioned != nil { |
| @@ -1346,7 +1353,7 @@ func walkEndedClean(err error) bool { |
| return err == nil || errors.Is(err, ErrBucketRetained) || errors.Is(err, ErrBucketHeld) || |
| errors.Is(err, ErrBucketBeingDestroyed) || errors.Is(err, ErrNotDeclared) || errors.Is(err, ErrNoPendingFinish) || |
| errors.Is(err, ErrUnfinishedTeardown) || errors.Is(err, ErrTwoBuckets) || errors.Is(err, ErrGrantUnfinished) || |
| - errors.Is(err, ErrStoreBeingTornDown) |
| + errors.Is(err, ErrStoreBeingTornDown) || errors.Is(err, ErrNotProvisioned) |
| } |
| // grantElapsed is how long a grant has been running. A seam, so a test can |
| @@ -1561,8 +1568,7 @@ func writeEnv(ctx context.Context, s release.Store, sealPub *ecdh.PublicKey, |
| // walk can follow. It is made by Get, the one reader every door, pass and |
| // walk uses, and by Put, so nothing acts on such a record and nothing |
| // writes one. A refused record is left exactly as found; an operator |
| -// repairs it by hand. (Inspect reads one anyway, for the holder inventory |
| -// and for display.) |
| +// repairs it by hand. (Inspect reads one anyway, for the holder inventory.) |
| // |
| // Two shapes: |
| // |