amadan
Review nine answered: holds, the latch and settlements are publication events; the snapshot reads again across a replay; a latch keeps its computed bound or is unresolved; replay closes on the settlement's test; the discovery instant floors closure; design revision 15
The ledger (internal/usage): writeHolds is the ledger's and dirties every day it writes with a fresh sequence, an admission dirties its day and a settlement the days it barred, and publish enumerates every latched discovery's days, so a day published complete is published again when a bound reaches it or a barrier lifts (a Publish-level test through a fake store, including a hold written during a publication's put). discoveryState.generation makes a snapshot read again when a replay commits across it; Replay, a replaying Discover and a closure that carries a replay take drainMu before l.mu. A failed discovery latches the bound and counted set it computed, or latches unresolved and bars every day; Replay recomputes the bound in its transaction and closes on maybeClose's test (every unsettled row and the pending count). uncertainties.discovered_at is stored and floors every closure. SetPendingCounter takes a lock-free *atomic.Int64. Repeats are keyed on the participant and outcome, not the instant. Every Part C write refuses a closed ledger with its own error. A zero bound term is absent, both absent refused without latching; one hold write is bounded. The inventory (internal/ops): the walk records the identity of every directory on the chain that listed each database and the ownership pass refuses a file whose kept chain is not that one (the same inode moved into a replacement directory is refused); the roots' recheck visits every root in order and a change wins over a failed look; ParseLitestreamDBs reads hand-written block YAML and errors on a form it does not take, which the shrink guard, restore-verify and the report propagate; inventoryCommand is the command without its process; registry.OpenReadOnly escapes the path; the report names its environment. The model's GateAnswers state check is killed by a recovery sequence. Dispositions "Review nine"; the gate on c824ed81 failed on an activator test this branch does not touch (amadan #206).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
22 files changed,
+1476
−225
DISPOSITIONS-replicator-pinned-root.md+24 −0docs/site/cli.md+7 −3docs/superpowers/plans/2026-10-03-replicator-authority-plan.md+3 −3docs/superpowers/specs/2026-10-03-replicator-pinned-roots-design.md+3 −3docs/superpowers/specs/2026-10-03-replicator-pinned-roots-spikes.md+1 −1internal/ops/admissionmodel/model_test.go+28 −0internal/ops/authoritybridge.go+64 −10internal/ops/litestream.go+126 −9internal/ops/litestream_metered_test.go+22 −6internal/ops/litestream_owners.go+17 −0internal/ops/litestream_owners_unix_test.go+119 −1internal/ops/ops.go+32 −14internal/ops/ops_test.go+57 −8internal/registry/readonly_test.go+28 −0internal/registry/registry.go+18 −6internal/usage/durabilityadmissions.go+48 −30internal/usage/durabilityadmissions_test.go+90 −14internal/usage/durabilityholds_test.go+414 −0internal/usage/durabilityledger.go+15 −0internal/usage/durabilitypublish.go+15 −0internal/usage/durabilityuncertainty.go+336 −111internal/usage/durabilityuncertainty_test.go+9 −6
diff --git a/DISPOSITIONS-replicator-pinned-root.md b/DISPOSITIONS-replicator-pinned-root.md| index a600ccc19..ffb8a26e6 100644 |
| --- a/DISPOSITIONS-replicator-pinned-root.md |
| +++ b/DISPOSITIONS-replicator-pinned-root.md |
| @@ -154,3 +154,27 @@ Opus: accept with fixes, no P1, one P2. Astra: accept with fixes, no P1, six P2. |
| | Astra 8: the recovery paragraph said all three timestamps; the ignored `Chmod`; the CLI doc | design revision 14 (missing timestamps, a completion before the crash keeps its instant); the `Chmod` checked; `ops inventory` documented. | |
| | Astra's notes: `openDirFDsUnder` stat-based; `cleanBefore` captured after the EIO look | the counter classifies the descriptor by `fstat` and keeps a deleted directory; the instant is captured before the look that cannot be taken, and that look is asserted not to count as clean. | |
| | Opus's three model mutants | one killed (above); two equivalent under the model, each with its reason in the spike note. | |
| + |
| +## Review nine (of `c824ed81`: the eighth review answered, Task 4's second and third slices) |
| + |
| +Opus: accept with fixes, one P1, two P2. Astra: reject, three P1, five P2. Reports: `logs/opus-replroot-review9.md`, `logs/astra-replroot-review9.md`. The gate on `c824ed81` failed on one test in `internal/activator` this branch does not touch (`TestMaintenanceFenceSerializesSidecarSweepReplacement`, a 15 ms deadline under the full run's load; it passes five of five alone under the gate's environment); the gate is run again on the answering tip, and the flake is filed for the activator's owners as amadan discussion #206 on carlos/platform. The owner's instruction (12:22): the publication findings and the other P2s are answered before slice 4 lands; slice 4 (built in a tree copy) lands as its own commit after them. |
| + |
| +| Finding | Disposition | |
| +|---|---| |
| +| Opus P1-1, Astra 2: holds, the latch, admissions and settlements never reached a published day | `writeHolds` is the ledger's, dirties each day it writes with a fresh sequence and persists it in the same transaction; an admission dirties its day and a settlement the days of `[admitted_at, settled_at]`; `publish` enumerates every latched discovery's days from its bound (or the ledger's earliest day, for an unresolved one) through today. A Publish-level test: a day published complete is republished incomplete under a latched discovery, then under the replayed hold; a closed day under the barrier republishes complete at settlement; a hold written during a publication's put (the store's hook) leaves the day dirty, since its sequence is above the watermark, and the next publication carries it. | |
| +| Astra 1: a snapshot could read neither the latch nor the replayed rows | `discoveryState.generation` changes with every latch and clear; `snapshot` reads it before and after and reads again on a change (the drain-locked fallback refuses instead, as it does for the failure latch); `Replay`, a `Discover` that replays and a closure that carries a replay take `drainMu` before `l.mu`; a test replays through a seam between the snapshot's database reads and its overlay read and asserts the hold is seen. | |
| +| Astra 3, Opus P2-1: a latch whose re-read failed kept a later bound and an empty counted set | `discover` returns the bound and counted set it computed and the latch keeps them; a failure before they could be read latches unresolved (zero bound), which bars every day and is enumerated from the ledger's earliest day; `Replay` recomputes the bound in its own transaction, takes the earlier of the two and the union of the counted sets; tests for both (the computed bound kept against the fallback; the unresolved latch barring a day before any bound and replayed to the right bound). The "never later" comment is gone. | |
| +| Astra 5, Opus P2-1: the replay could close over an unsettled row it had not counted | the replay closes on `maybeClose`'s test (every unsettled row of the fleet and the pending count); a registration pending at the failed discovery that commits before the replay keeps it open (test). | |
| +| Opus P2-2: the pending counter under the ledger's mutex | `SetPendingCounter(fleetID, *atomic.Int64)`, lock-free by contract, with the gate slice's contract stated in the plan: the ledger is never called with the discovery mutex held. | |
| +| Astra 4: the handoff compared the file's identity only; the same inode moved into a replacement directory passed | the walk records the identity of every directory from the tree's root to the one that listed each database (`fleetPins.listedChain`) and the ownership pass refuses a file whose kept chain is not, directory by directory, that one; the same-inode move is a fixture and its removal mutant is killed. The retained descriptors themselves are still not handed across; what is compared is the identity of the directories a retained descriptor would have been open on, which is what I1(b) asks the inspection to be: the directory that listed. | |
| +| Astra 6, Opus P3-1: a failing root hid another root's change | `fleetPins.recheck` visits every root in a fixed order and a change wins over a look that could not be taken; the fixture fails the first root examined and changes a later root with no owned file beneath it (killed). | |
| +| Astra 7, Opus P3-5: closure ignored the discovery instant | `uncertainties.discovered_at` is stored; closure is at the latest of the discovery, the settlement and every settlement of the fleet at or after the bound, plus the margin (a settlement reported out of order never closes before an interval written); a cancellation reported earlier than the discovery closes at the discovery plus the margin and materialises the margin's day across midnight (test). | |
| +| Astra 8: the live configuration was read with the generator's own format | `ParseLitestreamDBs` reads block-style YAML at any indentation, quoted or bare, comments, the path on the line after the dash, and returns an error for a form it does not take (flow style, an item with no path, a path key outside a list); `CoverageShortfall` and `RestoreVerify` return that error and `carlos ops litestream` refuses to write over a configuration it cannot read; the report refuses to compare one; a hand-written fixture. | |
| +| Astra 9, Opus P3-4: idempotency keyed on the instant | a repeat is the same participant and outcome whatever instant it carries (the drain retries with a fresh now until it commits): the transport's completion again, the same recorder outcome again, the registration again (settled or not) are nil and the first instant stands; a different recorder outcome is refused. | |
| +| Opus P3-2, P3-3, P3-6: the `closeNow` pending term; zero bound terms; closed checks and survivors | a discovery with nothing counted and one pending registration stays open (test); a zero term is absent and both absent is `ErrDiscoveryBound`, refused without latching; one hold write is bounded (an interval past 4000 days is an error, which latches conservatively); every Part C write refuses a closed ledger with `errLedgerClosed`, which the tests now distinguish from the database's refusal; a successful `Discover` under a latch replays it in its transaction and one under an open uncertainty is answered with it; a latched bound bars no day before it (test); the replay-failure seam leaves the latch (test). | |
| +| Opus P3-7: `OpenReadOnly`'s URI and the `-shm` | the path is URI-escaped (a `#` and a space tested; a `?` or a `%` the registry cannot take in any mode, since its own methods reopen by the path, which is on record); the comment says what a read-only open of a WAL database may still create. | |
| +| Opus P3-8: the report's required set is this process's environment | said in the report's required header, with the agent's unit environment named as the authority; tested. | |
| +| Astra 10: tests that still permitted removal of their subject | the participants' maximum against the last call with instants later than now; the closed checks by sentinel; an ambiguous commit that landed (a seam after the commit) read back and repeated; a settlement's own interval write with another admission pending; the replay's zero-unsettled closure and a failed discovery with no durable admission; the root-only change beside a failed root; the same-inode move; a hand-written configuration; the command against a missing registry; the enumeration's iteration error through a seam; the strict walk's second replacement between the second look and the acquisition (seam `treeBeforeReacquire`; the first acquisition with nil facts is killed); the report's "not required" line asserted. Multi-claim release accounting and constructor refusals stay inspection-only, as disclosed. | |
| +| Astra: the design's settlement formula and the plan's Task 3 | the formula names the clamp to now; Task 3 names `RequiredDatabases(registryPath)` and ownership. | |
| +| Astra: the `GateAnswers` equivalence claim | narrowed as asked: recovery leaves `admitted` without an answer; the named sequence kills the mutant; the spike note says so. | |
| +| Opus: mutants to add | the replay closing on an empty counted set over unsettled rows (killed by the pending-to-row test); closure without the discovery floor (killed by the edges test); a hold never published (killed by the Publish-level test). | |
diff --git a/docs/site/cli.md b/docs/site/cli.md| index 5f31f8112..7dbf87da3 100644 |
| --- a/docs/site/cli.md |
| +++ b/docs/site/cli.md |
| @@ -1442,9 +1442,13 @@ the deployment bucket so you can read it off-box. `inventory` is read-only: it |
| opens the registry without creating or migrating it and prints the platform |
| replicator's inventory (owned, skipped, unowned, problems, and the required |
| databases with whether each is reachable and owned) beside the external |
| -replicator's live configuration (`-config`, default `/etc/litestream.yml`) |
| -and the list a regeneration would write, every path quoted; the external |
| -lists and the inventory are separate looks, one after the other. |
| +replicator's live configuration (`-config`, default `/etc/litestream.yml`; |
| +block-style YAML, items at any indentation, quoted or bare paths; a |
| +configuration the reader cannot take is an error, never an empty list) and |
| +the list a regeneration would write, every path quoted; the external lists |
| +and the inventory are separate looks, one after the other, and the required |
| +set is read from this process's environment (`CONSOLE_DB`), which the |
| +report says. |
| ```sh |
| carlos ops restore-verify |
diff --git a/docs/superpowers/plans/2026-10-03-replicator-authority-plan.md b/docs/superpowers/plans/2026-10-03-replicator-authority-plan.md| index c76eae12d..7d54f0ac5 100644 |
| --- a/docs/superpowers/plans/2026-10-03-replicator-authority-plan.md |
| +++ b/docs/superpowers/plans/2026-10-03-replicator-authority-plan.md |
| @@ -24,7 +24,7 @@ Date: 2026-10-03. For `docs/superpowers/specs/2026-10-03-replicator-pinned-roots |
| ## Task 3 (built): required coverage (Part D) |
| -`RequiredDatabases(registryPath, dataDir)`: the registry this process has open and the console database when `CONSOLE_DB` names it. The design also names the console database when the box's role configuration says console box; no such role configuration exists on this branch, so `CONSOLE_DB` is the only way a console database is required here, and that gap is recorded for the owner with the deployment support conditions. `fleetInventory` looks at each required database through its root (`lookAt`: the retained chain, then `Lookat` of the name; a link in its place is "it is a link") and records `Required` with why each is unreachable; an unreachable one is a problem (the inventory is incomplete) and `MeteredFleetStore` tells the options' `Hold` the instant (the caller binds it to the durability ledger's `Incomplete`, today's box-wide durable marker; owner scope is decision 10) before it refuses the build. `carlos ops inventory` is the read-only operator command: the external replicator's configured list beside the platform replicator's inventory (owned, skipped, unowned, problems, each side's differences) and the required databases with their reachability; it builds no fleet and writes nothing. Tests: every required database reachable (no problem, the report says reachable); a link in the console database's place (missing, a problem, the report says MISSING); missing altogether (a fleet build holds the day once and is refused); restored; without `CONSOLE_DB` only the registry is required. |
| +`RequiredDatabases(registryPath)`: the registry this process has open and the console database when `CONSOLE_DB` names it. Required coverage is ownership: after the ownership pass a required path not owned is missing with its refusal as the reason (review eight). The design also names the console database when the box's role configuration says console box; no such role configuration exists on this branch, so `CONSOLE_DB` is the only way a console database is required here, and that gap is recorded for the owner with the deployment support conditions. `fleetInventory` looks at each required database through its root (`lookAt`: the retained chain, then `Lookat` of the name; a link in its place is "it is a link") and records `Required` with why each is unreachable; an unreachable one is a problem (the inventory is incomplete) and `MeteredFleetStore` tells the options' `Hold` the instant (the caller binds it to the durability ledger's `Incomplete`, today's box-wide durable marker; owner scope is decision 10) before it refuses the build. `carlos ops inventory` is the read-only operator command: the external replicator's configured list beside the platform replicator's inventory (owned, skipped, unowned, problems, each side's differences) and the required databases with their reachability; it builds no fleet and writes nothing. Tests: every required database reachable (no problem, the report says reachable); a link in the console database's place (missing, a problem, the report says MISSING); missing altogether (a fleet build holds the day once and is refused); restored; without `CONSOLE_DB` only the registry is required. |
| ## Task 4: Part C's ledger changes (after the model) |
| @@ -34,9 +34,9 @@ The admission model (`internal/ops/admissionmodel`) is the executable contract; |
| 1. **The tables and the admission's row (built: `internal/usage/durabilityadmissions.go`).** `admissions(token TEXT PRIMARY KEY, fleet_id, account, app, admitted_at, transport_done_at, recorder_started_at, recorder_done_at, settled_at, how)`, `fleets(fleet_id PRIMARY KEY, box, witness_generation, owners, opened_at, state, predecessor, acknowledged_by)`, `uncertainties(id, fleet_id, from_ns, to_ns NULL, reason)`, `inherited(fleet_id, uncertainty_id)`, `holds(day, account, app, uncertainty_id, written_at)`, all `IF NOT EXISTS` beside today's tables. The `DurabilityLedger` gains `Admit(token, fleet, owner, app, at) error` (the token allocated by the caller before the write, `NewAdmissionToken`, so an ambiguous commit is read back by it; one transaction, `synchronous=FULL`; an exact repeat is the write that landed, a repeat with other facts is refused; the gate's budget, `ErrAdmissionNotDurable` past it and the goroutine staying to mark a late commit `unissued`, are slice 5's, the gate's, and are not built here), `TransportDone(token, at, lost bool)` (`lost` is not recorded: transport-lost is never a value of `how`), `RecorderStarted(token, at)`, `RecorderDone(token, at, how)` (`finished` and `abandoned` need the recorder registered), settlement by whichever lands last in its own transaction at the latest of the two completions and now, and `Unissued(token, at)` (an edge from `admitted` alone: refused after any participant activity); every transition refuses an instant before the admission's, is idempotent for an exact repeat, and refuses on a closed ledger. Mirrors: `TestUnchangedFleetHoldsNothing` and `TestThreeMidnightsBothOrders` (the row's settlement in both orders; the interval written at settlement is slice 3's), `TestCommitAfterDiscoveryIsUnissued`, `TestHistoricalTermCountsUnrecordedSettlements`; `TestBudgetExpiryWithoutDiscovery` is slice 5's. |
| 2. **The barrier (built: `DurabilityContent.Barred`, counted in the snapshot's read transaction; `CompletenessFor` refuses while it is positive; box-wide, so the ancestry query waits for owner scope).** The publisher's snapshot (`snapshotOnce`) treats a day incomplete for an owner while a durable unsettled admission of that owner, or of any fleet whose unacknowledged ancestry includes the admission's fleet, is admitted on or before that day (box-wide: any owner); it is a query over `admissions` and `fleets`, writes nothing, and lifts at settlement. Mirrors: the model's barrier invariant and `TestSuccessorBarredByPredecessorAdmission`, `TestLandedAmbiguousCommitBarsBeforeReadBack`. |
| -3. **Discovery, the uncertainty and the latch (built in the ledger: `Discover` with the fixed bound and the counted set, `uncertainties` and box-wide `holds` rows in one transaction, closed at once with nothing counted or pending; the settlement's hook writing `[admitted_at, settled_at]` and closing when no row and no pending registration remains, the fleet's pending count by `SetPendingCounter`; `EmptySettlement`; a failed transaction latched in memory with its bound and counted tokens, barring publication through the snapshot's `Barred` until `Replay`, one transaction; `DurabilityContent.Held` refusing completeness; the fleet's discovery mutex, registration and the latch's refusal of requests are the gate slice).** `Discover(fleet, at, proof, lastCleanLook)`: under the fleet's discovery mutex (new in `MeteredFleet`: registration, the recheck and the start of a discovery serialise on it, the pending set counted), the fixed bound, the counted set, the `uncertainties` row and the holds of `[from, now]` for the owners reached in one transaction, closed at once with nothing pending; a failed write latches the pending discovery in the existing failure latch (extended with the bound, the owners reached and the counted tokens; the publisher's overlay reaches successors through the ancestry query) and the fleet refuses requests until `Replay`, one transaction as the design has it. Mirrors: `TestZeroPendingClosesAtOnceAcrossMidnight`, `TestFailedDiscoveryLatchBarsUntilReplay`, `TestFailedDiscoveryReplayKeepsSettlements`, `TestReplayHoldsTheLowerBoundWhileOpen`, `TestReplayWritesSettledTokenWhileOpen`, `TestLatchOverlayReachesSuccessor`, the lower-bound tests (the in-flight request across a clean look, the proof term, the earlier-of threshold, the fixed cutoff in both orders, the admission instant, the exact bound, the empty terminals excluded). |
| +3. **Discovery, the uncertainty and the latch (built in the ledger, with review nine's corrections: every hold write and every admission and settlement dirty their days through `touchDay` with a fresh sequence and `publish` enumerates the latched discoveries' days (`latchedDates`), so publication revisits a day a bound reaches or a barrier leaves; `discoveryState.generation` makes a snapshot read again across a replay; a failed discovery latches the bound and counted set it computed or latches unresolved; `Replay` recomputes the bound in its transaction and closes on `maybeClose`'s test; `uncertainties.discovered_at` floors every closure; `SetPendingCounter` takes an `*atomic.Int64`, lock-free by contract. `Discover` with the fixed bound and the counted set, `uncertainties` and box-wide `holds` rows in one transaction, closed at once with nothing counted or pending; the settlement's hook writing `[admitted_at, settled_at]` and closing when no row and no pending registration remains, the fleet's pending count by `SetPendingCounter`; `EmptySettlement`; a failed transaction latched in memory with its bound and counted tokens, barring publication through the snapshot's `Barred` until `Replay`, one transaction; `DurabilityContent.Held` refusing completeness; the fleet's discovery mutex, registration and the latch's refusal of requests are the gate slice).** `Discover(fleet, at, proof, lastCleanLook)`: under the fleet's discovery mutex (new in `MeteredFleet`: registration, the recheck and the start of a discovery serialise on it, the pending set counted), the fixed bound, the counted set, the `uncertainties` row and the holds of `[from, now]` for the owners reached in one transaction, closed at once with nothing pending; a failed write latches the pending discovery in the existing failure latch (extended with the bound, the owners reached and the counted tokens; the publisher's overlay reaches successors through the ancestry query) and the fleet refuses requests until `Replay`, one transaction as the design has it. Mirrors: `TestZeroPendingClosesAtOnceAcrossMidnight`, `TestFailedDiscoveryLatchBarsUntilReplay`, `TestFailedDiscoveryReplayKeepsSettlements`, `TestReplayHoldsTheLowerBoundWhileOpen`, `TestReplayWritesSettledTokenWhileOpen`, `TestLatchOverlayReachesSuccessor`, the lower-bound tests (the in-flight request across a clean look, the proof term, the earlier-of threshold, the fixed cutoff in both orders, the admission instant, the exact bound, the empty terminals excluded). |
| 4. **The fleet journal, construction backfill and acknowledgement.** `OpenFleet(predecessor, owners, witness)` writes the `open` row and the backfill of every interval its unacknowledged ancestry holds; `stale`, `closing`, `closed` as the design's transactions (closure carries a pending replay and refuses with an unsettled admission); `Acknowledge(successor)` root-first, copying the effective set into `inherited` and writing the holds for the closed ones. The ledger's reach query replaces the model's `reach`. Mirrors: `TestAncestryReachAndRootFirstAcknowledgement`, `TestHealthyChainContributesNothing`, the per-step inheritance invariant as a property test over a recorded schedule. |
| -5. **The gate and the fleet.** `MeteredFleet.admit` becomes the proof (unchanged) plus the admission: registration in the pending set under the discovery mutex, the ledger's `Admit`, the recheck, the token handed to the SDK middleware (`lease.RequestGate` returns the token; the meter's `Start` carries it; the finalize step writes `TransportDone`). `lookNow` is the discovery when it finds a change; `closeNow` is `closing` then `closed`. The helper process does not exist yet (Part B): the transport and the recorder run in the agent, so the lifecycle guard is the fleet's own, `transport-lost` is written at `Close` for a transport that never finalised, and `unrecorded` where the meter's `Start` failed. Mirrors: `TestHelperDeathAndRecorderGuard` (as the agent's close), `TestSendNeedsDurableIssuedAdmission` (the middleware refuses without a token), `TestReadBackOnlyAfterTheHandleCloses`, `TestBetweenCommitAndAnswer`. |
| +5. **The gate and the fleet.** Contract (review nine, Opus P2-2): the ledger is never called with the fleet's discovery mutex held, and the pending counter the ledger reads is the lock-free `*atomic.Int64`, so a settlement under the ledger's mutex never waits on the mutex a gate recheck holds while it writes `unissued`. `MeteredFleet.admit` becomes the proof (unchanged) plus the admission: registration in the pending set under the discovery mutex, the ledger's `Admit`, the recheck, the token handed to the SDK middleware (`lease.RequestGate` returns the token; the meter's `Start` carries it; the finalize step writes `TransportDone`). `lookNow` is the discovery when it finds a change; `closeNow` is `closing` then `closed`. The helper process does not exist yet (Part B): the transport and the recorder run in the agent, so the lifecycle guard is the fleet's own, `transport-lost` is written at `Close` for a transport that never finalised, and `unrecorded` where the meter's `Start` failed. Mirrors: `TestHelperDeathAndRecorderGuard` (as the agent's close), `TestSendNeedsDurableIssuedAdmission` (the middleware refuses without a token), `TestReadBackOnlyAfterTheHandleCloses`, `TestBetweenCommitAndAnswer`. |
| 6. **The producer fence and recovery.** `admissions.lock` flocked by the agent before its first admission on an open file description every helper will later inherit (today: the agent alone); `OpenDurabilityLedger`'s `recover` extended: with the fence, in one transaction, every unsettled admission `recovered-unknown` with its interval, every fleet not `closed` closed with its interval, every unacknowledged fleet acknowledged root-first by the first fleet built. Mirrors: `TestFenceOutlivesAgent`, `TestFenceHeldTokenSentAfterCrash`, `TestLandedAmbiguousThenCrash`, the kill-at-every-transition tests (a second process reopening the ledger after a kill at each boundary). |
| 7. **Part C2's reader.** The `holds` rows in the published day (`DurabilityContent`), the reader treating a box-wide row as failing every owner and an owner row as failing that owner alone, and the older-reader marker (a box-wide incompleteness marker whenever any hold exists). Box-wide stays the default; decision 10's per-owner fleets are not built. |
diff --git a/docs/superpowers/specs/2026-10-03-replicator-pinned-roots-design.md b/docs/superpowers/specs/2026-10-03-replicator-pinned-roots-design.md| index 6c9cfd792..3437b7015 100644 |
| --- a/docs/superpowers/specs/2026-10-03-replicator-pinned-roots-design.md |
| +++ b/docs/superpowers/specs/2026-10-03-replicator-pinned-roots-design.md |
| @@ -1,6 +1,6 @@ |
| # The replicator's authority: retained descriptors, a durable admission, claims by identity |
| -Date: 2026-10-03. Status: design, revision 14 (the eighth code review: the recovery paragraph agrees with the transition table, an orphan's missing timestamps become now and a completion recorded before the crash keeps its instant; required coverage stated as ownership, with the deployment condition for the console database). Revision 13 was (the sixth code review: the predicate's compatibility stated as within the pinned roots, with the two safe wider shapes and the kept platform-directory refusal; the walk's pre-pin absent exception; the link-shapes line extended). Revision 12 was (Part A's root rule stated as the plan has it: the platform walk ends where the platform's authority ends, a tenant-owned component's parent, and the chain beneath is retained and rechecked rather than refused). Revision 11 was (the fourth code review: the historical term's filter named `finished` and `abandoned` and so excluded `unrecorded`, a settlement whose transport did send; it now excludes only the empty terminals, `unissued` and `cancelled`; the look's order is stated as what safety needs rather than a sequence; the closing look agrees). Revision 10 was revision 9's correction of the lower bound refined after the third code review: the historical term's threshold is the earlier of the registry proof and the last clean look's start, since a witness change is known only to be after the proof; an empty terminal, unissued or cancelled, never sent and is not a settlement for that term; and every look reads the witness and rechecks every retained chain before it inspects files, so a clean look's start bounds every kind of change. Revision 9 was one mechanics correction to the uncertainty's lower bound, found by the second code review of the admission model: an admission admitted before the last clean look, still in flight across a change after it and settled clean before the change was noticed, was outside the bound; the bound now also takes the admission instant of every admission settled at or after the last clean look's start; the invariant I2 is unchanged and the model's named test and schedules cover it). Revision 8 was the accepted text: revision 7 was accepted with fixes by Astra (no P1; one P2: a negative read-back was not conclusive while the admission writer could still commit, and the pending set had three definitions; one P3: the restore wording made a salt mismatch imply a snapshot), both applied here. Revision 6 was rejected by Astra on one P1: a discovery whose write failed had no rule for the admissions that settled before its retry, so the replayed interval could lose a settlement; and three P2s: a helper dying before its recorder started left an admission nobody could finish; the transition table lacked the abort edges and the pending set's cleanup; the lineage record cited a durability discipline the shadow state does not have; each answered here. Revision 5 was rejected by Astra on two P1s: the admission's insertion had no ordering point against discovery, so a discovery between the proof and the commit counted nothing and a later send fell outside the closed uncertainty; construction did not backfill already-closed intervals for a successor's new owners before acknowledgement; and P2s: the transition table and the helper-death rule disagreed and the recorder participant was synthesised while its worker could still run; proof-only callers of admit would have created admissions with no terminal; a terminal `.` lost the directory requirement; conflicting summaries; each answered here. Revision 4 was rejected by Astra on four P1s: the helper has several connections of its own and needs SQLite's within-process lock bookkeeping between them; the producer fence did not cover a helper that outlives the agent with an admission in hand; an intermediate fleet closed before acknowledging broke transitive protection; a fleet could be marked closed with a discovery uncommitted; and four P2s: a dead helper's admissions had no terminal in the running agent, the sleeping binding persisted a reusable inode, the lineage predicate named header fields that identify nothing, the walk's grammar refused a `.` target the predicate accepts; each answered here. Revision 3 was rejected by Astra on five P1s: two SQLite VFSs in one process cannot coordinate through POSIX locks, since the library's own per-inode bookkeeping is in-process; a recorder that abandons early could settle an admission before its transport ended; an admission-only barrier does not cover a discovery whose uncertainty write failed after every admission settled; links at acknowledgement contradicted acknowledging only after settlement, and an inherited uncertainty vanished at the next successor; the ledger's lock does not prove the old process gone; and four P2s: closure at zero pending and the margin's extra day, sleeping claims with no binding, absolute and `.` spellings in the walk, `(dev, ino)` as a lineage; each answered here. Revision 2 was rejected by Astra on three P1s: the VFS delegated SQLite's Unix file methods, which open the SHM and check moved files by pathname themselves; every settled admission wrote a permanent hold, so ordinary replication never read complete; a late owner-scoped extension did not reach successors that had already acknowledged; and six P2s: orphan admissions never terminated, `..` and chain edges unspecified, `dup` sharing the directory offset, the platform-link predicate not preserved and terminal links unclassified, the uid producer without a lifecycle, one billing owner not one replicator; each answered here. Revision 1 was rejected by Astra on five P1s: SQLite canonicalises a `/proc` pathname before it opens anything and this driver passes no `nofollow`; `os.Root` confines but re-resolves every name, so a swap between a listing and an inspection still moves the answer; a `Start` row is not an admission, since the meter sends without one; the attempt-derived window lost a request completed before the change was noticed and the days between an admission and a finish; `closed_at` was not a handover; plus the directory open before `O_DIRECTORY`, per-owner holds the ledger cannot store, a uid with no producer, I3 against Part D, and tests). Branch `design-replicator-pinned-root`, off `feat/carloku-metered-replicator-20261001` at `e1280811`. It answers the eleven findings the twentieth review deferred (`DISPOSITIONS-metered-replicator.md`, review twenty) and keeps decisions 10 to 13 and approval H (`DECISIONS-FOR-PAUL.md`, section D, on the launch branch) as parameters. Nothing is built, wired or applied. |
| +Date: 2026-10-03. Status: design, revision 15 (the ninth code review: a hold, a latched discovery, an admission and its settlement are publication events, dirtying their days above any snapshot's watermark, and publication enumerates a latched discovery's days as it does the failure latch's; a snapshot reads again when the latched set changes across its read; a failed discovery keeps the bound and counted set its transaction computed, or latches unresolved and bars every day until the replay computes them; the replay closes on the settlement's own test, every unsettled row and the pending count; the discovery instant is stored and floors the closure; the settlement formula names the clamp to now; the handoff from the tree walk to the ownership checks compares the identity of the file and of every directory on the chain that listed it). Revision 14 was (the eighth code review: the recovery paragraph agrees with the transition table, an orphan's missing timestamps become now and a completion recorded before the crash keeps its instant; required coverage stated as ownership, with the deployment condition for the console database). Revision 13 was (the sixth code review: the predicate's compatibility stated as within the pinned roots, with the two safe wider shapes and the kept platform-directory refusal; the walk's pre-pin absent exception; the link-shapes line extended). Revision 12 was (Part A's root rule stated as the plan has it: the platform walk ends where the platform's authority ends, a tenant-owned component's parent, and the chain beneath is retained and rechecked rather than refused). Revision 11 was (the fourth code review: the historical term's filter named `finished` and `abandoned` and so excluded `unrecorded`, a settlement whose transport did send; it now excludes only the empty terminals, `unissued` and `cancelled`; the look's order is stated as what safety needs rather than a sequence; the closing look agrees). Revision 10 was revision 9's correction of the lower bound refined after the third code review: the historical term's threshold is the earlier of the registry proof and the last clean look's start, since a witness change is known only to be after the proof; an empty terminal, unissued or cancelled, never sent and is not a settlement for that term; and every look reads the witness and rechecks every retained chain before it inspects files, so a clean look's start bounds every kind of change. Revision 9 was one mechanics correction to the uncertainty's lower bound, found by the second code review of the admission model: an admission admitted before the last clean look, still in flight across a change after it and settled clean before the change was noticed, was outside the bound; the bound now also takes the admission instant of every admission settled at or after the last clean look's start; the invariant I2 is unchanged and the model's named test and schedules cover it). Revision 8 was the accepted text: revision 7 was accepted with fixes by Astra (no P1; one P2: a negative read-back was not conclusive while the admission writer could still commit, and the pending set had three definitions; one P3: the restore wording made a salt mismatch imply a snapshot), both applied here. Revision 6 was rejected by Astra on one P1: a discovery whose write failed had no rule for the admissions that settled before its retry, so the replayed interval could lose a settlement; and three P2s: a helper dying before its recorder started left an admission nobody could finish; the transition table lacked the abort edges and the pending set's cleanup; the lineage record cited a durability discipline the shadow state does not have; each answered here. Revision 5 was rejected by Astra on two P1s: the admission's insertion had no ordering point against discovery, so a discovery between the proof and the commit counted nothing and a later send fell outside the closed uncertainty; construction did not backfill already-closed intervals for a successor's new owners before acknowledgement; and P2s: the transition table and the helper-death rule disagreed and the recorder participant was synthesised while its worker could still run; proof-only callers of admit would have created admissions with no terminal; a terminal `.` lost the directory requirement; conflicting summaries; each answered here. Revision 4 was rejected by Astra on four P1s: the helper has several connections of its own and needs SQLite's within-process lock bookkeeping between them; the producer fence did not cover a helper that outlives the agent with an admission in hand; an intermediate fleet closed before acknowledging broke transitive protection; a fleet could be marked closed with a discovery uncommitted; and four P2s: a dead helper's admissions had no terminal in the running agent, the sleeping binding persisted a reusable inode, the lineage predicate named header fields that identify nothing, the walk's grammar refused a `.` target the predicate accepts; each answered here. Revision 3 was rejected by Astra on five P1s: two SQLite VFSs in one process cannot coordinate through POSIX locks, since the library's own per-inode bookkeeping is in-process; a recorder that abandons early could settle an admission before its transport ended; an admission-only barrier does not cover a discovery whose uncertainty write failed after every admission settled; links at acknowledgement contradicted acknowledging only after settlement, and an inherited uncertainty vanished at the next successor; the ledger's lock does not prove the old process gone; and four P2s: closure at zero pending and the margin's extra day, sleeping claims with no binding, absolute and `.` spellings in the walk, `(dev, ino)` as a lineage; each answered here. Revision 2 was rejected by Astra on three P1s: the VFS delegated SQLite's Unix file methods, which open the SHM and check moved files by pathname themselves; every settled admission wrote a permanent hold, so ordinary replication never read complete; a late owner-scoped extension did not reach successors that had already acknowledged; and six P2s: orphan admissions never terminated, `..` and chain edges unspecified, `dup` sharing the directory offset, the platform-link predicate not preserved and terminal links unclassified, the uid producer without a lifecycle, one billing owner not one replicator; each answered here. Revision 1 was rejected by Astra on five P1s: SQLite canonicalises a `/proc` pathname before it opens anything and this driver passes no `nofollow`; `os.Root` confines but re-resolves every name, so a swap between a listing and an inspection still moves the answer; a `Start` row is not an admission, since the meter sends without one; the attempt-derived window lost a request completed before the change was noticed and the days between an admission and a finish; `closed_at` was not a handover; plus the directory open before `O_DIRECTORY`, per-owner holds the ledger cannot store, a uid with no producer, I3 against Part D, and tests). Branch `design-replicator-pinned-root`, off `feat/carloku-metered-replicator-20261001` at `e1280811`. It answers the eleven findings the twentieth review deferred (`DISPOSITIONS-metered-replicator.md`, review twenty) and keeps decisions 10 to 13 and approval H (`DECISIONS-FOR-PAUL.md`, section D, on the launch branch) as parameters. Nothing is built, wired or applied. |
| ## What stands today |
| @@ -42,9 +42,9 @@ The VFS is the plan's first spike and it must prove, before anything else of Par |
| ## Part C. The durable admission, two kinds of hold, and a handover on disk (Astra 3, 4) |
| -**Proof and admission are two things.** Fleet construction, `Open` and every watch tick ask the ownership *proof* (today's `admit`: the witness unchanged, the fleet not stale) and create nothing; only the request gate (`lease.RequestGate`, asked by the helper for each physical request) performs an *admission*, which is a proof followed by a row, so an idle healthy fleet has no admission and nothing bars its days. **The admission is a row, written before the send, with one ordering point against discovery.** Under a *discovery mutex* on the fleet (a new requirement: today `admit` and `lookAgain` do not share one, and the admission comments allow a discovery to run unlocked; this design serialises registration, the recheck and the start of a discovery on it) the gate checks the proof and, if the fleet is not stale and no discovery is in progress, registers the admission in the fleet's in-memory *pending set* with its `admitted_at`; it releases the mutex and commits the row in the durability ledger's own database (`admissions`: `token`, `fleet_id`, `owner account and app`, `admitted_at`, `transport_done_at`, `recorder_started_at`, `recorder_done_at`, `settled_at`, `how`), `synchronous=FULL`, within a bounded budget; after the commit it takes the mutex again and, if the fleet went stale or a discovery began since registration, marks the row `unissued` and refuses. Discovery, under the same mutex, sets the in-progress flag and counts the pending set, durable or not yet, as unsettled admissions when it computes its lower bound, and a registered admission that the discovery has counted and that then commits is inside the uncertainty (it was counted); one that commits after the discovery began is refused. So an admission is either counted by every discovery that follows its registration or refused, and Astra's 23:58:10 sequence (proof, discovery, commit, a send on the 4th) ends in a refusal at the recheck, not a send outside the uncertainty. The gate waits for the commit. A commit that does not return within the budget refuses the request (`ErrAdmissionNotDurable`; litestream sees a failed upload and does not advance) and the goroutine that awaited it stays to see the commit land and then marks the row `how = unissued, settled_at = now` in its own transaction, so a late commit never authorises anything and never blocks a day; if the process dies between, recovery classifies the row as below. The meter's `Start` row carries the token (`Start` may still fail or be late, as today; what is now mandatory is the admission, not the `Start`). An admission has two participants, the transport and the recorder, and is settled by whichever finishes last. The transport writes `transport_done_at` from the SDK middleware's finalize step, or `transport-lost` when its helper is proven gone (below). The recorder writes `recorder_done_at` with its outcome: `finished` (`Finish`), `abandoned` (`Abandon`), or `unrecorded` (the request path, when `Start` failed and made no attempt row, which today calls neither; it now writes this terminal); the recorder participant is never synthesised while its worker can still run: today's meter lets recording workers outlive the request path, and the admission stays unsettled (barring) until that worker finishes or abandons. The recorder runs in the agent, fed by the helper's messages over the fleet socket (`Start`, `Finish`, `Abandon` with the token), and the agent *registers* the recorder participant when it receives `Start` (`recorder_started_at`, under an agent-owned lifecycle guard per helper); a `Start` that the agent receives after it has begun the helper's termination handling is refused (the guard is closed first, then the socket drained, so registration and exit handling cannot interleave). The participant that finds the other already done writes `settled_at = max(transport_done_at, recorder_done_at)` and `how`, and performs, in that same transaction, the uncertainty extension or closure below. **The transition table is authoritative and is the plan's model:** states `registered` (in memory only), `admitted` (durable), `transport done`, `recorder done`, `settled`; edges `registered → admitted` (the commit), `admitted → unissued` (the recheck after the commit found the fleet stale or a discovery begun, or the gate's budget elapsed before the answer was returned: the row is rewritten as the terminal and the helper never receives the token), `registered → cancelled` (the admission writer has returned with a definite failure, or the writer's handle is closed and a fresh read then finds no row: the registration leaves the pending set), `admitted → transport done | recorder done`, `transport done → settled` (the recorder's completion), `recorder done → settled` (the transport's completion or `transport-lost`), and `admitted | transport done | recorder done → recovered-unknown` at process recovery only (all missing timestamps `now`); `transport-lost` is a transport completion, not a terminal, so an admission whose recorder is still running after its helper died stays `recorder pending`. **The pending set has one contract.** It holds every registration not yet resolved, and an entry leaves it in exactly two ways, both under the discovery mutex: (a) the admission row is durable, at which point the row carries the responsibility (the removal follows the commit, so a discovery sees the entry or the row and never neither); (b) the cancellation is *conclusive*, at which point the empty-settlement result is recorded for every discovery that counted it (the pending-discovery latch's counted set, or the open uncertainty's closure test, which ranges over rows and pending registrations alike), so an uncertainty that waited only on it closes then. The same empty-settlement rule applies to a counted admission that ends `unissued`. **A cancellation is conclusive only when the admission writer can no longer commit:** the writer is one goroutine owning the transaction, the gate's budget expiring refuses the request but does not resolve the registration, and the registration waits for the writer to return. A writer that returns having committed takes `admitted` (then `unissued`, since the request was refused); one that returns with a definite failure takes `cancelled`; one that cannot tell (the connection lost around the commit) is resolved by a read-back by `token` that begins only after the writer's handle is closed, on a read transaction opened after that close, so nothing it fails to find can be committed afterwards: a row found takes `admitted → unissued`; none found takes `cancelled`; the read is retried until it answers. There is no edge out of `cancelled`. Astra's sequence (the writer still running at the budget's expiry and the discovery's count, a read at 23:58:00.040, the commit at 23:58:00.050) cannot declare a cancellation, because the read waits for the writer; the writer's commit resolves it as `unissued`, and the discovery that counted it is told once. Every terminal write is keyed by `token` and idempotent, retried until it commits. Admissions are the ledger's pending state moved to before the send and given an owner. |
| +**Proof and admission are two things.** Fleet construction, `Open` and every watch tick ask the ownership *proof* (today's `admit`: the witness unchanged, the fleet not stale) and create nothing; only the request gate (`lease.RequestGate`, asked by the helper for each physical request) performs an *admission*, which is a proof followed by a row, so an idle healthy fleet has no admission and nothing bars its days. **The admission is a row, written before the send, with one ordering point against discovery.** Under a *discovery mutex* on the fleet (a new requirement: today `admit` and `lookAgain` do not share one, and the admission comments allow a discovery to run unlocked; this design serialises registration, the recheck and the start of a discovery on it) the gate checks the proof and, if the fleet is not stale and no discovery is in progress, registers the admission in the fleet's in-memory *pending set* with its `admitted_at`; it releases the mutex and commits the row in the durability ledger's own database (`admissions`: `token`, `fleet_id`, `owner account and app`, `admitted_at`, `transport_done_at`, `recorder_started_at`, `recorder_done_at`, `settled_at`, `how`), `synchronous=FULL`, within a bounded budget; after the commit it takes the mutex again and, if the fleet went stale or a discovery began since registration, marks the row `unissued` and refuses. Discovery, under the same mutex, sets the in-progress flag and counts the pending set, durable or not yet, as unsettled admissions when it computes its lower bound, and a registered admission that the discovery has counted and that then commits is inside the uncertainty (it was counted); one that commits after the discovery began is refused. So an admission is either counted by every discovery that follows its registration or refused, and Astra's 23:58:10 sequence (proof, discovery, commit, a send on the 4th) ends in a refusal at the recheck, not a send outside the uncertainty. The gate waits for the commit. A commit that does not return within the budget refuses the request (`ErrAdmissionNotDurable`; litestream sees a failed upload and does not advance) and the goroutine that awaited it stays to see the commit land and then marks the row `how = unissued, settled_at = now` in its own transaction, so a late commit never authorises anything and never blocks a day; if the process dies between, recovery classifies the row as below. The meter's `Start` row carries the token (`Start` may still fail or be late, as today; what is now mandatory is the admission, not the `Start`). An admission has two participants, the transport and the recorder, and is settled by whichever finishes last. The transport writes `transport_done_at` from the SDK middleware's finalize step, or `transport-lost` when its helper is proven gone (below). The recorder writes `recorder_done_at` with its outcome: `finished` (`Finish`), `abandoned` (`Abandon`), or `unrecorded` (the request path, when `Start` failed and made no attempt row, which today calls neither; it now writes this terminal); the recorder participant is never synthesised while its worker can still run: today's meter lets recording workers outlive the request path, and the admission stays unsettled (barring) until that worker finishes or abandons. The recorder runs in the agent, fed by the helper's messages over the fleet socket (`Start`, `Finish`, `Abandon` with the token), and the agent *registers* the recorder participant when it receives `Start` (`recorder_started_at`, under an agent-owned lifecycle guard per helper); a `Start` that the agent receives after it has begun the helper's termination handling is refused (the guard is closed first, then the socket drained, so registration and exit handling cannot interleave). The participant that finds the other already done writes `settled_at = max(transport_done_at, recorder_done_at, now)` (the two completions as reported, and no earlier than the instant the ledger learns of the second: a completion reported late never settles a row in the past) and `how`, and performs, in that same transaction, the uncertainty extension or closure below. **The transition table is authoritative and is the plan's model:** states `registered` (in memory only), `admitted` (durable), `transport done`, `recorder done`, `settled`; edges `registered → admitted` (the commit), `admitted → unissued` (the recheck after the commit found the fleet stale or a discovery begun, or the gate's budget elapsed before the answer was returned: the row is rewritten as the terminal and the helper never receives the token), `registered → cancelled` (the admission writer has returned with a definite failure, or the writer's handle is closed and a fresh read then finds no row: the registration leaves the pending set), `admitted → transport done | recorder done`, `transport done → settled` (the recorder's completion), `recorder done → settled` (the transport's completion or `transport-lost`), and `admitted | transport done | recorder done → recovered-unknown` at process recovery only (all missing timestamps `now`); `transport-lost` is a transport completion, not a terminal, so an admission whose recorder is still running after its helper died stays `recorder pending`. **The pending set has one contract.** It holds every registration not yet resolved, and an entry leaves it in exactly two ways, both under the discovery mutex: (a) the admission row is durable, at which point the row carries the responsibility (the removal follows the commit, so a discovery sees the entry or the row and never neither); (b) the cancellation is *conclusive*, at which point the empty-settlement result is recorded for every discovery that counted it (the pending-discovery latch's counted set, or the open uncertainty's closure test, which ranges over rows and pending registrations alike), so an uncertainty that waited only on it closes then. The same empty-settlement rule applies to a counted admission that ends `unissued`. **A cancellation is conclusive only when the admission writer can no longer commit:** the writer is one goroutine owning the transaction, the gate's budget expiring refuses the request but does not resolve the registration, and the registration waits for the writer to return. A writer that returns having committed takes `admitted` (then `unissued`, since the request was refused); one that returns with a definite failure takes `cancelled`; one that cannot tell (the connection lost around the commit) is resolved by a read-back by `token` that begins only after the writer's handle is closed, on a read transaction opened after that close, so nothing it fails to find can be committed afterwards: a row found takes `admitted → unissued`; none found takes `cancelled`; the read is retried until it answers. There is no edge out of `cancelled`. Astra's sequence (the writer still running at the budget's expiry and the discovery's count, a read at 23:58:00.040, the commit at 23:58:00.050) cannot declare a cancellation, because the read waits for the writer; the writer's commit resolves it as `unissued`, and the discovery that counted it is told once. Every terminal write is keyed by `token` and idempotent, retried until it commits. Admissions are the ledger's pending state moved to before the send and given an owner. |
| -**Two kinds of hold.** A *barrier* is the ledger's completeness rule: a day is not published complete for an admission's owner while an admission of that owner admitted on or before that day is unsettled (box-wide while Part C2's owner scope is not in use); it is a query over `admissions`, writes nothing, and lifts at settlement. An *uncertainty* is a row (`uncertainties`: `id`, `fleet_id`, `from`, `to` null while open, `reason`) written when a fleet's authority is found changed (by the witness, by `Recheck`, by a changed file), in the transaction that also writes the holds for the days of `[from, now]` for every owner the uncertainty reaches: `bound = min(the registry proof, the last clean look's start)`; `from = min(bound, the earliest admitted_at among the fleet's admissions that are unsettled, or settled with settled_at >= bound and how not in {unissued, cancelled}) - margin`, the bound fixed before the rows are read and never the running minimum (revision 9: the first two terms alone missed an admission admitted before the look, in flight across the change, and settled clean before the discovery; such a settlement wrote nothing permanent, so its admission instant must bound the uncertainty; revision 10: the threshold is the bound itself, not the look alone; revision 11: every settlement whose transport ran counts, `unrecorded` included, and only the empty terminals, which never sent, are excluded; `transport-lost` is a transport completion, never a value of `how`; a `recovered-unknown` row has its own interval and need not be counted here). **What a look must do for its start to be the bound:** the look's start instant is stamped before its first observation, and every observation it makes (the witness, `Recheck` of every retained chain, the route's recorded invocation against systemd's, the files) sees any change that persists, so a look that finds nothing is clean against every kind of change from its start; the order of the observations within a look does not matter for that; the earlier-of rule (the proof and the look) is kept as a backstop for a look built without the witness observation only, since a registry proof can stand in for a missed witness reading and for nothing else (the chains, the invocation and the files are mandatory observations). If discovery finds no admission of the fleet unsettled, the uncertainty is opened and closed in that transaction (`to = discovery + margin`). While an uncertainty is open, settlements of that fleet's admissions extend it: each settlement, in its own transaction, writes the holds for every day of `[its admitted_at, its settled_at]` for every owner the uncertainty reaches, and the last settlement closes it with `to = max(discovery, that settlement) + margin` and materialises every day of the closed interval, the margin's extra day across midnight included; recovery uses the same helper and advances the publication watermark past the days it materialised. An admission that settles while no uncertainty of its fleet is open writes nothing permanent: the ordinary day of an unchanged fleet becomes complete when its barrier lifts. The multiple-midnights sequence (admitted on the 2nd, the change at 23:58:10, sent on the 3rd, finished on the 4th) holds the 2nd, 3rd and 4th through the settlement's interval, whichever participant finished last; the request completed before the change was noticed is inside the lower bound as today. **If the discovery transaction fails** the barrier is not enough (every admission may already be settled): discovery first latches a *pending discovery* in the ledger's in-memory failure overlay (`latchFailure`, extended to carry the lower bound and the owners reached), which the publisher's snapshot already overlays and detects across a drain (today's `snapshotOnce` and `drainFailures` behaviour, extended rather than replaced: a failed drain must not stop the overlay from making the snapshot incomplete), so every day from the bound reads incomplete for those owners until the uncertainty commits; the fleet refuses every request meanwhile and retries the write. The latch carries the lower bound, the owners reached and the tokens of the admissions the discovery counted; settlements of those admissions meanwhile find no uncertainty open and write no hold, but their rows record `settled_at` as always. **The replay is one transaction:** it opens the uncertainty with the original `from`; for each counted token whose row is settled it writes the holds for `[its admitted_at, its settled_at]` for every owner reached (and for a token that was cancelled or unissued, nothing); if any counted token is still unsettled the uncertainty stays open and the usual settlement rule closes it later; otherwise it closes with `to = replay instant + margin` (which is no earlier than the discovery and every counted settlement) and materialises every day of the closed interval; the overlay clears only after that commit, so no snapshot sees neither the latch nor the row. Astra's sequence (T counted at 23:58:10 on the 2nd, the write failing, T settling at 00:02 on the 4th, the replay at 00:03) writes the 2nd, 3rd and 4th through T's interval and closes at 00:04 on the 4th, and a successor constructed after the replay backfills that closed interval for its owners; the fleet cannot be marked `closed` while the discovery is uncommitted (above); a process that dies with the latch held leaves a fleet not `closed`, which recovery covers from the fleet's `opened_at`. |
| +**Two kinds of hold.** A *barrier* is the ledger's completeness rule: a day is not published complete for an admission's owner while an admission of that owner admitted on or before that day is unsettled (box-wide while Part C2's owner scope is not in use); it is a query over `admissions`, writes nothing, and lifts at settlement. An *uncertainty* is a row (`uncertainties`: `id`, `fleet_id`, `from`, `to` null while open, `reason`) written when a fleet's authority is found changed (by the witness, by `Recheck`, by a changed file), in the transaction that also writes the holds for the days of `[from, now]` for every owner the uncertainty reaches: `bound = min(the registry proof, the last clean look's start)`; `from = min(bound, the earliest admitted_at among the fleet's admissions that are unsettled, or settled with settled_at >= bound and how not in {unissued, cancelled}) - margin`, the bound fixed before the rows are read and never the running minimum (revision 9: the first two terms alone missed an admission admitted before the look, in flight across the change, and settled clean before the discovery; such a settlement wrote nothing permanent, so its admission instant must bound the uncertainty; revision 10: the threshold is the bound itself, not the look alone; revision 11: every settlement whose transport ran counts, `unrecorded` included, and only the empty terminals, which never sent, are excluded; `transport-lost` is a transport completion, never a value of `how`; a `recovered-unknown` row has its own interval and need not be counted here). **What a look must do for its start to be the bound:** the look's start instant is stamped before its first observation, and every observation it makes (the witness, `Recheck` of every retained chain, the route's recorded invocation against systemd's, the files) sees any change that persists, so a look that finds nothing is clean against every kind of change from its start; the order of the observations within a look does not matter for that; the earlier-of rule (the proof and the look) is kept as a backstop for a look built without the witness observation only, since a registry proof can stand in for a missed witness reading and for nothing else (the chains, the invocation and the files are mandatory observations). If discovery finds no admission of the fleet unsettled, the uncertainty is opened and closed in that transaction (`to = discovery + margin`). While an uncertainty is open, settlements of that fleet's admissions extend it: each settlement, in its own transaction, writes the holds for every day of `[its admitted_at, its settled_at]` for every owner the uncertainty reaches, and the last settlement closes it with `to = max(discovery, every settlement of the fleet at or after the bound) + margin` (the discovery instant is stored on the row; a settlement reported out of order never closes it before an interval already written) and materialises every day of the closed interval, the margin's extra day across midnight included; recovery uses the same helper and advances the publication watermark past the days it materialised. An admission that settles while no uncertainty of its fleet is open writes nothing permanent: the ordinary day of an unchanged fleet becomes complete when its barrier lifts. The multiple-midnights sequence (admitted on the 2nd, the change at 23:58:10, sent on the 3rd, finished on the 4th) holds the 2nd, 3rd and 4th through the settlement's interval, whichever participant finished last; the request completed before the change was noticed is inside the lower bound as today. **If the discovery transaction fails** the barrier is not enough (every admission may already be settled): discovery first latches a *pending discovery* in the ledger's in-memory failure overlay (`latchFailure`, extended to carry the lower bound and the owners reached), which the publisher's snapshot already overlays and detects across a drain (today's `snapshotOnce` and `drainFailures` behaviour, extended rather than replaced: a failed drain must not stop the overlay from making the snapshot incomplete), so every day from the bound reads incomplete for those owners until the uncertainty commits; the fleet refuses every request meanwhile and retries the write. The latch carries the lower bound, the owners reached and the tokens of the admissions the discovery counted; settlements of those admissions meanwhile find no uncertainty open and write no hold, but their rows record `settled_at` as always. **The replay is one transaction:** it opens the uncertainty with the original `from`; for each counted token whose row is settled it writes the holds for `[its admitted_at, its settled_at]` for every owner reached (and for a token that was cancelled or unissued, nothing); if any counted token is still unsettled the uncertainty stays open and the usual settlement rule closes it later; otherwise it closes with `to = max(discovery, replay instant) + margin` and materialises every day of the closed interval; the replay's closure test is the settlement's (no unsettled row of the fleet, nothing pending), not the set counted at the discovery, since a registration pending then may be a row by now; a latch keeps the bound and counted set its failed transaction computed, and one whose rows could not be read is unresolved and bars every day until the replay computes them; the overlay clears only after that commit, and a snapshot whose read the commit crossed reads again, so no snapshot sees neither the latch nor the row. Every hold written, and every admission and settlement, dirties the days it touches with a sequence above any snapshot's watermark, and publication enumerates a latched discovery's days from its bound, so a day already published complete is published again when a bound reaches it or a barrier lifts. Astra's sequence (T counted at 23:58:10 on the 2nd, the write failing, T settling at 00:02 on the 4th, the replay at 00:03) writes the 2nd, 3rd and 4th through T's interval and closes at 00:04 on the 4th, and a successor constructed after the replay backfills that closed interval for its owners; the fleet cannot be marked `closed` while the discovery is uncommitted (above); a process that dies with the latch held leaves a fleet not `closed`, which recovery covers from the fleet's `opened_at`. |
| **Who an uncertainty reaches, and what a fleet inherits.** Reach is computed over the chain, not stored per uncertainty: every fleet row names its predecessor, and a fleet's *unacknowledged ancestry* is the set of ancestors reached by following predecessors through fleets that have not been acknowledged (an acknowledged step ends the walk, because acknowledgement copies). An uncertainty, whenever it is opened, extended or closed, writes holds for the owners of its fleet and of every fleet whose unacknowledged ancestry contains that fleet (a query at write time, so an uncertainty discovered after a successor's construction reaches it, and so does one on a grandparent when the parent closed unopened without acknowledging); and **construction backfills**: the transaction that writes a new fleet's `open` row also writes, for the new fleet's owners, the holds of every interval already held by any uncertainty of its unacknowledged ancestry, open or closed (a successor built at 23:58:20 against a fleet whose uncertainty closed at 23:58:10 holds the 2nd for its owners from its first instant, before any acknowledgement), and the publisher's pending-discovery overlay reaches a fleet's owners through the same ancestry query; the barrier for an owner consults the unsettled admissions of its fleet and of its whole unacknowledged ancestry. Acknowledgement is terminal and **root-first**: a fleet may acknowledge its predecessor only when the predecessor is `closed` and has itself acknowledged its own predecessor (or has none), and it copies the predecessor's effective set (own and inherited, open and closed) into its inherited set (`inherited`: `fleet_id`, `uncertainty_id`) and writes holds for the closed ones its owners have not been held for, in one transaction; after that the chain walk stops there and the copy carries on. Astra's sequence: A has U open and an admission unsettled; B is built (its ancestry: A) and closes unopened without acknowledging; C is built against B (its ancestry: B, then A, since B has not acknowledged); on the 4th A's settlement extends U and the write reaches X, B's owners and C's owners through the ancestry; when A closes, B acknowledges A (root-first), then C acknowledges B and inherits U. Under owner scope the October sequence: Y, built at 23:58:20 with A in its ancestry, is barred on the 3rd by A's unsettled admission and held on the 4th by the settlement's write. |
diff --git a/docs/superpowers/specs/2026-10-03-replicator-pinned-roots-spikes.md b/docs/superpowers/specs/2026-10-03-replicator-pinned-roots-spikes.md| index 8955519e7..341b3a101 100644 |
| --- a/docs/superpowers/specs/2026-10-03-replicator-pinned-roots-spikes.md |
| +++ b/docs/superpowers/specs/2026-10-03-replicator-pinned-roots-spikes.md |
| @@ -22,7 +22,7 @@ Plan step 0's second spike is code, kept in tree at `internal/ops/admissionmodel |
| - **The boundaries are separate actions** (after the first code review): the writer's commit, the gate's answer (the recheck), a definite failure, an ambiguous return and its read-back, the send (the helper beginning the transport, which needs a durable issued row, recorded as a violation otherwise; a dead helper's send is a no-op, since a dead helper does nothing), the helper's proven exit (the guard closes) and the drain (transport-lost, unrecorded) are each their own step, so a schedule can put a discovery, a budget expiry or a crash between a commit and its answer, hold a token for a day before sending, or ask for a recorder's `Start` between the exit and the drain. The composites the named tests use (`WriterReturns`, `HelperDies`) are those steps in order. |
| - **Randomised schedules**: every action above in random interleavings over about four days, 120 steps a schedule, one owner per fleet (owner scope; the box-wide default is a superset); after each step, and right after every recovery, in the schedule and at quiescence, before anything else runs: the barrier, stated from the chain and not from the publisher (for every fleet and every owner of it, a durable unsettled admission of that fleet or of any fleet in its unacknowledged ancestry bars the owner from its day on, whether or not the token was ever handed out), the hazard invariant (once the ledger knows of a change, every day on which a transport overlapped the true change is incomplete for its owner; after a crash, every sent admission unsettled at the crash likewise; an issued token never sent has no transport and is covered by the barrier and recovery, not this oracle), a terminal state never changing, and a closed fleet having no latch or pending registration; at quiescence every fleet's owners hold every interval of every uncertainty on its whole chain, no uncertainty is open, and a day with no uncertainty reaching an owner is complete. `ADMISSION_MODEL_SEEDS=30000` passes in about six seconds; the default is 400. Publication is revisable by construction: a day published complete can be re-marked by a later hold (the design's materialisation and watermark), and the hazard invariant is satisfied by that retraction; the barrier's purpose is to make retraction rare, not to make it unnecessary. |
| - **What the runs found.** The first runs found the model's own bookkeeping wrong (an owner the fleet did not have; a crash-killed helper's transport end unset). After the second code review made the change its own action, discovered by a later look, Opus's sequence (an admission admitted before the last clean look, in flight across a change after it, settled clean before the discovery) showed the design's lower bound missing that admission's earlier day: **revision 9 of the design** adds the admission instant of every admission settled at or after the last clean look's start to the bound, and the mutant restoring revision 8's rule is killed by the named test. That is the one interleaving the design did not name that the model has found, within the actions it represents and the invariants it checks; the claim is no wider than that. |
| -- **Mutants, in a tree copy**, each killed by a named test or a schedule. After the eighth code review: `RecorderDone(finished|abandoned)` without `RecorderStart` (the guard test, and the schedules through a new check, a recorder outcome without a registration, which the schedule now attempts before Start); equivalent under this model: `GateAnswers` without its state check (`answered` is set only by the gate and nothing else leaves `admitted` before the answer), and the settlement's clamp to now (every participant instant is the model's own clock at its step, so the later completion is now; the ledger takes reported instants, where the clamp is live and its removal is killed by the ledger's clamp test). After the fourth code review: the historical term filtered to finished and abandoned (the unrecorded test), the cutoff as the running minimum (the fixed-bound test; the a1-first order kills it, the reversed order is the control), the historical term using the send instant, the historical term dropped, a later witness change ignored (the proof test). After the fifth: `>=` at the bound as `>` (the at-the-bound test), a witness change overwriting the first change's instant, and the invariant check right after a recovery is an oracle, not a mechanism: removing it fails nothing today, and it is kept so a recovery-mechanism fault shows at once rather than after later work repairs it. After the third code review: the threshold as the later of proof and look, as the look alone, the historical term including empty terminals, the proof not refusing a moved witness, the crash duplicating a landed row, `Close` without its unsettled loop, the latch overlay reaching the fleet's owners only. After the second code review: revision 8's bound (the in-flight test), the bound without the proof (the proof test), the bound without the clean look (the lower-bound test), a landed ambiguous row not durable until read-back, the open interval dropped from construction backfill (the per-step inheritance check), the replay's per-token writes dropped (the replay-while-open test), a read-back leaving the pending set entry. Removing the closed-implies-no-unsettled *check* removes an oracle; the mechanism mutant it exists for, `Close` without its unsettled-admissions loop, is killed by it (an unanswered durable row). Putting the send's terminal check before its gate is equivalent in the schedules tested (a settled, issued, unsent token exists after a clean stop, but no schedule sends it afterwards). After the first code review: the budget expiry a no-op (its own test, with no discovery); the barrier owner-only, or over issued rows only (the successor-barred test and the schedules); the pending set skipped at registration; the guard ignored, and the drain before the guard; the replay clearing the latch without the lower bound's holds (a named test: the schedules did not reach it in 3000 seeds); the send unchecked; a close without the replay (the closed-implies-no-latch check); acknowledgement copying the predecessor's own uncertainties only (not equivalent: a schedule found a fifth fleet's owner unheld for the first fleet's uncertainty, because the chain's copies are how a closed interval reaches a fleet built after its predecessor acknowledged); the answer ignoring the refusal; a read-back while the writer runs (the named test; the later commit would land on a cancelled registration, which the terminal check reports). Equivalent under this model: the unacknowledged-ancestry walk not stopping at an acknowledged step (over-reach writes the same holds). Not equivalent, though once claimed so here: the registry proof in the bound (Astra's schedule: a build, a clean look a day later, a discovery a day after that, with no admission: only the proof reaches the first day; and Opus's: a witness change known only to be after the proof; both are named tests). First: no construction backfill (reach); the pending set not counted (the commit-after-discovery tests); no recheck after commit (the same test, and a hazard at seed 264); the replay dropping counted settlements (hazard); the replay closing at the discovery instant (liveness and the latch tests); a settlement writing no interval (hazard); recovery ignoring the fence (the fence test); helper death without transport-lost (an uncertainty open at quiescence); the lower bound ignoring the clean look (the lower-bound test; the 30000-schedule sweep alone did not find it, which is why that sequence is a named test); the lower bound ignoring the earliest unsettled admission (liveness, because the model's proof then reached too far); unrecorded synthesised for a registered worker (the guard test); recovery skipping unclosed fleets (an uncertainty open at quiescence); acknowledgement not root-first and reach ignoring successors (reach). The equivalent mutants are named where they arise (the epoch check made redundant by the stale state; the unacknowledged-ancestry walk's over-reach; the send's terminal check before its gate), and the registry proof is not one. |
| +- **Mutants, in a tree copy**, each killed by a named test or a schedule. After the ninth code review: `GateAnswers` without its state check is not equivalent after all (Astra): recovery turns an admitted, unanswered row into `recovered-unknown`, and the gate answering it afterwards would leave a terminal; the named sequence kills it. After the eighth code review: `RecorderDone(finished|abandoned)` without `RecorderStart` (the guard test, and the schedules through a new check, a recorder outcome without a registration, which the schedule now attempts before Start); equivalent under this model: the settlement's clamp to now (every participant instant is the model's own clock at its step, so the later completion is now; the ledger takes reported instants, where the clamp is live and its removal is killed by the ledger's clamp test). After the fourth code review: the historical term filtered to finished and abandoned (the unrecorded test), the cutoff as the running minimum (the fixed-bound test; the a1-first order kills it, the reversed order is the control), the historical term using the send instant, the historical term dropped, a later witness change ignored (the proof test). After the fifth: `>=` at the bound as `>` (the at-the-bound test), a witness change overwriting the first change's instant, and the invariant check right after a recovery is an oracle, not a mechanism: removing it fails nothing today, and it is kept so a recovery-mechanism fault shows at once rather than after later work repairs it. After the third code review: the threshold as the later of proof and look, as the look alone, the historical term including empty terminals, the proof not refusing a moved witness, the crash duplicating a landed row, `Close` without its unsettled loop, the latch overlay reaching the fleet's owners only. After the second code review: revision 8's bound (the in-flight test), the bound without the proof (the proof test), the bound without the clean look (the lower-bound test), a landed ambiguous row not durable until read-back, the open interval dropped from construction backfill (the per-step inheritance check), the replay's per-token writes dropped (the replay-while-open test), a read-back leaving the pending set entry. Removing the closed-implies-no-unsettled *check* removes an oracle; the mechanism mutant it exists for, `Close` without its unsettled-admissions loop, is killed by it (an unanswered durable row). Putting the send's terminal check before its gate is equivalent in the schedules tested (a settled, issued, unsent token exists after a clean stop, but no schedule sends it afterwards). After the first code review: the budget expiry a no-op (its own test, with no discovery); the barrier owner-only, or over issued rows only (the successor-barred test and the schedules); the pending set skipped at registration; the guard ignored, and the drain before the guard; the replay clearing the latch without the lower bound's holds (a named test: the schedules did not reach it in 3000 seeds); the send unchecked; a close without the replay (the closed-implies-no-latch check); acknowledgement copying the predecessor's own uncertainties only (not equivalent: a schedule found a fifth fleet's owner unheld for the first fleet's uncertainty, because the chain's copies are how a closed interval reaches a fleet built after its predecessor acknowledged); the answer ignoring the refusal; a read-back while the writer runs (the named test; the later commit would land on a cancelled registration, which the terminal check reports). Equivalent under this model: the unacknowledged-ancestry walk not stopping at an acknowledged step (over-reach writes the same holds). Not equivalent, though once claimed so here: the registry proof in the bound (Astra's schedule: a build, a clean look a day later, a discovery a day after that, with no admission: only the proof reaches the first day; and Opus's: a witness change known only to be after the proof; both are named tests). First: no construction backfill (reach); the pending set not counted (the commit-after-discovery tests); no recheck after commit (the same test, and a hazard at seed 264); the replay dropping counted settlements (hazard); the replay closing at the discovery instant (liveness and the latch tests); a settlement writing no interval (hazard); recovery ignoring the fence (the fence test); helper death without transport-lost (an uncertainty open at quiescence); the lower bound ignoring the clean look (the lower-bound test; the 30000-schedule sweep alone did not find it, which is why that sequence is a named test); the lower bound ignoring the earliest unsettled admission (liveness, because the model's proof then reached too far); unrecorded synthesised for a registered worker (the guard test); recovery skipping unclosed fleets (an uncertainty open at quiescence); acknowledgement not root-first and reach ignoring successors (reach). The equivalent mutants are named where they arise (the epoch check made redundant by the stale state; the unacknowledged-ancestry walk's over-reach; the send's terminal check before its gate), and the registry proof is not one. |
| - **What the model does not cover**, by construction: the latch clearing after the commit as two instants (the replay is one step here; the rule stays with the ledger's tests), a snapshot between those instants, the recorder worker's lifetime and the socket drain as separate events, the fence as anything but helper liveness, retries of a failed terminal or replay write, the ledger's commit latency on a box's disk under the replicator's request rate (the spike's other half, which needs a box), Part C2's owner scope beyond the owner-scoped holds the model writes, and process kills inside a transaction (the model's transactions are atomic by fiat; the ledger's are SQLite's). |
| ## The fork's pathname entry points (read 2026-10-03 08:25, every call, not a sample) |
diff --git a/internal/ops/admissionmodel/model_test.go b/internal/ops/admissionmodel/model_test.go| index f1a53e3ad..5dabe3dfc 100644 |
| --- a/internal/ops/admissionmodel/model_test.go |
| +++ b/internal/ops/admissionmodel/model_test.go |
| @@ -921,6 +921,34 @@ func TestHistoricalCutoffIsTheFixedBound(t *testing.T) { |
| } |
| } |
| +// Review nine (Astra): the gate's state check is not implied by the |
| +// answered flag across recovery. An admitted, unanswered row survives a |
| +// crash; recovery classifies it recovered-unknown; a GateAnswers on it |
| +// afterwards is no edge, and the row keeps its terminal. |
| +func TestGateAnswersAfterRecoveryIsNoEdge(t *testing.T) { |
| + m := New(1) |
| + m.Now = at(1, 10, 0) |
| + f := m.Build(alice) |
| + a := m.Register(f, alice) |
| + m.WriterCommits(a) |
| + if a.State != Admitted || a.answered { |
| + t.Fatalf("precondition: %s answered=%v", a.State, a.answered) |
| + } |
| + m.Crash(func(*Fleet) bool { return false }) |
| + m.Now = at(1, 10, 5) |
| + if !m.Recover() { |
| + t.Fatal("recovery refused") |
| + } |
| + if a.State != RecoveredUnknown { |
| + t.Fatalf("after recovery: %s", a.State) |
| + } |
| + m.GateAnswers(a) |
| + if a.State != RecoveredUnknown || a.How != "recovered-unknown" { |
| + t.Fatalf("the gate answered a recovered row: %s %q", a.State, a.How) |
| + } |
| + must(t, m) |
| +} |
| + |
| // Review 3 (Astra 3): a landed ambiguous commit, then a crash before the |
| // read-back: one row, recovered-unknown, held. |
| func TestLandedAmbiguousThenCrash(t *testing.T) { |
diff --git a/internal/ops/authoritybridge.go b/internal/ops/authoritybridge.go| index 268a2588f..097d9880c 100644 |
| --- a/internal/ops/authoritybridge.go |
| +++ b/internal/ops/authoritybridge.go |
| @@ -3,6 +3,7 @@ package ops |
| import ( |
| "errors" |
| "path/filepath" |
| + "sort" |
| "strings" |
| "syscall" |
| @@ -38,9 +39,13 @@ type fleetPins struct { |
| pins *authority.Pins |
| roots map[string]*authority.Root |
| // listed is the facts of every database the strict tree walk listed, |
| - // by its spelled path: the identity the ownership checks must find |
| - // again (the listing-to-inspection rule across the handoff). |
| - listed map[string]authority.Facts |
| + // by its spelled path, and listedChain the facts of every directory |
| + // from the tree's root down to the one that listed it: the identities |
| + // the ownership checks must find again (the listing-to-inspection |
| + // rule across the handoff: the file, and the directory that listed |
| + // it, are the ones inspected, or the file is refused). |
| + listed map[string]authority.Facts |
| + listedChain map[string][]authority.Facts |
| } |
| func newFleetPins() (*fleetPins, error) { |
| @@ -48,7 +53,7 @@ func newFleetPins() (*fleetPins, error) { |
| if err != nil { |
| return nil, err |
| } |
| - p := &fleetPins{slash: slash, roots: map[string]*authority.Root{}, listed: map[string]authority.Facts{}} |
| + p := &fleetPins{slash: slash, roots: map[string]*authority.Root{}, listed: map[string]authority.Facts{}, listedChain: map[string][]authority.Facts{}} |
| p.pins = &authority.Pins{Slash: slash, Platform: platformFacts} |
| return p, nil |
| } |
| @@ -503,19 +508,68 @@ func (c *retainedChain) recheck() error { |
| } |
| // recheck asks every pinned root whether its ancestry still reads as it |
| -// did: a difference is a change the fleet goes stale on. |
| +// did: a difference is a change the fleet goes stale on. Every root is |
| +// asked, in a fixed order, and a change anywhere wins over a look that |
| +// could not be taken elsewhere; with no change, the first failed look is |
| +// returned. |
| func (p *fleetPins) recheck() error { |
| - for _, r := range p.roots { |
| - if changed, err := ancestryChanged(r); err != nil || changed { |
| - if err == nil { |
| - err = authority.ErrChanged |
| + paths := make([]string, 0, len(p.roots)) |
| + for path := range p.roots { |
| + paths = append(paths, path) |
| + } |
| + sort.Strings(paths) |
| + var first error |
| + for _, path := range paths { |
| + r := p.roots[path] |
| + changed, err := ancestryChanged(r) |
| + if err == nil && !changed { |
| + continue |
| + } |
| + if err == nil { |
| + err = authority.ErrChanged |
| + } |
| + err = &authority.EdgeError{Name: r.Path, Err: err} |
| + if isChange(err) { |
| + return err |
| + } |
| + if first == nil { |
| + first = err |
| + } |
| + } |
| + return first |
| +} |
| + |
| +// chainFacts is the identity of every directory from top down to d, by |
| +// the retained parent links; nil when top is not above d. |
| +func chainFacts(d, top *authority.Dir) []authority.Facts { |
| + var rev []authority.Facts |
| + for cur := d; cur != nil; cur = cur.Parent() { |
| + rev = append(rev, cur.Facts()) |
| + if cur == top { |
| + out := make([]authority.Facts, len(rev)) |
| + for i, f := range rev { |
| + out[len(rev)-1-i] = f |
| } |
| - return &authority.EdgeError{Name: r.Path, Err: err} |
| + return out |
| } |
| } |
| return nil |
| } |
| +// chainMatches reports whether the directories ending at d (d and its |
| +// retained parents, nearest first) are, by identity, the listed chain's |
| +// (its last entry being the listing directory). |
| +func chainMatches(d *authority.Dir, listed []authority.Facts) bool { |
| + cur := d |
| + for i := len(listed) - 1; i >= 0; i-- { |
| + if cur == nil || !cur.Facts().Same(listed[i]) { |
| + return false |
| + } |
| + cur = cur.Parent() |
| + } |
| + return true |
| +} |
| + |
| // ancestryChanged is a test seam over Root.AncestryChanged. |
| var ancestryChanged = func(r *authority.Root) (bool, error) { return r.AncestryChanged() } |
diff --git a/internal/ops/litestream.go b/internal/ops/litestream.go| index 3d405f3a5..815b05776 100644 |
| --- a/internal/ops/litestream.go |
| +++ b/internal/ops/litestream.go |
| @@ -176,14 +176,124 @@ func replicaKey(dbPath string) string { |
| // ParseLitestreamDBs recovers the database paths from a generated config. |
| // It reads only the exact shape LitestreamConfig writes (the round-trip is |
| // pinned by TestConfigRoundTrip) — it is not a YAML parser. |
| -func ParseLitestreamDBs(cfg string) []string { |
| +func ParseLitestreamDBs(cfg string) ([]string, error) { |
| + type item struct { |
| + indent int |
| + path string |
| + seen bool |
| + } |
| var dbs []string |
| - for _, line := range strings.Split(cfg, "\n") { |
| - if p, ok := strings.CutPrefix(line, " - path: "); ok { |
| - dbs = append(dbs, strings.TrimSpace(p)) |
| + inDBs := false |
| + var cur *item |
| + finish := func() error { |
| + if cur != nil && !cur.seen { |
| + return errors.New("litestream config: a dbs entry has no path") |
| + } |
| + if cur != nil { |
| + dbs = append(dbs, cur.path) |
| + } |
| + cur = nil |
| + return nil |
| + } |
| + sawPathKey := false |
| + for n, raw := range strings.Split(cfg, "\n") { |
| + line := strings.TrimRight(raw, " \t\r") |
| + trim := strings.TrimLeft(line, " \t") |
| + if trim == "" || strings.HasPrefix(trim, "#") { |
| + continue |
| + } |
| + indent := len(line) - len(trim) |
| + if !inDBs { |
| + if indent == 0 { |
| + if rest, ok := strings.CutPrefix(trim, "dbs:"); ok { |
| + rest = strings.TrimSpace(stripComment(rest)) |
| + switch rest { |
| + case "": |
| + inDBs = true |
| + case "[]": |
| + return nil, nil |
| + default: |
| + return nil, fmt.Errorf("litestream config line %d: dbs in a form this reader does not take (%q); block style only", n+1, rest) |
| + } |
| + } |
| + } |
| + if strings.HasPrefix(trim, "path:") || strings.HasPrefix(trim, "- path:") { |
| + sawPathKey = true |
| + } |
| + continue |
| + } |
| + if indent == 0 && !strings.HasPrefix(trim, "-") { |
| + if err := finish(); err != nil { |
| + return nil, err |
| + } |
| + inDBs = false |
| + continue |
| + } |
| + if strings.HasPrefix(trim, "-") && (cur == nil || indent <= cur.indent) { |
| + if err := finish(); err != nil { |
| + return nil, err |
| + } |
| + cur = &item{indent: indent} |
| + trim = strings.TrimLeft(strings.TrimPrefix(trim, "-"), " \t") |
| + if trim == "" { |
| + continue |
| + } |
| + indent += 2 // the item's first key on the dash's line |
| + } |
| + if cur == nil { |
| + return nil, fmt.Errorf("litestream config line %d: %q is under dbs but not in a list entry", n+1, trim) |
| } |
| + if v, ok := strings.CutPrefix(trim, "path:"); ok && indent > cur.indent { |
| + val, err := yamlScalar(strings.TrimSpace(stripComment(v))) |
| + if err != nil { |
| + return nil, fmt.Errorf("litestream config line %d: %v", n+1, err) |
| + } |
| + if cur.seen { |
| + return nil, fmt.Errorf("litestream config line %d: a dbs entry has two paths", n+1) |
| + } |
| + cur.path, cur.seen = val, true |
| + } |
| + } |
| + if err := finish(); err != nil { |
| + return nil, err |
| + } |
| + if len(dbs) == 0 && sawPathKey { |
| + return nil, errors.New("litestream config: path keys found outside a dbs list") |
| + } |
| + return dbs, nil |
| +} |
| + |
| +// stripComment removes a trailing YAML comment (a # after whitespace). |
| +func stripComment(s string) string { |
| + if i := strings.Index(s, " #"); i >= 0 { |
| + return s[:i] |
| + } |
| + if strings.HasPrefix(s, "#") { |
| + return "" |
| } |
| - return dbs |
| + return s |
| +} |
| + |
| +// yamlScalar reads a plain, single-quoted or double-quoted scalar. |
| +func yamlScalar(v string) (string, error) { |
| + switch { |
| + case v == "": |
| + return "", errors.New("an empty path") |
| + case strings.HasPrefix(v, "\""): |
| + s, err := strconv.Unquote(v) |
| + if err != nil { |
| + return "", fmt.Errorf("a double-quoted path that cannot be read: %s", v) |
| + } |
| + return s, nil |
| + case strings.HasPrefix(v, "'"): |
| + if len(v) < 2 || !strings.HasSuffix(v, "'") { |
| + return "", fmt.Errorf("a single-quoted path that cannot be read: %s", v) |
| + } |
| + return strings.ReplaceAll(v[1:len(v)-1], "''", "'"), nil |
| + case strings.HasPrefix(v, "[") || strings.HasPrefix(v, "{") || strings.HasPrefix(v, "&") || strings.HasPrefix(v, "*") || strings.HasPrefix(v, "|") || strings.HasPrefix(v, ">"): |
| + return "", fmt.Errorf("a path in a form this reader does not take: %s", v) |
| + } |
| + return v, nil |
| } |
| // CoverageShortfall names every database the EXISTING config replicates that |
| @@ -199,19 +309,23 @@ func ParseLitestreamDBs(cfg string) []string { |
| // |
| // Growing the set is always fine and needs no ceremony; shrinking it is a |
| // decision someone has to make on purpose. |
| -func CoverageShortfall(existingConfig string, proposed []string) []string { |
| +func CoverageShortfall(existingConfig string, proposed []string) ([]string, error) { |
| have := map[string]bool{} |
| for _, p := range proposed { |
| have[filepath.Clean(p)] = true |
| } |
| + existing, err := ParseLitestreamDBs(existingConfig) |
| + if err != nil { |
| + return nil, err // a configuration this reader cannot read is never quietly shrunk |
| + } |
| var missing []string |
| - for _, p := range ParseLitestreamDBs(existingConfig) { |
| + for _, p := range existing { |
| if !have[filepath.Clean(p)] { |
| missing = append(missing, p) |
| } |
| } |
| sort.Strings(missing) |
| - return missing |
| + return missing, nil |
| } |
| // FleetDBs lists the non-empty *.db files this box-wide replicator owns: |
| @@ -591,7 +705,10 @@ func RestoreVerify(litestreamBin, cfgPath, tmpDir string) error { |
| if err != nil { |
| return fmt.Errorf("restore-verify: %v", err) |
| } |
| - dbs := ParseLitestreamDBs(string(raw)) |
| + dbs, err := ParseLitestreamDBs(string(raw)) |
| + if err != nil { |
| + return fmt.Errorf("restore-verify: %s: %v", cfgPath, err) |
| + } |
| if len(dbs) == 0 { |
| return fmt.Errorf("restore-verify: no databases in %s — regenerate it: carlos ops litestream", cfgPath) |
| } |
diff --git a/internal/ops/litestream_metered_test.go b/internal/ops/litestream_metered_test.go| index 13d916371..a205ed98f 100644 |
| --- a/internal/ops/litestream_metered_test.go |
| +++ b/internal/ops/litestream_metered_test.go |
| @@ -2127,7 +2127,7 @@ func TestMeteredFleetDriftAtTheFinalLookStalesAndChangeWinsOverDrift(t *testing. |
| newFakeBucket(t) |
| dir := platformDir(t) |
| reg := openRegistry(t, filepath.Join(dir, "registry.db")) |
| - state := filepath.Join(platformDir(t), "unit-a") |
| + state := filepath.Join(platformDir(t), "a-unit-a") // sorts before the empty tree's root, so its failed look is met first |
| db := writeFile(t, filepath.Join(state, "data", "main.db")) |
| upsert(t, reg, registry.Route{Host: "a.example.com", Account: "bac", App: "one", DBPath: state, Backing: registry.BackingUnit, Version: "v1"}) |
| fleet, err := MeteredFleetStore(fleetOptions(reg, dir)) |
| @@ -2148,8 +2148,15 @@ func TestMeteredFleetDriftAtTheFinalLookStalesAndChangeWinsOverDrift(t *testing. |
| } |
| recheckHeld = func(inv *FleetInventory) error { return inv.recheck() } |
| - // change over drift: every root's ancestry look fails, and a chain |
| - // edge changed |
| + // change over drift, between roots: the first root examined (the |
| + // tree with the database, "a-...") cannot be looked at, and a later |
| + // root with no owned file beneath it ("z-...") changed; no chain |
| + // covers the changed root, so only the roots' recheck can see it |
| + empty := filepath.Join(platformDir(t), "z-empty-tree") |
| + if err := os.MkdirAll(empty, 0o755); err != nil { |
| + t.Fatal(err) |
| + } |
| + upsert(t, reg, registry.Route{Host: "e.example.com", Account: "bae", App: "three", DBPath: empty, Backing: registry.BackingUnit, Version: "v1"}) |
| fleet2, err := MeteredFleetStore(fleetOptions(reg, dir)) |
| if err != nil { |
| t.Fatal(err) |
| @@ -2159,14 +2166,23 @@ func TestMeteredFleetDriftAtTheFinalLookStalesAndChangeWinsOverDrift(t *testing. |
| t.Fatalf("a write before any change: %v", err) |
| } |
| clean2 := fleet2.lastCleanLook() |
| - ancestryChanged = func(*authority.Root) (bool, error) { return false, syscall.EIO } |
| + ancestryChanged = func(r *authority.Root) (bool, error) { |
| + if strings.Contains(r.Path, "unit-a") { |
| + return false, syscall.EIO |
| + } |
| + return r.AncestryChanged() |
| + } |
| t.Cleanup(func() { ancestryChanged = func(r *authority.Root) (bool, error) { return r.AncestryChanged() } }) |
| - if err := os.Chmod(filepath.Join(state, "data"), 0o700); err != nil { |
| + if err := os.Chmod(filepath.Dir(empty), 0o775); err != nil { // the empty root's parent fails the platform rule |
| t.Fatal(err) |
| } |
| + t.Cleanup(func() { os.Chmod(filepath.Dir(empty), 0o755) }) |
| fleet2.lookAgain() |
| if err := send(t, fleet2, db, 4); !errors.Is(err, ErrOwnershipStale) { |
| - t.Fatalf("a write after a chain edge changed beside a root that could not be looked at: %v, want stale (change wins over drift)", err) |
| + t.Fatalf("a write after a root changed behind a root that could not be looked at: %v, want stale (change wins over drift)", err) |
| + } |
| + if fleet2.drift.Load() { |
| + t.Fatal("drift recorded over a change") |
| } |
| if from, _, stale := fleet2.staleWindow(); !stale || !from.Equal(clean2.Add(-ownershipMargin)) { |
| t.Fatalf("the hold starts at %v, want the last clean look %v minus the margin", from, clean2) |
diff --git a/internal/ops/litestream_owners.go b/internal/ops/litestream_owners.go| index 4adfa7c6a..c0c460c43 100644 |
| --- a/internal/ops/litestream_owners.go |
| +++ b/internal/ops/litestream_owners.go |
| @@ -389,6 +389,15 @@ func fleetInventory(routes []registry.Route, registryPath, dataDir string) (Flee |
| } |
| continue |
| } |
| + // the directory that listed the file is the one it was checked |
| + // in, and so is every directory above it to the tree's root (the |
| + // same file moved into a replacement directory passes the file's |
| + // identity and not this) |
| + if lc, ok := pins.listedChain[p]; ok && facts.keep != nil && !chainMatches(facts.keep.dir, lc) { |
| + unowned(p, "it changed under the look (the directory it was checked in is not the directory that listed it)") |
| + facts.keep.release() |
| + continue |
| + } |
| inv.Owned = append(inv.Owned, OwnedDB{Path: p, Account: owner.Account, App: owner.App, Host: owner.Host, Platform: owner.Account == release.OpsAccount}) |
| inv.facts[p] = facts |
| } |
| @@ -863,6 +872,10 @@ func platformOwned(dir string) string { |
| // acquisition and its listing, or between the listing and what follows. |
| var treeAfterList, treeBeforeList func(d *authority.Dir, dir string) |
| +// treeBeforeReacquire is a test seam between the second look at a |
| +// replaced entry and its acquisition. |
| +var treeBeforeReacquire func(d *authority.Dir, path string) |
| + |
| // strictTreeMaxDepth bounds strictTreeDBs. Past treeDBWalkDepth a database |
| // is reported and not replicated; past this the walk says it cannot vouch |
| // for what lies below. |
| @@ -1041,6 +1054,9 @@ func strictTreeDBsOn(pins *fleetPins, root string) (dbs, skipped, problems []str |
| if k2 != authority.Directory { |
| continue // a file or a link now, or gone: churn, as a vanished entry is; the next look classifies it |
| } |
| + if treeBeforeReacquire != nil { |
| + treeBeforeReacquire(d, p) |
| + } |
| child, cerr = d.Child(name, &f2) |
| if cerr != nil { |
| problems = append(problems, p+": changed under the look") |
| @@ -1061,6 +1077,7 @@ func strictTreeDBsOn(pins *fleetPins, root string) (dbs, skipped, problems []str |
| case strings.HasSuffix(name, ".db") && !hidden && !strings.HasPrefix(name, "."): |
| dbs = append(dbs, p) |
| pins.listed[p] = e.Facts |
| + pins.listedChain[p] = chainFacts(d, top) |
| case sqliteName(name): |
| skipped = append(skipped, p) |
| default: |
diff --git a/internal/ops/litestream_owners_unix_test.go b/internal/ops/litestream_owners_unix_test.go| index 259ac14ae..eb999deea 100644 |
| --- a/internal/ops/litestream_owners_unix_test.go |
| +++ b/internal/ops/litestream_owners_unix_test.go |
| @@ -1007,6 +1007,54 @@ func TestStrictTreeBetweenListingAndLook(t *testing.T) { |
| if !contains(dbs, filepath.Join(root, "sub", "new.db")) || contains(dbs, filepath.Join(root, "sub", "old.db")) { |
| t.Fatalf("a swapped subdirectory: dbs %v (want the replacement, looked at again)", dbs) |
| } |
| + // replaced again between the second look and the acquisition: a |
| + // problem, and neither replacement's database is listed (the first |
| + // acquisition must carry the listed facts, or the first replacement |
| + // is walked and the second never seen) |
| + root2 := platformDir(t) |
| + mkdirs(t, filepath.Join(root2, "sub")) |
| + writeFile(t, filepath.Join(root2, "sub", "old.db")) |
| + swaps := 0 |
| + swap := func(name string) { |
| + if err := os.Rename(filepath.Join(root2, "sub"), filepath.Join(root2, "gone"+name)); err != nil { |
| + setupErr = err |
| + return |
| + } |
| + if err := os.MkdirAll(filepath.Join(root2, "sub"), 0o755); err != nil { |
| + setupErr = err |
| + return |
| + } |
| + if err := os.Chmod(filepath.Join(root2, "sub"), 0o755); err != nil { |
| + setupErr = err |
| + return |
| + } |
| + if err := os.WriteFile(filepath.Join(root2, "sub", name+".db"), []byte("x"), 0o644); err != nil { |
| + setupErr = err |
| + } |
| + } |
| + treeAfterList = func(d *authority.Dir, dir string) { |
| + if dir == root2 && swaps == 0 { |
| + swaps++ |
| + swap("first") |
| + } |
| + } |
| + treeBeforeReacquire = func(d *authority.Dir, p string) { |
| + if p == filepath.Join(root2, "sub") && swaps == 1 { |
| + swaps++ |
| + swap("second") |
| + } |
| + } |
| + t.Cleanup(func() { treeBeforeReacquire = nil }) |
| + dbs2, _, problems2 := strictTreeDBs(root2) |
| + if setupErr != nil { |
| + t.Fatal(setupErr) |
| + } |
| + if swaps != 2 { |
| + t.Fatalf("%d swaps ran, want both", swaps) |
| + } |
| + if containsPrefix(dbs2, filepath.Join(root2, "sub")) || !named(problems2, filepath.Join(root2, "sub"), "changed under the look") { |
| + t.Fatalf("replaced twice: dbs %v problems %v", dbs2, problems2) |
| + } |
| if !named(problems, filepath.Join(root, "plain.data"), "cannot be read") { |
| t.Fatalf("a FIFO in a file's place: %v", problems) |
| } |
| @@ -1386,10 +1434,49 @@ func TestFleetInventoryRefusesAFileNotTheOneListed(t *testing.T) { |
| if err != nil { |
| t.Fatal(err) |
| } |
| - defer inv.Close() |
| if _, owned := ownedByPath(inv)[db]; !owned { |
| t.Fatalf("the listed file is not owned: unowned %v problems %v", inv.Unowned, inv.Problems) |
| } |
| + inv.Close() |
| + // the same file (one inode) moved into a replacement directory: the |
| + // file's identity passes, the directory that listed it is not the one |
| + // it is checked in, and it is refused (Astra, review nine) |
| + moved := false |
| + inventoryAfterDiscovery = func() { |
| + if moved { |
| + return |
| + } |
| + moved = true |
| + if err := os.Rename(filepath.Join(state, "sub"), filepath.Join(state, "gone2")); err != nil { |
| + setupErr = err |
| + return |
| + } |
| + if err := os.MkdirAll(filepath.Join(state, "sub"), 0o755); err != nil { |
| + setupErr = err |
| + return |
| + } |
| + if err := os.Chmod(filepath.Join(state, "sub"), 0o755); err != nil { |
| + setupErr = err |
| + return |
| + } |
| + if err := os.Rename(filepath.Join(state, "gone2", "main.db"), db); err != nil { |
| + setupErr = err |
| + } |
| + } |
| + inv, err = fleetInventory(routes, reg.Path(), dir) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if setupErr != nil { |
| + t.Fatal(setupErr) |
| + } |
| + if _, owned := ownedByPath(inv)[db]; owned || !named(inv.Unowned, db, "not the directory that listed it") { |
| + t.Fatalf("the same file in a replacement directory: owned %+v unowned %v", inv.Owned, inv.Unowned) |
| + } |
| + inv.Close() |
| + if n := openDirFDsUnder(t, filepath.Dir(dir)); n != before { |
| + t.Fatalf("%d directory descriptors left after the refusal", n-before) |
| + } |
| } |
| // Review eight (Astra 5): the report compares the platform inventory with |
| @@ -1429,4 +1516,35 @@ func TestInventoryReportComparesTheLiveConfiguration(t *testing.T) { |
| if !strings.Contains(report, "live configuration: 0 databases") || !strings.Contains(report, "owned but not in the live configuration: "+strconv.Quote(owned)) { |
| t.Fatalf("report with no live configuration:\n%s", report) |
| } |
| + // a hand-written live configuration reads the same; one this reader |
| + // cannot take is an error, not an empty comparison |
| + hand := "dbs:\n- path: '" + stale + "'\n replicas:\n - url: s3://b/p\n" |
| + report, err = InventoryReport(reg, dir, hand) |
| + if err != nil || !strings.Contains(report, " live but not owned by the platform replicator: "+strconv.Quote(stale)) { |
| + t.Fatalf("hand-written live configuration: %v\n%s", err, report) |
| + } |
| + if _, err := InventoryReport(reg, dir, "dbs: [{path: /x.db}]\n"); err == nil { |
| + t.Fatal("a live configuration the reader cannot take was compared") |
| + } |
| + // the required set is this process's environment, and says so; a |
| + // console database at the conventional path that is owned but not |
| + // required is said |
| + t.Setenv("CONSOLE_DB", "") |
| + console := writeFile(t, filepath.Join(dir, "console", "console.db")) |
| + report, err = InventoryReport(reg, dir, "") |
| + if err != nil || !strings.Contains(report, "as this process's environment has it") || !strings.Contains(report, "not required: the console database "+strconv.Quote(console)+" is owned but CONSOLE_DB is not set") { |
| + t.Fatalf("the environment note and the not-required line: %v\n%s", err, report) |
| + } |
| + // the command: a missing registry is an error and creates nothing |
| + missing := filepath.Join(dir, "nope", "registry.db") |
| + if _, err := inventoryCommand(missing, dir, filepath.Join(dir, "no-such.yml")); err == nil { |
| + t.Fatal("the command ran against a missing registry") |
| + } |
| + if _, err := os.Stat(missing); !os.IsNotExist(err) { |
| + t.Fatalf("the command created the registry: %v", err) |
| + } |
| + out, err := inventoryCommand(reg.Path(), dir, filepath.Join(dir, "no-such.yml")) |
| + if err != nil || !strings.Contains(out, "no configuration at") || !strings.Contains(out, " owned "+strconv.Quote(owned)) { |
| + t.Fatalf("the command: %v\n%s", err, out) |
| + } |
| } |
diff --git a/internal/ops/ops.go b/internal/ops/ops.go| index f4a128931..6b4371f78 100644 |
| --- a/internal/ops/ops.go |
| +++ b/internal/ops/ops.go |
| @@ -142,7 +142,11 @@ func runLitestream(args []string) { |
| // CoverageShortfall. This is what makes a cutover from a hand-maintained |
| // config to a generated one safe to do unattended. |
| if prior, err := os.ReadFile(*out); err == nil { |
| - if missing := CoverageShortfall(string(prior), dbs); len(missing) > 0 && !*allowShrink { |
| + missing, perr := CoverageShortfall(string(prior), dbs) |
| + if perr != nil { |
| + log.Fatalf("litestream: REFUSING to write %s — it cannot be read, so what it names cannot be compared: %v\nNothing has been changed.", *out, perr) |
| + } |
| + if len(missing) > 0 && !*allowShrink { |
| log.Fatalf(`litestream: REFUSING to write %s — it would stop replicating %d database(s) that %s names today: |
| %s |
| Nothing has been changed. Either make the registry describe them (a route with |
| @@ -365,28 +369,38 @@ func runInventory(args []string) { |
| cfg := fs.String("config", defaultConfigPath, "the external replicator's configuration, compared against the inventory") |
| fs.Usage = cliflag.Usage(fs, "") |
| _ = fs.Parse(args) |
| - dir := dataDir(*data) |
| - rp := *regPath |
| - if rp == "" { |
| - rp = dir + "/registry.db" |
| + report, err := inventoryCommand(*regPath, dataDir(*data), *cfg) |
| + if err != nil { |
| + log.Fatalf("inventory: %v", err) |
| } |
| - reg, err := registry.OpenReadOnly(rp) |
| + fmt.Print(report) |
| +} |
| + |
| +// inventoryCommand is carlos ops inventory without its process: the |
| +// registry opened read-only (a missing or wrong path is an error, and |
| +// nothing is created or migrated), the live configuration read if it is |
| +// there, and the report. |
| +func inventoryCommand(regPath, dir, cfgPath string) (string, error) { |
| + if regPath == "" { |
| + regPath = dir + "/registry.db" |
| + } |
| + reg, err := registry.OpenReadOnly(regPath) |
| if err != nil { |
| - log.Fatalf("inventory: open registry %s read-only: %v", rp, err) |
| + return "", fmt.Errorf("open registry %s read-only: %v", regPath, err) |
| } |
| defer reg.Close() |
| - live, err := os.ReadFile(*cfg) |
| + live, err := os.ReadFile(cfgPath) |
| if err != nil && !os.IsNotExist(err) { |
| - log.Fatalf("inventory: read %s: %v", *cfg, err) |
| + return "", fmt.Errorf("read %s: %v", cfgPath, err) |
| } |
| report, err := InventoryReport(reg, dir, string(live)) |
| if err != nil { |
| - log.Fatalf("inventory: %v", err) |
| + return "", err |
| } |
| if live == nil { |
| - fmt.Printf("external replicator: no configuration at %q\n", *cfg) |
| + report = fmt.Sprintf("external replicator: no configuration at %q\n", cfgPath) + report |
| } |
| - fmt.Print(report) |
| + return report, nil |
| } |
| // InventoryReport renders, for one box, the databases the external |
| @@ -407,8 +421,12 @@ func InventoryReport(reg *registry.Registry, dir, liveConfig string) (string, er |
| if err != nil { |
| return "", err |
| } |
| + parsed, err := ParseLitestreamDBs(liveConfig) |
| + if err != nil { |
| + return "", fmt.Errorf("the live configuration cannot be read, so it cannot be compared: %v", err) |
| + } |
| var live []string |
| - for _, p := range ParseLitestreamDBs(liveConfig) { |
| + for _, p := range parsed { |
| live = append(live, filepath.Clean(p)) |
| } |
| sort.Strings(live) |
| @@ -467,7 +485,7 @@ func InventoryReport(reg *registry.Registry, dir, liveConfig string) (string, er |
| fmt.Fprintf(&b, " owned but a regeneration would not write it: %q\n", o.Path) |
| } |
| } |
| - fmt.Fprintf(&b, "required: %d databases\n", len(inv.Required)) |
| + fmt.Fprintf(&b, "required: %d databases (as this process's environment has it: CONSOLE_DB and the registry path are read here, and the agent's unit environment is the authority for what a build requires)\n", len(inv.Required)) |
| required := map[string]bool{} |
| for _, r := range inv.Required { |
| required[r.Path] = true |
diff --git a/internal/ops/ops_test.go b/internal/ops/ops_test.go| index edfcb71ae..3ffdf8c28 100644 |
| --- a/internal/ops/ops_test.go |
| +++ b/internal/ops/ops_test.go |
| @@ -185,7 +185,7 @@ func TestFleetDBsDedupesUnitPathUnderDataDir(t *testing.T) { |
| // ParseLitestreamDBs must recover exactly. |
| func TestConfigRoundTrip(t *testing.T) { |
| in := []string{"/data/registry.db", "/data/a.example.com.db", "/data/b.example.com.db"} |
| - got := ParseLitestreamDBs(LitestreamConfig(in, "b", "p", "eu-west-1")) |
| + got, _ := ParseLitestreamDBs(LitestreamConfig(in, "b", "p", "eu-west-1")) |
| if len(got) != len(in) { |
| t.Fatalf("round-trip lost databases: in %d, out %d (%v)", len(in), len(got), got) |
| } |
| @@ -201,11 +201,11 @@ func TestConfigRoundTrip(t *testing.T) { |
| } |
| func TestParseLitestreamDBsIgnoresNoise(t *testing.T) { |
| - got := ParseLitestreamDBs("# comment\ndbs:\n - path: /data/x.db\n replicas:\n - url: s3://b/p/x.db\n region: eu-west-1\n\n") |
| + got, _ := ParseLitestreamDBs("# comment\ndbs:\n - path: /data/x.db\n replicas:\n - url: s3://b/p/x.db\n region: eu-west-1\n\n") |
| if len(got) != 1 || got[0] != "/data/x.db" { |
| t.Fatalf("ParseLitestreamDBs = %#v, want [/data/x.db]", got) |
| } |
| - if got := ParseLitestreamDBs(""); len(got) != 0 { |
| + if got, _ := ParseLitestreamDBs(""); len(got) != 0 { |
| t.Fatalf("ParseLitestreamDBs(\"\") = %#v, want none", got) |
| } |
| } |
| @@ -341,7 +341,7 @@ func TestConfigPinsSnapshotRetention(t *testing.T) { |
| t.Fatalf("SnapshotRetention = %s, which is no better than litestream's default", SnapshotRetention) |
| } |
| // And the round-trip the restore-verify verb depends on must survive it. |
| - if got := ParseLitestreamDBs(cfg); len(got) != 1 || got[0] != "/data/registry.db" { |
| + if got, _ := ParseLitestreamDBs(cfg); len(got) != 1 || got[0] != "/data/registry.db" { |
| t.Fatalf("ParseLitestreamDBs broke on the retention stanza: %v", got) |
| } |
| } |
| @@ -687,20 +687,20 @@ func TestCoverageShortfall(t *testing.T) { |
| existing := LitestreamConfig([]string{"/data/a.db", "/data/b.db", "/data/registry.db"}, "b", "p", "eu-west-1") |
| // A superset is fine — that is the ordinary case (console.db arriving). |
| - if missing := CoverageShortfall(existing, []string{"/data/a.db", "/data/b.db", "/data/registry.db", "/data/console/console.db"}); len(missing) != 0 { |
| + if missing, _ := CoverageShortfall(existing, []string{"/data/a.db", "/data/b.db", "/data/registry.db", "/data/console/console.db"}); len(missing) != 0 { |
| t.Fatalf("a superset was reported as a shortfall: %v", missing) |
| } |
| // Identical is fine. |
| - if missing := CoverageShortfall(existing, []string{"/data/registry.db", "/data/b.db", "/data/a.db"}); len(missing) != 0 { |
| + if missing, _ := CoverageShortfall(existing, []string{"/data/registry.db", "/data/b.db", "/data/a.db"}); len(missing) != 0 { |
| t.Fatalf("an identical set was reported as a shortfall: %v", missing) |
| } |
| // Dropping one is the thing to catch, and the caller must be told WHICH. |
| - missing := CoverageShortfall(existing, []string{"/data/a.db", "/data/registry.db"}) |
| + missing, _ := CoverageShortfall(existing, []string{"/data/a.db", "/data/registry.db"}) |
| if len(missing) != 1 || missing[0] != "/data/b.db" { |
| t.Fatalf("CoverageShortfall = %v, want [/data/b.db]", missing) |
| } |
| // No existing config (a fresh box) can never be a shortfall. |
| - if missing := CoverageShortfall("", []string{"/data/a.db"}); len(missing) != 0 { |
| + if missing, _ := CoverageShortfall("", []string{"/data/a.db"}); len(missing) != 0 { |
| t.Fatalf("a fresh box reported a shortfall: %v", missing) |
| } |
| } |
| @@ -790,3 +790,52 @@ func TestFleetDBsSkipsUnclaimedFilesAndSaysSo(t *testing.T) { |
| t.Fatalf("the unclaimed file was dropped SILENTLY — skipped = %v", skipped) |
| } |
| } |
| + |
| +// Review nine (Astra 8): the live configuration is hand-maintained YAML as |
| +// often as the generator's output: items at any indentation, quoted paths, |
| +// comments, the path on the line after the dash. A form this reader does |
| +// not take is an error, never an empty list; a path key outside a dbs list |
| +// is an error too. |
| +func TestParseLitestreamDBsReadsHandWrittenConfigurations(t *testing.T) { |
| + cfg := `# hand-maintained |
| +access-key-id: x |
| +dbs: |
| +- path: /data/retired.db |
| + replicas: |
| + - url: s3://b/p/retired.db |
| +- path: "/data/with space.db" # quoted |
| + replicas: [] |
| +- |
| + path: '/data/it''s.db' |
| +- replicas: [] |
| + path: /data/last.db |
| +logging: |
| + level: info |
| +` |
| + got, err := ParseLitestreamDBs(cfg) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + want := []string{"/data/retired.db", "/data/with space.db", "/data/it's.db", "/data/last.db"} |
| + if strings.Join(got, ",") != strings.Join(want, ",") { |
| + t.Fatalf("ParseLitestreamDBs = %q, want %q", got, want) |
| + } |
| + for name, bad := range map[string]string{ |
| + "flow style": "dbs: [{path: /data/a.db}]\n", |
| + "no path": "dbs:\n - replicas: []\n", |
| + "two paths": "dbs:\n - path: /a.db\n path: /b.db\n", |
| + "bad quote": "dbs:\n - path: \"/a.db\n", |
| + "stray path": "replicas:\n - path: /a.db\n", |
| + "not in a list": "dbs:\n path: /a.db\n", |
| + } { |
| + if got, err := ParseLitestreamDBs(bad); err == nil { |
| + t.Fatalf("%s: read as %q, want an error", name, got) |
| + } |
| + } |
| + if _, err := CoverageShortfall("dbs: [{path: /data/a.db}]\n", []string{"/data/a.db"}); err == nil { |
| + t.Fatal("the shrink guard compared against a configuration it could not read") |
| + } |
| + if got, err := ParseLitestreamDBs("dbs: []\n"); err != nil || len(got) != 0 { |
| + t.Fatalf("an empty list: %q %v", got, err) |
| + } |
| +} |
diff --git a/internal/registry/readonly_test.go b/internal/registry/readonly_test.go| index dfba95e82..ff7ef1b52 100644 |
| --- a/internal/registry/readonly_test.go |
| +++ b/internal/registry/readonly_test.go |
| @@ -42,4 +42,32 @@ func TestOpenReadOnly(t *testing.T) { |
| if _, err := OpenReadOnly(dir); err == nil { |
| t.Fatal("a directory opened as a registry") |
| } |
| + // a # and a space in the path are the path's, not the URI's (the |
| + // registry's own methods reopen by its path, so a path the driver |
| + // reads as a URI, with a ? or a %, is unusable for a registry in any |
| + // mode; the registry is made at a plain path and its directory renamed) |
| + plain := filepath.Join(dir, "plain", "registry.db") |
| + if err := os.MkdirAll(filepath.Dir(plain), 0o755); err != nil { |
| + t.Fatal(err) |
| + } |
| + rw2, err := Open(plain) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := rw2.Upsert(Route{Host: "c.example.com", Account: "bac", App: "one"}); err != nil { |
| + t.Fatal(err) |
| + } |
| + rw2.Close() |
| + odd := filepath.Join(dir, "odd # dir", "registry.db") |
| + if err := os.Rename(filepath.Dir(plain), filepath.Dir(odd)); err != nil { |
| + t.Fatal(err) |
| + } |
| + ro2, err := OpenReadOnly(odd) |
| + if err != nil { |
| + t.Fatalf("an odd path: %v", err) |
| + } |
| + defer ro2.Close() |
| + if routes, err := ro2.List(); err != nil || len(routes) != 1 || routes[0].Host != "c.example.com" { |
| + t.Fatalf("list through the odd path: %v %+v", err, routes) |
| + } |
| } |
diff --git a/internal/registry/registry.go b/internal/registry/registry.go| index dd1ba5e36..29dfd932b 100644 |
| --- a/internal/registry/registry.go |
| +++ b/internal/registry/registry.go |
| @@ -13,7 +13,9 @@ import ( |
| "crypto/sha256" |
| "database/sql" |
| "fmt" |
| + "net/url" |
| "os" |
| + "path/filepath" |
| "regexp" |
| "strings" |
| "time" |
| @@ -412,11 +414,16 @@ func Open(path string) (*Registry, error) { |
| return r, nil |
| } |
| -// OpenReadOnly opens an existing registry for reading alone: no file is |
| -// created, no journal mode is set and nothing is migrated, so a report |
| -// standing on a box (carlos ops inventory) writes nothing, and a wrong |
| -// path is an error rather than an empty registry. A write through the |
| -// handle fails. |
| +// OpenReadOnly opens an existing registry for reading alone: the registry |
| +// is not created, no journal mode is set and nothing is migrated, so a |
| +// report standing on a box (carlos ops inventory) changes nothing in it, |
| +// and a wrong path is an error rather than an empty registry. A write |
| +// through the handle fails. What a read-only open of a WAL database may |
| +// still do is create the -shm file beside it when no other connection |
| +// holds one open; on a box the agent holds the registry open, so it is |
| +// there already. The path is URI-escaped, so a # or a space in it is the |
| +// path's (a ? or a % the registry cannot take in any mode: its own |
| +// methods reopen by the path). |
| func OpenReadOnly(path string) (*Registry, error) { |
| fi, err := os.Stat(path) |
| if err != nil { |
| @@ -425,7 +432,12 @@ func OpenReadOnly(path string) (*Registry, error) { |
| if !fi.Mode().IsRegular() { |
| return nil, fmt.Errorf("registry %s: not a regular file", path) |
| } |
| - d, err := sql.Open("sqlite", "file:"+path+"?mode=ro") |
| + abs, err := filepath.Abs(path) |
| + if err != nil { |
| + return nil, err |
| + } |
| + u := url.URL{Scheme: "file", Path: abs, RawQuery: "mode=ro"} |
| + d, err := sql.Open("sqlite", u.String()) |
| if err != nil { |
| return nil, err |
| } |
diff --git a/internal/usage/durabilityadmissions.go b/internal/usage/durabilityadmissions.go| index e427c7d53..1435ba78c 100644 |
| --- a/internal/usage/durabilityadmissions.go |
| +++ b/internal/usage/durabilityadmissions.go |
| @@ -51,6 +51,7 @@ var admissionSchema = []string{ |
| fleet_id TEXT NOT NULL, |
| from_ns INTEGER NOT NULL, |
| to_ns INTEGER, |
| + discovered_at INTEGER NOT NULL DEFAULT 0, |
| reason TEXT NOT NULL DEFAULT '' |
| )`, |
| `CREATE TABLE IF NOT EXISTS inherited ( |
| @@ -141,7 +142,7 @@ func (l *DurabilityLedger) Admit(token, fleetID, account, app string, at time.Ti |
| l.mu.Lock() |
| defer l.mu.Unlock() |
| if l.closed { |
| - return errors.New("usage ledger: closed") |
| + return errLedgerClosed |
| } |
| tx, err := l.db.Begin() |
| if err != nil { |
| @@ -160,17 +161,29 @@ func (l *DurabilityLedger) Admit(token, fleetID, account, app string, at time.Ti |
| token, fleetID, account, app, nsOf(at)); err != nil { |
| return err |
| } |
| + // the barrier appears: the admitted day is a publication event |
| + if err := l.dirtyInterval(tx, at, at); err != nil { |
| + return err |
| + } |
| if admitFail != nil { |
| if err := admitFail(); err != nil { |
| return err |
| } |
| } |
| - return tx.Commit() |
| + if err := tx.Commit(); err != nil { |
| + return err |
| + } |
| + if admitAfterCommit != nil { |
| + return admitAfterCommit() // the commit landed and the caller hears an error: the ambiguous case, read back by the token |
| + } |
| + return nil |
| } |
| // admitFail is a test seam: an error here fails the admission's |
| // transaction before its commit, as a full disk would, after the insert. |
| -var admitFail func() error |
| +// admitAfterCommit is the other seam: the commit has landed and the |
| +// caller is told an error all the same (a lost response). |
| +var admitFail, admitAfterCommit func() error |
| // Unissued is the terminal of a committed row whose request was refused |
| // after all (the recheck found the fleet stale or a discovery begun, or |
| @@ -189,7 +202,7 @@ func (l *DurabilityLedger) terminal(token string, at time.Time, how string) erro |
| l.mu.Lock() |
| defer l.mu.Unlock() |
| if l.closed { |
| - return errors.New("usage ledger: closed") |
| + return errLedgerClosed |
| } |
| tx, err := l.db.Begin() |
| if err != nil { |
| @@ -202,7 +215,7 @@ func (l *DurabilityLedger) terminal(token string, at time.Time, how string) erro |
| } |
| if a.Settled() { |
| if a.How == how { |
| - return nil // the same terminal again: it landed |
| + return nil // the same terminal again, whatever instant the retry carries: it landed |
| } |
| return ErrAdmissionSettled |
| } |
| @@ -225,6 +238,10 @@ func (l *DurabilityLedger) terminal(token string, at time.Time, how string) erro |
| ns, ns, ns, how, token); err != nil { |
| return err |
| } |
| + // the barrier lifts: the days it barred are a publication event |
| + if err := l.dirtyInterval(tx, a.AdmittedAt, at); err != nil { |
| + return err |
| + } |
| a.How = how |
| if err := l.settled(tx, a, at); err != nil { // an empty settlement for an open uncertainty that counted it |
| return err |
| @@ -255,7 +272,7 @@ func (l *DurabilityLedger) RecorderStarted(token string, at time.Time) error { |
| l.mu.Lock() |
| defer l.mu.Unlock() |
| if l.closed { |
| - return errors.New("usage ledger: closed") |
| + return errLedgerClosed |
| } |
| tx, err := l.db.Begin() |
| if err != nil { |
| @@ -266,15 +283,12 @@ func (l *DurabilityLedger) RecorderStarted(token string, at time.Time) error { |
| if err != nil { |
| return err |
| } |
| + if !a.RecorderStartedAt.IsZero() { |
| + return nil // the same registration again, whatever instant the retry carries: the first stands |
| + } |
| if a.Settled() { |
| return ErrAdmissionSettled |
| } |
| - if !a.RecorderStartedAt.IsZero() { |
| - if a.RecorderStartedAt.Equal(at) { |
| - return nil // the same registration again |
| - } |
| - return ErrAdmissionParticipant |
| - } |
| if !a.RecorderDoneAt.IsZero() { |
| return ErrAdmissionParticipant |
| } |
| @@ -313,7 +327,7 @@ func (l *DurabilityLedger) participant(token string, at time.Time, which, how st |
| l.mu.Lock() |
| defer l.mu.Unlock() |
| if l.closed { |
| - return errors.New("usage ledger: closed") |
| + return errLedgerClosed |
| } |
| tx, err := l.db.Begin() |
| if err != nil { |
| @@ -324,12 +338,24 @@ func (l *DurabilityLedger) participant(token string, at time.Time, which, how st |
| if err != nil { |
| return err |
| } |
| - if a.Settled() { |
| - // the same completion again on a settled row is the write that |
| - // landed; anything else is refused |
| - if which == "transport" && a.TransportDoneAt.Equal(at) || which == "recorder" && a.RecorderDoneAt.Equal(at) && a.How == how { |
| + // a participant completes once: the same completion again (the same |
| + // participant, the same outcome, whatever instant the retry carries, |
| + // since the drain retries with a fresh now until it commits) is the |
| + // write that landed; a different outcome from the same participant is |
| + // refused |
| + if which == "transport" && !a.TransportDoneAt.IsZero() { |
| + return nil |
| + } |
| + if which == "recorder" && !a.RecorderDoneAt.IsZero() { |
| + if a.How == how { |
| return nil |
| } |
| + if a.Settled() { |
| + return ErrAdmissionSettled |
| + } |
| + return ErrAdmissionParticipant |
| + } |
| + if a.Settled() { |
| return ErrAdmissionSettled |
| } |
| if at.Before(a.AdmittedAt) { |
| @@ -338,23 +364,11 @@ func (l *DurabilityLedger) participant(token string, at time.Time, which, how st |
| ns := nsOf(at) |
| switch which { |
| case "transport": |
| - if !a.TransportDoneAt.IsZero() { |
| - if a.TransportDoneAt.Equal(at) { |
| - return nil |
| - } |
| - return ErrAdmissionParticipant |
| - } |
| a.TransportDoneAt = at |
| if _, err := tx.Exec(`UPDATE admissions SET transport_done_at = ? WHERE token = ?`, ns, token); err != nil { |
| return err |
| } |
| case "recorder": |
| - if !a.RecorderDoneAt.IsZero() { |
| - if a.RecorderDoneAt.Equal(at) && a.How == how { |
| - return nil |
| - } |
| - return ErrAdmissionParticipant |
| - } |
| if how != AdmissionUnrecorded && a.RecorderStartedAt.IsZero() { |
| return ErrAdmissionEdge // finished and abandoned are a registered worker's outcomes |
| } |
| @@ -378,6 +392,10 @@ func (l *DurabilityLedger) participant(token string, at time.Time, which, how st |
| if _, err := tx.Exec(`UPDATE admissions SET settled_at = ? WHERE token = ?`, nsOf(settled), token); err != nil { |
| return err |
| } |
| + // the barrier lifts: the days it barred are a publication event |
| + if err := l.dirtyInterval(tx, a.AdmittedAt, settled); err != nil { |
| + return err |
| + } |
| if err := l.settled(tx, a, settled); err != nil { |
| return err |
| } |
| @@ -452,7 +470,7 @@ func (l *DurabilityLedger) UnsettledAdmissions(fleetIDs ...string) ([]Admission, |
| } |
| tokens = append(tokens, t) |
| } |
| - if err := rows.Err(); err != nil { // a partial enumeration is no enumeration: the barrier and a discovery must not read it as nothing unsettled |
| + if err := rowsErr(rows); err != nil { // a partial enumeration is no enumeration: the barrier and a discovery must not read it as nothing unsettled |
| rows.Close() |
| return nil, err |
| } |
diff --git a/internal/usage/durabilityadmissions_test.go b/internal/usage/durabilityadmissions_test.go| index f73687c07..cb2c4cbfa 100644 |
| --- a/internal/usage/durabilityadmissions_test.go |
| +++ b/internal/usage/durabilityadmissions_test.go |
| @@ -2,6 +2,7 @@ package usage |
| import ( |
| "context" |
| + "database/sql" |
| "errors" |
| "strings" |
| "testing" |
| @@ -86,8 +87,8 @@ func TestAdmissionSettlesAtTheLaterParticipant(t *testing.T) { |
| if err := second(secondAt); err != nil { |
| t.Fatalf("the second participant's repeat: %v", err) |
| } |
| - if err := l.TransportDone(tok, time.Time{}, false); !errors.Is(err, ErrAdmissionSettled) { |
| - t.Fatalf("a transition on a settled row: %v", err) |
| + if err := l.TransportDone(tok, time.Time{}, false); err != nil { |
| + t.Fatalf("the transport's completion again on a settled row (the same participant, whatever the instant): %v", err) |
| } |
| if err := l.Unissued(tok, time.Time{}); !errors.Is(err, ErrAdmissionSettled) { |
| t.Fatalf("unissued after settlement: %v", err) |
| @@ -122,9 +123,15 @@ func TestAdmissionUnissuedAndParticipants(t *testing.T) { |
| if err := l.TransportDone(tok2, time.Time{}, false); err != nil { |
| t.Fatalf("the same completion again: %v", err) |
| } |
| + // a retry at another instant (the drain retries with a fresh now until |
| + // it commits) is the same completion: nil, and the first instant stands |
| + firstTransport := l.now().UTC() |
| advance(time.Minute) |
| - if err := l.TransportDone(tok2, time.Time{}, false); !errors.Is(err, ErrAdmissionParticipant) { |
| - t.Fatalf("the transport twice: %v", err) |
| + if err := l.TransportDone(tok2, time.Time{}, false); err != nil { |
| + t.Fatalf("the transport again at another instant: %v", err) |
| + } |
| + if a, _ := l.ReadAdmission(tok2); !a.TransportDoneAt.Equal(firstTransport) { |
| + t.Fatalf("the retry moved the transport's instant: %+v", a) |
| } |
| // unissued is an edge from admitted alone (the model's GateAnswers |
| // requires admitted and unanswered) |
| @@ -147,9 +154,13 @@ func TestAdmissionUnissuedAndParticipants(t *testing.T) { |
| if err := l.RecorderStarted(tok2, time.Time{}); err != nil { |
| t.Fatal(err) |
| } |
| + firstStart := l.now().UTC() |
| advance(time.Minute) |
| - if err := l.RecorderStarted(tok2, time.Time{}); !errors.Is(err, ErrAdmissionParticipant) { |
| - t.Fatalf("Start twice: %v", err) |
| + if err := l.RecorderStarted(tok2, time.Time{}); err != nil { |
| + t.Fatalf("Start again at another instant: %v", err) |
| + } |
| + if a, _ := l.ReadAdmission(tok2); !a.RecorderStartedAt.Equal(firstStart) { |
| + t.Fatalf("the retry moved the registration's instant: %+v", a) |
| } |
| if err := l.RecorderDone("nope", time.Time{}, AdmissionFinished); !errors.Is(err, ErrAdmissionUnknown) { |
| t.Fatalf("an unknown token: %v", err) |
| @@ -276,8 +287,8 @@ func TestAdmissionEdgesAndFleets(t *testing.T) { |
| t.Fatalf("unissued after the recorder registered: %v", err) |
| } |
| advance(time.Minute) |
| - if err := l.RecorderStarted(tok, time.Time{}); !errors.Is(err, ErrAdmissionParticipant) { |
| - t.Fatalf("a second registration: %v", err) |
| + if err := l.RecorderStarted(tok, time.Time{}); err != nil { |
| + t.Fatalf("the registration again at another instant: %v", err) |
| } |
| tok2 := mustAdmit(t, l, "fleet-1", "alice", "app", time.Time{}) |
| if err := l.RecorderDone(tok2, time.Time{}, AdmissionUnrecorded); err != nil { |
| @@ -286,10 +297,14 @@ func TestAdmissionEdgesAndFleets(t *testing.T) { |
| if err := l.RecorderStarted(tok2, time.Time{}); !errors.Is(err, ErrAdmissionParticipant) { |
| t.Fatalf("registered after the outcome: %v", err) |
| } |
| - // a second recorder completion with the transport still pending |
| + // a second recorder completion with the transport still pending: the |
| + // same outcome again is nil, another outcome is refused |
| advance(time.Minute) |
| - if err := l.RecorderDone(tok2, time.Time{}, AdmissionUnrecorded); !errors.Is(err, ErrAdmissionParticipant) { |
| - t.Fatalf("the recorder completed twice with the transport pending: %v", err) |
| + if err := l.RecorderDone(tok2, time.Time{}, AdmissionUnrecorded); err != nil { |
| + t.Fatalf("the same outcome again with the transport pending: %v", err) |
| + } |
| + if err := l.RecorderDone(tok2, time.Time{}, AdmissionAbandoned); !errors.Is(err, ErrAdmissionParticipant) { |
| + t.Fatalf("another outcome with the transport pending: %v", err) |
| } |
| if a, _ := l.ReadAdmission(tok2); a.Settled() { |
| t.Fatalf("settled with the transport pending: %+v", a) |
| @@ -377,9 +392,12 @@ func TestAdmissionInstantsAreClampedToNow(t *testing.T) { |
| if err := l.RecorderDone(tok, t2, AdmissionFinished); err != nil { |
| t.Fatal(err) |
| } |
| - if err := l.RecorderDone(tok, t1, AdmissionFinished); !errors.Is(err, ErrAdmissionParticipant) { |
| + if err := l.RecorderDone(tok, t1, AdmissionFinished); err != nil { |
| t.Fatalf("the outcome again at another instant: %v", err) |
| } |
| + if err := l.RecorderDone(tok, t1, AdmissionAbandoned); !errors.Is(err, ErrAdmissionParticipant) { |
| + t.Fatalf("another outcome: %v", err) |
| + } |
| if err := l.TransportDone(tok, t1, false); err != nil { |
| t.Fatal(err) |
| } |
| @@ -456,8 +474,66 @@ func TestAdmissionClosedTokensAndReadBack(t *testing.T) { |
| "started": func() error { return l.RecorderStarted(tok, time.Time{}) }, |
| "done": func() error { return l.RecorderDone(tok, time.Time{}, AdmissionUnrecorded) }, |
| } { |
| - if err := f(); err == nil { |
| - t.Fatalf("%s on a closed ledger", name) |
| + if err := f(); !errors.Is(err, errLedgerClosed) { |
| + t.Fatalf("%s on a closed ledger: %v, want the ledger's own refusal", name, err) |
| } |
| } |
| } |
| + |
| +// Review nine (Astra 10): the participants' maximum, not the last call's |
| +// instant. Instants later than now are allowed (a clock ahead of the |
| +// ledger's), so the clamp to now cannot mask the choice: the recorder |
| +// reports the later instant first, the transport the earlier one second, |
| +// and the settlement is the recorder's. |
| +func TestAdmissionSettlesAtTheParticipantsMaximum(t *testing.T) { |
| + l, _ := openAdmissionLedger(t) |
| + tok := mustAdmit(t, l, "fleet-1", "alice", "app", time.Time{}) |
| + now := l.now().UTC() |
| + later, earlier := now.Add(2*time.Hour), now.Add(time.Hour) |
| + if err := l.RecorderStarted(tok, now); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := l.RecorderDone(tok, later, AdmissionFinished); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := l.TransportDone(tok, earlier, false); err != nil { |
| + t.Fatal(err) |
| + } |
| + a, _ := l.ReadAdmission(tok) |
| + if !a.Settled() || !a.SettledAt.Equal(later) { |
| + t.Fatalf("settled at %v, want the later participant %v (last call %v, now %v)", a.SettledAt, later, earlier, now) |
| + } |
| +} |
| + |
| +// Review nine (Astra 10): an ambiguous commit that landed. The commit is |
| +// durable and the caller hears an error; the read-back by the token finds |
| +// the row, and the same Admit again is the write that landed. An |
| +// enumeration whose iteration fails is an error, never an empty list. |
| +func TestAdmissionAmbiguousLandedCommitAndEnumerationError(t *testing.T) { |
| + l, _ := openAdmissionLedger(t) |
| + tok, _ := NewAdmissionToken() |
| + at := l.now().UTC() |
| + lost := errors.New("response lost") |
| + admitAfterCommit = func() error { return lost } |
| + err := l.Admit(tok, "fleet-1", "alice", "app", at) |
| + admitAfterCommit = nil |
| + if !errors.Is(err, lost) { |
| + t.Fatalf("the lost response: %v", err) |
| + } |
| + if a, err := l.ReadAdmission(tok); err != nil || !a.AdmittedAt.Equal(at) { |
| + t.Fatalf("the landed commit is not read back: %+v %v", a, err) |
| + } |
| + if err := l.Admit(tok, "fleet-1", "alice", "app", at); err != nil { |
| + t.Fatalf("the same admission after the lost response: %v", err) |
| + } |
| + if open, _ := l.UnsettledAdmissions("fleet-1"); len(open) != 1 { |
| + t.Fatalf("unsettled: %+v", open) |
| + } |
| + boom := errors.New("iteration failed") |
| + rowsErr = func(*sql.Rows) error { return boom } |
| + _, err = l.UnsettledAdmissions("fleet-1") |
| + rowsErr = func(rows *sql.Rows) error { return rows.Err() } |
| + if !errors.Is(err, boom) { |
| + t.Fatalf("a failed enumeration: %v, want the error", err) |
| + } |
| +} |
diff --git a/internal/usage/durabilityholds_test.go b/internal/usage/durabilityholds_test.go| new file mode 100644 |
| index 000000000..ede5ac1fa |
| --- /dev/null |
| +++ b/internal/usage/durabilityholds_test.go |
| @@ -0,0 +1,414 @@ |
| +package usage |
| + |
| +import ( |
| + "context" |
| + "database/sql" |
| + "errors" |
| + "strings" |
| + "sync/atomic" |
| + "testing" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/release" |
| +) |
| + |
| +// finishAdmission settles an admission: the recorder registered and |
| +// finished, then the transport done. |
| +func finishAdmission(t *testing.T, l *DurabilityLedger, tok string) { |
| + t.Helper() |
| + if err := l.RecorderStarted(tok, time.Time{}); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := l.RecorderDone(tok, time.Time{}, AdmissionFinished); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := l.TransportDone(tok, time.Time{}, false); err != nil { |
| + t.Fatal(err) |
| + } |
| +} |
| + |
| +// Review nine (Opus P1-1, Astra 2): a hold, a latched discovery, an |
| +// admission and its settlement are publication events. A day published |
| +// complete is republished incomplete when a discovery's bound reaches it |
| +// (through a committed hold, and through the latch while the write has |
| +// failed), a closed day barred by an admission republishes complete when |
| +// it settles, and a hold written during a publication's put leaves the |
| +// day dirty for the next one. |
| +func TestHoldsAndBarriersRepublishTheirDays(t *testing.T) { |
| + ctx := context.Background() |
| + clock := newLedgerClock(ledgerDay0.Add(-time.Hour)) |
| + store := &countingStore{Store: release.NewDirStore(t.TempDir())} |
| + l := openTestLedger(t, t.TempDir(), "box1", clock, true) |
| + c, err := l.Bind(tenantAttr) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + day0, day2, day3 := DayOf(ledgerDay0), DayOf(ledgerDay0.Add(48*time.Hour)), DayOf(ledgerDay0.Add(72*time.Hour)) |
| + clock.set(ledgerDay0.Add(10 * time.Hour)) |
| + recordOne(t, c, clock, "PutObject", 200) |
| + clock.set(ledgerDay0.Add(24*time.Hour + time.Second)) |
| + if err := l.Seal(); err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, err := l.Publish(ctx, store); err != nil { |
| + t.Fatal(err) |
| + } |
| + if ok, why := readDurabilityDay(t, store, "box1", day0).Completeness(); !ok { |
| + t.Fatalf("precondition: day 0 complete before any hold: %s", why) |
| + } |
| + // a failed discovery on day 1 whose bound reaches day 0: latched, and |
| + // the latch alone republishes day 0 incomplete |
| + clock.set(ledgerDay0.Add(34 * time.Hour)) |
| + discoverFail = func() error { return errors.New("disk full") } |
| + _, err = l.Discover(Discovery{FleetID: "f", At: clock.now(), Proof: ledgerDay0.Add(23*time.Hour + 50*time.Minute), LastCleanLook: ledgerDay0.Add(33 * time.Hour)}) |
| + discoverFail = nil |
| + if err == nil { |
| + t.Fatal("the failed discovery did not say so") |
| + } |
| + if n, err := l.Publish(ctx, store); err != nil || n < 1 { |
| + t.Fatalf("publish with a latched discovery: n=%d err=%v", n, err) |
| + } |
| + if ok, why := readDurabilityDay(t, store, "box1", day0).Completeness(); ok || !strings.Contains(why, "unsettled") { |
| + t.Fatalf("day 0 under the latch: complete=%v %q", ok, why) |
| + } |
| + // the replay commits the hold: republished, now held |
| + if _, err := l.Replay("f"); err != nil { |
| + t.Fatal(err) |
| + } |
| + if n, err := l.Publish(ctx, store); err != nil || n < 1 { |
| + t.Fatalf("publish after the replay: n=%d err=%v", n, err) |
| + } |
| + if ok, why := readDurabilityDay(t, store, "box1", day0).Completeness(); ok || !strings.Contains(why, "hold") { |
| + t.Fatalf("day 0 under the hold: complete=%v %q", ok, why) |
| + } |
| + if n, _ := l.Publish(ctx, store); n != 0 { |
| + t.Fatalf("nothing changed and %d days republished", n) |
| + } |
| + // the barrier: an admission on day 2 (day 1 is held by the replayed |
| + // discovery, which closed on it), the day sealed and published |
| + // incomplete; its settlement republishes it complete |
| + clock.set(ledgerDay0.Add(60 * time.Hour)) |
| + tok := mustAdmit(t, l, "g", "alice", "app", time.Time{}) |
| + clock.set(ledgerDay0.Add(72*time.Hour + time.Second)) |
| + if err := l.Seal(); err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, err := l.Publish(ctx, store); err != nil { |
| + t.Fatal(err) |
| + } |
| + if ok, why := readDurabilityDay(t, store, "box1", day2).Completeness(); ok || !strings.Contains(why, "unsettled") { |
| + t.Fatalf("day 2 under the barrier: complete=%v %q", ok, why) |
| + } |
| + clock.set(ledgerDay0.Add(73 * time.Hour)) |
| + finishAdmission(t, l, tok) |
| + if n, err := l.Publish(ctx, store); err != nil || n < 1 { |
| + t.Fatalf("publish after the settlement: n=%d err=%v", n, err) |
| + } |
| + if ok, why := readDurabilityDay(t, store, "box1", day2).Completeness(); !ok { |
| + t.Fatalf("day 2 after the barrier lifted: %q", why) |
| + } |
| + // a hold written while a publication's put is in flight: the day's |
| + // sequence is above that publication's watermark, so the |
| + // acknowledgement leaves it dirty and the next publication carries |
| + // the hold |
| + clock.set(ledgerDay0.Add(82 * time.Hour)) |
| + recordOne(t, c, clock, "PutObject", 200) |
| + clock.set(ledgerDay0.Add(96*time.Hour + time.Second)) |
| + if err := l.Seal(); err != nil { |
| + t.Fatal(err) |
| + } |
| + raced := false |
| + store.beforePutHook = func() { |
| + if raced { |
| + return |
| + } |
| + raced = true |
| + if _, err := l.Discover(Discovery{FleetID: "h", At: clock.now(), Proof: ledgerDay0.Add(74 * time.Hour), LastCleanLook: ledgerDay0.Add(95 * time.Hour)}); err != nil { |
| + t.Error(err) |
| + } |
| + } |
| + if _, err := l.Publish(ctx, store); err != nil { |
| + t.Fatal(err) |
| + } |
| + store.beforePutHook = nil |
| + if !raced { |
| + t.Fatal("the put never ran") |
| + } |
| + if n, err := l.Publish(ctx, store); err != nil || n < 1 { |
| + t.Fatalf("the day a hold landed on during its put was not republished: n=%d err=%v", n, err) |
| + } |
| + if ok, why := readDurabilityDay(t, store, "box1", day3).Completeness(); ok || !strings.Contains(why, "hold") { |
| + t.Fatalf("day 3 after the racing hold: complete=%v %q", ok, why) |
| + } |
| +} |
| + |
| +// Review nine (Astra 1): a snapshot whose database read predates a |
| +// replay's commit and whose latch read follows it would see neither; the |
| +// discovery state's generation makes it read again. |
| +func TestSnapshotRetriesAcrossAReplay(t *testing.T) { |
| + l, set := openLedgerAt(t, at(2, 9, 0)) |
| + set(at(2, 23, 58)) |
| + tok := mustAdmit(t, l, "f", "alice", "app", time.Time{}) |
| + discoverFail = func() error { return errors.New("disk full") } |
| + _, err := l.Discover(Discovery{FleetID: "f", At: at(2, 23, 58), Proof: at(2, 23, 58), LastCleanLook: at(2, 23, 58)}) |
| + discoverFail = nil |
| + if err == nil { |
| + t.Fatal("the failed discovery did not say so") |
| + } |
| + set(at(3, 0, 5)) |
| + finishAdmission(t, l, tok) // nothing unsettled in the rows: the latch is the only bar |
| + replayed := false |
| + snapshotBeforeOverlay = func() { |
| + if replayed { |
| + return |
| + } |
| + replayed = true |
| + if _, err := l.Replay("f"); err != nil { |
| + t.Error(err) |
| + } |
| + } |
| + defer func() { snapshotBeforeOverlay = nil }() |
| + c, err := l.Snapshot("2026-10-02") |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if !replayed { |
| + t.Fatal("the replay never ran") |
| + } |
| + if c.Held == 0 { |
| + t.Fatalf("the snapshot across the replay read neither the latch nor the rows: %+v", c) |
| + } |
| +} |
| + |
| +// Review nine (Astra 3, Opus P2-1): a failed discovery keeps the bound and |
| +// counted set its transaction computed; one whose rows could not be read |
| +// is unresolved, bars every day, and the replay computes the bound before |
| +// anything clears it. |
| +func TestLatchKeepsTheComputedBoundOrStaysUnresolved(t *testing.T) { |
| + l, set := openLedgerAt(t, at(1, 23, 0)) |
| + a1 := mustAdmit(t, l, "f", "alice", "app", time.Time{}) |
| + set(at(2, 10, 0)) |
| + finishAdmission(t, l, a1) // settled after the bound: the historical term reaches its admission on the 1st |
| + set(at(2, 12, 0)) |
| + d := Discovery{FleetID: "f", At: at(2, 12, 0), Proof: at(2, 9, 0), LastCleanLook: at(2, 11, 0)} |
| + discoverFail = func() error { return errors.New("disk full") } |
| + _, err := l.Discover(d) |
| + discoverFail = nil |
| + if err == nil { |
| + t.Fatal("the failed discovery did not say so") |
| + } |
| + if from, ok := l.Latched("f"); !ok || !from.Equal(at(1, 22, 59)) { |
| + t.Fatalf("the latch lost the computed bound: %v %v (the fallback would be %v)", from, ok, at(2, 8, 59)) |
| + } |
| + if c, _ := l.Snapshot("2026-10-01"); c.Barred == 0 { |
| + t.Fatal("the latched bound does not bar the 1st") |
| + } |
| + // the rows could not be read: unresolved, every day barred, the bound |
| + // computed at the replay |
| + l2, set2 := openLedgerAt(t, at(1, 23, 0)) |
| + b1 := mustAdmit(t, l2, "g", "alice", "app", time.Time{}) |
| + set2(at(2, 10, 0)) |
| + finishAdmission(t, l2, b1) |
| + set2(at(2, 12, 0)) |
| + rowsErr = func(*sql.Rows) error { return errors.New("read failed") } |
| + _, err = l2.Discover(Discovery{FleetID: "g", At: at(2, 12, 0), Proof: at(2, 9, 0), LastCleanLook: at(2, 11, 0)}) |
| + rowsErr = func(rows *sql.Rows) error { return rows.Err() } |
| + if err == nil { |
| + t.Fatal("a discovery whose rows could not be read reported as committed") |
| + } |
| + if from, ok := l2.Latched("g"); !ok || !from.IsZero() { |
| + t.Fatalf("an unreadable bound latched as resolved: %v %v", from, ok) |
| + } |
| + if c, _ := l2.Snapshot("2026-09-20"); c.Barred == 0 { |
| + t.Fatal("an unresolved latch does not bar a day before any bound") |
| + } |
| + set2(at(2, 12, 30)) |
| + id, err := l2.Replay("g") |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + u, _ := l2.ReadUncertainty(id) |
| + if !u.From.Equal(at(1, 22, 59)) || u.Open() || !u.To.Equal(at(2, 12, 31)) || !u.DiscoveredAt.Equal(at(2, 12, 0)) { |
| + t.Fatalf("the replay of an unresolved latch: %+v", u) |
| + } |
| + held(t, l2, true, "2026-10-01", "2026-10-02") |
| + if c, _ := l2.Snapshot("2026-09-20"); c.Barred != 0 { |
| + t.Fatal("the replayed latch still bars") |
| + } |
| +} |
| + |
| +// Review nine (Astra 5, Opus P2-1): the replay closes on the same test as |
| +// a settlement, every unsettled row of the fleet and the pending count, |
| +// not the set counted at the discovery: a registration pending at the |
| +// failed discovery that commits before the replay keeps it open; with |
| +// everything settled the replay closes at the latest of the discovery and |
| +// the replay plus the margin; a replay whose transaction fails leaves the |
| +// latch. |
| +func TestReplayClosesOnEveryUnsettledRow(t *testing.T) { |
| + l, set := openLedgerAt(t, at(2, 23, 50)) |
| + var pending atomic.Int64 |
| + pending.Store(1) |
| + l.SetPendingCounter("f", &pending) |
| + discoverFail = func() error { return errors.New("disk full") } |
| + _, err := l.Discover(Discovery{FleetID: "f", At: at(2, 23, 50), Proof: at(2, 23, 0), LastCleanLook: at(2, 23, 40), Pending: []time.Time{at(2, 23, 45)}}) |
| + discoverFail = nil |
| + if err == nil { |
| + t.Fatal("the failed discovery did not say so") |
| + } |
| + if from, ok := l.Latched("f"); !ok || !from.Equal(at(2, 22, 59)) { |
| + t.Fatalf("latched %v %v", from, ok) |
| + } |
| + // the registration commits its row and leaves the pending set |
| + set(at(2, 23, 55)) |
| + tok := mustAdmit(t, l, "f", "alice", "app", at(2, 23, 45)) |
| + pending.Store(0) |
| + set(at(3, 0, 10)) |
| + id, err := l.Replay("f") |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if u, _ := l.ReadUncertainty(id); !u.Open() { |
| + t.Fatalf("the replay closed over an unsettled row it had not counted: %+v", u) |
| + } |
| + if err := l.Unissued(tok, time.Time{}); err != nil { |
| + t.Fatal(err) |
| + } |
| + if u, _ := l.ReadUncertainty(id); u.Open() || !u.To.Equal(at(3, 0, 11)) { |
| + t.Fatalf("after the empty settlement: %+v", u) |
| + } |
| + // everything settled before the replay: closed by the replay itself, |
| + // no earlier than the discovery plus the margin |
| + l2, set2 := openLedgerAt(t, at(2, 23, 50)) |
| + tok2 := mustAdmit(t, l2, "g", "alice", "app", time.Time{}) |
| + discoverFail = func() error { return errors.New("disk full") } |
| + l2.Discover(Discovery{FleetID: "g", At: at(2, 23, 59), Proof: at(2, 23, 0), LastCleanLook: at(2, 23, 40)}) |
| + discoverFail = nil |
| + set2(at(2, 23, 59)) |
| + finishAdmission(t, l2, tok2) |
| + replayFail = func() error { return errors.New("disk full") } |
| + _, err = l2.Replay("g") |
| + replayFail = nil |
| + if err == nil { |
| + t.Fatal("a replay whose transaction failed returned nil") |
| + } |
| + if _, ok := l2.Latched("g"); !ok { |
| + t.Fatal("the latch was cleared by a replay that did not commit") |
| + } |
| + id2, err := l2.Replay("g") |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if u, _ := l2.ReadUncertainty(id2); u.Open() || !u.To.Equal(at(3, 0, 0)) { |
| + t.Fatalf("the replay with everything settled: %+v (want closed at the discovery plus the margin)", u) |
| + } |
| + held(t, l2, true, "2026-10-02", "2026-10-03") |
| + if _, ok := l2.Latched("g"); ok { |
| + t.Fatal("the latch survived the commit") |
| + } |
| +} |
| + |
| +// Review nine (Opus P3-2, P3-3, P3-6; Astra 7): a discovery with nothing |
| +// counted and one pending registration stays open; a zero bound term is |
| +// absent and both absent is refused without latching; a latched bound bars |
| +// no day before it; the closure's floor is the discovery, whatever |
| +// earlier instant a settlement reports; a second discovery under an open |
| +// uncertainty is answered with it, and one under a latch replays it; a |
| +// closed ledger refuses with its own error. |
| +func TestDiscoveryEdges(t *testing.T) { |
| + l, set := openLedgerAt(t, at(3, 23, 59)) |
| + var pending atomic.Int64 |
| + pending.Store(1) |
| + l.SetPendingCounter("f", &pending) |
| + id, err := l.Discover(Discovery{FleetID: "f", At: at(3, 23, 59), Proof: time.Time{}, LastCleanLook: at(3, 23, 0), Pending: []time.Time{at(3, 23, 30)}}) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + u, _ := l.ReadUncertainty(id) |
| + if !u.Open() || !u.From.Equal(at(3, 22, 59)) { |
| + t.Fatalf("with a registration pending: %+v (a zero proof is absent)", u) |
| + } |
| + if again, err := l.Discover(Discovery{FleetID: "f", At: at(4, 0, 30), LastCleanLook: at(3, 23, 0)}); err != nil || again != id { |
| + t.Fatalf("a second discovery under an open uncertainty: %d %v, want %d", again, err, id) |
| + } |
| + if _, err := l.Discover(Discovery{FleetID: "z", At: at(3, 23, 59)}); !errors.Is(err, ErrDiscoveryBound) { |
| + t.Fatalf("a discovery with no bound term: %v", err) |
| + } |
| + if _, ok := l.Latched("z"); ok { |
| + t.Fatal("a refused discovery latched") |
| + } |
| + // the floor: the registration cancels, reported at an earlier instant |
| + // than the discovery; the uncertainty closes at the discovery plus the |
| + // margin, which is the 4th |
| + pending.Store(0) |
| + set(at(3, 23, 59)) |
| + if err := l.EmptySettlement("f", at(3, 23, 0)); err != nil { |
| + t.Fatal(err) |
| + } |
| + if u, _ := l.ReadUncertainty(id); u.Open() || !u.To.Equal(at(4, 0, 0)) { |
| + t.Fatalf("closed before the discovery plus the margin: %+v", u) |
| + } |
| + held(t, l, true, "2026-10-03", "2026-10-04") |
| + // a latched bound bars from its day on, not before |
| + l2, _ := openLedgerAt(t, at(3, 12, 0)) |
| + discoverFail = func() error { return errors.New("disk full") } |
| + l2.Discover(Discovery{FleetID: "g", At: at(3, 12, 0), Proof: at(3, 9, 0), LastCleanLook: at(3, 11, 0)}) |
| + discoverFail = nil |
| + if c, _ := l2.Snapshot("2026-10-02"); c.Barred != 0 { |
| + t.Fatal("the latch bars a day before its bound") |
| + } |
| + if c, _ := l2.Snapshot("2026-10-03"); c.Barred == 0 { |
| + t.Fatal("the latch does not bar its bound's day") |
| + } |
| + // a discovery under the latch replays it: one row, the latch cleared |
| + rid, err := l2.Discover(Discovery{FleetID: "g", At: at(3, 12, 30), Proof: at(3, 9, 0), LastCleanLook: at(3, 11, 0)}) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, ok := l2.Latched("g"); ok { |
| + t.Fatal("the latch survived the discovery that replayed it") |
| + } |
| + if u, _ := l2.ReadUncertainty(rid); !u.DiscoveredAt.Equal(at(3, 12, 0)) || !u.From.Equal(at(3, 8, 59)) { |
| + t.Fatalf("the replayed discovery: %+v", u) |
| + } |
| + if err := l2.Close(t.Context(), nil); err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, err := l2.Discover(Discovery{FleetID: "g", At: at(3, 13, 0), Proof: at(3, 9, 0)}); !errors.Is(err, errLedgerClosed) { |
| + t.Fatalf("discover on a closed ledger: %v", err) |
| + } |
| + if _, err := l2.Replay("g"); err != nil { // nothing latched: nil before the closed check is reached |
| + t.Fatal(err) |
| + } |
| + if err := l2.EmptySettlement("g", time.Time{}); !errors.Is(err, errLedgerClosed) { |
| + t.Fatalf("empty settlement on a closed ledger: %v", err) |
| + } |
| +} |
| + |
| +// Review nine (Astra 10): a settlement under an open uncertainty writes |
| +// its own interval while another admission keeps the uncertainty open. |
| +func TestSettlementWritesItsOwnInterval(t *testing.T) { |
| + l, set := openLedgerAt(t, at(1, 23, 0)) |
| + a1 := mustAdmit(t, l, "f", "alice", "app", time.Time{}) |
| + set(at(2, 10, 0)) |
| + a2 := mustAdmit(t, l, "f", "alice", "app", time.Time{}) |
| + set(at(2, 12, 0)) |
| + id, err := l.Discover(Discovery{FleetID: "f", At: at(2, 12, 0), Proof: at(2, 9, 0), LastCleanLook: at(2, 11, 0)}) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + held(t, l, true, "2026-10-01", "2026-10-02") |
| + held(t, l, false, "2026-10-03") |
| + set(at(3, 1, 0)) |
| + finishAdmission(t, l, a1) |
| + if u, _ := l.ReadUncertainty(id); !u.Open() { |
| + t.Fatalf("closed with a2 unsettled: %+v", u) |
| + } |
| + held(t, l, true, "2026-10-03") // a1's own interval reaches the 3rd |
| + held(t, l, false, "2026-10-04") |
| + set(at(4, 1, 0)) |
| + finishAdmission(t, l, a2) |
| + if u, _ := l.ReadUncertainty(id); u.Open() || !u.To.Equal(at(4, 1, 1)) { |
| + t.Fatalf("%+v", u) |
| + } |
| + held(t, l, true, "2026-10-04") |
| +} |
diff --git a/internal/usage/durabilityledger.go b/internal/usage/durabilityledger.go| index 4b3f77a74..6eaa44bf0 100644 |
| --- a/internal/usage/durabilityledger.go |
| +++ b/internal/usage/durabilityledger.go |
| @@ -1813,10 +1813,14 @@ func (l *DurabilityLedger) Snapshot(date string) (DurabilityContent, error) { |
| func (l *DurabilityLedger) snapshot(ctx context.Context, date string) (DurabilityContent, dayLineage, error) { |
| for range 8 { |
| _, before := l.latch.pendingAt(date) |
| + dbefore := l.disc().gen() |
| content, lineage, err := l.snapshotOnce(ctx, date, before, false) |
| if errors.Is(err, errSnapshotLatchChanged) { |
| continue |
| } |
| + if err == nil && l.disc().gen() != dbefore { |
| + continue // a replay committed across the read: its rows or its latch, never neither |
| + } |
| if err != nil { |
| return DurabilityContent{}, dayLineage{}, err |
| } |
| @@ -1828,15 +1832,23 @@ func (l *DurabilityLedger) snapshot(ctx context.Context, date string) (Durabilit |
| l.drainMu.Lock() |
| defer l.drainMu.Unlock() |
| _, before := l.latch.pendingAt(date) |
| + dbefore := l.disc().gen() |
| content, lineage, err := l.snapshotOnce(ctx, date, before, true) |
| if errors.Is(err, errSnapshotLatchChanged) { |
| return DurabilityContent{}, dayLineage{}, errors.New("usage ledger: failure drain changed during locked snapshot") |
| } |
| + if err == nil && l.disc().gen() != dbefore { |
| + return DurabilityContent{}, dayLineage{}, errors.New("usage ledger: discovery replay changed during locked snapshot") |
| + } |
| return content, lineage, err |
| } |
| var errSnapshotLatchChanged = errors.New("usage ledger: failure drain crossed snapshot") |
| +// snapshotBeforeOverlay is a test seam between a snapshot's database |
| +// reads and its read of the discovery latch. |
| +var snapshotBeforeOverlay func() |
| + |
| // snapshotOnce builds the content and reads the day's publication lineage in |
| // ONE read transaction on the reader connection: the watermark is the |
| // ledger's next_seq as that transaction sees it, so every row with a |
| @@ -2011,6 +2023,9 @@ func (l *DurabilityLedger) snapshotOnce(ctx context.Context, date string, before |
| if err := tx.QueryRow(`SELECT COUNT(*) FROM holds WHERE day = ?`, date).Scan(&c.Held); err != nil { |
| return DurabilityContent{}, dayLineage{}, err |
| } |
| + if snapshotBeforeOverlay != nil { |
| + snapshotBeforeOverlay() |
| + } |
| c.Barred += l.latchedBarring(dayEnd) |
| } |
| if err := tx.Commit(); err != nil { |
diff --git a/internal/usage/durabilitypublish.go b/internal/usage/durabilitypublish.go| index a966b2194..80af1705d 100644 |
| --- a/internal/usage/durabilitypublish.go |
| +++ b/internal/usage/durabilitypublish.go |
| @@ -103,6 +103,21 @@ func (l *DurabilityLedger) publish(ctx context.Context, store release.Store) (in |
| dates = append(dates, d) |
| } |
| } |
| + // A latched discovery's days likewise: from its bound (or, unresolved, |
| + // from the earliest day this ledger has) through today, so the bucket |
| + // never reads complete over a discovery the process already knows of. |
| + floor := l.now().UTC() |
| + if len(dates) > 0 { |
| + if f, _, err := DayBounds(dates[0]); err == nil { |
| + floor = f |
| + } |
| + } |
| + for _, d := range l.latchedDates(floor, l.now()) { |
| + if !seen[d] { |
| + seen[d] = true |
| + dates = append(dates, d) |
| + } |
| + } |
| sort.Strings(dates) |
| n := 0 |
| for _, date := range dates { |
diff --git a/internal/usage/durabilityuncertainty.go b/internal/usage/durabilityuncertainty.go| index 01f1fb7f5..532e7c564 100644 |
| --- a/internal/usage/durabilityuncertainty.go |
| +++ b/internal/usage/durabilityuncertainty.go |
| @@ -5,6 +5,7 @@ import ( |
| "errors" |
| "fmt" |
| "sync" |
| + "sync/atomic" |
| "time" |
| ) |
| @@ -12,20 +13,31 @@ import ( |
| // found changed, from the design's fixed bound and the counted admissions, |
| // held for the fleet's owners (box-wide here: every hold row names the box, |
| // decision 10 being a parameter), open until every counted admission is |
| -// settled, then closed at the latest settlement plus the margin. A |
| -// discovery whose transaction fails is latched in memory, bars publication |
| -// from its bound, and is replayed as one transaction. Mirrors |
| -// internal/ops/admissionmodel's Discover, Replay, settle and maybeClose. |
| +// settled, then closed at the latest of the discovery and the settlements |
| +// plus the margin. A discovery whose transaction fails is latched in |
| +// memory, bars publication from its bound, and is replayed as one |
| +// transaction. Every hold written and every settlement dirties the days it |
| +// touches with a sequence above any snapshot's watermark, so publication |
| +// revisits them; a latched discovery's days are enumerated by publication |
| +// as the failure latch's are. Mirrors internal/ops/admissionmodel's |
| +// Discover, Replay, settle and maybeClose. |
| // AdmissionMargin is the design's margin around a bound. |
| const AdmissionMargin = time.Minute |
| +// errLedgerClosed is every Part C write's answer on a closed ledger, |
| +// distinct from the database's own refusal. |
| +var errLedgerClosed = errors.New("usage ledger: closed") |
| + |
| +// ErrDiscoveryBound is a discovery with neither lower-bound term. |
| +var ErrDiscoveryBound = errors.New("usage ledger: a discovery needs a proof instant or a last clean look") |
| + |
| // Discovery is what a fleet found changed, as the ledger needs it. |
| type Discovery struct { |
| FleetID string |
| At time.Time // the discovery instant |
| // Proof and LastCleanLook are the fleet's two lower-bound terms; the |
| - // bound is the earlier. |
| + // bound is the earlier. A zero term is absent; both absent is refused. |
| Proof, LastCleanLook time.Time |
| // Pending is the admission instant of every registration in the |
| // fleet's in-memory pending set (registered, not yet committed): each |
| @@ -36,44 +48,62 @@ type Discovery struct { |
| // Uncertainty is one row, read back. |
| type Uncertainty struct { |
| - ID int64 |
| - FleetID string |
| - From time.Time |
| - To time.Time // zero while open |
| - Reason string |
| + ID int64 |
| + FleetID string |
| + From time.Time |
| + To time.Time // zero while open |
| + DiscoveredAt time.Time |
| + Reason string |
| } |
| // Open reports whether the uncertainty is open. |
| func (u Uncertainty) Open() bool { return u.To.IsZero() } |
| // pendingDiscovery is a discovery whose transaction failed: latched in |
| -// memory, barring every day from its bound until Replay commits. |
| +// memory, barring every day from its bound until Replay commits. One |
| +// whose bound could not be read either is unresolved: it bars every day, |
| +// and the replay computes the bound before anything clears it. |
| type pendingDiscovery struct { |
| - d Discovery |
| - from time.Time |
| - counted map[string]bool // the tokens counted at the discovery |
| + d Discovery |
| + from time.Time |
| + counted map[string]bool // the tokens counted at the discovery |
| + unresolved bool |
| } |
| // discoveryState is the ledger's in-memory side of Part C: the latched |
| // discoveries by fleet and each open fleet's pending-registration counter |
| // (the closure test ranges over rows and pending registrations alike). |
| +// generation changes whenever the latched set changes, so a snapshot |
| +// whose database read predates a replay's commit retries rather than |
| +// reading neither the latch nor the rows. |
| type discoveryState struct { |
| - mu sync.Mutex |
| - latched map[string]*pendingDiscovery |
| - pending map[string]func() int |
| + mu sync.Mutex |
| + generation uint64 |
| + latched map[string]*pendingDiscovery |
| + pending map[string]*atomic.Int64 |
| } |
| func (l *DurabilityLedger) disc() *discoveryState { |
| l.discOnce.Do(func() { |
| - l.discovery = &discoveryState{latched: map[string]*pendingDiscovery{}, pending: map[string]func() int{}} |
| + l.discovery = &discoveryState{latched: map[string]*pendingDiscovery{}, pending: map[string]*atomic.Int64{}} |
| }) |
| return l.discovery |
| } |
| +func (ds *discoveryState) gen() uint64 { |
| + ds.mu.Lock() |
| + defer ds.mu.Unlock() |
| + return ds.generation |
| +} |
| + |
| // SetPendingCounter tells the ledger how many registrations the fleet |
| // holds in memory for a fleet: an open uncertainty of that fleet does not |
| -// close while any remains. nil clears it. |
| -func (l *DurabilityLedger) SetPendingCounter(fleetID string, n func() int) { |
| +// close while any remains. The counter is lock-free on purpose: it is read |
| +// under the ledger's mutex, and the fleet's discovery mutex may be held by |
| +// a caller that then writes to the ledger (the gate's recheck writing |
| +// unissued), so a counter that took that mutex would deadlock. nil clears |
| +// it. |
| +func (l *DurabilityLedger) SetPendingCounter(fleetID string, n *atomic.Int64) { |
| ds := l.disc() |
| ds.mu.Lock() |
| defer ds.mu.Unlock() |
| @@ -84,7 +114,7 @@ func (l *DurabilityLedger) SetPendingCounter(fleetID string, n func() int) { |
| ds.pending[fleetID] = n |
| } |
| -func (l *DurabilityLedger) pendingCount(fleetID string) int { |
| +func (l *DurabilityLedger) pendingCount(fleetID string) int64 { |
| ds := l.disc() |
| ds.mu.Lock() |
| n := ds.pending[fleetID] |
| @@ -92,7 +122,7 @@ func (l *DurabilityLedger) pendingCount(fleetID string) int { |
| if n == nil { |
| return 0 |
| } |
| - return n() |
| + return n.Load() |
| } |
| func minTime(a, b time.Time) time.Time { |
| @@ -102,6 +132,28 @@ func minTime(a, b time.Time) time.Time { |
| return a |
| } |
| +func maxTime(a, b time.Time) time.Time { |
| + if b.After(a) { |
| + return b |
| + } |
| + return a |
| +} |
| + |
| +// fixedBound is the earlier of the discovery's two terms, a zero term |
| +// being absent. |
| +func fixedBound(d Discovery) (time.Time, error) { |
| + p, c := d.Proof.UTC(), d.LastCleanLook.UTC() |
| + switch { |
| + case d.Proof.IsZero() && d.LastCleanLook.IsZero(): |
| + return time.Time{}, ErrDiscoveryBound |
| + case d.Proof.IsZero(): |
| + return c, nil |
| + case d.LastCleanLook.IsZero(): |
| + return p, nil |
| + } |
| + return minTime(p, c), nil |
| +} |
| + |
| // bound is the design's fixed bound and lower bound for a discovery: the |
| // earlier of the proof and the last clean look's start; then the earliest |
| // admission instant among the fleet's rows that are unsettled, or settled |
| @@ -109,7 +161,10 @@ func minTime(a, b time.Time) time.Time { |
| // registrations; minus the margin. The cutoff is the bound, fixed before |
| // the rows are read. Also returns the tokens counted (unsettled rows). |
| func (l *DurabilityLedger) bound(tx *sql.Tx, d Discovery) (from time.Time, counted map[string]bool, err error) { |
| - b := minTime(d.Proof.UTC(), d.LastCleanLook.UTC()) |
| + b, err := fixedBound(d) |
| + if err != nil { |
| + return time.Time{}, nil, err |
| + } |
| from = b |
| counted = map[string]bool{} |
| rows, err := tx.Query(`SELECT token, admitted_at, settled_at, how FROM admissions WHERE fleet_id = ?`, d.FleetID) |
| @@ -134,7 +189,7 @@ func (l *DurabilityLedger) bound(tx *sql.Tx, d Discovery) (from time.Time, count |
| from = at |
| } |
| } |
| - if err := rows.Err(); err != nil { |
| + if err := rowsErr(rows); err != nil { |
| return time.Time{}, nil, err |
| } |
| for _, p := range d.Pending { |
| @@ -145,100 +200,182 @@ func (l *DurabilityLedger) bound(tx *sql.Tx, d Discovery) (from time.Time, count |
| return from.Add(-AdmissionMargin), counted, nil |
| } |
| -// writeHolds writes the box-wide hold rows for every day of [from, to]. |
| -func writeHolds(tx *sql.Tx, from, to time.Time, uncertainty int64, at time.Time) error { |
| - for d := from.UTC().Truncate(24 * time.Hour); !d.After(to.UTC()); d = d.Add(24 * time.Hour) { |
| +// rowsErr is rows.Err behind a test seam. |
| +var rowsErr = func(rows *sql.Rows) error { return rows.Err() } |
| + |
| +// holdsMaxDays bounds one hold write: an interval longer than this is an |
| +// error (the write fails, the discovery latches, and the latch bars |
| +// conservatively) rather than a transaction looping over centuries. |
| +const holdsMaxDays = 4000 |
| + |
| +// writeHolds writes the box-wide hold rows for every day of [from, to] and |
| +// dirties each day with a fresh sequence, so the publisher revisits a day |
| +// already published complete. Caller holds l.mu; the sequence is |
| +// persisted with the transaction. |
| +func (l *DurabilityLedger) writeHolds(tx *sql.Tx, from, to time.Time, uncertainty int64, at time.Time) error { |
| + from, to = from.UTC(), to.UTC() |
| + if to.Before(from) { |
| + from, to = to, from |
| + } |
| + if to.Sub(from) > holdsMaxDays*24*time.Hour { |
| + return fmt.Errorf("usage ledger: a hold over %s to %s is longer than %d days", from.Format(time.RFC3339), to.Format(time.RFC3339), holdsMaxDays) |
| + } |
| + seq := l.allocSeq() |
| + for d := from.Truncate(24 * time.Hour); !d.After(to); d = d.Add(24 * time.Hour) { |
| if _, err := tx.Exec(`INSERT OR IGNORE INTO holds (day, account, app, uncertainty_id, written_at) VALUES (?, '', '', ?, ?)`, DayOf(d), uncertainty, nsOf(at)); err != nil { |
| return err |
| } |
| + if err := touchDay(tx, DayOf(d), seq); err != nil { |
| + return err |
| + } |
| } |
| - return nil |
| + return l.persistSeq(tx) |
| +} |
| + |
| +// dirtyInterval dirties every day of [from, to] with a fresh sequence: |
| +// the barrier appearing (an admission) and lifting (its settlement) are |
| +// publication events. Caller holds l.mu. |
| +func (l *DurabilityLedger) dirtyInterval(tx *sql.Tx, from, to time.Time) error { |
| + if err := touchDays(tx, from, to, l.allocSeq()); err != nil { |
| + return err |
| + } |
| + return l.persistSeq(tx) |
| } |
| // Discover writes the uncertainty and its first holds in one transaction: |
| // the fleet's authority was found changed at d.At. With nothing counted |
| // and nothing pending, it is opened and closed at once at d.At plus the |
| -// margin. If the transaction fails the discovery is latched (Latched, |
| -// Replay) and the error returned; the caller refuses requests meanwhile. |
| +// margin. A fleet with a latched discovery has that replayed in this |
| +// transaction instead (one open uncertainty per fleet), and a fleet whose |
| +// uncertainty is already open is answered with it. If the transaction |
| +// fails the discovery is latched (Latched, Replay) with the bound and the |
| +// counted set it computed, or unresolved when even those could not be |
| +// read, and the error returned; the caller refuses requests meanwhile. |
| // Mirrors admissionmodel.Discover. |
| func (l *DurabilityLedger) Discover(d Discovery) (int64, error) { |
| d.At = d.At.UTC() |
| if d.At.IsZero() { |
| d.At = l.now().UTC() |
| } |
| + if _, err := fixedBound(d); err != nil { |
| + return 0, err // refused, not latched: nothing was found that a bound could describe |
| + } |
| + l.drainMu.Lock() |
| + defer l.drainMu.Unlock() |
| l.mu.Lock() |
| defer l.mu.Unlock() |
| - id, err := l.discover(d) |
| + if l.closed { |
| + return 0, errLedgerClosed |
| + } |
| + ds := l.disc() |
| + ds.mu.Lock() |
| + pd := ds.latched[d.FleetID] |
| + ds.mu.Unlock() |
| + if pd != nil { |
| + id, err := l.replayLocked(d.FleetID, pd, d.At) |
| + if err != nil { |
| + return 0, err |
| + } |
| + return id, nil |
| + } |
| + id, from, counted, err := l.discover(d) |
| if err != nil { |
| - l.latchDiscovery(d) |
| + l.latchDiscovery(d, from, counted) |
| } |
| return id, err |
| } |
| -func (l *DurabilityLedger) discover(d Discovery) (int64, error) { |
| +// discover is the discovery's transaction. On failure it returns the |
| +// bound and counted set it computed, zero and nil when it did not get |
| +// that far. |
| +func (l *DurabilityLedger) discover(d Discovery) (int64, time.Time, map[string]bool, error) { |
| tx, err := l.db.Begin() |
| if err != nil { |
| - return 0, err |
| + return 0, time.Time{}, nil, err |
| } |
| defer tx.Rollback() //nolint:errcheck |
| + if id, _, ok, err := openUncertainty(tx, d.FleetID); err != nil { |
| + return 0, time.Time{}, nil, err |
| + } else if ok { |
| + return id, time.Time{}, nil, nil // already stale under an open uncertainty: the same one |
| + } |
| from, counted, err := l.bound(tx, d) |
| if err != nil { |
| - return 0, err |
| + return 0, time.Time{}, nil, err |
| } |
| - res, err := tx.Exec(`INSERT INTO uncertainties (fleet_id, from_ns, to_ns, reason) VALUES (?, ?, NULL, ?)`, d.FleetID, nsOf(from), d.Reason) |
| + res, err := tx.Exec(`INSERT INTO uncertainties (fleet_id, from_ns, to_ns, discovered_at, reason) VALUES (?, ?, NULL, ?, ?)`, d.FleetID, nsOf(from), nsOf(d.At), d.Reason) |
| if err != nil { |
| - return 0, err |
| + return 0, from, counted, err |
| } |
| id, err := res.LastInsertId() |
| if err != nil { |
| - return 0, err |
| + return 0, from, counted, err |
| + } |
| + if err := markStale(tx, d.FleetID); err != nil { |
| + return 0, from, counted, err |
| } |
| to := d.At |
| closeNow := len(counted) == 0 && len(d.Pending) == 0 |
| if closeNow { |
| to = d.At.Add(AdmissionMargin) |
| if _, err := tx.Exec(`UPDATE uncertainties SET to_ns = ? WHERE id = ?`, nsOf(to), id); err != nil { |
| - return 0, err |
| + return 0, from, counted, err |
| } |
| } |
| - if err := writeHolds(tx, from, to, id, d.At); err != nil { |
| - return 0, err |
| + if err := l.writeHolds(tx, from, to, id, d.At); err != nil { |
| + return 0, from, counted, err |
| } |
| if discoverFail != nil { |
| if err := discoverFail(); err != nil { |
| - return 0, err |
| + return 0, from, counted, err |
| } |
| } |
| if err := tx.Commit(); err != nil { |
| - return 0, err |
| + return 0, from, counted, err |
| } |
| - return id, nil |
| + return id, from, counted, nil |
| +} |
| + |
| +// openUncertainty is the fleet's open uncertainty, if any. |
| +func openUncertainty(tx *sql.Tx, fleetID string) (id int64, from time.Time, ok bool, err error) { |
| + var fromNs int64 |
| + err = tx.QueryRow(`SELECT id, from_ns FROM uncertainties WHERE fleet_id = ? AND to_ns IS NULL ORDER BY id LIMIT 1`, fleetID).Scan(&id, &fromNs) |
| + if errors.Is(err, sql.ErrNoRows) { |
| + return 0, time.Time{}, false, nil |
| + } |
| + if err != nil { |
| + return 0, time.Time{}, false, err |
| + } |
| + return id, time.Unix(0, fromNs).UTC(), true, nil |
| } |
| // discoverFail is a test seam: an error returned here fails the |
| // discovery's transaction before its commit, as a full disk would. |
| var discoverFail func() error |
| -func (l *DurabilityLedger) latchDiscovery(d Discovery) { |
| - // the counted set and the bound are computed again outside a failed |
| - // transaction, from what the rows say now; a read that fails leaves |
| - // the bound at the earlier of the two terms, which is never later |
| - b := minTime(d.Proof.UTC(), d.LastCleanLook.UTC()).Add(-AdmissionMargin) |
| - pd := &pendingDiscovery{d: d, from: b, counted: map[string]bool{}} |
| - if tx, err := l.db.Begin(); err == nil { |
| - if from, counted, err := l.bound(tx, d); err == nil { |
| - pd.from, pd.counted = from, counted |
| - } |
| - tx.Rollback() //nolint:errcheck |
| +// replayFail is a test seam: an error returned here fails the replay's |
| +// transaction before its commit. |
| +var replayFail func() error |
| + |
| +// latchDiscovery keeps a failed discovery in memory with the bound and |
| +// counted set its transaction computed; a failure before those could be |
| +// read latches unresolved, which bars every day until the replay |
| +// computes them. Caller holds l.mu. |
| +func (l *DurabilityLedger) latchDiscovery(d Discovery, from time.Time, counted map[string]bool) { |
| + pd := &pendingDiscovery{d: d, from: from, counted: counted, unresolved: from.IsZero()} |
| + if pd.counted == nil { |
| + pd.counted = map[string]bool{} |
| } |
| ds := l.disc() |
| ds.mu.Lock() |
| ds.latched[d.FleetID] = pd |
| + ds.generation++ |
| ds.mu.Unlock() |
| } |
| // Latched reports whether a fleet has a discovery awaiting replay, and |
| -// the bound it bars publication from. |
| +// the bound it bars publication from (zero when the bound itself is |
| +// unresolved: every day is barred). |
| func (l *DurabilityLedger) Latched(fleetID string) (time.Time, bool) { |
| ds := l.disc() |
| ds.mu.Lock() |
| @@ -251,25 +388,56 @@ func (l *DurabilityLedger) Latched(fleetID string) (time.Time, bool) { |
| } |
| // latchedBarring is the publisher's overlay: the number of latched |
| -// discoveries whose bound is on or before the given day's end. |
| +// discoveries whose bound is on or before the given day's end, an |
| +// unresolved one counting for every day. |
| func (l *DurabilityLedger) latchedBarring(dayEnd time.Time) int64 { |
| ds := l.disc() |
| ds.mu.Lock() |
| defer ds.mu.Unlock() |
| var n int64 |
| for _, pd := range ds.latched { |
| - if pd.from.Before(dayEnd) { |
| + if pd.unresolved || pd.from.Before(dayEnd) { |
| n++ |
| } |
| } |
| return n |
| } |
| +// latchedDates is every day a latched discovery bars, from its bound (or |
| +// the given floor, for an unresolved one) through now, for publication's |
| +// enumeration; bounded like a gap's touch. |
| +func (l *DurabilityLedger) latchedDates(floor, now time.Time) []string { |
| + ds := l.disc() |
| + ds.mu.Lock() |
| + defer ds.mu.Unlock() |
| + var out []string |
| + for _, pd := range ds.latched { |
| + from := pd.from |
| + if pd.unresolved || from.IsZero() { |
| + from = floor |
| + } |
| + if from.IsZero() { |
| + from = now |
| + } |
| + n := 0 |
| + for d := from.UTC().Truncate(24 * time.Hour); !d.After(now.UTC()); d = d.Add(24 * time.Hour) { |
| + if n++; n > ledgerGapTouchMax { |
| + break |
| + } |
| + out = append(out, DayOf(d)) |
| + } |
| + } |
| + return out |
| +} |
| + |
| // Replay is the latched discovery's one transaction: the uncertainty at |
| -// the original bound, the holds of [bound, discovery], the interval of |
| -// every counted admission that settled meanwhile, and closure at the |
| -// replay instant plus the margin when none remains unsettled; the latch |
| -// clears only after the commit. Mirrors admissionmodel.Replay. |
| +// the earlier of the latched bound and the bound as the rows read now |
| +// (a latch that could not read them takes the latter), the holds of |
| +// [bound, discovery], the interval of every counted admission that |
| +// settled meanwhile, and closure at the latest of the discovery and the |
| +// replay plus the margin when no admission of the fleet is unsettled and |
| +// nothing is pending (the same test as a settlement's); the latch clears |
| +// only after the commit. Mirrors admissionmodel.Replay. |
| func (l *DurabilityLedger) Replay(fleetID string) (int64, error) { |
| ds := l.disc() |
| ds.mu.Lock() |
| @@ -279,14 +447,67 @@ func (l *DurabilityLedger) Replay(fleetID string) (int64, error) { |
| return 0, nil |
| } |
| now := l.now().UTC() |
| + l.drainMu.Lock() |
| + defer l.drainMu.Unlock() |
| l.mu.Lock() |
| defer l.mu.Unlock() |
| + if l.closed { |
| + return 0, errLedgerClosed |
| + } |
| + return l.replayLocked(fleetID, pd, now) |
| +} |
| + |
| +// replayLocked is Replay under drainMu and l.mu: its own transaction, the |
| +// latch cleared after the commit. |
| +func (l *DurabilityLedger) replayLocked(fleetID string, pd *pendingDiscovery, now time.Time) (int64, error) { |
| tx, err := l.db.Begin() |
| if err != nil { |
| return 0, err |
| } |
| defer tx.Rollback() //nolint:errcheck |
| - res, err := tx.Exec(`INSERT INTO uncertainties (fleet_id, from_ns, to_ns, reason) VALUES (?, ?, NULL, ?)`, fleetID, nsOf(pd.from), pd.d.Reason) |
| + id, err := l.replayTx(tx, fleetID, pd, now) |
| + if err != nil { |
| + return 0, err |
| + } |
| + if replayFail != nil { |
| + if err := replayFail(); err != nil { |
| + return 0, err |
| + } |
| + } |
| + if err := tx.Commit(); err != nil { |
| + return 0, err |
| + } |
| + l.clearLatch(fleetID) |
| + return id, nil |
| +} |
| + |
| +// clearLatch forgets a replayed discovery; after its commit only. |
| +func (l *DurabilityLedger) clearLatch(fleetID string) { |
| + ds := l.disc() |
| + ds.mu.Lock() |
| + delete(ds.latched, fleetID) |
| + ds.generation++ |
| + ds.mu.Unlock() |
| +} |
| + |
| +// replayTx is the replay's writes inside a transaction the caller |
| +// commits: Replay's own, a discovery's that finds the latch, or a |
| +// closure's that carries the pending discovery with it. The latch is the |
| +// caller's to clear after the commit. |
| +func (l *DurabilityLedger) replayTx(tx *sql.Tx, fleetID string, pd *pendingDiscovery, now time.Time) (int64, error) { |
| + // the bound as the rows read now, and the earlier of it and the |
| + // latched one; the counted set is the union |
| + from, counted, err := l.bound(tx, pd.d) |
| + if err != nil { |
| + return 0, err |
| + } |
| + if !pd.unresolved && pd.from.Before(from) { |
| + from = pd.from |
| + } |
| + for tok := range pd.counted { |
| + counted[tok] = true |
| + } |
| + res, err := tx.Exec(`INSERT INTO uncertainties (fleet_id, from_ns, to_ns, discovered_at, reason) VALUES (?, ?, NULL, ?, ?)`, fleetID, nsOf(from), nsOf(pd.d.At), pd.d.Reason) |
| if err != nil { |
| return 0, err |
| } |
| @@ -294,68 +515,52 @@ func (l *DurabilityLedger) Replay(fleetID string) (int64, error) { |
| if err != nil { |
| return 0, err |
| } |
| - if err := writeHolds(tx, pd.from, pd.d.At, id, now); err != nil { |
| + if err := markStale(tx, fleetID); err != nil { |
| return 0, err |
| } |
| - unsettled := 0 |
| - for token := range pd.counted { |
| + if err := l.writeHolds(tx, from, pd.d.At, id, now); err != nil { |
| + return 0, err |
| + } |
| + for token := range counted { |
| a, err := readAdmission(tx, token) |
| if err != nil { |
| return 0, err |
| } |
| - if a.Settled() { |
| - if a.How != AdmissionUnissued { |
| - if err := writeHolds(tx, a.AdmittedAt, a.SettledAt, id, now); err != nil { |
| - return 0, err |
| - } |
| + if a.Settled() && a.How != AdmissionUnissued { |
| + if err := l.writeHolds(tx, a.AdmittedAt, a.SettledAt, id, now); err != nil { |
| + return 0, err |
| } |
| - continue |
| - } |
| - unsettled++ |
| - } |
| - if unsettled == 0 && l.pendingCount(fleetID) == 0 { |
| - to := now.Add(AdmissionMargin) |
| - if _, err := tx.Exec(`UPDATE uncertainties SET to_ns = ? WHERE id = ?`, nsOf(to), id); err != nil { |
| - return 0, err |
| - } |
| - if err := writeHolds(tx, pd.from, to, id, now); err != nil { |
| - return 0, err |
| } |
| } |
| - if err := tx.Commit(); err != nil { |
| + if err := l.maybeClose(tx, fleetID, id, from, now); err != nil { |
| return 0, err |
| } |
| - ds.mu.Lock() |
| - delete(ds.latched, fleetID) |
| - ds.mu.Unlock() |
| return id, nil |
| } |
| // settled is the settlement's hook (participant's transaction): under an |
| // open uncertainty of the admission's fleet it writes the holds of |
| -// [admitted_at, settled_at] and closes the uncertainty when no counted |
| -// admission (row or pending registration) remains; under none it writes |
| -// nothing permanent. Mirrors admissionmodel.settle and maybeClose. |
| +// [admitted_at, settled_at] and closes the uncertainty when no admission |
| +// of the fleet is unsettled and no registration is pending; under none it |
| +// writes nothing permanent. Mirrors admissionmodel.settle and maybeClose. |
| func (l *DurabilityLedger) settled(tx *sql.Tx, a Admission, at time.Time) error { |
| - var id int64 |
| - var fromNs int64 |
| - err := tx.QueryRow(`SELECT id, from_ns FROM uncertainties WHERE fleet_id = ? AND to_ns IS NULL ORDER BY id LIMIT 1`, a.FleetID).Scan(&id, &fromNs) |
| - if errors.Is(err, sql.ErrNoRows) { |
| - return nil |
| - } |
| - if err != nil { |
| + id, from, ok, err := openUncertainty(tx, a.FleetID) |
| + if err != nil || !ok { |
| return err |
| } |
| if a.How != AdmissionUnissued { |
| - if err := writeHolds(tx, a.AdmittedAt, at, id, at); err != nil { |
| + if err := l.writeHolds(tx, a.AdmittedAt, at, id, at); err != nil { |
| return err |
| } |
| } |
| - return l.maybeClose(tx, a.FleetID, id, time.Unix(0, fromNs).UTC(), at) |
| + return l.maybeClose(tx, a.FleetID, id, from, at) |
| } |
| -// maybeClose closes an open uncertainty when nothing counted remains: |
| -// no unsettled row of the fleet and no pending registration. |
| +// maybeClose closes an open uncertainty when nothing counted remains (no |
| +// unsettled row of the fleet and no pending registration), at the latest |
| +// of its discovery, this settlement and every settlement of the fleet at |
| +// or after its bound, plus the margin: a settlement reported out of order |
| +// never closes it before an interval already written. |
| func (l *DurabilityLedger) maybeClose(tx *sql.Tx, fleetID string, id int64, from, settlement time.Time) error { |
| var open int |
| if err := tx.QueryRow(`SELECT COUNT(*) FROM admissions WHERE fleet_id = ? AND settled_at IS NULL`, fleetID).Scan(&open); err != nil { |
| @@ -364,11 +569,23 @@ func (l *DurabilityLedger) maybeClose(tx *sql.Tx, fleetID string, id int64, from |
| if open > 0 || l.pendingCount(fleetID) > 0 { |
| return nil |
| } |
| - to := settlement.Add(AdmissionMargin) |
| + var discovered int64 |
| + if err := tx.QueryRow(`SELECT discovered_at FROM uncertainties WHERE id = ?`, id).Scan(&discovered); err != nil { |
| + return err |
| + } |
| + latest := maxTime(settlement, time.Unix(0, discovered).UTC()) |
| + var last sql.NullInt64 |
| + if err := tx.QueryRow(`SELECT MAX(settled_at) FROM admissions WHERE fleet_id = ? AND settled_at >= ? AND how != ?`, fleetID, nsOf(from), AdmissionUnissued).Scan(&last); err != nil { |
| + return err |
| + } |
| + if last.Valid { |
| + latest = maxTime(latest, time.Unix(0, last.Int64).UTC()) |
| + } |
| + to := latest.Add(AdmissionMargin) |
| if _, err := tx.Exec(`UPDATE uncertainties SET to_ns = ? WHERE id = ? AND to_ns IS NULL`, nsOf(to), id); err != nil { |
| return err |
| } |
| - return writeHolds(tx, from, to, id, settlement) |
| + return l.writeHolds(tx, from, to, id, settlement) |
| } |
| // EmptySettlement tells an open uncertainty that a counted registration |
| @@ -381,20 +598,19 @@ func (l *DurabilityLedger) EmptySettlement(fleetID string, at time.Time) error { |
| } |
| l.mu.Lock() |
| defer l.mu.Unlock() |
| + if l.closed { |
| + return errLedgerClosed |
| + } |
| tx, err := l.db.Begin() |
| if err != nil { |
| return err |
| } |
| defer tx.Rollback() //nolint:errcheck |
| - var id, fromNs int64 |
| - err = tx.QueryRow(`SELECT id, from_ns FROM uncertainties WHERE fleet_id = ? AND to_ns IS NULL ORDER BY id LIMIT 1`, fleetID).Scan(&id, &fromNs) |
| - if errors.Is(err, sql.ErrNoRows) { |
| - return nil |
| - } |
| - if err != nil { |
| + id, from, ok, err := openUncertainty(tx, fleetID) |
| + if err != nil || !ok { |
| return err |
| } |
| - if err := l.maybeClose(tx, fleetID, id, time.Unix(0, fromNs).UTC(), at); err != nil { |
| + if err := l.maybeClose(tx, fleetID, id, from, at); err != nil { |
| return err |
| } |
| return tx.Commit() |
| @@ -405,9 +621,9 @@ func (l *DurabilityLedger) ReadUncertainty(id int64) (Uncertainty, error) { |
| l.mu.Lock() |
| defer l.mu.Unlock() |
| var u Uncertainty |
| - var from int64 |
| + var from, discovered int64 |
| var to sql.NullInt64 |
| - err := l.db.QueryRow(`SELECT id, fleet_id, from_ns, to_ns, reason FROM uncertainties WHERE id = ?`, id).Scan(&u.ID, &u.FleetID, &from, &to, &u.Reason) |
| + err := l.db.QueryRow(`SELECT id, fleet_id, from_ns, to_ns, discovered_at, reason FROM uncertainties WHERE id = ?`, id).Scan(&u.ID, &u.FleetID, &from, &to, &discovered, &u.Reason) |
| if errors.Is(err, sql.ErrNoRows) { |
| return Uncertainty{}, fmt.Errorf("usage ledger: no uncertainty %d", id) |
| } |
| @@ -415,12 +631,21 @@ func (l *DurabilityLedger) ReadUncertainty(id int64) (Uncertainty, error) { |
| return Uncertainty{}, err |
| } |
| u.From = time.Unix(0, from).UTC() |
| + u.DiscoveredAt = time.Unix(0, discovered).UTC() |
| if to.Valid { |
| u.To = time.Unix(0, to.Int64).UTC() |
| } |
| return u, nil |
| } |
| +// markStale writes the journal's stale state when an uncertainty of the |
| +// fleet is opened (inside the discovery's transaction); a fleet with no |
| +// journal row is left alone. The journal's states are the fleet slice's. |
| +func markStale(tx *sql.Tx, fleetID string) error { |
| + _, err := tx.Exec(`UPDATE fleets SET state = 'stale' WHERE fleet_id = ? AND state = 'open'`, fleetID) |
| + return err |
| +} |
| + |
| // Held reports whether a day has a hold row (box-wide). |
| func (l *DurabilityLedger) Held(day string) (bool, error) { |
| l.mu.Lock() |
diff --git a/internal/usage/durabilityuncertainty_test.go b/internal/usage/durabilityuncertainty_test.go| index bc87bb3f2..03b2c0ef8 100644 |
| --- a/internal/usage/durabilityuncertainty_test.go |
| +++ b/internal/usage/durabilityuncertainty_test.go |
| @@ -3,6 +3,7 @@ package usage |
| import ( |
| "errors" |
| "strings" |
| + "sync/atomic" |
| "testing" |
| "time" |
| ) |
| @@ -112,8 +113,9 @@ func TestUncertaintyExtendsToTheSettlement(t *testing.T) { |
| func TestEmptySettlementsCloseTheUncertainty(t *testing.T) { |
| l, set := openLedgerAt(t, at(2, 23, 57)) |
| tok := mustAdmit(t, l, "f", "alice", "app", time.Time{}) |
| - pending := 1 |
| - l.SetPendingCounter("f", func() int { return pending }) |
| + var pending atomic.Int64 |
| + pending.Store(1) |
| + l.SetPendingCounter("f", &pending) |
| set(at(2, 23, 58)) |
| id, err := l.Discover(Discovery{FleetID: "f", At: at(2, 23, 58), Proof: at(2, 23, 57), LastCleanLook: at(2, 23, 55), Pending: []time.Time{at(2, 23, 50)}}) |
| if err != nil { |
| @@ -129,7 +131,7 @@ func TestEmptySettlementsCloseTheUncertainty(t *testing.T) { |
| if u, _ := l.ReadUncertainty(id); !u.Open() { |
| t.Fatal("closed with a registration pending") |
| } |
| - pending = 0 |
| + pending.Store(0) |
| if err := l.EmptySettlement("f", time.Time{}); err != nil { |
| t.Fatal(err) |
| } |
| @@ -178,8 +180,9 @@ func TestLatchedDiscoveryAndReplay(t *testing.T) { |
| t.Fatal("the latch stopped barring before the replay") |
| } |
| set(at(4, 0, 3)) |
| - pending := 1 // a registration still unresolved: the replay stays open, so the settled token's days come from its own write |
| - l.SetPendingCounter("f", func() int { return pending }) |
| + var pending atomic.Int64 // a registration still unresolved: the replay stays open, so the settled token's days come from its own write |
| + pending.Store(1) |
| + l.SetPendingCounter("f", &pending) |
| id, err := l.Replay("f") |
| if err != nil || id == 0 { |
| t.Fatal(id, err) |
| @@ -189,7 +192,7 @@ func TestLatchedDiscoveryAndReplay(t *testing.T) { |
| t.Fatalf("%+v", u) |
| } |
| held(t, l, true, "2026-10-02", "2026-10-03", "2026-10-04") |
| - pending = 0 |
| + pending.Store(0) |
| set(at(4, 0, 4)) |
| if err := l.EmptySettlement("f", time.Time{}); err != nil { |
| t.Fatal(err) |