amadan
Review eight answered; plan Task 4's second and third slices (the barrier; discovery, the uncertainty and the latch); design revision 14
The ledger (internal/usage): the admission's transitions follow the table's edges (unissued from admitted alone; finished and abandoned need the recorder registered), the caller allocates the token before the write so an ambiguous commit is read back by it, an exact repeat is the write that landed, an instant before the admission is refused, a settlement is no earlier than now, every transition refuses on a closed ledger, a partial enumeration is an error. Slice 2: DurabilityContent.Barred counted in the snapshot's read transaction; CompletenessFor refuses on it. Slice 3: Discover with the fixed bound and the counted set, uncertainties and box-wide holds in one transaction, the settlement hook, the in-memory latch and Replay, DurabilityContent.Held. The inventory (internal/ops): the strict walk's listed facts carried into the ownership checks (the identity rule across the handoff), the recheck visiting everything so a change wins over drift, required coverage as ownership with the refusal as the reason, Hold returning an error that the refusal says, carlos ops inventory opening the registry read-only (registry.OpenReadOnly) and comparing the live configuration as well as a regeneration's list, paths quoted. The model gains the recorder-outcome check; one of Opus's three mutants killed, two recorded as equivalent. Tests for each (ENOENT rename-away, drift at the final look, change over drift, the swapped listed directory, the live configuration fixture, the read-only open, the ledger's edges, instants, fleets, closed checks and synchronous=FULL); nineteen mutants killed in tree copies; the 30000-seed sweep passes. Dispositions "Review eight"; plan slice 1 text corrected; docs/site/cli.md documents ops inventory.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
22 files changed,
+1806
−115
DISPOSITIONS-replicator-pinned-root.md+24 −0docs/site/cli.md+8 −1docs/superpowers/plans/2026-10-03-replicator-authority-plan.md+3 −3docs/superpowers/specs/2026-10-03-replicator-pinned-roots-design.md+3 −3docs/superpowers/specs/2026-10-03-replicator-pinned-roots-spikes.md+1 −1internal/ops/admissionmodel/model.go+7 −0internal/ops/admissionmodel/model_test.go+13 −2internal/ops/authoritybridge.go+9 −2internal/ops/litestream.go+11 −0internal/ops/litestream_metered.go+7 −3internal/ops/litestream_metered_test.go+108 −4internal/ops/litestream_owners.go+76 −10internal/ops/litestream_owners_unix_test.go+157 −8internal/ops/ops.go+79 −30internal/registry/readonly_test.go+45 −0internal/registry/registry.go+29 −0internal/usage/durability.go+18 −1internal/usage/durabilityadmissions.go+121 −31internal/usage/durabilityadmissions_test.go+334 −16internal/usage/durabilityledger.go+20 −0internal/usage/durabilityuncertainty.go+433 −0internal/usage/durabilityuncertainty_test.go+300 −0
CI passed — run details
diff --git a/DISPOSITIONS-replicator-pinned-root.md b/DISPOSITIONS-replicator-pinned-root.md| index 24f6de7b1..a600ccc19 100644 |
| --- a/DISPOSITIONS-replicator-pinned-root.md |
| +++ b/DISPOSITIONS-replicator-pinned-root.md |
| @@ -130,3 +130,27 @@ Opus: accept with fixes, no P1, two P2. Astra: accept with fixes, no P1, two P2. |
| Plan Task 3 (required coverage) landed at `f0ed32b7` between these reviews and is in this delta's review scope; plan Task 4's first slice (the admission tables and the row's transitions in the durability ledger, `internal/usage/durabilityadmissions.go`, with tests mirroring the model) lands with these answers. |
| + |
| +## Review eight (of `47efe06f`: the seventh review answered, plan Task 3, Task 4's first slice) |
| + |
| +Opus: accept with fixes, no P1, one P2. Astra: accept with fixes, no P1, six P2. Reports: `logs/opus-replroot-review8.md`, `logs/astra-replroot-review8.md`. The gate on `47efe06f` passed. Task 4's second and third slices (the barrier; discovery, the uncertainty and the latch) land with these answers. |
| + |
| +| Finding | Disposition | |
| +|---|---| |
| +| Opus P2-1, Astra 3: the ledger accepted transitions the table has no edge for (`unissued` after a participant; `finished` and `abandoned` with no recorder registered) | `terminal(unissued)` refuses after any participant activity (`ErrAdmissionEdge`); `RecorderDone(finished|abandoned)` refuses without `RecorderStarted`; the refusals write nothing; tests for each, and the later-participant test registers the recorder in both orders. The model's set gains the recorder-outcome check and the schedule attempts the outcome before Start (spike note). | |
| +| Astra 1: `Admit` allocated the token after the write, so an ambiguous commit could not be read back | `NewAdmissionToken` allocates before the write and `Admit(token, ...)` takes it; an exact repeat under the token is the write that landed, other facts are refused; a seam fails the transaction after the insert and the read-back says unknown (conclusive cancellation). | |
| +| Astra 2: `UnsettledAdmissions` dropped `rows.Err()` | checked: a partial enumeration is an error, never an empty list. | |
| +| Astra 4, Opus P3-8: `carlos ops inventory` created and migrated the registry, printed raw paths, took two looks unsaid | `registry.OpenReadOnly` (an existing regular file, `mode=ro`, no journal mode, no migration; a write through it fails; tested); every path quoted; the report says the external lists and the inventory are separate looks. | |
| +| Astra 5: the report compared against the regenerated list, not the live configuration | `InventoryReport(reg, dir, liveConfig)` reads the live file (`-config`, default `/etc/litestream.yml`) and compares both the live list and the regenerated proposal against the inventory, each labelled; a fixture whose live configuration names a database the registry no longer describes. `docs/site/cli.md` documents the command. | |
| +| Astra 6: discovery handed ownership checking pathnames; the listed directory swapped between the two was owned | the strict walk records the facts of every database it lists (`fleetPins.listed`); the ownership pass refuses a file whose looked-at identity is not the listed one, and a file whose checked chain's identity is not the looked-at one; the pathname post-filter in `fleetDBsFromRoutes` no longer stats what the walk listed; a `fleetInventory` test swaps the listed directory between discovery and the checks (seam `inventoryAfterDiscovery`) and the removal mutant is killed; the second comparison (looked-at to checked) has no seam between its two looks and is inspection-only. The retained `Dir`s themselves are not handed across (the walk releases its descriptors at its end and the ownership check acquires and keeps its own chain): the identity comparison is I1's rule, the file inspected is the file listed, which is what the handoff lacked. | |
| +| Opus P3-1: one `isChange` branch tested; the first recheck error won | a chain directory renamed away (ENOENT) stales from the last clean look; drift at the final look stales at Close; the recheck visits everything so a change wins over a look that could not be taken (seam `ancestryChanged` fails every root while a chain edge changes: stale, not drift); each mutant killed. | |
| +| Opus P3-2: transitions not idempotent | an exact repeat (same participant and instant; same `how`; `unissued` on an unissued row; the same registration) returns nil; a conflicting repeat stays an error; tested with explicit instants. | |
| +| Opus P3-3: backdated instants | an instant before the admission's is refused (`ErrAdmissionBackdated`, nothing written); the settlement is the latest of the two completions and now; `unissued`'s timestamps are now; `lost` documented as not recorded; a backdated pair settles now (test). | |
| +| Opus P3-4: the plan's slice-1 text overstated | rewritten: the signature as built, the budget and `ErrAdmissionNotDurable` named as slice 5's, the mirrors trimmed. | |
| +| Opus P3-5, Astra 7: ledger tests that would survive | a second fleet with admission order against insertion order; out-of-order instants; `abandoned`; a duplicate recorder completion with the transport pending; `RecorderStarted` after an outcome; every transition on a closed ledger; `synchronous=FULL` read back from the pragma; the absence of a permanent hold is asserted by slice 3's discovery tests (`held(..., false, day)`). Commit ambiguity is the seam test above; crash durability is not claimed. | |
| +| Opus P3-6: required coverage checked reachability | after the ownership pass a required path not owned is missing with its refusal as the reason, a problem, and holds the day (a route claiming the console database: tested); `RequiredDatabases(registryPath)` loses its unused parameter and its comment describes itself. | |
| +| Opus P3-7: the `CONSOLE_DB`-only rule probably covers nothing in production | stated as a deployment condition in the design (the agent's unit on a console box must carry it); the report says when the console database at the conventional path is owned but not required, so the gap is visible on the box. | |
| +| Opus P3-9: `Hold` returned nothing | `Hold func(time.Time) error`; a hold not written is said in the refusal (tested); the restored step builds a fleet and asserts no hold. | |
| +| Astra 8: the recovery paragraph said all three timestamps; the ignored `Chmod`; the CLI doc | design revision 14 (missing timestamps, a completion before the crash keeps its instant); the `Chmod` checked; `ops inventory` documented. | |
| +| Astra's notes: `openDirFDsUnder` stat-based; `cleanBefore` captured after the EIO look | the counter classifies the descriptor by `fstat` and keeps a deleted directory; the instant is captured before the look that cannot be taken, and that look is asserted not to count as clean. | |
| +| Opus's three model mutants | one killed (above); two equivalent under the model, each with its reason in the spike note. | |
diff --git a/docs/site/cli.md b/docs/site/cli.md| index dba6553c5..5f31f8112 100644 |
| --- a/docs/site/cli.md |
| +++ b/docs/site/cli.md |
| @@ -1438,10 +1438,17 @@ to restart replication. `restore-verify` restores every configured replica |
| from S3 and integrity-checks it. `hibernate-verify` does the same drill for |
| the replicas the config does not name — hibernating instances, whose |
| replication belongs to the activator — and publishes a per-host verdict to |
| -the deployment bucket so you can read it off-box. |
| +the deployment bucket so you can read it off-box. `inventory` is read-only: it |
| +opens the registry without creating or migrating it and prints the platform |
| +replicator's inventory (owned, skipped, unowned, problems, and the required |
| +databases with whether each is reachable and owned) beside the external |
| +replicator's live configuration (`-config`, default `/etc/litestream.yml`) |
| +and the list a regeneration would write, every path quoted; the external |
| +lists and the inventory are separate looks, one after the other. |
| ```sh |
| carlos ops restore-verify |
| +carlos ops inventory -config /etc/litestream.yml |
| ``` |
| ## Everything else |
diff --git a/docs/superpowers/plans/2026-10-03-replicator-authority-plan.md b/docs/superpowers/plans/2026-10-03-replicator-authority-plan.md| index 88d149eda..c76eae12d 100644 |
| --- a/docs/superpowers/plans/2026-10-03-replicator-authority-plan.md |
| +++ b/docs/superpowers/plans/2026-10-03-replicator-authority-plan.md |
| @@ -32,9 +32,9 @@ The admission model (`internal/ops/admissionmodel`) is the executable contract; |
| **Slices, each its own reviewed commit with the model's test named beside each ledger test.** |
| -1. **The tables and the admission's row (built: `internal/usage/durabilityadmissions.go`).** `admissions(token TEXT PRIMARY KEY, fleet_id, account, app, admitted_at, transport_done_at, recorder_started_at, recorder_done_at, settled_at, how)`, `fleets(fleet_id PRIMARY KEY, box, witness_generation, owners, opened_at, state, predecessor, acknowledged_by)`, `uncertainties(id, fleet_id, from_ns, to_ns NULL, reason)`, `inherited(fleet_id, uncertainty_id)`, `holds(day, account, app, uncertainty_id, written_at)`, all `IF NOT EXISTS` beside today's tables. The `DurabilityLedger` gains `Admit(fleet, owner, at) (token, error)` (one transaction, `synchronous=FULL`, within the gate's budget: `ErrAdmissionNotDurable` past it, the goroutine staying to mark a late commit `unissued`), `TransportDone(token, at, lost bool)`, `RecorderStarted(token, at)`, `RecorderDone(token, at, how)`, settlement by whichever lands last in its own transaction, and `Unissued(token)`. Mirrors: `TestUnchangedFleetHoldsNothing`, `TestThreeMidnightsBothOrders` (the interval written at settlement), `TestCommitAfterDiscoveryIsUnissued`, `TestBudgetExpiryWithoutDiscovery`, `TestHistoricalTermCountsUnrecordedSettlements`. |
| -2. **The barrier.** The publisher's snapshot (`snapshotOnce`) treats a day incomplete for an owner while a durable unsettled admission of that owner, or of any fleet whose unacknowledged ancestry includes the admission's fleet, is admitted on or before that day (box-wide: any owner); it is a query over `admissions` and `fleets`, writes nothing, and lifts at settlement. Mirrors: the model's barrier invariant and `TestSuccessorBarredByPredecessorAdmission`, `TestLandedAmbiguousCommitBarsBeforeReadBack`. |
| -3. **Discovery, the uncertainty and the latch.** `Discover(fleet, at, proof, lastCleanLook)`: under the fleet's discovery mutex (new in `MeteredFleet`: registration, the recheck and the start of a discovery serialise on it, the pending set counted), the fixed bound, the counted set, the `uncertainties` row and the holds of `[from, now]` for the owners reached in one transaction, closed at once with nothing pending; a failed write latches the pending discovery in the existing failure latch (extended with the bound, the owners reached and the counted tokens; the publisher's overlay reaches successors through the ancestry query) and the fleet refuses requests until `Replay`, one transaction as the design has it. Mirrors: `TestZeroPendingClosesAtOnceAcrossMidnight`, `TestFailedDiscoveryLatchBarsUntilReplay`, `TestFailedDiscoveryReplayKeepsSettlements`, `TestReplayHoldsTheLowerBoundWhileOpen`, `TestReplayWritesSettledTokenWhileOpen`, `TestLatchOverlayReachesSuccessor`, the lower-bound tests (the in-flight request across a clean look, the proof term, the earlier-of threshold, the fixed cutoff in both orders, the admission instant, the exact bound, the empty terminals excluded). |
| +1. **The tables and the admission's row (built: `internal/usage/durabilityadmissions.go`).** `admissions(token TEXT PRIMARY KEY, fleet_id, account, app, admitted_at, transport_done_at, recorder_started_at, recorder_done_at, settled_at, how)`, `fleets(fleet_id PRIMARY KEY, box, witness_generation, owners, opened_at, state, predecessor, acknowledged_by)`, `uncertainties(id, fleet_id, from_ns, to_ns NULL, reason)`, `inherited(fleet_id, uncertainty_id)`, `holds(day, account, app, uncertainty_id, written_at)`, all `IF NOT EXISTS` beside today's tables. The `DurabilityLedger` gains `Admit(token, fleet, owner, app, at) error` (the token allocated by the caller before the write, `NewAdmissionToken`, so an ambiguous commit is read back by it; one transaction, `synchronous=FULL`; an exact repeat is the write that landed, a repeat with other facts is refused; the gate's budget, `ErrAdmissionNotDurable` past it and the goroutine staying to mark a late commit `unissued`, are slice 5's, the gate's, and are not built here), `TransportDone(token, at, lost bool)` (`lost` is not recorded: transport-lost is never a value of `how`), `RecorderStarted(token, at)`, `RecorderDone(token, at, how)` (`finished` and `abandoned` need the recorder registered), settlement by whichever lands last in its own transaction at the latest of the two completions and now, and `Unissued(token, at)` (an edge from `admitted` alone: refused after any participant activity); every transition refuses an instant before the admission's, is idempotent for an exact repeat, and refuses on a closed ledger. Mirrors: `TestUnchangedFleetHoldsNothing` and `TestThreeMidnightsBothOrders` (the row's settlement in both orders; the interval written at settlement is slice 3's), `TestCommitAfterDiscoveryIsUnissued`, `TestHistoricalTermCountsUnrecordedSettlements`; `TestBudgetExpiryWithoutDiscovery` is slice 5's. |
| +2. **The barrier (built: `DurabilityContent.Barred`, counted in the snapshot's read transaction; `CompletenessFor` refuses while it is positive; box-wide, so the ancestry query waits for owner scope).** The publisher's snapshot (`snapshotOnce`) treats a day incomplete for an owner while a durable unsettled admission of that owner, or of any fleet whose unacknowledged ancestry includes the admission's fleet, is admitted on or before that day (box-wide: any owner); it is a query over `admissions` and `fleets`, writes nothing, and lifts at settlement. Mirrors: the model's barrier invariant and `TestSuccessorBarredByPredecessorAdmission`, `TestLandedAmbiguousCommitBarsBeforeReadBack`. |
| +3. **Discovery, the uncertainty and the latch (built in the ledger: `Discover` with the fixed bound and the counted set, `uncertainties` and box-wide `holds` rows in one transaction, closed at once with nothing counted or pending; the settlement's hook writing `[admitted_at, settled_at]` and closing when no row and no pending registration remains, the fleet's pending count by `SetPendingCounter`; `EmptySettlement`; a failed transaction latched in memory with its bound and counted tokens, barring publication through the snapshot's `Barred` until `Replay`, one transaction; `DurabilityContent.Held` refusing completeness; the fleet's discovery mutex, registration and the latch's refusal of requests are the gate slice).** `Discover(fleet, at, proof, lastCleanLook)`: under the fleet's discovery mutex (new in `MeteredFleet`: registration, the recheck and the start of a discovery serialise on it, the pending set counted), the fixed bound, the counted set, the `uncertainties` row and the holds of `[from, now]` for the owners reached in one transaction, closed at once with nothing pending; a failed write latches the pending discovery in the existing failure latch (extended with the bound, the owners reached and the counted tokens; the publisher's overlay reaches successors through the ancestry query) and the fleet refuses requests until `Replay`, one transaction as the design has it. Mirrors: `TestZeroPendingClosesAtOnceAcrossMidnight`, `TestFailedDiscoveryLatchBarsUntilReplay`, `TestFailedDiscoveryReplayKeepsSettlements`, `TestReplayHoldsTheLowerBoundWhileOpen`, `TestReplayWritesSettledTokenWhileOpen`, `TestLatchOverlayReachesSuccessor`, the lower-bound tests (the in-flight request across a clean look, the proof term, the earlier-of threshold, the fixed cutoff in both orders, the admission instant, the exact bound, the empty terminals excluded). |
| 4. **The fleet journal, construction backfill and acknowledgement.** `OpenFleet(predecessor, owners, witness)` writes the `open` row and the backfill of every interval its unacknowledged ancestry holds; `stale`, `closing`, `closed` as the design's transactions (closure carries a pending replay and refuses with an unsettled admission); `Acknowledge(successor)` root-first, copying the effective set into `inherited` and writing the holds for the closed ones. The ledger's reach query replaces the model's `reach`. Mirrors: `TestAncestryReachAndRootFirstAcknowledgement`, `TestHealthyChainContributesNothing`, the per-step inheritance invariant as a property test over a recorded schedule. |
| 5. **The gate and the fleet.** `MeteredFleet.admit` becomes the proof (unchanged) plus the admission: registration in the pending set under the discovery mutex, the ledger's `Admit`, the recheck, the token handed to the SDK middleware (`lease.RequestGate` returns the token; the meter's `Start` carries it; the finalize step writes `TransportDone`). `lookNow` is the discovery when it finds a change; `closeNow` is `closing` then `closed`. The helper process does not exist yet (Part B): the transport and the recorder run in the agent, so the lifecycle guard is the fleet's own, `transport-lost` is written at `Close` for a transport that never finalised, and `unrecorded` where the meter's `Start` failed. Mirrors: `TestHelperDeathAndRecorderGuard` (as the agent's close), `TestSendNeedsDurableIssuedAdmission` (the middleware refuses without a token), `TestReadBackOnlyAfterTheHandleCloses`, `TestBetweenCommitAndAnswer`. |
| 6. **The producer fence and recovery.** `admissions.lock` flocked by the agent before its first admission on an open file description every helper will later inherit (today: the agent alone); `OpenDurabilityLedger`'s `recover` extended: with the fence, in one transaction, every unsettled admission `recovered-unknown` with its interval, every fleet not `closed` closed with its interval, every unacknowledged fleet acknowledged root-first by the first fleet built. Mirrors: `TestFenceOutlivesAgent`, `TestFenceHeldTokenSentAfterCrash`, `TestLandedAmbiguousThenCrash`, the kill-at-every-transition tests (a second process reopening the ledger after a kill at each boundary). |
diff --git a/docs/superpowers/specs/2026-10-03-replicator-pinned-roots-design.md b/docs/superpowers/specs/2026-10-03-replicator-pinned-roots-design.md| index 21405af50..6c9cfd792 100644 |
| --- a/docs/superpowers/specs/2026-10-03-replicator-pinned-roots-design.md |
| +++ b/docs/superpowers/specs/2026-10-03-replicator-pinned-roots-design.md |
| @@ -1,6 +1,6 @@ |
| # The replicator's authority: retained descriptors, a durable admission, claims by identity |
| -Date: 2026-10-03. Status: design, revision 13 (the sixth code review: the predicate's compatibility stated as within the pinned roots, with the two safe wider shapes and the kept platform-directory refusal; the walk's pre-pin absent exception; the link-shapes line extended). Revision 12 was (Part A's root rule stated as the plan has it: the platform walk ends where the platform's authority ends, a tenant-owned component's parent, and the chain beneath is retained and rechecked rather than refused). Revision 11 was (the fourth code review: the historical term's filter named `finished` and `abandoned` and so excluded `unrecorded`, a settlement whose transport did send; it now excludes only the empty terminals, `unissued` and `cancelled`; the look's order is stated as what safety needs rather than a sequence; the closing look agrees). Revision 10 was revision 9's correction of the lower bound refined after the third code review: the historical term's threshold is the earlier of the registry proof and the last clean look's start, since a witness change is known only to be after the proof; an empty terminal, unissued or cancelled, never sent and is not a settlement for that term; and every look reads the witness and rechecks every retained chain before it inspects files, so a clean look's start bounds every kind of change. Revision 9 was one mechanics correction to the uncertainty's lower bound, found by the second code review of the admission model: an admission admitted before the last clean look, still in flight across a change after it and settled clean before the change was noticed, was outside the bound; the bound now also takes the admission instant of every admission settled at or after the last clean look's start; the invariant I2 is unchanged and the model's named test and schedules cover it). Revision 8 was the accepted text: revision 7 was accepted with fixes by Astra (no P1; one P2: a negative read-back was not conclusive while the admission writer could still commit, and the pending set had three definitions; one P3: the restore wording made a salt mismatch imply a snapshot), both applied here. Revision 6 was rejected by Astra on one P1: a discovery whose write failed had no rule for the admissions that settled before its retry, so the replayed interval could lose a settlement; and three P2s: a helper dying before its recorder started left an admission nobody could finish; the transition table lacked the abort edges and the pending set's cleanup; the lineage record cited a durability discipline the shadow state does not have; each answered here. Revision 5 was rejected by Astra on two P1s: the admission's insertion had no ordering point against discovery, so a discovery between the proof and the commit counted nothing and a later send fell outside the closed uncertainty; construction did not backfill already-closed intervals for a successor's new owners before acknowledgement; and P2s: the transition table and the helper-death rule disagreed and the recorder participant was synthesised while its worker could still run; proof-only callers of admit would have created admissions with no terminal; a terminal `.` lost the directory requirement; conflicting summaries; each answered here. Revision 4 was rejected by Astra on four P1s: the helper has several connections of its own and needs SQLite's within-process lock bookkeeping between them; the producer fence did not cover a helper that outlives the agent with an admission in hand; an intermediate fleet closed before acknowledging broke transitive protection; a fleet could be marked closed with a discovery uncommitted; and four P2s: a dead helper's admissions had no terminal in the running agent, the sleeping binding persisted a reusable inode, the lineage predicate named header fields that identify nothing, the walk's grammar refused a `.` target the predicate accepts; each answered here. Revision 3 was rejected by Astra on five P1s: two SQLite VFSs in one process cannot coordinate through POSIX locks, since the library's own per-inode bookkeeping is in-process; a recorder that abandons early could settle an admission before its transport ended; an admission-only barrier does not cover a discovery whose uncertainty write failed after every admission settled; links at acknowledgement contradicted acknowledging only after settlement, and an inherited uncertainty vanished at the next successor; the ledger's lock does not prove the old process gone; and four P2s: closure at zero pending and the margin's extra day, sleeping claims with no binding, absolute and `.` spellings in the walk, `(dev, ino)` as a lineage; each answered here. Revision 2 was rejected by Astra on three P1s: the VFS delegated SQLite's Unix file methods, which open the SHM and check moved files by pathname themselves; every settled admission wrote a permanent hold, so ordinary replication never read complete; a late owner-scoped extension did not reach successors that had already acknowledged; and six P2s: orphan admissions never terminated, `..` and chain edges unspecified, `dup` sharing the directory offset, the platform-link predicate not preserved and terminal links unclassified, the uid producer without a lifecycle, one billing owner not one replicator; each answered here. Revision 1 was rejected by Astra on five P1s: SQLite canonicalises a `/proc` pathname before it opens anything and this driver passes no `nofollow`; `os.Root` confines but re-resolves every name, so a swap between a listing and an inspection still moves the answer; a `Start` row is not an admission, since the meter sends without one; the attempt-derived window lost a request completed before the change was noticed and the days between an admission and a finish; `closed_at` was not a handover; plus the directory open before `O_DIRECTORY`, per-owner holds the ledger cannot store, a uid with no producer, I3 against Part D, and tests). Branch `design-replicator-pinned-root`, off `feat/carloku-metered-replicator-20261001` at `e1280811`. It answers the eleven findings the twentieth review deferred (`DISPOSITIONS-metered-replicator.md`, review twenty) and keeps decisions 10 to 13 and approval H (`DECISIONS-FOR-PAUL.md`, section D, on the launch branch) as parameters. Nothing is built, wired or applied. |
| +Date: 2026-10-03. Status: design, revision 14 (the eighth code review: the recovery paragraph agrees with the transition table, an orphan's missing timestamps become now and a completion recorded before the crash keeps its instant; required coverage stated as ownership, with the deployment condition for the console database). Revision 13 was (the sixth code review: the predicate's compatibility stated as within the pinned roots, with the two safe wider shapes and the kept platform-directory refusal; the walk's pre-pin absent exception; the link-shapes line extended). Revision 12 was (Part A's root rule stated as the plan has it: the platform walk ends where the platform's authority ends, a tenant-owned component's parent, and the chain beneath is retained and rechecked rather than refused). Revision 11 was (the fourth code review: the historical term's filter named `finished` and `abandoned` and so excluded `unrecorded`, a settlement whose transport did send; it now excludes only the empty terminals, `unissued` and `cancelled`; the look's order is stated as what safety needs rather than a sequence; the closing look agrees). Revision 10 was revision 9's correction of the lower bound refined after the third code review: the historical term's threshold is the earlier of the registry proof and the last clean look's start, since a witness change is known only to be after the proof; an empty terminal, unissued or cancelled, never sent and is not a settlement for that term; and every look reads the witness and rechecks every retained chain before it inspects files, so a clean look's start bounds every kind of change. Revision 9 was one mechanics correction to the uncertainty's lower bound, found by the second code review of the admission model: an admission admitted before the last clean look, still in flight across a change after it and settled clean before the change was noticed, was outside the bound; the bound now also takes the admission instant of every admission settled at or after the last clean look's start; the invariant I2 is unchanged and the model's named test and schedules cover it). Revision 8 was the accepted text: revision 7 was accepted with fixes by Astra (no P1; one P2: a negative read-back was not conclusive while the admission writer could still commit, and the pending set had three definitions; one P3: the restore wording made a salt mismatch imply a snapshot), both applied here. Revision 6 was rejected by Astra on one P1: a discovery whose write failed had no rule for the admissions that settled before its retry, so the replayed interval could lose a settlement; and three P2s: a helper dying before its recorder started left an admission nobody could finish; the transition table lacked the abort edges and the pending set's cleanup; the lineage record cited a durability discipline the shadow state does not have; each answered here. Revision 5 was rejected by Astra on two P1s: the admission's insertion had no ordering point against discovery, so a discovery between the proof and the commit counted nothing and a later send fell outside the closed uncertainty; construction did not backfill already-closed intervals for a successor's new owners before acknowledgement; and P2s: the transition table and the helper-death rule disagreed and the recorder participant was synthesised while its worker could still run; proof-only callers of admit would have created admissions with no terminal; a terminal `.` lost the directory requirement; conflicting summaries; each answered here. Revision 4 was rejected by Astra on four P1s: the helper has several connections of its own and needs SQLite's within-process lock bookkeeping between them; the producer fence did not cover a helper that outlives the agent with an admission in hand; an intermediate fleet closed before acknowledging broke transitive protection; a fleet could be marked closed with a discovery uncommitted; and four P2s: a dead helper's admissions had no terminal in the running agent, the sleeping binding persisted a reusable inode, the lineage predicate named header fields that identify nothing, the walk's grammar refused a `.` target the predicate accepts; each answered here. Revision 3 was rejected by Astra on five P1s: two SQLite VFSs in one process cannot coordinate through POSIX locks, since the library's own per-inode bookkeeping is in-process; a recorder that abandons early could settle an admission before its transport ended; an admission-only barrier does not cover a discovery whose uncertainty write failed after every admission settled; links at acknowledgement contradicted acknowledging only after settlement, and an inherited uncertainty vanished at the next successor; the ledger's lock does not prove the old process gone; and four P2s: closure at zero pending and the margin's extra day, sleeping claims with no binding, absolute and `.` spellings in the walk, `(dev, ino)` as a lineage; each answered here. Revision 2 was rejected by Astra on three P1s: the VFS delegated SQLite's Unix file methods, which open the SHM and check moved files by pathname themselves; every settled admission wrote a permanent hold, so ordinary replication never read complete; a late owner-scoped extension did not reach successors that had already acknowledged; and six P2s: orphan admissions never terminated, `..` and chain edges unspecified, `dup` sharing the directory offset, the platform-link predicate not preserved and terminal links unclassified, the uid producer without a lifecycle, one billing owner not one replicator; each answered here. Revision 1 was rejected by Astra on five P1s: SQLite canonicalises a `/proc` pathname before it opens anything and this driver passes no `nofollow`; `os.Root` confines but re-resolves every name, so a swap between a listing and an inspection still moves the answer; a `Start` row is not an admission, since the meter sends without one; the attempt-derived window lost a request completed before the change was noticed and the days between an admission and a finish; `closed_at` was not a handover; plus the directory open before `O_DIRECTORY`, per-owner holds the ledger cannot store, a uid with no producer, I3 against Part D, and tests). Branch `design-replicator-pinned-root`, off `feat/carloku-metered-replicator-20261001` at `e1280811`. It answers the eleven findings the twentieth review deferred (`DISPOSITIONS-metered-replicator.md`, review twenty) and keeps decisions 10 to 13 and approval H (`DECISIONS-FOR-PAUL.md`, section D, on the launch branch) as parameters. Nothing is built, wired or applied. |
| ## What stands today |
| @@ -50,7 +50,7 @@ The VFS is the plan's first spike and it must prove, before anything else of Par |
| **The fleet journal and the handover.** One row per fleet (`fleets`: `fleet_id`, `box`, `witness_generation`, `owners`, `opened_at`, `state`, `predecessor`, `acknowledged_by`). States and the transaction each is written in: `open` at build, before any admission, with the construction links above; `stale` when an uncertainty is opened; `closing` when the helper has exited and the fleet has released its witness; `closed` when every admission of the fleet is settled and no discovery of the fleet is uncommitted (a discovery whose write failed is retried, and the closure transaction carries any still-pending discovery's uncertainty with it, so a fleet is closed with that uncertainty committed or not closed at all; the recorder exposes a wait on settlement; `closeNow` waits, bounded, and past the bound the row stays `closing`, with its admissions still barring and its uncertainties still open: a late worker is a late settlement, and nothing about it is lost). Before the helper is told to stop, `closeNow` keeps today's ordering: replication stops first, then one last look (the witness, the chains and the files, as any look), then the witness is released; a change found by that last look is a discovery like any other. A successor acknowledges a predecessor only when the predecessor is `closed`, or when recovery has closed it; acknowledgement writes `acknowledged_by`, copies the predecessor's effective uncertainties (its own and inherited, open and closed) into the successor's inherited set, and writes holds for the closed ones the successor's owners have not yet been held for, in one transaction. A `closed` predecessor whose effective set is empty contributes nothing: routine replacement of a healthy chain holds no day. An unopened intermediate fleet (built and closed without replicating) has its construction links and is acknowledged like any other, so the chain's barriers and inheritance pass through it. |
| -**The producer fence, and recovery at process start.** The ledger's own lock does not prove the old process gone: `DurabilityLedger.Close` releases it when its bounded wait ends with work in flight, and a request admitted before could still be sent after. So dispatch is fenced separately: a *producer fence*, an `flock` on `admissions.lock` in the ledger's directory, taken by the agent before its first admission on an open file description it then passes to every helper it starts (over the same `SCM_RIGHTS` message as the directories, before the helper may send anything; a helper that has not received it refuses to dispatch), never released by a timeout and released by the kernel only when the agent and every helper holding that description have exited; every admission's commit requires the fence held (checked under the ledger's mutex before `BEGIN`). A helper that outlives its agent with an admission in hand therefore still holds the fence, and a new agent cannot take it, and so cannot recover, until that helper is gone; a helper's exit is observed by the agent as its child's `waitpid`, never inferred from a closed socket, and `PR_SET_PDEATHSIG` is set in the helper as a convenience that shortens the wait and proves nothing. Recovery requires the fence, which is the verified termination of every old producer. Recovery takes the fence first and, holding it, in one transaction before any fleet is built: every admission of this box that is unsettled is classified `how = recovered-unknown` with all three timestamps `now`, an uncertainty opened and closed for it at once over `[admitted_at - margin, now + margin]` for its owner (box-wide while owner scope is not in use), its days materialised; every fleet not `closed` is marked `closed` with an uncertainty over `[min(opened_at, its earliest admission) - margin, now + margin]` if it had none open, or its open uncertainty closed at `now + margin` and materialised; every fleet with no `acknowledged_by` is acknowledged, root-first, by the first fleet the new process builds, with the inheritance and holds above. Orphans therefore terminate, conservatively, and nothing blocks beyond its recovery; a producer that is somehow still alive cannot send, because the fence is no longer its. |
| +**The producer fence, and recovery at process start.** The ledger's own lock does not prove the old process gone: `DurabilityLedger.Close` releases it when its bounded wait ends with work in flight, and a request admitted before could still be sent after. So dispatch is fenced separately: a *producer fence*, an `flock` on `admissions.lock` in the ledger's directory, taken by the agent before its first admission on an open file description it then passes to every helper it starts (over the same `SCM_RIGHTS` message as the directories, before the helper may send anything; a helper that has not received it refuses to dispatch), never released by a timeout and released by the kernel only when the agent and every helper holding that description have exited; every admission's commit requires the fence held (checked under the ledger's mutex before `BEGIN`). A helper that outlives its agent with an admission in hand therefore still holds the fence, and a new agent cannot take it, and so cannot recover, until that helper is gone; a helper's exit is observed by the agent as its child's `waitpid`, never inferred from a closed socket, and `PR_SET_PDEATHSIG` is set in the helper as a convenience that shortens the wait and proves nothing. Recovery requires the fence, which is the verified termination of every old producer. Recovery takes the fence first and, holding it, in one transaction before any fleet is built: every admission of this box that is unsettled is classified `how = recovered-unknown` with its missing timestamps `now` (a completion recorded before the crash keeps its instant, as the table's recovery edge has it), an uncertainty opened and closed for it at once over `[admitted_at - margin, now + margin]` for its owner (box-wide while owner scope is not in use), its days materialised; every fleet not `closed` is marked `closed` with an uncertainty over `[min(opened_at, its earliest admission) - margin, now + margin]` if it had none open, or its open uncertainty closed at `now + margin` and materialised; every fleet with no `acknowledged_by` is acknowledged, root-first, by the first fleet the new process builds, with the inheritance and holds above. Orphans therefore terminate, conservatively, and nothing blocks beyond its recovery; a producer that is somehow still alive cannot send, because the fence is no longer its. |
| **Part C2, who is held.** The ledger's failure rows and `Incomplete` are box-wide today (`DurabilityClient.Incomplete` discards the client's identity; `CompletenessFor(account, …)` fails on any box failure). Owner-scoped holds are a ledger change: a `holds` table `(day, account, app or empty for the box, uncertainty_id, written_at)`, the publisher's snapshot carrying them, the reader treating a box-wide row as failing every owner and an owner row as failing that owner alone, and older readers of a published day seeing a box-wide incompleteness marker whenever any hold exists (so an older console never reads a held day as complete). The design specifies the table, the reader and the marker and keeps **box-wide the default**: every hold this design writes names the box until decision 10 is taken. Decision 10's implementation boundary is also stated, so it is not mistaken for a switch: per-tenant refusal means per-owner fleets (a fleet per `(account, app)`, each with its own witness view, uncertainties and holds, and a tenant's unacceptable file refusing only its fleet), which is a change to `MeteredFleetStore`'s shape beyond the holds table. |
| @@ -62,7 +62,7 @@ The VFS is the plan's first spike and it must prove, before anything else of Par |
| **Lineage.** A replica key today is `(account, path)`, and an inode number is an observation, not a database: a deleted and recreated file can reuse it across a restart. The lineage is therefore a durable record in the shadow state (platform-owned; one record per replica key, written with a protocol this design states, since the shadow state has none to borrow (the fork's LTX writes sync the file and rename it without syncing the directory, and the shadow directory's creation syncs nothing): the record is written to a temporary name in the shadow directory, `fsync`ed, renamed over its final name, the directory `fsync`ed, and every directory created for it `fsync`ed in its parent at creation; replication authority for the key is granted only after the directory's `fsync` returns, so no replica write precedes a lineage whose name and ancestry are durable; the migration that adopts existing prefixes is marked by a record written the same way, and once that marker exists a key whose replica data exists with no record or a corrupt record is a fault, not a first build: its replication authority is withheld and an operator review raised, never re-adopted): `lineage_id` (random), `owner`, the accepted spelling, `(dev, ino)` and the file's birth time as `statx` reports it with `STATX_BTIME` set. Continuity means the same filesystem incarnation of the file, nothing more: within one fleet's life another accepted spelling of the same `(dev, ino)` reuses the key (one stream, one file); across a restart a key is resumed only when `(dev, ino)` and a valid birth time match the record; a filesystem that reports no valid birth time, or a mismatch, rotates to a new lineage (a new prefix) after the hold the change causes. A database replaced in place over the same incarnation (a restore written into the same inode) is not distinguished by this record and is not claimed to be; it is the owner's own replacement; what applies to it is the fork's existing verification, which chooses a new snapshot, incremental continuation or an error according to its own checks (a salt change whose last page still matches continues incrementally, for one), and this record neither detects arbitrary in-place replacement nor establishes that any restore is supported. No SQLite header field identifies a database incarnation and none is used. Existing path-based prefixes are adopted as the first lineage of the file found under that path at the first build after this design, recorded then; a later mismatch rotates as above. |
| -**Required coverage.** The platform's required databases are a list: the registry this process has open, the console database when the agent's role configuration says the box is a console box, `CONSOLE_DB` when set. A required database unreachable through its root is a problem that makes the inventory incomplete and is written as a durable box-wide hold for the day before and independent of whether a fleet is built. Before any cutover, a read-only operator command lists this inventory beside the external replicator's configuration. |
| +**Required coverage.** The platform's required databases are a list: the registry this process has open, the console database when the agent's role configuration says the box is a console box, `CONSOLE_DB` when set. A required database unreachable through its root is a problem that makes the inventory incomplete and is written as a durable box-wide hold for the day before and independent of whether a fleet is built. Required coverage is ownership, not reachability: a required database that is reachable but refused by the ownership checks (a hard link, a wrong owner, a route claiming it too) is missing, with the refusal as its reason, and holds the day the same way. Deployment condition: no role configuration names a console box on this branch, so the console database is required only where `CONSOLE_DB` is set in the environment of the process that builds the fleet (the agent's; the variable is the console process's today, so a console box's agent unit must carry it too, or the console database already gone at a build is not held for); a console database at the conventional path that is owned but not required is said by the read-only report, so the gap is visible on the box rather than silent; during a fleet's life the owned console database's disappearance stales the fleet regardless. Before any cutover, a read-only operator command (`carlos ops inventory`) lists this inventory beside the external replicator's live configuration and the list a regeneration would write, opening the registry without creating or migrating it. |
| **Route identity (Astra 10, approval H): a producer and a lifecycle, not a number.** The current whole-app backend starts a unit and checks its health; it reads no uid or invocation, units restart on their own (`Restart=always`, socket activation), and a stale pair would survive an autonomous restart. The design names the mechanism: the identity of a route's running workload is read from systemd, not inferred: the unit observer the activator already runs for isolated exec is extended to every unit route and reads, at every start, stop and restart it is told of (D-Bus `PropertiesChanged` on the unit, with a poll of `systemctl show -p InvocationID,MainPID,ActiveState` as the fallback), the `InvocationID` and the main process's uid from `/proc/<MainPID>/status`; it writes `route_uid` and `invocation_id` on the route row, under the ownership witness (the generation covers the columns), as a conditional write keyed on the previously recorded invocation (a write that finds another invocation recorded clears both and lets the next observation set them); it clears both at stop. A claim is honoured only when the route's recorded invocation is current (the observer's last reading agrees with systemd now) and the claim root's owning uid equals `route_uid`; unknown identity refuses the claim. Hibernation capability is a route property; a hibernating route that is awake has an invocation and claims as a live unit; one that is asleep has none, and reaching its private directory through the platform root proves which directory was reached, not that this route owns it, so a sleeping route claims only against a **binding** the observer recorded while it was awake: `state_dir_identity`, the `(dev, ino, birth time)` of the state directory the live workload's uid owned at the observer's last coherent reading, with that uid and the platform link's target spelling, written on the route row under the witness beside `route_uid` and `invocation_id` and kept across sleep; the binding survives an ordinary sleep (the observer clears `route_uid` and `invocation_id` at stop and leaves the binding) and is invalidated by these and only these: the route removed; its `db_path`, `backing` or hibernation capability changed; its state directory removed or recreated (the observer sees the identity or birth time differ at the next start); the live workload's uid differing from the binding's at the next start; a sleeping route with no binding, whose private spelling resolves to a directory of another identity or birth time or owner, or whose filesystem reports no valid birth time, is refused: continuity that cannot be established is refused, as with lineage below. A coherent reading is: `InvocationID` and `MainPID` from systemd, the uid from `/proc/<MainPID>/status`, then `InvocationID` and `MainPID` from systemd again, equal to the first; a reading that is not coherent is retried and, failing, leaves the route unknown. Freshness is checked where claims are made: at the inventory's build and at every look (`lookNow`), by comparing the recorded invocation with systemd's current one, and a mismatch stales the fleet directly (the observer may not have written the change yet, so the witness is not relied on for it); it is not checked per admission, and the bound is stated as the looks give it: freshness holds as of each completed look, and an uncertainty opened by a look reaches back to the last clean look's start (the lower bound), so a unit restarted between two looks is held from the look before the change; the watch's interval is when the next look is scheduled, not a deadline on when it completes. An exec or named-file claim of a non-unit route (the activator's own processes) is claimed under the activator's identity, which is a trusted workload and so a parameter. Which uids are the platform's own workloads, and whether a root-run unit may claim, are `trustedUIDs` (empty) and approval H. |
diff --git a/docs/superpowers/specs/2026-10-03-replicator-pinned-roots-spikes.md b/docs/superpowers/specs/2026-10-03-replicator-pinned-roots-spikes.md| index 11b67e0d8..8955519e7 100644 |
| --- a/docs/superpowers/specs/2026-10-03-replicator-pinned-roots-spikes.md |
| +++ b/docs/superpowers/specs/2026-10-03-replicator-pinned-roots-spikes.md |
| @@ -22,7 +22,7 @@ Plan step 0's second spike is code, kept in tree at `internal/ops/admissionmodel |
| - **The boundaries are separate actions** (after the first code review): the writer's commit, the gate's answer (the recheck), a definite failure, an ambiguous return and its read-back, the send (the helper beginning the transport, which needs a durable issued row, recorded as a violation otherwise; a dead helper's send is a no-op, since a dead helper does nothing), the helper's proven exit (the guard closes) and the drain (transport-lost, unrecorded) are each their own step, so a schedule can put a discovery, a budget expiry or a crash between a commit and its answer, hold a token for a day before sending, or ask for a recorder's `Start` between the exit and the drain. The composites the named tests use (`WriterReturns`, `HelperDies`) are those steps in order. |
| - **Randomised schedules**: every action above in random interleavings over about four days, 120 steps a schedule, one owner per fleet (owner scope; the box-wide default is a superset); after each step, and right after every recovery, in the schedule and at quiescence, before anything else runs: the barrier, stated from the chain and not from the publisher (for every fleet and every owner of it, a durable unsettled admission of that fleet or of any fleet in its unacknowledged ancestry bars the owner from its day on, whether or not the token was ever handed out), the hazard invariant (once the ledger knows of a change, every day on which a transport overlapped the true change is incomplete for its owner; after a crash, every sent admission unsettled at the crash likewise; an issued token never sent has no transport and is covered by the barrier and recovery, not this oracle), a terminal state never changing, and a closed fleet having no latch or pending registration; at quiescence every fleet's owners hold every interval of every uncertainty on its whole chain, no uncertainty is open, and a day with no uncertainty reaching an owner is complete. `ADMISSION_MODEL_SEEDS=30000` passes in about six seconds; the default is 400. Publication is revisable by construction: a day published complete can be re-marked by a later hold (the design's materialisation and watermark), and the hazard invariant is satisfied by that retraction; the barrier's purpose is to make retraction rare, not to make it unnecessary. |
| - **What the runs found.** The first runs found the model's own bookkeeping wrong (an owner the fleet did not have; a crash-killed helper's transport end unset). After the second code review made the change its own action, discovered by a later look, Opus's sequence (an admission admitted before the last clean look, in flight across a change after it, settled clean before the discovery) showed the design's lower bound missing that admission's earlier day: **revision 9 of the design** adds the admission instant of every admission settled at or after the last clean look's start to the bound, and the mutant restoring revision 8's rule is killed by the named test. That is the one interleaving the design did not name that the model has found, within the actions it represents and the invariants it checks; the claim is no wider than that. |
| -- **Mutants, in a tree copy**, each killed by a named test or a schedule. After the fourth code review: the historical term filtered to finished and abandoned (the unrecorded test), the cutoff as the running minimum (the fixed-bound test; the a1-first order kills it, the reversed order is the control), the historical term using the send instant, the historical term dropped, a later witness change ignored (the proof test). After the fifth: `>=` at the bound as `>` (the at-the-bound test), a witness change overwriting the first change's instant, and the invariant check right after a recovery is an oracle, not a mechanism: removing it fails nothing today, and it is kept so a recovery-mechanism fault shows at once rather than after later work repairs it. After the third code review: the threshold as the later of proof and look, as the look alone, the historical term including empty terminals, the proof not refusing a moved witness, the crash duplicating a landed row, `Close` without its unsettled loop, the latch overlay reaching the fleet's owners only. After the second code review: revision 8's bound (the in-flight test), the bound without the proof (the proof test), the bound without the clean look (the lower-bound test), a landed ambiguous row not durable until read-back, the open interval dropped from construction backfill (the per-step inheritance check), the replay's per-token writes dropped (the replay-while-open test), a read-back leaving the pending set entry. Removing the closed-implies-no-unsettled *check* removes an oracle; the mechanism mutant it exists for, `Close` without its unsettled-admissions loop, is killed by it (an unanswered durable row). Putting the send's terminal check before its gate is equivalent in the schedules tested (a settled, issued, unsent token exists after a clean stop, but no schedule sends it afterwards). After the first code review: the budget expiry a no-op (its own test, with no discovery); the barrier owner-only, or over issued rows only (the successor-barred test and the schedules); the pending set skipped at registration; the guard ignored, and the drain before the guard; the replay clearing the latch without the lower bound's holds (a named test: the schedules did not reach it in 3000 seeds); the send unchecked; a close without the replay (the closed-implies-no-latch check); acknowledgement copying the predecessor's own uncertainties only (not equivalent: a schedule found a fifth fleet's owner unheld for the first fleet's uncertainty, because the chain's copies are how a closed interval reaches a fleet built after its predecessor acknowledged); the answer ignoring the refusal; a read-back while the writer runs (the named test; the later commit would land on a cancelled registration, which the terminal check reports). Equivalent under this model: the unacknowledged-ancestry walk not stopping at an acknowledged step (over-reach writes the same holds). Not equivalent, though once claimed so here: the registry proof in the bound (Astra's schedule: a build, a clean look a day later, a discovery a day after that, with no admission: only the proof reaches the first day; and Opus's: a witness change known only to be after the proof; both are named tests). First: no construction backfill (reach); the pending set not counted (the commit-after-discovery tests); no recheck after commit (the same test, and a hazard at seed 264); the replay dropping counted settlements (hazard); the replay closing at the discovery instant (liveness and the latch tests); a settlement writing no interval (hazard); recovery ignoring the fence (the fence test); helper death without transport-lost (an uncertainty open at quiescence); the lower bound ignoring the clean look (the lower-bound test; the 30000-schedule sweep alone did not find it, which is why that sequence is a named test); the lower bound ignoring the earliest unsettled admission (liveness, because the model's proof then reached too far); unrecorded synthesised for a registered worker (the guard test); recovery skipping unclosed fleets (an uncertainty open at quiescence); acknowledgement not root-first and reach ignoring successors (reach). The equivalent mutants are named where they arise (the epoch check made redundant by the stale state; the unacknowledged-ancestry walk's over-reach; the send's terminal check before its gate), and the registry proof is not one. |
| +- **Mutants, in a tree copy**, each killed by a named test or a schedule. After the eighth code review: `RecorderDone(finished|abandoned)` without `RecorderStart` (the guard test, and the schedules through a new check, a recorder outcome without a registration, which the schedule now attempts before Start); equivalent under this model: `GateAnswers` without its state check (`answered` is set only by the gate and nothing else leaves `admitted` before the answer), and the settlement's clamp to now (every participant instant is the model's own clock at its step, so the later completion is now; the ledger takes reported instants, where the clamp is live and its removal is killed by the ledger's clamp test). After the fourth code review: the historical term filtered to finished and abandoned (the unrecorded test), the cutoff as the running minimum (the fixed-bound test; the a1-first order kills it, the reversed order is the control), the historical term using the send instant, the historical term dropped, a later witness change ignored (the proof test). After the fifth: `>=` at the bound as `>` (the at-the-bound test), a witness change overwriting the first change's instant, and the invariant check right after a recovery is an oracle, not a mechanism: removing it fails nothing today, and it is kept so a recovery-mechanism fault shows at once rather than after later work repairs it. After the third code review: the threshold as the later of proof and look, as the look alone, the historical term including empty terminals, the proof not refusing a moved witness, the crash duplicating a landed row, `Close` without its unsettled loop, the latch overlay reaching the fleet's owners only. After the second code review: revision 8's bound (the in-flight test), the bound without the proof (the proof test), the bound without the clean look (the lower-bound test), a landed ambiguous row not durable until read-back, the open interval dropped from construction backfill (the per-step inheritance check), the replay's per-token writes dropped (the replay-while-open test), a read-back leaving the pending set entry. Removing the closed-implies-no-unsettled *check* removes an oracle; the mechanism mutant it exists for, `Close` without its unsettled-admissions loop, is killed by it (an unanswered durable row). Putting the send's terminal check before its gate is equivalent in the schedules tested (a settled, issued, unsent token exists after a clean stop, but no schedule sends it afterwards). After the first code review: the budget expiry a no-op (its own test, with no discovery); the barrier owner-only, or over issued rows only (the successor-barred test and the schedules); the pending set skipped at registration; the guard ignored, and the drain before the guard; the replay clearing the latch without the lower bound's holds (a named test: the schedules did not reach it in 3000 seeds); the send unchecked; a close without the replay (the closed-implies-no-latch check); acknowledgement copying the predecessor's own uncertainties only (not equivalent: a schedule found a fifth fleet's owner unheld for the first fleet's uncertainty, because the chain's copies are how a closed interval reaches a fleet built after its predecessor acknowledged); the answer ignoring the refusal; a read-back while the writer runs (the named test; the later commit would land on a cancelled registration, which the terminal check reports). Equivalent under this model: the unacknowledged-ancestry walk not stopping at an acknowledged step (over-reach writes the same holds). Not equivalent, though once claimed so here: the registry proof in the bound (Astra's schedule: a build, a clean look a day later, a discovery a day after that, with no admission: only the proof reaches the first day; and Opus's: a witness change known only to be after the proof; both are named tests). First: no construction backfill (reach); the pending set not counted (the commit-after-discovery tests); no recheck after commit (the same test, and a hazard at seed 264); the replay dropping counted settlements (hazard); the replay closing at the discovery instant (liveness and the latch tests); a settlement writing no interval (hazard); recovery ignoring the fence (the fence test); helper death without transport-lost (an uncertainty open at quiescence); the lower bound ignoring the clean look (the lower-bound test; the 30000-schedule sweep alone did not find it, which is why that sequence is a named test); the lower bound ignoring the earliest unsettled admission (liveness, because the model's proof then reached too far); unrecorded synthesised for a registered worker (the guard test); recovery skipping unclosed fleets (an uncertainty open at quiescence); acknowledgement not root-first and reach ignoring successors (reach). The equivalent mutants are named where they arise (the epoch check made redundant by the stale state; the unacknowledged-ancestry walk's over-reach; the send's terminal check before its gate), and the registry proof is not one. |
| - **What the model does not cover**, by construction: the latch clearing after the commit as two instants (the replay is one step here; the rule stays with the ledger's tests), a snapshot between those instants, the recorder worker's lifetime and the socket drain as separate events, the fence as anything but helper liveness, retries of a failed terminal or replay write, the ledger's commit latency on a box's disk under the replicator's request rate (the spike's other half, which needs a box), Part C2's owner scope beyond the owner-scoped holds the model writes, and process kills inside a transaction (the model's transactions are atomic by fiat; the ledger's are SQLite's). |
| ## The fork's pathname entry points (read 2026-10-03 08:25, every call, not a sample) |
diff --git a/internal/ops/admissionmodel/model.go b/internal/ops/admissionmodel/model.go| index 2bbd9c9cc..7e71c218d 100644 |
| --- a/internal/ops/admissionmodel/model.go |
| +++ b/internal/ops/admissionmodel/model.go |
| @@ -1004,6 +1004,13 @@ func (m *Model) Check() error { |
| m.final[a.Token] = a.State |
| } |
| } |
| + // finished and abandoned are a registered worker's outcomes: a |
| + // recorder outcome without a Start is not an edge of the table |
| + for _, a := range m.Admissions { |
| + if (a.How == "finished" || a.How == "abandoned") && a.RecorderStartedAt == unset { |
| + return fmt.Errorf("T%d %s with no recorder registered", a.Token, a.How) |
| + } |
| + } |
| // one durable row per token |
| for _, f := range m.Fleets { |
| seen := map[int]bool{} |
diff --git a/internal/ops/admissionmodel/model_test.go b/internal/ops/admissionmodel/model_test.go| index 482c1f23d..f1a53e3ad 100644 |
| --- a/internal/ops/admissionmodel/model_test.go |
| +++ b/internal/ops/admissionmodel/model_test.go |
| @@ -377,6 +377,14 @@ func TestHelperDeathAndRecorderGuard(t *testing.T) { |
| f := m.Build(alice) |
| never := issue(t, m, f, alice) |
| live := issue(t, m, f, alice) |
| + // finished and abandoned are a registered worker's outcomes: before |
| + // Start they are not edges, and the admission is unchanged |
| + for _, how := range []string{"finished", "abandoned"} { |
| + m.RecorderDone(never, how) |
| + if never.RecorderDoneAt != unset || never.How != "" || never.State != Admitted { |
| + t.Fatalf("%s accepted with no recorder registered: %s %q", how, never.State, never.How) |
| + } |
| + } |
| m.RecorderStart(live) |
| m.Advance(1) |
| m.HelperDies(f, false) |
| @@ -1193,9 +1201,12 @@ func runSchedule(seed int64, steps int) (err error) { |
| case 9, 10: |
| if a := pick(recorderless); a != nil { |
| if a.RecorderStartedAt == unset { |
| - if m.rnd.Intn(5) == 0 { |
| + switch m.rnd.Intn(7) { |
| + case 0: |
| m.RecorderDone(a, "unrecorded") |
| - } else { |
| + case 1: |
| + m.RecorderDone(a, []string{"finished", "abandoned"}[m.rnd.Intn(2)]) // not an edge before Start: a no-op the check watches |
| + default: |
| m.RecorderStart(a) |
| } |
| } else { |
diff --git a/internal/ops/authoritybridge.go b/internal/ops/authoritybridge.go| index ee52ccf1e..268a2588f 100644 |
| --- a/internal/ops/authoritybridge.go |
| +++ b/internal/ops/authoritybridge.go |
| @@ -37,6 +37,10 @@ type fleetPins struct { |
| slash *authority.Dir |
| pins *authority.Pins |
| roots map[string]*authority.Root |
| + // listed is the facts of every database the strict tree walk listed, |
| + // by its spelled path: the identity the ownership checks must find |
| + // again (the listing-to-inspection rule across the handoff). |
| + listed map[string]authority.Facts |
| } |
| func newFleetPins() (*fleetPins, error) { |
| @@ -44,7 +48,7 @@ func newFleetPins() (*fleetPins, error) { |
| if err != nil { |
| return nil, err |
| } |
| - p := &fleetPins{slash: slash, roots: map[string]*authority.Root{}} |
| + p := &fleetPins{slash: slash, roots: map[string]*authority.Root{}, listed: map[string]authority.Facts{}} |
| p.pins = &authority.Pins{Slash: slash, Platform: platformFacts} |
| return p, nil |
| } |
| @@ -502,7 +506,7 @@ func (c *retainedChain) recheck() error { |
| // did: a difference is a change the fleet goes stale on. |
| func (p *fleetPins) recheck() error { |
| for _, r := range p.roots { |
| - if changed, err := r.AncestryChanged(); err != nil || changed { |
| + if changed, err := ancestryChanged(r); err != nil || changed { |
| if err == nil { |
| err = authority.ErrChanged |
| } |
| @@ -512,6 +516,9 @@ func (p *fleetPins) recheck() error { |
| return nil |
| } |
| +// ancestryChanged is a test seam over Root.AncestryChanged. |
| +var ancestryChanged = func(r *authority.Root) (bool, error) { return r.AncestryChanged() } |
| + |
| // isChange reports whether a recheck's error is a change of what was |
| // retained (an edge that differs, a link or something else where a |
| // directory was, a name gone, a root failing the platform rule), as |
diff --git a/internal/ops/litestream.go b/internal/ops/litestream.go| index 7a8bb1175..3d405f3a5 100644 |
| --- a/internal/ops/litestream.go |
| +++ b/internal/ops/litestream.go |
| @@ -512,6 +512,17 @@ func fleetDBsFromRoutes(routes []registry.Route, registryPath, dataDir string, t |
| if seen[p] { |
| continue // a unit DBPath that happens to also sit under dataDir |
| } |
| + if tree != nil && discovered[p] { |
| + // listed by the strict walk on retained descriptors as a |
| + // regular, non-empty database: not looked at again by name |
| + // here; the ownership checks look at it through its chain and |
| + // refuse it if it is not the file listed |
| + if !hibernating[p] { |
| + seen[p] = true |
| + dbs = append(dbs, p) |
| + continue |
| + } |
| + } |
| fi, err := os.Stat(p) |
| if err != nil || !fi.Mode().IsRegular() || fi.Size() == 0 { |
| // Gone, odd, or an empty placeholder: nothing to replicate. For |
diff --git a/internal/ops/litestream_metered.go b/internal/ops/litestream_metered.go| index 5bdc7711b..6b9efa5e7 100644 |
| --- a/internal/ops/litestream_metered.go |
| +++ b/internal/ops/litestream_metered.go |
| @@ -103,8 +103,10 @@ type MeteredFleetOptions struct { |
| // fleet was being built: the caller binds it to the box's durability |
| // ledger (DurabilityClient.Incomplete), so the day is held, durably |
| // and box-wide, whether or not a fleet is then built. The build is |
| - // refused as well, since the inventory is incomplete. |
| - Hold func(at time.Time) |
| + // refused as well, since the inventory is incomplete. An error is the |
| + // hold not written: it is said in the refusal, since the day is then |
| + // not held durably. |
| + Hold func(at time.Time) error |
| // Replaces is the fleet this one is built to take over from, if any. |
| // The old fleet must be closed before this one is opened, and closing |
| // it looks at the files and asks the witness one last time each (a |
| @@ -911,7 +913,9 @@ func MeteredFleetStore(o MeteredFleetOptions) (fleet *MeteredFleet, err error) { |
| return nil, err |
| } |
| if missing := inv.RequiredMissing(); len(missing) > 0 && o.Hold != nil { |
| - o.Hold(time.Now()) |
| + if herr := o.Hold(time.Now()); herr != nil { |
| + inv.Problems = append(inv.Problems, "the box's day could not be held for the required databases that cannot be reached ("+herr.Error()+")") |
| + } |
| } |
| if len(inv.Problems) > 0 { |
| inv.Close() |
diff --git a/internal/ops/litestream_metered_test.go b/internal/ops/litestream_metered_test.go| index d6a37fe2b..13d916371 100644 |
| --- a/internal/ops/litestream_metered_test.go |
| +++ b/internal/ops/litestream_metered_test.go |
| @@ -5,6 +5,7 @@ import ( |
| "context" |
| "database/sql" |
| "errors" |
| + "github.com/carlosframework/platform/internal/authority" |
| "io" |
| "net/http" |
| "net/http/httptest" |
| @@ -2019,7 +2020,9 @@ func TestMeteredFleetStalesWhenAChainEdgeChanges(t *testing.T) { |
| if err := send(t, fleet, db, 3); err != nil { |
| t.Fatalf("a write after a clean look: %v", err) |
| } |
| - // a look that cannot be taken (no change) is drift, not stale |
| + // a look that cannot be taken (no change) is drift, not stale; the |
| + // hold a later change opens starts at the clean look before it |
| + cleanBefore := fleet.lastCleanLook() |
| recheckHeld = func(*FleetInventory) error { return syscall.EIO } |
| fleet.lookAgain() |
| recheckHeld = func(inv *FleetInventory) error { return inv.recheck() } |
| @@ -2029,7 +2032,9 @@ func TestMeteredFleetStalesWhenAChainEdgeChanges(t *testing.T) { |
| if !fleet.drift.Load() { |
| t.Fatal("a look that could not be taken did not record drift") |
| } |
| - cleanBefore := fleet.lastCleanLook() |
| + if !fleet.lastCleanLook().Equal(cleanBefore) { |
| + t.Fatal("a look that could not be taken counted as a clean look") |
| + } |
| if err := os.Chmod(filepath.Join(state, "data"), 0o700); err != nil { // a mode change on the chain: only the retained edge sees it |
| t.Fatal(err) |
| } |
| @@ -2059,12 +2064,111 @@ func openDirFDsUnder(t *testing.T, prefix string) int { |
| n := 0 |
| for _, e := range ents { |
| target, err := os.Readlink(filepath.Join("/proc/self/fd", e.Name())) |
| - if err != nil || !strings.HasPrefix(target, prefix+string(filepath.Separator)) && target != prefix { |
| + if err != nil { |
| + continue |
| + } |
| + target = strings.TrimSuffix(target, " (deleted)") // an open directory renamed away or removed still counts |
| + if !strings.HasPrefix(target, prefix+string(filepath.Separator)) && target != prefix { |
| + continue |
| + } |
| + // the descriptor itself, not its textual target: what it is open |
| + // on is what counts, whatever that name leads to now |
| + fd, err := strconv.Atoi(e.Name()) |
| + if err != nil { |
| continue |
| } |
| - if fi, err := os.Stat(target); err == nil && fi.IsDir() { |
| + var st syscall.Stat_t |
| + if syscall.Fstat(fd, &st) == nil && st.Mode&syscall.S_IFMT == syscall.S_IFDIR { |
| n++ |
| } |
| } |
| return n |
| } |
| + |
| +// Review eight (Opus P3-1): the other change classes. A chain directory |
| +// renamed away (ENOENT at the retained edge, no replacement) stales the |
| +// fleet from the last clean look. |
| +func TestMeteredFleetStalesWhenAChainDirectoryIsRenamedAway(t *testing.T) { |
| + newFakeBucket(t) |
| + dir := platformDir(t) |
| + reg := openRegistry(t, filepath.Join(dir, "registry.db")) |
| + state := filepath.Join(platformDir(t), "unit-a") |
| + db := writeFile(t, filepath.Join(state, "data", "main.db")) |
| + upsert(t, reg, registry.Route{Host: "a.example.com", Account: "bac", App: "one", DBPath: state, Backing: registry.BackingUnit, Version: "v1"}) |
| + fleet, err := MeteredFleetStore(fleetOptions(reg, dir)) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + defer fleet.Close() |
| + if err := send(t, fleet, db, 2); err != nil { |
| + t.Fatalf("a write before any change: %v", err) |
| + } |
| + cleanBefore := fleet.lastCleanLook() |
| + if err := os.Rename(filepath.Join(state, "data"), filepath.Join(state, "gone")); err != nil { |
| + t.Fatal(err) |
| + } |
| + fleet.lookAgain() |
| + if err := send(t, fleet, db, 3); !errors.Is(err, ErrOwnershipStale) { |
| + t.Fatalf("a write after a chain directory was renamed away: %v, want stale", err) |
| + } |
| + if from, _, stale := fleet.staleWindow(); !stale || !from.Equal(cleanBefore.Add(-ownershipMargin)) { |
| + t.Fatalf("the hold starts at %v, want the last clean look %v minus the margin", from, cleanBefore) |
| + } |
| + if fleet.drift.Load() { |
| + t.Fatal("a name gone is a change, not drift") |
| + } |
| +} |
| + |
| +// Review eight (Opus P3-1): drift at the final look, where there is no |
| +// look again, stales the fleet at Close from the last clean look; and a |
| +// change wins over drift: a root whose ancestry cannot be looked at beside |
| +// a chain edge that changed is stale at once, not drift. |
| +func TestMeteredFleetDriftAtTheFinalLookStalesAndChangeWinsOverDrift(t *testing.T) { |
| + newFakeBucket(t) |
| + dir := platformDir(t) |
| + reg := openRegistry(t, filepath.Join(dir, "registry.db")) |
| + state := filepath.Join(platformDir(t), "unit-a") |
| + db := writeFile(t, filepath.Join(state, "data", "main.db")) |
| + upsert(t, reg, registry.Route{Host: "a.example.com", Account: "bac", App: "one", DBPath: state, Backing: registry.BackingUnit, Version: "v1"}) |
| + fleet, err := MeteredFleetStore(fleetOptions(reg, dir)) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := send(t, fleet, db, 2); err != nil { |
| + t.Fatalf("a write before any change: %v", err) |
| + } |
| + cleanBefore := fleet.lastCleanLook() |
| + recheckHeld = func(*FleetInventory) error { return syscall.EIO } |
| + t.Cleanup(func() { recheckHeld = func(inv *FleetInventory) error { return inv.recheck() } }) |
| + if err := fleet.Close(); err != nil { |
| + t.Fatal(err) |
| + } |
| + if from, _, stale := fleet.staleWindow(); !stale || !from.Equal(cleanBefore.Add(-ownershipMargin)) { |
| + t.Fatalf("after a final look that could not be taken: stale=%v from %v, want stale from the last clean look %v minus the margin", stale, from, cleanBefore) |
| + } |
| + recheckHeld = func(inv *FleetInventory) error { return inv.recheck() } |
| + |
| + // change over drift: every root's ancestry look fails, and a chain |
| + // edge changed |
| + fleet2, err := MeteredFleetStore(fleetOptions(reg, dir)) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + defer fleet2.Close() |
| + if err := send(t, fleet2, db, 3); err != nil { |
| + t.Fatalf("a write before any change: %v", err) |
| + } |
| + clean2 := fleet2.lastCleanLook() |
| + ancestryChanged = func(*authority.Root) (bool, error) { return false, syscall.EIO } |
| + t.Cleanup(func() { ancestryChanged = func(r *authority.Root) (bool, error) { return r.AncestryChanged() } }) |
| + if err := os.Chmod(filepath.Join(state, "data"), 0o700); err != nil { |
| + t.Fatal(err) |
| + } |
| + fleet2.lookAgain() |
| + if err := send(t, fleet2, db, 4); !errors.Is(err, ErrOwnershipStale) { |
| + t.Fatalf("a write after a chain edge changed beside a root that could not be looked at: %v, want stale (change wins over drift)", err) |
| + } |
| + if from, _, stale := fleet2.staleWindow(); !stale || !from.Equal(clean2.Add(-ownershipMargin)) { |
| + t.Fatalf("the hold starts at %v, want the last clean look %v minus the margin", from, clean2) |
| + } |
| +} |
diff --git a/internal/ops/litestream_owners.go b/internal/ops/litestream_owners.go| index 2549c33d0..4adfa7c6a 100644 |
| --- a/internal/ops/litestream_owners.go |
| +++ b/internal/ops/litestream_owners.go |
| @@ -97,7 +97,7 @@ type RequiredDB struct { |
| // configuration exists on this branch, so CONSOLE_DB is the only way a |
| // console database is required here, and the conventional path is only |
| // listed when it exists (as fleetInventory lists it today). |
| -func RequiredDatabases(registryPath, dataDir string) []string { |
| +func RequiredDatabases(registryPath string) []string { |
| var out []string |
| if registryPath != "" { |
| out = append(out, filepath.Clean(registryPath)) |
| @@ -162,23 +162,39 @@ func (inv *FleetInventory) Descriptors() int { |
| } |
| // recheck is the fleet's look at what it retains: every pinned root's |
| -// ancestry and every owned file's chain and name. The first difference is |
| -// returned with its edge. |
| +// ancestry and every owned file's chain and name. Everything is visited: |
| +// a change anywhere is returned (with its edge) over a look that could |
| +// not be taken elsewhere, so an EMFILE on one root never hides a changed |
| +// chain behind it; with no change, the first failed look is returned. |
| func (inv *FleetInventory) recheck() error { |
| if inv.pins == nil { |
| return nil |
| } |
| - if err := inv.pins.recheck(); err != nil { |
| - return err |
| + var first error |
| + note := func(err error) bool { |
| + if err == nil { |
| + return false |
| + } |
| + if isChange(err) { |
| + first = err |
| + return true |
| + } |
| + if first == nil { |
| + first = err |
| + } |
| + return false |
| + } |
| + if note(inv.pins.recheck()) { |
| + return first |
| } |
| for _, f := range inv.facts { |
| if f.keep != nil { |
| - if err := f.keep.recheck(); err != nil { |
| - return err |
| + if note(f.keep.recheck()) { |
| + return first |
| } |
| } |
| } |
| - return nil |
| + return first |
| } |
| // fleetInventory is FleetDBs with an owner for each database, for the |
| @@ -221,7 +237,10 @@ func fleetInventory(routes []registry.Route, registryPath, dataDir string) (Flee |
| // Required coverage: each required database must be reachable through |
| // its root as a regular file, before and independent of whether a |
| // fleet is built over the rest. |
| - for _, p := range RequiredDatabases(registryPath, dataDir) { |
| + if inventoryAfterDiscovery != nil { |
| + inventoryAfterDiscovery() |
| + } |
| + for _, p := range RequiredDatabases(registryPath) { |
| why := "" |
| if id, lerr := pins.lookAt(p); lerr != "" { |
| why = lerr |
| @@ -243,7 +262,13 @@ func fleetInventory(routes []registry.Route, registryPath, dataDir string) (Flee |
| if registryPath != "" { |
| platform[filepath.Clean(registryPath)] = OwnedDB{Account: release.OpsAccount, App: PlatformRegistryApp, Host: PlatformRegistryHost, Platform: true} |
| } |
| - unowned := func(p, why string) { inv.Unowned = append(inv.Unowned, p+": "+why) } |
| + whyUnowned := map[string]string{} |
| + unowned := func(p, why string) { |
| + inv.Unowned = append(inv.Unowned, p+": "+why) |
| + if _, seen := whyUnowned[p]; !seen { |
| + whyUnowned[p] = why |
| + } |
| + } |
| // The file each listed path is. Two listed paths that are one file are |
| // both unowned: one stream must never be written twice, nor two |
| // owners billed for it. |
| @@ -259,6 +284,13 @@ func fleetInventory(routes []registry.Route, registryPath, dataDir string) (Flee |
| unowned(p, why) |
| continue |
| } |
| + // the identity rule across the handoff: a file the tree walk listed |
| + // is the file looked at here, or it is refused (the listed |
| + // directory swapped for another holding a file of the same name) |
| + if lf, ok := pins.listed[p]; ok && !lf.Same(id) { |
| + unowned(p, "it changed under the look (the file looked at is not the file listed)") |
| + continue |
| + } |
| files = append(files, listed{p, id}) |
| } |
| for i, f := range files { |
| @@ -283,6 +315,12 @@ func fleetInventory(routes []registry.Route, registryPath, dataDir string) (Flee |
| continue |
| } |
| facts, why := checkOwnedFileOn(pins, filepath.Dir(p), p, platformDirectory) |
| + if why == "" && (facts.Dev != f.id.Dev || facts.Ino != f.id.Ino) { |
| + why = "it changed under the look (the file checked is not the file looked at)" |
| + if facts.keep != nil { |
| + facts.keep.release() |
| + } |
| + } |
| if why != "" { |
| unowned(p, why) |
| continue |
| @@ -322,8 +360,12 @@ func fleetInventory(routes []registry.Route, registryPath, dataDir string) (Flee |
| // later checks. |
| var facts fileFacts |
| refused := false |
| + want := f.id |
| for i, c := range claimChecks(claims, p, dataDir) { |
| f, why := checkOwnedFileOn(pins, c.root, p, c.kind) |
| + if why == "" && (f.Dev != want.Dev || f.Ino != want.Ino) { |
| + why = "it changed under the look (the file checked is not the file looked at)" |
| + } |
| if why == "" && i > 0 && (f.Dev != facts.Dev || f.Ino != facts.Ino) { |
| why = "it changed under the look (its claims' checks did not all see one file)" |
| } |
| @@ -374,11 +416,34 @@ func fleetInventory(routes []registry.Route, registryPath, dataDir string) (Flee |
| kept = append(kept, o) |
| } |
| inv.Owned = kept |
| + // Required coverage is ownership, not reachability: a required |
| + // database that is reachable but refused (a hard link, a wrong owner, |
| + // a route claiming it too) is missing, with the refusal as its reason, |
| + // so the day is held and the build refused for it. |
| + ownedNow := map[string]bool{} |
| + for _, o := range inv.Owned { |
| + ownedNow[o.Path] = true |
| + } |
| + for i := range inv.Required { |
| + r := &inv.Required[i] |
| + if r.Why != "" || ownedNow[r.Path] { |
| + continue |
| + } |
| + r.Why = whyUnowned[r.Path] |
| + if r.Why == "" { |
| + r.Why = "it is not in the inventory" |
| + } |
| + inv.Problems = append(inv.Problems, r.Path+": a required database is not owned ("+r.Why+")") |
| + } |
| sort.Slice(inv.Owned, func(i, j int) bool { return inv.Owned[i].Path < inv.Owned[j].Path }) |
| sort.Strings(inv.Unowned) |
| return inv, nil |
| } |
| +// inventoryAfterDiscovery is a test seam between the discovery of the |
| +// fleet's files and the ownership checks over them. |
| +var inventoryAfterDiscovery func() |
| + |
| // routeClaiming is the host of a route that claims p, by naming it or (a |
| // unit route) a directory above it, or "". It is the same predicate the |
| // inventory uses to gather a file's claims. |
| @@ -995,6 +1060,7 @@ func strictTreeDBsOn(pins *fleetPins, root string) (dbs, skipped, problems []str |
| case e.Facts.Size == 0: |
| case strings.HasSuffix(name, ".db") && !hidden && !strings.HasPrefix(name, "."): |
| dbs = append(dbs, p) |
| + pins.listed[p] = e.Facts |
| case sqliteName(name): |
| skipped = append(skipped, p) |
| default: |
diff --git a/internal/ops/litestream_owners_unix_test.go b/internal/ops/litestream_owners_unix_test.go| index c16354de5..259ac14ae 100644 |
| --- a/internal/ops/litestream_owners_unix_test.go |
| +++ b/internal/ops/litestream_owners_unix_test.go |
| @@ -969,7 +969,10 @@ func TestStrictTreeBetweenListingAndLook(t *testing.T) { |
| setupErr = err |
| return |
| } |
| - os.Chmod(filepath.Join(root, "sub"), 0o755) |
| + if err := os.Chmod(filepath.Join(root, "sub"), 0o755); err != nil { |
| + setupErr = err |
| + return |
| + } |
| if err := os.WriteFile(filepath.Join(root, "sub", "new.db"), []byte("x"), 0o644); err != nil { |
| setupErr = err |
| return |
| @@ -1232,8 +1235,8 @@ func TestRequiredCoverage(t *testing.T) { |
| if len(inv.Problems) != 0 { |
| t.Fatalf("problems with every required database reachable: %v", inv.Problems) |
| } |
| - report, err := InventoryReport(reg, dir) |
| - if err != nil || !strings.Contains(report, "required "+console+" reachable") || !strings.Contains(report, "required: 2 databases") { |
| + report, err := InventoryReport(reg, dir, "") |
| + if err != nil || !strings.Contains(report, "required "+strconv.Quote(console)+" reachable and owned") || !strings.Contains(report, "required: 2 databases") { |
| t.Fatalf("report: %v\n%s", err, report) |
| } |
| // the console database replaced by a link to itself: not reachable |
| @@ -1250,8 +1253,8 @@ func TestRequiredCoverage(t *testing.T) { |
| if m := inv.RequiredMissing(); len(m) != 1 || m[0] != console || !named(inv.Problems, console, "a required database cannot be reached through its root") || !named(inv.Problems, console, "it is a link") { |
| t.Fatalf("a link in the required database's place: missing %v problems %v", m, inv.Problems) |
| } |
| - report, _ = InventoryReport(reg, dir) |
| - if !strings.Contains(report, "required "+console+" MISSING: it is a link") { |
| + report, _ = InventoryReport(reg, dir, "") |
| + if !strings.Contains(report, "required "+strconv.Quote(console)+" MISSING: it is a link") { |
| t.Fatalf("report:\n%s", report) |
| } |
| // missing altogether: a problem, and a fleet build holds the day once |
| @@ -1260,24 +1263,170 @@ func TestRequiredCoverage(t *testing.T) { |
| os.Remove(console + ".real") |
| var held []time.Time |
| o := fleetOptions(reg, dir) |
| - o.Hold = func(at time.Time) { held = append(held, at) } |
| + o.Hold = func(at time.Time) error { held = append(held, at); return nil } |
| if _, err := MeteredFleetStore(o); err == nil || !strings.Contains(err.Error(), "a required database cannot be reached") { |
| t.Fatalf("a fleet built with a required database missing: %v", err) |
| } |
| if len(held) != 1 { |
| t.Fatalf("the day was held %d times, want once", len(held)) |
| } |
| - // reachable again: no hold, no problem |
| + // a hold that cannot be written is said in the refusal: the day is |
| + // then not held durably, which the operator must know |
| + o.Hold = func(time.Time) error { return errors.New("ledger: disk full") } |
| + if _, err := MeteredFleetStore(o); err == nil || !strings.Contains(err.Error(), "could not be held") || !strings.Contains(err.Error(), "disk full") { |
| + t.Fatalf("a hold that failed to write: %v", err) |
| + } |
| + // reachable again: no hold, no problem, and a fleet builds |
| writeFile(t, console) |
| held = nil |
| + o.Hold = func(at time.Time) error { held = append(held, at); return nil } |
| inv, err = FleetDBOwners(reg, dir) |
| if err != nil || len(inv.RequiredMissing()) != 0 || len(inv.Problems) != 0 { |
| t.Fatalf("restored: %v missing %v problems %v", err, inv.RequiredMissing(), inv.Problems) |
| } |
| - // without CONSOLE_DB only the registry is required |
| + fleet, err := MeteredFleetStore(o) |
| + if err != nil { |
| + t.Fatalf("restored: a fleet is refused: %v", err) |
| + } |
| + fleet.Close() |
| + if len(held) != 0 { |
| + t.Fatalf("the day was held %d times with every required database reachable", len(held)) |
| + } |
| + // required coverage is ownership, not reachability: the console |
| + // database reachable but refused (a route claims it too) is missing, |
| + // with the refusal as its reason; the build holds the day and refuses |
| + upsert(t, reg, registry.Route{Host: "claims.example.com", Account: "bac", App: "one", DBPath: console, Backing: registry.BackingUnit}) |
| + inv, err = FleetDBOwners(reg, dir) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if m := inv.RequiredMissing(); len(m) != 1 || m[0] != console || !named(inv.Problems, console, "a required database is not owned") || !named(inv.Problems, console, "also claimed by the route") { |
| + t.Fatalf("a required database reachable but refused: missing %v problems %v required %+v", m, inv.Problems, inv.Required) |
| + } |
| + report, _ = InventoryReport(reg, dir, "") |
| + if !strings.Contains(report, "required "+strconv.Quote(console)+" MISSING: the platform's own database is also claimed") { |
| + t.Fatalf("report:\n%s", report) |
| + } |
| + held = nil |
| + if _, err := MeteredFleetStore(o); err == nil || !strings.Contains(err.Error(), "a required database is not owned") { |
| + t.Fatalf("a fleet built with a required database refused: %v", err) |
| + } |
| + if len(held) != 1 { |
| + t.Fatalf("the day was held %d times, want once", len(held)) |
| + } |
| + // without CONSOLE_DB only the registry is required, and the report |
| + // says the console database at the conventional path is not |
| t.Setenv("CONSOLE_DB", "") |
| inv, _ = FleetDBOwners(reg, dir) |
| if len(inv.Required) != 1 || inv.Required[0].Path != reg.Path() { |
| t.Fatalf("required without CONSOLE_DB: %+v", inv.Required) |
| } |
| + if got := RequiredDatabases(reg.Path()); len(got) != 1 || got[0] != reg.Path() { |
| + t.Fatalf("RequiredDatabases without CONSOLE_DB: %v", got) |
| + } |
| +} |
| + |
| +// Review eight (Astra 6): the identity rule across the handoff from the |
| +// strict tree walk to the ownership checks. The listed directory swapped |
| +// for another holding a database of the same name between the two: the |
| +// file the checks find is not the file listed, and it is refused rather |
| +// than owned; with the file unchanged it is owned. |
| +func TestFleetInventoryRefusesAFileNotTheOneListed(t *testing.T) { |
| + dir := platformDir(t) |
| + reg := openRegistry(t, filepath.Join(dir, "registry.db")) |
| + state := filepath.Join(platformDir(t), "unit-a") |
| + db := writeFile(t, filepath.Join(state, "sub", "main.db")) |
| + upsert(t, reg, registry.Route{Host: "a.example.com", Account: "bac", App: "one", DBPath: state, Backing: registry.BackingUnit}) |
| + routes, err := reg.List() |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + var setupErr error |
| + swapped := false |
| + inventoryAfterDiscovery = func() { |
| + if swapped { |
| + return |
| + } |
| + swapped = true |
| + if err := os.Rename(filepath.Join(state, "sub"), filepath.Join(state, "gone")); err != nil { |
| + setupErr = err |
| + return |
| + } |
| + if err := os.MkdirAll(filepath.Join(state, "sub"), 0o755); err != nil { |
| + setupErr = err |
| + return |
| + } |
| + if err := os.Chmod(filepath.Join(state, "sub"), 0o755); err != nil { |
| + setupErr = err |
| + return |
| + } |
| + if err := os.WriteFile(db, []byte("another"), 0o644); err != nil { |
| + setupErr = err |
| + } |
| + } |
| + t.Cleanup(func() { inventoryAfterDiscovery = nil }) |
| + before := openDirFDsUnder(t, filepath.Dir(dir)) |
| + inv, err := fleetInventory(routes, reg.Path(), dir) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if setupErr != nil { |
| + t.Fatal(setupErr) |
| + } |
| + if _, owned := ownedByPath(inv)[db]; owned || !named(inv.Unowned, db, "not the file listed") { |
| + t.Fatalf("a file that is not the one listed: owned %+v unowned %v problems %v", inv.Owned, inv.Unowned, inv.Problems) |
| + } |
| + inv.Close() |
| + if n := openDirFDsUnder(t, filepath.Dir(dir)); n != before { |
| + t.Fatalf("%d directory descriptors left after the refusal", n-before) |
| + } |
| + // the same file still there: owned |
| + inventoryAfterDiscovery = nil |
| + inv, err = fleetInventory(routes, reg.Path(), dir) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + defer inv.Close() |
| + if _, owned := ownedByPath(inv)[db]; !owned { |
| + t.Fatalf("the listed file is not owned: unowned %v problems %v", inv.Unowned, inv.Problems) |
| + } |
| +} |
| + |
| +// Review eight (Astra 5): the report compares the platform inventory with |
| +// the external replicator's live configuration, which may name a database |
| +// the registry no longer describes (the shrink guard keeps it) and may |
| +// lack one the registry does; both are said, apart from what a |
| +// regeneration would write; paths are quoted. |
| +func TestInventoryReportComparesTheLiveConfiguration(t *testing.T) { |
| + dir := platformDir(t) |
| + reg := openRegistry(t, filepath.Join(dir, "registry.db")) |
| + owned := writeFile(t, filepath.Join(dir, "a.example.com.db")) |
| + upsert(t, reg, registry.Route{Host: "a.example.com", Account: "bac", App: "one", DBPath: owned}) |
| + stale := filepath.Join(dir, "b.example.com.db") |
| + live := LitestreamConfig([]string{reg.Path(), stale}, "bucket", "prefix", "eu-west-1") |
| + report, err := InventoryReport(reg, dir, live) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + for _, want := range []string{ |
| + "external replicator, live configuration: 2 databases", |
| + " live " + strconv.Quote(stale), |
| + " live but not owned by the platform replicator: " + strconv.Quote(stale), |
| + " owned but not in the live configuration: " + strconv.Quote(owned), |
| + " proposed " + strconv.Quote(owned), |
| + " owned " + strconv.Quote(owned) + " bac/one a.example.com", |
| + "(a separate look)", |
| + } { |
| + if !strings.Contains(report, want) { |
| + t.Fatalf("report lacks %q:\n%s", want, report) |
| + } |
| + } |
| + if strings.Contains(report, "owned but a regeneration would not write it") || strings.Contains(report, "proposed but not owned") { |
| + t.Fatalf("the proposal and the inventory agree, the report says otherwise:\n%s", report) |
| + } |
| + // no live configuration: nothing live, everything owned is said so |
| + report, _ = InventoryReport(reg, dir, "") |
| + if !strings.Contains(report, "live configuration: 0 databases") || !strings.Contains(report, "owned but not in the live configuration: "+strconv.Quote(owned)) { |
| + t.Fatalf("report with no live configuration:\n%s", report) |
| + } |
| } |
diff --git a/internal/ops/ops.go b/internal/ops/ops.go| index bac280379..f4a128931 100644 |
| --- a/internal/ops/ops.go |
| +++ b/internal/ops/ops.go |
| @@ -6,6 +6,8 @@ package ops |
| import ( |
| "github.com/carlosframework/platform/internal/cliflag" |
| + "path/filepath" |
| + "sort" |
| "context" |
| "errors" |
| @@ -352,12 +354,15 @@ func openDeployStore(ctx context.Context) (release.Store, error) { |
| } |
| // runInventory prints the platform replicator's inventory beside the |
| -// external replicator's configured list. Read-only: nothing is written, |
| -// no fleet is built, no hold is placed. |
| +// external replicator's configuration as it stands on the box, and the |
| +// list a regeneration would write. Read-only: the registry is opened |
| +// without creating or migrating it, nothing is written, no fleet is built, |
| +// no hold is placed. |
| func runInventory(args []string) { |
| fs := flag.NewFlagSet("ops inventory", flag.ExitOnError) |
| regPath := fs.String("registry", "", "registry database (default <data>/registry.db)") |
| data := fs.String("data", "", "data directory holding the fleet's *.db files (default $CARLOS_DATA_DIR or /data)") |
| + cfg := fs.String("config", defaultConfigPath, "the external replicator's configuration, compared against the inventory") |
| fs.Usage = cliflag.Usage(fs, "") |
| _ = fs.Parse(args) |
| dir := dataDir(*data) |
| @@ -365,25 +370,36 @@ func runInventory(args []string) { |
| if rp == "" { |
| rp = dir + "/registry.db" |
| } |
| - reg, err := registry.Open(rp) |
| + reg, err := registry.OpenReadOnly(rp) |
| if err != nil { |
| - log.Fatalf("inventory: open registry %s: %v", rp, err) |
| + log.Fatalf("inventory: open registry %s read-only: %v", rp, err) |
| } |
| defer reg.Close() |
| - report, err := InventoryReport(reg, dir) |
| + live, err := os.ReadFile(*cfg) |
| + if err != nil && !os.IsNotExist(err) { |
| + log.Fatalf("inventory: read %s: %v", *cfg, err) |
| + } |
| + report, err := InventoryReport(reg, dir, string(live)) |
| if err != nil { |
| log.Fatalf("inventory: %v", err) |
| } |
| + if live == nil { |
| + fmt.Printf("external replicator: no configuration at %q\n", *cfg) |
| + } |
| fmt.Print(report) |
| } |
| -// InventoryReport renders, for one box, the external replicator's |
| -// configured databases (what carlos ops litestream would write), the |
| -// platform replicator's inventory (FleetDBOwners), and the required |
| -// databases with whether each is reachable through its root. It reads |
| -// and writes nothing but the report. |
| -func InventoryReport(reg *registry.Registry, dir string) (string, error) { |
| - dbs, skipped, err := FleetDBsWithSkipped(reg, dir) |
| +// InventoryReport renders, for one box, the databases the external |
| +// replicator's live configuration names (liveConfig, the file as it stands |
| +// on the box: the list the shrink guard may have kept past a route's |
| +// removal), the list a regeneration would write, the platform replicator's |
| +// inventory (FleetDBOwners), and the required databases with whether each |
| +// is reachable through its root and owned. The external lists and the |
| +// inventory are separate looks, one after the other: a file renamed |
| +// between them shows on one side only. Every path is quoted. It writes |
| +// nothing but the report. |
| +func InventoryReport(reg *registry.Registry, dir, liveConfig string) (string, error) { |
| + proposed, skipped, err := FleetDBsWithSkipped(reg, dir) |
| if err != nil { |
| return "", err |
| } |
| @@ -391,51 +407,84 @@ func InventoryReport(reg *registry.Registry, dir string) (string, error) { |
| if err != nil { |
| return "", err |
| } |
| + var live []string |
| + for _, p := range ParseLitestreamDBs(liveConfig) { |
| + live = append(live, filepath.Clean(p)) |
| + } |
| + sort.Strings(live) |
| var b strings.Builder |
| - fmt.Fprintf(&b, "external replicator: %d databases configured, %d skipped\n", len(dbs), len(skipped)) |
| - for _, p := range dbs { |
| - fmt.Fprintf(&b, " configured %s\n", p) |
| + fmt.Fprintf(&b, "external replicator, live configuration: %d databases\n", len(live)) |
| + for _, p := range live { |
| + fmt.Fprintf(&b, " live %q\n", p) |
| + } |
| + fmt.Fprintf(&b, "external replicator, a regeneration would write: %d databases, %d skipped\n", len(proposed), len(skipped)) |
| + for _, p := range proposed { |
| + fmt.Fprintf(&b, " proposed %q\n", p) |
| } |
| for _, p := range skipped { |
| - fmt.Fprintf(&b, " skipped %s\n", p) |
| + fmt.Fprintf(&b, " skipped %q\n", p) |
| } |
| - fmt.Fprintf(&b, "platform replicator: %d owned, %d skipped, %d unowned, %d problems\n", len(inv.Owned), len(inv.Skipped), len(inv.Unowned), len(inv.Problems)) |
| + fmt.Fprintf(&b, "platform replicator: %d owned, %d skipped, %d unowned, %d problems (a separate look)\n", len(inv.Owned), len(inv.Skipped), len(inv.Unowned), len(inv.Problems)) |
| for _, o := range inv.Owned { |
| - fmt.Fprintf(&b, " owned %s %s/%s %s\n", o.Path, o.Account, o.App, o.Host) |
| + fmt.Fprintf(&b, " owned %q %s/%s %s\n", o.Path, o.Account, o.App, o.Host) |
| } |
| for _, p := range inv.Skipped { |
| - fmt.Fprintf(&b, " skipped %s\n", p) |
| + fmt.Fprintf(&b, " skipped %q\n", p) |
| } |
| for _, p := range inv.Unowned { |
| - fmt.Fprintf(&b, " unowned %s\n", p) |
| + fmt.Fprintf(&b, " unowned %q\n", p) |
| } |
| for _, p := range inv.Problems { |
| - fmt.Fprintf(&b, " problem %s\n", p) |
| + fmt.Fprintf(&b, " problem %q\n", p) |
| } |
| owned := map[string]bool{} |
| for _, o := range inv.Owned { |
| owned[o.Path] = true |
| } |
| - for _, p := range dbs { |
| + for _, p := range live { |
| if !owned[p] { |
| - fmt.Fprintf(&b, " configured but not owned by the platform replicator: %s\n", p) |
| + fmt.Fprintf(&b, " live but not owned by the platform replicator: %q\n", p) |
| } |
| } |
| - configured := map[string]bool{} |
| - for _, p := range dbs { |
| - configured[p] = true |
| + for _, p := range proposed { |
| + if !owned[p] { |
| + fmt.Fprintf(&b, " proposed but not owned by the platform replicator: %q\n", p) |
| + } |
| + } |
| + isLive := map[string]bool{} |
| + for _, p := range live { |
| + isLive[p] = true |
| + } |
| + isProposed := map[string]bool{} |
| + for _, p := range proposed { |
| + isProposed[p] = true |
| } |
| for _, o := range inv.Owned { |
| - if !configured[o.Path] { |
| - fmt.Fprintf(&b, " owned but not configured for the external replicator: %s\n", o.Path) |
| + if !isLive[o.Path] { |
| + fmt.Fprintf(&b, " owned but not in the live configuration: %q\n", o.Path) |
| + } |
| + if !isProposed[o.Path] { |
| + fmt.Fprintf(&b, " owned but a regeneration would not write it: %q\n", o.Path) |
| } |
| } |
| fmt.Fprintf(&b, "required: %d databases\n", len(inv.Required)) |
| + required := map[string]bool{} |
| for _, r := range inv.Required { |
| + required[r.Path] = true |
| if r.Why == "" { |
| - fmt.Fprintf(&b, " required %s reachable\n", r.Path) |
| + fmt.Fprintf(&b, " required %q reachable and owned\n", r.Path) |
| } else { |
| - fmt.Fprintf(&b, " required %s MISSING: %s\n", r.Path, r.Why) |
| + fmt.Fprintf(&b, " required %q MISSING: %s\n", r.Path, r.Why) |
| + } |
| + } |
| + // the console's database is required only when CONSOLE_DB says so (the |
| + // agent's environment, not the console's): one at the conventional |
| + // path that is not required is said, so the gap is visible on the box |
| + if console := filepath.Clean(consoleDBPath(dir)); !required[console] { |
| + for _, o := range inv.Owned { |
| + if o.Path == console { |
| + fmt.Fprintf(&b, " not required: the console database %q is owned but CONSOLE_DB is not set for this process, so a build would not hold the day were it gone\n", console) |
| + } |
| } |
| } |
| return b.String(), nil |
diff --git a/internal/registry/readonly_test.go b/internal/registry/readonly_test.go| new file mode 100644 |
| index 000000000..dfba95e82 |
| --- /dev/null |
| +++ b/internal/registry/readonly_test.go |
| @@ -0,0 +1,45 @@ |
| +package registry |
| + |
| +import ( |
| + "os" |
| + "path/filepath" |
| + "testing" |
| +) |
| + |
| +// OpenReadOnly opens what is there and creates nothing: a missing path is |
| +// an error with no file made, an existing registry lists and refuses a |
| +// write. |
| +func TestOpenReadOnly(t *testing.T) { |
| + dir := t.TempDir() |
| + missing := filepath.Join(dir, "nope", "registry.db") |
| + if _, err := OpenReadOnly(missing); err == nil { |
| + t.Fatal("a missing registry opened read-only") |
| + } |
| + if _, err := os.Stat(missing); !os.IsNotExist(err) { |
| + t.Fatalf("the missing registry was created: %v", err) |
| + } |
| + path := filepath.Join(dir, "registry.db") |
| + rw, err := Open(path) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := rw.Upsert(Route{Host: "a.example.com", Account: "bac", App: "one"}); err != nil { |
| + t.Fatal(err) |
| + } |
| + rw.Close() |
| + ro, err := OpenReadOnly(path) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + defer ro.Close() |
| + routes, err := ro.List() |
| + if err != nil || len(routes) != 1 || routes[0].Host != "a.example.com" { |
| + t.Fatalf("list through the read-only handle: %v %+v", err, routes) |
| + } |
| + if err := ro.Upsert(Route{Host: "b.example.com", Account: "bac", App: "one"}); err == nil { |
| + t.Fatal("a write through the read-only handle") |
| + } |
| + if _, err := OpenReadOnly(dir); err == nil { |
| + t.Fatal("a directory opened as a registry") |
| + } |
| +} |
diff --git a/internal/registry/registry.go b/internal/registry/registry.go| index 726e74efd..dd1ba5e36 100644 |
| --- a/internal/registry/registry.go |
| +++ b/internal/registry/registry.go |
| @@ -412,6 +412,35 @@ func Open(path string) (*Registry, error) { |
| return r, nil |
| } |
| +// OpenReadOnly opens an existing registry for reading alone: no file is |
| +// created, no journal mode is set and nothing is migrated, so a report |
| +// standing on a box (carlos ops inventory) writes nothing, and a wrong |
| +// path is an error rather than an empty registry. A write through the |
| +// handle fails. |
| +func OpenReadOnly(path string) (*Registry, error) { |
| + fi, err := os.Stat(path) |
| + if err != nil { |
| + return nil, err |
| + } |
| + if !fi.Mode().IsRegular() { |
| + return nil, fmt.Errorf("registry %s: not a regular file", path) |
| + } |
| + d, err := sql.Open("sqlite", "file:"+path+"?mode=ro") |
| + if err != nil { |
| + return nil, err |
| + } |
| + d.SetMaxOpenConns(1) |
| + if _, err := d.Exec("PRAGMA busy_timeout=5000"); err != nil { |
| + d.Close() |
| + return nil, err |
| + } |
| + if _, err := d.Exec("SELECT 1 FROM routes LIMIT 0"); err != nil { // the schema is whatever is there: a registry, or said |
| + d.Close() |
| + return nil, fmt.Errorf("registry %s: %w", path, err) |
| + } |
| + return &Registry{sql: d, path: path}, nil |
| +} |
| + |
| // Path returns the file the registry was opened from. |
| func (r *Registry) Path() string { return r.path } |
diff --git a/internal/usage/durability.go b/internal/usage/durability.go| index ab115c2f9..78666846b 100644 |
| --- a/internal/usage/durability.go |
| +++ b/internal/usage/durability.go |
| @@ -298,6 +298,17 @@ type DurabilityContent struct { |
| // Late counts attempts recorded under this date after it was closed |
| // (a wall clock stepped back across midnight). |
| Late int64 `json:"late,omitempty"` |
| + // Barred counts the replicator's admissions (the replicator's authority |
| + // design, Part C) admitted on or before this day and not yet settled: |
| + // a request may still be in flight, so the day is not complete while |
| + // any remains (the barrier, temporary and box-wide; it writes nothing |
| + // and lifts at settlement). An older reader ignores the field; the |
| + // design's box-wide marker for older readers is a later slice. |
| + Barred int64 `json:"barred,omitempty"` |
| + // Held counts the hold rows on this day (Part C's uncertainty: a |
| + // fleet's authority was found changed and this day is inside the |
| + // interval it may have been changed in); permanent and box-wide here. |
| + Held int64 `json:"held,omitempty"` |
| } |
| // DurabilitySnapshot is one epoch's published snapshot of a day, with the |
| @@ -584,6 +595,12 @@ func (c DurabilityContent) CompletenessFor(account string, from, to time.Time) ( |
| if c.RecordingFailures > 0 { |
| return false, fmt.Sprintf("%d recording failure(s): requests happened that the ledger could not record", c.RecordingFailures) |
| } |
| + if c.Barred > 0 { |
| + return false, fmt.Sprintf("%d replication admission(s) unsettled: a request admitted on or before this day may still be in flight", c.Barred) |
| + } |
| + if c.Held > 0 { |
| + return false, fmt.Sprintf("%d replication hold(s): a fleet's authority was found changed within an interval that covers this day", c.Held) |
| + } |
| if c.Late > 0 { |
| return false, fmt.Sprintf("%d attempt(s) recorded after the day was closed", c.Late) |
| } |
| @@ -729,7 +746,7 @@ func validateDurabilityContent(c *DurabilityContent) error { |
| if !durabilityEpochRE.MatchString(c.Epoch) { |
| return fmt.Errorf("usage: durability content: epoch %q", c.Epoch) |
| } |
| - if c.Watermark < 0 || c.RecordingFailures < 0 || c.Late < 0 { |
| + if c.Watermark < 0 || c.RecordingFailures < 0 || c.Late < 0 || c.Barred < 0 || c.Held < 0 { |
| return errors.New("usage: durability content: negative counter") |
| } |
| var sum DurabilityTotals |
diff --git a/internal/usage/durabilityadmissions.go b/internal/usage/durabilityadmissions.go| index 2bb80dc7d..e427c7d53 100644 |
| --- a/internal/usage/durabilityadmissions.go |
| +++ b/internal/usage/durabilityadmissions.go |
| @@ -83,6 +83,12 @@ var ( |
| // ErrAdmissionSettled is returned for a transition on a row that is |
| // already terminal: a terminal state is final (the model's check). |
| ErrAdmissionSettled = errors.New("usage ledger: the admission is already settled") |
| + // ErrAdmissionEdge is returned for a transition the design's table has |
| + // no edge for: unissued after any participant activity, a recorder |
| + // outcome other than unrecorded without the recorder registered. |
| + ErrAdmissionEdge = errors.New("usage ledger: the transition is not an edge of the admission's table") |
| + // ErrAdmissionBackdated is returned for an instant before the admission's. |
| + ErrAdmissionBackdated = errors.New("usage ledger: an instant before the admission") |
| // ErrAdmissionUnknown is returned for a token with no row. |
| ErrAdmissionUnknown = errors.New("usage ledger: no such admission") |
| // ErrAdmissionParticipant is returned when a participant completes twice. |
| @@ -105,7 +111,10 @@ type Admission struct { |
| // Settled reports whether no participant remains. |
| func (a Admission) Settled() bool { return !a.SettledAt.IsZero() } |
| -func newToken() (string, error) { |
| +// NewAdmissionToken allocates a token before the registration, so that a |
| +// commit whose outcome is ambiguous can be read back by it (the design's |
| +// conclusive cancellation) and a retry never allocates a second one. |
| +func NewAdmissionToken() (string, error) { |
| var b [16]byte |
| if _, err := rand.Read(b[:]); err != nil { |
| return "", err |
| @@ -113,36 +122,56 @@ func newToken() (string, error) { |
| return hex.EncodeToString(b[:]), nil |
| } |
| -// Admit writes the admission's row, durably (synchronous=FULL, the |
| -// commit waited for), and returns its token. It is the gate's second half |
| +// Admit writes the admission's row under the caller's token, durably |
| +// (synchronous=FULL, the commit waited for). It is the gate's second half |
| // in the design (the proof is the fleet's; the registration in the |
| // fleet's pending set and the recheck after the commit are the gate slice |
| -// of the plan): nothing is sent before this returns. Mirrors |
| -// admissionmodel.WriterCommits. |
| -func (l *DurabilityLedger) Admit(fleetID, account, app string, at time.Time) (string, error) { |
| - if fleetID == "" || account == "" { |
| - return "", errors.New("usage ledger: an admission needs a fleet and an owner") |
| +// of the plan): nothing is sent before this returns. A repeat with the |
| +// same token and the same facts is the same admission (idempotent); a |
| +// repeat with other facts is refused. An error leaves the caller its |
| +// token for the read-back. Mirrors admissionmodel.WriterCommits. |
| +func (l *DurabilityLedger) Admit(token, fleetID, account, app string, at time.Time) error { |
| + if token == "" || fleetID == "" || account == "" { |
| + return errors.New("usage ledger: an admission needs a token, a fleet and an owner") |
| } |
| at = at.UTC() |
| if at.IsZero() { |
| at = l.now().UTC() |
| } |
| - token, err := newToken() |
| - if err != nil { |
| - return "", err |
| - } |
| l.mu.Lock() |
| defer l.mu.Unlock() |
| if l.closed { |
| - return "", errors.New("usage ledger: closed") |
| + return errors.New("usage ledger: closed") |
| + } |
| + tx, err := l.db.Begin() |
| + if err != nil { |
| + return err |
| + } |
| + defer tx.Rollback() //nolint:errcheck |
| + if a, err := readAdmission(tx, token); err == nil { |
| + if a.FleetID == fleetID && a.Account == account && a.App == app && a.AdmittedAt.Equal(at) { |
| + return nil // the same write again: it landed |
| + } |
| + return fmt.Errorf("usage ledger: token %s already names another admission", token) |
| + } else if !errors.Is(err, ErrAdmissionUnknown) { |
| + return err |
| } |
| - if _, err := l.db.Exec(`INSERT INTO admissions (token, fleet_id, account, app, admitted_at) VALUES (?, ?, ?, ?, ?)`, |
| + if _, err := tx.Exec(`INSERT INTO admissions (token, fleet_id, account, app, admitted_at) VALUES (?, ?, ?, ?, ?)`, |
| token, fleetID, account, app, nsOf(at)); err != nil { |
| - return "", err |
| + return err |
| + } |
| + if admitFail != nil { |
| + if err := admitFail(); err != nil { |
| + return err |
| + } |
| } |
| - return token, nil |
| + return tx.Commit() |
| } |
| +// admitFail is a test seam: an error here fails the admission's |
| +// transaction before its commit, as a full disk would, after the insert. |
| +var admitFail func() error |
| + |
| // Unissued is the terminal of a committed row whose request was refused |
| // after all (the recheck found the fleet stale or a discovery begun, or |
| // the gate's budget elapsed): the helper never receives the token, every |
| @@ -159,6 +188,9 @@ func (l *DurabilityLedger) terminal(token string, at time.Time, how string) erro |
| } |
| l.mu.Lock() |
| defer l.mu.Unlock() |
| + if l.closed { |
| + return errors.New("usage ledger: closed") |
| + } |
| tx, err := l.db.Begin() |
| if err != nil { |
| return err |
| @@ -169,21 +201,43 @@ func (l *DurabilityLedger) terminal(token string, at time.Time, how string) erro |
| return err |
| } |
| if a.Settled() { |
| + if a.How == how { |
| + return nil // the same terminal again: it landed |
| + } |
| return ErrAdmissionSettled |
| } |
| + // unissued is an edge from admitted alone: a token never handed out |
| + // has no participant activity |
| + if how == AdmissionUnissued && (!a.TransportDoneAt.IsZero() || !a.RecorderStartedAt.IsZero() || !a.RecorderDoneAt.IsZero()) { |
| + return ErrAdmissionEdge |
| + } |
| + if at.Before(a.AdmittedAt) { |
| + return ErrAdmissionBackdated |
| + } |
| + // the missing timestamps are now (the design's late-commit rule): a |
| + // reported instant earlier than the ledger's clock does not settle |
| + // the row before the ledger learns of it |
| + if now := l.now().UTC(); now.After(at) { |
| + at = now |
| + } |
| ns := nsOf(at) |
| if _, err := tx.Exec(`UPDATE admissions SET transport_done_at = COALESCE(transport_done_at, ?), recorder_done_at = COALESCE(recorder_done_at, ?), settled_at = ?, how = ? WHERE token = ?`, |
| ns, ns, ns, how, token); err != nil { |
| return err |
| } |
| + a.How = how |
| + if err := l.settled(tx, a, at); err != nil { // an empty settlement for an open uncertainty that counted it |
| + return err |
| + } |
| return tx.Commit() |
| } |
| // TransportDone is the SDK middleware's finalize step (lost: the helper |
| // proven gone with the transport not finalised, a completion and not a |
| -// terminal). When the recorder is done too, the row settles at the later |
| -// of the two. Mirrors admissionmodel.TransportDone and the drain's |
| -// transport-lost. |
| +// terminal; it is not recorded on the row, since transport-lost is never |
| +// a value of how). When the recorder is done too, the row settles at the |
| +// latest of the two completions and now. Mirrors |
| +// admissionmodel.TransportDone and the drain's transport-lost. |
| func (l *DurabilityLedger) TransportDone(token string, at time.Time, lost bool) error { |
| return l.participant(token, at, "transport", "") |
| } |
| @@ -200,6 +254,9 @@ func (l *DurabilityLedger) RecorderStarted(token string, at time.Time) error { |
| } |
| l.mu.Lock() |
| defer l.mu.Unlock() |
| + if l.closed { |
| + return errors.New("usage ledger: closed") |
| + } |
| tx, err := l.db.Begin() |
| if err != nil { |
| return err |
| @@ -212,19 +269,29 @@ func (l *DurabilityLedger) RecorderStarted(token string, at time.Time) error { |
| if a.Settled() { |
| return ErrAdmissionSettled |
| } |
| - if !a.RecorderStartedAt.IsZero() || !a.RecorderDoneAt.IsZero() { |
| + if !a.RecorderStartedAt.IsZero() { |
| + if a.RecorderStartedAt.Equal(at) { |
| + return nil // the same registration again |
| + } |
| + return ErrAdmissionParticipant |
| + } |
| + if !a.RecorderDoneAt.IsZero() { |
| return ErrAdmissionParticipant |
| } |
| + if at.Before(a.AdmittedAt) { |
| + return ErrAdmissionBackdated |
| + } |
| if _, err := tx.Exec(`UPDATE admissions SET recorder_started_at = ? WHERE token = ?`, nsOf(at), token); err != nil { |
| return err |
| } |
| return tx.Commit() |
| } |
| -// RecorderDone is the recorder's outcome: finished, abandoned, or |
| -// unrecorded (Start failed and made no attempt row, or never registered |
| -// before the drain). When the transport is done too, the row settles at |
| -// the later of the two. Mirrors admissionmodel.RecorderDone. |
| +// RecorderDone is the recorder's outcome: finished or abandoned, which |
| +// need the recorder registered (RecorderStarted), or unrecorded (Start |
| +// failed and made no attempt row, or never registered before the drain). |
| +// When the transport is done too, the row settles at the latest of the |
| +// two completions and now. Mirrors admissionmodel.RecorderDone. |
| func (l *DurabilityLedger) RecorderDone(token string, at time.Time, how string) error { |
| switch how { |
| case AdmissionFinished, AdmissionAbandoned, AdmissionUnrecorded: |
| @@ -245,6 +312,9 @@ func (l *DurabilityLedger) participant(token string, at time.Time, which, how st |
| } |
| l.mu.Lock() |
| defer l.mu.Unlock() |
| + if l.closed { |
| + return errors.New("usage ledger: closed") |
| + } |
| tx, err := l.db.Begin() |
| if err != nil { |
| return err |
| @@ -255,12 +325,23 @@ func (l *DurabilityLedger) participant(token string, at time.Time, which, how st |
| return err |
| } |
| if a.Settled() { |
| + // the same completion again on a settled row is the write that |
| + // landed; anything else is refused |
| + if which == "transport" && a.TransportDoneAt.Equal(at) || which == "recorder" && a.RecorderDoneAt.Equal(at) && a.How == how { |
| + return nil |
| + } |
| return ErrAdmissionSettled |
| } |
| + if at.Before(a.AdmittedAt) { |
| + return ErrAdmissionBackdated |
| + } |
| ns := nsOf(at) |
| switch which { |
| case "transport": |
| if !a.TransportDoneAt.IsZero() { |
| + if a.TransportDoneAt.Equal(at) { |
| + return nil |
| + } |
| return ErrAdmissionParticipant |
| } |
| a.TransportDoneAt = at |
| @@ -269,8 +350,14 @@ func (l *DurabilityLedger) participant(token string, at time.Time, which, how st |
| } |
| case "recorder": |
| if !a.RecorderDoneAt.IsZero() { |
| + if a.RecorderDoneAt.Equal(at) && a.How == how { |
| + return nil |
| + } |
| return ErrAdmissionParticipant |
| } |
| + if how != AdmissionUnrecorded && a.RecorderStartedAt.IsZero() { |
| + return ErrAdmissionEdge // finished and abandoned are a registered worker's outcomes |
| + } |
| a.RecorderDoneAt = at |
| a.How = how |
| if _, err := tx.Exec(`UPDATE admissions SET recorder_done_at = ?, how = ? WHERE token = ?`, ns, how, token); err != nil { |
| @@ -278,10 +365,16 @@ func (l *DurabilityLedger) participant(token string, at time.Time, which, how st |
| } |
| } |
| if !a.TransportDoneAt.IsZero() && !a.RecorderDoneAt.IsZero() { |
| + // settled at the latest of the two completions and now: a |
| + // completion reported late still settles no earlier than the |
| + // instant the ledger learns of it (the model's clamp) |
| settled := a.TransportDoneAt |
| if a.RecorderDoneAt.After(settled) { |
| settled = a.RecorderDoneAt |
| } |
| + if now := l.now().UTC(); now.After(settled) { |
| + settled = now |
| + } |
| if _, err := tx.Exec(`UPDATE admissions SET settled_at = ? WHERE token = ?`, nsOf(settled), token); err != nil { |
| return err |
| } |
| @@ -292,13 +385,6 @@ func (l *DurabilityLedger) participant(token string, at time.Time, which, how st |
| return tx.Commit() |
| } |
| -// settled is the hook the uncertainty slice fills: a settlement under an |
| -// open uncertainty of the admission's fleet writes the holds of |
| -// [admitted_at, settled_at] for every owner the uncertainty reaches and |
| -// closes it when it is the last; one under no open uncertainty writes |
| -// nothing permanent. Here it writes nothing. |
| -func (l *DurabilityLedger) settled(tx *sql.Tx, a Admission, at time.Time) error { return nil } |
| - |
| // ReadAdmission reads one row. |
| func (l *DurabilityLedger) ReadAdmission(token string) (Admission, error) { |
| l.mu.Lock() |
| @@ -366,6 +452,10 @@ func (l *DurabilityLedger) UnsettledAdmissions(fleetIDs ...string) ([]Admission, |
| } |
| tokens = append(tokens, t) |
| } |
| + if err := rows.Err(); err != nil { // a partial enumeration is no enumeration: the barrier and a discovery must not read it as nothing unsettled |
| + rows.Close() |
| + return nil, err |
| + } |
| rows.Close() |
| tx, err := l.db.Begin() |
| if err != nil { |
diff --git a/internal/usage/durabilityadmissions_test.go b/internal/usage/durabilityadmissions_test.go| index afa23c9c2..f73687c07 100644 |
| --- a/internal/usage/durabilityadmissions_test.go |
| +++ b/internal/usage/durabilityadmissions_test.go |
| @@ -1,11 +1,26 @@ |
| package usage |
| import ( |
| + "context" |
| "errors" |
| + "strings" |
| "testing" |
| "time" |
| ) |
| +// mustAdmit allocates a token and admits under it. |
| +func mustAdmit(t *testing.T, l *DurabilityLedger, fleet, account, app string, at time.Time) string { |
| + t.Helper() |
| + tok, err := NewAdmissionToken() |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := l.Admit(tok, fleet, account, app, at); err != nil { |
| + t.Fatal(err) |
| + } |
| + return tok |
| +} |
| + |
| func openAdmissionLedger(t *testing.T) (*DurabilityLedger, func(time.Duration)) { |
| t.Helper() |
| dir := t.TempDir() |
| @@ -26,10 +41,7 @@ func openAdmissionLedger(t *testing.T) (*DurabilityLedger, func(time.Duration)) |
| func TestAdmissionSettlesAtTheLaterParticipant(t *testing.T) { |
| for _, transportLast := range []bool{false, true} { |
| l, advance := openAdmissionLedger(t) |
| - tok, err := l.Admit("fleet-1", "alice", "app", time.Time{}) |
| - if err != nil { |
| - t.Fatal(err) |
| - } |
| + tok := mustAdmit(t, l, "fleet-1", "alice", "app", time.Time{}) |
| a, err := l.ReadAdmission(tok) |
| if err != nil || a.Settled() || a.Account != "alice" || a.AdmittedAt.IsZero() { |
| t.Fatalf("after Admit: %+v %v", a, err) |
| @@ -38,22 +50,24 @@ func TestAdmissionSettlesAtTheLaterParticipant(t *testing.T) { |
| t.Fatalf("unsettled: %+v", open) |
| } |
| advance(12*time.Hour + 3*time.Minute) // the 3rd |
| - first := func() error { return l.TransportDone(tok, time.Time{}, false) } |
| - second := func() error { return l.RecorderDone(tok, time.Time{}, AdmissionFinished) } |
| + if err := l.RecorderStarted(tok, time.Time{}); err != nil { |
| + t.Fatal(err) |
| + } |
| + first := func(at time.Time) error { return l.TransportDone(tok, at, false) } |
| + second := func(at time.Time) error { return l.RecorderDone(tok, at, AdmissionFinished) } |
| if transportLast { |
| first, second = second, first |
| - if err := l.RecorderStarted(tok, time.Time{}); err != nil { |
| - t.Fatal(err) |
| - } |
| } |
| - if err := first(); err != nil { |
| + firstAt := l.now().UTC() |
| + if err := first(firstAt); err != nil { |
| t.Fatal(err) |
| } |
| if a, _ := l.ReadAdmission(tok); a.Settled() { |
| t.Fatal("settled with one participant pending") |
| } |
| advance(12*time.Hour + 2*time.Minute) // the 4th |
| - if err := second(); err != nil { |
| + secondAt := l.now().UTC() |
| + if err := second(secondAt); err != nil { |
| t.Fatal(err) |
| } |
| a, _ = l.ReadAdmission(tok) |
| @@ -63,13 +77,24 @@ func TestAdmissionSettlesAtTheLaterParticipant(t *testing.T) { |
| if open, _ := l.UnsettledAdmissions(); len(open) != 0 { |
| t.Fatalf("still unsettled: %+v", open) |
| } |
| - // a terminal is final |
| + // the same completion again is the write that landed; a terminal |
| + // is final against anything else |
| + advance(time.Minute) |
| + if err := first(firstAt); err != nil { |
| + t.Fatalf("the first participant's repeat: %v", err) |
| + } |
| + if err := second(secondAt); err != nil { |
| + t.Fatalf("the second participant's repeat: %v", err) |
| + } |
| if err := l.TransportDone(tok, time.Time{}, false); !errors.Is(err, ErrAdmissionSettled) { |
| t.Fatalf("a transition on a settled row: %v", err) |
| } |
| if err := l.Unissued(tok, time.Time{}); !errors.Is(err, ErrAdmissionSettled) { |
| t.Fatalf("unissued after settlement: %v", err) |
| } |
| + if err := l.RecorderDone(tok, time.Time{}, AdmissionAbandoned); !errors.Is(err, ErrAdmissionSettled) { |
| + t.Fatalf("another outcome on a settled row: %v", err) |
| + } |
| } |
| } |
| @@ -78,7 +103,7 @@ func TestAdmissionSettlesAtTheLaterParticipant(t *testing.T) { |
| // now; a participant cannot complete twice; an unknown token is said. |
| func TestAdmissionUnissuedAndParticipants(t *testing.T) { |
| l, advance := openAdmissionLedger(t) |
| - tok, _ := l.Admit("fleet-1", "alice", "app", time.Time{}) |
| + tok := mustAdmit(t, l, "fleet-1", "alice", "app", time.Time{}) |
| advance(time.Minute) |
| if err := l.Unissued(tok, time.Time{}); err != nil { |
| t.Fatal(err) |
| @@ -87,16 +112,42 @@ func TestAdmissionUnissuedAndParticipants(t *testing.T) { |
| if !a.Settled() || a.How != AdmissionUnissued || a.TransportDoneAt.IsZero() || a.RecorderDoneAt.IsZero() || !a.SettledAt.Equal(a.TransportDoneAt) { |
| t.Fatalf("unissued: %+v", a) |
| } |
| - tok2, _ := l.Admit("fleet-1", "alice", "app", time.Time{}) |
| + if err := l.Unissued(tok, time.Time{}); err != nil { |
| + t.Fatalf("unissued again on an unissued row: %v", err) |
| + } |
| + tok2 := mustAdmit(t, l, "fleet-1", "alice", "app", time.Time{}) |
| if err := l.TransportDone(tok2, time.Time{}, false); err != nil { |
| t.Fatal(err) |
| } |
| + if err := l.TransportDone(tok2, time.Time{}, false); err != nil { |
| + t.Fatalf("the same completion again: %v", err) |
| + } |
| + advance(time.Minute) |
| if err := l.TransportDone(tok2, time.Time{}, false); !errors.Is(err, ErrAdmissionParticipant) { |
| t.Fatalf("the transport twice: %v", err) |
| } |
| + // unissued is an edge from admitted alone (the model's GateAnswers |
| + // requires admitted and unanswered) |
| + if err := l.Unissued(tok2, time.Time{}); !errors.Is(err, ErrAdmissionEdge) { |
| + t.Fatalf("unissued after the transport: %v", err) |
| + } |
| + if a, _ := l.ReadAdmission(tok2); a.Settled() || a.How != "" { |
| + t.Fatalf("the refused unissued wrote: %+v", a) |
| + } |
| + // finished and abandoned are a registered worker's outcomes |
| + if err := l.RecorderDone(tok2, time.Time{}, AdmissionFinished); !errors.Is(err, ErrAdmissionEdge) { |
| + t.Fatalf("finished with no recorder registered: %v", err) |
| + } |
| + if err := l.RecorderDone(tok2, time.Time{}, AdmissionAbandoned); !errors.Is(err, ErrAdmissionEdge) { |
| + t.Fatalf("abandoned with no recorder registered: %v", err) |
| + } |
| + if a, _ := l.ReadAdmission(tok2); !a.RecorderDoneAt.IsZero() { |
| + t.Fatalf("the refused outcome wrote: %+v", a) |
| + } |
| if err := l.RecorderStarted(tok2, time.Time{}); err != nil { |
| t.Fatal(err) |
| } |
| + advance(time.Minute) |
| if err := l.RecorderStarted(tok2, time.Time{}); !errors.Is(err, ErrAdmissionParticipant) { |
| t.Fatalf("Start twice: %v", err) |
| } |
| @@ -118,7 +169,7 @@ func TestAdmissionUnrecordedSettlesAndSurvivesReopen(t *testing.T) { |
| if err != nil { |
| t.Fatal(err) |
| } |
| - tok, _ := l.Admit("fleet-1", "alice", "app", time.Time{}) |
| + tok := mustAdmit(t, l, "fleet-1", "alice", "app", time.Time{}) |
| now = now.Add(90 * time.Minute) |
| if err := l.TransportDone(tok, time.Time{}, false); err != nil { |
| t.Fatal(err) |
| @@ -126,7 +177,7 @@ func TestAdmissionUnrecordedSettlesAndSurvivesReopen(t *testing.T) { |
| if err := l.RecorderDone(tok, time.Time{}, AdmissionUnrecorded); err != nil { |
| t.Fatal(err) |
| } |
| - pending, _ := l.Admit("fleet-1", "alice", "app", time.Time{}) |
| + pending := mustAdmit(t, l, "fleet-1", "alice", "app", time.Time{}) |
| if err := l.Close(t.Context(), nil); err != nil { |
| t.Fatal(err) |
| } |
| @@ -143,3 +194,270 @@ func TestAdmissionUnrecordedSettlesAndSurvivesReopen(t *testing.T) { |
| t.Fatalf("the pending row after reopen: %+v", open) |
| } |
| } |
| + |
| +// Mirrors the admission model's barrier invariant and |
| +// TestSuccessorBarredByPredecessorAdmission (box-wide here: every owner): |
| +// a day is not complete while a durable admission admitted on or before |
| +// it is unsettled; it lifts at settlement; an admission admitted the next |
| +// day does not bar the day before. |
| +func TestAdmissionBarsTheDayUntilSettled(t *testing.T) { |
| + l, advance := openAdmissionLedger(t) // 2026-10-02 23:57 |
| + day := "2026-10-02" |
| + before, err := l.Snapshot(day) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if before.Barred != 0 { |
| + t.Fatalf("barred with no admission: %+v", before) |
| + } |
| + tok := mustAdmit(t, l, "fleet-1", "alice", "app", time.Time{}) |
| + c, err := l.Snapshot(day) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if c.Barred != 1 { |
| + t.Fatalf("barred = %d after an admission", c.Barred) |
| + } |
| + if ok, why := c.Completeness(); ok || !strings.Contains(why, "admission") { |
| + t.Fatalf("complete with an unsettled admission: %v %q", ok, why) |
| + } |
| + // the next day is barred too while it stays unsettled; the day before |
| + // the admission is not |
| + advance(5 * time.Minute) // the 3rd |
| + if c3, _ := l.Snapshot("2026-10-03"); c3.Barred != 1 { |
| + t.Fatalf("the next day: barred = %d", c3.Barred) |
| + } |
| + if c1, _ := l.Snapshot("2026-10-01"); c1.Barred != 0 { |
| + t.Fatalf("the day before: barred = %d", c1.Barred) |
| + } |
| + if err := l.TransportDone(tok, time.Time{}, false); err != nil { |
| + t.Fatal(err) |
| + } |
| + if c2, _ := l.Snapshot(day); c2.Barred != 1 { |
| + t.Fatal("lifted with the recorder pending") |
| + } |
| + if err := l.RecorderStarted(tok, time.Time{}); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := l.RecorderDone(tok, time.Time{}, AdmissionFinished); err != nil { |
| + t.Fatal(err) |
| + } |
| + for _, d := range []string{"2026-10-02", "2026-10-03"} { |
| + if c, _ := l.Snapshot(d); c.Barred != 0 { |
| + t.Fatalf("%s still barred after settlement", d) |
| + } |
| + } |
| + // an unissued terminal lifts too |
| + tok2 := mustAdmit(t, l, "fleet-1", "alice", "app", time.Time{}) |
| + if c, _ := l.Snapshot("2026-10-03"); c.Barred != 1 { |
| + t.Fatal("not barred by the second admission") |
| + } |
| + if err := l.Unissued(tok2, time.Time{}); err != nil { |
| + t.Fatal(err) |
| + } |
| + if c, _ := l.Snapshot("2026-10-03"); c.Barred != 0 { |
| + t.Fatal("still barred after unissued") |
| + } |
| +} |
| + |
| +// Review eight: a registered recorder refuses unissued too, and |
| +// RecorderStarted after an outcome; a second fleet keeps its own |
| +// unsettled list, in admission order. |
| +func TestAdmissionEdgesAndFleets(t *testing.T) { |
| + l, advance := openAdmissionLedger(t) |
| + tok := mustAdmit(t, l, "fleet-1", "alice", "app", time.Time{}) |
| + if err := l.RecorderStarted(tok, time.Time{}); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := l.RecorderStarted(tok, time.Time{}); err != nil { |
| + t.Fatalf("the same registration again: %v", err) |
| + } |
| + if err := l.Unissued(tok, time.Time{}); !errors.Is(err, ErrAdmissionEdge) { |
| + t.Fatalf("unissued after the recorder registered: %v", err) |
| + } |
| + advance(time.Minute) |
| + if err := l.RecorderStarted(tok, time.Time{}); !errors.Is(err, ErrAdmissionParticipant) { |
| + t.Fatalf("a second registration: %v", err) |
| + } |
| + tok2 := mustAdmit(t, l, "fleet-1", "alice", "app", time.Time{}) |
| + if err := l.RecorderDone(tok2, time.Time{}, AdmissionUnrecorded); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := l.RecorderStarted(tok2, time.Time{}); !errors.Is(err, ErrAdmissionParticipant) { |
| + t.Fatalf("registered after the outcome: %v", err) |
| + } |
| + // a second recorder completion with the transport still pending |
| + advance(time.Minute) |
| + if err := l.RecorderDone(tok2, time.Time{}, AdmissionUnrecorded); !errors.Is(err, ErrAdmissionParticipant) { |
| + t.Fatalf("the recorder completed twice with the transport pending: %v", err) |
| + } |
| + if a, _ := l.ReadAdmission(tok2); a.Settled() { |
| + t.Fatalf("settled with the transport pending: %+v", a) |
| + } |
| + // abandoned: a registered worker's other outcome, settling with the |
| + // transport |
| + tok3 := mustAdmit(t, l, "fleet-1", "alice", "app", time.Time{}) |
| + if err := l.RecorderStarted(tok3, time.Time{}); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := l.RecorderDone(tok3, time.Time{}, AdmissionAbandoned); err != nil { |
| + t.Fatal(err) |
| + } |
| + if a, _ := l.ReadAdmission(tok3); a.Settled() || a.How != AdmissionAbandoned { |
| + t.Fatalf("abandoned with the transport pending: %+v", a) |
| + } |
| + if err := l.TransportDone(tok3, time.Time{}, true); err != nil { |
| + t.Fatal(err) |
| + } |
| + if a, _ := l.ReadAdmission(tok3); !a.Settled() || a.How != AdmissionAbandoned { |
| + t.Fatalf("abandoned then transport done: %+v", a) |
| + } |
| + // the ledger's commits are synchronous=FULL |
| + var sync int |
| + if err := l.db.QueryRow("PRAGMA synchronous").Scan(&sync); err != nil || sync != 2 { |
| + t.Fatalf("PRAGMA synchronous = %d (%v), want 2 (FULL)", sync, err) |
| + } |
| + if err := l.Unissued(tok2, time.Time{}); !errors.Is(err, ErrAdmissionEdge) { |
| + t.Fatalf("unissued after the recorder's outcome: %v", err) |
| + } |
| + // a second fleet: its own list, each in admission order |
| + advance(time.Minute) |
| + later := mustAdmit(t, l, "fleet-2", "bob", "app", time.Time{}) |
| + advance(-time.Hour) // admitted earlier, inserted later |
| + earlier := mustAdmit(t, l, "fleet-2", "bob", "app", time.Time{}) |
| + advance(2 * time.Hour) |
| + open1, err := l.UnsettledAdmissions("fleet-1") |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if len(open1) != 2 || open1[0].Token != tok || open1[1].Token != tok2 { |
| + t.Fatalf("fleet-1 unsettled: %+v", open1) |
| + } |
| + if all, _ := l.UnsettledAdmissions(); len(all) != 4 { |
| + t.Fatalf("every fleet before the second fleet's settlement: %+v", all) |
| + } |
| + open2, _ := l.UnsettledAdmissions("fleet-2") |
| + if len(open2) != 2 || open2[0].Token != earlier || open2[1].Token != later { |
| + t.Fatalf("fleet-2 unsettled, in admission order: %+v", open2) |
| + } |
| + if all, _ := l.UnsettledAdmissions(); len(all) != 4 || all[0].Token != earlier || all[1].Token != tok { |
| + t.Fatalf("every fleet, in admission order: %+v", all) |
| + } |
| + if none, _ := l.UnsettledAdmissions("fleet-3"); len(none) != 0 { |
| + t.Fatalf("an unknown fleet: %+v", none) |
| + } |
| +} |
| + |
| +// Review eight: an instant before the admission is refused; a settlement |
| +// is no earlier than now, whatever instants the participants report |
| +// (the model's clamp); the earlier participant's instant is kept. |
| +func TestAdmissionInstantsAreClampedToNow(t *testing.T) { |
| + l, advance := openAdmissionLedger(t) |
| + admitted := l.now().UTC() |
| + tok := mustAdmit(t, l, "fleet-1", "alice", "app", time.Time{}) |
| + before := admitted.Add(-time.Second) |
| + if err := l.TransportDone(tok, before, false); !errors.Is(err, ErrAdmissionBackdated) { |
| + t.Fatalf("transport before the admission: %v", err) |
| + } |
| + if err := l.RecorderStarted(tok, before); !errors.Is(err, ErrAdmissionBackdated) { |
| + t.Fatalf("recorder registered before the admission: %v", err) |
| + } |
| + if err := l.Unissued(tok, before); !errors.Is(err, ErrAdmissionBackdated) { |
| + t.Fatalf("unissued before the admission: %v", err) |
| + } |
| + if a, _ := l.ReadAdmission(tok); !a.TransportDoneAt.IsZero() || !a.RecorderStartedAt.IsZero() || a.Settled() { |
| + t.Fatalf("a refused instant wrote: %+v", a) |
| + } |
| + advance(time.Hour) |
| + t1 := admitted.Add(time.Minute) |
| + t2 := admitted.Add(2 * time.Minute) |
| + if err := l.RecorderStarted(tok, t1); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := l.RecorderDone(tok, t2, AdmissionFinished); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := l.RecorderDone(tok, t1, AdmissionFinished); !errors.Is(err, ErrAdmissionParticipant) { |
| + t.Fatalf("the outcome again at another instant: %v", err) |
| + } |
| + if err := l.TransportDone(tok, t1, false); err != nil { |
| + t.Fatal(err) |
| + } |
| + a, _ := l.ReadAdmission(tok) |
| + now := l.now().UTC() |
| + if !a.Settled() || !a.TransportDoneAt.Equal(t1) || !a.RecorderDoneAt.Equal(t2) || !a.SettledAt.Equal(now) { |
| + t.Fatalf("a backdated pair settles now (%s): %+v", now, a) |
| + } |
| + // unissued with a reported instant: the timestamps it fills are now, |
| + // and now is no earlier than the reported instant |
| + tok2 := mustAdmit(t, l, "fleet-1", "alice", "app", time.Time{}) |
| + advance(time.Hour) |
| + if err := l.Unissued(tok2, l.now().Add(-30*time.Minute)); err != nil { |
| + t.Fatal(err) |
| + } |
| + if a, _ := l.ReadAdmission(tok2); !a.SettledAt.Equal(l.now().UTC()) { |
| + t.Fatalf("unissued settled before now: %+v", a) |
| + } |
| +} |
| + |
| +// Review eight: a closed ledger refuses every transition; an admission |
| +// under a token is idempotent for the same facts and refused for others; |
| +// a failed admission leaves no row, so a read-back under the token is |
| +// conclusive. |
| +func TestAdmissionClosedTokensAndReadBack(t *testing.T) { |
| + l, _ := openAdmissionLedger(t) |
| + tok, err := NewAdmissionToken() |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + at := l.now().UTC() |
| + if err := l.Admit(tok, "fleet-1", "alice", "app", at); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := l.Admit(tok, "fleet-1", "alice", "app", at); err != nil { |
| + t.Fatalf("the same admission again: %v", err) |
| + } |
| + if err := l.Admit(tok, "fleet-1", "bob", "app", at); err == nil { |
| + t.Fatal("the token reused for another admission") |
| + } |
| + if err := l.Admit("", "fleet-1", "alice", "app", at); err == nil { |
| + t.Fatal("admitted with no token") |
| + } |
| + if err := l.Admit(tok, "", "alice", "app", at); err == nil { |
| + t.Fatal("admitted with no fleet") |
| + } |
| + if err := l.Admit(tok, "fleet-1", "", "app", at); err == nil { |
| + t.Fatal("admitted with no owner") |
| + } |
| + // a commit that fails after the insert leaves nothing: the read-back |
| + // says unknown and the caller cancels conclusively |
| + tok2, _ := NewAdmissionToken() |
| + boom := errors.New("disk full") |
| + admitFail = func() error { return boom } |
| + err = l.Admit(tok2, "fleet-1", "alice", "app", at) |
| + admitFail = nil |
| + if !errors.Is(err, boom) { |
| + t.Fatalf("the failed commit: %v", err) |
| + } |
| + if _, err := l.ReadAdmission(tok2); !errors.Is(err, ErrAdmissionUnknown) { |
| + t.Fatalf("read back after the failed commit: %v", err) |
| + } |
| + if open, _ := l.UnsettledAdmissions("fleet-1"); len(open) != 1 { |
| + t.Fatalf("the failed admission counted: %+v", open) |
| + } |
| + if err := l.Close(context.Background(), nil); err != nil { |
| + t.Fatal(err) |
| + } |
| + tok3, _ := NewAdmissionToken() |
| + for name, f := range map[string]func() error{ |
| + "admit": func() error { return l.Admit(tok3, "fleet-1", "alice", "app", at) }, |
| + "unissued": func() error { return l.Unissued(tok, time.Time{}) }, |
| + "transport": func() error { return l.TransportDone(tok, time.Time{}, false) }, |
| + "started": func() error { return l.RecorderStarted(tok, time.Time{}) }, |
| + "done": func() error { return l.RecorderDone(tok, time.Time{}, AdmissionUnrecorded) }, |
| + } { |
| + if err := f(); err == nil { |
| + t.Fatalf("%s on a closed ledger", name) |
| + } |
| + } |
| +} |
diff --git a/internal/usage/durabilityledger.go b/internal/usage/durabilityledger.go| index fad2a74e6..4b3f77a74 100644 |
| --- a/internal/usage/durabilityledger.go |
| +++ b/internal/usage/durabilityledger.go |
| @@ -86,6 +86,10 @@ import ( |
| // |
| // Publish and the bucket-side rules live in durabilitypublish.go. |
| type DurabilityLedger struct { |
| + // discovery is Part C's in-memory side (durabilityuncertainty.go). |
| + discovery *discoveryState |
| + discOnce sync.Once |
| + |
| dir string |
| box string |
| now func() time.Time |
| @@ -1993,6 +1997,22 @@ func (l *DurabilityLedger) snapshotOnce(ctx context.Context, date string, before |
| if err := tx.QueryRow(`SELECT pub_revision, pub_hash, intent_revision, intent_hash FROM days WHERE date = ?`, date).Scan(&lineage.pubRevision, &lineage.pubHash, &lineage.intentRevision, &lineage.intentHash); err != nil && !errors.Is(err, sql.ErrNoRows) { |
| return DurabilityContent{}, dayLineage{}, err |
| } |
| + // The barrier (the replicator's authority design, Part C): a durable |
| + // admission admitted on or before this day and not settled bars the |
| + // day, box-wide, until it settles. A query in the same read |
| + // transaction as the rows, so the snapshot and the barrier are one |
| + // instant of the ledger; mirrors the admission model's barrier. |
| + if _, dayEnd, err := DayBounds(date); err == nil { |
| + if err := tx.QueryRow(`SELECT COUNT(*) FROM admissions WHERE settled_at IS NULL AND admitted_at < ?`, nsOf(dayEnd)).Scan(&c.Barred); err != nil { |
| + return DurabilityContent{}, dayLineage{}, err |
| + } |
| + // the permanent holds of the day (box-wide rows), and the latched |
| + // discoveries not yet replayed whose bound reaches the day |
| + if err := tx.QueryRow(`SELECT COUNT(*) FROM holds WHERE day = ?`, date).Scan(&c.Held); err != nil { |
| + return DurabilityContent{}, dayLineage{}, err |
| + } |
| + c.Barred += l.latchedBarring(dayEnd) |
| + } |
| if err := tx.Commit(); err != nil { |
| return DurabilityContent{}, dayLineage{}, err |
| } |
diff --git a/internal/usage/durabilityuncertainty.go b/internal/usage/durabilityuncertainty.go| new file mode 100644 |
| index 000000000..01f1fb7f5 |
| --- /dev/null |
| +++ b/internal/usage/durabilityuncertainty.go |
| @@ -0,0 +1,433 @@ |
| +package usage |
| + |
| +import ( |
| + "database/sql" |
| + "errors" |
| + "fmt" |
| + "sync" |
| + "time" |
| +) |
| + |
| +// Part C's uncertainty: an interval written when a fleet's authority is |
| +// found changed, from the design's fixed bound and the counted admissions, |
| +// held for the fleet's owners (box-wide here: every hold row names the box, |
| +// decision 10 being a parameter), open until every counted admission is |
| +// settled, then closed at the latest settlement plus the margin. A |
| +// discovery whose transaction fails is latched in memory, bars publication |
| +// from its bound, and is replayed as one transaction. Mirrors |
| +// internal/ops/admissionmodel's Discover, Replay, settle and maybeClose. |
| + |
| +// AdmissionMargin is the design's margin around a bound. |
| +const AdmissionMargin = time.Minute |
| + |
| +// Discovery is what a fleet found changed, as the ledger needs it. |
| +type Discovery struct { |
| + FleetID string |
| + At time.Time // the discovery instant |
| + // Proof and LastCleanLook are the fleet's two lower-bound terms; the |
| + // bound is the earlier. |
| + Proof, LastCleanLook time.Time |
| + // Pending is the admission instant of every registration in the |
| + // fleet's in-memory pending set (registered, not yet committed): each |
| + // is counted as unsettled. |
| + Pending []time.Time |
| + Reason string |
| +} |
| + |
| +// Uncertainty is one row, read back. |
| +type Uncertainty struct { |
| + ID int64 |
| + FleetID string |
| + From time.Time |
| + To time.Time // zero while open |
| + Reason string |
| +} |
| + |
| +// Open reports whether the uncertainty is open. |
| +func (u Uncertainty) Open() bool { return u.To.IsZero() } |
| + |
| +// pendingDiscovery is a discovery whose transaction failed: latched in |
| +// memory, barring every day from its bound until Replay commits. |
| +type pendingDiscovery struct { |
| + d Discovery |
| + from time.Time |
| + counted map[string]bool // the tokens counted at the discovery |
| +} |
| + |
| +// discoveryState is the ledger's in-memory side of Part C: the latched |
| +// discoveries by fleet and each open fleet's pending-registration counter |
| +// (the closure test ranges over rows and pending registrations alike). |
| +type discoveryState struct { |
| + mu sync.Mutex |
| + latched map[string]*pendingDiscovery |
| + pending map[string]func() int |
| +} |
| + |
| +func (l *DurabilityLedger) disc() *discoveryState { |
| + l.discOnce.Do(func() { |
| + l.discovery = &discoveryState{latched: map[string]*pendingDiscovery{}, pending: map[string]func() int{}} |
| + }) |
| + return l.discovery |
| +} |
| + |
| +// SetPendingCounter tells the ledger how many registrations the fleet |
| +// holds in memory for a fleet: an open uncertainty of that fleet does not |
| +// close while any remains. nil clears it. |
| +func (l *DurabilityLedger) SetPendingCounter(fleetID string, n func() int) { |
| + ds := l.disc() |
| + ds.mu.Lock() |
| + defer ds.mu.Unlock() |
| + if n == nil { |
| + delete(ds.pending, fleetID) |
| + return |
| + } |
| + ds.pending[fleetID] = n |
| +} |
| + |
| +func (l *DurabilityLedger) pendingCount(fleetID string) int { |
| + ds := l.disc() |
| + ds.mu.Lock() |
| + n := ds.pending[fleetID] |
| + ds.mu.Unlock() |
| + if n == nil { |
| + return 0 |
| + } |
| + return n() |
| +} |
| + |
| +func minTime(a, b time.Time) time.Time { |
| + if b.Before(a) { |
| + return b |
| + } |
| + return a |
| +} |
| + |
| +// bound is the design's fixed bound and lower bound for a discovery: the |
| +// earlier of the proof and the last clean look's start; then the earliest |
| +// admission instant among the fleet's rows that are unsettled, or settled |
| +// (not an empty terminal) at or after that bound, and the pending |
| +// registrations; minus the margin. The cutoff is the bound, fixed before |
| +// the rows are read. Also returns the tokens counted (unsettled rows). |
| +func (l *DurabilityLedger) bound(tx *sql.Tx, d Discovery) (from time.Time, counted map[string]bool, err error) { |
| + b := minTime(d.Proof.UTC(), d.LastCleanLook.UTC()) |
| + from = b |
| + counted = map[string]bool{} |
| + rows, err := tx.Query(`SELECT token, admitted_at, settled_at, how FROM admissions WHERE fleet_id = ?`, d.FleetID) |
| + if err != nil { |
| + return time.Time{}, nil, err |
| + } |
| + defer rows.Close() |
| + for rows.Next() { |
| + var token, how string |
| + var admitted int64 |
| + var settled sql.NullInt64 |
| + if err := rows.Scan(&token, &admitted, &settled, &how); err != nil { |
| + return time.Time{}, nil, err |
| + } |
| + at := time.Unix(0, admitted).UTC() |
| + unsettled := !settled.Valid |
| + historical := settled.Valid && !time.Unix(0, settled.Int64).UTC().Before(b) && how != AdmissionUnissued && how != AdmissionRecoveredUnknown |
| + if unsettled { |
| + counted[token] = true |
| + } |
| + if (unsettled || historical) && at.Before(from) { |
| + from = at |
| + } |
| + } |
| + if err := rows.Err(); err != nil { |
| + return time.Time{}, nil, err |
| + } |
| + for _, p := range d.Pending { |
| + if p.UTC().Before(from) { |
| + from = p.UTC() |
| + } |
| + } |
| + return from.Add(-AdmissionMargin), counted, nil |
| +} |
| + |
| +// writeHolds writes the box-wide hold rows for every day of [from, to]. |
| +func writeHolds(tx *sql.Tx, from, to time.Time, uncertainty int64, at time.Time) error { |
| + for d := from.UTC().Truncate(24 * time.Hour); !d.After(to.UTC()); d = d.Add(24 * time.Hour) { |
| + if _, err := tx.Exec(`INSERT OR IGNORE INTO holds (day, account, app, uncertainty_id, written_at) VALUES (?, '', '', ?, ?)`, DayOf(d), uncertainty, nsOf(at)); err != nil { |
| + return err |
| + } |
| + } |
| + return nil |
| +} |
| + |
| +// Discover writes the uncertainty and its first holds in one transaction: |
| +// the fleet's authority was found changed at d.At. With nothing counted |
| +// and nothing pending, it is opened and closed at once at d.At plus the |
| +// margin. If the transaction fails the discovery is latched (Latched, |
| +// Replay) and the error returned; the caller refuses requests meanwhile. |
| +// Mirrors admissionmodel.Discover. |
| +func (l *DurabilityLedger) Discover(d Discovery) (int64, error) { |
| + d.At = d.At.UTC() |
| + if d.At.IsZero() { |
| + d.At = l.now().UTC() |
| + } |
| + l.mu.Lock() |
| + defer l.mu.Unlock() |
| + id, err := l.discover(d) |
| + if err != nil { |
| + l.latchDiscovery(d) |
| + } |
| + return id, err |
| +} |
| + |
| +func (l *DurabilityLedger) discover(d Discovery) (int64, error) { |
| + tx, err := l.db.Begin() |
| + if err != nil { |
| + return 0, err |
| + } |
| + defer tx.Rollback() //nolint:errcheck |
| + from, counted, err := l.bound(tx, d) |
| + if err != nil { |
| + return 0, err |
| + } |
| + res, err := tx.Exec(`INSERT INTO uncertainties (fleet_id, from_ns, to_ns, reason) VALUES (?, ?, NULL, ?)`, d.FleetID, nsOf(from), d.Reason) |
| + if err != nil { |
| + return 0, err |
| + } |
| + id, err := res.LastInsertId() |
| + if err != nil { |
| + return 0, err |
| + } |
| + to := d.At |
| + closeNow := len(counted) == 0 && len(d.Pending) == 0 |
| + if closeNow { |
| + to = d.At.Add(AdmissionMargin) |
| + if _, err := tx.Exec(`UPDATE uncertainties SET to_ns = ? WHERE id = ?`, nsOf(to), id); err != nil { |
| + return 0, err |
| + } |
| + } |
| + if err := writeHolds(tx, from, to, id, d.At); err != nil { |
| + return 0, err |
| + } |
| + if discoverFail != nil { |
| + if err := discoverFail(); err != nil { |
| + return 0, err |
| + } |
| + } |
| + if err := tx.Commit(); err != nil { |
| + return 0, err |
| + } |
| + return id, nil |
| +} |
| + |
| +// discoverFail is a test seam: an error returned here fails the |
| +// discovery's transaction before its commit, as a full disk would. |
| +var discoverFail func() error |
| + |
| +func (l *DurabilityLedger) latchDiscovery(d Discovery) { |
| + // the counted set and the bound are computed again outside a failed |
| + // transaction, from what the rows say now; a read that fails leaves |
| + // the bound at the earlier of the two terms, which is never later |
| + b := minTime(d.Proof.UTC(), d.LastCleanLook.UTC()).Add(-AdmissionMargin) |
| + pd := &pendingDiscovery{d: d, from: b, counted: map[string]bool{}} |
| + if tx, err := l.db.Begin(); err == nil { |
| + if from, counted, err := l.bound(tx, d); err == nil { |
| + pd.from, pd.counted = from, counted |
| + } |
| + tx.Rollback() //nolint:errcheck |
| + } |
| + ds := l.disc() |
| + ds.mu.Lock() |
| + ds.latched[d.FleetID] = pd |
| + ds.mu.Unlock() |
| +} |
| + |
| +// Latched reports whether a fleet has a discovery awaiting replay, and |
| +// the bound it bars publication from. |
| +func (l *DurabilityLedger) Latched(fleetID string) (time.Time, bool) { |
| + ds := l.disc() |
| + ds.mu.Lock() |
| + defer ds.mu.Unlock() |
| + pd, ok := ds.latched[fleetID] |
| + if !ok { |
| + return time.Time{}, false |
| + } |
| + return pd.from, true |
| +} |
| + |
| +// latchedBarring is the publisher's overlay: the number of latched |
| +// discoveries whose bound is on or before the given day's end. |
| +func (l *DurabilityLedger) latchedBarring(dayEnd time.Time) int64 { |
| + ds := l.disc() |
| + ds.mu.Lock() |
| + defer ds.mu.Unlock() |
| + var n int64 |
| + for _, pd := range ds.latched { |
| + if pd.from.Before(dayEnd) { |
| + n++ |
| + } |
| + } |
| + return n |
| +} |
| + |
| +// Replay is the latched discovery's one transaction: the uncertainty at |
| +// the original bound, the holds of [bound, discovery], the interval of |
| +// every counted admission that settled meanwhile, and closure at the |
| +// replay instant plus the margin when none remains unsettled; the latch |
| +// clears only after the commit. Mirrors admissionmodel.Replay. |
| +func (l *DurabilityLedger) Replay(fleetID string) (int64, error) { |
| + ds := l.disc() |
| + ds.mu.Lock() |
| + pd, ok := ds.latched[fleetID] |
| + ds.mu.Unlock() |
| + if !ok { |
| + return 0, nil |
| + } |
| + now := l.now().UTC() |
| + l.mu.Lock() |
| + defer l.mu.Unlock() |
| + tx, err := l.db.Begin() |
| + if err != nil { |
| + return 0, err |
| + } |
| + defer tx.Rollback() //nolint:errcheck |
| + res, err := tx.Exec(`INSERT INTO uncertainties (fleet_id, from_ns, to_ns, reason) VALUES (?, ?, NULL, ?)`, fleetID, nsOf(pd.from), pd.d.Reason) |
| + if err != nil { |
| + return 0, err |
| + } |
| + id, err := res.LastInsertId() |
| + if err != nil { |
| + return 0, err |
| + } |
| + if err := writeHolds(tx, pd.from, pd.d.At, id, now); err != nil { |
| + return 0, err |
| + } |
| + unsettled := 0 |
| + for token := range pd.counted { |
| + a, err := readAdmission(tx, token) |
| + if err != nil { |
| + return 0, err |
| + } |
| + if a.Settled() { |
| + if a.How != AdmissionUnissued { |
| + if err := writeHolds(tx, a.AdmittedAt, a.SettledAt, id, now); err != nil { |
| + return 0, err |
| + } |
| + } |
| + continue |
| + } |
| + unsettled++ |
| + } |
| + if unsettled == 0 && l.pendingCount(fleetID) == 0 { |
| + to := now.Add(AdmissionMargin) |
| + if _, err := tx.Exec(`UPDATE uncertainties SET to_ns = ? WHERE id = ?`, nsOf(to), id); err != nil { |
| + return 0, err |
| + } |
| + if err := writeHolds(tx, pd.from, to, id, now); err != nil { |
| + return 0, err |
| + } |
| + } |
| + if err := tx.Commit(); err != nil { |
| + return 0, err |
| + } |
| + ds.mu.Lock() |
| + delete(ds.latched, fleetID) |
| + ds.mu.Unlock() |
| + return id, nil |
| +} |
| + |
| +// settled is the settlement's hook (participant's transaction): under an |
| +// open uncertainty of the admission's fleet it writes the holds of |
| +// [admitted_at, settled_at] and closes the uncertainty when no counted |
| +// admission (row or pending registration) remains; under none it writes |
| +// nothing permanent. Mirrors admissionmodel.settle and maybeClose. |
| +func (l *DurabilityLedger) settled(tx *sql.Tx, a Admission, at time.Time) error { |
| + var id int64 |
| + var fromNs int64 |
| + err := tx.QueryRow(`SELECT id, from_ns FROM uncertainties WHERE fleet_id = ? AND to_ns IS NULL ORDER BY id LIMIT 1`, a.FleetID).Scan(&id, &fromNs) |
| + if errors.Is(err, sql.ErrNoRows) { |
| + return nil |
| + } |
| + if err != nil { |
| + return err |
| + } |
| + if a.How != AdmissionUnissued { |
| + if err := writeHolds(tx, a.AdmittedAt, at, id, at); err != nil { |
| + return err |
| + } |
| + } |
| + return l.maybeClose(tx, a.FleetID, id, time.Unix(0, fromNs).UTC(), at) |
| +} |
| + |
| +// maybeClose closes an open uncertainty when nothing counted remains: |
| +// no unsettled row of the fleet and no pending registration. |
| +func (l *DurabilityLedger) maybeClose(tx *sql.Tx, fleetID string, id int64, from, settlement time.Time) error { |
| + var open int |
| + if err := tx.QueryRow(`SELECT COUNT(*) FROM admissions WHERE fleet_id = ? AND settled_at IS NULL`, fleetID).Scan(&open); err != nil { |
| + return err |
| + } |
| + if open > 0 || l.pendingCount(fleetID) > 0 { |
| + return nil |
| + } |
| + to := settlement.Add(AdmissionMargin) |
| + if _, err := tx.Exec(`UPDATE uncertainties SET to_ns = ? WHERE id = ? AND to_ns IS NULL`, nsOf(to), id); err != nil { |
| + return err |
| + } |
| + return writeHolds(tx, from, to, id, settlement) |
| +} |
| + |
| +// EmptySettlement tells an open uncertainty that a counted registration |
| +// ended without a row (a conclusive cancellation) or as unissued, so it |
| +// closes if nothing else remains. Mirrors admissionmodel.emptySettlement. |
| +func (l *DurabilityLedger) EmptySettlement(fleetID string, at time.Time) error { |
| + at = at.UTC() |
| + if at.IsZero() { |
| + at = l.now().UTC() |
| + } |
| + l.mu.Lock() |
| + defer l.mu.Unlock() |
| + tx, err := l.db.Begin() |
| + if err != nil { |
| + return err |
| + } |
| + defer tx.Rollback() //nolint:errcheck |
| + var id, fromNs int64 |
| + err = tx.QueryRow(`SELECT id, from_ns FROM uncertainties WHERE fleet_id = ? AND to_ns IS NULL ORDER BY id LIMIT 1`, fleetID).Scan(&id, &fromNs) |
| + if errors.Is(err, sql.ErrNoRows) { |
| + return nil |
| + } |
| + if err != nil { |
| + return err |
| + } |
| + if err := l.maybeClose(tx, fleetID, id, time.Unix(0, fromNs).UTC(), at); err != nil { |
| + return err |
| + } |
| + return tx.Commit() |
| +} |
| + |
| +// ReadUncertainty reads one row. |
| +func (l *DurabilityLedger) ReadUncertainty(id int64) (Uncertainty, error) { |
| + l.mu.Lock() |
| + defer l.mu.Unlock() |
| + var u Uncertainty |
| + var from int64 |
| + var to sql.NullInt64 |
| + err := l.db.QueryRow(`SELECT id, fleet_id, from_ns, to_ns, reason FROM uncertainties WHERE id = ?`, id).Scan(&u.ID, &u.FleetID, &from, &to, &u.Reason) |
| + if errors.Is(err, sql.ErrNoRows) { |
| + return Uncertainty{}, fmt.Errorf("usage ledger: no uncertainty %d", id) |
| + } |
| + if err != nil { |
| + return Uncertainty{}, err |
| + } |
| + u.From = time.Unix(0, from).UTC() |
| + if to.Valid { |
| + u.To = time.Unix(0, to.Int64).UTC() |
| + } |
| + return u, nil |
| +} |
| + |
| +// Held reports whether a day has a hold row (box-wide). |
| +func (l *DurabilityLedger) Held(day string) (bool, error) { |
| + l.mu.Lock() |
| + defer l.mu.Unlock() |
| + var n int |
| + if err := l.db.QueryRow(`SELECT COUNT(*) FROM holds WHERE day = ?`, day).Scan(&n); err != nil { |
| + return false, err |
| + } |
| + return n > 0, nil |
| +} |
diff --git a/internal/usage/durabilityuncertainty_test.go b/internal/usage/durabilityuncertainty_test.go| new file mode 100644 |
| index 000000000..bc87bb3f2 |
| --- /dev/null |
| +++ b/internal/usage/durabilityuncertainty_test.go |
| @@ -0,0 +1,300 @@ |
| +package usage |
| + |
| +import ( |
| + "errors" |
| + "strings" |
| + "testing" |
| + "time" |
| +) |
| + |
| +func at(d, hh, mm int) time.Time { return time.Date(2026, 10, d, hh, mm, 0, 0, time.UTC) } |
| + |
| +func openLedgerAt(t *testing.T, start time.Time) (*DurabilityLedger, func(time.Time)) { |
| + t.Helper() |
| + dir := t.TempDir() |
| + now := start |
| + l, err := OpenDurabilityLedger(dir, "box", func() time.Time { return now }, DurabilityLedgerOptions{}) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + t.Cleanup(func() { l.Close(t.Context(), nil) }) |
| + return l, func(to time.Time) { now = to } |
| +} |
| + |
| +func held(t *testing.T, l *DurabilityLedger, want bool, days ...string) { |
| + t.Helper() |
| + for _, d := range days { |
| + if got, err := l.Held(d); err != nil || got != want { |
| + t.Fatalf("%s held=%v (%v), want %v", d, got, err, want) |
| + } |
| + } |
| +} |
| + |
| +// Mirrors admissionmodel.TestZeroPendingClosesAtOnceAcrossMidnight: a |
| +// discovery with nothing counted opens and closes at once at the |
| +// discovery plus the margin, holding the margin's extra day across |
| +// midnight; and TestUnchangedFleetHoldsNothing: a settlement under no |
| +// uncertainty writes nothing permanent. |
| +func TestDiscoveryWithNothingPendingClosesAtOnce(t *testing.T) { |
| + l, set := openLedgerAt(t, at(1, 9, 0)) |
| + tok := mustAdmit(t, l, "f", "alice", "app", time.Time{}) |
| + set(at(1, 9, 5)) |
| + if err := l.TransportDone(tok, time.Time{}, false); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := l.RecorderStarted(tok, time.Time{}); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := l.RecorderDone(tok, time.Time{}, AdmissionFinished); err != nil { |
| + t.Fatal(err) |
| + } |
| + held(t, l, false, "2026-10-01") |
| + set(at(1, 23, 59)) |
| + id, err := l.Discover(Discovery{FleetID: "f", At: at(1, 23, 59), Proof: at(1, 9, 0), LastCleanLook: at(1, 23, 0)}) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + u, _ := l.ReadUncertainty(id) |
| + if u.Open() || !u.To.Equal(at(2, 0, 0)) || !u.From.Equal(at(1, 8, 59)) { |
| + t.Fatalf("%+v", u) |
| + } |
| + held(t, l, true, "2026-10-01", "2026-10-02") |
| + c, _ := l.Snapshot("2026-10-02") |
| + if ok, why := c.Completeness(); ok || !strings.Contains(why, "hold") { |
| + t.Fatalf("the margin's day complete: %v %q", ok, why) |
| + } |
| +} |
| + |
| +// Mirrors TestThreeMidnightsBothOrders: admitted on the 2nd, the change |
| +// found on the 3rd, settled on the 4th in either order; the 2nd, 3rd and |
| +// 4th are held and the uncertainty closes at the settlement plus the |
| +// margin. |
| +func TestUncertaintyExtendsToTheSettlement(t *testing.T) { |
| + for _, transportLast := range []bool{false, true} { |
| + l, set := openLedgerAt(t, at(2, 23, 57)) |
| + tok := mustAdmit(t, l, "f", "alice", "app", time.Time{}) |
| + set(at(3, 0, 1)) |
| + id, err := l.Discover(Discovery{FleetID: "f", At: at(3, 0, 1), Proof: at(2, 23, 57), LastCleanLook: at(2, 9, 0)}) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if u, _ := l.ReadUncertainty(id); !u.Open() { |
| + t.Fatal("closed with the admission unsettled") |
| + } |
| + set(at(3, 12, 0)) |
| + if err := l.RecorderStarted(tok, time.Time{}); err != nil { |
| + t.Fatal(err) |
| + } |
| + first, second := func() error { return l.TransportDone(tok, time.Time{}, false) }, func() error { return l.RecorderDone(tok, time.Time{}, AdmissionFinished) } |
| + if transportLast { |
| + first, second = second, first |
| + } |
| + if err := first(); err != nil { |
| + t.Fatal(err) |
| + } |
| + set(at(4, 0, 2)) |
| + if err := second(); err != nil { |
| + t.Fatal(err) |
| + } |
| + u, _ := l.ReadUncertainty(id) |
| + if u.Open() || !u.To.Equal(at(4, 0, 3)) { |
| + t.Fatalf("transportLast=%v: %+v", transportLast, u) |
| + } |
| + held(t, l, true, "2026-10-02", "2026-10-03", "2026-10-04") |
| + held(t, l, false, "2026-10-05") |
| + } |
| +} |
| + |
| +// Mirrors TestCommitAfterDiscoveryIsUnissued and TestBudgetExpiryThenCommit: |
| +// a row counted by the discovery that ends unissued is an empty |
| +// settlement, and the uncertainty closes; a pending registration (the |
| +// fleet's counter) keeps it open until it is resolved. |
| +func TestEmptySettlementsCloseTheUncertainty(t *testing.T) { |
| + l, set := openLedgerAt(t, at(2, 23, 57)) |
| + tok := mustAdmit(t, l, "f", "alice", "app", time.Time{}) |
| + pending := 1 |
| + l.SetPendingCounter("f", func() int { return pending }) |
| + set(at(2, 23, 58)) |
| + id, err := l.Discover(Discovery{FleetID: "f", At: at(2, 23, 58), Proof: at(2, 23, 57), LastCleanLook: at(2, 23, 55), Pending: []time.Time{at(2, 23, 50)}}) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if u, _ := l.ReadUncertainty(id); !u.From.Equal(at(2, 23, 49)) { |
| + t.Fatalf("the pending registration did not lower the bound: %+v", u) |
| + } |
| + set(at(3, 0, 5)) |
| + if err := l.Unissued(tok, time.Time{}); err != nil { |
| + t.Fatal(err) |
| + } |
| + if u, _ := l.ReadUncertainty(id); !u.Open() { |
| + t.Fatal("closed with a registration pending") |
| + } |
| + pending = 0 |
| + if err := l.EmptySettlement("f", time.Time{}); err != nil { |
| + t.Fatal(err) |
| + } |
| + if u, _ := l.ReadUncertainty(id); u.Open() || !u.To.Equal(at(3, 0, 6)) { |
| + t.Fatalf("%+v", u) |
| + } |
| +} |
| + |
| +// Mirrors TestFailedDiscoveryLatchBarsUntilReplay, |
| +// TestFailedDiscoveryReplayKeepsSettlements and |
| +// TestReplayHoldsTheLowerBoundWhileOpen: a discovery whose transaction |
| +// fails is latched and bars publication from its bound; the replay is one |
| +// transaction that opens at the original bound, writes each counted |
| +// settlement's interval, stays open while a counted row is unsettled, and |
| +// closes at the replay plus the margin otherwise. |
| +func TestLatchedDiscoveryAndReplay(t *testing.T) { |
| + l, set := openLedgerAt(t, at(2, 9, 0)) |
| + set(at(2, 23, 58)) |
| + tok := mustAdmit(t, l, "f", "alice", "app", time.Time{}) |
| + discoverFail = func() error { return errors.New("disk full") } |
| + _, err := l.Discover(Discovery{FleetID: "f", At: at(2, 23, 58), Proof: at(2, 23, 58), LastCleanLook: at(2, 23, 58)}) |
| + discoverFail = nil |
| + if err == nil { |
| + t.Fatal("a failed transaction reported as committed") |
| + } |
| + if from, ok := l.Latched("f"); !ok || !from.Equal(at(2, 23, 57)) { |
| + t.Fatalf("latched %v %v", from, ok) |
| + } |
| + c, _ := l.Snapshot("2026-10-02") |
| + if c.Barred == 0 { |
| + t.Fatal("the latch does not bar the day") |
| + } |
| + held(t, l, false, "2026-10-02") |
| + set(at(4, 0, 2)) |
| + if err := l.TransportDone(tok, time.Time{}, false); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := l.RecorderStarted(tok, time.Time{}); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := l.RecorderDone(tok, time.Time{}, AdmissionFinished); err != nil { |
| + t.Fatal(err) |
| + } |
| + held(t, l, false, "2026-10-03") // nothing permanent yet: no uncertainty row |
| + if c, _ := l.Snapshot("2026-10-04"); c.Barred == 0 { |
| + t.Fatal("the latch stopped barring before the replay") |
| + } |
| + set(at(4, 0, 3)) |
| + pending := 1 // a registration still unresolved: the replay stays open, so the settled token's days come from its own write |
| + l.SetPendingCounter("f", func() int { return pending }) |
| + id, err := l.Replay("f") |
| + if err != nil || id == 0 { |
| + t.Fatal(id, err) |
| + } |
| + u, _ := l.ReadUncertainty(id) |
| + if !u.Open() || !u.From.Equal(at(2, 23, 57)) { |
| + t.Fatalf("%+v", u) |
| + } |
| + held(t, l, true, "2026-10-02", "2026-10-03", "2026-10-04") |
| + pending = 0 |
| + set(at(4, 0, 4)) |
| + if err := l.EmptySettlement("f", time.Time{}); err != nil { |
| + t.Fatal(err) |
| + } |
| + if u, _ := l.ReadUncertainty(id); u.Open() || !u.To.Equal(at(4, 0, 5)) { |
| + t.Fatalf("after the registration resolved: %+v", u) |
| + } |
| + if _, ok := l.Latched("f"); ok { |
| + t.Fatal("the latch did not clear after the commit") |
| + } |
| + if c, _ := l.Snapshot("2026-10-04"); c.Barred != 0 || c.Held == 0 { |
| + t.Fatalf("after the replay: %+v", c) |
| + } |
| + // replay while a counted row is unsettled: open, the bound held |
| + l2, set2 := openLedgerAt(t, at(1, 9, 0)) |
| + set2(at(1, 23, 0)) |
| + l2.Discover(Discovery{FleetID: "g", At: at(1, 23, 0), Proof: at(1, 9, 0), LastCleanLook: at(1, 23, 0)}) // a clean look is not a discovery; this just advances nothing |
| + set2(at(1, 23, 40)) |
| + t0 := mustAdmit(t, l2, "h", "alice", "app", time.Time{}) |
| + set2(at(1, 23, 50)) |
| + l2.TransportDone(t0, time.Time{}, false) |
| + l2.RecorderStarted(t0, time.Time{}) |
| + l2.RecorderDone(t0, time.Time{}, AdmissionFinished) |
| + set2(at(2, 0, 30)) |
| + t1 := mustAdmit(t, l2, "h", "alice", "app", time.Time{}) |
| + set2(at(2, 1, 0)) |
| + discoverFail = func() error { return errors.New("disk full") } |
| + l2.Discover(Discovery{FleetID: "h", At: at(2, 1, 0), Proof: at(2, 0, 30), LastCleanLook: at(1, 23, 0)}) |
| + discoverFail = nil |
| + set2(at(2, 1, 5)) |
| + id2, err := l2.Replay("h") |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if u, _ := l2.ReadUncertainty(id2); !u.Open() || !u.From.Equal(at(1, 22, 59)) { |
| + t.Fatalf("%+v", u) |
| + } |
| + held(t, l2, true, "2026-10-01", "2026-10-02") |
| + _ = t1 |
| +} |
| + |
| +// Mirrors the lower-bound tests: the earlier of the proof and the clean |
| +// look is the threshold and the base (TestLowerBoundThresholdIsTheEarlierOfProofAndLook, |
| +// TestLowerBoundProofTerm); the historical term uses the admission |
| +// instant of a settlement at or after it, whatever the recorder's outcome |
| +// (unrecorded counts), excludes unissued, and the cutoff is fixed before |
| +// the rows are read (TestHistoricalCutoffIsTheFixedBound, both row orders). |
| +func TestLowerBound(t *testing.T) { |
| + // the proof earlier than the look, with no admission: the proof's day |
| + l, set := openLedgerAt(t, at(1, 9, 0)) |
| + set(at(3, 9, 0)) |
| + id, _ := l.Discover(Discovery{FleetID: "f", At: at(3, 9, 0), Proof: at(1, 9, 0), LastCleanLook: at(2, 9, 0)}) |
| + if u, _ := l.ReadUncertainty(id); !u.From.Equal(at(1, 8, 59)) { |
| + t.Fatalf("proof term: %+v", u) |
| + } |
| + // a settlement at the bound counts, an unrecorded one counts, an |
| + // unissued one does not; the cutoff is the fixed bound in both orders |
| + // a1 (admitted 23:55, unsettled) and a2 (admitted 23:50, settled at |
| + // 23:58, before the bound): in either insertion order the bound is |
| + // a1's admission minus the margin; a running cutoff would let a2 count |
| + // once a1 had lowered it |
| + for _, a1First := range []bool{false, true} { |
| + l, set := openLedgerAt(t, at(1, 9, 0)) |
| + set(at(1, 23, 58)) |
| + var a1, a2 string |
| + if a1First { |
| + a1 = mustAdmit(t, l, "f", "alice", "app", at(1, 23, 55)) |
| + a2 = mustAdmit(t, l, "f", "alice", "app", at(1, 23, 50)) |
| + } else { |
| + a2 = mustAdmit(t, l, "f", "alice", "app", at(1, 23, 50)) |
| + a1 = mustAdmit(t, l, "f", "alice", "app", at(1, 23, 55)) |
| + } |
| + l.TransportDone(a2, time.Time{}, false) |
| + l.RecorderDone(a2, time.Time{}, AdmissionUnrecorded) // settled before the bound: must not count |
| + set(at(2, 0, 30)) |
| + id, _ := l.Discover(Discovery{FleetID: "f", At: at(2, 0, 40), Proof: at(2, 0, 35), LastCleanLook: at(2, 0, 30)}) |
| + u, _ := l.ReadUncertainty(id) |
| + if want := at(1, 23, 54); !u.From.Equal(want) { |
| + t.Fatalf("a1First=%v: from %v, want the unsettled row's admission minus the margin %v", a1First, u.From, want) |
| + } |
| + _ = a1 |
| + } |
| + // settled exactly at the bound, unrecorded: counts; unissued after the bound: does not |
| + l3, set3 := openLedgerAt(t, at(1, 9, 0)) |
| + set3(at(1, 23, 0)) |
| + tb := mustAdmit(t, l3, "f", "alice", "app", time.Time{}) |
| + set3(at(1, 23, 30)) |
| + tu := mustAdmit(t, l3, "f", "alice", "app", time.Time{}) |
| + set3(at(2, 0, 30)) |
| + l3.TransportDone(tb, time.Time{}, false) |
| + l3.RecorderDone(tb, time.Time{}, AdmissionUnrecorded) // settled at 00:30 == the bound |
| + l3.Unissued(tu, time.Time{}) |
| + set3(at(2, 1, 0)) |
| + id3, _ := l3.Discover(Discovery{FleetID: "f", At: at(2, 1, 0), Proof: at(2, 0, 35), LastCleanLook: at(2, 0, 30)}) |
| + if u, _ := l3.ReadUncertainty(id3); !u.From.Equal(at(1, 22, 59)) { |
| + t.Fatalf("at the bound, unrecorded: %+v", u) |
| + } |
| + l4, set4 := openLedgerAt(t, at(1, 9, 0)) |
| + set4(at(1, 23, 0)) |
| + tu4 := mustAdmit(t, l4, "f", "alice", "app", time.Time{}) |
| + set4(at(2, 0, 30)) |
| + l4.Unissued(tu4, time.Time{}) |
| + set4(at(2, 1, 0)) |
| + id4, _ := l4.Discover(Discovery{FleetID: "f", At: at(2, 1, 0), Proof: at(2, 0, 35), LastCleanLook: at(2, 0, 30)}) |
| + if u, _ := l4.ReadUncertainty(id4); !u.From.Equal(at(2, 0, 29)) { |
| + t.Fatalf("an unissued row reached the bound: %+v", u) |
| + } |
| +} |