amadan
Review seven answered; Part C's first ledger slice: the root governing a resolved stack tracked across an absolute platform link, a later ascent re-pinning higher in that pin's ancestry and re-walking the link; what an acquisition holds released directory by directory to the resolved chain (authority.Result.Release), so the descriptor count agrees with /proc on the systemd and composed shapes; a recheck error that is not a change is drift, stale only at the final look; a root with no owned file beneath it rechecked on its own; the stale window's start asserted; the design's parameter row, the authority doc, the pins comment, the churn list and the spike note reconciled; the durability ledger gains the admissions, fleets, uncertainties, inherited and holds tables and the admission row's transitions (Admit, Unissued, TransportDone, RecorderStarted, RecorderDone, settled by the later participant, a terminal final), with tests mirroring the admission model
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
14 files changed,
+839
−40
DISPOSITIONS-replicator-pinned-root.md+18 −2docs/superpowers/plans/2026-10-03-replicator-authority-plan.md+14 −2docs/superpowers/specs/2026-10-03-replicator-pinned-roots-design.md+1 −1docs/superpowers/specs/2026-10-03-replicator-pinned-roots-spikes.md+1 −1internal/authority/authority.go+17 −0internal/authority/doc.go+3 −1internal/ops/authoritybridge.go+94 −22internal/ops/litestream_metered.go+15 −2internal/ops/litestream_metered_test.go+16 −1internal/ops/litestream_owners.go+6 −3internal/ops/litestream_owners_unix_test.go+113 −4internal/usage/durabilityadmissions.go+395 −0internal/usage/durabilityadmissions_test.go+145 −0internal/usage/durabilityledger.go+1 −1
CI passed — run details
diff --git a/DISPOSITIONS-replicator-pinned-root.md b/DISPOSITIONS-replicator-pinned-root.md| index 21822bd63..24f6de7b1 100644 |
| --- a/DISPOSITIONS-replicator-pinned-root.md |
| +++ b/DISPOSITIONS-replicator-pinned-root.md |
| @@ -107,10 +107,26 @@ Opus: accept with fixes, no P1, one P2. Astra: accept with fixes, no P1, four P2 |
| | Astra 4, Opus P3-5: the tests did not distinguish retention of the directory already listed, per-entry errors, or the listing's `ENOENT` at depth | `TestStrictTreeInspectsTheDirectoryItListed`: the listed directory renamed away and another put at its spelling between the listing and the inspection (the headers read and the links classified are the retained directory's, and its `Recheck` reports the changed edge); a directory removed between its acquisition and its listing (a second seam, `treeBeforeList`) is churn; a directory without search permission is "cannot be read" as a whole (a per-entry failure with search permission present, `Entry.Err`, cannot be produced here and the test says so). | |
| | Opus P2-1: an absolute tenant link spelled through the tree's own root (the `STATE_DIRECTORY` shape) became a link to a link | walked relative from the confined root, since the predicate vetted that link when the root was acquired, and `Within` still judges the result; a test; the link-shapes line names it. | |
| | Astra 5, Opus P3-1: the plan, design and dispositions still said "exactly", the first link's directory, a changed directory skipped, Astra 2's layout refused | the plan's root rule, predicate, inventory and tests paragraphs rewritten; design revision 13 (Part A's predicate "within the pinned roots", the two safe wider shapes, the kept platform-directory refusal, the walk's pre-pin absent exception); the leftover venv sentence removed. | |
| -| Opus P3-2: reason strings changed | the climb's messages name the resolved spelling again (`pathOf`, from the retained names, never a look); the rest are listed here as changed: the predicate's "in a directory that is not the platform's" and "not under the platform's" carry no `: <why>` suffix for a relative target and `root()`'s phrase for an absolute one; a dangling target is "leads through another link: X cannot be looked at"; a relative further link names the target spelling; the non-unix message is "cannot be looked at". None is an Accept key. | |
| +| Opus P3-2: reason strings changed | the climb's messages name the resolved spelling again (`pathOf`, from the retained names, never a look); the rest are listed here as changed: the predicate's "in a directory that is not the platform's" carries no `: <why>` suffix, "not under the platform's" names the resolved parent and the rule for a relative target and `root()`'s phrase for an absolute one; a dangling target is "leads through another link: X cannot be looked at"; a relative further link names the target spelling; the non-unix message is "cannot be looked at". None is an Accept key. | |
| | Opus P3-3: the predicate comment overclaimed for relative targets | the comment says which components are examined (as today, a relative target's named components between the popped ancestor and the parent are not). | |
| | Astra 6, Opus P3-7: the final recovery in the schedule's quiescence ran unchecked; the spike note's mutant list and wording | checked right after it; the list and wording corrected; the fixed-bound test's comment says which order kills. | |
| | Opus (hygiene): `t.Fatal` on the walk's goroutine; an ignored `Chmod` | errors recorded and checked after the walk; the `Chmod` checked. | |
| -The third slice lands with these answers: `fileFacts.keep` (the retained chain), `FleetInventory.Close`, `recheck` and `Descriptors`, `checkOwnedFileOn` and `strictTreeDBsOn` against a fleet's pins, `MeteredFleet.held` rechecked before every look's rebuild and released at Close, a report keeping no descriptors; tests for retention, the mode-only change on a chain directory that only the retained edge sees, the fresh inventory released after a clean look, and descriptors released at Close. The roots' own recheck is observable only where a chain's `Recheck` sees the same edge (every root sits on an owned file's stack), and is kept as the design's ancestry rebuild. |
| +The third slice lands with these answers: `fileFacts.keep` (the retained chain), `FleetInventory.Close`, `recheck` and `Descriptors`, `checkOwnedFileOn` and `strictTreeDBsOn` against a fleet's pins, `MeteredFleet.held` rechecked before every look's rebuild and released at Close, a report keeping no descriptors; tests for retention, the mode-only change on a chain directory that only the retained edge sees, the fresh inventory released after a clean look, and descriptors released at Close. The roots' own recheck is observable on its own for a root with no owned file beneath it (a unit route whose tree holds no database still pins its root); the retention test has that case. |
| + |
| +## Review seven (of `28e7f4d6`: the sixth review answered, the inventory's third slice) |
| + |
| +Opus: accept with fixes, no P1, two P2. Astra: accept with fixes, no P1, two P2. Reports: `logs/opus-replroot-review7.md`, `logs/astra-replroot-review7.md`. The gates on `28e7f4d6` and on `f0ed32b7` (plan Task 3, required coverage, which lands beside these answers) passed. |
| + |
| +| Finding | Disposition | |
| +|---|---| |
| +| Opus P2-1, Astra 2: the descriptor count omitted directories held off the final stack (those descended to a link and popped past by its target; an earlier link's target an absolute link left behind) | at the end of acquisition everything not on the resolved chain is released, directory by directory (`authority.Result.Release`: two walks' stacks share directories when a later walk pops onto an earlier walk's, so a result is never closed whole), and the count agrees with `/proc`; the retention test adds the systemd shape and the composed shape (an absolute link then a relative ascent above its pin) and asserts the equality on both, counting directory descriptors beneath the test root; `retainedChain.depth` (dead) removed; the rendering of the count in the fleet's status is deferred to wiring and the plan says so (Opus P3-4). | |
| +| Opus P2-2: any recheck error staled the fleet | a change (`ErrChanged`, `ErrLink`, `ErrNotPlatform`, `ENOENT`, `ENOTDIR`, `ELOOP`) stales from the last clean look; any other error is drift, and stales only at the final look; a seam (`recheckHeld`) lets the test inject `EIO`: the write after it is sent, drift is recorded, and the next change stales with the hold starting at the last clean look minus the margin (Opus P3-5). | |
| +| Astra 1, Opus P3-1: a relative ascent after an absolute platform link was measured against the spelled root | the root governing the resolved stack is tracked (the absolute link's pin after it); an ascent needing more re-pins higher in that pin's ancestry by the platform walk and re-walks the absolute link from the state before it (each restart strictly higher); the composed fixture (an absolute link, then a relative ascent above its pin) is walked. | |
| +| Astra 3, Opus P3-2: the roots' recheck was said to be unobservable | a unit route whose tree holds no database pins a root no owned file's chain includes; the retention test makes that root's ancestry fail the platform rule and the inventory's recheck sees it; the removal mutant is killed; the earlier sentence is corrected above. | |
| +| Astra 4, Opus P3-3: contract statements still in tension; the slash pin's effect on classification strings | the design's parameter row, the authority package's doc, the pins comment, the "on purpose" list, the suffix description and the retry-failure note reconciled; a pre-pin link's class is "out of the tree" until some spelling pins the slash and "a link to a link" after, which this row records. | |
| +| Astra 5: the spike note counted an oracle's removal as a killed mutant | reworded: the check after a recovery is an oracle, kept so a recovery-mechanism fault shows at once. | |
| +| Opus P3-5: tests that would survive | the window's start asserted; descriptor accounting on a refused build and a multi-claim file stays unasserted (a release missed there would leak only on refusal and is covered by inspection); the `need < 0` clamp is reached by the "above every root" fixture. | |
| + |
| +Plan Task 3 (required coverage) landed at `f0ed32b7` between these reviews and is in this delta's review scope; plan Task 4's first slice (the admission tables and the row's transitions in the durability ledger, `internal/usage/durabilityadmissions.go`, with tests mirroring the model) lands with these answers. |
diff --git a/docs/superpowers/plans/2026-10-03-replicator-authority-plan.md b/docs/superpowers/plans/2026-10-03-replicator-authority-plan.md| index 6424da614..88d149eda 100644 |
| --- a/docs/superpowers/plans/2026-10-03-replicator-authority-plan.md |
| +++ b/docs/superpowers/plans/2026-10-03-replicator-authority-plan.md |
| @@ -14,7 +14,7 @@ Date: 2026-10-03. For `docs/superpowers/specs/2026-10-03-replicator-pinned-roots |
| **`strictTreeDBs` on a `Dir`.** `strictTreeDBs(root string)` keeps its signature and its three string lists (paths spelled as today, `root` joined with the names the listing gave, so the Accept list and every existing test compare unchanged) and becomes a walk on `Dir`s: `List` for the entries; a subdirectory by `Child(name, &entry.Facts)` (a changed entry is looked at again once and, if still a directory, acquired with the fresh facts and walked, a second change being "changed under the look"; a failed look is "cannot be looked at"; `ENOTDIR`, `ErrLink` or absent is churn, as a vanished entry is today; any other error is "cannot be read"); a listing's per-entry failure is that entry's problem and the rest is walked; a listing's `ENOENT` at depth is churn; a regular file's header by `OpenRegular(name, entry.Facts)`; a link's target by `Readlink` and `Walk(target, pins)` from the directory that listed it, classified by the design's explicit classes and nothing silent: `Directory`, or a database-named link, is "a link that may lead to a database"; `Regular` within the tenant's root (`Result.Within(tenantRoot)`, whether the target was spelled relative or absolute) is a skipped link entry, as today; `Regular` or `Directory` not within it (a sibling tenant under the same pin, another pin) is "a link out of the tree"; `Link` is "a link to a link that may lead to a database" (today's classification followed it; this refuses rather than follows); `Other` is "a link to something that is neither a file nor a directory"; `Absent` is churn; `ErrAscentPastRoot`, `ErrOutside` and `ErrNotPlatform` are "a link out of the tree"; `ErrAscentAfterName`, `ErrNotDirectory` and every other error are "a link whose target cannot be looked at". Depth and dot-directory rules are unchanged. `statTarget` and `linktarget_*.go` are deleted. |
| -**`checkOwnedFile` on a chain.** The chain is acquired downward from the platform root as `Dir`s (the root, the predicate's result, each component by `Child`), never by `Lstat` of a joined path; each check it makes today (no link, one filesystem, a regular file with one name, the owner rules by kind, the `-wal`/`-shm`/`-journal` siblings regular with one name, by `Lookat` from the file's directory) is made on the retained chain; `fileFacts` gains the file's retained chain (`keep`: the directory `Dir` with its stack, the file's name and facts; the design's one descriptor stack per owned file, counted by `FleetInventory.Descriptors`), kept for the fleet's life (`MeteredFleet.held`) so every look runs `Recheck` on every chain and `Lookat` of the file's name and `AncestryChanged` on every root before it rebuilds the inventory, and so litestream's opens (Part B, later) take it; `fleetInventory` closes the chains of files it does not keep, a report (`FleetDBOwners`) keeps none, and the fleet releases its chains at Close. The third slice builds this. |
| +**`checkOwnedFile` on a chain.** The chain is acquired downward from the platform root as `Dir`s (the root, the predicate's result, each component by `Child`), never by `Lstat` of a joined path; each check it makes today (no link, one filesystem, a regular file with one name, the owner rules by kind, the `-wal`/`-shm`/`-journal` siblings regular with one name, by `Lookat` from the file's directory) is made on the retained chain; `fileFacts` gains the file's retained chain (`keep`: the directory `Dir` with its stack, the file's name and facts; the design's one descriptor stack per owned file, counted by `FleetInventory.Descriptors`), kept for the fleet's life (`MeteredFleet.held`) so every look runs `Recheck` on every chain and `Lookat` of the file's name and `AncestryChanged` on every root before it rebuilds the inventory, and so litestream's opens (Part B, later) take it; `fleetInventory` closes the chains of files it does not keep, a report (`FleetDBOwners`) keeps none, and the fleet releases its chains at Close. The third slice builds this; the count is `FleetInventory.Descriptors` and its rendering in the fleet's status is deferred to wiring (the drift comparison's `render` must not carry it). |
| **Before wiring.** A census of the link classes on each box's live trees (the venv and `node_modules` shapes are problems under the new classes, where today they are skipped entries) and the owner's decision on them beside decisions 10 to 13; the live `DBPath`s checked against the root rule. |
| @@ -28,7 +28,19 @@ Date: 2026-10-03. For `docs/superpowers/specs/2026-10-03-replicator-pinned-roots |
| ## Task 4: Part C's ledger changes (after the model) |
| -The `admissions`, `uncertainties`, `fleets`, `inherited` and `holds` tables, the request gate's admission with the discovery mutex, the barrier, the latch and the replay, construction backfill, acknowledgement, the producer fence and recovery, exactly as the model has them (the model is the executable contract; each ledger test names the model test it mirrors), with the kill-at-every-transition tests the design lists. Box-wide holds by default. |
| +The admission model (`internal/ops/admissionmodel`) is the executable contract; every ledger test below names the model test it mirrors. The ledger is `internal/usage/durabilityledger.go` (SQLite, `synchronous=FULL`, one writer under `l.mu`, a separate reader for snapshots, the in-memory failure latch the publisher overlays); the gate is `lease.RequestGate` asked by the SDK middleware before each physical request; the fleet is `ops.MeteredFleet`, whose `admit` is today's proof. Box-wide holds by default (decision 10 is a parameter): every hold this task writes names the box, and the `holds` table carries the owner columns for when it is taken. |
| + |
| +**Slices, each its own reviewed commit with the model's test named beside each ledger test.** |
| + |
| +1. **The tables and the admission's row (built: `internal/usage/durabilityadmissions.go`).** `admissions(token TEXT PRIMARY KEY, fleet_id, account, app, admitted_at, transport_done_at, recorder_started_at, recorder_done_at, settled_at, how)`, `fleets(fleet_id PRIMARY KEY, box, witness_generation, owners, opened_at, state, predecessor, acknowledged_by)`, `uncertainties(id, fleet_id, from_ns, to_ns NULL, reason)`, `inherited(fleet_id, uncertainty_id)`, `holds(day, account, app, uncertainty_id, written_at)`, all `IF NOT EXISTS` beside today's tables. The `DurabilityLedger` gains `Admit(fleet, owner, at) (token, error)` (one transaction, `synchronous=FULL`, within the gate's budget: `ErrAdmissionNotDurable` past it, the goroutine staying to mark a late commit `unissued`), `TransportDone(token, at, lost bool)`, `RecorderStarted(token, at)`, `RecorderDone(token, at, how)`, settlement by whichever lands last in its own transaction, and `Unissued(token)`. Mirrors: `TestUnchangedFleetHoldsNothing`, `TestThreeMidnightsBothOrders` (the interval written at settlement), `TestCommitAfterDiscoveryIsUnissued`, `TestBudgetExpiryWithoutDiscovery`, `TestHistoricalTermCountsUnrecordedSettlements`. |
| +2. **The barrier.** The publisher's snapshot (`snapshotOnce`) treats a day incomplete for an owner while a durable unsettled admission of that owner, or of any fleet whose unacknowledged ancestry includes the admission's fleet, is admitted on or before that day (box-wide: any owner); it is a query over `admissions` and `fleets`, writes nothing, and lifts at settlement. Mirrors: the model's barrier invariant and `TestSuccessorBarredByPredecessorAdmission`, `TestLandedAmbiguousCommitBarsBeforeReadBack`. |
| +3. **Discovery, the uncertainty and the latch.** `Discover(fleet, at, proof, lastCleanLook)`: under the fleet's discovery mutex (new in `MeteredFleet`: registration, the recheck and the start of a discovery serialise on it, the pending set counted), the fixed bound, the counted set, the `uncertainties` row and the holds of `[from, now]` for the owners reached in one transaction, closed at once with nothing pending; a failed write latches the pending discovery in the existing failure latch (extended with the bound, the owners reached and the counted tokens; the publisher's overlay reaches successors through the ancestry query) and the fleet refuses requests until `Replay`, one transaction as the design has it. Mirrors: `TestZeroPendingClosesAtOnceAcrossMidnight`, `TestFailedDiscoveryLatchBarsUntilReplay`, `TestFailedDiscoveryReplayKeepsSettlements`, `TestReplayHoldsTheLowerBoundWhileOpen`, `TestReplayWritesSettledTokenWhileOpen`, `TestLatchOverlayReachesSuccessor`, the lower-bound tests (the in-flight request across a clean look, the proof term, the earlier-of threshold, the fixed cutoff in both orders, the admission instant, the exact bound, the empty terminals excluded). |
| +4. **The fleet journal, construction backfill and acknowledgement.** `OpenFleet(predecessor, owners, witness)` writes the `open` row and the backfill of every interval its unacknowledged ancestry holds; `stale`, `closing`, `closed` as the design's transactions (closure carries a pending replay and refuses with an unsettled admission); `Acknowledge(successor)` root-first, copying the effective set into `inherited` and writing the holds for the closed ones. The ledger's reach query replaces the model's `reach`. Mirrors: `TestAncestryReachAndRootFirstAcknowledgement`, `TestHealthyChainContributesNothing`, the per-step inheritance invariant as a property test over a recorded schedule. |
| +5. **The gate and the fleet.** `MeteredFleet.admit` becomes the proof (unchanged) plus the admission: registration in the pending set under the discovery mutex, the ledger's `Admit`, the recheck, the token handed to the SDK middleware (`lease.RequestGate` returns the token; the meter's `Start` carries it; the finalize step writes `TransportDone`). `lookNow` is the discovery when it finds a change; `closeNow` is `closing` then `closed`. The helper process does not exist yet (Part B): the transport and the recorder run in the agent, so the lifecycle guard is the fleet's own, `transport-lost` is written at `Close` for a transport that never finalised, and `unrecorded` where the meter's `Start` failed. Mirrors: `TestHelperDeathAndRecorderGuard` (as the agent's close), `TestSendNeedsDurableIssuedAdmission` (the middleware refuses without a token), `TestReadBackOnlyAfterTheHandleCloses`, `TestBetweenCommitAndAnswer`. |
| +6. **The producer fence and recovery.** `admissions.lock` flocked by the agent before its first admission on an open file description every helper will later inherit (today: the agent alone); `OpenDurabilityLedger`'s `recover` extended: with the fence, in one transaction, every unsettled admission `recovered-unknown` with its interval, every fleet not `closed` closed with its interval, every unacknowledged fleet acknowledged root-first by the first fleet built. Mirrors: `TestFenceOutlivesAgent`, `TestFenceHeldTokenSentAfterCrash`, `TestLandedAmbiguousThenCrash`, the kill-at-every-transition tests (a second process reopening the ledger after a kill at each boundary). |
| +7. **Part C2's reader.** The `holds` rows in the published day (`DurabilityContent`), the reader treating a box-wide row as failing every owner and an owner row as failing that owner alone, and the older-reader marker (a box-wide incompleteness marker whenever any hold exists). Box-wide stays the default; decision 10's per-owner fleets are not built. |
| + |
| +**What waits.** The helper process (Part B) and the fence's descriptor transfer, until the VFS spike; owner-scoped holds (decision 10); the activator-to-fleet handover (not designed). |
| ## Later, each behind its gate |
diff --git a/docs/superpowers/specs/2026-10-03-replicator-pinned-roots-design.md b/docs/superpowers/specs/2026-10-03-replicator-pinned-roots-design.md| index 264255a87..21405af50 100644 |
| --- a/docs/superpowers/specs/2026-10-03-replicator-pinned-roots-design.md |
| +++ b/docs/superpowers/specs/2026-10-03-replicator-pinned-roots-design.md |
| @@ -74,7 +74,7 @@ The VFS is the plan's first spike and it must prove, before anything else of Par |
| | A tenant's database in a platform-owned directory. | 11 | Unowned and refused, as today; `Accept` applies to skipped entries today, so acceptance by name of an unowned file is new mechanism if 11 says so. | |
| | Another user's directory inside a tenant's tree. | 12 | Today's tree behaviour (walked; its databases listed under the tenant's claim); refusing it is 12's change, not this design's; Part B's sidecar residual has two possible authors until then. | |
| | Which uids are the platform's workloads; a root-run unit's claims; the activator's own claims. | 13, approval H | `trustedUIDs` empty: refused. | |
| -| The link shapes a tenant root may be reached through. | compatibility, not policy | Today's predicate, kept as one function, unchanged. | |
| +| The link shapes a tenant root may be reached through. | compatibility, not policy | Today's predicate, kept as one function, evaluated within the pinned roots (Part A: two safe wider shapes, the platform-directory refusal kept, a target that leaves every pinned root refused). | |
| | Tenant link shapes the inventory refuses | Under Part A's classes a venv's `bin/python3 -> python` is a link to a link and `python -> /usr/bin/python3` a file out of the tree, and a `node_modules/.bin` entry likewise; today these are skipped entries compared with the Accept list, and problems are never compared, so once wired they refuse the build unconditionally. An absolute target with a group-writable or other-owned component above its parent (a `0775` `/srv/data`) is out of the tree too. An absolute link spelled through the tree's own root (a DynamicUser app spelling its `STATE_DIRECTORY`, the platform link's path) is walked relative from the confined root and is not a problem. Whether the refused shapes refuse the box, the tenant, or are named and passed over is the owner's, beside decisions 10 to 13; a census of link classes on each box's live trees is part of the pre-wiring check. A dangling link, relative or absolute, stays churn. | |
| | An isolated instance's own mount namespace; bind mounts and device nodes inside a tenant tree. | declared support conditions before wiring | A route outside the declared conditions is refused, not classified silently; the conditions are written in the inventory's rendering per route. | |
| | Handover of a database from the activator's hibernation path to the fleet. | not designed | Refused as a conflict of authorities, as today. | |
diff --git a/docs/superpowers/specs/2026-10-03-replicator-pinned-roots-spikes.md b/docs/superpowers/specs/2026-10-03-replicator-pinned-roots-spikes.md| index e69d0b75a..11b67e0d8 100644 |
| --- a/docs/superpowers/specs/2026-10-03-replicator-pinned-roots-spikes.md |
| +++ b/docs/superpowers/specs/2026-10-03-replicator-pinned-roots-spikes.md |
| @@ -22,7 +22,7 @@ Plan step 0's second spike is code, kept in tree at `internal/ops/admissionmodel |
| - **The boundaries are separate actions** (after the first code review): the writer's commit, the gate's answer (the recheck), a definite failure, an ambiguous return and its read-back, the send (the helper beginning the transport, which needs a durable issued row, recorded as a violation otherwise; a dead helper's send is a no-op, since a dead helper does nothing), the helper's proven exit (the guard closes) and the drain (transport-lost, unrecorded) are each their own step, so a schedule can put a discovery, a budget expiry or a crash between a commit and its answer, hold a token for a day before sending, or ask for a recorder's `Start` between the exit and the drain. The composites the named tests use (`WriterReturns`, `HelperDies`) are those steps in order. |
| - **Randomised schedules**: every action above in random interleavings over about four days, 120 steps a schedule, one owner per fleet (owner scope; the box-wide default is a superset); after each step, and right after every recovery, in the schedule and at quiescence, before anything else runs: the barrier, stated from the chain and not from the publisher (for every fleet and every owner of it, a durable unsettled admission of that fleet or of any fleet in its unacknowledged ancestry bars the owner from its day on, whether or not the token was ever handed out), the hazard invariant (once the ledger knows of a change, every day on which a transport overlapped the true change is incomplete for its owner; after a crash, every sent admission unsettled at the crash likewise; an issued token never sent has no transport and is covered by the barrier and recovery, not this oracle), a terminal state never changing, and a closed fleet having no latch or pending registration; at quiescence every fleet's owners hold every interval of every uncertainty on its whole chain, no uncertainty is open, and a day with no uncertainty reaching an owner is complete. `ADMISSION_MODEL_SEEDS=30000` passes in about six seconds; the default is 400. Publication is revisable by construction: a day published complete can be re-marked by a later hold (the design's materialisation and watermark), and the hazard invariant is satisfied by that retraction; the barrier's purpose is to make retraction rare, not to make it unnecessary. |
| - **What the runs found.** The first runs found the model's own bookkeeping wrong (an owner the fleet did not have; a crash-killed helper's transport end unset). After the second code review made the change its own action, discovered by a later look, Opus's sequence (an admission admitted before the last clean look, in flight across a change after it, settled clean before the discovery) showed the design's lower bound missing that admission's earlier day: **revision 9 of the design** adds the admission instant of every admission settled at or after the last clean look's start to the bound, and the mutant restoring revision 8's rule is killed by the named test. That is the one interleaving the design did not name that the model has found, within the actions it represents and the invariants it checks; the claim is no wider than that. |
| -- **Mutants, in a tree copy**, each killed by a named test or a schedule. After the fourth code review: the historical term filtered to finished and abandoned (the unrecorded test), the cutoff as the running minimum (the fixed-bound test; the a1-first order kills it, the reversed order is the control), the historical term using the send instant, the historical term dropped, a later witness change ignored (the proof test). After the fifth: `>=` at the bound as `>` (the at-the-bound test), a witness change overwriting the first change's instant, the invariant check after a recovery removed (observed by the schedules that recover). After the third code review: the threshold as the later of proof and look, as the look alone, the historical term including empty terminals, the proof not refusing a moved witness, the crash duplicating a landed row, `Close` without its unsettled loop, the latch overlay reaching the fleet's owners only. After the second code review: revision 8's bound (the in-flight test), the bound without the proof (the proof test), the bound without the clean look (the lower-bound test), a landed ambiguous row not durable until read-back, the open interval dropped from construction backfill (the per-step inheritance check), the replay's per-token writes dropped (the replay-while-open test), a read-back leaving the pending set entry. Removing the closed-implies-no-unsettled *check* removes an oracle; the mechanism mutant it exists for, `Close` without its unsettled-admissions loop, is killed by it (an unanswered durable row). Putting the send's terminal check before its gate is equivalent in the schedules tested (a settled, issued, unsent token exists after a clean stop, but no schedule sends it afterwards). After the first code review: the budget expiry a no-op (its own test, with no discovery); the barrier owner-only, or over issued rows only (the successor-barred test and the schedules); the pending set skipped at registration; the guard ignored, and the drain before the guard; the replay clearing the latch without the lower bound's holds (a named test: the schedules did not reach it in 3000 seeds); the send unchecked; a close without the replay (the closed-implies-no-latch check); acknowledgement copying the predecessor's own uncertainties only (not equivalent: a schedule found a fifth fleet's owner unheld for the first fleet's uncertainty, because the chain's copies are how a closed interval reaches a fleet built after its predecessor acknowledged); the answer ignoring the refusal; a read-back while the writer runs (the named test; the later commit would land on a cancelled registration, which the terminal check reports). Equivalent under this model: the unacknowledged-ancestry walk not stopping at an acknowledged step (over-reach writes the same holds). Not equivalent, though once claimed so here: the registry proof in the bound (Astra's schedule: a build, a clean look a day later, a discovery a day after that, with no admission: only the proof reaches the first day; and Opus's: a witness change known only to be after the proof; both are named tests). First: no construction backfill (reach); the pending set not counted (the commit-after-discovery tests); no recheck after commit (the same test, and a hazard at seed 264); the replay dropping counted settlements (hazard); the replay closing at the discovery instant (liveness and the latch tests); a settlement writing no interval (hazard); recovery ignoring the fence (the fence test); helper death without transport-lost (an uncertainty open at quiescence); the lower bound ignoring the clean look (the lower-bound test; the 30000-schedule sweep alone did not find it, which is why that sequence is a named test); the lower bound ignoring the earliest unsettled admission (liveness, because the model's proof then reached too far); unrecorded synthesised for a registered worker (the guard test); recovery skipping unclosed fleets (an uncertainty open at quiescence); acknowledgement not root-first and reach ignoring successors (reach). The equivalent mutants are named where they arise (the epoch check made redundant by the stale state; the unacknowledged-ancestry walk's over-reach; the send's terminal check before its gate), and the registry proof is not one. |
| +- **Mutants, in a tree copy**, each killed by a named test or a schedule. After the fourth code review: the historical term filtered to finished and abandoned (the unrecorded test), the cutoff as the running minimum (the fixed-bound test; the a1-first order kills it, the reversed order is the control), the historical term using the send instant, the historical term dropped, a later witness change ignored (the proof test). After the fifth: `>=` at the bound as `>` (the at-the-bound test), a witness change overwriting the first change's instant, and the invariant check right after a recovery is an oracle, not a mechanism: removing it fails nothing today, and it is kept so a recovery-mechanism fault shows at once rather than after later work repairs it. After the third code review: the threshold as the later of proof and look, as the look alone, the historical term including empty terminals, the proof not refusing a moved witness, the crash duplicating a landed row, `Close` without its unsettled loop, the latch overlay reaching the fleet's owners only. After the second code review: revision 8's bound (the in-flight test), the bound without the proof (the proof test), the bound without the clean look (the lower-bound test), a landed ambiguous row not durable until read-back, the open interval dropped from construction backfill (the per-step inheritance check), the replay's per-token writes dropped (the replay-while-open test), a read-back leaving the pending set entry. Removing the closed-implies-no-unsettled *check* removes an oracle; the mechanism mutant it exists for, `Close` without its unsettled-admissions loop, is killed by it (an unanswered durable row). Putting the send's terminal check before its gate is equivalent in the schedules tested (a settled, issued, unsent token exists after a clean stop, but no schedule sends it afterwards). After the first code review: the budget expiry a no-op (its own test, with no discovery); the barrier owner-only, or over issued rows only (the successor-barred test and the schedules); the pending set skipped at registration; the guard ignored, and the drain before the guard; the replay clearing the latch without the lower bound's holds (a named test: the schedules did not reach it in 3000 seeds); the send unchecked; a close without the replay (the closed-implies-no-latch check); acknowledgement copying the predecessor's own uncertainties only (not equivalent: a schedule found a fifth fleet's owner unheld for the first fleet's uncertainty, because the chain's copies are how a closed interval reaches a fleet built after its predecessor acknowledged); the answer ignoring the refusal; a read-back while the writer runs (the named test; the later commit would land on a cancelled registration, which the terminal check reports). Equivalent under this model: the unacknowledged-ancestry walk not stopping at an acknowledged step (over-reach writes the same holds). Not equivalent, though once claimed so here: the registry proof in the bound (Astra's schedule: a build, a clean look a day later, a discovery a day after that, with no admission: only the proof reaches the first day; and Opus's: a witness change known only to be after the proof; both are named tests). First: no construction backfill (reach); the pending set not counted (the commit-after-discovery tests); no recheck after commit (the same test, and a hazard at seed 264); the replay dropping counted settlements (hazard); the replay closing at the discovery instant (liveness and the latch tests); a settlement writing no interval (hazard); recovery ignoring the fence (the fence test); helper death without transport-lost (an uncertainty open at quiescence); the lower bound ignoring the clean look (the lower-bound test; the 30000-schedule sweep alone did not find it, which is why that sequence is a named test); the lower bound ignoring the earliest unsettled admission (liveness, because the model's proof then reached too far); unrecorded synthesised for a registered worker (the guard test); recovery skipping unclosed fleets (an uncertainty open at quiescence); acknowledgement not root-first and reach ignoring successors (reach). The equivalent mutants are named where they arise (the epoch check made redundant by the stale state; the unacknowledged-ancestry walk's over-reach; the send's terminal check before its gate), and the registry proof is not one. |
| - **What the model does not cover**, by construction: the latch clearing after the commit as two instants (the replay is one step here; the rule stays with the ledger's tests), a snapshot between those instants, the recorder worker's lifetime and the socket drain as separate events, the fence as anything but helper liveness, retries of a failed terminal or replay write, the ledger's commit latency on a box's disk under the replicator's request rate (the spike's other half, which needs a box), Part C2's owner scope beyond the owner-scoped holds the model writes, and process kills inside a transaction (the model's transactions are atomic by fiat; the ledger's are SQLite's). |
| ## The fork's pathname entry points (read 2026-10-03 08:25, every call, not a sample) |
diff --git a/internal/authority/authority.go b/internal/authority/authority.go| index b1aeb0b3d..cce8aca16 100644 |
| --- a/internal/authority/authority.go |
| +++ b/internal/authority/authority.go |
| @@ -161,6 +161,23 @@ func (r *Result) Within(root *Dir) bool { |
| return false |
| } |
| +// Release closes the directories the walk acquired that keep does not |
| +// name, and keeps the rest for Close. A caller that holds several results |
| +// whose stacks share directories (a later walk popped onto an earlier |
| +// walk's directory) releases what is off the final chain this way, never |
| +// by closing a whole result. |
| +func (r *Result) Release(keep func(*Dir) bool) { |
| + kept := r.owned[:0] |
| + for _, d := range r.owned { |
| + if keep(d) { |
| + kept = append(kept, d) |
| + } else { |
| + d.Close() |
| + } |
| + } |
| + r.owned = kept |
| +} |
| + |
| // Close closes the directories the walk acquired, deepest first. |
| func (r *Result) Close() { |
| for i := len(r.owned) - 1; i >= 0; i-- { |
diff --git a/internal/authority/doc.go b/internal/authority/doc.go| index a3839367a..4489c767c 100644 |
| --- a/internal/authority/doc.go |
| +++ b/internal/authority/doc.go |
| @@ -11,7 +11,9 @@ |
| // link target by components from an anchor, popping the retained stack for |
| // a leading run of ".." rather than asking the kernel for ".."; a relative |
| // walk never leaves the root it started in, and an absolute one ends |
| -// beneath a pinned root or is refused, and says which root (Result.Pin). A Root is a Dir reached from a pinned "/" |
| +// beneath a pinned root or is refused (a name absent before any pin is |
| +// entered answers absent, as a dangling target), and says which root |
| +// (Result.Pin). A Root is a Dir reached from a pinned "/" |
| // by a walk whose every component is the platform's. |
| // |
| // Two choices differ from today's pathname code on purpose. An absolute |
diff --git a/internal/ops/authoritybridge.go b/internal/ops/authoritybridge.go| index fedacf9ba..ee52ccf1e 100644 |
| --- a/internal/ops/authoritybridge.go |
| +++ b/internal/ops/authoritybridge.go |
| @@ -4,6 +4,7 @@ import ( |
| "errors" |
| "path/filepath" |
| "strings" |
| + "syscall" |
| "github.com/carlosframework/platform/internal/authority" |
| ) |
| @@ -23,9 +24,9 @@ var authFacts = func(name string, f authority.Facts) authority.Facts { return f |
| func platformFacts(f authority.Facts) bool { return platformUID(f.UID) } |
| // fleetPins is the pinned "/" and the platform roots an inventory walks |
| -// from, built for one tree walk or one claim check today (the fleet will |
| -// hold them for its life once owned files keep their chains) and closed |
| -// with it. A root is where the platform's authority ends in a spelled |
| +// from, built once per inventory and held with it: a report closes them |
| +// before it returns, and a fleet keeps them, with every owned file's |
| +// chain, for its life (FleetInventory.Close). A root is where the platform's authority ends in a spelled |
| // path (the plan's rule): above the first link's directory by as many |
| // levels as the link's target ascends (systemd's "../private/..." pops |
| // from the link's directory to the root and no further; "../../x" needs |
| @@ -209,7 +210,7 @@ func (p *fleetPins) acquire(dir string) (*authority.Dir, func(), bool, string) { |
| } |
| return nil, nil, false, "its directory is not beneath the pinned root" |
| } |
| - var owned []interface{ Close() error } |
| + var owned []*authority.Dir |
| var results []*authority.Result |
| release := func() { |
| for i := len(results) - 1; i >= 0; i-- { |
| @@ -280,6 +281,16 @@ descend: |
| } |
| cur := r.Dir |
| throughLink := false |
| + // gov is the root governing the resolved stack: the spelled root until |
| + // an absolute link is crossed, that link's pin after it. A later |
| + // relative ascent that needs more than gov allows re-pins higher in |
| + // gov's ancestry and re-walks the absolute link from where it was met |
| + // (abs*: the state just before it), so the composed sequence of |
| + // platform links is acquired as the pathname predicate accepted it. |
| + gov := r |
| + absAt, absOwned, absResults := -1, 0, 0 |
| + var absCur *authority.Dir |
| + var absGov *authority.Root |
| for i := rootN; i < len(comps); i++ { |
| c := comps[i] |
| throughLink = false |
| @@ -328,23 +339,48 @@ descend: |
| } |
| n++ |
| } |
| - // the pops available on the resolved stack before the root in |
| - // use; what the ascent needs beyond them comes off the root's own |
| - // spelled components (the root is reached by spelling) |
| + // the pops available on the resolved stack before the governing |
| + // root; what the ascent needs beyond them comes off that root's |
| + // own ancestry, which the platform walk reached by spelling |
| avail := 0 |
| - for d := cur; d != nil && d != r.Dir; d = d.Parent() { |
| + for d := cur; d != nil && d != gov.Dir; d = d.Parent() { |
| avail++ |
| } |
| if n > avail { |
| - need := rootN - (n - avail) |
| - if need < 0 { |
| - need = 0 |
| - } |
| - if need < rootN { // strictly higher than the root in use: the restart terminates |
| - release() |
| - owned, results = nil, nil |
| - rootN = need |
| - goto descend |
| + if gov == r { |
| + need := rootN - (n - avail) |
| + if need < 0 { |
| + need = 0 |
| + } |
| + if need < rootN { // strictly higher than the root in use: the restart terminates |
| + release() |
| + owned, results = nil, nil |
| + rootN = need |
| + goto descend |
| + } |
| + } else if absAt >= 0 { |
| + pinComps, _ := p.comps(p.pathOf(gov.Dir)) |
| + need := len(pinComps) - (n - avail) |
| + if need < 0 { |
| + need = 0 |
| + } |
| + if need < len(pinComps) { // strictly higher than the pin in use: the restart terminates |
| + if _, why := p.root(pinComps[:need]); why != "" { |
| + release() |
| + return nil, nil, false, "a link in its path leads to a directory that is not under the platform's: " + why |
| + } |
| + for k := len(owned) - 1; k >= absOwned; k-- { |
| + owned[k].Close() |
| + } |
| + owned = owned[:absOwned] |
| + for k := len(results) - 1; k >= absResults; k-- { |
| + results[k].Close() |
| + } |
| + results = results[:absResults] |
| + cur, gov = absCur, absGov |
| + i = absAt - 1 |
| + continue |
| + } |
| } |
| } |
| } |
| @@ -394,6 +430,13 @@ descend: |
| release() |
| return nil, nil, false, "a link in its path does not lead to a directory" |
| } |
| + if filepath.IsAbs(target) { |
| + // the state before this absolute link, for a restart from it |
| + absAt, absOwned, absResults, absCur, absGov = i, len(owned), len(results), cur, gov |
| + if res.Pin != nil { |
| + gov = res.Pin |
| + } |
| + } |
| results = append(results, &res) |
| if parent := res.Dir.Parent(); parent != nil { |
| if pf := authFacts(parent.Name(), parent.Facts()); !platformUID(pf.UID) || pf.Mode&0o022 != 0 { |
| @@ -403,6 +446,26 @@ descend: |
| } |
| cur = res.Dir |
| } |
| + // what is held is exactly the resolved chain: the directories descended |
| + // to reach a link that its target's ascent popped past, and an earlier |
| + // link's target an absolute link left behind, are not on it and are |
| + // released now, so the count is honest (Opus and Astra, review seven) |
| + onStack := map[*authority.Dir]bool{} |
| + for d := cur; d != nil; d = d.Parent() { |
| + onStack[d] = true |
| + } |
| + keptOwned := owned[:0] |
| + for _, d := range owned { |
| + if onStack[d] { |
| + keptOwned = append(keptOwned, d) |
| + } else { |
| + d.Close() |
| + } |
| + } |
| + owned = keptOwned |
| + for _, res := range results { // per directory: results' stacks share directories |
| + res.Release(func(d *authority.Dir) bool { return onStack[d] }) |
| + } |
| if cur != p.slash { // the slash is every walk's boundary already and is shared |
| cur.Confine() |
| } |
| @@ -435,11 +498,6 @@ func (c *retainedChain) recheck() error { |
| return nil |
| } |
| -// depth is the number of descriptors the chain holds beneath the pinned |
| -// slash (the file's directory and every directory above it to its root's |
| -// stack top), for the fleet's rendering. |
| -func (c *retainedChain) depth() int { return c.dir.Depth() } |
| - |
| // recheck asks every pinned root whether its ancestry still reads as it |
| // did: a difference is a change the fleet goes stale on. |
| func (p *fleetPins) recheck() error { |
| @@ -454,6 +512,20 @@ func (p *fleetPins) recheck() error { |
| return nil |
| } |
| +// isChange reports whether a recheck's error is a change of what was |
| +// retained (an edge that differs, a link or something else where a |
| +// directory was, a name gone, a root failing the platform rule), as |
| +// opposed to a look that could not be taken (EMFILE, EIO, ENOMEM), which |
| +// is drift until the final look. |
| +func isChange(err error) bool { |
| + var ee *authority.EdgeError |
| + if errors.As(err, &ee) { |
| + err = ee.Err |
| + } |
| + return errors.Is(err, authority.ErrChanged) || errors.Is(err, authority.ErrLink) || errors.Is(err, authority.ErrNotPlatform) || |
| + errors.Is(err, syscall.ENOENT) || errors.Is(err, syscall.ENOTDIR) || errors.Is(err, syscall.ELOOP) |
| +} |
| + |
| // lookAt is one look at a spelled file through its directory's retained |
| // chain: its identity, for the inventory's "two listed paths are one file" |
| // check, with nothing resolved by pathname. |
diff --git a/internal/ops/litestream_metered.go b/internal/ops/litestream_metered.go| index c54b98072..5bdc7711b 100644 |
| --- a/internal/ops/litestream_metered.go |
| +++ b/internal/ops/litestream_metered.go |
| @@ -635,6 +635,10 @@ func (f *MeteredFleet) lookAgain() { |
| f.lookNow(false) |
| } |
| +// recheckHeld is a test seam over the held inventory's recheck, so a test |
| +// can make a look fail without a change. |
| +var recheckHeld = func(inv *FleetInventory) error { return inv.recheck() } |
| + |
| // lookNow is lookAgain without its guard: the look itself, from the |
| // routes this fleet was built over, held from the last clean look when it |
| // finds a change. A look that cannot be taken at all leaves the watch to |
| @@ -650,8 +654,17 @@ func (f *MeteredFleet) lookNow(final bool) { |
| // owned file's chain and name, by descriptor (a directory moved, |
| // replaced, re-owned or re-moded anywhere on a chain is a changed |
| // edge, whether or not the pathname still leads somewhere). |
| - if err := f.held.recheck(); err != nil { |
| - f.goStaleFrom(f.lastCleanLook(), time.Now()) |
| + if err := recheckHeld(&f.held); err != nil { |
| + if isChange(err) { |
| + f.goStaleFrom(f.lastCleanLook(), time.Now()) |
| + return |
| + } |
| + // a look that could not be taken: drift, and at the final look, |
| + // where there is no again, doubt from the last clean look |
| + f.drift.Store(true) |
| + if final { |
| + f.goStaleFrom(f.lastCleanLook(), time.Now()) |
| + } |
| return |
| } |
| inv, err := fleetInventory(f.routes, f.registryPath, f.dataDir) |
diff --git a/internal/ops/litestream_metered_test.go b/internal/ops/litestream_metered_test.go| index aaf781f10..d6a37fe2b 100644 |
| --- a/internal/ops/litestream_metered_test.go |
| +++ b/internal/ops/litestream_metered_test.go |
| @@ -14,6 +14,7 @@ import ( |
| "strings" |
| "sync" |
| "sync/atomic" |
| + "syscall" |
| "testing" |
| "time" |
| @@ -2018,13 +2019,27 @@ func TestMeteredFleetStalesWhenAChainEdgeChanges(t *testing.T) { |
| if err := send(t, fleet, db, 3); err != nil { |
| t.Fatalf("a write after a clean look: %v", err) |
| } |
| + // a look that cannot be taken (no change) is drift, not stale |
| + recheckHeld = func(*FleetInventory) error { return syscall.EIO } |
| + fleet.lookAgain() |
| + recheckHeld = func(inv *FleetInventory) error { return inv.recheck() } |
| + if err := send(t, fleet, db, 4); err != nil { |
| + t.Fatalf("a write after a look that could not be taken: %v, want it sent (drift, not stale)", err) |
| + } |
| + if !fleet.drift.Load() { |
| + t.Fatal("a look that could not be taken did not record drift") |
| + } |
| + cleanBefore := fleet.lastCleanLook() |
| if err := os.Chmod(filepath.Join(state, "data"), 0o700); err != nil { // a mode change on the chain: only the retained edge sees it |
| t.Fatal(err) |
| } |
| fleet.lookAgain() |
| - if err := send(t, fleet, db, 4); !errors.Is(err, ErrOwnershipStale) { |
| + if err := send(t, fleet, db, 5); !errors.Is(err, ErrOwnershipStale) { |
| t.Fatalf("a write after a chain edge changed: %v, want stale", err) |
| } |
| + if from, _, stale := fleet.staleWindow(); !stale || !from.Equal(cleanBefore.Add(-ownershipMargin)) { |
| + t.Fatalf("the hold starts at %v, want the last clean look %v minus the margin", from, cleanBefore) |
| + } |
| if err := fleet.Close(); err != nil { |
| t.Fatal(err) |
| } |
diff --git a/internal/ops/litestream_owners.go b/internal/ops/litestream_owners.go| index 93173e7cb..2549c33d0 100644 |
| --- a/internal/ops/litestream_owners.go |
| +++ b/internal/ops/litestream_owners.go |
| @@ -851,9 +851,12 @@ const strictTreeMaxDepth = 32 |
| // What it passes over on purpose: an entry that vanishes between being |
| // listed and being looked at, or that is a link when listed and something |
| // else when read, or a directory replaced once between the listing and |
| -// its acquisition (it is looked at again and the replacement walked; a |
| -// second replacement under the same look is a problem, "changed under the |
| -// look"), or a directory removed between its acquisition and its listing |
| +// its acquisition (it is looked at again: a directory is acquired with |
| +// the fresh facts and walked, and a file, a link or nothing by then is |
| +// churn like any other; a second replacement under the same look is a |
| +// problem, "changed under the look"; a look that fails there is a |
| +// problem, "cannot be looked at"), or a directory removed between its |
| +// acquisition and its listing |
| // (ordinary churn: SQLite's journals come and go, deploys swap a link for |
| // a directory; the next look sees what is there); an empty regular file |
| // (it holds nothing; the fleet's periodic look notices when it stops |
diff --git a/internal/ops/litestream_owners_unix_test.go b/internal/ops/litestream_owners_unix_test.go| index 260d31e01..c16354de5 100644 |
| --- a/internal/ops/litestream_owners_unix_test.go |
| +++ b/internal/ops/litestream_owners_unix_test.go |
| @@ -810,6 +810,22 @@ func TestStrictTreeThroughAPlatformLink(t *testing.T) { |
| if dbs, _, problems := strictTreeDBs(composed); len(problems) != 0 || !contains(dbs, filepath.Join(composed, "a.db")) { |
| t.Fatalf("two platform links, the second ascending higher: dbs %v problems %v", dbs, problems) |
| } |
| + // an absolute platform link followed by a relative one whose ascent |
| + // goes above the absolute target's pin: the pin is re-established |
| + // higher in that target's ancestry and the tree walked |
| + mkdirs(t, filepath.Join(top, "opt", "state"), filepath.Join(top, "var", "state", "alice")) |
| + writeFile(t, filepath.Join(top, "var", "state", "alice", "v.db")) |
| + if err := os.Symlink(filepath.Join(top, "opt", "state"), filepath.Join(srv, "public", "start2")); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := os.Symlink("../../var/state", filepath.Join(top, "opt", "state", "b")); err != nil { |
| + t.Fatal(err) |
| + } |
| + withOwners(t, map[string]uint32{"srv": 0, "public": 0, "opt": 0, "state": 0, "var": 0, "alice": tenant, "v.db": tenant, "link:start2": 0, "link:b": 0}) |
| + mixed := filepath.Join(srv, "public", "start2", "b", "alice") |
| + if dbs, _, problems := strictTreeDBs(mixed); len(problems) != 0 || !contains(dbs, filepath.Join(mixed, "v.db")) { |
| + t.Fatalf("an absolute link then a relative ascent above its pin: dbs %v problems %v", dbs, problems) |
| + } |
| // a platform link whose absolute target is the slash itself pins the |
| // slash, and a tree spelled through it is walked |
| viaSlash := filepath.Join(public, "viaslash") |
| @@ -899,6 +915,11 @@ func TestStrictTreeInspectsTheDirectoryItListed(t *testing.T) { |
| t.Fatalf("a directory removed before its listing: %v", problems) |
| } |
| treeBeforeList = nil |
| + // the re-look's own failure (a changed directory whose fresh look |
| + // fails with EACCES or EIO, reported as "cannot be looked at") cannot |
| + // be produced here: the look needs search permission on the listed |
| + // directory, which the listing itself needed, so that branch is |
| + // untested by this fixture |
| // a directory that can be read but not searched (0444): its fresh |
| // listing description cannot be opened without search permission, so |
| // the directory as a whole is "cannot be read" (nothing in it was |
| @@ -1026,7 +1047,7 @@ func TestFleetInventoryRetainsAndRechecksChains(t *testing.T) { |
| if err != nil { |
| t.Fatal(err) |
| } |
| - before := openFDCount(t) |
| + before := openDirFDsUnder(t, filepath.Dir(dir)) |
| inv, err := fleetInventory(routes, reg.Path(), dir) |
| if err != nil { |
| t.Fatal(err) |
| @@ -1037,7 +1058,7 @@ func TestFleetInventoryRetainsAndRechecksChains(t *testing.T) { |
| if inv.facts[db].keep == nil { |
| t.Fatal("the owned file's chain is not retained") |
| } |
| - if n, d := openFDCount(t)-before, inv.Descriptors(); n != d || d < 3 { |
| + if n, d := openDirFDsUnder(t, filepath.Dir(dir))-before, inv.Descriptors(); n != d || d < 3 { |
| t.Fatalf("%d descriptors open, the inventory counts %d", n, d) |
| } |
| if err := inv.recheck(); err != nil { |
| @@ -1086,18 +1107,106 @@ func TestFleetInventoryRetainsAndRechecksChains(t *testing.T) { |
| } |
| inv.Close() |
| inv.Close() |
| - if n := openFDCount(t); n != before { |
| + if n := openDirFDsUnder(t, filepath.Dir(dir)); n != before { |
| t.Fatalf("after Close: %d open, %d before", n, before) |
| } |
| if inv.facts[db].keep != nil || inv.Descriptors() != 0 { |
| t.Fatal("Close left a chain") |
| } |
| + // the systemd shape: the directories descended to the link and popped |
| + // past by its target are released, so what is held is the chain and |
| + // the count agrees with /proc |
| + top := platformDir(t) |
| + public := filepath.Join(top, "carlos-app") |
| + private := filepath.Join(top, "private", "apps", "h.example.com") |
| + mkdirs(t, public, private) |
| + linked := writeFile(t, filepath.Join(private, "l.db")) |
| + if err := os.Symlink("../private/apps/h.example.com", filepath.Join(public, "h.example.com")); err != nil { |
| + t.Fatal(err) |
| + } |
| + withOwners(t, map[string]uint32{"carlos-app": 0, "private": 0, "apps": 0, "h.example.com": 44001, "l.db": 44001, "link:h.example.com": 0}) |
| + upsert(t, reg, registry.Route{Host: "h.example.com", Account: "bad", App: "two", DBPath: filepath.Join(public, "h.example.com"), Backing: registry.BackingUnit}) |
| + routes, _ = reg.List() |
| + before2 := openDirFDsUnder(t, filepath.Dir(dir)) |
| + inv2, err := fleetInventory(routes, reg.Path(), dir) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, ok := ownedByPath(inv2)[filepath.Join(public, "h.example.com", "l.db")]; !ok { |
| + t.Fatalf("the linked tree's database: owned %+v unowned %v problems %v", inv2.Owned, inv2.Unowned, inv2.Problems) |
| + } |
| + _ = linked |
| + if n, d := openDirFDsUnder(t, filepath.Dir(dir))-before2, inv2.Descriptors(); n != d { |
| + t.Fatalf("with a platform link: %d descriptors open, the inventory counts %d", n, d) |
| + } |
| + inv2.Close() |
| + if n := openDirFDsUnder(t, filepath.Dir(dir)); n != before2 { |
| + t.Fatalf("after Close with a platform link: %d open, %d before", n, before2) |
| + } |
| + // the composed shape (an absolute platform link, then a relative ascent |
| + // above its pin): what is held is the resolved chain alone, and the |
| + // count agrees with /proc |
| + mkdirs(t, filepath.Join(top, "srv", "public"), filepath.Join(top, "opt", "state"), filepath.Join(top, "var", "state", "c.example.com")) |
| + writeFile(t, filepath.Join(top, "var", "state", "c.example.com", "c.db")) |
| + if err := os.Symlink(filepath.Join(top, "opt", "state"), filepath.Join(top, "srv", "public", "start")); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := os.Symlink("../../var/state", filepath.Join(top, "opt", "state", "b")); err != nil { |
| + t.Fatal(err) |
| + } |
| + withOwners(t, map[string]uint32{"srv": 0, "public": 0, "opt": 0, "state": 0, "var": 0, "c.example.com": 44001, "c.db": 44001, "link:start": 0, "link:b": 0}) |
| + composed := filepath.Join(top, "srv", "public", "start", "b", "c.example.com") |
| + upsert(t, reg, registry.Route{Host: "c.example.com", Account: "baf", App: "four", DBPath: composed, Backing: registry.BackingUnit}) |
| + routes, _ = reg.List() |
| + before4 := openDirFDsUnder(t, filepath.Dir(dir)) |
| + inv4, err := fleetInventory(routes, reg.Path(), dir) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, ok := ownedByPath(inv4)[filepath.Join(composed, "c.db")]; !ok { |
| + t.Fatalf("the composed tree's database: owned %+v unowned %v problems %v", inv4.Owned, inv4.Unowned, inv4.Problems) |
| + } |
| + if n, d := openDirFDsUnder(t, filepath.Dir(dir))-before4, inv4.Descriptors(); n != d { |
| + t.Fatalf("with composed links: %d descriptors open, the inventory counts %d", n, d) |
| + } |
| + inv4.Close() |
| + if err := reg.Delete("c.example.com"); err != nil { |
| + t.Fatal(err) |
| + } |
| + // a route whose tree holds no database still pins its root; a change in |
| + // that root's ancestry is seen by the roots' recheck alone |
| + empty := filepath.Join(platformDir(t), "empty-tree") |
| + mkdirs(t, empty) |
| + upsert(t, reg, registry.Route{Host: "e.example.com", Account: "bae", App: "three", DBPath: empty, Backing: registry.BackingUnit}) |
| + routes, _ = reg.List() |
| + inv3, err := fleetInventory(routes, reg.Path(), dir) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := inv3.recheck(); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := os.Chmod(filepath.Dir(empty), 0o775); err != nil { |
| + t.Fatal(err) |
| + } |
| + t.Cleanup(func() { os.Chmod(filepath.Dir(empty), 0o755) }) |
| + if err := inv3.recheck(); err == nil { |
| + t.Fatal("a root with no owned file beneath it, failing the platform rule, was not seen") |
| + } |
| + inv3.Close() |
| + os.Chmod(filepath.Dir(empty), 0o755) |
| + if err := reg.Delete("e.example.com"); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := reg.Delete("h.example.com"); err != nil { |
| + t.Fatal(err) |
| + } |
| // a report through FleetDBOwners keeps no descriptors |
| rep, err := FleetDBOwners(reg, dir) |
| if err != nil { |
| t.Fatal(err) |
| } |
| - if n := openFDCount(t); n != before || rep.Descriptors() != 0 { |
| + if n := openDirFDsUnder(t, filepath.Dir(dir)); n != before || rep.Descriptors() != 0 { |
| t.Fatalf("a report holds %d descriptors (%d open, %d before)", rep.Descriptors(), n, before) |
| } |
| } |
diff --git a/internal/usage/durabilityadmissions.go b/internal/usage/durabilityadmissions.go| new file mode 100644 |
| index 000000000..2bb80dc7d |
| --- /dev/null |
| +++ b/internal/usage/durabilityadmissions.go |
| @@ -0,0 +1,395 @@ |
| +package usage |
| + |
| +import ( |
| + "crypto/rand" |
| + "database/sql" |
| + "encoding/hex" |
| + "errors" |
| + "fmt" |
| + "time" |
| +) |
| + |
| +// Part C of the replicator's authority design |
| +// (docs/superpowers/specs/2026-10-03-replicator-pinned-roots-design.md): |
| +// the durable admission. No replica request is sent until a row naming it |
| +// (fleet, owner, admission instant) is durable; the row has two |
| +// participants, the transport and the recorder, and is settled by |
| +// whichever finishes last. The executable contract is |
| +// internal/ops/admissionmodel; each test of this file names the model |
| +// test it mirrors. This first slice is the tables and the row's |
| +// transitions; the barrier, the uncertainties, the fleet journal, the |
| +// gate, the fence and recovery are the plan's later slices. |
| + |
| +// admissionSchema is created beside the ledger's own tables. |
| +var admissionSchema = []string{ |
| + `CREATE TABLE IF NOT EXISTS admissions ( |
| + token TEXT PRIMARY KEY, |
| + fleet_id TEXT NOT NULL, |
| + account TEXT NOT NULL, |
| + app TEXT NOT NULL, |
| + admitted_at INTEGER NOT NULL, |
| + transport_done_at INTEGER, |
| + recorder_started_at INTEGER, |
| + recorder_done_at INTEGER, |
| + settled_at INTEGER, |
| + how TEXT NOT NULL DEFAULT '' |
| + )`, |
| + `CREATE INDEX IF NOT EXISTS admissions_open ON admissions(fleet_id) WHERE settled_at IS NULL`, |
| + `CREATE INDEX IF NOT EXISTS admissions_day ON admissions(admitted_at)`, |
| + `CREATE TABLE IF NOT EXISTS fleets ( |
| + fleet_id TEXT PRIMARY KEY, |
| + box TEXT NOT NULL, |
| + witness_generation INTEGER NOT NULL, |
| + owners TEXT NOT NULL, |
| + opened_at INTEGER NOT NULL, |
| + state TEXT NOT NULL, |
| + predecessor TEXT NOT NULL DEFAULT '', |
| + acknowledged_by TEXT NOT NULL DEFAULT '' |
| + )`, |
| + `CREATE TABLE IF NOT EXISTS uncertainties ( |
| + id INTEGER PRIMARY KEY, |
| + fleet_id TEXT NOT NULL, |
| + from_ns INTEGER NOT NULL, |
| + to_ns INTEGER, |
| + reason TEXT NOT NULL DEFAULT '' |
| + )`, |
| + `CREATE TABLE IF NOT EXISTS inherited ( |
| + fleet_id TEXT NOT NULL, |
| + uncertainty_id INTEGER NOT NULL, |
| + PRIMARY KEY (fleet_id, uncertainty_id) |
| + )`, |
| + `CREATE TABLE IF NOT EXISTS holds ( |
| + day TEXT NOT NULL, |
| + account TEXT NOT NULL DEFAULT '', |
| + app TEXT NOT NULL DEFAULT '', |
| + uncertainty_id INTEGER NOT NULL, |
| + written_at INTEGER NOT NULL, |
| + PRIMARY KEY (day, account, app, uncertainty_id) |
| + )`, |
| +} |
| + |
| +// The recorder's outcomes and the admission's terminals, as the design's |
| +// transition table names them. transport-lost is a transport completion, |
| +// never a value of how. |
| +const ( |
| + AdmissionFinished = "finished" |
| + AdmissionAbandoned = "abandoned" |
| + AdmissionUnrecorded = "unrecorded" |
| + AdmissionUnissued = "unissued" |
| + AdmissionRecoveredUnknown = "recovered-unknown" |
| +) |
| + |
| +var ( |
| + // ErrAdmissionSettled is returned for a transition on a row that is |
| + // already terminal: a terminal state is final (the model's check). |
| + ErrAdmissionSettled = errors.New("usage ledger: the admission is already settled") |
| + // ErrAdmissionUnknown is returned for a token with no row. |
| + ErrAdmissionUnknown = errors.New("usage ledger: no such admission") |
| + // ErrAdmissionParticipant is returned when a participant completes twice. |
| + ErrAdmissionParticipant = errors.New("usage ledger: the participant has already completed") |
| +) |
| + |
| +// Admission is one row, read back. |
| +type Admission struct { |
| + Token string |
| + FleetID string |
| + Account, App string |
| + AdmittedAt time.Time |
| + TransportDoneAt time.Time // zero while pending |
| + RecorderStartedAt time.Time // zero until Start reached the agent |
| + RecorderDoneAt time.Time // zero while pending |
| + SettledAt time.Time // zero while unsettled |
| + How string // the recorder's outcome, or a terminal |
| +} |
| + |
| +// Settled reports whether no participant remains. |
| +func (a Admission) Settled() bool { return !a.SettledAt.IsZero() } |
| + |
| +func newToken() (string, error) { |
| + var b [16]byte |
| + if _, err := rand.Read(b[:]); err != nil { |
| + return "", err |
| + } |
| + return hex.EncodeToString(b[:]), nil |
| +} |
| + |
| +// Admit writes the admission's row, durably (synchronous=FULL, the |
| +// commit waited for), and returns its token. It is the gate's second half |
| +// in the design (the proof is the fleet's; the registration in the |
| +// fleet's pending set and the recheck after the commit are the gate slice |
| +// of the plan): nothing is sent before this returns. Mirrors |
| +// admissionmodel.WriterCommits. |
| +func (l *DurabilityLedger) Admit(fleetID, account, app string, at time.Time) (string, error) { |
| + if fleetID == "" || account == "" { |
| + return "", errors.New("usage ledger: an admission needs a fleet and an owner") |
| + } |
| + at = at.UTC() |
| + if at.IsZero() { |
| + at = l.now().UTC() |
| + } |
| + token, err := newToken() |
| + if err != nil { |
| + return "", err |
| + } |
| + l.mu.Lock() |
| + defer l.mu.Unlock() |
| + if l.closed { |
| + return "", errors.New("usage ledger: closed") |
| + } |
| + if _, err := l.db.Exec(`INSERT INTO admissions (token, fleet_id, account, app, admitted_at) VALUES (?, ?, ?, ?, ?)`, |
| + token, fleetID, account, app, nsOf(at)); err != nil { |
| + return "", err |
| + } |
| + return token, nil |
| +} |
| + |
| +// Unissued is the terminal of a committed row whose request was refused |
| +// after all (the recheck found the fleet stale or a discovery begun, or |
| +// the gate's budget elapsed): the helper never receives the token, every |
| +// missing timestamp is now, and nothing was sent. Mirrors |
| +// admissionmodel.GateAnswers's refusal and the late-commit rule. |
| +func (l *DurabilityLedger) Unissued(token string, at time.Time) error { |
| + return l.terminal(token, at, AdmissionUnissued) |
| +} |
| + |
| +func (l *DurabilityLedger) terminal(token string, at time.Time, how string) error { |
| + at = at.UTC() |
| + if at.IsZero() { |
| + at = l.now().UTC() |
| + } |
| + l.mu.Lock() |
| + defer l.mu.Unlock() |
| + tx, err := l.db.Begin() |
| + if err != nil { |
| + return err |
| + } |
| + defer tx.Rollback() //nolint:errcheck |
| + a, err := readAdmission(tx, token) |
| + if err != nil { |
| + return err |
| + } |
| + if a.Settled() { |
| + return ErrAdmissionSettled |
| + } |
| + ns := nsOf(at) |
| + if _, err := tx.Exec(`UPDATE admissions SET transport_done_at = COALESCE(transport_done_at, ?), recorder_done_at = COALESCE(recorder_done_at, ?), settled_at = ?, how = ? WHERE token = ?`, |
| + ns, ns, ns, how, token); err != nil { |
| + return err |
| + } |
| + return tx.Commit() |
| +} |
| + |
| +// TransportDone is the SDK middleware's finalize step (lost: the helper |
| +// proven gone with the transport not finalised, a completion and not a |
| +// terminal). When the recorder is done too, the row settles at the later |
| +// of the two. Mirrors admissionmodel.TransportDone and the drain's |
| +// transport-lost. |
| +func (l *DurabilityLedger) TransportDone(token string, at time.Time, lost bool) error { |
| + return l.participant(token, at, "transport", "") |
| +} |
| + |
| +// RecorderStarted registers the recorder participant: the meter's Start |
| +// reached the agent. A row whose recorder never registered is settled as |
| +// unrecorded by the drain after the helper's proven exit; one that did is |
| +// kept until its worker finishes or abandons. Mirrors |
| +// admissionmodel.RecorderStart. |
| +func (l *DurabilityLedger) RecorderStarted(token string, at time.Time) error { |
| + at = at.UTC() |
| + if at.IsZero() { |
| + at = l.now().UTC() |
| + } |
| + l.mu.Lock() |
| + defer l.mu.Unlock() |
| + tx, err := l.db.Begin() |
| + if err != nil { |
| + return err |
| + } |
| + defer tx.Rollback() //nolint:errcheck |
| + a, err := readAdmission(tx, token) |
| + if err != nil { |
| + return err |
| + } |
| + if a.Settled() { |
| + return ErrAdmissionSettled |
| + } |
| + if !a.RecorderStartedAt.IsZero() || !a.RecorderDoneAt.IsZero() { |
| + return ErrAdmissionParticipant |
| + } |
| + if _, err := tx.Exec(`UPDATE admissions SET recorder_started_at = ? WHERE token = ?`, nsOf(at), token); err != nil { |
| + return err |
| + } |
| + return tx.Commit() |
| +} |
| + |
| +// RecorderDone is the recorder's outcome: finished, abandoned, or |
| +// unrecorded (Start failed and made no attempt row, or never registered |
| +// before the drain). When the transport is done too, the row settles at |
| +// the later of the two. Mirrors admissionmodel.RecorderDone. |
| +func (l *DurabilityLedger) RecorderDone(token string, at time.Time, how string) error { |
| + switch how { |
| + case AdmissionFinished, AdmissionAbandoned, AdmissionUnrecorded: |
| + default: |
| + return fmt.Errorf("usage ledger: %q is not a recorder outcome", how) |
| + } |
| + return l.participant(token, at, "recorder", how) |
| +} |
| + |
| +// participant writes one participant's completion and settles the row |
| +// when it is the last. The settlement's hold writing (the uncertainty's |
| +// extension or closure, in the same transaction) is the plan's third |
| +// slice and hooks in at settle. |
| +func (l *DurabilityLedger) participant(token string, at time.Time, which, how string) error { |
| + at = at.UTC() |
| + if at.IsZero() { |
| + at = l.now().UTC() |
| + } |
| + l.mu.Lock() |
| + defer l.mu.Unlock() |
| + tx, err := l.db.Begin() |
| + if err != nil { |
| + return err |
| + } |
| + defer tx.Rollback() //nolint:errcheck |
| + a, err := readAdmission(tx, token) |
| + if err != nil { |
| + return err |
| + } |
| + if a.Settled() { |
| + return ErrAdmissionSettled |
| + } |
| + ns := nsOf(at) |
| + switch which { |
| + case "transport": |
| + if !a.TransportDoneAt.IsZero() { |
| + return ErrAdmissionParticipant |
| + } |
| + a.TransportDoneAt = at |
| + if _, err := tx.Exec(`UPDATE admissions SET transport_done_at = ? WHERE token = ?`, ns, token); err != nil { |
| + return err |
| + } |
| + case "recorder": |
| + if !a.RecorderDoneAt.IsZero() { |
| + return ErrAdmissionParticipant |
| + } |
| + a.RecorderDoneAt = at |
| + a.How = how |
| + if _, err := tx.Exec(`UPDATE admissions SET recorder_done_at = ?, how = ? WHERE token = ?`, ns, how, token); err != nil { |
| + return err |
| + } |
| + } |
| + if !a.TransportDoneAt.IsZero() && !a.RecorderDoneAt.IsZero() { |
| + settled := a.TransportDoneAt |
| + if a.RecorderDoneAt.After(settled) { |
| + settled = a.RecorderDoneAt |
| + } |
| + if _, err := tx.Exec(`UPDATE admissions SET settled_at = ? WHERE token = ?`, nsOf(settled), token); err != nil { |
| + return err |
| + } |
| + if err := l.settled(tx, a, settled); err != nil { |
| + return err |
| + } |
| + } |
| + return tx.Commit() |
| +} |
| + |
| +// settled is the hook the uncertainty slice fills: a settlement under an |
| +// open uncertainty of the admission's fleet writes the holds of |
| +// [admitted_at, settled_at] for every owner the uncertainty reaches and |
| +// closes it when it is the last; one under no open uncertainty writes |
| +// nothing permanent. Here it writes nothing. |
| +func (l *DurabilityLedger) settled(tx *sql.Tx, a Admission, at time.Time) error { return nil } |
| + |
| +// ReadAdmission reads one row. |
| +func (l *DurabilityLedger) ReadAdmission(token string) (Admission, error) { |
| + l.mu.Lock() |
| + defer l.mu.Unlock() |
| + tx, err := l.db.Begin() |
| + if err != nil { |
| + return Admission{}, err |
| + } |
| + defer tx.Rollback() //nolint:errcheck |
| + return readAdmission(tx, token) |
| +} |
| + |
| +func readAdmission(tx *sql.Tx, token string) (Admission, error) { |
| + var a Admission |
| + var admitted int64 |
| + var transport, started, done, settled sql.NullInt64 |
| + err := tx.QueryRow(`SELECT token, fleet_id, account, app, admitted_at, transport_done_at, recorder_started_at, recorder_done_at, settled_at, how FROM admissions WHERE token = ?`, token). |
| + Scan(&a.Token, &a.FleetID, &a.Account, &a.App, &admitted, &transport, &started, &done, &settled, &a.How) |
| + if errors.Is(err, sql.ErrNoRows) { |
| + return Admission{}, ErrAdmissionUnknown |
| + } |
| + if err != nil { |
| + return Admission{}, err |
| + } |
| + a.AdmittedAt = time.Unix(0, admitted).UTC() |
| + if transport.Valid { |
| + a.TransportDoneAt = time.Unix(0, transport.Int64).UTC() |
| + } |
| + if started.Valid { |
| + a.RecorderStartedAt = time.Unix(0, started.Int64).UTC() |
| + } |
| + if done.Valid { |
| + a.RecorderDoneAt = time.Unix(0, done.Int64).UTC() |
| + } |
| + if settled.Valid { |
| + a.SettledAt = time.Unix(0, settled.Int64).UTC() |
| + } |
| + return a, nil |
| +} |
| + |
| +// UnsettledAdmissions lists the unsettled rows of the given fleets (all |
| +// fleets when none is given), oldest first: what the barrier and a |
| +// discovery count. |
| +func (l *DurabilityLedger) UnsettledAdmissions(fleetIDs ...string) ([]Admission, error) { |
| + l.mu.Lock() |
| + defer l.mu.Unlock() |
| + q := `SELECT token FROM admissions WHERE settled_at IS NULL ORDER BY admitted_at, token` |
| + var args []any |
| + if len(fleetIDs) > 0 { |
| + q = `SELECT token FROM admissions WHERE settled_at IS NULL AND fleet_id IN (` + placeholders(len(fleetIDs)) + `) ORDER BY admitted_at, token` |
| + for _, f := range fleetIDs { |
| + args = append(args, f) |
| + } |
| + } |
| + rows, err := l.db.Query(q, args...) |
| + if err != nil { |
| + return nil, err |
| + } |
| + var tokens []string |
| + for rows.Next() { |
| + var t string |
| + if err := rows.Scan(&t); err != nil { |
| + rows.Close() |
| + return nil, err |
| + } |
| + tokens = append(tokens, t) |
| + } |
| + rows.Close() |
| + tx, err := l.db.Begin() |
| + if err != nil { |
| + return nil, err |
| + } |
| + defer tx.Rollback() //nolint:errcheck |
| + out := make([]Admission, 0, len(tokens)) |
| + for _, t := range tokens { |
| + a, err := readAdmission(tx, t) |
| + if err != nil { |
| + return nil, err |
| + } |
| + out = append(out, a) |
| + } |
| + return out, nil |
| +} |
| + |
| +func placeholders(n int) string { |
| + s := "" |
| + for i := 0; i < n; i++ { |
| + if i > 0 { |
| + s += "," |
| + } |
| + s += "?" |
| + } |
| + return s |
| +} |
diff --git a/internal/usage/durabilityadmissions_test.go b/internal/usage/durabilityadmissions_test.go| new file mode 100644 |
| index 000000000..afa23c9c2 |
| --- /dev/null |
| +++ b/internal/usage/durabilityadmissions_test.go |
| @@ -0,0 +1,145 @@ |
| +package usage |
| + |
| +import ( |
| + "errors" |
| + "testing" |
| + "time" |
| +) |
| + |
| +func openAdmissionLedger(t *testing.T) (*DurabilityLedger, func(time.Duration)) { |
| + t.Helper() |
| + dir := t.TempDir() |
| + now := time.Date(2026, 10, 2, 23, 57, 0, 0, time.UTC) |
| + clock := func() time.Time { return now } |
| + l, err := OpenDurabilityLedger(dir, "box", clock, DurabilityLedgerOptions{}) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + t.Cleanup(func() { l.Close(t.Context(), nil) }) |
| + return l, func(d time.Duration) { now = now.Add(d) } |
| +} |
| + |
| +// Mirrors admissionmodel.TestUnchangedFleetHoldsNothing and |
| +// TestThreeMidnightsBothOrders: a row is unsettled until both participants |
| +// complete, settles at the later completion whichever is last, and a |
| +// terminal is final. |
| +func TestAdmissionSettlesAtTheLaterParticipant(t *testing.T) { |
| + for _, transportLast := range []bool{false, true} { |
| + l, advance := openAdmissionLedger(t) |
| + tok, err := l.Admit("fleet-1", "alice", "app", time.Time{}) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + a, err := l.ReadAdmission(tok) |
| + if err != nil || a.Settled() || a.Account != "alice" || a.AdmittedAt.IsZero() { |
| + t.Fatalf("after Admit: %+v %v", a, err) |
| + } |
| + if open, _ := l.UnsettledAdmissions("fleet-1"); len(open) != 1 || open[0].Token != tok { |
| + t.Fatalf("unsettled: %+v", open) |
| + } |
| + advance(12*time.Hour + 3*time.Minute) // the 3rd |
| + first := func() error { return l.TransportDone(tok, time.Time{}, false) } |
| + second := func() error { return l.RecorderDone(tok, time.Time{}, AdmissionFinished) } |
| + if transportLast { |
| + first, second = second, first |
| + if err := l.RecorderStarted(tok, time.Time{}); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + if err := first(); err != nil { |
| + t.Fatal(err) |
| + } |
| + if a, _ := l.ReadAdmission(tok); a.Settled() { |
| + t.Fatal("settled with one participant pending") |
| + } |
| + advance(12*time.Hour + 2*time.Minute) // the 4th |
| + if err := second(); err != nil { |
| + t.Fatal(err) |
| + } |
| + a, _ = l.ReadAdmission(tok) |
| + if !a.Settled() || !a.SettledAt.Equal(time.Date(2026, 10, 4, 0, 2, 0, 0, time.UTC)) || a.How != AdmissionFinished { |
| + t.Fatalf("transportLast=%v: %+v", transportLast, a) |
| + } |
| + if open, _ := l.UnsettledAdmissions(); len(open) != 0 { |
| + t.Fatalf("still unsettled: %+v", open) |
| + } |
| + // a terminal is final |
| + if err := l.TransportDone(tok, time.Time{}, false); !errors.Is(err, ErrAdmissionSettled) { |
| + t.Fatalf("a transition on a settled row: %v", err) |
| + } |
| + if err := l.Unissued(tok, time.Time{}); !errors.Is(err, ErrAdmissionSettled) { |
| + t.Fatalf("unissued after settlement: %v", err) |
| + } |
| + } |
| +} |
| + |
| +// Mirrors admissionmodel.TestCommitAfterDiscoveryIsUnissued and the |
| +// late-commit rule: unissued is a terminal with every missing timestamp |
| +// now; a participant cannot complete twice; an unknown token is said. |
| +func TestAdmissionUnissuedAndParticipants(t *testing.T) { |
| + l, advance := openAdmissionLedger(t) |
| + tok, _ := l.Admit("fleet-1", "alice", "app", time.Time{}) |
| + advance(time.Minute) |
| + if err := l.Unissued(tok, time.Time{}); err != nil { |
| + t.Fatal(err) |
| + } |
| + a, _ := l.ReadAdmission(tok) |
| + if !a.Settled() || a.How != AdmissionUnissued || a.TransportDoneAt.IsZero() || a.RecorderDoneAt.IsZero() || !a.SettledAt.Equal(a.TransportDoneAt) { |
| + t.Fatalf("unissued: %+v", a) |
| + } |
| + tok2, _ := l.Admit("fleet-1", "alice", "app", time.Time{}) |
| + if err := l.TransportDone(tok2, time.Time{}, false); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := l.TransportDone(tok2, time.Time{}, false); !errors.Is(err, ErrAdmissionParticipant) { |
| + t.Fatalf("the transport twice: %v", err) |
| + } |
| + if err := l.RecorderStarted(tok2, time.Time{}); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := l.RecorderStarted(tok2, time.Time{}); !errors.Is(err, ErrAdmissionParticipant) { |
| + t.Fatalf("Start twice: %v", err) |
| + } |
| + if err := l.RecorderDone("nope", time.Time{}, AdmissionFinished); !errors.Is(err, ErrAdmissionUnknown) { |
| + t.Fatalf("an unknown token: %v", err) |
| + } |
| + if err := l.RecorderDone(tok2, time.Time{}, "lost"); err == nil { |
| + t.Fatal("an outcome outside the alphabet accepted") |
| + } |
| +} |
| + |
| +// Mirrors admissionmodel.TestHistoricalTermCountsUnrecordedSettlements's |
| +// premise: unrecorded is a settlement (the transport ran), and the rows |
| +// survive the ledger's close and reopen. |
| +func TestAdmissionUnrecordedSettlesAndSurvivesReopen(t *testing.T) { |
| + dir := t.TempDir() |
| + now := time.Date(2026, 10, 1, 23, 0, 0, 0, time.UTC) |
| + l, err := OpenDurabilityLedger(dir, "box", func() time.Time { return now }, DurabilityLedgerOptions{}) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + tok, _ := l.Admit("fleet-1", "alice", "app", time.Time{}) |
| + now = now.Add(90 * time.Minute) |
| + if err := l.TransportDone(tok, time.Time{}, false); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := l.RecorderDone(tok, time.Time{}, AdmissionUnrecorded); err != nil { |
| + t.Fatal(err) |
| + } |
| + pending, _ := l.Admit("fleet-1", "alice", "app", time.Time{}) |
| + if err := l.Close(t.Context(), nil); err != nil { |
| + t.Fatal(err) |
| + } |
| + l2, err := OpenDurabilityLedger(dir, "box", func() time.Time { return now }, DurabilityLedgerOptions{}) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + defer l2.Close(t.Context(), nil) |
| + a, err := l2.ReadAdmission(tok) |
| + if err != nil || !a.Settled() || a.How != AdmissionUnrecorded { |
| + t.Fatalf("after reopen: %+v %v", a, err) |
| + } |
| + if open, _ := l2.UnsettledAdmissions(); len(open) != 1 || open[0].Token != pending { |
| + t.Fatalf("the pending row after reopen: %+v", open) |
| + } |
| +} |
diff --git a/internal/usage/durabilityledger.go b/internal/usage/durabilityledger.go| index 37c6f4013..fad2a74e6 100644 |
| --- a/internal/usage/durabilityledger.go |
| +++ b/internal/usage/durabilityledger.go |
| @@ -667,7 +667,7 @@ func (l *DurabilityLedger) recover(handoff string) error { |
| return fmt.Errorf("usage ledger %s: %w", l.dir, err) |
| } |
| l.db = db |
| - for _, stmt := range ledgerSchema { |
| + for _, stmt := range append(append([]string(nil), ledgerSchema...), admissionSchema...) { |
| if _, err := db.Exec(stmt); err != nil { |
| return fmt.Errorf("usage ledger %s: schema: %w", l.dir, err) |
| } |