amadan
Review six answered; inventory on retained descriptors, third slice: the fleet holds its pins and each owned file's retained chain for its life (fileFacts.keep, FleetInventory.Close, recheck and Descriptors; checkOwnedFileOn and strictTreeDBsOn against a fleet's pins), rechecks every chain, name and root by descriptor before each look's rebuild and releases them at Close, a report keeping none; the root re-established higher when a later platform link's ascent needs it, a slash target pinned, the platform-directory claim refusing a link at the claimed directory, a relative claim refused, a failed look after a changed directory a problem, an absolute tenant link spelled through the tree's own root walked relative from the confined root; tests through two composed links, the slash target, the directory already listed replaced under the inspection, a directory removed before its listing, an unsearchable directory, a mode-only change on a chain that only the retained edge sees, descriptors released; the model checks right after the final recovery; design revision 13 and the plan reconciled with the implemented boundary
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
12 files changed,
+721
−81
DISPOSITIONS-replicator-pinned-root.md+19 −0docs/superpowers/plans/2026-10-03-replicator-authority-plan.md+6 −6docs/superpowers/specs/2026-10-03-replicator-pinned-roots-design.md+4 −4docs/superpowers/specs/2026-10-03-replicator-pinned-roots-spikes.md+2 −2internal/ops/admissionmodel/model_test.go+8 −2internal/ops/authoritybridge.go+163 −24internal/ops/filefacts_other.go+1 −0internal/ops/filefacts_unix.go+1 −0internal/ops/litestream_metered.go+22 −1internal/ops/litestream_metered_test.go+70 −0internal/ops/litestream_owners.go+173 −34internal/ops/litestream_owners_unix_test.go+252 −8
CI passed — run details
diff --git a/DISPOSITIONS-replicator-pinned-root.md b/DISPOSITIONS-replicator-pinned-root.md| index c020524ab..21822bd63 100644 |
| --- a/DISPOSITIONS-replicator-pinned-root.md |
| +++ b/DISPOSITIONS-replicator-pinned-root.md |
| @@ -95,3 +95,22 @@ Opus: accept with fixes, no P1, two P2. Astra: accept with fixes, no P1, two P2. |
| The second slice (`checkOwnedFile` on the retained chain, `resolvePlatformLinks`, `walkWithoutLinks`, `sqliteHeader` and `openRegularNoFollow` retired, `Facts.Nlink`) lands with these answers; Astra's note that pins live per call and Opus P3-6 stand until the fleet holds its chains. |
| +## Review six (of `cc1500a9`: the fifth review answered, the inventory's second slice) |
| + |
| +Opus: accept with fixes, no P1, one P2. Astra: accept with fixes, no P1, four P2. Reports: `logs/opus-replroot-review6.md`, `logs/astra-replroot-review6.md`. |
| + |
| +| Finding | Disposition | |
| +|---|---| |
| +| Astra 1: a later platform link needing a higher root was refused; an absolute target of the slash was not pinned | a link met beneath the root whose ascent needs more pops than the resolved stack has before the root re-establishes the root that many levels up by the same platform walk and the descent restarts (terminating: each restart is strictly higher); a target of the slash pins the slash; the slash is never confined. Tests: two links, the second ascending higher, the composed tree walked. | |
| +| Astra 2: a failed look after a changed directory was churn | "cannot be looked at"; a directory that is a file or a link now, or gone, is churn, named in the "on purpose" list (Opus P3-4). The failure itself cannot be forced in a test here and the test file says so. | |
| +| Astra 3: the platform-directory claim through a link at the claimed directory was accepted; a relative claim was not refused | `acquire` says whether the spelled directory itself was a link and the platform-directory claim refuses it with today's string; a relative spelling is "its directory is not an absolute path". Tests for both. The tree claim one level below a platform link, which today's `Lstat` of the parent refused, stays accepted (safe: the link is vetted) and is recorded in the design and plan. | |
| +| Astra 4, Opus P3-5: the tests did not distinguish retention of the directory already listed, per-entry errors, or the listing's `ENOENT` at depth | `TestStrictTreeInspectsTheDirectoryItListed`: the listed directory renamed away and another put at its spelling between the listing and the inspection (the headers read and the links classified are the retained directory's, and its `Recheck` reports the changed edge); a directory removed between its acquisition and its listing (a second seam, `treeBeforeList`) is churn; a directory without search permission is "cannot be read" as a whole (a per-entry failure with search permission present, `Entry.Err`, cannot be produced here and the test says so). | |
| +| Opus P2-1: an absolute tenant link spelled through the tree's own root (the `STATE_DIRECTORY` shape) became a link to a link | walked relative from the confined root, since the predicate vetted that link when the root was acquired, and `Within` still judges the result; a test; the link-shapes line names it. | |
| +| Astra 5, Opus P3-1: the plan, design and dispositions still said "exactly", the first link's directory, a changed directory skipped, Astra 2's layout refused | the plan's root rule, predicate, inventory and tests paragraphs rewritten; design revision 13 (Part A's predicate "within the pinned roots", the two safe wider shapes, the kept platform-directory refusal, the walk's pre-pin absent exception); the leftover venv sentence removed. | |
| +| Opus P3-2: reason strings changed | the climb's messages name the resolved spelling again (`pathOf`, from the retained names, never a look); the rest are listed here as changed: the predicate's "in a directory that is not the platform's" and "not under the platform's" carry no `: <why>` suffix for a relative target and `root()`'s phrase for an absolute one; a dangling target is "leads through another link: X cannot be looked at"; a relative further link names the target spelling; the non-unix message is "cannot be looked at". None is an Accept key. | |
| +| Opus P3-3: the predicate comment overclaimed for relative targets | the comment says which components are examined (as today, a relative target's named components between the popped ancestor and the parent are not). | |
| +| Astra 6, Opus P3-7: the final recovery in the schedule's quiescence ran unchecked; the spike note's mutant list and wording | checked right after it; the list and wording corrected; the fixed-bound test's comment says which order kills. | |
| +| Opus (hygiene): `t.Fatal` on the walk's goroutine; an ignored `Chmod` | errors recorded and checked after the walk; the `Chmod` checked. | |
| + |
| +The third slice lands with these answers: `fileFacts.keep` (the retained chain), `FleetInventory.Close`, `recheck` and `Descriptors`, `checkOwnedFileOn` and `strictTreeDBsOn` against a fleet's pins, `MeteredFleet.held` rechecked before every look's rebuild and released at Close, a report keeping no descriptors; tests for retention, the mode-only change on a chain directory that only the retained edge sees, the fresh inventory released after a clean look, and descriptors released at Close. The roots' own recheck is observable only where a chain's `Recheck` sees the same edge (every root sits on an owned file's stack), and is kept as the design's ancestry rebuild. |
| + |
diff --git a/docs/superpowers/plans/2026-10-03-replicator-authority-plan.md b/docs/superpowers/plans/2026-10-03-replicator-authority-plan.md| index 7aad067f9..7d81d6544 100644 |
| --- a/docs/superpowers/plans/2026-10-03-replicator-authority-plan.md |
| +++ b/docs/superpowers/plans/2026-10-03-replicator-authority-plan.md |
| @@ -1,6 +1,6 @@ |
| # The replicator's authority: the task plan for the policy-independent parts |
| -Date: 2026-10-03. For `docs/superpowers/specs/2026-10-03-replicator-pinned-roots-design.md` (revision 11). This is the plan's steps 1 to 3 as tasks, each its own reviewed commit on `design-replicator-pinned-root`, each with the full `internal/ops`, `internal/lease`, `internal/usage`, `internal/registry` and `internal/activator` run. Step 0's admission model and step 1's first task (`internal/authority`) are done, four code reviews answered; the VFS spike (step 0's other half) needs a box with amd64 and arm64 and gates Part B only. Decisions 10 to 13 and approval H are parameters: nothing below chooses them, and nothing below is wired into a running fleet until the design's support conditions are declared or checked. |
| +Date: 2026-10-03. For `docs/superpowers/specs/2026-10-03-replicator-pinned-roots-design.md` (revision 13). This is the plan's steps 1 to 3 as tasks, each its own reviewed commit on `design-replicator-pinned-root`, each with the full `internal/ops`, `internal/lease`, `internal/usage`, `internal/registry` and `internal/activator` run. Step 0's admission model and step 1's first task (`internal/authority`) are done, four code reviews answered; the VFS spike (step 0's other half) needs a box with amd64 and arm64 and gates Part B only. Decisions 10 to 13 and approval H are parameters: nothing below chooses them, and nothing below is wired into a running fleet until the design's support conditions are declared or checked. |
| ## Task 1 (done): `internal/authority` |
| @@ -8,19 +8,19 @@ Date: 2026-10-03. For `docs/superpowers/specs/2026-10-03-replicator-pinned-roots |
| ## Task 2: the inventory on `Dir`s (Part A's second half) |
| -**Which roots a fleet pins.** The design says a root is built once per fleet from `/` and kept, and that a tenant root is reached from a platform root through the platform-link predicate; it does not list the roots. The rule for this task, where the platform's authority ends: for every spelled directory the inventory is asked about (a unit route's `db_path` tree, the activator's data directory, the registry's and console's directories, a file claim's directory), the pinned platform root is **the parent of the first link's directory** when the spelling contains a platform link (`/var/lib/carlos-app/<host>` is a link in `/var/lib/carlos-app`, so the root is `/var/lib`, and systemd's `../private/carlos-app/<host>` pops from `carlos-app` to the root and no further), else **the parent of the first component that is not the platform's** (another user's, or writable by group or others: a tenant's tree top, so a file claim deep in a tenant's tree gets the tenant's parent as its root), else **the parent of the spelled directory**; and **the parent of an absolute platform-link target** is pinned too, so that walk enters it (today's predicate accepted an all-platform absolute target anywhere; the pin is established by the same platform walk). `OpenRoot(slash, root, platform)` requires every component from `/` to the root to pass the platform rule. A tenant-owned component above a platform-owned one (Astra 2's layout) puts the root above it; the directories beneath are the tenant's chain, retained and rechecked, so the swap that layout allows is a changed edge at the next look, never a steer, and the claim's own rules (the directory above a tree's root the platform's) decide what is owned. Roots are keyed by path and shared across the fleet's spellings; `Pins` carries them all with the pinned `/`; the slash itself is pinned deliberately (`SlashRoot`) when the rule names it. |
| +**Which roots a fleet pins.** The design says a root is built once per fleet from `/` and kept, and that a tenant root is reached from a platform root through the platform-link predicate; it does not list the roots. The rule for this task, where the platform's authority ends: for every spelled directory the inventory is asked about (a unit route's `db_path` tree, the activator's data directory, the registry's and console's directories, a file claim's directory), the pinned platform root is **as many levels above the first link's directory as its target's leading ascent needs, one at least** when the spelling contains a platform link (`/var/lib/carlos-app/<host>` is a link in `/var/lib/carlos-app` with the target `../private/carlos-app/<host>`, so the root is `/var/lib`, and the ascent pops from `carlos-app` to the root and no further; a later link whose ascent needs a higher root re-establishes it by the same platform walk and the descent restarts), else **the parent of the first component that is not the platform's** (another user's, or writable by group or others: a tenant's tree top, so a file claim deep in a tenant's tree gets the tenant's parent as its root), else **the parent of the spelled directory**; and **the parent of an absolute platform-link target** is pinned too, so that walk enters it (today's predicate accepted an all-platform absolute target anywhere; the pin is established by the same platform walk). `OpenRoot(slash, root, platform)` requires every component from `/` to the root to pass the platform rule. A tenant-owned component above a platform-owned one (Astra 2's layout) puts the root above it; the directories beneath are the tenant's chain, retained and rechecked, so the swap that layout allows is a changed edge at the next look, never a steer, and the claim's own rules (the directory above a tree's root the platform's) decide what is owned. Roots are keyed by path and shared across the fleet's spellings; `Pins` carries them all with the pinned `/`; the slash itself is pinned deliberately (`SlashRoot`) when the rule names it. |
| -**The platform-link predicate on descriptors.** `resolvePlatformLinks(dir string)` becomes `platformDir(pins, dir string) (*authority.Dir, string)`: from the root's `Dir`, each remaining component is `Child(name, nil)`; a component that is a link (`ErrLink`) is read with `Readlink` from its directory's `Dir` and must satisfy exactly today's predicate (the link's directory the platform's, the link itself the platform's by `Lookat`'s facts, the target walked with `Walk(target, pins)`: relative from the link's directory, absolute from the pinned `/`, with `..` only in the leading run, which `Walk` enforces, no further link, which `Walk`'s `Link` class refuses, ending at a directory whose parent is the platform's); the result's `Dir` is the next component's parent and the walk continues beneath it. A tenant root so reached is `Confine`d once by the inventory before anything walks from it. The predicate's restrictions do not change; its error strings stay, because the Accept list and the rendering compare them. |
| +**The platform-link predicate on descriptors.** `resolvePlatformLinks(dir string)` becomes `platformDir(pins, dir string) (*authority.Dir, string)`: from the root's `Dir`, each remaining component is `Child(name, nil)`; a component that is a link (`ErrLink`) is read with `Readlink` from its directory's `Dir` and must satisfy today's predicate within the pinned roots (a target that leaves every pinned root is refused where the pathname walk followed it) (the link's directory the platform's, the link itself the platform's by `Lookat`'s facts, the target walked with `Walk(target, pins)`: relative from the link's directory, absolute from the pinned `/`, with `..` only in the leading run, which `Walk` enforces, no further link, which `Walk`'s `Link` class refuses, ending at a directory whose parent is the platform's); the result's `Dir` is the next component's parent and the walk continues beneath it. A tenant root so reached is `Confine`d once by the inventory before anything walks from it. The predicate's restrictions do not loosen (two shapes are wider than today and safe: a tree claim spelled one level below a platform link, which today's `Lstat` of the parent refused, and a link's target parent checked on the retained chain); its problem strings are rendered, not compared (the Accept list compares skipped entries only), and the changed ones are listed in the dispositions. |
| -**`strictTreeDBs` on a `Dir`.** `strictTreeDBs(root string)` keeps its signature and its three string lists (paths spelled as today, `root` joined with the names the listing gave, so the Accept list and every existing test compare unchanged) and becomes a walk on `Dir`s: `List` for the entries; a subdirectory by `Child(name, &entry.Facts)` (a changed entry, `ENOTDIR`, `ErrLink` or absent is churn, as a vanished entry is today; any other error is "cannot be read"); a regular file's header by `OpenRegular(name, entry.Facts)`; a link's target by `Readlink` and `Walk(target, pins)` from the directory that listed it, classified by the design's explicit classes and nothing silent: `Directory`, or a database-named link, is "a link that may lead to a database"; `Regular` within the tenant's root (`Result.Within(tenantRoot)`, whether the target was spelled relative or absolute) is a skipped link entry, as today; `Regular` or `Directory` not within it (a sibling tenant under the same pin, another pin) is "a link out of the tree"; `Link` is "a link to a link that may lead to a database" (today's classification followed it; this refuses rather than follows); `Other` is "a link to something that is neither a file nor a directory"; `Absent` is churn; `ErrAscentPastRoot`, `ErrOutside` and `ErrNotPlatform` are "a link out of the tree"; `ErrAscentAfterName`, `ErrNotDirectory` and every other error are "a link whose target cannot be looked at". Depth and dot-directory rules are unchanged. `statTarget` and `linktarget_*.go` are deleted. |
| +**`strictTreeDBs` on a `Dir`.** `strictTreeDBs(root string)` keeps its signature and its three string lists (paths spelled as today, `root` joined with the names the listing gave, so the Accept list and every existing test compare unchanged) and becomes a walk on `Dir`s: `List` for the entries; a subdirectory by `Child(name, &entry.Facts)` (a changed entry is looked at again once and, if still a directory, acquired with the fresh facts and walked, a second change being "changed under the look"; a failed look is "cannot be looked at"; `ENOTDIR`, `ErrLink` or absent is churn, as a vanished entry is today; any other error is "cannot be read"); a listing's per-entry failure is that entry's problem and the rest is walked; a listing's `ENOENT` at depth is churn; a regular file's header by `OpenRegular(name, entry.Facts)`; a link's target by `Readlink` and `Walk(target, pins)` from the directory that listed it, classified by the design's explicit classes and nothing silent: `Directory`, or a database-named link, is "a link that may lead to a database"; `Regular` within the tenant's root (`Result.Within(tenantRoot)`, whether the target was spelled relative or absolute) is a skipped link entry, as today; `Regular` or `Directory` not within it (a sibling tenant under the same pin, another pin) is "a link out of the tree"; `Link` is "a link to a link that may lead to a database" (today's classification followed it; this refuses rather than follows); `Other` is "a link to something that is neither a file nor a directory"; `Absent` is churn; `ErrAscentPastRoot`, `ErrOutside` and `ErrNotPlatform` are "a link out of the tree"; `ErrAscentAfterName`, `ErrNotDirectory` and every other error are "a link whose target cannot be looked at". Depth and dot-directory rules are unchanged. `statTarget` and `linktarget_*.go` are deleted. |
| -**`checkOwnedFile` on a chain.** The chain is acquired downward from the platform root as `Dir`s (the root, the predicate's result, each component by `Child`), never by `Lstat` of a joined path; each check it makes today (no link, one filesystem, a regular file with one name, the owner rules by kind, the `-wal`/`-shm`/`-journal` siblings regular with one name, by `Lookat` from the file's directory) is made on the retained chain; `fileFacts` gains the file's directory `Dir` (the design: one descriptor stack per owned file, counted in the fleet's rendering), kept for the fleet's life so every later look runs `Recheck` on it and `Lookat` of the file's name, and so litestream's opens (Part B, later) take it. `fleetInventory` closes the chains of files it does not keep. |
| +**`checkOwnedFile` on a chain.** The chain is acquired downward from the platform root as `Dir`s (the root, the predicate's result, each component by `Child`), never by `Lstat` of a joined path; each check it makes today (no link, one filesystem, a regular file with one name, the owner rules by kind, the `-wal`/`-shm`/`-journal` siblings regular with one name, by `Lookat` from the file's directory) is made on the retained chain; `fileFacts` gains the file's retained chain (`keep`: the directory `Dir` with its stack, the file's name and facts; the design's one descriptor stack per owned file, counted by `FleetInventory.Descriptors`), kept for the fleet's life (`MeteredFleet.held`) so every look runs `Recheck` on every chain and `Lookat` of the file's name and `AncestryChanged` on every root before it rebuilds the inventory, and so litestream's opens (Part B, later) take it; `fleetInventory` closes the chains of files it does not keep, a report (`FleetDBOwners`) keeps none, and the fleet releases its chains at Close. The third slice builds this. |
| **Before wiring.** A census of the link classes on each box's live trees (the venv and `node_modules` shapes are problems under the new classes, where today they are skipped entries) and the owner's decision on them beside decisions 10 to 13; the live `DBPath`s checked against the root rule. |
| **The namespace condition.** A route whose instance runs in its own mount namespace is classified in the agent's namespace and rendered as such; until the support condition is declared it is a refusal rule (a problem naming the route), not a silence. |
| -**Tests.** Every existing owners test keeps passing with its strings; added, from the design's Part A list at the inventory level: each link class rendered as written and none silent (a directory target, a database-named link, a regular file in the tree, a regular file under another pin, a regular file in a sibling tenant under the same pin, an absolute target inside the tenant's own tree (skipped, not out of the tree), a link to a link, a FIFO target, an absent target, an ascent past the tenant root, an absolute target outside every pin, a non-platform component in an absolute target); a tenant root reached through systemd's relative link shape and through an absolute one; Astra 2's layout refused at the root; a swap of the tenant directory between the listing and the classification changing nothing; the retained chain's `Recheck` at the next look reporting a moved or replaced directory; the descriptor count per fleet equal to the retained stacks' total. |
| +**Tests.** Every existing owners test keeps passing with its strings; added, from the design's Part A list at the inventory level: each link class rendered as written and none silent (a directory target, a database-named link, a regular file in the tree, a regular file under another pin, a regular file in a sibling tenant under the same pin, an absolute target inside the tenant's own tree (skipped, not out of the tree), a link to a link, a FIFO target, an absent target, an ascent past the tenant root, an absolute target outside every pin, a non-platform component in an absolute target); a tenant root reached through systemd's relative link shape and through an absolute one; Astra 2's layout walked, with the root above the tenant-owned component and the chain beneath retained; a swap of the tenant directory between the listing and the classification changing nothing; the retained chain's `Recheck` at the next look reporting a moved or replaced directory; the descriptor count per fleet equal to the retained stacks' total. |
| ## Task 3: required coverage (Part D) |
diff --git a/docs/superpowers/specs/2026-10-03-replicator-pinned-roots-design.md b/docs/superpowers/specs/2026-10-03-replicator-pinned-roots-design.md| index 1cd4774ac..264255a87 100644 |
| --- a/docs/superpowers/specs/2026-10-03-replicator-pinned-roots-design.md |
| +++ b/docs/superpowers/specs/2026-10-03-replicator-pinned-roots-design.md |
| @@ -1,6 +1,6 @@ |
| # The replicator's authority: retained descriptors, a durable admission, claims by identity |
| -Date: 2026-10-03. Status: design, revision 12 (Part A's root rule stated as the plan has it: the platform walk ends where the platform's authority ends, a tenant-owned component's parent, and the chain beneath is retained and rechecked rather than refused). Revision 11 was (the fourth code review: the historical term's filter named `finished` and `abandoned` and so excluded `unrecorded`, a settlement whose transport did send; it now excludes only the empty terminals, `unissued` and `cancelled`; the look's order is stated as what safety needs rather than a sequence; the closing look agrees). Revision 10 was revision 9's correction of the lower bound refined after the third code review: the historical term's threshold is the earlier of the registry proof and the last clean look's start, since a witness change is known only to be after the proof; an empty terminal, unissued or cancelled, never sent and is not a settlement for that term; and every look reads the witness and rechecks every retained chain before it inspects files, so a clean look's start bounds every kind of change. Revision 9 was one mechanics correction to the uncertainty's lower bound, found by the second code review of the admission model: an admission admitted before the last clean look, still in flight across a change after it and settled clean before the change was noticed, was outside the bound; the bound now also takes the admission instant of every admission settled at or after the last clean look's start; the invariant I2 is unchanged and the model's named test and schedules cover it). Revision 8 was the accepted text: revision 7 was accepted with fixes by Astra (no P1; one P2: a negative read-back was not conclusive while the admission writer could still commit, and the pending set had three definitions; one P3: the restore wording made a salt mismatch imply a snapshot), both applied here. Revision 6 was rejected by Astra on one P1: a discovery whose write failed had no rule for the admissions that settled before its retry, so the replayed interval could lose a settlement; and three P2s: a helper dying before its recorder started left an admission nobody could finish; the transition table lacked the abort edges and the pending set's cleanup; the lineage record cited a durability discipline the shadow state does not have; each answered here. Revision 5 was rejected by Astra on two P1s: the admission's insertion had no ordering point against discovery, so a discovery between the proof and the commit counted nothing and a later send fell outside the closed uncertainty; construction did not backfill already-closed intervals for a successor's new owners before acknowledgement; and P2s: the transition table and the helper-death rule disagreed and the recorder participant was synthesised while its worker could still run; proof-only callers of admit would have created admissions with no terminal; a terminal `.` lost the directory requirement; conflicting summaries; each answered here. Revision 4 was rejected by Astra on four P1s: the helper has several connections of its own and needs SQLite's within-process lock bookkeeping between them; the producer fence did not cover a helper that outlives the agent with an admission in hand; an intermediate fleet closed before acknowledging broke transitive protection; a fleet could be marked closed with a discovery uncommitted; and four P2s: a dead helper's admissions had no terminal in the running agent, the sleeping binding persisted a reusable inode, the lineage predicate named header fields that identify nothing, the walk's grammar refused a `.` target the predicate accepts; each answered here. Revision 3 was rejected by Astra on five P1s: two SQLite VFSs in one process cannot coordinate through POSIX locks, since the library's own per-inode bookkeeping is in-process; a recorder that abandons early could settle an admission before its transport ended; an admission-only barrier does not cover a discovery whose uncertainty write failed after every admission settled; links at acknowledgement contradicted acknowledging only after settlement, and an inherited uncertainty vanished at the next successor; the ledger's lock does not prove the old process gone; and four P2s: closure at zero pending and the margin's extra day, sleeping claims with no binding, absolute and `.` spellings in the walk, `(dev, ino)` as a lineage; each answered here. Revision 2 was rejected by Astra on three P1s: the VFS delegated SQLite's Unix file methods, which open the SHM and check moved files by pathname themselves; every settled admission wrote a permanent hold, so ordinary replication never read complete; a late owner-scoped extension did not reach successors that had already acknowledged; and six P2s: orphan admissions never terminated, `..` and chain edges unspecified, `dup` sharing the directory offset, the platform-link predicate not preserved and terminal links unclassified, the uid producer without a lifecycle, one billing owner not one replicator; each answered here. Revision 1 was rejected by Astra on five P1s: SQLite canonicalises a `/proc` pathname before it opens anything and this driver passes no `nofollow`; `os.Root` confines but re-resolves every name, so a swap between a listing and an inspection still moves the answer; a `Start` row is not an admission, since the meter sends without one; the attempt-derived window lost a request completed before the change was noticed and the days between an admission and a finish; `closed_at` was not a handover; plus the directory open before `O_DIRECTORY`, per-owner holds the ledger cannot store, a uid with no producer, I3 against Part D, and tests). Branch `design-replicator-pinned-root`, off `feat/carloku-metered-replicator-20261001` at `e1280811`. It answers the eleven findings the twentieth review deferred (`DISPOSITIONS-metered-replicator.md`, review twenty) and keeps decisions 10 to 13 and approval H (`DECISIONS-FOR-PAUL.md`, section D, on the launch branch) as parameters. Nothing is built, wired or applied. |
| +Date: 2026-10-03. Status: design, revision 13 (the sixth code review: the predicate's compatibility stated as within the pinned roots, with the two safe wider shapes and the kept platform-directory refusal; the walk's pre-pin absent exception; the link-shapes line extended). Revision 12 was (Part A's root rule stated as the plan has it: the platform walk ends where the platform's authority ends, a tenant-owned component's parent, and the chain beneath is retained and rechecked rather than refused). Revision 11 was (the fourth code review: the historical term's filter named `finished` and `abandoned` and so excluded `unrecorded`, a settlement whose transport did send; it now excludes only the empty terminals, `unissued` and `cancelled`; the look's order is stated as what safety needs rather than a sequence; the closing look agrees). Revision 10 was revision 9's correction of the lower bound refined after the third code review: the historical term's threshold is the earlier of the registry proof and the last clean look's start, since a witness change is known only to be after the proof; an empty terminal, unissued or cancelled, never sent and is not a settlement for that term; and every look reads the witness and rechecks every retained chain before it inspects files, so a clean look's start bounds every kind of change. Revision 9 was one mechanics correction to the uncertainty's lower bound, found by the second code review of the admission model: an admission admitted before the last clean look, still in flight across a change after it and settled clean before the change was noticed, was outside the bound; the bound now also takes the admission instant of every admission settled at or after the last clean look's start; the invariant I2 is unchanged and the model's named test and schedules cover it). Revision 8 was the accepted text: revision 7 was accepted with fixes by Astra (no P1; one P2: a negative read-back was not conclusive while the admission writer could still commit, and the pending set had three definitions; one P3: the restore wording made a salt mismatch imply a snapshot), both applied here. Revision 6 was rejected by Astra on one P1: a discovery whose write failed had no rule for the admissions that settled before its retry, so the replayed interval could lose a settlement; and three P2s: a helper dying before its recorder started left an admission nobody could finish; the transition table lacked the abort edges and the pending set's cleanup; the lineage record cited a durability discipline the shadow state does not have; each answered here. Revision 5 was rejected by Astra on two P1s: the admission's insertion had no ordering point against discovery, so a discovery between the proof and the commit counted nothing and a later send fell outside the closed uncertainty; construction did not backfill already-closed intervals for a successor's new owners before acknowledgement; and P2s: the transition table and the helper-death rule disagreed and the recorder participant was synthesised while its worker could still run; proof-only callers of admit would have created admissions with no terminal; a terminal `.` lost the directory requirement; conflicting summaries; each answered here. Revision 4 was rejected by Astra on four P1s: the helper has several connections of its own and needs SQLite's within-process lock bookkeeping between them; the producer fence did not cover a helper that outlives the agent with an admission in hand; an intermediate fleet closed before acknowledging broke transitive protection; a fleet could be marked closed with a discovery uncommitted; and four P2s: a dead helper's admissions had no terminal in the running agent, the sleeping binding persisted a reusable inode, the lineage predicate named header fields that identify nothing, the walk's grammar refused a `.` target the predicate accepts; each answered here. Revision 3 was rejected by Astra on five P1s: two SQLite VFSs in one process cannot coordinate through POSIX locks, since the library's own per-inode bookkeeping is in-process; a recorder that abandons early could settle an admission before its transport ended; an admission-only barrier does not cover a discovery whose uncertainty write failed after every admission settled; links at acknowledgement contradicted acknowledging only after settlement, and an inherited uncertainty vanished at the next successor; the ledger's lock does not prove the old process gone; and four P2s: closure at zero pending and the margin's extra day, sleeping claims with no binding, absolute and `.` spellings in the walk, `(dev, ino)` as a lineage; each answered here. Revision 2 was rejected by Astra on three P1s: the VFS delegated SQLite's Unix file methods, which open the SHM and check moved files by pathname themselves; every settled admission wrote a permanent hold, so ordinary replication never read complete; a late owner-scoped extension did not reach successors that had already acknowledged; and six P2s: orphan admissions never terminated, `..` and chain edges unspecified, `dup` sharing the directory offset, the platform-link predicate not preserved and terminal links unclassified, the uid producer without a lifecycle, one billing owner not one replicator; each answered here. Revision 1 was rejected by Astra on five P1s: SQLite canonicalises a `/proc` pathname before it opens anything and this driver passes no `nofollow`; `os.Root` confines but re-resolves every name, so a swap between a listing and an inspection still moves the answer; a `Start` row is not an admission, since the meter sends without one; the attempt-derived window lost a request completed before the change was noticed and the days between an admission and a finish; `closed_at` was not a handover; plus the directory open before `O_DIRECTORY`, per-owner holds the ledger cannot store, a uid with no producer, I3 against Part D, and tests). Branch `design-replicator-pinned-root`, off `feat/carloku-metered-replicator-20261001` at `e1280811`. It answers the eleven findings the twentieth review deferred (`DISPOSITIONS-metered-replicator.md`, review twenty) and keeps decisions 10 to 13 and approval H (`DECISIONS-FOR-PAUL.md`, section D, on the launch branch) as parameters. Nothing is built, wired or applied. |
| ## What stands today |
| @@ -18,9 +18,9 @@ The replicator (`internal/ops/litestream_metered.go`, `litestream_owners.go`; pa |
| **The primitive is a hand-walked, retained directory descriptor, not `os.Root`.** `os.Root` gives containment and re-resolves every name beneath its root; it follows links inside the root, `os.OpenRoot(path)` follows links in the path it is given, Go 1.26.5 walks per component with `openat` and there is no public constructor over a descriptor one has already verified. I1(b) and (c) need the descriptor itself. A small package, `internal/authority`: |
| -- `Dir`: a retained descriptor opened with `openat(parent, name, O_DIRECTORY|O_NOFOLLOW|O_RDONLY|O_CLOEXEC|O_NOCTTY)` (the kernel refuses a FIFO or device with `ENOTDIR` at lookup, before any open effect, which is the one existing directory-open test kept through the new shape), with its `(dev, ino, uid, mode)` read by `fstat` at acquisition and kept. Every method takes a basename (one component, not `.`, `..` or anything with a separator; anything else is refused before any syscall) and resolves it relative to this descriptor alone. `Child(name, expect) (*Dir, error)`: `openat` as above, then `fstat` compared with `expect` (the facts of the entry it is acquired from, when the caller has them); the child keeps a pointer to its parent `Dir` and its name, so the retained chain is a stack. `List() ([]Entry, error)`: a fresh open file description each time (`openat(fd, ".", O_DIRECTORY|O_RDONLY|O_CLOEXEC)`, `fstat`-compared with this descriptor, then `fdopendir` on it), never a `dup` (a `dup` shares the offset, so a second listing would start where the first ended); each entry's facts by `fstatat(fd, name, AT_SYMLINK_NOFOLLOW)`. `Readlink(name)`: `readlinkat`. `Walk(spelling, anchor)`: a captured link target, normalised first (repeated separators collapsed, `.` components dropped, a trailing separator or a terminal `.` component remembered as a requirement that the result be a directory, so `file/.` and `file/` are errors as today; a spelling that was empty before normalisation is refused, and one that is only `.` or only a run of `..` resolves to the anchor or the popped directory as a directory result, which is what today's predicate does with `.`), then walked by components from an anchor: a relative spelling from this `Dir`, an absolute one from the pinned `/` root with the platform walk's rules (every component platform-owned) until it enters a pinned platform root, and from there as a relative walk beneath that root; a leading run of `..` pops the retained stack (never `openat(fd, "..")`, whose answer follows a moved directory to its new parent) and is refused when the stack would pass the anchor's root; a `..` after a named component is refused; each named component is `Child`; the last component is `fstatat(..., AT_SYMLINK_NOFOLLOW)` and its kind is returned: directory, regular, link, other (a FIFO, a socket, a device), absent, or error; a spelling with a trailing separator whose last component is not a directory is an error (`regular-file/` does not become the file). The confinement boundary of a walk is the anchor's root: a relative walk cannot leave the tenant root it started in; an absolute walk must end beneath a pinned platform root or is refused. `OpenRegular(name, looked)`: `openat(fd, name, O_RDONLY|O_NOFOLLOW|O_NONBLOCK|O_NOCTTY|O_CLOEXEC)` then `fstat`, refused unless regular and equal to `looked`'s `(dev, ino)`. `Recheck()`: for this `Dir` and every ancestor on its stack, `fstatat(parent, name, AT_SYMLINK_NOFOLLOW)` compared with the retained child's facts; a difference anywhere is a changed edge. `Facts()`. |
| +- `Dir`: a retained descriptor opened with `openat(parent, name, O_DIRECTORY|O_NOFOLLOW|O_RDONLY|O_CLOEXEC|O_NOCTTY)` (the kernel refuses a FIFO or device with `ENOTDIR` at lookup, before any open effect, which is the one existing directory-open test kept through the new shape), with its `(dev, ino, uid, mode)` read by `fstat` at acquisition and kept. Every method takes a basename (one component, not `.`, `..` or anything with a separator; anything else is refused before any syscall) and resolves it relative to this descriptor alone. `Child(name, expect) (*Dir, error)`: `openat` as above, then `fstat` compared with `expect` (the facts of the entry it is acquired from, when the caller has them); the child keeps a pointer to its parent `Dir` and its name, so the retained chain is a stack. `List() ([]Entry, error)`: a fresh open file description each time (`openat(fd, ".", O_DIRECTORY|O_RDONLY|O_CLOEXEC)`, `fstat`-compared with this descriptor, then `fdopendir` on it), never a `dup` (a `dup` shares the offset, so a second listing would start where the first ended); each entry's facts by `fstatat(fd, name, AT_SYMLINK_NOFOLLOW)`. `Readlink(name)`: `readlinkat`. `Walk(spelling, anchor)`: a captured link target, normalised first (repeated separators collapsed, `.` components dropped, a trailing separator or a terminal `.` component remembered as a requirement that the result be a directory, so `file/.` and `file/` are errors as today; a spelling that was empty before normalisation is refused, and one that is only `.` or only a run of `..` resolves to the anchor or the popped directory as a directory result, which is what today's predicate does with `.`), then walked by components from an anchor: a relative spelling from this `Dir`, an absolute one from the pinned `/` root with the platform walk's rules (every component platform-owned) until it enters a pinned platform root, and from there as a relative walk beneath that root; a leading run of `..` pops the retained stack (never `openat(fd, "..")`, whose answer follows a moved directory to its new parent) and is refused when the stack would pass the anchor's root; a `..` after a named component is refused; each named component is `Child`; the last component is `fstatat(..., AT_SYMLINK_NOFOLLOW)` and its kind is returned: directory, regular, link, other (a FIFO, a socket, a device), absent, or error; a spelling with a trailing separator whose last component is not a directory is an error (`regular-file/` does not become the file). The confinement boundary of a walk is the anchor's root: a relative walk cannot leave the tenant root it started in; an absolute walk must end beneath a pinned platform root or is refused, except that a name absent before any pin is entered answers absent (a dangling target is churn, and nothing beneath a platform-owned directory a tenant cannot write in is reached through it). `OpenRegular(name, looked)`: `openat(fd, name, O_RDONLY|O_NOFOLLOW|O_NONBLOCK|O_NOCTTY|O_CLOEXEC)` then `fstat`, refused unless regular and equal to `looked`'s `(dev, ino)`. `Recheck()`: for this `Dir` and every ancestor on its stack, `fstatat(parent, name, AT_SYMLINK_NOFOLLOW)` compared with the retained child's facts; a difference anywhere is a changed edge. `Facts()`. |
| - `Root`: a `Dir` reached by a walk from `/` (`open("/")` then `Child` per component), each component required to be the platform's (root's or this process's user's), not writable by group or others; the walk's `(dev, ino)` list is the root's *ancestry*. A root is built once per fleet and kept; `lookNow` rebuilds the ancestry from `/` and compares it, and a difference is a change the fleet goes stale on. |
| -- Tenant roots: a unit route's tree is reached from a platform root through the platform-link predicate exactly as `resolvePlatformLinks` has it today (the link itself the platform's, its directory the platform's, its target's `..` only in a leading run, the target's parent the platform's, no further link in the target), evaluated on descriptors: the link read with `Readlink` from its directory's `Dir`, its target walked with `Walk` (anchored as the spelling says: relative from the link's directory, absolute from the pinned `/`), and the resulting `Dir` the tenant root; that predicate is one function, kept separate from the inventory's classification of links (below), and its restrictions do not change here. A file claim's chain top is found as today (upward by ownership from the file's directory) but every directory of the chain is then acquired downward from the platform root as `Dir`s, and the chain is the file's retained stack. Astra 2's layout (a tenant-owned component above a platform-owned one) is where the platform walk ends: the root is the tenant-owned component's parent, no root is built over the tenant's directory, and the directories beneath are the tenant's chain, retained and rechecked, so the swap that layout allows is a changed edge at the next look and never a steer (the plan's root rule, revision 12). |
| +- Tenant roots: a unit route's tree is reached from a platform root through the platform-link predicate as `resolvePlatformLinks` has it today (the link itself the platform's, its directory the platform's, its target's `..` only in a leading run, the target's parent the platform's, no further link in the target), evaluated on descriptors and within the pinned roots (a target that leaves every pinned root is refused where the pathname walk followed it; an absolute target's parent is pinned by the platform walk; a relative target's ascent sets how high the root is pinned): the link read with `Readlink` from its directory's `Dir`, its target walked with `Walk` (anchored as the spelling says: relative from the link's directory, absolute from the pinned `/`), and the resulting `Dir` the tenant root; that predicate is one function, kept separate from the inventory's classification of links (below); its restrictions do not loosen, and two shapes it accepts that today's pathname checks refused are safe and recorded (a tree claim spelled one level below a platform link, and a target parent checked on the retained chain); a platform-directory claim through a link at the claimed directory stays refused. A file claim's chain top is found as today (upward by ownership from the file's directory) but every directory of the chain is then acquired downward from the platform root as `Dir`s, and the chain is the file's retained stack. Astra 2's layout (a tenant-owned component above a platform-owned one) is where the platform walk ends: the root is the tenant-owned component's parent, no root is built over the tenant's directory, and the directories beneath are the tenant's chain, retained and rechecked, so the swap that layout allows is a changed edge at the next look and never a steer (the plan's root rule, revision 12). |
| **Every inventory operation runs on `Dir`s**: `strictTreeDBs` takes a `*Dir`, lists with `List`, descends with `Child(name, entry facts)`, reads a header with `OpenRegular`, and classifies a link's captured spelling with `Walk` on the directory that listed it. The classification is explicit and conservative, and nothing disappears through an unnamed case: a target that is a directory, or a link whose name is a database name, is "a link that may lead to a database" (a problem, as today); a regular file is a skipped link entry (as today); a target that is itself a link is "a link to a link that may lead to a database" (a problem; today's classification followed it, and the only honest replacement is to refuse rather than follow); a target that is other than a file or directory is "a link to something that is neither a file nor a directory" (a problem); a target outside the root is "a link out of the tree" (a problem); absent is churn (as today); any error is "a link whose target cannot be looked at" (as today). The namespace question for an isolated instance (its own mount namespace may resolve a spelling elsewhere) is a declared support condition before wiring: a route whose instance runs in its own mount namespace is classified in the agent's namespace and rendered as such, and until the condition is declared it is a refusal rule, not a silence. |
| @@ -75,7 +75,7 @@ The VFS is the plan's first spike and it must prove, before anything else of Par |
| | Another user's directory inside a tenant's tree. | 12 | Today's tree behaviour (walked; its databases listed under the tenant's claim); refusing it is 12's change, not this design's; Part B's sidecar residual has two possible authors until then. | |
| | Which uids are the platform's workloads; a root-run unit's claims; the activator's own claims. | 13, approval H | `trustedUIDs` empty: refused. | |
| | The link shapes a tenant root may be reached through. | compatibility, not policy | Today's predicate, kept as one function, unchanged. | |
| -| Tenant link shapes the inventory refuses | Under Part A's classes a venv's `bin/python3 -> python -> /usr/bin/python3` is a link to a link and a file out of the tree, and a `node_modules/.bin` entry likewise; today these are skipped entries compared with the Accept list, and problems are never compared, so once wired they refuse the build unconditionally. Whether such links refuse the box, the tenant, or are named and passed over is the owner's, beside decisions 10 to 13; a census of link classes on each box's live trees is part of the pre-wiring check. A dangling link, relative or absolute, stays churn. | |
| +| Tenant link shapes the inventory refuses | Under Part A's classes a venv's `bin/python3 -> python` is a link to a link and `python -> /usr/bin/python3` a file out of the tree, and a `node_modules/.bin` entry likewise; today these are skipped entries compared with the Accept list, and problems are never compared, so once wired they refuse the build unconditionally. An absolute target with a group-writable or other-owned component above its parent (a `0775` `/srv/data`) is out of the tree too. An absolute link spelled through the tree's own root (a DynamicUser app spelling its `STATE_DIRECTORY`, the platform link's path) is walked relative from the confined root and is not a problem. Whether the refused shapes refuse the box, the tenant, or are named and passed over is the owner's, beside decisions 10 to 13; a census of link classes on each box's live trees is part of the pre-wiring check. A dangling link, relative or absolute, stays churn. | |
| | An isolated instance's own mount namespace; bind mounts and device nodes inside a tenant tree. | declared support conditions before wiring | A route outside the declared conditions is refused, not classified silently; the conditions are written in the inventory's rendering per route. | |
| | Handover of a database from the activator's hibernation path to the fleet. | not designed | Refused as a conflict of authorities, as today. | |
diff --git a/docs/superpowers/specs/2026-10-03-replicator-pinned-roots-spikes.md b/docs/superpowers/specs/2026-10-03-replicator-pinned-roots-spikes.md| index 8df19ac29..e69d0b75a 100644 |
| --- a/docs/superpowers/specs/2026-10-03-replicator-pinned-roots-spikes.md |
| +++ b/docs/superpowers/specs/2026-10-03-replicator-pinned-roots-spikes.md |
| @@ -20,9 +20,9 @@ Plan step 0's second spike is code, kept in tree at `internal/ops/admissionmodel |
| - **The named sequences are tests**, one each: an unchanged fleet holds nothing; the three-midnight interval in both completion orders (closes at the settlement plus the margin); Astra's commit-after-discovery (unissued, the uncertainty closed by the empty settlement); review 7's budget expiry then commit; a definite failure cancels and an ambiguous commit is read back either way with the pending set empty after; zero-pending discovery closes at once and holds the margin's extra day across midnight; a failed discovery's latch bars and refuses until the replay, which with nothing pending closes at the replay instant plus the margin; review 6's replay with a counted settlement (the 2nd, 3rd and 4th held, closed at 00:04, a successor built after the replay backfilled); Astra's A, B, C construction and root-first acknowledgement (C refused before B acknowledged A; C inherits U); helper death before and after recorder registration, a `Start` refused after the guard closed; the fence outliving the agent (recovery refused until the helper is gone; recovered-unknown covers the transport that finished after the crash); a healthy chain contributing nothing; and the lower bound reaching a request completed before the change was noticed (the clean look minus the margin). |
| - **The boundaries are separate actions** (after the first code review): the writer's commit, the gate's answer (the recheck), a definite failure, an ambiguous return and its read-back, the send (the helper beginning the transport, which needs a durable issued row, recorded as a violation otherwise; a dead helper's send is a no-op, since a dead helper does nothing), the helper's proven exit (the guard closes) and the drain (transport-lost, unrecorded) are each their own step, so a schedule can put a discovery, a budget expiry or a crash between a commit and its answer, hold a token for a day before sending, or ask for a recorder's `Start` between the exit and the drain. The composites the named tests use (`WriterReturns`, `HelperDies`) are those steps in order. |
| -- **Randomised schedules**: every action above in random interleavings over about four days, 120 steps a schedule, one owner per fleet (owner scope; the box-wide default is a superset); after each step, and right after a recovery before anything else runs: the barrier, stated from the chain and not from the publisher (for every fleet and every owner of it, a durable unsettled admission of that fleet or of any fleet in its unacknowledged ancestry bars the owner from its day on, whether or not the token was ever handed out), the hazard invariant (once the ledger knows of a change, every day on which a transport overlapped the true change is incomplete for its owner; after a crash, every sent admission unsettled at the crash likewise; an issued token never sent has no transport and is covered by the barrier and recovery, not this oracle), a terminal state never changing, and a closed fleet having no latch or pending registration; at quiescence every fleet's owners hold every interval of every uncertainty on its whole chain, no uncertainty is open, and a day with no uncertainty reaching an owner is complete. `ADMISSION_MODEL_SEEDS=30000` passes in about six seconds; the default is 400. Publication is revisable by construction: a day published complete can be re-marked by a later hold (the design's materialisation and watermark), and the hazard invariant is satisfied by that retraction; the barrier's purpose is to make retraction rare, not to make it unnecessary. |
| +- **Randomised schedules**: every action above in random interleavings over about four days, 120 steps a schedule, one owner per fleet (owner scope; the box-wide default is a superset); after each step, and right after every recovery, in the schedule and at quiescence, before anything else runs: the barrier, stated from the chain and not from the publisher (for every fleet and every owner of it, a durable unsettled admission of that fleet or of any fleet in its unacknowledged ancestry bars the owner from its day on, whether or not the token was ever handed out), the hazard invariant (once the ledger knows of a change, every day on which a transport overlapped the true change is incomplete for its owner; after a crash, every sent admission unsettled at the crash likewise; an issued token never sent has no transport and is covered by the barrier and recovery, not this oracle), a terminal state never changing, and a closed fleet having no latch or pending registration; at quiescence every fleet's owners hold every interval of every uncertainty on its whole chain, no uncertainty is open, and a day with no uncertainty reaching an owner is complete. `ADMISSION_MODEL_SEEDS=30000` passes in about six seconds; the default is 400. Publication is revisable by construction: a day published complete can be re-marked by a later hold (the design's materialisation and watermark), and the hazard invariant is satisfied by that retraction; the barrier's purpose is to make retraction rare, not to make it unnecessary. |
| - **What the runs found.** The first runs found the model's own bookkeeping wrong (an owner the fleet did not have; a crash-killed helper's transport end unset). After the second code review made the change its own action, discovered by a later look, Opus's sequence (an admission admitted before the last clean look, in flight across a change after it, settled clean before the discovery) showed the design's lower bound missing that admission's earlier day: **revision 9 of the design** adds the admission instant of every admission settled at or after the last clean look's start to the bound, and the mutant restoring revision 8's rule is killed by the named test. That is the one interleaving the design did not name that the model has found, within the actions it represents and the invariants it checks; the claim is no wider than that. |
| -- **Mutants, in a tree copy**, each killed by a named test or a schedule. After the fourth code review: the historical term filtered to finished and abandoned (the unrecorded test), the cutoff as the running minimum (the fixed-bound test in both row orders), the historical term using the send instant, the historical term dropped, a later witness change ignored (the proof test). After the third code review: the threshold as the later of proof and look, as the look alone, the historical term including empty terminals, the proof not refusing a moved witness, the crash duplicating a landed row, `Close` without its unsettled loop, the latch overlay reaching the fleet's owners only. After the second code review: revision 8's bound (the in-flight test), the bound without the proof (the proof test), the bound without the clean look (the lower-bound test), a landed ambiguous row not durable until read-back, the open interval dropped from construction backfill (the per-step inheritance check), the replay's per-token writes dropped (the replay-while-open test), a read-back leaving the pending set entry. Removing the closed-implies-no-unsettled *check* removes an oracle; the mechanism mutant it exists for, `Close` without its unsettled-admissions loop, is killed by it (an unanswered durable row). Putting the send's terminal check before its gate is equivalent in the schedules tested (a settled, issued, unsent token exists after a clean stop, but no schedule sends it afterwards). After the first code review: the budget expiry a no-op (its own test, with no discovery); the barrier owner-only, or over issued rows only (the successor-barred test and the schedules); the pending set skipped at registration; the guard ignored, and the drain before the guard; the replay clearing the latch without the lower bound's holds (a named test: the schedules did not reach it in 3000 seeds); the send unchecked; a close without the replay (the closed-implies-no-latch check); acknowledgement copying the predecessor's own uncertainties only (not equivalent: a schedule found a fifth fleet's owner unheld for the first fleet's uncertainty, because the chain's copies are how a closed interval reaches a fleet built after its predecessor acknowledged); the answer ignoring the refusal; a read-back while the writer runs (the named test; the later commit would land on a cancelled registration, which the terminal check reports). Equivalent under this model: the unacknowledged-ancestry walk not stopping at an acknowledged step (over-reach writes the same holds). Not equivalent, though once claimed so here: the registry proof in the bound (Astra's schedule: a build, a clean look a day later, a discovery a day after that, with no admission: only the proof reaches the first day; and Opus's: a witness change known only to be after the proof; both are named tests). First: no construction backfill (reach); the pending set not counted (the commit-after-discovery tests); no recheck after commit (the same test, and a hazard at seed 264); the replay dropping counted settlements (hazard); the replay closing at the discovery instant (liveness and the latch tests); a settlement writing no interval (hazard); recovery ignoring the fence (the fence test); helper death without transport-lost (an uncertainty open at quiescence); the lower bound ignoring the clean look (the lower-bound test; the 30000-schedule sweep alone did not find it, which is why that sequence is a named test); the lower bound ignoring the earliest unsettled admission (liveness, because the model's proof then reached too far); unrecorded synthesised for a registered worker (the guard test); recovery skipping unclosed fleets (an uncertainty open at quiescence); acknowledgement not root-first and reach ignoring successors (reach). An epoch check made redundant by the stale state is the one equivalent mutant, noted in the model. |
| +- **Mutants, in a tree copy**, each killed by a named test or a schedule. After the fourth code review: the historical term filtered to finished and abandoned (the unrecorded test), the cutoff as the running minimum (the fixed-bound test; the a1-first order kills it, the reversed order is the control), the historical term using the send instant, the historical term dropped, a later witness change ignored (the proof test). After the fifth: `>=` at the bound as `>` (the at-the-bound test), a witness change overwriting the first change's instant, the invariant check after a recovery removed (observed by the schedules that recover). After the third code review: the threshold as the later of proof and look, as the look alone, the historical term including empty terminals, the proof not refusing a moved witness, the crash duplicating a landed row, `Close` without its unsettled loop, the latch overlay reaching the fleet's owners only. After the second code review: revision 8's bound (the in-flight test), the bound without the proof (the proof test), the bound without the clean look (the lower-bound test), a landed ambiguous row not durable until read-back, the open interval dropped from construction backfill (the per-step inheritance check), the replay's per-token writes dropped (the replay-while-open test), a read-back leaving the pending set entry. Removing the closed-implies-no-unsettled *check* removes an oracle; the mechanism mutant it exists for, `Close` without its unsettled-admissions loop, is killed by it (an unanswered durable row). Putting the send's terminal check before its gate is equivalent in the schedules tested (a settled, issued, unsent token exists after a clean stop, but no schedule sends it afterwards). After the first code review: the budget expiry a no-op (its own test, with no discovery); the barrier owner-only, or over issued rows only (the successor-barred test and the schedules); the pending set skipped at registration; the guard ignored, and the drain before the guard; the replay clearing the latch without the lower bound's holds (a named test: the schedules did not reach it in 3000 seeds); the send unchecked; a close without the replay (the closed-implies-no-latch check); acknowledgement copying the predecessor's own uncertainties only (not equivalent: a schedule found a fifth fleet's owner unheld for the first fleet's uncertainty, because the chain's copies are how a closed interval reaches a fleet built after its predecessor acknowledged); the answer ignoring the refusal; a read-back while the writer runs (the named test; the later commit would land on a cancelled registration, which the terminal check reports). Equivalent under this model: the unacknowledged-ancestry walk not stopping at an acknowledged step (over-reach writes the same holds). Not equivalent, though once claimed so here: the registry proof in the bound (Astra's schedule: a build, a clean look a day later, a discovery a day after that, with no admission: only the proof reaches the first day; and Opus's: a witness change known only to be after the proof; both are named tests). First: no construction backfill (reach); the pending set not counted (the commit-after-discovery tests); no recheck after commit (the same test, and a hazard at seed 264); the replay dropping counted settlements (hazard); the replay closing at the discovery instant (liveness and the latch tests); a settlement writing no interval (hazard); recovery ignoring the fence (the fence test); helper death without transport-lost (an uncertainty open at quiescence); the lower bound ignoring the clean look (the lower-bound test; the 30000-schedule sweep alone did not find it, which is why that sequence is a named test); the lower bound ignoring the earliest unsettled admission (liveness, because the model's proof then reached too far); unrecorded synthesised for a registered worker (the guard test); recovery skipping unclosed fleets (an uncertainty open at quiescence); acknowledgement not root-first and reach ignoring successors (reach). The equivalent mutants are named where they arise (the epoch check made redundant by the stale state; the unacknowledged-ancestry walk's over-reach; the send's terminal check before its gate), and the registry proof is not one. |
| - **What the model does not cover**, by construction: the latch clearing after the commit as two instants (the replay is one step here; the rule stays with the ledger's tests), a snapshot between those instants, the recorder worker's lifetime and the socket drain as separate events, the fence as anything but helper liveness, retries of a failed terminal or replay write, the ledger's commit latency on a box's disk under the replicator's request rate (the spike's other half, which needs a box), Part C2's owner scope beyond the owner-scoped holds the model writes, and process kills inside a transaction (the model's transactions are atomic by fiat; the ledger's are SQLite's). |
| ## The fork's pathname entry points (read 2026-10-03 08:25, every call, not a sample) |
diff --git a/internal/ops/admissionmodel/model_test.go b/internal/ops/admissionmodel/model_test.go| index 1f3292b50..482c1f23d 100644 |
| --- a/internal/ops/admissionmodel/model_test.go |
| +++ b/internal/ops/admissionmodel/model_test.go |
| @@ -876,7 +876,9 @@ func TestWitnessChangeKeepsTheFirstChangeInstant(t *testing.T) { |
| // Review 4 (both): the cutoff is the bound, fixed before the rows are |
| // read, so a settlement before the bound never qualifies through another |
| -// row's earlier admission, in either row order. |
| +// row's earlier admission. The a1-first order is the one that kills the |
| +// running-cutoff mutant (a1 lowers the running minimum before a2 is |
| +// read); the reversed order is the control, where both agree. |
| func TestHistoricalCutoffIsTheFixedBound(t *testing.T) { |
| for _, reversed := range []bool{false, true} { |
| m := New(1) |
| @@ -1255,7 +1257,11 @@ func runSchedule(seed int64, steps int) (err error) { |
| for _, g := range m.Fleets { |
| m.SurvivingHelperDies(g) |
| } |
| - m.Recover() |
| + if m.Recover() { |
| + if err := m.Check(); err != nil { // right after this recovery too, before anything repairs it |
| + return fmt.Errorf("after the final recovery: %v\n%s", err, dump(m)) |
| + } |
| + } |
| } |
| for _, a := range m.Admissions { |
| switch { |
diff --git a/internal/ops/authoritybridge.go b/internal/ops/authoritybridge.go| index 5658db846..69b181708 100644 |
| --- a/internal/ops/authoritybridge.go |
| +++ b/internal/ops/authoritybridge.go |
| @@ -58,6 +58,9 @@ func (p *fleetPins) Close() { |
| // comps splits a spelled absolute path into its components beneath the |
| // pinned slash; a path not beneath it is not something the fleet can look at. |
| func (p *fleetPins) comps(path string) ([]string, bool) { |
| + if !filepath.IsAbs(path) { |
| + return nil, false // a spelling relative to nothing this inventory knows |
| + } |
| path = filepath.Clean(path) |
| base := filepath.Clean(slashPath) |
| if base != "/" { |
| @@ -115,6 +118,16 @@ func (p *fleetPins) root(comps []string) (*authority.Root, string) { |
| return r, "" |
| } |
| +// pathOf is a retained directory's resolved spelling from the pinned slash, |
| +// built from the names on its stack: for messages only, never for a look. |
| +func (p *fleetPins) pathOf(d *authority.Dir) string { |
| + var names []string |
| + for ; d != nil && d != p.slash; d = d.Parent() { |
| + names = append([]string{d.Name()}, names...) |
| + } |
| + return filepath.Join(append([]string{filepath.Clean(slashPath)}, names...)...) |
| +} |
| + |
| // linkIn descends the components from the slash with no rule but |
| // O_NOFOLLOW and names the first that is a link, or "": the predicate's |
| // "holds no further link" said before anything about ownership. |
| @@ -167,17 +180,34 @@ func (p *fleetPins) walk(d *authority.Dir, dest string) (authority.Result, error |
| // walk to its root and then the platform-link predicate on descriptors |
| // (today's rule, within the pinned roots: a link is followed when the link |
| // and its directory are the platform's, its target holds no further link |
| -// and no ".." past its leading run, the target's directories from the |
| -// root down are the platform's, and the target's parent is the |
| -// platform's; a target that leaves every pinned root is refused where |
| -// today's pathname walk would have followed it). The result is confined: a relative walk from it or beneath |
| +// and no ".." past its leading run, and the target's parent is the |
| +// platform's; for an absolute target every directory from the slash to |
| +// the parent is the platform's as well, since the pin is established by |
| +// the platform walk, while a relative target's named components between |
| +// the popped ancestor and the parent are not examined, as today; a target |
| +// that leaves every pinned root is refused where today's pathname walk |
| +// would have followed it). The result is confined: a relative walk from it or beneath |
| // it cannot leave it. The caller releases what was acquired with the |
| // returned func; the root stays with the pins. The reason, when the |
| // directory cannot be reached, is a string the inventory renders. |
| func (p *fleetPins) tenantRoot(dir string) (*authority.Dir, func(), string) { |
| + d, release, _, why := p.acquire(dir) |
| + return d, release, why |
| +} |
| + |
| +// acquire is tenantRoot, saying as well whether the spelled directory |
| +// itself was a link (a platform-directory claim refuses that, as today's |
| +// Lstat did). A link met beneath the root whose target ascends above the |
| +// root re-establishes the root higher, by the same platform walk, and the |
| +// descent starts again from it; so a sequence of platform links each |
| +// needing more ancestry is acquired as the pathname predicate accepted it. |
| +func (p *fleetPins) acquire(dir string) (*authority.Dir, func(), bool, string) { |
| comps, ok := p.comps(dir) |
| if !ok { |
| - return nil, nil, "its directory is not beneath the pinned root" |
| + if !filepath.IsAbs(dir) { |
| + return nil, nil, false, "its directory is not an absolute path" |
| + } |
| + return nil, nil, false, "its directory is not beneath the pinned root" |
| } |
| var owned []interface{ Close() error } |
| var results []*authority.Result |
| @@ -243,12 +273,16 @@ func (p *fleetPins) tenantRoot(dir string) (*authority.Dir, func(), string) { |
| if rootN < 0 { |
| rootN = 0 |
| } |
| +descend: |
| r, why := p.root(comps[:rootN]) |
| if why != "" { |
| - return nil, nil, why |
| + return nil, nil, false, why |
| } |
| cur := r.Dir |
| - for _, c := range comps[rootN:] { |
| + throughLink := false |
| + for i := rootN; i < len(comps); i++ { |
| + c := comps[i] |
| + throughLink = false |
| child, err := cur.Child(c, nil) |
| if err == nil { |
| owned = append(owned, child) |
| @@ -258,43 +292,81 @@ func (p *fleetPins) tenantRoot(dir string) (*authority.Dir, func(), string) { |
| var ee *authority.EdgeError |
| if !errors.As(err, &ee) || !errors.Is(ee.Err, authority.ErrLink) { |
| release() |
| - return nil, nil, "a directory in its path cannot be looked at" |
| + return nil, nil, false, "a directory in its path cannot be looked at" |
| } |
| + throughLink = true |
| // a link: the platform's, in the platform's directory, to a |
| // directory beneath the platform's, through no further link |
| if df := authFacts(cur.Name(), cur.Facts()); !platformUID(df.UID) || df.Mode&0o022 != 0 { |
| release() |
| - return nil, nil, "a link in its path is in a directory that is not the platform's" |
| + return nil, nil, false, "a link in its path is in a directory that is not the platform's" |
| } |
| _, lf, lerr := cur.Lookat(c) |
| if lerr != nil { |
| release() |
| - return nil, nil, "a link in its path cannot be looked at" |
| + return nil, nil, false, "a link in its path cannot be looked at" |
| } |
| if !platformUID(authFacts(c, lf).UID) { |
| release() |
| - return nil, nil, "a link in its path is not the platform's" |
| + return nil, nil, false, "a link in its path is not the platform's" |
| } |
| target, rerr := cur.Readlink(c) |
| if rerr != nil { |
| release() |
| - return nil, nil, "a link in its path cannot be read" |
| + return nil, nil, false, "a link in its path cannot be read" |
| + } |
| + // a relative target that ascends above this root: the root is |
| + // re-established that many levels up, and the descent restarts |
| + if !filepath.IsAbs(target) { |
| + n := 0 |
| + for _, part := range strings.Split(target, "/") { |
| + if part == "" || part == "." { |
| + continue |
| + } |
| + if part != ".." { |
| + break |
| + } |
| + n++ |
| + } |
| + // the pops available on the resolved stack before the root in |
| + // use; what the ascent needs beyond them comes off the root's own |
| + // spelled components (the root is reached by spelling) |
| + avail := 0 |
| + for d := cur; d != nil && d != r.Dir; d = d.Parent() { |
| + avail++ |
| + } |
| + if n > avail { |
| + need := rootN - (n - avail) |
| + if need < 0 { |
| + need = 0 |
| + } |
| + if need < rootN { // strictly higher than the root in use: the restart terminates |
| + release() |
| + owned, results = nil, nil |
| + rootN = need |
| + goto descend |
| + } |
| + } |
| } |
| // An absolute platform-link target pins its parent as a root (the |
| // platform walk from the slash requires every component to the |
| // parent to be the platform's, as today's predicate did), so the |
| // walk enters it; a relative one pops to this spelling's root. |
| if filepath.IsAbs(target) { |
| - if tc, ok := p.comps(target); ok && len(tc) > 0 { |
| + if tc, ok := p.comps(target); ok { |
| // a further link anywhere in the target is said before its |
| // directories' ownership, as today's predicate said it |
| if at := p.linkIn(tc); at != "" { |
| release() |
| - return nil, nil, "a link in its path leads through another link: " + at + " is a link" |
| + return nil, nil, false, "a link in its path leads through another link: " + at + " is a link" |
| + } |
| + parent := tc |
| + if len(parent) > 0 { |
| + parent = parent[:len(parent)-1] |
| } |
| - if _, why := p.root(tc[:len(tc)-1]); why != "" { |
| + if _, why := p.root(parent); why != "" { // a target of the slash itself pins the slash |
| release() |
| - return nil, nil, "a link in its path leads to a directory that is not under the platform's: " + why |
| + return nil, nil, false, "a link in its path leads to a directory that is not under the platform's: " + why |
| } |
| } |
| } |
| @@ -302,31 +374,98 @@ func (p *fleetPins) tenantRoot(dir string) (*authority.Dir, func(), string) { |
| switch { |
| case errors.Is(werr, authority.ErrAscentAfterName): |
| release() |
| - return nil, nil, "a link in its path leads through another link: " + target + " steps up after a named component" |
| + return nil, nil, false, "a link in its path leads through another link: " + target + " steps up after a named component" |
| case errors.Is(werr, authority.ErrAscentPastRoot), errors.Is(werr, authority.ErrOutside), errors.Is(werr, authority.ErrNotPlatform): |
| release() |
| - return nil, nil, "a link in its path leads out of its platform root: " + target |
| + return nil, nil, false, "a link in its path leads out of its platform root: " + target |
| case werr != nil: |
| release() |
| - return nil, nil, "a link in its path leads through another link: " + target + " cannot be looked at" |
| + return nil, nil, false, "a link in its path leads through another link: " + target + " cannot be looked at" |
| + case res.Kind == authority.Absent: |
| + res.Close() |
| + release() |
| + return nil, nil, false, "a link in its path leads through another link: " + target + " cannot be looked at" |
| case res.Kind == authority.Link: |
| res.Close() |
| release() |
| - return nil, nil, "a link in its path leads through another link: " + target + " is a link" |
| + return nil, nil, false, "a link in its path leads through another link: " + target + " is a link" |
| case res.Kind != authority.Directory: |
| res.Close() |
| release() |
| - return nil, nil, "a link in its path does not lead to a directory" |
| + return nil, nil, false, "a link in its path does not lead to a directory" |
| } |
| results = append(results, &res) |
| if parent := res.Dir.Parent(); parent != nil { |
| if pf := authFacts(parent.Name(), parent.Facts()); !platformUID(pf.UID) || pf.Mode&0o022 != 0 { |
| release() |
| - return nil, nil, "a link in its path leads to a directory that is not under the platform's" |
| + return nil, nil, false, "a link in its path leads to a directory that is not under the platform's: " + p.pathOf(parent) + " belongs to a user other than root or the platform's, or can be written by its group or by anyone" |
| } |
| } |
| cur = res.Dir |
| } |
| - cur.Confine() |
| - return cur, release, "" |
| + if cur != p.slash { // the slash is every walk's boundary already and is shared |
| + cur.Confine() |
| + } |
| + return cur, release, throughLink, "" |
| +} |
| + |
| +// retainedChain is an owned file's directory with its retained stack, kept |
| +// for a fleet's life (one per owned file; the design counts them in the |
| +// fleet's rendering). recheck is what every look does with it: every edge |
| +// of the stack by fstatat in its parent, then the file's name in its |
| +// directory, compared by identity with what was checked. |
| +type retainedChain struct { |
| + dir *authority.Dir |
| + name string |
| + facts authority.Facts |
| + release func() |
| +} |
| + |
| +func (c *retainedChain) recheck() error { |
| + if err := c.dir.Recheck(); err != nil { |
| + return err |
| + } |
| + k, f, err := c.dir.Lookat(c.name) |
| + if err != nil { |
| + return err |
| + } |
| + if k != authority.Regular || !f.Same(c.facts) { |
| + return &authority.EdgeError{Name: c.name, Err: authority.ErrChanged} |
| + } |
| + return nil |
| +} |
| + |
| +// depth is the number of descriptors the chain holds beneath the pinned |
| +// slash (the file's directory and every directory above it to its root's |
| +// stack top), for the fleet's rendering. |
| +func (c *retainedChain) depth() int { return c.dir.Depth() } |
| + |
| +// recheck asks every pinned root whether its ancestry still reads as it |
| +// did: a difference is a change the fleet goes stale on. |
| +func (p *fleetPins) recheck() error { |
| + for _, r := range p.roots { |
| + if changed, err := r.AncestryChanged(); err != nil || changed { |
| + if err == nil { |
| + err = authority.ErrChanged |
| + } |
| + return &authority.EdgeError{Name: r.Path, Err: err} |
| + } |
| + } |
| + return nil |
| +} |
| + |
| +// lookAt is one look at a spelled file through its directory's retained |
| +// chain: its identity, for the inventory's "two listed paths are one file" |
| +// check, with nothing resolved by pathname. |
| +func (p *fleetPins) lookAt(path string) (authority.Facts, string) { |
| + dir, release, why := p.tenantRoot(filepath.Dir(path)) |
| + if why != "" { |
| + return authority.Facts{}, "it cannot be looked at" |
| + } |
| + defer release() |
| + k, f, err := dir.Lookat(filepath.Base(path)) |
| + if err != nil || k == authority.Absent { |
| + return authority.Facts{}, "it cannot be looked at" |
| + } |
| + return f, "" |
| } |
diff --git a/internal/ops/filefacts_other.go b/internal/ops/filefacts_other.go| index d535997a8..e0bc85d78 100644 |
| --- a/internal/ops/filefacts_other.go |
| +++ b/internal/ops/filefacts_other.go |
| @@ -11,6 +11,7 @@ type fileFacts struct { |
| Dev, Ino uint64 |
| Nlink uint64 |
| UID uint32 |
| + keep *retainedChain // the file's retained chain while the inventory is held; nil in a report |
| } |
| var factsOf = func(fs.FileInfo) (fileFacts, bool) { return fileFacts{}, false } |
diff --git a/internal/ops/filefacts_unix.go b/internal/ops/filefacts_unix.go| index 78360f3ed..f8669abf1 100644 |
| --- a/internal/ops/filefacts_unix.go |
| +++ b/internal/ops/filefacts_unix.go |
| @@ -16,6 +16,7 @@ type fileFacts struct { |
| Dev, Ino uint64 |
| Nlink uint64 |
| UID uint32 |
| + keep *retainedChain // the file's retained chain while the inventory is held; nil in a report |
| } |
| // factsOf is a variable so a test can give a chain of directories owners |
diff --git a/internal/ops/litestream_metered.go b/internal/ops/litestream_metered.go| index 96e287677..f84f0a9e7 100644 |
| --- a/internal/ops/litestream_metered.go |
| +++ b/internal/ops/litestream_metered.go |
| @@ -133,6 +133,7 @@ type MeteredFleet struct { |
| registryPath string |
| dataDir string |
| facts map[string]fileFacts // by owned path: the file each was when it was checked |
| + held FleetInventory // the inventory's pins and the owned files' chains, retained for the fleet's life |
| inventory string // the inventory this fleet was built over, rendered |
| recorders []lease.RequestRecorder |
| replaces *MeteredFleet |
| @@ -638,6 +639,14 @@ func (f *MeteredFleet) lookNow(final bool) { |
| if f.lookDelay > 0 { |
| time.Sleep(f.lookDelay) |
| } |
| + // What this fleet retains first: every pinned root's ancestry and every |
| + // owned file's chain and name, by descriptor (a directory moved, |
| + // replaced, re-owned or re-moded anywhere on a chain is a changed |
| + // edge, whether or not the pathname still leads somewhere). |
| + if err := f.held.recheck(); err != nil { |
| + f.goStaleFrom(f.lastCleanLook(), time.Now()) |
| + return |
| + } |
| inv, err := fleetInventory(f.routes, f.registryPath, f.dataDir) |
| if err != nil { |
| f.drift.Store(true) |
| @@ -646,6 +655,7 @@ func (f *MeteredFleet) lookNow(final bool) { |
| } |
| return |
| } |
| + defer inv.Close() // the retained chains were rechecked above and are kept; the fresh inventory's go |
| for path, was := range f.facts { |
| now, still := inv.facts[path] |
| if still && now.Dev == was.Dev && now.Ino == was.Ino { |
| @@ -768,6 +778,7 @@ func (f *MeteredFleet) closeNow() error { |
| f.mu.Lock() |
| defer f.mu.Unlock() |
| if f.obs == nil { |
| + f.held.Close() |
| f.closed.Store(true) |
| return first |
| } |
| @@ -797,6 +808,7 @@ func (f *MeteredFleet) closeNow() error { |
| // A fleet closed without ever opening still passes on what the fleet |
| // it replaced held. |
| f.inheritHold() |
| + f.held.Close() |
| f.closed.Store(true) |
| err := f.obs.Close() |
| f.obs = nil |
| @@ -875,9 +887,11 @@ func MeteredFleetStore(o MeteredFleetOptions) (fleet *MeteredFleet, err error) { |
| }() |
| inv, err := fleetInventory(built.routes, built.registryPath, built.dataDir) |
| if err != nil { |
| + inv.Close() |
| return nil, err |
| } |
| if len(inv.Problems) > 0 { |
| + inv.Close() |
| return nil, fmt.Errorf("%w: places a database could be that cannot be vouched for: %s", ErrUnmeterable, quotedList(inv.Problems)) |
| } |
| // A plain path is accepted as cleaned; a link entry (link "path" -> |
| @@ -904,15 +918,19 @@ func MeteredFleetStore(o MeteredFleetOptions) (fleet *MeteredFleet, err error) { |
| } |
| } |
| if len(unseen) > 0 { |
| + inv.Close() |
| return nil, fmt.Errorf("%w: files that look like databases, not replicated and not named as accepted: %s", ErrUnmeterable, quotedList(unseen)) |
| } |
| if len(inv.Unowned) > 0 { |
| + inv.Close() |
| return nil, fmt.Errorf("%w: %s", ErrUnmeterable, quotedList(inv.Unowned)) |
| } |
| if len(inv.Owned) == 0 { |
| + inv.Close() |
| return nil, errors.New("ops: no databases to replicate") |
| } |
| if err := built.admit(context.Background()); err != nil { |
| + inv.Close() |
| return nil, fmt.Errorf("%w (while ownership was being read)", err) |
| } |
| // A replica lineage and its shadow state belong to an owner and a |
| @@ -928,6 +946,7 @@ func MeteredFleetStore(o MeteredFleetOptions) (fleet *MeteredFleet, err error) { |
| } |
| metas, err := prepareShadowState(built.dataDir, keys) |
| if err != nil { |
| + inv.Close() |
| return nil, err |
| } |
| prefix := strings.Trim(o.Prefix, "/") |
| @@ -935,6 +954,7 @@ func MeteredFleetStore(o MeteredFleetOptions) (fleet *MeteredFleet, err error) { |
| for _, owned := range inv.Owned { |
| rec := o.Meter(owned) |
| if rec == nil { |
| + inv.Close() |
| return nil, fmt.Errorf("%w: no recorder for %s (%s/%s)", ErrUnmeterable, strconv.Quote(owned.Path), owned.Account, owned.App) |
| } |
| built.recorders = append(built.recorders, rec) |
| @@ -947,13 +967,14 @@ func MeteredFleetStore(o MeteredFleetOptions) (fleet *MeteredFleet, err error) { |
| } |
| store, err := lease.NewMeteredStore(o.Bucket, o.Region, dbs) |
| if err != nil { |
| + inv.Close() |
| return nil, err |
| } |
| // The fleet's recovery window, not litestream's one-day default: the |
| // external replicator this replaces keeps a week of snapshots. |
| store.SnapshotInterval, store.SnapshotRetention = SnapshotInterval, SnapshotRetention |
| built.store, built.Owned, built.Skipped = store, inv.Owned, inv.Skipped |
| - built.facts, built.inventory = inv.facts, inv.render() |
| + built.facts, built.held, built.inventory = inv.facts, inv, inv.render() |
| // What the fleet this one replaces held is passed on at Open, once the |
| // old fleet has closed and said its last. |
| return built, nil |
diff --git a/internal/ops/litestream_metered_test.go b/internal/ops/litestream_metered_test.go| index 54fe41681..aaf781f10 100644 |
| --- a/internal/ops/litestream_metered_test.go |
| +++ b/internal/ops/litestream_metered_test.go |
| @@ -1983,3 +1983,73 @@ func TestRenderQuotesEveryEntry(t *testing.T) { |
| } |
| } |
| } |
| + |
| +// A fleet rechecks what it retains at every look, by descriptor: a mode |
| +// change on a directory of an owned file's chain, which no rebuilt |
| +// inventory looks at (the tenant's own directory inside its tree), is a |
| +// changed edge, and the fleet goes stale at the next look; a clean look |
| +// before it keeps exactly the retained descriptors (the fresh inventory |
| +// it compares against is released); its descriptors are released at |
| +// Close. |
| +func TestMeteredFleetStalesWhenAChainEdgeChanges(t *testing.T) { |
| + newFakeBucket(t) |
| + dir := platformDir(t) |
| + reg := openRegistry(t, filepath.Join(dir, "registry.db")) |
| + state := filepath.Join(platformDir(t), "unit-a") |
| + db := writeFile(t, filepath.Join(state, "data", "main.db")) |
| + upsert(t, reg, registry.Route{Host: "a.example.com", Account: "bac", App: "one", DBPath: state, Backing: registry.BackingUnit, Version: "v1"}) |
| + under := filepath.Dir(dir) // the test's temp root: every directory descriptor the fleet retains is beneath it |
| + before := openDirFDsUnder(t, under) |
| + fleet, err := MeteredFleetStore(fleetOptions(reg, dir)) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if held := fleet.held.Descriptors(); held == 0 || openDirFDsUnder(t, under) != before+held { |
| + t.Fatalf("the fleet retains %d descriptors by its count, %d by /proc (before %d)", held, openDirFDsUnder(t, under)-before, before) |
| + } |
| + if err := send(t, fleet, db, 2); err != nil { |
| + t.Fatalf("a write before any change: %v", err) |
| + } |
| + held := fleet.held.Descriptors() |
| + fleet.lookAgain() // clean: the fresh inventory it compared against is released |
| + if n := openDirFDsUnder(t, under); n != before+held { |
| + t.Fatalf("after a clean look: %d directory descriptors, want the retained %d", n-before, held) |
| + } |
| + if err := send(t, fleet, db, 3); err != nil { |
| + t.Fatalf("a write after a clean look: %v", err) |
| + } |
| + if err := os.Chmod(filepath.Join(state, "data"), 0o700); err != nil { // a mode change on the chain: only the retained edge sees it |
| + t.Fatal(err) |
| + } |
| + fleet.lookAgain() |
| + if err := send(t, fleet, db, 4); !errors.Is(err, ErrOwnershipStale) { |
| + t.Fatalf("a write after a chain edge changed: %v, want stale", err) |
| + } |
| + if err := fleet.Close(); err != nil { |
| + t.Fatal(err) |
| + } |
| + if n := openDirFDsUnder(t, under); n != before || fleet.held.Descriptors() != 0 { |
| + t.Fatalf("after Close: %d directory descriptors under the test root, %d before the fleet; the fleet counts %d", n, before, fleet.held.Descriptors()) |
| + } |
| +} |
| + |
| +// openDirFDsUnder counts this process's open directory descriptors whose |
| +// target lies beneath prefix. |
| +func openDirFDsUnder(t *testing.T, prefix string) int { |
| + t.Helper() |
| + ents, err := os.ReadDir("/proc/self/fd") |
| + if err != nil { |
| + t.Skip("no /proc/self/fd here") |
| + } |
| + n := 0 |
| + for _, e := range ents { |
| + target, err := os.Readlink(filepath.Join("/proc/self/fd", e.Name())) |
| + if err != nil || !strings.HasPrefix(target, prefix+string(filepath.Separator)) && target != prefix { |
| + continue |
| + } |
| + if fi, err := os.Stat(target); err == nil && fi.IsDir() { |
| + n++ |
| + } |
| + } |
| + return n |
| +} |
diff --git a/internal/ops/litestream_owners.go b/internal/ops/litestream_owners.go| index 9f2711569..f258ad649 100644 |
| --- a/internal/ops/litestream_owners.go |
| +++ b/internal/ops/litestream_owners.go |
| @@ -42,6 +42,12 @@ const ( |
| // FleetInventory is one strict reading of what the box replicates and who |
| // pays for each database's replication writes. |
| type FleetInventory struct { |
| + // pins are the pinned slash and the platform roots this inventory was |
| + // walked from; the chains of the owned files (facts[p].keep) are |
| + // retained beneath them until Close. A caller that keeps the inventory |
| + // (the fleet) rechecks both at every look; one that does not closes it. |
| + pins *fleetPins |
| + |
| // Owned is every database with a single owner, sorted by path. |
| Owned []OwnedDB |
| // Unowned names the databases no single account can be shown to own, |
| @@ -67,7 +73,70 @@ func FleetDBOwners(reg *registry.Registry, dataDir string) (FleetInventory, erro |
| if err != nil { |
| return FleetInventory{}, err |
| } |
| - return fleetInventory(routes, reg.Path(), dataDir) |
| + inv, err := fleetInventory(routes, reg.Path(), dataDir) |
| + inv.Close() // a report keeps no descriptors; the facts stay |
| + return inv, err |
| +} |
| + |
| +// Close releases every retained chain and the pins. The facts stay, their |
| +// chains gone; a second Close is harmless. |
| +func (inv *FleetInventory) Close() { |
| + for p, f := range inv.facts { |
| + if f.keep != nil { |
| + f.keep.release() |
| + f.keep = nil |
| + inv.facts[p] = f |
| + } |
| + } |
| + if inv.pins != nil { |
| + inv.pins.Close() |
| + inv.pins = nil |
| + } |
| +} |
| + |
| +// Descriptors is the number of directory descriptors the inventory holds: |
| +// the pinned slash, every root's stack and every owned file's chain |
| +// (the design: the count is part of the fleet's rendering and a test's |
| +// assertion). |
| +func (inv *FleetInventory) Descriptors() int { |
| + if inv.pins == nil { |
| + return 0 |
| + } |
| + seen := map[*authority.Dir]bool{inv.pins.slash: true} |
| + walk := func(d *authority.Dir) { |
| + for ; d != nil; d = d.Parent() { |
| + seen[d] = true |
| + } |
| + } |
| + for _, r := range inv.pins.roots { |
| + walk(r.Dir) |
| + } |
| + for _, f := range inv.facts { |
| + if f.keep != nil { |
| + walk(f.keep.dir) |
| + } |
| + } |
| + return len(seen) |
| +} |
| + |
| +// recheck is the fleet's look at what it retains: every pinned root's |
| +// ancestry and every owned file's chain and name. The first difference is |
| +// returned with its edge. |
| +func (inv *FleetInventory) recheck() error { |
| + if inv.pins == nil { |
| + return nil |
| + } |
| + if err := inv.pins.recheck(); err != nil { |
| + return err |
| + } |
| + for _, f := range inv.facts { |
| + if f.keep != nil { |
| + if err := f.keep.recheck(); err != nil { |
| + return err |
| + } |
| + } |
| + } |
| + return nil |
| } |
| // fleetInventory is FleetDBs with an owner for each database, for the |
| @@ -95,8 +164,15 @@ func FleetDBOwners(reg *registry.Registry, dataDir string) (FleetInventory, erro |
| // that does not establish. |
| func fleetInventory(routes []registry.Route, registryPath, dataDir string) (FleetInventory, error) { |
| inv := FleetInventory{facts: map[string]fileFacts{}} |
| - dbs, skipped, problems, err := fleetDBsFromRoutes(routes, registryPath, dataDir, strictTreeDBs) |
| + pins, perr := newFleetPins() |
| + if perr != nil { |
| + return inv, perr |
| + } |
| + inv.pins = pins |
| + tree := func(root string) (dbs, skipped, problems []string) { return strictTreeDBsOn(pins, root) } |
| + dbs, skipped, problems, err := fleetDBsFromRoutes(routes, registryPath, dataDir, tree) |
| if err != nil { |
| + inv.Close() |
| return inv, err |
| } |
| inv.Skipped, inv.Problems = skipped, problems |
| @@ -116,23 +192,23 @@ func fleetInventory(routes []registry.Route, registryPath, dataDir string) (Flee |
| // owners billed for it. |
| type listed struct { |
| path string |
| - fi os.FileInfo |
| + id authority.Facts |
| } |
| var files []listed |
| for _, p := range dbs { |
| p = filepath.Clean(p) |
| - fi, serr := os.Stat(p) |
| - if serr != nil { |
| - unowned(p, "it cannot be looked at") |
| + id, why := pins.lookAt(p) |
| + if why != "" { |
| + unowned(p, why) |
| continue |
| } |
| - files = append(files, listed{p, fi}) |
| + files = append(files, listed{p, id}) |
| } |
| for i, f := range files { |
| p := f.path |
| twice := false |
| for j, other := range files { |
| - if i != j && os.SameFile(f.fi, other.fi) { |
| + if i != j && f.id.Same(other.id) { |
| twice = true |
| break |
| } |
| @@ -149,7 +225,7 @@ func fleetInventory(routes []registry.Route, registryPath, dataDir string) (Flee |
| unowned(p, "the platform's own database is also claimed by the route "+host) |
| continue |
| } |
| - facts, why := checkOwnedFile(filepath.Dir(p), p, platformDirectory) |
| + facts, why := checkOwnedFileOn(pins, filepath.Dir(p), p, platformDirectory) |
| if why != "" { |
| unowned(p, why) |
| continue |
| @@ -190,20 +266,28 @@ func fleetInventory(routes []registry.Route, registryPath, dataDir string) (Flee |
| var facts fileFacts |
| refused := false |
| for i, c := range claimChecks(claims, p, dataDir) { |
| - f, why := checkOwnedFile(c.root, p, c.kind) |
| + f, why := checkOwnedFileOn(pins, c.root, p, c.kind) |
| if why == "" && i > 0 && (f.Dev != facts.Dev || f.Ino != facts.Ino) { |
| why = "it changed under the look (its claims' checks did not all see one file)" |
| } |
| if why != "" { |
| unowned(p, why) |
| refused = true |
| + if f.keep != nil { |
| + f.keep.release() |
| + } |
| break |
| } |
| if i == 0 { |
| - facts = f |
| + facts = f // the first check's chain is the one kept |
| + } else if f.keep != nil { |
| + f.keep.release() |
| } |
| } |
| if refused { |
| + if facts.keep != nil { |
| + facts.keep.release() |
| + } |
| continue |
| } |
| inv.Owned = append(inv.Owned, OwnedDB{Path: p, Account: owner.Account, App: owner.App, Host: owner.Host, Platform: owner.Account == release.OpsAccount}) |
| @@ -224,6 +308,9 @@ func fleetInventory(routes []registry.Route, registryPath, dataDir string) (Flee |
| f := inv.facts[o.Path] |
| if len(byInode[[2]uint64{f.Dev, f.Ino}]) > 1 { |
| unowned(o.Path, "another owned path became this same file under the look") |
| + if f.keep != nil { |
| + f.keep.release() |
| + } |
| delete(inv.facts, o.Path) |
| continue |
| } |
| @@ -393,16 +480,28 @@ const ( |
| // RESOLVE_BENEATH and RESOLVE_NO_SYMLINKS) and to keep the opened |
| // identity, which is a change to litestream and is not made. |
| func checkOwnedFile(root, path string, kind claimKind) (fileFacts, string) { |
| - root, path = filepath.Clean(root), filepath.Clean(path) |
| - rel, err := filepath.Rel(root, path) |
| - if err != nil || rel == "." || rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) { |
| - return fileFacts{}, "it is not beneath the directory it is claimed from" |
| - } |
| pins, perr := newFleetPins() |
| if perr != nil { |
| return fileFacts{}, "the directory it is claimed from cannot be looked at" |
| } |
| defer pins.Close() |
| + f, why := checkOwnedFileOn(pins, root, path, kind) |
| + if f.keep != nil { |
| + f.keep.release() |
| + f.keep = nil |
| + } |
| + return f, why |
| +} |
| + |
| +// checkOwnedFileOn is checkOwnedFile against a fleet's pins; the facts it |
| +// returns on success keep the file's retained chain (keep), which the |
| +// caller holds for the fleet's life and releases. |
| +func checkOwnedFileOn(pins *fleetPins, root, path string, kind claimKind) (fileFacts, string) { |
| + root, path = filepath.Clean(root), filepath.Clean(path) |
| + rel, err := filepath.Rel(root, path) |
| + if err != nil || rel == "." || rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) { |
| + return fileFacts{}, "it is not beneath the directory it is claimed from" |
| + } |
| // The spelled directory is acquired through the platform walk to its |
| // root and the platform-link predicate on descriptors (systemd's |
| // DynamicUser shape, /var/lib/carlos-app/<host> -> ../private/..., is |
| @@ -412,14 +511,25 @@ func checkOwnedFile(root, path string, kind claimKind) (fileFacts, string) { |
| // refused). The result is the resolved directory with its retained |
| // chain up to the platform root, and nothing below is resolved by |
| // pathname again. |
| - dir, release, why := pins.tenantRoot(root) |
| + dir, release, throughLink, why := pins.acquire(root) |
| if why != "" { |
| - if strings.HasPrefix(why, "a link in its path") { |
| + if strings.HasPrefix(why, "a link in its path") || why == "its directory is not an absolute path" { |
| return fileFacts{}, why |
| } |
| return fileFacts{}, "the directory it is claimed from cannot be looked at" |
| } |
| - defer release() |
| + if kind == platformDirectory && throughLink { |
| + // the platform's own directory is named directly, never through a |
| + // link, as today's Lstat refused |
| + release() |
| + return fileFacts{}, fmt.Sprintf("%s is not a directory that can be looked at", root) |
| + } |
| + kept := false |
| + defer func() { |
| + if !kept { |
| + release() |
| + } |
| + }() |
| facts := func(d *authority.Dir) authority.Facts { return authFacts(d.Name(), d.Facts()) } |
| // platformOwnedDir is platformOwned on a retained directory. |
| platformOwnedDir := func(d *authority.Dir, spelled string) string { |
| @@ -491,11 +601,11 @@ func checkOwnedFile(root, path string, kind claimKind) (fileFacts, string) { |
| break |
| } |
| if pf.Dev != rootFacts.Dev { |
| - return fileFacts{}, "its path crosses onto another filesystem (" + parent.Name() + ")" |
| + return fileFacts{}, "its path crosses onto another filesystem (" + pins.pathOf(parent) + ")" |
| } |
| top = parent |
| } |
| - if why := platformOwnedDir(top.Parent(), "the directory above the top of its chain ("+top.Parent().Name()+")"); why != "" { |
| + if why := platformOwnedDir(top.Parent(), pins.pathOf(top.Parent())); why != "" { |
| return fileFacts{}, why |
| } |
| rootFacts = facts(top) |
| @@ -507,6 +617,9 @@ func checkOwnedFile(root, path string, kind claimKind) (fileFacts, string) { |
| cur := dir |
| var opened []*authority.Dir |
| defer func() { |
| + if kept { |
| + return |
| + } |
| for i := len(opened) - 1; i >= 0; i-- { |
| opened[i].Close() |
| } |
| @@ -590,7 +703,15 @@ func checkOwnedFile(root, path string, kind claimKind) (fileFacts, string) { |
| return fileFacts{}, "its " + suffix + " file has more than one name (a hard link)" |
| } |
| } |
| - return fileFacts{Dev: last.Dev, Ino: last.Ino, Nlink: last.Nlink, UID: last.UID}, "" |
| + kept = true |
| + chainDir, chainOpened, chainRelease := cur, opened, release |
| + keep := &retainedChain{dir: chainDir, name: name, facts: lf, release: func() { |
| + for i := len(chainOpened) - 1; i >= 0; i-- { |
| + chainOpened[i].Close() |
| + } |
| + chainRelease() |
| + }} |
| + return fileFacts{Dev: last.Dev, Ino: last.Ino, Nlink: last.Nlink, UID: last.UID, keep: keep}, "" |
| } |
| // platformOwned says why dir is not the platform's, or nothing: it must be |
| @@ -614,10 +735,11 @@ func platformOwned(dir string) string { |
| return "" |
| } |
| -// treeAfterList is a test seam: called after each directory's listing in |
| -// strictTreeDBs with the retained directory and its spelled path, so a |
| -// test can change the directory between the listing and what follows. |
| -var treeAfterList func(d *authority.Dir, dir string) |
| +// treeAfterList and treeBeforeList are test seams: called after (before) |
| +// each directory's listing in strictTreeDBs with the retained directory |
| +// and its spelled path, so a test can change the directory between its |
| +// acquisition and its listing, or between the listing and what follows. |
| +var treeAfterList, treeBeforeList func(d *authority.Dir, dir string) |
| // strictTreeMaxDepth bounds strictTreeDBs. Past treeDBWalkDepth a database |
| // is reported and not replicated; past this the walk says it cannot vouch |
| @@ -667,11 +789,7 @@ const strictTreeMaxDepth = 32 |
| // problems are never compared with the Accept list: before this is |
| // wired, the owner's decision on link shapes stands beside decisions |
| // 10 to 13, and a census of the link classes on each box's live trees |
| -// is part of the pre-wiring check. The links tenant software makes in numbers (a venv's, a |
| -// node_modules' .bin) are all named this way: whether one tenant's |
| -// unaccepted links should refuse the whole box's next build, as they |
| -// do today through Accept, or that tenant's databases alone, is to be |
| -// decided before this is wired. |
| +// is part of the pre-wiring check. A link re-pointed later reads as a change as described. |
| // |
| // What it passes over on purpose: an entry that vanishes between being |
| // listed and being looked at, or that is a link when listed and something |
| @@ -692,6 +810,11 @@ func strictTreeDBs(root string) (dbs, skipped, problems []string) { |
| return nil, nil, []string{root + ": cannot be read"} |
| } |
| defer pins.Close() |
| + return strictTreeDBsOn(pins, root) |
| +} |
| + |
| +// strictTreeDBsOn is strictTreeDBs against a fleet's pins. |
| +func strictTreeDBsOn(pins *fleetPins, root string) (dbs, skipped, problems []string) { |
| top, release, why := pins.tenantRoot(root) |
| if why != "" { |
| return nil, nil, []string{root + ": cannot be read (" + why + ")"} |
| @@ -699,6 +822,9 @@ func strictTreeDBs(root string) (dbs, skipped, problems []string) { |
| defer release() |
| var walk func(d *authority.Dir, dir string, depth int, hidden bool) |
| walk = func(d *authority.Dir, dir string, depth int, hidden bool) { |
| + if treeBeforeList != nil { |
| + treeBeforeList(d, dir) |
| + } |
| entries, err := d.List() |
| if err != nil { |
| var ee *authority.EdgeError |
| @@ -735,7 +861,16 @@ func strictTreeDBs(root string) (dbs, skipped, problems []string) { |
| problems = append(problems, p+": a link that cannot be read") |
| continue |
| } |
| - res, werr := pins.walk(d, dest) |
| + // an absolute target spelled through the tree's own root (a |
| + // DynamicUser app spells its STATE_DIRECTORY, which is the |
| + // platform link's path) is walked relative from the confined |
| + // root: the predicate vetted that link when the root was |
| + // acquired, and Within still judges the result |
| + walkFrom, walkDest := d, dest |
| + if spelled := filepath.Clean(root); filepath.IsAbs(dest) && (dest == spelled || strings.HasPrefix(dest, spelled+"/")) { |
| + walkFrom, walkDest = top, "./"+strings.TrimPrefix(strings.TrimPrefix(dest, spelled), "/") |
| + } |
| + res, werr := pins.walk(walkFrom, walkDest) |
| quoted := p + " -> " + strconv.Quote(dest) |
| switch { |
| case sqliteName(name): |
| @@ -774,8 +909,12 @@ func strictTreeDBs(root string) (dbs, skipped, problems []string) { |
| // looked at again once, and the replacement walked if |
| // it is a directory; replaced again is a problem |
| k2, f2, lerr := d.Lookat(name) |
| - if lerr != nil || k2 != authority.Directory { |
| - continue // something else now: churn, as a vanished entry is |
| + if lerr != nil { |
| + problems = append(problems, p+": cannot be looked at") // a failed look is not a vanishing |
| + continue |
| + } |
| + if k2 != authority.Directory { |
| + continue // a file or a link now, or gone: churn, as a vanished entry is; the next look classifies it |
| } |
| child, cerr = d.Child(name, &f2) |
| if cerr != nil { |
diff --git a/internal/ops/litestream_owners_unix_test.go b/internal/ops/litestream_owners_unix_test.go| index 22f03889f..8ad9cf4d8 100644 |
| --- a/internal/ops/litestream_owners_unix_test.go |
| +++ b/internal/ops/litestream_owners_unix_test.go |
| @@ -608,7 +608,9 @@ func TestStrictTreeNamesEveryLinkClass(t *testing.T) { |
| wdir := filepath.Join(filepath.Dir(plat), "w", "x") |
| mkdirs(t, wdir) |
| writeFile(t, filepath.Join(wdir, "w.db")) |
| - os.Chmod(filepath.Dir(wdir), 0o775) |
| + if err := os.Chmod(filepath.Dir(wdir), 0o775); err != nil { |
| + t.Fatal(err) |
| + } |
| links := map[string]string{ |
| "to-other-dir": "../other", |
| "abs-other": other, |
| @@ -777,6 +779,145 @@ func TestStrictTreeThroughAPlatformLink(t *testing.T) { |
| if _, _, problems := strictTreeDBs(out); len(problems) != 1 || !named(problems, out, "leads out of its platform root") { |
| t.Fatalf("a target above every root: %v", problems) |
| } |
| + // an absolute link inside the tree spelled through the tree's own |
| + // root (a DynamicUser app spells its STATE_DIRECTORY, the platform |
| + // link's path): walked relative from the confined root, a file in the |
| + // tree, skipped with its target |
| + withOwners(t, map[string]uint32{"carlos-app": 0, "private": 0, "apps": 0, "keymail.dev": tenant, "main.db": tenant, "link:keymail.dev": 0}) |
| + writeFile(t, filepath.Join(private, "releases", "5.sqlite")) |
| + if err := os.Symlink(filepath.Join(link, "releases", "5.sqlite"), filepath.Join(private, "current")); err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, skipped, problems := strictTreeDBs(link); len(problems) != 0 || !contains(skipped, linkEntry(filepath.Join(link, "current"), filepath.Join(link, "releases", "5.sqlite"))) { |
| + t.Fatalf("an absolute link through the tree's own spelled root: skipped %v problems %v", skipped, problems) |
| + } |
| + if err := os.Remove(filepath.Join(private, "current")); err != nil { // spelled through this root only; the cases below spell the tree otherwise |
| + t.Fatal(err) |
| + } |
| + // two platform links in one spelling, the second ascending higher |
| + // than the first's root: the root is re-established and the tree walked |
| + srv := filepath.Join(platformDir(t), "srv") |
| + mkdirs(t, filepath.Join(srv, "public"), filepath.Join(srv, "state"), filepath.Join(filepath.Dir(srv), "opt", "state", "alice")) |
| + writeFile(t, filepath.Join(filepath.Dir(srv), "opt", "state", "alice", "a.db")) |
| + if err := os.Symlink("../state", filepath.Join(srv, "public", "start")); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := os.Symlink("../../opt/state", filepath.Join(srv, "state", "b")); err != nil { |
| + t.Fatal(err) |
| + } |
| + withOwners(t, map[string]uint32{"srv": 0, "public": 0, "state": 0, "opt": 0, "alice": tenant, "a.db": tenant, "link:start": 0, "link:b": 0}) |
| + composed := filepath.Join(srv, "public", "start", "b", "alice") |
| + if dbs, _, problems := strictTreeDBs(composed); len(problems) != 0 || !contains(dbs, filepath.Join(composed, "a.db")) { |
| + t.Fatalf("two platform links, the second ascending higher: dbs %v problems %v", dbs, problems) |
| + } |
| + // a platform link whose absolute target is the slash itself pins the |
| + // slash, and a tree spelled through it is walked |
| + viaSlash := filepath.Join(public, "viaslash") |
| + if err := os.Symlink(filepath.Dir(top), viaSlash); err != nil { |
| + t.Fatal(err) |
| + } |
| + relPrivate, err := filepath.Rel(filepath.Dir(top), private) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + withOwners(t, map[string]uint32{"carlos-app": 0, "private": 0, "apps": 0, "keymail.dev": tenant, "main.db": tenant, "link:viaslash": 0}) |
| + if dbs, _, problems := strictTreeDBs(filepath.Join(viaSlash, relPrivate)); len(problems) != 0 || !contains(dbs, filepath.Join(viaSlash, relPrivate, "main.db")) { |
| + t.Fatalf("a platform link to the slash: dbs %v problems %v", dbs, problems) |
| + } |
| + // a platform-directory claim through a link at the claimed directory |
| + // is refused, as today's Lstat refused it; a relative spelling too |
| + if _, why := checkOwnedFile(link, filepath.Join(link, "main.db"), platformDirectory); !strings.Contains(why, "is not a directory that can be looked at") { |
| + t.Fatalf("a platform directory claimed through a link: %q", why) |
| + } |
| + if _, why := checkOwnedFile("srv/state", "srv/state/x.db", tenantFile); why != "its directory is not an absolute path" { |
| + t.Fatalf("a relative claim: %q", why) |
| + } |
| +} |
| + |
| +// The directory that was listed is the directory inspected: when the |
| +// listed directory is renamed away and another put at its spelling |
| +// between the listing and the inspection, the headers read and the links |
| +// classified are the retained directory's, and its Recheck reports the |
| +// changed edge. A directory removed between its acquisition and its |
| +// listing is churn. A directory that cannot be searched is a problem as |
| +// a whole. |
| +func TestStrictTreeInspectsTheDirectoryItListed(t *testing.T) { |
| + root := platformDir(t) |
| + mkdirs(t, filepath.Join(root, "sub")) |
| + if err := os.WriteFile(filepath.Join(root, "sub", "hdr.data"), []byte("SQLite format 3\x00 and more bytes after it"), 0o644); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := os.Symlink("hdr.data", filepath.Join(root, "sub", "l")); err != nil { |
| + t.Fatal(err) |
| + } |
| + var recheckErr, setupErr error |
| + done := false |
| + treeAfterList = func(d *authority.Dir, dir string) { |
| + if dir != filepath.Join(root, "sub") || done { |
| + return |
| + } |
| + done = true |
| + if err := os.Rename(filepath.Join(root, "sub"), filepath.Join(root, "sub.real")); err != nil { |
| + setupErr = err |
| + return |
| + } |
| + mkdirs(t, filepath.Join(root, "sub")) |
| + // the replacement: the same names, other content |
| + if err := os.WriteFile(filepath.Join(root, "sub", "hdr.data"), []byte("not a database, long enough to read"), 0o644); err != nil { |
| + setupErr = err |
| + return |
| + } |
| + if err := os.Symlink("../nothing", filepath.Join(root, "sub", "l")); err != nil { |
| + setupErr = err |
| + return |
| + } |
| + recheckErr = d.Recheck() |
| + } |
| + t.Cleanup(func() { treeAfterList = nil }) |
| + _, skipped, problems := strictTreeDBs(root) |
| + if setupErr != nil { |
| + t.Fatal(setupErr) |
| + } |
| + if !contains(skipped, filepath.Join(root, "sub", "hdr.data")) { |
| + t.Fatalf("the header read was not the listed directory's: skipped %v problems %v", skipped, problems) |
| + } |
| + if !contains(skipped, linkEntry(filepath.Join(root, "sub", "l"), "hdr.data")) { |
| + t.Fatalf("the link classified was not the listed directory's: skipped %v problems %v", skipped, problems) |
| + } |
| + if recheckErr == nil { |
| + t.Fatal("the listed directory's edge was not reported changed") |
| + } |
| + // removed between acquisition and listing: churn |
| + treeBeforeList = func(d *authority.Dir, dir string) { |
| + if dir == filepath.Join(root, "gone") { |
| + os.Remove(dir) |
| + } |
| + } |
| + t.Cleanup(func() { treeBeforeList = nil }) |
| + mkdirs(t, filepath.Join(root, "gone")) |
| + if _, _, problems := strictTreeDBs(root); containsPrefix(problems, filepath.Join(root, "gone")) { |
| + t.Fatalf("a directory removed before its listing: %v", problems) |
| + } |
| + treeBeforeList = nil |
| + // a directory that can be read but not searched (0444): its fresh |
| + // listing description cannot be opened without search permission, so |
| + // the directory as a whole is "cannot be read" (nothing in it was |
| + // looked at); a per-entry failure with search permission present |
| + // (Entry.Err: EIO, an overflow) cannot be produced here, and that |
| + // branch is untested by this fixture |
| + if os.Geteuid() == 0 { |
| + return |
| + } |
| + mkdirs(t, filepath.Join(root, "sealed")) |
| + writeFile(t, filepath.Join(root, "sealed", "x.db")) |
| + if err := os.Chmod(filepath.Join(root, "sealed"), 0o444); err != nil { |
| + t.Fatal(err) |
| + } |
| + t.Cleanup(func() { os.Chmod(filepath.Join(root, "sealed"), 0o755) }) |
| + dbs, _, problems := strictTreeDBs(root) |
| + if containsPrefix(dbs, filepath.Join(root, "sealed")) || !named(problems, filepath.Join(root, "sealed"), "cannot be read") { |
| + t.Fatalf("a directory without search permission: dbs %v problems %v", dbs, problems) |
| + } |
| } |
| // Between the listing and what follows: a subdirectory swapped is looked at |
| @@ -792,23 +933,32 @@ func TestStrictTreeBetweenListingAndLook(t *testing.T) { |
| t.Fatal(err) |
| } |
| swapped := false |
| - var confineErr error |
| - treeAfterList = func(d *authority.Dir, dir string) { |
| + var confineErr, setupErr error |
| + treeAfterList = func(d *authority.Dir, dir string) { // on the walk's goroutine: errors are recorded, not fatal here |
| if dir != root || swapped { |
| return |
| } |
| swapped = true |
| _, confineErr = d.Walk("..", nil) |
| if err := os.Rename(filepath.Join(root, "sub"), filepath.Join(root, "gone")); err != nil { |
| - t.Fatal(err) |
| + setupErr = err |
| + return |
| + } |
| + if err := os.MkdirAll(filepath.Join(root, "sub"), 0o755); err != nil { |
| + setupErr = err |
| + return |
| + } |
| + os.Chmod(filepath.Join(root, "sub"), 0o755) |
| + if err := os.WriteFile(filepath.Join(root, "sub", "new.db"), []byte("x"), 0o644); err != nil { |
| + setupErr = err |
| + return |
| } |
| - mkdirs(t, filepath.Join(root, "sub")) |
| - writeFile(t, filepath.Join(root, "sub", "new.db")) |
| if err := os.Remove(filepath.Join(root, "plain.data")); err != nil { |
| - t.Fatal(err) |
| + setupErr = err |
| + return |
| } |
| if err := syscall.Mkfifo(filepath.Join(root, "plain.data"), 0o644); err != nil { |
| - t.Fatal(err) |
| + setupErr = err |
| } |
| } |
| t.Cleanup(func() { treeAfterList = nil }) |
| @@ -824,6 +974,9 @@ func TestStrictTreeBetweenListingAndLook(t *testing.T) { |
| case <-time.After(10 * time.Second): |
| t.Fatal("the walk blocked") |
| } |
| + if setupErr != nil { |
| + t.Fatal(setupErr) |
| + } |
| if !errors.Is(confineErr, authority.ErrAscentPastRoot) { |
| t.Fatalf("the tenant root is not confined: %v", confineErr) |
| } |
| @@ -857,3 +1010,94 @@ func openFDCount(t *testing.T) int { |
| } |
| return len(ents) |
| } |
| + |
| +// An inventory a fleet keeps retains the owned files' chains and the |
| +// pinned roots as descriptors (counted), rechecks them by descriptor (a |
| +// directory on a chain replaced by a link to the moved original reads as |
| +// a changed edge though the pathname still leads to the same file), and |
| +// releases everything at Close; a report keeps nothing. |
| +func TestFleetInventoryRetainsAndRechecksChains(t *testing.T) { |
| + dir := platformDir(t) |
| + reg := openRegistry(t, filepath.Join(dir, "registry.db")) |
| + state := filepath.Join(platformDir(t), "unit-a") |
| + db := writeFile(t, filepath.Join(state, "data", "main.db")) |
| + upsert(t, reg, registry.Route{Host: "a.example.com", Account: "bac", App: "one", DBPath: state, Backing: registry.BackingUnit}) |
| + routes, err := reg.List() |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + before := openFDCount(t) |
| + inv, err := fleetInventory(routes, reg.Path(), dir) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, ok := ownedByPath(inv)[db]; !ok { |
| + t.Fatalf("owned %+v unowned %v problems %v", inv.Owned, inv.Unowned, inv.Problems) |
| + } |
| + if inv.facts[db].keep == nil { |
| + t.Fatal("the owned file's chain is not retained") |
| + } |
| + if n, d := openFDCount(t)-before, inv.Descriptors(); n != d || d < 3 { |
| + t.Fatalf("%d descriptors open, the inventory counts %d", n, d) |
| + } |
| + if err := inv.recheck(); err != nil { |
| + t.Fatalf("an unchanged inventory rechecks as changed: %v", err) |
| + } |
| + // the file replaced under its name: the name's identity differs |
| + if err := os.Rename(db, db+".old"); err != nil { |
| + t.Fatal(err) |
| + } |
| + writeFile(t, db) |
| + if err := inv.recheck(); err == nil { |
| + t.Fatal("the file replaced under its name was not seen") |
| + } |
| + if err := os.Rename(db+".old", db); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := inv.recheck(); err != nil { |
| + t.Fatalf("restored, and still read as changed: %v", err) |
| + } |
| + // the chain's edge replaced by a link to the moved original: the |
| + // pathname still leads to the same file; the descriptor says changed |
| + if err := os.Rename(filepath.Join(state, "data"), filepath.Join(state, "data.real")); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := os.Symlink("data.real", filepath.Join(state, "data")); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := inv.recheck(); err == nil { |
| + t.Fatal("a chain edge replaced by a link was not seen") |
| + } |
| + // a root's ancestry: the directory above the tenant's made writable |
| + // by its group fails the platform rule at the next look |
| + if err := os.Chmod(filepath.Dir(state), 0o775); err != nil { |
| + t.Fatal(err) |
| + } |
| + t.Cleanup(func() { os.Chmod(filepath.Dir(state), 0o755) }) |
| + // (the chain edge above is restored so the roots are what the look sees) |
| + if err := os.Remove(filepath.Join(state, "data")); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := os.Rename(filepath.Join(state, "data.real"), filepath.Join(state, "data")); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := inv.recheck(); err == nil { |
| + t.Fatal("a root's ancestry failing the platform rule was not seen") |
| + } |
| + inv.Close() |
| + inv.Close() |
| + if n := openFDCount(t); n != before { |
| + t.Fatalf("after Close: %d open, %d before", n, before) |
| + } |
| + if inv.facts[db].keep != nil || inv.Descriptors() != 0 { |
| + t.Fatal("Close left a chain") |
| + } |
| + // a report through FleetDBOwners keeps no descriptors |
| + rep, err := FleetDBOwners(reg, dir) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if n := openFDCount(t); n != before || rep.Descriptors() != 0 { |
| + t.Fatalf("a report holds %d descriptors (%d open, %d before)", rep.Descriptors(), n, before) |
| + } |
| +} |