amadan
plan 4 rev 12: the e2e wake fixture carries the real mirror and a fake instance; a path-aware trust seam with a portable declaration; the first-arming witnesses assert the waiting rule
Applies the three changes of Astra's READY WITH CHANGES review of rev 11.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1 file changed,
+22
−19
docs/superpowers/plans/2026-10-22-self-serve-moves-4-cleanup.md+22 −19
CI failed — run details
CI log
=== gate === sigwho: pid 2 ppid 1 pgid 0 sid 0; child pid 3 .amadan/test-gate.sh gate: 2026-10-24T12:06:07Z on aarch64, 32 cpus, 57630 MB available, go1.26.8 linux/arm64 gate: +0s start launchctx gate: +0s start vet gate: +0s start gofmt gate: +1s start rest gate: +25s start internal_console@0 gate: +25s start internal_console@1 gate: +25s start internal_console@2 gate: +25s start internal_console@3 gate: +25s start internal_console@4 gate: +25s start internal_console@5 gate: +25s start internal_console@6 gate: +25s start internal_console@7 gate: +25s waiting for internal_console's shards before the next split package gate: +208s start root@0 gate: +208s start root@1 gate: +208s start root@2 gate: +208s waiting for root's shards before the next split package amadan: step gate failed: signal: terminated amadan: job failed at step gate — its output is above, not at the tail of this log
diff --git a/docs/superpowers/plans/2026-10-22-self-serve-moves-4-cleanup.md b/docs/superpowers/plans/2026-10-22-self-serve-moves-4-cleanup.md| index 9e11c6cef..b4a537cbb 100644 |
| --- a/docs/superpowers/plans/2026-10-22-self-serve-moves-4-cleanup.md |
| +++ b/docs/superpowers/plans/2026-10-22-self-serve-moves-4-cleanup.md |
| @@ -2,7 +2,9 @@ |
| > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| -Rev 11, 2026-10-24. Review 10 (`after236/selfserve-plan4-review10.md`) closed W3 and found three gaps; they are answered under the controller's rulings X1 to X3. **Finding 1 (a delayed tombstone creation rebuilds a lifted body that an outstanding mark names), X1:** every lock body this plan's code writes, `LiftTombstone`'s, `MarkCleaning`'s and `UnmarkCleaning`'s, carries `write_id`, 128 random bits drawn afresh for each HTTP attempt, and each attempt is sent with the SDK's retryer off. So each such body reaches the bucket in at most one request, the lock's ETag is unique to that write, and a conditional write can land only while the one body it was conditioned on is current; once that body is replaced it never exists again (Global Constraint "No lock body recurs", where the argument is stated once). The seal's tombstone is litestream's write, not ours (`Tombstone` calls `AcquireLease`, `internal/lease/tombstone.go:192`; `Reassert` calls `RenewLease`, `:248`), and carries no `write_id`; when a delayed creation lands after an ordinary release deleted the lock, the landing's lift of it writes a new body with a fresh `write_id`, so a mark conditioned on the earlier lifted body gets 412. A mark is never conditioned on a body without a `write_id`: it first rewrites such a lifted tombstone as ours (Task 3). The `cleanups` count stays, for the log; nothing rests on it. Older readers ignore both fields (`readLock`, `tombstone.go:136-137`; `LockState`, `lockstate.go:71-72`; litestream's `readLease`, `s3/leaser.go:217-218`). Witness: `TestDelayedTombstoneCreationCannotRestoreAMarkCondition` (Task 3), review 10's interleaving step by step, through the production calls. **Finding 2 (a disarm before root's first gated pass strands a request), X2:** the agent arms only once root's `gated` marker exists. Root writes `gated`, with its nonce, at its first adopt pass after the switch is on, before any publisher stage and whether or not any obligation exists (Task 6b, `enterRootGate`). Until a cleanup run finds the marker, the run visits nothing: no reservation, no mark, no obligation that could owe root work, and one log line per agent start, "cleanup waits for root's first gated pass" (Task 4d). No code removes `gated`, so every obligation past `finished` was reserved after `gated` existed, and root serves its request whatever the switch says later. Witnesses that start with both markers absent: `TestCleanupWaitsForRootsFirstGatedPass` (Task 4d), `TestDisarmAfterTheFirstArmingStillServesTheRequest` and `TestRootArmsBeforeAnyPublisherWhateverTheObligations` (Task 6b), and `TestFirstArmingThenDisarmBeforeTheFirstDrop` (Task 9b). **Finding 3 (the end-to-end fixtures cannot carry the witnesses), X3:** Task 9 is split. Task 9a changes the fixtures, each named with its file: `faultStore` gains a `PutOnce` that keeps its faults; the bridge's `fakeWakeBackend` takes its lease and its restore from a real `activator.ExecBackend` over the box's own bucket, so a wake's refusal, the lease's settlement and the restored rows are real; `leasetest.Holder` stands in front of the bridge's S3 server for held requests; the bridge gains controls for the switch, the agent's uid, a cleanup run, root's pass, the markers and a wake; and `offlinecleanup`'s ownership checks take a trust root that only the bridge sets, so root's step runs as the test's uid. Task 9b holds the witnesses and the census. The plan has 20 tasks. |
| +Rev 12, 2026-10-24. Review 11 (`after236/selfserve-plan4-review11.md`) closed X1 and X2 and asked for three corrections, made here under the controller's rulings Y1 to Y3; nothing else in the plan changed. **Item 3 (the real-wake fixture is incompletely wired), Y1:** Task 9a's `ExecBackend` now carries the box's own S3 mirror (`Mirror`, `lease.NewS3Mirror(ctx, region, region)`, built as the agent builds it, `agent.go:586`) and `Phase1: false`, the legacy promotion mode of a box whose Phase 1 switch is off. So `restorePlain` runs `promoteStagedReplica` (`backend_exec.go:817`), and a wake after the unmark restores the replica a legacy return staged; without the mirror the restore read only the ordinary prefix. The wake backend also carries a fake instance (`fakeWakeBackend.inst`, one per box, on a short socket path, with a handler), because `Start` dereferences `b.inst` (`e2e_bridge.go:697-701`) and a successful wake after the unmark would otherwise panic. A new control, `BoxHandle.Sleep`, stops an awake instance through `Service.Bounce` and waits for `Quiesced`, by which time replication has closed and the lease is released. An agent restart shuts the previous activator down first (`Service.Shutdown`), and so does the world's teardown, before the S3 server closes. **Item 4 (the trust seam cannot see where a file is), Y2:** `RootOwned(fi fs.FileInfo)` becomes `RootOwned(parent *os.File, name string)`, which judges an entry through the directory it was opened under and composes with a new opened-parent check, `RootOnlyDir(parent *os.File)`: the `fstat` that Task 6c's step made inline. Every ownership check of Tasks 6b and 6c goes through `RootOnlyChain`, `RootOnlyDir` or `RootOwned`: the agent's read of `gated`, `QuiesceRootCleanup`'s read of it, the acknowledgement's trust and the step's opened parent. So `SetTrustForE2E` accepts the test's uid only for a place inside its bases, located by the path or by the opened parent's name, and a test-owned file outside them is still refused. `SetTrustForE2E` is also declared in `layout_other.go`, where it records nothing, its restore does nothing and every check stays false, so the portable bridge that calls it builds for darwin and windows. **Item 5 (the first-arming witness contradicts the waiting rule), Y3:** a waiting run visits nothing, so Task 9b's `TestFirstArmingThenDisarmBeforeTheFirstDrop` asserts that the obligation is still `open`, as `gone` wrote it, while cleanup waits, and that it progresses (`finished`, then reserved, marked and deleted) only after root arms. Root's first pass in that witness and in Task 6b's `TestDisarmAfterTheFirstArmingStillServesTheRequest` gets a nonempty census, one other adopt pass alive, so it arms but writes no `quiesced`; certification comes from the pass after the disarm, whose census is empty. The plan still has 20 tasks. |
| + |
| +Rev 11's answers that stand (review 11 closed X1 and X2; Y1 completes X3). Review 10 (`after236/selfserve-plan4-review10.md`) closed W3 and found three gaps; they are answered under the controller's rulings X1 to X3. **Finding 1 (a delayed tombstone creation rebuilds a lifted body that an outstanding mark names), X1:** every lock body this plan's code writes, `LiftTombstone`'s, `MarkCleaning`'s and `UnmarkCleaning`'s, carries `write_id`, 128 random bits drawn afresh for each HTTP attempt, and each attempt is sent with the SDK's retryer off. So each such body reaches the bucket in at most one request, the lock's ETag is unique to that write, and a conditional write can land only while the one body it was conditioned on is current; once that body is replaced it never exists again (Global Constraint "No lock body recurs", where the argument is stated once). The seal's tombstone is litestream's write, not ours (`Tombstone` calls `AcquireLease`, `internal/lease/tombstone.go:192`; `Reassert` calls `RenewLease`, `:248`), and carries no `write_id`; when a delayed creation lands after an ordinary release deleted the lock, the landing's lift of it writes a new body with a fresh `write_id`, so a mark conditioned on the earlier lifted body gets 412. A mark is never conditioned on a body without a `write_id`: it first rewrites such a lifted tombstone as ours (Task 3). The `cleanups` count stays, for the log; nothing rests on it. Older readers ignore both fields (`readLock`, `tombstone.go:136-137`; `LockState`, `lockstate.go:71-72`; litestream's `readLease`, `s3/leaser.go:217-218`). Witness: `TestDelayedTombstoneCreationCannotRestoreAMarkCondition` (Task 3), review 10's interleaving step by step, through the production calls. **Finding 2 (a disarm before root's first gated pass strands a request), X2:** the agent arms only once root's `gated` marker exists. Root writes `gated`, with its nonce, at its first adopt pass after the switch is on, before any publisher stage and whether or not any obligation exists (Task 6b, `enterRootGate`). Until a cleanup run finds the marker, the run visits nothing: no reservation, no mark, no obligation that could owe root work, and one log line per agent start, "cleanup waits for root's first gated pass" (Task 4d). No code removes `gated`, so every obligation past `finished` was reserved after `gated` existed, and root serves its request whatever the switch says later. Witnesses that start with both markers absent: `TestCleanupWaitsForRootsFirstGatedPass` (Task 4d), `TestDisarmAfterTheFirstArmingStillServesTheRequest` and `TestRootArmsBeforeAnyPublisherWhateverTheObligations` (Task 6b), and `TestFirstArmingThenDisarmBeforeTheFirstDrop` (Task 9b). **Finding 3 (the end-to-end fixtures cannot carry the witnesses), X3:** Task 9 is split. Task 9a changes the fixtures, each named with its file: `faultStore` gains a `PutOnce` that keeps its faults; the bridge's `fakeWakeBackend` takes its lease and its restore from a real `activator.ExecBackend` over the box's own bucket, so a wake's refusal, the lease's settlement and the restored rows are real; `leasetest.Holder` stands in front of the bridge's S3 server for held requests; the bridge gains controls for the switch, the agent's uid, a cleanup run, root's pass, the markers and a wake; and `offlinecleanup`'s ownership checks take a trust root that only the bridge sets, so root's step runs as the test's uid. Task 9b holds the witnesses and the census. The plan has 20 tasks. |
| Rev 10's answers that stand (review 10 closed W3 and found W1 and W2 sound as far as they reached; X1 and X2 complete them). **W1 (review 9, finding 1):** the unmark never reconstructs an earlier lock body. It writes this move's lifted tombstone with the `cleanups` count it read plus one, and **always** writes when it finds our mark or this move's lifted tombstone, even with no mark there, so a mark attempt still travelling names a body that is gone. The mark's and the unmark's lock writes are single attempts (`aws.NopRetryer{}` per call), sent again until answered, and a 412 after an unanswered attempt is answered by reading the lock (Task 3, `TestDelayedMarkCannotLandAfterTheUnmark`, `TestUnmarkSettlesAnUncertainMark`; Task 4a, `TestSupersessionAfterAnUncertainMarkSettlesIt`). Rev 11 keeps all of it and rests the uniqueness on `write_id` instead of the count (X1), because a count cannot tell apart two lifts of a recreated tombstone. **W2 (review 9, finding 2):** `carlos system cleanup-ungate` is gone and no code removes `gated` or `quiesced`. A fresh arming (a gated pass that finds `gated` absent: the box's first, or one after an operator removed a malformed marker by hand) writes `gated` afresh with a new nonce, which voids the standing `quiesced` and forces a new census. Disarming leaves both, so root keeps serving the requests of obligations in flight at the disarm, including one published after it, because root's gate is judged by the markers, not by the switch (Task 6b, `TestDisarmedPublicationIsStillServed`, `TestNothingRemovesTheGateMarkers`). Rev 11 adds the order that makes this hold from the first arming on (X2). **W3 (closed by review 10):** Task 9b's disarm witness restarts the console after `drained` and only then asserts that creates succeed, because the drain leaves admission closed until a restart. |
| @@ -65,7 +67,7 @@ Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and |
| ### Task 1: The obligation and its evidence at `gone` |
| **Files:** |
| -- Create: `internal/offlinecleanup/wire.go` (standard library only; every wire type of this plan, `Retired`, `CorruptStub`, `InventoryKey`, `InventoryVersion`, `InFlightDelete`, `KeyOutcome`, `Event`, `PendingPage`, `Report`, `FileIdentity`, `LocalPath`, `KeptPath`, `RootCleanupRequest`, `RootCleanupPath`, `RootCleanupAck`, with Tasks 5a and 6b giving the last ones their semantics), `internal/offlinecleanup/layout_linux.go` and `layout_other.go` (`Identify`, `RootOnlyChain`, `RootOwned`), `internal/host/movescleanup_retired.go` |
| +- Create: `internal/offlinecleanup/wire.go` (standard library only; every wire type of this plan, `Retired`, `CorruptStub`, `InventoryKey`, `InventoryVersion`, `InFlightDelete`, `KeyOutcome`, `Event`, `PendingPage`, `Report`, `FileIdentity`, `LocalPath`, `KeptPath`, `RootCleanupRequest`, `RootCleanupPath`, `RootCleanupAck`, with Tasks 5a and 6b giving the last ones their semantics), `internal/offlinecleanup/layout_linux.go` and `layout_other.go` (`Identify`, `RootOnlyChain`, `RootOnlyDir`, `RootOwned`; Task 9a adds `SetTrustForE2E` to both), `internal/host/movescleanup_retired.go` |
| - Modify: `internal/moves/offline.go` (`OfflineOpDir`, `RetiredFile`, `CleanupReportKey`), `internal/host/movesoffline.go` (`offlineSourceGone`, line 1493: with `rt == nil` and this box the move's sealer by `cfg.offlineRoles(m)`, `w.recordRetired(ctx, s, rec)` runs before the gone report; an error writes no gone report this pass; "untracked" is not an error), `internal/release/programhistory.go` (export `ProgramsPath(bakeDir, host)`, today's `programPath`) |
| - Test: `internal/offlinecleanup/wire_test.go`, `internal/offlinecleanup/layout_linux_test.go`, `internal/host/movescleanup_retired_test.go` |
| @@ -74,7 +76,7 @@ Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and |
| - `type Retired struct { V int; ID, MoveID, Host, Account, App, Box string; Incarnation uint64; From, To string; OpenAt time.Time; FinalTXID uint64; TombstoneOwner string; Bucket, BucketRegion, Prefix string; BucketInventory []InventoryKey; LocalInventory []LocalPath; LocalLeft []KeptPath; WrittenAt time.Time; State, Outcome, Superseding string; FinishedAt, ReservedAt, MarkedAt, RootRequestedAt, CancelRequestedAt, EndedAt time.Time; ReserveUncertain bool; InFlight *InFlightDelete; Keys map[string]KeyOutcome; Events []Event; PagesPending []PendingPage; Root *RootCleanupAck; Rev, PublishedRev uint64 }`, every field tagged snake_case, times `omitzero`, `V = 1`. |
| - **States** (`State`, the only lifecycle field): `open`, `finished`, `cleaning`, `publishing`, `root-requested`, `cancel-pending`, `releasing`, `ended`, `corrupt`. `Outcome` is "" until `releasing`, then `cleaned`, `superseded` or `cancelled` (Task 4a's outcome table). |
| - `type InventoryKey struct { Key, Latest string; Versions []InventoryVersion }` with `Versions` in deletion order and `Latest` the version id of the last (the one current at `gone`); `type InventoryVersion struct { Seq int; VersionID, ETag string; Size int64; DeleteMarker, Latest bool; LastModified time.Time }` (`Seq` the entry's index in `ListVersions`' answer for the whole prefix; `ETag` unquoted, as `lease.ObjectVersion` carries it, `internal/lease/versioned.go:25-33`; "" for a marker); `type InFlightDelete struct { Key, VersionID string; At time.Time }` (`At` the first send, kept on every resend); `type KeyOutcome struct { State string (pending, deleted, superseded, unsettled); Done int (versions, in deletion order, known done); At time.Time }`; `type Event`, `type PendingPage` (Task 5a); `type FileIdentity struct { Dev, Inode uint64; Mode string (symlink or regular); LinkTarget string }`; `type LocalPath struct { Which string (live-link or programs-file); Identity FileIdentity }` (root derives the path itself; the agent's file names none); `type KeptPath struct { Which, Reason string }`. |
| -- `func Identify(path string) (FileIdentity, bool, error)`: `Lstat` (never follows), `Readlink` for a symlink; absent → `false`; any other error fails the step. `func RootOnlyChain(dir string) (bool, error)`: `Lstat` of every component from `/` to `dir`: each a directory (a symlink component answers false), owned by uid 0, `mode & 0o022 == 0`. `func RootOwned(fi fs.FileInfo) bool`: a file owned by uid 0 with `mode & 0o022 == 0`. Every root-only and root-owned check of Tasks 4d, 6b and 6c is one of these two functions, so Task 9a's trust root reaches all of them. Off Linux all three answer absent and false (the cleanup is Linux-only, as the box is). |
| +- `func Identify(path string) (FileIdentity, bool, error)`: `Lstat` (never follows), `Readlink` for a symlink; absent → `false`; any other error fails the step. `func RootOnlyChain(dir string) (bool, error)`: `Lstat` of every component from `/` to `dir`: each a directory (a symlink component answers false), owned by uid 0, `mode & 0o022 == 0`. `func RootOnlyDir(parent *os.File) (bool, error)`, the opened-parent check: `fstat` of a directory the caller opened `O_DIRECTORY|O_NOFOLLOW` after `RootOnlyChain` passed for its path: a directory owned by uid 0 with `mode & 0o022 == 0`; an `fstat` error → false with it. `func RootOwned(parent *os.File, name string) (bool, error)` composes with it: `RootOnlyDir(parent)` first (false or an error → that answer), then `fstatat(parent, name, AT_SYMLINK_NOFOLLOW)`: a regular file owned by uid 0 with `mode & 0o022 == 0` → true; `ENOENT`, another file type (a symlink included), another owner or a write bit → false, nil; any other error → false with it. An entry is judged only under its opened parent, so each check knows where it looks: `RootOnlyChain` by its path, the other two by `filepath.Clean(parent.Name())`, the path the parent was opened by, which is what Task 9a's trust root is scoped by. Every root-only and root-owned check of Tasks 4d, 6b and 6c is one of these three functions, so Task 9a's trust root reaches all of them. Off Linux `Identify` answers absent and the three checks answer false (the cleanup is Linux-only, as the box is). |
| - `ValidateRetired(r Retired) error`: `V == 1`; `ID == CleanupID(Box, Host, MoveID)`; `MoveID` a plain segment; `Host` passes the host shape; `Account`, `App`, `Box` non-empty; `Incarnation > 0`; `From != To`; `State` in the set and not `corrupt`; `Outcome` "" before `releasing` and in the set after; `Superseding` "" or a supersession reason, and set only in `cleaning` with `InFlight` set; `WrittenAt` non-zero; `Prefix` ends with `/`; inventory keys unique, under `Prefix`, neither excluded key; every key has at least one version; every version id passes `offlinecleanup.UsableVersionID` (non-empty and not `null`: `lease.UsableVersionID`'s rule, `internal/lease/staging.go:414`, restated here because this package imports the standard library only, and pinned to it by `TestUsableVersionIDMatchesTheLease`) and is unique within the key; `Seq` unique across the inventory; exactly one version has `Latest` set, it is the last, and `InventoryKey.Latest` is its id; the versions before it in the recorded order (`LastModified` ascending, ties by `Seq` descending); `ETag` non-empty exactly for the versions that are not markers; `Which` values unique across `LocalInventory` and `LocalLeft`, and known; every `Keys` entry names an inventory key with `Done` in `0..len(Versions)`; `InFlight`, when set, names an inventoried version of a key whose `Keys` state is `pending`. |
| - `type CorruptStub struct { Corrupt bool; ID, Host string; FoundAt time.Time }` (`{corrupt: true, id, host, found_at}`); `ValidateCorruptStub(c, box, host, move string) error`: `Corrupt`, `Host == host`, `ID == CleanupID(box, host, move)`, `FoundAt` non-zero. |
| - `func OfflineOpDir(dataDir, host, op string) string` (`<dataDir>/.offline-moves/<host>/<op>`, "" on a bad name, the checks `OfflineParkDir` applies); `func RetiredFile(dataDir, host, op string) string`; `func CleanupReportKey(host, op, box string) string` (`OfflineReportPrefix(host, op) + "cleanup/" + box + ".json"`). |
| @@ -82,7 +84,7 @@ Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and |
| - Writes: `createRetired` is create-only (temp file, `fsync`, `os.Link` into place, directory `fsync`; an existing file is never replaced); `writeRetired` (temp, `fsync`, rename, directory `fsync`) increments `Rev` on every write except `markPublished` (Task 5a), which sets `PublishedRev` alone. |
| - **Corrupt (N6):** a file that is a valid `CorruptStub` stays as it is, never rewritten. A file that is neither a valid `Retired` nor a valid stub is moved aside create-only (`os.Link` then remove): to `retired.json.corrupt-1` when that name is free, else to the next free `retired.json.corrupt-<n>`, so the first damaged file is never overwritten; then the stub is written with `FoundAt` = `retired.json.corrupt-1`'s mtime. A stub keeps the obligation listed as `corrupt` and pages hourly from `FoundAt` (Task 5a) until an operator repairs it (a valid `retired.json` restored from `.corrupt-1`, or the directory removed after the entries and the mark are released by hand); it is never reaped and never released, so any entry or mark it held stays until the repair. |
| -- [ ] **Step 1: Write the failing tests**: `TestRetiredJSONWrittenOnceAtGoneBySealer` (fields; a key with three versions and a key whose current entry is a marker each recorded whole, in deletion order with the latest last and each version's ETag; `FinalTXID` from the seal; a non-sealer source box writes nothing; a second gone pass does not rewrite; an identity error writes nothing and no gone report); `TestUntrackedLifetimeWritesNoRetiredAndReportsGone` (incarnation 0 with no slot, a migrating record, a stale verdict: no file, the gone report written; an unreadable lifetime: no file and no gone report this pass, both next pass); `TestRetiredPrefixIsNormalisedOnce` (`cfg.OwnPrefix = "instances"` → `Prefix == "instances/shop.example.com/"`; the fake versioned store was listed with exactly that prefix; `instances/shop.example.com/lock.json` and `.../build-state.json` excluded with all their versions; `instances/shop.example.co` keys not listed; `ValidateRetired` passes on what gone wrote); `TestInventoryOrderIsOldestFirstLatestLast` (versions with equal `LastModified`: ordered by `Seq` descending, recorded, and the latest last and named whatever the tie order: `k.Latest == k.Versions[len(k.Versions)-1].VersionID && k.Versions[len(k.Versions)-1].Latest`; a listing with two `IsLatest` entries for one key → an error and no file); `TestNullVersionIDWritesNoRetired` (a listing with a `null` id under the prefix: `recordRetired` returns nil, no file exists, the gone report is written; `ValidateRetired` of an otherwise valid inventory with one `null` id → an error naming it); `TestRealIDsOnASuspendedBucketStillWriteRetired` (U5's limit, pinned: a fake bucket marked suspended whose listing under the prefix holds only real ids → `retired.json` is written and validates; the listing check is about ids, and the status is the rollout's check); `TestUsableVersionIDMatchesTheLease` (external test package: for `""`, `"null"`, `"Null"`, `"3HL4kqtJlcpXroDTDmJ+rmSpXd3dIbrHY"`, `offlinecleanup.UsableVersionID(v) == lease.UsableVersionID(v)`); `TestWritablePathsAreLeftNotRequested` (a path under a directory the test uid owns → `LocalLeft`, "writable layout"; a path under a root-only chain the test cannot build is covered by injecting `RootOnlyChain`); `TestCleanupIDIsDeterministic`; `TestValidateRetired` (each rule); `TestCorruptStubIsValidAndKeepsTheFirstDamage` (damaged → `.corrupt-1` and a stub that validates; a second scan changes nothing; a damaged stub → `.corrupt-2`, `FoundAt` still `.corrupt-1`'s mtime); `TestIdentifyUsesLstat` (a dangling link has an identity; a regular file; absent); `TestRootOnlyChain` (`/usr` true on a normal system; a temp directory false; a symlink component false). |
| +- [ ] **Step 1: Write the failing tests**: `TestRetiredJSONWrittenOnceAtGoneBySealer` (fields; a key with three versions and a key whose current entry is a marker each recorded whole, in deletion order with the latest last and each version's ETag; `FinalTXID` from the seal; a non-sealer source box writes nothing; a second gone pass does not rewrite; an identity error writes nothing and no gone report); `TestUntrackedLifetimeWritesNoRetiredAndReportsGone` (incarnation 0 with no slot, a migrating record, a stale verdict: no file, the gone report written; an unreadable lifetime: no file and no gone report this pass, both next pass); `TestRetiredPrefixIsNormalisedOnce` (`cfg.OwnPrefix = "instances"` → `Prefix == "instances/shop.example.com/"`; the fake versioned store was listed with exactly that prefix; `instances/shop.example.com/lock.json` and `.../build-state.json` excluded with all their versions; `instances/shop.example.co` keys not listed; `ValidateRetired` passes on what gone wrote); `TestInventoryOrderIsOldestFirstLatestLast` (versions with equal `LastModified`: ordered by `Seq` descending, recorded, and the latest last and named whatever the tie order: `k.Latest == k.Versions[len(k.Versions)-1].VersionID && k.Versions[len(k.Versions)-1].Latest`; a listing with two `IsLatest` entries for one key → an error and no file); `TestNullVersionIDWritesNoRetired` (a listing with a `null` id under the prefix: `recordRetired` returns nil, no file exists, the gone report is written; `ValidateRetired` of an otherwise valid inventory with one `null` id → an error naming it); `TestRealIDsOnASuspendedBucketStillWriteRetired` (U5's limit, pinned: a fake bucket marked suspended whose listing under the prefix holds only real ids → `retired.json` is written and validates; the listing check is about ids, and the status is the rollout's check); `TestUsableVersionIDMatchesTheLease` (external test package: for `""`, `"null"`, `"Null"`, `"3HL4kqtJlcpXroDTDmJ+rmSpXd3dIbrHY"`, `offlinecleanup.UsableVersionID(v) == lease.UsableVersionID(v)`); `TestWritablePathsAreLeftNotRequested` (a path under a directory the test uid owns → `LocalLeft`, "writable layout"; a path under a root-only chain the test cannot build is covered by injecting `RootOnlyChain`); `TestCleanupIDIsDeterministic`; `TestValidateRetired` (each rule); `TestCorruptStubIsValidAndKeepsTheFirstDamage` (damaged → `.corrupt-1` and a stub that validates; a second scan changes nothing; a damaged stub → `.corrupt-2`, `FoundAt` still `.corrupt-1`'s mtime); `TestIdentifyUsesLstat` (a dangling link has an identity; a regular file; absent); `TestRootOnlyChain` (`/usr` true on a normal system; a temp directory false; a symlink component false); `TestRootOwnedJudgesTheEntryUnderItsOpenedParent` (`/etc` opened: `RootOnlyDir` true and `RootOwned(etc, "passwd")` true on a normal system; a temp directory opened: `RootOnlyDir` false, and `RootOwned` of a file the test wrote there false; a symlink entry false; an absent name `false, nil`). |
| - [ ] **Step 2: Run to verify they fail**: `go test ./internal/offlinecleanup -count=1`, `go test ./internal/host -run 'RetiredJSON|Untracked|RetiredPrefix|InventoryOrder|NullVersionID|RealIDsOnASuspended|WritablePaths|CorruptStub' -count=1`. |
| - [ ] **Step 3: Implement. Step 4: Run to verify they pass.** |
| - [ ] **Step 5: Commit** `moves: the sealing source box records retired.json at gone for a tracked lifetime, with every real version id in a recorded deletion order and the key's latest named, a normalised prefix and a deterministic cleanup id`. |
| @@ -315,8 +317,8 @@ Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and |
| **Interfaces:** |
| - **The root pass lock (R5, F5; T7).** `func LockRootPass(optDir string) (unlock func(), err error)`, in this order, so no directory is created under an ancestry that has not passed: (1) `offlinecleanup.RootOnlyChain(optDir)` on the **existing** chain from `/` to `optDir`: false → `ErrNotRootOnly`, nothing created; an error → itself; (2) `RootCleanupBase(optDir) = <optDir>/root-cleanup/` created by root `0755` when absent (`os.Mkdir`; `EEXIST` is fine); (3) `RootOnlyChain` on the base (a base another uid made, or a symlink, answers false → `ErrNotRootOnly`); (4) `pass.lock` in it opened with `safefile.OpenFile(O_CREATE|O_RDWR, 0o600)` and `lockFileEx` (blocking `flock`), as `lockExclusive` does (`adoptlock.go:55-68`). `ErrNotRootOnly` is a definite answer about the box, not a failure to lock: only root can make a chain root-only or not, so every pass gets the same answer, and root's step can never run there (a writable `/opt` is a writable layout). **Why the step does not depend on the adopt-pass lock's identity:** that lock lives in the agent-writable `/data` (`adoptlock.go:16`), where a shared-uid writer can replace or plant its name (`safefile` then refuses it, and the pass continues unserialized, `agent.go:4655-4661`); the root pass lock lives on a root-only chain, which no other uid can change, so its acquisition needs no identity check. The adopt-pass lock is still required for the step (it is held, so the sweep keeps its own rule), but nothing relies on its identity. |
| -- **The gate (T2).** In `runAdoptUnderRootRelease`, right after `lockAdoptPass` (`agent.go:4655-4661`) and whatever that answered, in every mode (hourly, `-stage-only`, `-config-only`, `-restart-only`), before `ensureUnits` (`:4664`) and every later stage: `g, unlockRoot, rootLockErr := enterRootGate(release.OptDir(), os.Getenv("CARLOS_OFFLINE_CLEANUP") == "on")` (`unlockRoot` deferred when the lock was taken). `func enterRootGate(optDir string, switchOn bool) (g rootGate, unlock func(), err error)` (`adoptrootgate.go`; Task 9a's bridge calls the same function) is: `unlock, err := release.LockRootPass(optDir)`; `gated := switchOn || release.RootCleanupGated(optDir)`; `g = adoptRootGate(gated, err)`, which answers one of three; and, when `g` serves and `gated`, the arming. **Serve** (`rootLockErr == nil`): when `gated` and `<optDir>/root-cleanup/gated` is absent, `ArmRootCleanup` (below) runs here, **before `ensureUnits` and every other stage**, in each of the four modes and whether or not any drop or obligation exists (X2; review 10, finding 2: the agent arms only once it finds the marker, so this write is what lets the first obligation reserve, and it comes before anything that obligation could ask of root); an arming that fails is logged (`ALERT: adopt: arming root cleanup: %v`), the pass goes on under the root pass lock, which keeps it safe, and the next pass arms. Then the pass runs as today, and its last stage is root's (the stage below: the agent check, `QuiesceRootCleanup`, and only then root's step). **Publish nothing** (`gated`, and `rootLockErr` neither nil nor `ErrNotRootOnly`): one `log.Printf("ALERT: adopt: root pass lock %s: %v: publishing nothing this pass", ...)` and `return nil`, so the pass runs none of `ensureUnits`, `recoverRootRestarts` (`:4695`; its resumes call `PointLive` at `rootrollback_resume.go:125` and `roothistory_recovery.go:216`), `rootJudgePrev` (`:4698`), `sweepStagedVersionsPass` (`:4720`), the `-restart-only` branch (`:4758`), config sync and the `-config-only` branch (`:4794`), `capabilityPass` (`:4837`), `linkRootHistory` (`:4888`), `release.Pass` (`:4920`: adoption and rollback, `adopt.go:1751`, `:1870`, `rollback.go:124`, and named recovery inside it) and `heldStagePass` (`:4932`: `materializeheld.go:460`, `:494`); the next pass tries again. **Publish without serving** (not `gated`, or `ErrNotRootOnly`): today's fail-open adoption, logged as the adopt-pass lock's failure is, no root step and no arming (on a box whose `/opt` is not root-only the agent's runs therefore keep waiting, and its leftovers stay the runbook's). `func RootCleanupGated(optDir string) bool`: `Lstat` of `<optDir>/root-cleanup/gated`: present → true; `ENOENT` → false; any other error → true (fail closed). `func RootCleanupArmed(optDir string) (bool, error)`, the agent's read (Task 4d, X2): `Lstat` of the same file: a regular file for which `offlinecleanup.RootOwned` holds, under a `RootCleanupBase` that passes `offlinecleanup.RootOnlyChain` → true; `ENOENT`, another file type, another owner or a writable base → false (logged by the agent); any other error → false with it. It fails closed the other way from root's: an unknown marker gates root's passes and leaves the agent waiting. No code removes the marker (W2; review 9, finding 2): it records that cleanup was armed on this box, so a disarmed box, whose obligations may still wait for root or may still publish a request (Global Constraint "Rolling plan 4 back is disarming cleanup"), keeps serving them under the same rule; the gate is judged by the markers, not by the switch. **Why this closes R5:** on a gated box every pass that publishes holds the root pass lock, and root's step runs only as the last stage of a pass holding it, so no root publisher of the live link or the programs file runs while root's step runs; the agent's own publishers are excluded by the freeze and the held-stage lock as before. On a box never gated no root step can run (it needs the lock and the switch or the marker), so fail-open adoption there races nothing. The moment a box first becomes gated is closed by the arming boundary. |
| -- **The arming boundary (U1; review 7's finding 2).** The gate leaves one interleaving: a pass that read the switch off and found no marker, and whose two locks failed (a planted `/data/adopt-pass.lock` and a root-level fault on `LockRootPass`), publishes fail-open; if it is still running when a later pass has armed the box and runs root's step, a publication can land between root's `fstatat` and `unlinkat`. Setting the switch or creating the marker does not stop a pass already past its gate. **How passes run, verified:** `carlos adopt` runs one pass per process and exits (`runAdopt`, `agent.go:4568`); it is started by four root oneshot units, `carlos-adopt.service` (`carlos-adopt.timer`, hourly), `carlos-stage.service` (`carlos-stage.path`, the agent's `/data/stage-requested`), `carlos-config-sync.service` (`carlos-adopt.path`, `/data/config-sync-requested`) and `carlos-restart.service` (`carlos-restart.path`, `/data/restart-requested`), each a separate process, and passes of different units can overlap ("the four units that exec this verb can start in the same second", `agent.go:4645-4648`). Within one unit nothing overlaps: systemd runs one instance of a unit, a start request for a oneshot unit that is still activating joins its job, and a unit is inactive again only once its control group is empty (the adopt units keep the default `KillMode=control-group`), so a restarted pass's predecessor has exited and its kernel operations have completed before the new one runs. An in-process mutex would therefore serialise nothing (no process runs two passes), and none is added; the boundary is a census of processes instead. **The census, by executable (V1).** `func CarlosProcesses(procDir, optDir string) ([]CarlosProcess, error)` (Linux; `type CarlosProcess struct { PID int; Verb string; RealUID, EffectiveUID uint32 }`): every numeric entry of `procDir` (`/proc` in production; the adopt units set neither `ProtectProc=` nor `PrivatePIDs=`, unlike `carlos-relay.service:32` and `carlos-instance-sidecar@.service:117-119`, so a root pass sees every process) **whose executable is the installed binary**: `readlink(<pid>/exe)`, with a trailing ` (deleted)` removed (what a task still running a binary an apply has renamed over reads, `internal/host/rootgeneration_services_linux.go:344-350`), equal to `<optDir>/carlos` or `<optDir>/carlos.prev`; or `stat(<pid>/exe)`, which follows to the task's image, naming the device and inode of either. `Verb` is `argv[1]` of `cmdline` split on NUL; the uids are the first two fields of `status`'s `Uid:` line. `argv[0]` is never read, so a renamed path or a hard link of the installed binary is still counted, and **an unrelated binary that runs `adopt` is outside support** (Self-review notes). An entry that vanishes while it is read (`ENOENT`, `ESRCH`) has exited and is skipped; any other error is returned, and an unknown census is never an empty one. `func AdoptPasses(procDir, optDir string, self int) ([]int, error)`: the processes with `Verb == "adopt"` and `RealUID == 0`, other than `self`. A process of another uid is not counted, so the agent's uid cannot hold the boundary shut. **The agent check (V1).** `func AgentRunsUnprivileged(ctx context.Context, procDir, optDir string, unitUser func(context.Context) (string, error)) (ok bool, reason string, err error)`, with `unitUser` in production `systemctl show -p User --value carlos-edge.service` (systemd's merged value, drop-ins included; root already asks systemd about this unit, `internal/host/claimstop_production.go:52`): a user "" (systemd's default, which is root), `root` or `0` → false, "carlos-edge.service runs the agent as root"; then `CarlosProcesses`: any with `Verb == "agent"` and `RealUID == 0 || EffectiveUID == 0` → false, "a carlos agent runs as root (pid <n>)"; an error from either → the error; else true. Off Linux `CarlosProcesses` and `AgentRunsUnprivileged` answer `ErrNotRootOnly`. **The arming and its nonce (V2; review 8, item 2).** `func ArmRootCleanup(optDir string) error`, called by `enterRootGate` for a serving gated pass when `gated` is absent, before any other stage: a fresh nonce, 16 bytes from `crypto/rand` as 32 hex digits, written as `{"nonce": N, "at": <now>}` to a temp file in `RootCleanupBase`, `fsync`ed, `os.Link`ed to `gated` (create-only: `EEXIST` means another pass armed first, which is fine), the temp removed, the base `fsync`ed; root `0644`. Every creation of `gated` is an arming and has its own nonce. `gated` is absent only before the box's first arming, or after an operator removed it by hand, which is the one repair for a malformed marker and is made with the switch on, so every pass stays gated meanwhile. So arming again after a disarm, with `gated` still standing, keeps its nonce and its certificate, which is sound because every pass since that arming was gated. `func QuiesceRootCleanup(optDir string, census func() ([]int, error)) (bool, error)`, called only by a pass holding both locks with `gated` true, after every other stage and after the agent check: (1) `gated`: `Lstat` not a regular file owned by uid 0 with no group or other write bit, or its body (read bounded, 512 bytes) without a nonce of 32 hex digits → false, logged (fail closed: an operator repairs it by removing `gated` by hand with the switch on, and the next serving pass arms afresh with a new nonce); (2) `<optDir>/root-cleanup/quiesced` present and recording that nonce → true; present with another nonce, or with none (written for an earlier arming, by hand, or by rev 8's form, which recorded the `(dev, inode)` of `gated`) → **void**: it certifies nothing and is left in place, and on to (3); (3) the census: an error → false with it; any pid → false, one `log.Printf("root-cleanup: waiting for adopt pass %v, which may predate arming, to end before the first cleanup step", pids)`; none → `quiesced` written as `{"nonce": N, "at": <now>}` (temp, `fsync`, `rename` over any void certificate, base `fsync`; two writers are never concurrent, because each holds the root pass lock) → true. Nothing removes `quiesced` (W2): a void one is replaced only by a certificate for the current nonce, after its census. Nothing reads the file identity of `gated`: a filesystem can give a recreated `gated` the inode of the removed one (review 8, item 2), and a nonce is never written twice. **Why it closes the first arming:** a pass that publishes without the root pass lock is one whose gate answered "publish without serving", so it read the switch off and found no `gated` marker; `quiesced` is written for a nonce only after the `gated` carrying that nonce existed, at a moment the census found no other root adopt process of the installed binary alive; every such process alive after that moment was started after it, so after `gated` existed, and reads the marker (`RootCleanupGated` fails closed on any `Lstat` error but `ENOENT`), so it holds the root pass lock or publishes nothing. A `/proc` walk lists every process that exists for the whole walk, and one started during the walk started after `gated` existed. A certificate for an earlier arming certifies nothing about this one, because its nonce differs, and is replaced only after a census for this one. So once `quiesced` stands for the current nonce, no pass admitted before this arming is still running, and root's step, which requires it, never meets one. A census that keeps finding a process (a pass blocked behind this one's lock counts too) costs only time: the next pass, at the latest the next hour, tries again, and the obligation pages `root-wait` meanwhile. |
| +- **The gate (T2).** In `runAdoptUnderRootRelease`, right after `lockAdoptPass` (`agent.go:4655-4661`) and whatever that answered, in every mode (hourly, `-stage-only`, `-config-only`, `-restart-only`), before `ensureUnits` (`:4664`) and every later stage: `g, unlockRoot, rootLockErr := enterRootGate(release.OptDir(), os.Getenv("CARLOS_OFFLINE_CLEANUP") == "on")` (`unlockRoot` deferred when the lock was taken). `func enterRootGate(optDir string, switchOn bool) (g rootGate, unlock func(), err error)` (`adoptrootgate.go`; Task 9a's bridge calls the same function) is: `unlock, err := release.LockRootPass(optDir)`; `gated := switchOn || release.RootCleanupGated(optDir)`; `g = adoptRootGate(gated, err)`, which answers one of three; and, when `g` serves and `gated`, the arming. **Serve** (`rootLockErr == nil`): when `gated` and `<optDir>/root-cleanup/gated` is absent, `ArmRootCleanup` (below) runs here, **before `ensureUnits` and every other stage**, in each of the four modes and whether or not any drop or obligation exists (X2; review 10, finding 2: the agent arms only once it finds the marker, so this write is what lets the first obligation reserve, and it comes before anything that obligation could ask of root); an arming that fails is logged (`ALERT: adopt: arming root cleanup: %v`), the pass goes on under the root pass lock, which keeps it safe, and the next pass arms. Then the pass runs as today, and its last stage is root's (the stage below: the agent check, `QuiesceRootCleanup`, and only then root's step). **Publish nothing** (`gated`, and `rootLockErr` neither nil nor `ErrNotRootOnly`): one `log.Printf("ALERT: adopt: root pass lock %s: %v: publishing nothing this pass", ...)` and `return nil`, so the pass runs none of `ensureUnits`, `recoverRootRestarts` (`:4695`; its resumes call `PointLive` at `rootrollback_resume.go:125` and `roothistory_recovery.go:216`), `rootJudgePrev` (`:4698`), `sweepStagedVersionsPass` (`:4720`), the `-restart-only` branch (`:4758`), config sync and the `-config-only` branch (`:4794`), `capabilityPass` (`:4837`), `linkRootHistory` (`:4888`), `release.Pass` (`:4920`: adoption and rollback, `adopt.go:1751`, `:1870`, `rollback.go:124`, and named recovery inside it) and `heldStagePass` (`:4932`: `materializeheld.go:460`, `:494`); the next pass tries again. **Publish without serving** (not `gated`, or `ErrNotRootOnly`): today's fail-open adoption, logged as the adopt-pass lock's failure is, no root step and no arming (on a box whose `/opt` is not root-only the agent's runs therefore keep waiting, and its leftovers stay the runbook's). `func RootCleanupGated(optDir string) bool`: `Lstat` of `<optDir>/root-cleanup/gated`: present → true; `ENOENT` → false; any other error → true (fail closed). `func RootCleanupArmed(optDir string) (bool, error)`, the agent's read (Task 4d, X2): `offlinecleanup.RootOnlyChain(RootCleanupBase(optDir))`, then the base opened `O_DIRECTORY|O_NOFOLLOW` and `offlinecleanup.RootOwned(base, "gated")`, which checks the opened base with `RootOnlyDir` before the file: both hold → true; `ENOENT` (of the base or of the file), another file type, another owner or a writable base → false (logged by the agent); any other error → false with it. It fails closed the other way from root's: an unknown marker gates root's passes and leaves the agent waiting. No code removes the marker (W2; review 9, finding 2): it records that cleanup was armed on this box, so a disarmed box, whose obligations may still wait for root or may still publish a request (Global Constraint "Rolling plan 4 back is disarming cleanup"), keeps serving them under the same rule; the gate is judged by the markers, not by the switch. **Why this closes R5:** on a gated box every pass that publishes holds the root pass lock, and root's step runs only as the last stage of a pass holding it, so no root publisher of the live link or the programs file runs while root's step runs; the agent's own publishers are excluded by the freeze and the held-stage lock as before. On a box never gated no root step can run (it needs the lock and the switch or the marker), so fail-open adoption there races nothing. The moment a box first becomes gated is closed by the arming boundary. |
| +- **The arming boundary (U1; review 7's finding 2).** The gate leaves one interleaving: a pass that read the switch off and found no marker, and whose two locks failed (a planted `/data/adopt-pass.lock` and a root-level fault on `LockRootPass`), publishes fail-open; if it is still running when a later pass has armed the box and runs root's step, a publication can land between root's `fstatat` and `unlinkat`. Setting the switch or creating the marker does not stop a pass already past its gate. **How passes run, verified:** `carlos adopt` runs one pass per process and exits (`runAdopt`, `agent.go:4568`); it is started by four root oneshot units, `carlos-adopt.service` (`carlos-adopt.timer`, hourly), `carlos-stage.service` (`carlos-stage.path`, the agent's `/data/stage-requested`), `carlos-config-sync.service` (`carlos-adopt.path`, `/data/config-sync-requested`) and `carlos-restart.service` (`carlos-restart.path`, `/data/restart-requested`), each a separate process, and passes of different units can overlap ("the four units that exec this verb can start in the same second", `agent.go:4645-4648`). Within one unit nothing overlaps: systemd runs one instance of a unit, a start request for a oneshot unit that is still activating joins its job, and a unit is inactive again only once its control group is empty (the adopt units keep the default `KillMode=control-group`), so a restarted pass's predecessor has exited and its kernel operations have completed before the new one runs. An in-process mutex would therefore serialise nothing (no process runs two passes), and none is added; the boundary is a census of processes instead. **The census, by executable (V1).** `func CarlosProcesses(procDir, optDir string) ([]CarlosProcess, error)` (Linux; `type CarlosProcess struct { PID int; Verb string; RealUID, EffectiveUID uint32 }`): every numeric entry of `procDir` (`/proc` in production; the adopt units set neither `ProtectProc=` nor `PrivatePIDs=`, unlike `carlos-relay.service:32` and `carlos-instance-sidecar@.service:117-119`, so a root pass sees every process) **whose executable is the installed binary**: `readlink(<pid>/exe)`, with a trailing ` (deleted)` removed (what a task still running a binary an apply has renamed over reads, `internal/host/rootgeneration_services_linux.go:344-350`), equal to `<optDir>/carlos` or `<optDir>/carlos.prev`; or `stat(<pid>/exe)`, which follows to the task's image, naming the device and inode of either. `Verb` is `argv[1]` of `cmdline` split on NUL; the uids are the first two fields of `status`'s `Uid:` line. `argv[0]` is never read, so a renamed path or a hard link of the installed binary is still counted, and **an unrelated binary that runs `adopt` is outside support** (Self-review notes). An entry that vanishes while it is read (`ENOENT`, `ESRCH`) has exited and is skipped; any other error is returned, and an unknown census is never an empty one. `func AdoptPasses(procDir, optDir string, self int) ([]int, error)`: the processes with `Verb == "adopt"` and `RealUID == 0`, other than `self`. A process of another uid is not counted, so the agent's uid cannot hold the boundary shut. **The agent check (V1).** `func AgentRunsUnprivileged(ctx context.Context, procDir, optDir string, unitUser func(context.Context) (string, error)) (ok bool, reason string, err error)`, with `unitUser` in production `systemctl show -p User --value carlos-edge.service` (systemd's merged value, drop-ins included; root already asks systemd about this unit, `internal/host/claimstop_production.go:52`): a user "" (systemd's default, which is root), `root` or `0` → false, "carlos-edge.service runs the agent as root"; then `CarlosProcesses`: any with `Verb == "agent"` and `RealUID == 0 || EffectiveUID == 0` → false, "a carlos agent runs as root (pid <n>)"; an error from either → the error; else true. Off Linux `CarlosProcesses` and `AgentRunsUnprivileged` answer `ErrNotRootOnly`. **The arming and its nonce (V2; review 8, item 2).** `func ArmRootCleanup(optDir string) error`, called by `enterRootGate` for a serving gated pass when `gated` is absent, before any other stage: a fresh nonce, 16 bytes from `crypto/rand` as 32 hex digits, written as `{"nonce": N, "at": <now>}` to a temp file in `RootCleanupBase`, `fsync`ed, `os.Link`ed to `gated` (create-only: `EEXIST` means another pass armed first, which is fine), the temp removed, the base `fsync`ed; root `0644`. Every creation of `gated` is an arming and has its own nonce. `gated` is absent only before the box's first arming, or after an operator removed it by hand, which is the one repair for a malformed marker and is made with the switch on, so every pass stays gated meanwhile. So arming again after a disarm, with `gated` still standing, keeps its nonce and its certificate, which is sound because every pass since that arming was gated. `func QuiesceRootCleanup(optDir string, census func() ([]int, error)) (bool, error)`, called only by a pass holding both locks with `gated` true, after every other stage and after the agent check: (1) `gated`: `offlinecleanup.RootOwned` false for it under the opened `RootCleanupBase` (not a regular file owned by uid 0 with no group or other write bit), or its body (read bounded, 512 bytes) without a nonce of 32 hex digits → false, logged (fail closed: an operator repairs it by removing `gated` by hand with the switch on, and the next serving pass arms afresh with a new nonce); (2) `<optDir>/root-cleanup/quiesced` present and recording that nonce → true; present with another nonce, or with none (written for an earlier arming, by hand, or by rev 8's form, which recorded the `(dev, inode)` of `gated`) → **void**: it certifies nothing and is left in place, and on to (3); (3) the census: an error → false with it; any pid → false, one `log.Printf("root-cleanup: waiting for adopt pass %v, which may predate arming, to end before the first cleanup step", pids)`; none → `quiesced` written as `{"nonce": N, "at": <now>}` (temp, `fsync`, `rename` over any void certificate, base `fsync`; two writers are never concurrent, because each holds the root pass lock) → true. Nothing removes `quiesced` (W2): a void one is replaced only by a certificate for the current nonce, after its census. Nothing reads the file identity of `gated`: a filesystem can give a recreated `gated` the inode of the removed one (review 8, item 2), and a nonce is never written twice. **Why it closes the first arming:** a pass that publishes without the root pass lock is one whose gate answered "publish without serving", so it read the switch off and found no `gated` marker; `quiesced` is written for a nonce only after the `gated` carrying that nonce existed, at a moment the census found no other root adopt process of the installed binary alive; every such process alive after that moment was started after it, so after `gated` existed, and reads the marker (`RootCleanupGated` fails closed on any `Lstat` error but `ENOENT`), so it holds the root pass lock or publishes nothing. A `/proc` walk lists every process that exists for the whole walk, and one started during the walk started after `gated` existed. A certificate for an earlier arming certifies nothing about this one, because its nonce differs, and is replaced only after a census for this one. So once `quiesced` stands for the current nonce, no pass admitted before this arming is still running, and root's step, which requires it, never meets one. A census that keeps finding a process (a pass blocked behind this one's lock counts too) costs only time: the next pass, at the latest the next hour, tries again, and the obligation pages `root-wait` meanwhile. |
| - **The stage (S12).** `rootCleanupPass(...)` runs after `heldStagePass` (line 4932), so after `release.Pass`, inside which named recovery runs (`recovery.passOption`, `agent.go:4890`, `namedrecovery_pass.go:15`), and only when `passLockErr == nil && rootLockErr == nil && gated`; nonfatal. It first runs the agent check, `st.AgentCheck(ctx)`: an error → nothing this pass; false → one `log.Printf("ALERT: root-cleanup: %s; cleanup needs the agent to run as its own user", reason)`, no `quiesced` written, and every request this pass would serve (each drop that passes the checks (0) and (1) below and is taken, and each taken request) answered `cancelled` with `Reason: "cleanup needs the agent to run as its own user"`, with nothing touched (no path step, no authority call); true → `QuiesceRootCleanup(st.OptDir, st.Census)`, and only when that answers true, `ServeRootCleanupRequests`. The check runs once per pass, before any path is touched. `-config-only` and `-restart-only` return before it (`agent.go:4758-4767`, `:4794-4801`) but have taken the root pass lock, so they wait for a running step and a running step waits for them. Inside one pass the stages run one after another, so no root publisher of this pass runs while root's step runs, and every other adopt pass is blocked on the root pass lock. |
| - **Files.** The agent's drop `<OfflineOpDir>/root-cleanup.json` (create-only) and cancel `<OfflineOpDir>/root-cleanup.cancel` (create-only). Root's directory `release.RootCleanupDir(optDir, id) = <optDir>/root-cleanup/<id>/` holding `request.json` (root's copy), `journal` (Task 6c) and exactly one of `done.json`, `cancelled.json`. Each `<id>/` is created by root `0755` only after `RootOnlyChain` has passed on the base (T7: the existing ancestry is checked before any missing directory is created), and used only when the chain passes on it too. **When it does not (S16), root writes nothing there**, removes nothing, and leaves the drop where it is: it logs `root-cleanup: <id> refused: <dir> is not root-only; repair its ownership, or remove <drop> to give up` to the adopt unit's journal (`log.Printf`, the systemd journal of `carlos-adopt.service`), which is the only record of the refusal. A repaired ancestry lets the next pass take the request. |
| - The wire types (declared in Task 1): `type RootCleanupRequest struct { V int; CleanupID, MoveID, Host, Account, App, Box string; Incarnation uint64; From, To string; Bucket, BucketRegion, Prefix string; Paths []RootCleanupPath; WrittenAt time.Time }`; `type RootCleanupPath struct { Which string; Identity FileIdentity }`; `type RootCleanupAck struct { V int; CleanupID, MoveID, Host, Outcome string (done or cancelled); Removed, Absent []string; Kept []KeptPath; Reason string; At time.Time }`. |
| @@ -324,11 +326,11 @@ Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and |
| - **Discovery (F6):** each pass walks, with `safefile` (no symlinks), (i) the drops `<DataDir>/.offline-moves/<host>/<move>/root-cleanup.json` (host passing `ValidHostShape`, move a plain segment; `id = CleanupID(o.Box, host, move)`) and (ii) root's own `<OptDir>/root-cleanup/<id>/` directories that hold `request.json` and no acknowledgement, so a request root took and then crashed on is served from its own copy whether or not a drop remains. |
| - **Root, per drop:** (0) `RootOnlyChain` on `RootCleanupBase` and, when it exists, `RootCleanupDir(id)`, before anything is created: false → the S16 refusal above, next drop. (1) Read bounded (4096 bytes) and validate: `Host`, `MoveID` equal the path; `Box == o.Box`; `CleanupID == id`; `From == o.Region`; `Which` values known and unique. Invalid → `cancelled` under `id` with `Reason: "malformed request"`, the drop removed. (2) Copy: `request.json` create-only (temp, `fsync`, `os.Link`, directory `fsync`) with the drop's bytes; the journal's first line `taken <at>` (`fsync`); then the drop is removed. A `request.json` already present with other bytes: the drop is removed and logged, root's copy stands (a later drop never changes or cancels a taken request). From here root reads only its copy. |
| - **Root, per taken request** (from (2) or from discovery (ii)): Task 6c's journal recovery; `o.Authorize(ctx, req)`: `ErrRootCleanupRefused` → `cancelled` with its reason, nothing touched; another error → the next pass; each path not yet terminal through Task 6c's step; every path terminal → `cancelled.json` when the authority refused or any path was kept for "cancelled", else `done.json` (temp, `fsync`, rename, directory `fsync`). `<optDir>/root-cleanup/<id>/` older than 30 days past its acknowledgement is removed by root. |
| -- `func ReadRootCleanupAck(optDir, id string) (ack *offlinecleanup.RootCleanupAck, trusted bool, err error)`: `trusted` only when `RootOnlyChain(RootCleanupDir(optDir, id))` passes and the file, opened with `safefile` (no symlinks), is owned by uid 0 with no group or other write bit; an acknowledgement the agent cannot trust is advisory (Task 4a's outcome table: `superseded`, every requested path recorded kept "writable layout"). |
| +- `func ReadRootCleanupAck(optDir, id string) (ack *offlinecleanup.RootCleanupAck, trusted bool, err error)`: `trusted` only when `RootOnlyChain(RootCleanupDir(optDir, id))` passes and the file, opened with `safefile` (no symlinks), is one `offlinecleanup.RootOwned(dir, name)` holds for, `dir` being that directory opened `O_DIRECTORY|O_NOFOLLOW` (owned by uid 0 with no group or other write bit); an acknowledgement the agent cannot trust is advisory (Task 4a's outcome table: `superseded`, every requested path recorded kept "writable layout"). |
| - `func rootCleanupPass(ctx context.Context, st rootStage) error`, with `type rootStage struct { DataDir, OptDir string; Reg *registry.Registry; Store release.Store; Judge *provision.LifetimeJudge; Box, Region string; Hib func() (*activator.Config, error); AgentCheck func(context.Context) (bool, string, error); Census func() ([]int, error) }` (`runAdoptUnderRootRelease` fills `Hib` with `activator.ConfigFromEnv`, `AgentCheck` with `release.AgentRunsUnprivileged(ctx, "/proc", OptDir, unitUser)` and `Census` with `release.AdoptPasses("/proc", OptDir, os.Getpid())`; Task 9a's bridge fills them for its box): the stage above, then `ServeRootCleanupRequests` with `Authorize` = Task 6a's callback over `Store`, `Judge`, `Box`, `Region` and `Hib`, `Freeze` = `reg.WithHostFrozen` (a non-nil row is `rowPresent`), `HostLock` = `withHeldStageLock(dataDir, ·, ·)`, `Processes` reading `/proc/*/exe` (`ENOENT`/`ESRCH` skipped as exited, any other error returned). |
| - **The agent's rows** (Task 4a's row 6): `publishing`: `supersededBy` → supersede; the drop absent and `RootCleanupDir` absent → `writeRootCleanupRequest` (create-only; an existing drop with equal bytes is fine) → touch the stage-request marker (`holdStageRequester(dataDir)(Host)`, `moveshold.go:277`) → `RootRequestedAt = now`, `root-requested`, event; either present (a crash after the drop) → `root-requested` with `RootRequestedAt` restored, in this order: the drop's mtime when the drop exists; else the `at` of root's journal `taken` line; else `now`. `root-requested`: `supersededBy` → supersede (to `cancel-pending`); `ReadRootCleanupAck(optDir, ID)` → an acknowledgement whose `CleanupID`, `Host`, `MoveID` match → `Root = ack`, event, `releasing` with the outcome table's answer; a mismatch → page `corrupt-<h>`, stay; none → the drop is read **first** and then `RootCleanupDir`: both absent (an operator removed the drop before root took it; root takes a request into its directory before it removes the drop) → `releasing`, `Outcome = superseded`; else, after `rootRefusedAfter` (1 h) since `RootRequestedAt` with the agent's own `RootOnlyChain` check of `RootCleanupBase` (or of `RootCleanupDir` when it exists) failing → event `root-refused` and the hourly `root-refused-<h>` page, stay (**never a release**; the mark, which has no expiry, and the entries stay until an operator repairs the ancestry or removes the drop); else the `root-wait` pages. `cancel-pending`: the cancel file written create-only and `CancelRequestedAt` set (once); an acknowledgement → `releasing`, `Outcome = superseded`; else the drop is read **first** and then `RootCleanupDir`: both absent → `releasing`, `Outcome = superseded` (root takes a request into its directory before it removes the drop, so a drop already gone means root's directory exists, and no drop was written after `publishing`); else the `cancel-wait` pages. |
| -- [ ] **Step 1: Write the failing tests**: `TestRootCleanupStepRunsInsideThePass` (a pass driven through `runAdoptUnderRootRelease`'s stages with fakes: the step runs after `release.Pass` and `heldStagePass`; with the adopt-pass lock failed, with the root pass lock failed, and on `-config-only` and `-restart-only`, no step; every mode takes the root pass lock before its first publisher); `TestRootPassLockIsRootOnly` (a base directory the test uid owns → `ErrNotRootOnly`, no lock; the lock's chain check runs as any uid against an injected chain check); `TestLockRootPassChecksAncestryBeforeCreating` (T7: an `optDir` whose existing chain fails → `errors.Is(err, release.ErrNotRootOnly)` and `os.Lstat(base)` answers `fs.ErrNotExist` afterwards; a root-only `optDir` without a base → created, checked, locked, in that order in the injected syscall log; a base that is a symlink → `ErrNotRootOnly`, nothing created through it); `TestGatedPassPublishesNothingWithoutTheRootPassLock` (T2: a pass driven through `runAdoptUnderRootRelease`'s stages with a recording fake per stage; the switch on and `LockRootPass` failing with an injected `EIO` → `len(calls) == 0` across `ensureUnits`, `recoverRootRestarts`, `rootJudgePrev`, `sweepStagedVersionsPass`, the restart pass, config sync, `capabilityPass`, `linkRootHistory`, `release.Pass`, `heldStagePass` and the root step, in each of the four modes, and one log line; the switch off with the marker present → the same; the next pass with the lock → every stage in today's order and the step last); `TestNeverArmedBoxKeepsFailOpenAdoption` (the switch off and no marker, the lock failing → every stage runs as today, no step, no marker created; `ErrNotRootOnly` with the switch on → the same); `TestFirstArmingWaitsForAFailOpenPassToEnd` (U1's witness, review 7's finding 2 interleaving, through `runAdoptUnderRootRelease`'s stages with the production lock functions, an injected `LockRootPass` fault for one pass and an injected census: pass A, with the switch off, no marker, `/data/adopt-pass.lock` replaced by a symlink and `LockRootPass` failing with `EIO`, runs fail-open and is paused inside `release.Pass` after reading its route; the switch goes on and a drop is written; pass B takes both locks, arms the box (`gated` with a nonce), its agent check answers true, and its census answers A's pid: `QuiesceRootCleanup` answers false, `os.Lstat(quiesced)` answers `fs.ErrNotExist`, and root's step is not called, `len(stepCalls) == 0`; A resumes, publishes a new live link for the host and exits, and the census answers no pid; pass C writes `quiesced` and runs root's step, which finds the link A published, not the recorded one: `kept`, "identity changed", and `release.LiveLink(optDir, host)` still names A's target); `TestAdoptPassesCensus` (V1: a fake `/proc` tree whose `exe` entries are symlinks into a fake `optDir`: uid-0 processes running `adopt --stage-only` through `<opt>/carlos`, through `<opt>/carlos.prev`, through an `exe` that reads `<opt>/carlos (deleted)`, and through a hard link of `<opt>/carlos` at another path (`stat` names its inode) are each counted, and so is one whose `argv[0]` reads `x`; a uid-0 `adopt` through `/usr/local/bin/carlos`, a copy with another inode, is not (outside support), nor are `self`, a uid-1001 process through `<opt>/carlos` running `adopt`, a uid-0 `<opt>/carlos agent` and an entry removed during the walk; an unreadable `status` → the error); `TestStaleCertificateIsVoid` (V2's witness, forcing identity reuse: `gated` armed with nonce N1; beside it, in turn, a `quiesced` the test writes with nonce N0, and one in rev 8's form whose `gated_dev` and `gated_ino` equal the current `gated`'s `Lstat` identity and which has no nonce; with the injected census answering a pid, `QuiesceRootCleanup` answers false, `quiesced` is left byte for byte as the test wrote it (nothing removes it), and `len(stepCalls) == 0`; with the census then answering none, it is replaced by one with `nonce == N1` and the step runs; a `gated` that is a symlink, is owned by another uid, or holds `"nonce": "zz"` → false and nothing written); `TestArmingWritesAFreshNonce` (two armings with `gated` removed by hand between, the only way a box is armed twice: `n1 != n2`, each matching `^[0-9a-f]{32}$`; the injected syscall log of one arming is write temp, `fsync` temp, link `gated`, unlink temp, `fsync` base; two passes arming at once: one link wins, the other's `os.Link` answers `EEXIST`, and both then read the winner's nonce); `TestDisarmedPublicationIsStillServed` (W2's witness, review 9's finding 2 interleaving: an armed box, with `gated` and `quiesced` standing for nonce N; an obligation reaches `publishing` and `cleanupBeforeDropHook` pauses its visit immediately before its first drop; `CARLOS_OFFLINE_CLEANUP` is unset in the test's `host.env` and the agent restarts with `cfg.CleanupSwitch == false`, the paused worker discarded; the new worker's visit writes the drop and touches the stage-request marker; root's next pass, driven through `runAdoptUnderRootRelease`'s stages with the switch off: `adoptRootGate` answers serve, because `gated` stands, the agent check answers true, `QuiesceRootCleanup` answers true with no census call (`quiesced` names N), and root takes the request and acknowledges `done`, with `release.LiveLink(optDir, host)` absent afterwards; the agent's next visit reads the acknowledgement: `releasing`, `r.Outcome == "cleaned"`, the mark lifted, the release withheld with `release-withheld` owed, and `len(fake.PutsAfter(mark, moves.OfflineActiveKey)) == 0`; `gated` and `quiesced` are byte for byte what they were before the disarm); `TestDisarmAfterTheFirstArmingStillServesTheRequest` (X2's witness, review 10's finding 2 interleaving, with both markers absent at the start: `os.Lstat(gated)` and `os.Lstat(quiesced)` answer `fs.ErrNotExist`; `CARLOS_OFFLINE_CLEANUP=on` in the test's `host.env` and the agent restarted with `cfg.CleanupSwitch == true` and production's `RootArmed`; an obligation 25 h past `FinishedAt`; two agent runs: `r.State == "finished"`, `len(fake.PutsAfter(mark, moves.OfflineActiveKey)) == 0` and `fake.LockPuts(location) == 0`; root's pass through `runAdoptUnderRootRelease`'s stages with the switch on and no drop anywhere: `calls[0] == "arm"`, before `ensureUnits`, and `gated` holds a nonce matching `^[0-9a-f]{32}$`; the agent's next run reserves, marks, deletes every key and reaches `publishing`, where `cleanupBeforeDropHook` pauses it before its first drop; the switch is unset and the agent restarts with `cfg.CleanupSwitch == false`, the paused worker discarded; the new worker's visit writes the drop; root's next pass with the switch off: `adoptRootGate` answers serve, `QuiesceRootCleanup` takes its census (none) and writes `quiesced` for that nonce, and root takes the request and acknowledges `done`, `release.LiveLink(optDir, host)` absent afterwards; the agent's next visit: `r.State == "releasing" && r.Outcome == "cleaned"`, the mark lifted, `release-withheld` owed; and with rev 10's arming (the agent armed from the switch alone) the first run reserves before `gated` exists, root's pass with the switch off publishes fail-open without serving, and the obligation stays `root-requested` for good); `TestRootArmsBeforeAnyPublisherWhateverTheObligations` (X2: markers absent, the switch on, no drop and no obligation on the box: in each of the four modes the pass's recording fakes log `arm` first, before `ensureUnits`, and `gated` exists afterwards; a second pass arms nothing and `gated` is byte for byte unchanged; `LockRootPass` answering `ErrNotRootOnly` → no arming and no marker; `ArmRootCleanup` failing with an injected `EIO` → one ALERT line, every stage still runs under the lock, and the next pass arms); `TestAgentReadOfGatedFailsClosedToUnarmed` (`RootCleanupArmed`: absent → `false, nil`; a symlink, a file of another owner, a group-writable file, a writable base → `false, nil`; an injected `EACCES` → `false` and the error; a root-owned regular file under a root-only base → `true, nil`, the last checked against an injected ownership check as any uid); `TestNothingRemovesTheGateMarkers` (W2: `go/ast` with `go/types` over the non-test files of `internal/release` and `internal/host`: the marker names are referenced only by `ArmRootCleanup`, `RootCleanupGated` and `QuiesceRootCleanup`, and no `os.Remove`, `os.RemoveAll`, `unix.Unlinkat` or `os.Rename` takes `gated` as its path, or `quiesced` as anything but a rename's target; and in behaviour, an armed box disarmed and driven through ten passes, with and without the root pass lock, and through root's 30-day removal of its `<id>/` directories, keeps both files byte for byte); `TestRootStepRefusesARootAgent` (V1: `unitUser` answering "", "root" and "0", and separately `carlos` with a fake `/proc` holding a uid-0 `<opt>/carlos agent`: `len(stepCalls) == 0`, `len(authorizeCalls) == 0`, `os.Lstat(quiesced)` answers `fs.ErrNotExist`, every drop and taken request is answered `cancelled` with `ack.Reason == "cleanup needs the agent to run as its own user"`, and one ALERT line; the agent's next visit reads the acknowledgement and goes to `releasing` with `Outcome == "cancelled"`; `unitUser` failing → nothing answered and nothing written this pass; `carlos` and no root agent → the step runs); `TestArmedPassLeavesTheGatedMarker` (an armed pass holding the lock creates the marker; a later pass with the switch off is gated; a marker `Lstat` error other than `ENOENT` → gated); `TestRootFindsItsOwnRequestsOnRestart` (root takes a request, removes the drop and stops: the next pass serves it from `request.json`); `TestForgedAgentFilesAreHarmless` (a `done.json` planted in the agent's directory is never read; a drop with another id is cancelled under the expected id; a second drop with other bytes after the take is removed and changes nothing; a forged cancel stops root and ends the obligation `cancelled`, nothing else); `TestRejectedAncestryGetsNoWrite` (a `RootCleanupBase` or `<id>/` that fails the chain check: nothing created, written or removed under it, the drop kept, one log line; repaired, the next pass takes it); `TestAgentPagesARefusedAncestryAndNeverReleases` (no answer for 1 h with the chain failing from the agent's view: hourly `root-refused` pages, the obligation stays `root-requested` with both entries and the mark for a simulated week; the drop removed by an operator → `releasing`, `superseded`); `TestAgentReadsOnlyRootsAcknowledgement` (an untrusted acknowledgement → `superseded`, paths kept "writable layout"); `TestRootRequestedAtIsRestored` (a crash after the drop: from its mtime; after the take: from the `taken` line); `TestUnansweredRootRequestPagesEveryTenMinutes`; `TestCancelBeforeAndDuringRootsWork`; `TestCancelPendingResolvesWithoutARequest` (a crash after `publishing` before the drop; root never wrote a directory); `TestUnsupportedPlatformCancels`. |
| +- [ ] **Step 1: Write the failing tests**: `TestRootCleanupStepRunsInsideThePass` (a pass driven through `runAdoptUnderRootRelease`'s stages with fakes: the step runs after `release.Pass` and `heldStagePass`; with the adopt-pass lock failed, with the root pass lock failed, and on `-config-only` and `-restart-only`, no step; every mode takes the root pass lock before its first publisher); `TestRootPassLockIsRootOnly` (a base directory the test uid owns → `ErrNotRootOnly`, no lock; the lock's chain check runs as any uid against an injected chain check); `TestLockRootPassChecksAncestryBeforeCreating` (T7: an `optDir` whose existing chain fails → `errors.Is(err, release.ErrNotRootOnly)` and `os.Lstat(base)` answers `fs.ErrNotExist` afterwards; a root-only `optDir` without a base → created, checked, locked, in that order in the injected syscall log; a base that is a symlink → `ErrNotRootOnly`, nothing created through it); `TestGatedPassPublishesNothingWithoutTheRootPassLock` (T2: a pass driven through `runAdoptUnderRootRelease`'s stages with a recording fake per stage; the switch on and `LockRootPass` failing with an injected `EIO` → `len(calls) == 0` across `ensureUnits`, `recoverRootRestarts`, `rootJudgePrev`, `sweepStagedVersionsPass`, the restart pass, config sync, `capabilityPass`, `linkRootHistory`, `release.Pass`, `heldStagePass` and the root step, in each of the four modes, and one log line; the switch off with the marker present → the same; the next pass with the lock → every stage in today's order and the step last); `TestNeverArmedBoxKeepsFailOpenAdoption` (the switch off and no marker, the lock failing → every stage runs as today, no step, no marker created; `ErrNotRootOnly` with the switch on → the same); `TestFirstArmingWaitsForAFailOpenPassToEnd` (U1's witness, review 7's finding 2 interleaving, through `runAdoptUnderRootRelease`'s stages with the production lock functions, an injected `LockRootPass` fault for one pass and an injected census: pass A, with the switch off, no marker, `/data/adopt-pass.lock` replaced by a symlink and `LockRootPass` failing with `EIO`, runs fail-open and is paused inside `release.Pass` after reading its route; the switch goes on and a drop is written; pass B takes both locks, arms the box (`gated` with a nonce), its agent check answers true, and its census answers A's pid: `QuiesceRootCleanup` answers false, `os.Lstat(quiesced)` answers `fs.ErrNotExist`, and root's step is not called, `len(stepCalls) == 0`; A resumes, publishes a new live link for the host and exits, and the census answers no pid; pass C writes `quiesced` and runs root's step, which finds the link A published, not the recorded one: `kept`, "identity changed", and `release.LiveLink(optDir, host)` still names A's target); `TestAdoptPassesCensus` (V1: a fake `/proc` tree whose `exe` entries are symlinks into a fake `optDir`: uid-0 processes running `adopt --stage-only` through `<opt>/carlos`, through `<opt>/carlos.prev`, through an `exe` that reads `<opt>/carlos (deleted)`, and through a hard link of `<opt>/carlos` at another path (`stat` names its inode) are each counted, and so is one whose `argv[0]` reads `x`; a uid-0 `adopt` through `/usr/local/bin/carlos`, a copy with another inode, is not (outside support), nor are `self`, a uid-1001 process through `<opt>/carlos` running `adopt`, a uid-0 `<opt>/carlos agent` and an entry removed during the walk; an unreadable `status` → the error); `TestStaleCertificateIsVoid` (V2's witness, forcing identity reuse: `gated` armed with nonce N1; beside it, in turn, a `quiesced` the test writes with nonce N0, and one in rev 8's form whose `gated_dev` and `gated_ino` equal the current `gated`'s `Lstat` identity and which has no nonce; with the injected census answering a pid, `QuiesceRootCleanup` answers false, `quiesced` is left byte for byte as the test wrote it (nothing removes it), and `len(stepCalls) == 0`; with the census then answering none, it is replaced by one with `nonce == N1` and the step runs; a `gated` that is a symlink, is owned by another uid, or holds `"nonce": "zz"` → false and nothing written); `TestArmingWritesAFreshNonce` (two armings with `gated` removed by hand between, the only way a box is armed twice: `n1 != n2`, each matching `^[0-9a-f]{32}$`; the injected syscall log of one arming is write temp, `fsync` temp, link `gated`, unlink temp, `fsync` base; two passes arming at once: one link wins, the other's `os.Link` answers `EEXIST`, and both then read the winner's nonce); `TestDisarmedPublicationIsStillServed` (W2's witness, review 9's finding 2 interleaving: an armed box, with `gated` and `quiesced` standing for nonce N; an obligation reaches `publishing` and `cleanupBeforeDropHook` pauses its visit immediately before its first drop; `CARLOS_OFFLINE_CLEANUP` is unset in the test's `host.env` and the agent restarts with `cfg.CleanupSwitch == false`, the paused worker discarded; the new worker's visit writes the drop and touches the stage-request marker; root's next pass, driven through `runAdoptUnderRootRelease`'s stages with the switch off: `adoptRootGate` answers serve, because `gated` stands, the agent check answers true, `QuiesceRootCleanup` answers true with no census call (`quiesced` names N), and root takes the request and acknowledges `done`, with `release.LiveLink(optDir, host)` absent afterwards; the agent's next visit reads the acknowledgement: `releasing`, `r.Outcome == "cleaned"`, the mark lifted, the release withheld with `release-withheld` owed, and `len(fake.PutsAfter(mark, moves.OfflineActiveKey)) == 0`; `gated` and `quiesced` are byte for byte what they were before the disarm); `TestDisarmAfterTheFirstArmingStillServesTheRequest` (X2's witness, review 10's finding 2 interleaving, with both markers absent at the start: `os.Lstat(gated)` and `os.Lstat(quiesced)` answer `fs.ErrNotExist`; `CARLOS_OFFLINE_CLEANUP=on` in the test's `host.env` and the agent restarted with `cfg.CleanupSwitch == true` and production's `RootArmed`; an obligation 25 h past `FinishedAt`; two agent runs: `r.State == "finished"`, `len(fake.PutsAfter(mark, moves.OfflineActiveKey)) == 0` and `fake.LockPuts(location) == 0`; root's pass through `runAdoptUnderRootRelease`'s stages with the switch on, no drop anywhere and a census answering one other adopt pass (`[]int{4242}`; this first pass is a full one, and with an empty census its `QuiesceRootCleanup` would write `quiesced` itself): `calls[0] == "arm"`, before `ensureUnits`, `gated` holds a nonce matching `^[0-9a-f]{32}$`, and `os.Lstat(quiesced)` still answers `fs.ErrNotExist`; the agent's next run reserves, marks, deletes every key and reaches `publishing`, where `cleanupBeforeDropHook` pauses it before its first drop; the switch is unset and the agent restarts with `cfg.CleanupSwitch == false`, the paused worker discarded; the new worker's visit writes the drop; root's next pass with the switch off: `adoptRootGate` answers serve, `QuiesceRootCleanup` takes its census (none) and writes `quiesced` for that nonce, and root takes the request and acknowledges `done`, `release.LiveLink(optDir, host)` absent afterwards; the agent's next visit: `r.State == "releasing" && r.Outcome == "cleaned"`, the mark lifted, `release-withheld` owed; and with rev 10's arming (the agent armed from the switch alone) the first run reserves before `gated` exists, root's pass with the switch off publishes fail-open without serving, and the obligation stays `root-requested` for good); `TestRootArmsBeforeAnyPublisherWhateverTheObligations` (X2: markers absent, the switch on, no drop and no obligation on the box: in each of the four modes the pass's recording fakes log `arm` first, before `ensureUnits`, and `gated` exists afterwards; a second pass arms nothing and `gated` is byte for byte unchanged; `LockRootPass` answering `ErrNotRootOnly` → no arming and no marker; `ArmRootCleanup` failing with an injected `EIO` → one ALERT line, every stage still runs under the lock, and the next pass arms); `TestAgentReadOfGatedFailsClosedToUnarmed` (`RootCleanupArmed`: absent → `false, nil`; a symlink, a file of another owner, a group-writable file, a writable base → `false, nil`; an injected `EACCES` → `false` and the error; a root-owned regular file under a root-only base → `true, nil`, the last checked against an injected ownership check as any uid); `TestNothingRemovesTheGateMarkers` (W2: `go/ast` with `go/types` over the non-test files of `internal/release` and `internal/host`: the marker names are referenced only by `ArmRootCleanup`, `RootCleanupGated` and `QuiesceRootCleanup`, and no `os.Remove`, `os.RemoveAll`, `unix.Unlinkat` or `os.Rename` takes `gated` as its path, or `quiesced` as anything but a rename's target; and in behaviour, an armed box disarmed and driven through ten passes, with and without the root pass lock, and through root's 30-day removal of its `<id>/` directories, keeps both files byte for byte); `TestRootStepRefusesARootAgent` (V1: `unitUser` answering "", "root" and "0", and separately `carlos` with a fake `/proc` holding a uid-0 `<opt>/carlos agent`: `len(stepCalls) == 0`, `len(authorizeCalls) == 0`, `os.Lstat(quiesced)` answers `fs.ErrNotExist`, every drop and taken request is answered `cancelled` with `ack.Reason == "cleanup needs the agent to run as its own user"`, and one ALERT line; the agent's next visit reads the acknowledgement and goes to `releasing` with `Outcome == "cancelled"`; `unitUser` failing → nothing answered and nothing written this pass; `carlos` and no root agent → the step runs); `TestArmedPassLeavesTheGatedMarker` (an armed pass holding the lock creates the marker; a later pass with the switch off is gated; a marker `Lstat` error other than `ENOENT` → gated); `TestRootFindsItsOwnRequestsOnRestart` (root takes a request, removes the drop and stops: the next pass serves it from `request.json`); `TestForgedAgentFilesAreHarmless` (a `done.json` planted in the agent's directory is never read; a drop with another id is cancelled under the expected id; a second drop with other bytes after the take is removed and changes nothing; a forged cancel stops root and ends the obligation `cancelled`, nothing else); `TestRejectedAncestryGetsNoWrite` (a `RootCleanupBase` or `<id>/` that fails the chain check: nothing created, written or removed under it, the drop kept, one log line; repaired, the next pass takes it); `TestAgentPagesARefusedAncestryAndNeverReleases` (no answer for 1 h with the chain failing from the agent's view: hourly `root-refused` pages, the obligation stays `root-requested` with both entries and the mark for a simulated week; the drop removed by an operator → `releasing`, `superseded`); `TestAgentReadsOnlyRootsAcknowledgement` (an untrusted acknowledgement → `superseded`, paths kept "writable layout"); `TestRootRequestedAtIsRestored` (a crash after the drop: from its mtime; after the take: from the `taken` line); `TestUnansweredRootRequestPagesEveryTenMinutes`; `TestCancelBeforeAndDuringRootsWork`; `TestCancelPendingResolvesWithoutARequest` (a crash after `publishing` before the drop; root never wrote a directory); `TestUnsupportedPlatformCancels`. |
| - [ ] **Step 2: Run to verify they fail**: `go test ./internal/release -run 'RootFinds|ForgedAgent|RejectedAncestry|RootPassLock|LockRootPassChecks|CancelBefore|Unsupported' -count=1`, `go test ./internal/release -run 'AdoptPassesCensus|StaleCertificate|ArmingWrites|NothingRemovesTheGate|AgentReadOfGated' -count=1`, `go test ./internal/host -run 'RootCleanupStep|GatedPass|NeverArmedBox|ArmedPassLeaves|FirstArming|DisarmedPublication|DisarmAfterTheFirst|RootArmsBefore|RootStepRefuses|AgentPages|AgentReadsOnly|RootRequestedAt|Unanswered|CancelPending' -count=1`. |
| - [ ] **Step 3: Implement** (until Task 6c lands, the per-path step answers kept with reason "not built", so this commit removes nothing). **Step 4: Run to verify they pass**, then `GOOS=darwin go build ./internal/release/ ./internal/host/` and `GOOS=windows go vet ./internal/release/ ./internal/host/`. |
| - [ ] **Step 5: Commit** `root: a stage of the adopt pass under a root-only pass lock, which an armed box's passes must hold to publish, arms the box before any publisher of its first pass after the switch, runs only beside an agent that is not root and first runs once no adopt process of the installed binary that predates this arming's nonce is alive, keeps its gate markers for the life of the box, takes a cleanup request into its own directory, writes nothing where the ancestry is not root's, and answers in a file the agent trusts only on a root-only chain`. |
| @@ -342,7 +344,7 @@ Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and |
| **Interfaces:** |
| - **The journal** `<RootCleanupDir>/journal`: append-only JSON lines, each `fsync`ed before the next syscall: `{at, step: "taken"}`; per path `{at, which, step: "intent", identity}` then exactly one of `{at, which, step: "removed"}`, `{at, which, step: "kept", reason}`, `{at, which, step: "absent"}`. A path's result is its last result line; the acknowledgement is built from results alone, so no outcome is unknown. |
| - **Recovery (F6),** per path whose last line is `intent`: `fstatat` the name in the opened parent again: absent → journal `absent` (removed by us before the crash or by anyone: recorded as absent, never as removed); present with the recorded identity → the step is retried from the start below; present with another identity → journal `kept`, "identity changed". Nothing was ever renamed, so nothing needs restoring after a cancel or a refusal. |
| -- **The step,** per path not yet terminal (`live-link` first; root derives the path: `LiveLink(OptDir, host)`, `ProgramsPath(DataDir, host)`): for the live link only, outside the freeze: the build directory it targets; another link under `<OptDir>/live/` resolving to the same build → `kept`, "the build is still referenced"; `o.Processes()` naming an executable under the build → `kept`, "a process runs the build"; `Processes` failing → the next pass (unknown is never unreferenced). The programs file has no reference check: its schema (`{schema, programs}`, Go main-package paths, `programhistory.go:28-34`) belongs to this host alone and no reader resolves another host through it. Then `o.Authorize` again (the same two answers). Then `RootOnlyChain(parent)` false → `kept`, "writable layout". Then `o.Freeze(host, func(rowPresent) { rowPresent → kept, "the instance has a row on this box"; o.HostLock(host, func() { root-cleanup.cancel present (Lstat) → kept, "cancelled"; open the parent (O_DIRECTORY|O_NOFOLLOW) and fstat it: not uid 0 or group/other writable → kept, "writable layout"; journal intent; fstatat(parent, name, AT_SYMLINK_NOFOLLOW): ENOENT → journal absent; identity (dev, inode, file type, readlinkat target) differs → journal kept, "identity changed"; equal → unlinkat(parent, name, 0): ENOENT → journal absent; success → fsync the parent, journal removed }) })`. Between `fstatat` and `unlinkat` only root can change an entry of a root-only directory, and every root writer of these names is excluded: the root adoption publishers by the pass (this step is a stage of it, and every other adopt pass waits on the root pass lock), held publication by the held-stage lock, and anything that needs a row by the freeze. |
| +- **The step,** per path not yet terminal (`live-link` first; root derives the path: `LiveLink(OptDir, host)`, `ProgramsPath(DataDir, host)`): for the live link only, outside the freeze: the build directory it targets; another link under `<OptDir>/live/` resolving to the same build → `kept`, "the build is still referenced"; `o.Processes()` naming an executable under the build → `kept`, "a process runs the build"; `Processes` failing → the next pass (unknown is never unreferenced). The programs file has no reference check: its schema (`{schema, programs}`, Go main-package paths, `programhistory.go:28-34`) belongs to this host alone and no reader resolves another host through it. Then `o.Authorize` again (the same two answers). Then `RootOnlyChain(parent)` false → `kept`, "writable layout". Then `o.Freeze(host, func(rowPresent) { rowPresent → kept, "the instance has a row on this box"; o.HostLock(host, func() { root-cleanup.cancel present (Lstat) → kept, "cancelled"; open the parent (O_DIRECTORY|O_NOFOLLOW) and `offlinecleanup.RootOnlyDir(parent)` (its fstat: not uid 0 or group/other writable) false → kept, "writable layout"; journal intent; fstatat(parent, name, AT_SYMLINK_NOFOLLOW): ENOENT → journal absent; identity (dev, inode, file type, readlinkat target) differs → journal kept, "identity changed"; equal → unlinkat(parent, name, 0): ENOENT → journal absent; success → fsync the parent, journal removed }) })`. Between `fstatat` and `unlinkat` only root can change an entry of a root-only directory, and every root writer of these names is excluded: the root adoption publishers by the pass (this step is a stage of it, and every other adopt pass waits on the root pass lock), held publication by the held-stage lock, and anything that needs a row by the freeze. |
| - **The writers of these names** (`TestRootOnlyLiveLinkWritersAreExcluded`, `go/ast` with `go/types` over non-test files under `./internal/...`): every call of `release.PointLive`, `release.RecordProgram` and `(*activator.ExecBackend).ensureLiveLink` is listed with what keeps it off the name during root's step: `root pass lock` (reached only from `runAdoptUnderRootRelease`'s stages after `adoptRootGate`, which on a gated box runs them only while the pass holds the root pass lock for its whole run, and runs root's step as a later stage: `adopt.go:1751`, `:1870`, `rollback.go:124`, `rootrollback_resume.go:125`, `roothistory_recovery.go:216`, and named recovery's `namedrecovery.go:120` and `namedrecovery_active.go:191` through `recovery.passOption` inside `release.Pass`, `agent.go:4890`, `namedrecovery_pass.go:15`), `held-stage lock` (held publication: `materializeheld.go:460`, `:494`), `needs a row` (the freeze), `agent process` (the agent, which cleanup requires to run as its own user, cannot write a root-only directory: `ensureLiveLink`, the agent's own adopt passes at `agent.go:1326`, `:2652` and `:2852`; on a box whose agent runs as root no root step runs, Task 6b's agent check) or `e2e only`; the test also proves each `root pass lock` site's caller chain reaches only `runAdoptUnderRootRelease`. A new site, or a removed one, fails until the list is edited, and a site the implementer cannot place in a class is a question for the reviewer, not a guess. |
| - [ ] **Step 1: Write the failing tests**: `TestRootRemovesOnlyTheRecordedIdentity` (Review Focus 3: a publication replaces the link between the reference check and the `fstatat` → kept, the replacement at its name; the recorded entry → removed); `TestNoPublisherRunsBetweenFstatatAndUnlinkat` (the S12 witness: a hook between `fstatat` and `unlinkat` starts a second adopt pass through the production lock functions, with `/data/adopt-pass.lock` replaced by a symlink so its adopt-pass lock fails, and a held publication of the same host: the second pass blocks in `LockRootPass` and the held publication in the held-stage lock until the step returns; the recorded entry is removed, and the second pass's publication lands after it; and, separately on the gated box, a second pass whose `LockRootPass` fails with an injected fault while the step runs: it publishes nothing, `len(calls) == 0`, and the recorded entry is removed with no replacement at its name); `TestWritableLayoutIsNeverRemoved` (a parent the test uid owns; an ancestor writable by group: kept "writable layout", nothing unlinked, no intent line); `TestRootRowCheckIsAtomicWithTheUnlink` (an `Upsert` issued during the freeze returns after it; a row present at the freeze → kept); `TestRootJournalResolvesEveryIntent` (a crash after each journal line and after the `unlinkat`: each restart ends in removed, kept or absent; a crash after the unlink and before `removed` → `absent`); `TestReferencedBuildIsKept` (another host's link to the build; a process; `Processes` failing → left); `TestProgramsFileHasNoReferenceCheck`; `TestRootOnlyLiveLinkWritersAreExcluded`. The Linux tests that need a root-owned parent run when the test runs as root (the root runner, `root-runner.py`) and are skipped otherwise; the identity, journal and lock-order logic run as any uid against an injected chain check. |
| @@ -422,19 +424,19 @@ Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and |
| **Files:** |
| - Modify: `internal/e2e/selfserve/job_world_test.go` (`faultStore.PutOnce`, `faultStore.write`, `heldPut.once`, `faultStore.release`) |
| - Create: `internal/e2e/selfserve/cleanup_world_test.go` (`boxStore`, `cleanupWorld`, `newCleanupWorld`, and the fixture's own checks) |
| -- Modify: `internal/host/e2e_bridge.go` (`fakeWakeBackend.eb`, `E2ERealWake`, `landOpts.realWake`, `offBox.restart` and `landWorld.arm`, `newE2EEnv` returning the holder, `E2E.holder` and `(*E2E).Holder`, `NewE2E` setting the trust root, `E2ERows`) |
| +- Modify: `internal/host/e2e_bridge.go` (`fakeWakeBackend.eb`, `offBox.inst` built in `offWorld.box`, `E2ERealWake`, `landOpts.realWake`, `offBox.restart` and `landWorld.arm`, each shutting the previous activator down, `newE2EEnv` returning the holder, `E2E.holder` and `(*E2E).Holder`, `NewE2E` setting the trust root and shutting the activators down at teardown, `E2ERows`) |
| - Create: `internal/host/e2e_cleanup_bridge.go` (the `BoxHandle` cleanup controls below) |
| - Modify: `internal/lease/leasetest/hold.go` (`HoldDeletes`; `HeldRequest.Method` and `VersionID`) |
| -- Modify: `internal/offlinecleanup/layout_linux.go` (`SetTrustForE2E`), `internal/host/movespass.go` (`moveCfg.CleanupManual`), `internal/host/movescleanup.go` (`kickCleanup` starts nothing when `cfg.CleanupManual`) |
| +- Modify: `internal/offlinecleanup/layout_linux.go` and `layout_other.go` (`SetTrustForE2E`), `internal/host/movespass.go` (`moveCfg.CleanupManual`), `internal/host/movescleanup.go` (`kickCleanup` starts nothing when `cfg.CleanupManual`) |
| - Test: `internal/e2e/selfserve/cleanup_world_test.go`, `internal/host/e2e_cleanup_bridge_test.go`, `internal/offlinecleanup/trust_test.go`, `internal/lease/leasetest/hold_test.go` |
| **Interfaces:** |
| -- Consumes: `release.PutOnce` (Task 2a), `leasetest.Holder` (Task 3), `cfg.CleanupSwitch`, `cfg.RootArmed`, `cleanupRun` (Task 4d), `enterRootGate`, `rootCleanupPass`, `rootStage`, `cleanupBeforeDropHook` (Task 6b), `rootCleanupAuthority` (Task 6a), `offlinecleanup.RootOnlyChain` and `RootOwned` (Task 1). |
| +- Consumes: `release.PutOnce` (Task 2a), `leasetest.Holder` (Task 3), `cfg.CleanupSwitch`, `cfg.RootArmed`, `cleanupRun` (Task 4d), `enterRootGate`, `rootCleanupPass`, `rootStage`, `cleanupBeforeDropHook` (Task 6b), `rootCleanupAuthority` (Task 6a), `offlinecleanup.RootOnlyChain`, `RootOnlyDir` and `RootOwned` (Task 1). |
| - **`faultStore.PutOnce` (review 10, finding 3, first mismatch).** `faultStore` embeds `release.Store` (`job_world_test.go:37-39`), so `DirStore.PutOnce` is not promoted through it, and `release.PutOnce` would answer `ErrNoSingleAttemptPut` for every table write the console makes from Task 2a on. `Put` and the new `func (f *faultStore) PutOnce(ctx context.Context, key string, body []byte, ifMatch string) (string, error)` become one `f.write(ctx, key, body, ifMatch, once bool)`: the same `seen` hook, serial count, `put` decision and `failing` rule for record keys, and the same held, lost and failed answers. A write that runs goes to `release.PutOnce(ctx, f.Store, key, body, ifMatch)` when `once`, else to `f.Store.Put`, so a store under it without `PutOnce` still answers `ErrNoSingleAttemptPut`. `heldPut` gains `once bool`, and `release` runs each held write through the method it arrived by. Nothing else about the fixture changes, so plan 2's and plan 3's witnesses over it are unchanged. |
| - **`boxStore`** (`cleanup_world_test.go`): the boxes' deployment store, `raw` under a recorder that forwards `Put`, `PutOnce` (through `release.PutOnce`, so the boxes keep `DirStore`'s single attempts), `Get` and `DeleteIfMatch`, and records every write of `moves.OfflineActiveKey` with the world's mark sequence. `w.mark() int`; `w.boxTableWritesSince(m int) []string` (from `boxStore`) and `w.consoleTableWritesSince(m int) []string` (from `faultStore.seen`) count the table writes by writer. `newCleanupWorld(t *testing.T) *cleanupWorld` is `newJobWorld`'s construction (`job_world_test.go:296-330`) with `host.NewE2E(t, boxStore, host.E2ELifetime(), host.E2ERealWake())`. |
| -- **A real wake (review 10, finding 3, second mismatch).** `fakeWakeBackend.Acquire` answers `nopLease{}` without reading the bucket (`e2e_bridge.go:689-691`) and `Restore` answers without reading it (`:693`), so no witness over the bridge could see a mark refuse a wake, a lease settle, or a restore read what the cleanup left. `fakeWakeBackend` gains `eb *activator.ExecBackend`; when it is set, `Acquire` is `b.eb.Acquire(ctx, rt)` and `Restore` is `b.eb.Restore(ctx, rt)`, and everything else (`Start`, `NextDue`, sidecars, workers) stays the fake instance. The `ExecBackend` is the one the host's own offline witnesses take the source's lease with (`internal/host/movesoffline_e2e_test.go:142`, `:234`): `&activator.ExecBackend{Cfg: activator.Config{Bucket: bucket, Region: region, Prefix: sealPrefix}, DataDir: b.dd, OptDir: b.opt}`, over the box's own hibernate bucket on the loopback S3 server. So a wake under a mark meets Task 3's refusal in the real `lease.AcquireTTL` and gets the real `serveWakeError` answer (503, `Retry-After: 30`); an acquisition writes a real lease to `lock.json`, which `execLease.Release` deletes through litestream's `ReleaseLease` (`s3/leaser.go:179-192`); and the restore reads the replica the bucket holds, promoting a staged replica through the real `promoteStagedReplica`. `func E2ERealWake() E2EOption` sets `landOpts.realWake`; `offBox.restart` (`e2e_bridge.go:190-191`) and `landWorld.arm` (`:476`) build `&fakeWakeBackend{eb: ...}` when it is set; without the option every bridge is as today. `const E2ERows = e2eRows` (`e2e_bridge.go:749`, 30) names the rows the source holds. |
| +- **A real wake (review 10, finding 3, second mismatch; review 11, item 3).** `fakeWakeBackend.Acquire` answers `nopLease{}` without reading the bucket (`e2e_bridge.go:689-691`) and `Restore` answers without reading it (`:693`), so no witness over the bridge could see a mark refuse a wake, a lease settle, or a restore read what the cleanup left. `fakeWakeBackend` gains `eb *activator.ExecBackend`; when it is set, `Acquire` is `b.eb.Acquire(ctx, rt)` and `Restore` is `b.eb.Restore(ctx, rt)`, and everything else (`Start`, `NextDue`, sidecars, workers) stays the fake instance. **The backend.** It is the one the host's own offline witnesses take the source's lease with (`internal/host/movesoffline_e2e_test.go:142`, `:234`), plus the box's mirror and the legacy promotion mode: `&activator.ExecBackend{Cfg: activator.Config{Bucket: bucket, Region: region, Prefix: sealPrefix}, DataDir: b.dd, OptDir: b.opt, Mirror: m, Phase1: false}`. `bucket, region` are the box's `CARLOS_HIBERNATE_BUCKET` and `CARLOS_HIBERNATE_REGION`, read as `landWorld.arm` reads them (`e2e_bridge.go:475`), and `m, err := lease.NewS3Mirror(ctx, region, region)` is the box's own same-region mirror client, built as the agent builds it (`agent.go:586`) and reaching the loopback S3 server as `sealWorld.arm`'s mirror does (`e2e_bridge.go:299-301`); an error fails the test. `Phase1: false` is the mode of a box whose Phase 1 switch is off (`armPhase1` is the field's only writer, `movespass.go:351`, `:722`), so `restorePlain`'s `b.Mirror != nil && !b.Phase1` (`backend_exec.go:817`) holds: the restore runs `promoteStagedReplica` (`:3397`, which lists `activator.MovesPrefix(host)`) before `restoreReplica`, and `dropStagedReplica` (`:3437`) once `lease.CheckDB` vouches for what it restored. So a wake under a mark meets Task 3's refusal in the real `lease.AcquireTTL` and gets the real `serveWakeError` answer (503, `Retry-After: 30`); an acquisition writes a real lease to `lock.json`, which `execLease.Release` deletes through litestream's `ReleaseLease` (`s3/leaser.go:179-192`); and after a legacy return has staged the host's replica under `moves/<host>/`, the wake's restore promotes it and reads it, so Task 9b's promotion witnesses (`TestPromotionBackIntoTheRegionKeepsItsData`, `TestLegacyMoveIntoTheSourceRegionIsTolerated`, `TestCleanupHoldsTheMarkUntilItsDeleteIsAnswered`) observe the real promotion. **The instance.** `Start` dereferences `b.inst` (`e2e_bridge.go:697-701`), and both bridge constructors build `&fakeWakeBackend{}` today, so a wake that acquired and restored would panic there. `offBox` gains `inst *fakeInstance`, built once per box under the option in `offWorld.box` (`e2e_bridge.go:165-167`): `&fakeInstance{socket: filepath.Join(sock, "w.sock"), handler: http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { w.WriteHeader(http.StatusNoContent) })}`, with `sock` from `os.MkdirTemp("", "cw")`, removed in `t.Cleanup`, so the unix socket path stays under its 108-byte limit whatever the test's name. `offBox.restart` (`e2e_bridge.go:190-191`) and `landWorld.arm` (`:476`) build `&fakeWakeBackend{inst: b.inst, eb: ...}` under the option. `Start` makes the instance listen on its socket; `fakeInstanceProc.Stop` closes its server, which removes the socket, so the next wake listens again. No request reaches the handler: the activator proxies to the handler its caller passes (`Service.ServeHTTP`, `activator.go:5296`), and `Wake` passes its own. **Stop, release and teardown.** A witness stops an awake instance with `BoxHandle.Sleep` (below), which ends in the activator's own stop path, so the lease is released by replication's close. Under the option, `offBox.restart` and `landWorld.arm` first shut the box's previous activator down, `b.svc.Shutdown(ctx)` under a 30 s budget, as an agent going down takes its instances with it (`activator.go:5108-5112`), so no restart leaves an instance holding the old lease; and `NewE2E` registers a `t.Cleanup` that shuts every box's current activator down the same way. It is registered after `newE2EEnv`'s `s3srv.Close` (`e2e_bridge.go:767`) and so runs before it, and an instance still awake at the end releases its lease while the S3 server answers. `func E2ERealWake() E2EOption` sets `landOpts.realWake`; without the option every bridge is as today (`&fakeWakeBackend{}`, no `inst`, no shutdown). `const E2ERows = e2eRows` (`e2e_bridge.go:749`, 30) names the rows the source holds. |
| - **Held requests.** `newE2EEnv` (`e2e_bridge.go:753-775`) serves `leasetest.NewHolder(h)` in place of `h` and returns it with the bucket model; `NewE2E` keeps it, `func (e *E2E) Holder() *leasetest.Holder`. `leasetest.Holder` gains `func (h *Holder) HoldDeletes(key string, n int) (release func())`, which holds the next `n` `DELETE`s of `key` as `HoldPuts` holds `PUT`s, and every `HeldRequest` gains `Method string` and `VersionID string` (the request's `versionId`). Each `release` lets through only the requests its own call held, in arrival order. |
| -- **The trust root.** `offlinecleanup.RootOnlyChain` and `RootOwned` (Task 1) answer for uid 0 from `/`. The bridge runs as the test's uid on temporary directories, where both answer false, so at `gone` every path would be `LocalLeft` and root's step could never run. `func SetTrustForE2E(uid int, bases ...string) (restore func())` (`layout_linux.go`, one package variable under a mutex): while it is set, a directory under one of `bases` is root-only when every component from that base down to it is a directory owned by `uid` with `mode & 0o022 == 0`, and a file under one of `bases` owned by `uid` with neither bit is root's; outside every base the answers are unchanged. `NewE2E` sets it with the test's uid and the two boxes' opt directories only, and restores it in `t.Cleanup`. Each box's data directory stays outside the trust root, as `/data` is the agent's on every built box, so the programs file is still `LocalLeft` "writable layout". `TestTrustForE2EIsSetOnlyByTheBridge` (`go/ast` with `go/types` over the non-test files under `./...`): the one call of `SetTrustForE2E` is in `NewE2E`, and no package under `cmd/`, and not the root package, reaches `NewE2E`. |
| +- **The trust root (review 11, item 4).** `offlinecleanup.RootOnlyChain`, `RootOnlyDir` and `RootOwned` (Task 1) answer for uid 0 from `/`. The bridge runs as the test's uid on temporary directories, where all three answer false, so at `gone` every path would be `LocalLeft` and root's step could never run. `func SetTrustForE2E(uid int, bases ...string) (restore func())` (`layout_linux.go`): one package variable under a mutex holding `uid` and the bases, each absolute and `filepath.Clean`ed. A place is inside a base when its path is the base or lies below it (`filepath.Rel` answers without a leading `..`, so `<base>x` is outside). While the variable is set, and only inside a base: `RootOnlyChain(dir)` holds when every component from the base down to `dir` is a directory (a symlink component answers false) owned by `uid` with `mode & 0o022 == 0`; `RootOnlyDir(parent)` accepts `uid` as the owner when `filepath.Clean(parent.Name())` is inside a base; and `RootOwned(parent, name)` accepts `uid` for the entry on the same condition, after `RootOnlyDir(parent)` has held. Every place outside every base gets the answer it gets without the trust root (uid 0 only), so a test-owned file outside the bases is still refused: the scope is decided by where a check looks, never by the uid alone. `layout_other.go` declares the same function for every other OS: it records nothing and returns a no-op restore, and the checks keep answering false, so `internal/host/e2e_bridge.go`, which is portable and calls it, builds for darwin and windows. `NewE2E` sets it with the test's uid and the two boxes' opt directories only, and restores it in `t.Cleanup`. Each box's data directory stays outside the trust root, as `/data` is the agent's on every built box, so the programs file is still `LocalLeft` "writable layout". `TestTrustForE2EIsSetOnlyByTheBridge` (`go/ast` with `go/types` over the non-test files under `./...`): the one call of `SetTrustForE2E` is in `NewE2E`, and no package under `cmd/`, and not the root package, reaches `NewE2E`. |
| - **The cleanup controls** (`e2e_cleanup_bridge.go`, methods of `BoxHandle`; each runs on the caller's goroutine): |
| - `func (h BoxHandle) SetCleanup(on bool)`: `CARLOS_OFFLINE_CLEANUP` set to `on` in the box's environment, or removed, then an agent restart (`e.rearm`), so `armOffline` reads it as production does. |
| - `func (h BoxHandle) SetAgentRunsAsRoot(root bool) (restore func())`: the `agentRunsAsRoot` seam (`agent.go:2206`), then an agent restart. |
| @@ -444,10 +446,11 @@ Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and |
| - `func (h BoxHandle) RootPass(ctx context.Context, c RootPassControls) (RootPassResult, error)`: root's adopt pass as far as cleanup sees it, over this box's opt and data directories: `enterRootGate(opt, on)` with `on` read from the box's environment (the file root's units read in production), then, when the gate serves, `rootCleanupPass(ctx, rootStage{...})` with Task 6a's authority over the deployment store, the box's judge, box id, region and hibernation settings (`Hib` returns `&activator.Config{Bucket, Region, Prefix: sealPrefix}` from the box's environment), the box's registry and held-stage lock, and `c.Census` (nil: no other adopt pass) and `c.AgentCheck` (nil: the agent runs as its own user). The real pass's publisher stages are not run, because the bridge's boxes publish through their own ticks. `type RootPassControls struct { Census func() ([]int, error); AgentCheck func(context.Context) (bool, string, error); LockFault error }` (`LockFault`, when set, is what `LockRootPass` answers instead of taking the lock); `type RootPassResult struct { Gate string; Armed, Quiesced bool; Acks []offlinecleanup.RootCleanupAck }` (`Gate` is `serve`, `publish-nothing` or `fail-open`; `Armed`: this pass wrote `gated`; `Acks`: the acknowledgements this pass wrote). |
| - `func (h BoxHandle) Markers() (gated, quiesced []byte, err error)`: the two files' bytes, nil when absent. |
| - `func (h BoxHandle) Wake(ctx context.Context, host string) WakeAnswer`: one HTTP request for `host` through the box's activator (`b.svc.ServeHTTP` with an `httptest.ResponseRecorder`, the host's registry row and a proxy that answers 200 "served"); `type WakeAnswer struct { Code int; RetryAfter, Body string }`. `func (h BoxHandle) DBRows(host string) (int, error)`: the rows of the database at the host's `rt.DBPath`, which the last wake restored, counted by the bridge's own query (`countRows` in `movesoffline_seal_test.go:98` is test-only and does the same). |
| + - `func (h BoxHandle) Sleep(ctx context.Context, host string) error`: the stop half of a wake cycle. `b.svc.Bounce(host)`, then a wait (10 ms polls; `ctx`'s error when it ends first) for `b.svc.Quiesced(host)`, which holds once `finishStop` has stopped the instance, closed replication (the final sync and the lease release) and deleted the entry (`activator.go:4339-4373`). |
| -- [ ] **Step 1: Write the failing tests**: `TestFaultStorePutOnceKeepsItsFaults` (a record key armed `putHeld` and written through `PutOnce` answers `context.DeadlineExceeded`, is held, and `release` runs it through `PutOnce`; `seen` sees the console's table write; the console's first `moves.PutTable` over `faultStore` answers without `errors.Is(err, release.ErrNoSingleAttemptPut)`; a `faultStore` over a store without `PutOnce` → `errors.Is(err, release.ErrNoSingleAttemptPut)` and nothing written); `TestBoxStoreCountsTableWritesByWriter` (one box `Reserve` and one console `reserveBoxes` after `m := w.mark()`: `len(w.boxTableWritesSince(m)) == 1 && len(w.consoleTableWritesSince(m)) == 1`); `TestBridgeWakeReadsTheRealLock` (with `E2ERealWake`: a mark written by `lease.MarkCleaning` over the source's lifted tombstone → `ans.Code == 503 && ans.RetryAfter == "30"` and `ans.Body` begins "instance is being cleaned up after a move. retry shortly"; after `UnmarkCleaning` → `ans.Code == 200`, `lock.json` holds a lease of the box while the instance runs, and after its release `LockState` answers owner "" and a zero expiry; `h.DBRows(host) == host.E2ERows`; without the option the wake answers 200 whatever the lock says, as today); `TestBridgeRootPassArmsAndServes` (markers absent and the switch on: `res.Armed && res.Gate == "serve"`, `gated` matching `"nonce": "[0-9a-f]{32}"`; a second pass: `!res.Armed`; `LockFault: io.ErrUnexpectedEOF` with `gated` present → `res.Gate == "publish-nothing"`; the switch off with no marker → `res.Gate == "serve" && !res.Armed`, and no file under `<opt>/root-cleanup/` but `pass.lock`); `TestBridgeCleanupRunsOnlyWhenAsked` (with `cfg.CleanupManual` a box tick starts no cleanup run; `h.Cleanup(ctx)` runs one); `TestTrustForE2EIsSetOnlyByTheBridge`; `TestTrustForE2EStaysInsideItsBases` (`RootOnlyChain` of a directory outside every base, and `RootOwned` of a file there, answer as without the trust root; after `restore()` a base answers false again); `TestHolderHoldsDeletesByVersion` (`HoldDeletes(key, 1)`: the held request's `Method == http.MethodDelete` and `VersionID` the one sent; `release()` delivers it to the bucket, which applies it then). |
| +- [ ] **Step 1: Write the failing tests**: `TestFaultStorePutOnceKeepsItsFaults` (a record key armed `putHeld` and written through `PutOnce` answers `context.DeadlineExceeded`, is held, and `release` runs it through `PutOnce`; `seen` sees the console's table write; the console's first `moves.PutTable` over `faultStore` answers without `errors.Is(err, release.ErrNoSingleAttemptPut)`; a `faultStore` over a store without `PutOnce` → `errors.Is(err, release.ErrNoSingleAttemptPut)` and nothing written); `TestBoxStoreCountsTableWritesByWriter` (one box `Reserve` and one console `reserveBoxes` after `m := w.mark()`: `len(w.boxTableWritesSince(m)) == 1 && len(w.consoleTableWritesSince(m)) == 1`); `TestBridgeWakeReadsTheRealLock` (with `E2ERealWake`: a mark written by `lease.MarkCleaning` over the source's lifted tombstone → `ans.Code == 503 && ans.RetryAfter == "30"` and `ans.Body` begins "instance is being cleaned up after a move. retry shortly"; after `UnmarkCleaning` → `ans.Code == 200` with the fake instance listening on its socket, `lock.json` holds a lease of the box while the instance runs, and after `h.Sleep(ctx, host)` the socket is gone and `LockState` answers owner "" and a zero expiry; a second wake answers 200 again; `h.DBRows(host) == host.E2ERows`; the box's backend has `Mirror != nil && !Phase1`, and with no database file on the box and the source's replica copied under `activator.MovesPrefix(host)`, a wake restores `host.E2ERows` rows and `MovesPrefix(host)` lists nothing afterwards; without the option the wake answers 200 whatever the lock says, as today); `TestBridgeRootPassArmsAndServes` (markers absent and the switch on: `res.Armed && res.Gate == "serve"`, `gated` matching `"nonce": "[0-9a-f]{32}"`; a second pass: `!res.Armed`; `LockFault: io.ErrUnexpectedEOF` with `gated` present → `res.Gate == "publish-nothing"`; the switch off with no marker → `res.Gate == "serve" && !res.Armed`, and no file under `<opt>/root-cleanup/` but `pass.lock`); `TestBridgeCleanupRunsOnlyWhenAsked` (with `cfg.CleanupManual` a box tick starts no cleanup run; `h.Cleanup(ctx)` runs one); `TestTrustForE2EIsSetOnlyByTheBridge`; `TestTrustForE2EStaysInsideItsBases` (`RootOnlyChain` of a test-owned directory outside every base, `RootOnlyDir` of it opened, and `RootOwned` of a test-owned file under it, answer false as without the trust root, while the same shapes inside a base answer true; a sibling `<base>x` is outside; after `restore()` a base answers false again); `TestHolderHoldsDeletesByVersion` (`HoldDeletes(key, 1)`: the held request's `Method == http.MethodDelete` and `VersionID` the one sent; `release()` delivers it to the bucket, which applies it then). |
| - [ ] **Step 2: Run to verify they fail**: `go test -tags e2e ./internal/e2e/selfserve -run 'FaultStorePutOnce|BoxStoreCounts' -count=1`, `go test ./internal/host -run 'BridgeWake|BridgeRootPass|BridgeCleanupRuns' -count=1`, `go test ./internal/offlinecleanup -run 'TrustForE2E' -count=1`, `go test ./internal/lease/leasetest -run 'HolderHolds' -count=1`. |
| -- [ ] **Step 3: Implement. Step 4: Run to verify they pass**, plus `go test -tags e2e ./internal/e2e/selfserve -count=1` (plan 2's and plan 3's witnesses green over the changed fixtures). |
| +- [ ] **Step 3: Implement. Step 4: Run to verify they pass**, plus `go test -tags e2e ./internal/e2e/selfserve -count=1` (plan 2's and plan 3's witnesses green over the changed fixtures), and `GOOS=darwin go build ./internal/host/... ./internal/offlinecleanup/...` and the same with `GOOS=windows` (the bridge's call of `SetTrustForE2E` compiles against `layout_other.go`). |
| - [ ] **Step 5: Commit** `e2e: fixtures for the cleanup witnesses: a fault-preserving single-attempt write, a wake whose lease and restore are real, held requests at the bridge's S3 server, a trust root only the bridge sets, and controls for the switch, the agent's uid, a cleanup run, root's pass and the markers`. |
| ### Task 9b: End to end and the census |
| @@ -475,7 +478,7 @@ Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and |
| - `TestDisarmEndsObligationsWithoutTableWrites` (plan 4's rollback, V3 and W3: cleanups on both boxes, one `cleaning` with a delete held by `HoldDeletes` and one `root-requested`; `h.SetCleanup(false)` on both at `m := w.mark()`: from then on `len(w.boxTableWritesSince(m)) == 0`; the delete is answered under the mark; `h.RootPass` with the switch off answers `res.Gate == "serve"` because `gated` stands, and acknowledges; both obligations reach `releasing` with their marks lifted and their entries held, and the page sink holds "cleanup of <host> waits for its reservation to be released: cleanup is off on this box" once per obligation; `carlos system reservations drain` releases both entries on their reports, settles a placement and a create paused between `BeginAllocation` and its reservation, and answers `drained`; a create of each host is then refused 503 with "this console is draining its reservations. try again later"; the console is restarted (a new `Server` over the same store and bucket), and only then does a create of each host succeed). |
| - `TestRootRunAgentNeverCleansUp` (V1: `restore := h.SetAgentRunsAsRoot(true)` with the switch on: at 24 h nothing is reserved, marked or deleted, and the page sink holds "cleanup needs the agent to run as its own user" once; a request left from an earlier armed run is answered by `h.RootPass(ctx, host.RootPassControls{AgentCheck: func(context.Context) (bool, string, error) { return false, "carlos-edge.service runs the agent as root", nil }})` with `res.Acks[0].Outcome == "cancelled" && res.Acks[0].Reason == "cleanup needs the agent to run as its own user"`, and `release.LiveLink(h.OptDir(), host)` still present). |
| - `TestCleanupHoldsTheMarkUntilItsDeleteIsAnswered` (a delete held by `HoldDeletes` and no `h.Cleanup` for 25 h of the test clock while a legacy move returns the host to ie: `h.Wake` answers 503 with `Retry-After: 30`; the next `h.Cleanup` resends the same version id (two `DELETE`s of it in `e.Holder().Requests(key)`), is answered, unmarks and releases; then `h.Wake(ctx, host).Code == 200` and `h.DBRows(host) == host.E2ERows`). |
| - - `TestFirstArmingThenDisarmBeforeTheFirstDrop` (X2 end to end, review 10's finding 2 interleaving; this witness skips the common start: `gated, quiesced, _ := h.Markers()` both nil; `h.SetCleanup(true)`; the forward move; at 24 h, with `log.SetOutput(&buf)`, two `h.Cleanup` runs: `r.State == "finished"`, `len(w.boxTableWritesSince(m)) == 0` and `strings.Count(buf.String(), "cleanup waits for root's first gated pass") == 1`; `res, _ := h.RootPass(ctx, host.RootPassControls{})` with no drop anywhere: `res.Armed && res.Gate == "serve"`; `h.BeforeDrop` blocks on a channel, and `go h.Cleanup(runCtx)` reserves, marks, deletes every key and stops at the hook; `h.SetCleanup(false)` restarts the agent, `runCancel()` and the hook's release end the discarded worker (a drop it writes has equal bytes, which the agent's row accepts); the new worker's `h.Cleanup` writes the drop: `r.State == "root-requested"`; `h.RootPass` with the switch off: `res.Gate == "serve" && res.Quiesced` and `res.Acks[0].Outcome == "done"`; the next `h.Cleanup`: `r.State == "releasing" && r.Outcome == "cleaned"`, no box table write since the disarm, and `h.Markers()` byte for byte what root's first pass left plus `quiesced`). |
| + - `TestFirstArmingThenDisarmBeforeTheFirstDrop` (X2 end to end, review 10's finding 2 interleaving; this witness skips the common start: `gated, quiesced, _ := h.Markers()` both nil; `h.SetCleanup(true)`; the forward move; at 24 h, `m := w.mark()`, with `log.SetOutput(&buf)`, two `h.Cleanup` runs: `r.State == "open" && r.Rev == 1`, unchanged since `gone` wrote it (a waiting run visits nothing, so not even the finish is observed), `len(w.boxTableWritesSince(m)) == 0` and `strings.Count(buf.String(), "cleanup waits for root's first gated pass") == 1`; `res, _ := h.RootPass(ctx, host.RootPassControls{Census: func() ([]int, error) { return []int{4242}, nil }})` (one other adopt pass alive, so this pass arms and certifies nothing) with no drop anywhere: `res.Armed && res.Gate == "serve"`, and `h.Markers()` answers `gated` set and `quiesced` nil; `h.BeforeDrop` blocks on a channel, and `go h.Cleanup(runCtx)` takes the obligation out of `open` (`finished`, with `FinishedAt` the move's `LastMove.At`, already 24 h past), then reserves, marks, deletes every key and stops at the hook; `h.SetCleanup(false)` restarts the agent, `runCancel()` and the hook's release end the discarded worker (a drop it writes has equal bytes, which the agent's row accepts); the new worker's `h.Cleanup` writes the drop: `r.State == "root-requested"`; `h.RootPass` with the switch off: `res.Gate == "serve" && res.Quiesced` and `res.Acks[0].Outcome == "done"`; the next `h.Cleanup`: `r.State == "releasing" && r.Outcome == "cleaned"`, no box table write since the disarm, and `h.Markers()` byte for byte what root's first pass left plus `quiesced`). |
| - [ ] **Step 2: Run them**: `go test -tags e2e ./internal/e2e/selfserve -run 'Cleanup|CreateUnderCleanup|RowMinted|LateAttempt|PromotionBack|OperatorReturnAfter|LegacyMoveInto|UntrackedLifetime|OlderCleanup|DisarmEnds|RootRunAgent|FirstArmingThenDisarm' -count=1`; fix what they find within this plan's scope. |
| - [ ] **Step 3: Run both census tests and the cross-builds**; write the census proposal with the loader block, the changed envelopes (`internal/host`, `internal/release`, `internal/registry`, `internal/lease`, `internal/moves`, `internal/console`, `internal/offlinecleanup`, the root package) and the units. |
| - [ ] **Step 4: Commit** `self-serve moves: cleanup end to end`; push. |
| @@ -492,7 +495,7 @@ Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and |
| - **The lock's write id (X1).** Rev 10's count kept apart the lifted bodies of one cleanup's own sequence, but a lifted body can also be rebuilt from outside it: a live tombstone recreated by a delayed creation after an ordinary release deleted the lock, then lifted by a later landing at the same generation and count (review 10, finding 1). A count, or the cleanup id, describes the history its writer knows; a write id drawn for each HTTP attempt depends on no history, so the argument needs no case analysis of who wrote the lock before. It is drawn per attempt rather than per operation because two identical attempts are two requests carrying one body, and the second could land later on a recurrence of the first's condition; for the same reason the lift leaves the SDK's retryer. The ruling's "every lock body our code writes" is read that way (rev 10's resends were identical; rev 11's differ only in `write_id`). The seal's tombstone creation and renewal stay litestream's (`tombstone.go:192`, `:248`): their bodies can recur, but no write of the cleanup is conditioned on one, because `MarkCleaning` first rewrites a lifted tombstone that has no `write_id` (Task 3). `cleanups` is kept for the log. A higher generation would also make a body new, but litestream derives the next lease's generation from it (`leaser.go:103-106`), so the plan leaves it alone. |
| - **The agent waits for root's first arming (X2).** Rev 10 armed the agent from its switch and uid alone, so an obligation could reserve, clean and reach `publishing` before any root pass had written `gated`; a disarm then left root fail-open with the switch off, and the request was never served (review 10, finding 2). The order is now fixed at both ends: root writes `gated` in its first pass after the switch goes on, before any publisher stage and with no condition on obligations (Task 6b), and the agent's cleanup run reads it first and does nothing without it (Task 4d). The cost is a delay of at most one adopt timer period (an hour, or less when a stage request runs a pass) between the switch and a box's first cleanup; a box whose `/opt` is not root-only never arms, which matches root's step, which can never run there. |
| - Plan 2 amendments (built code, changed here): move selections hold `host/<hostname>`; the switch-off rule applies between moves only, so a cleanup neither blocks nor is blocked by an operator move on another box; recovery mode counts `move` entries only; the cleanup reclaim interface is deleted; `releaseBoxes` writes an empty table instead of deleting it; every table write carries `epoch` and `rev`. |
| -- Built-code changes this plan makes, by task: T1 `moves/offline.go`, `host/movesoffline.go`, `release/programhistory.go` (export); T2a `moves/offline.go` (kinds, epoch, rev, `valid()`), `moves/reserve.go` (new, with `PeekActive`), `moves/tablewrite.go` (new), `release/store.go` (`PutOnce`); T2b `console/offlinereserve.go`, `console/selfserve_pass.go`, `console/lifetimes_store.go`, `console/server.go` (`tableWrites`); T2c `console/lifetimes_recovery.go`, `console/selfserve_pages.go`; T3 `lease/cleaning.go` (new: the mark, the unmark, `lockBody`'s count and write id, and the single-attempt lock writes), `lease/leasetest/hold.go` (new, test support), `lease/lease.go`, `lease/tombstone.go` (the refusal, `readLockBody`, `LiftTombstone` through `putLockUntilAnswered`), `lease/lockstate.go` (the refusals), `host/movesoffline_land.go` (the landing's wait), `activator/backend_exec.go` and `activator/activator.go` (the wake's answer); T4b, T4c `host` (new files only); T4d `host/movespass.go` (`CleanupSwitch`, `RootArmed`, the root agent's page), `host/movescleanup.go` (`armedTable`, the run's mode); T5a `host/movespaging.go`, `console/apiinstancemove_offline.go`; T5b `console/authcli.go`, `move.go`, `docs/site/cli.md`, the copy record; T6a `registry`; T6b `host/agent.go` (the lock and the gate, through `enterRootGate`), `host/adoptrootgate.go` (new), `release/rootcleanup.go` (new, with `RootCleanupArmed`), `release/carlosprocs_linux.go` and `_other.go` (new), `host/adoptlock.go` and `host/movesheldlock.go` (comments), `host/claimcensus.go`; T7a `console/store.go`, `lifetimes_store.go` (`host_key`, `Allocation`), `lifetimes_create.go`, `lifetimes_recovery.go` (the reload); T7b `fleets.go`, `console/store.go` (`claimRawSetAppPlacement`), `console/view.go` (`consoleRegistry`), `registry/registry.go` (`SetPlacementContext`), `registry/remote_dispatch.go` (the deadline), `offlinereserve.go`, `residence_sweep.go`; T7c `console/authcli.go`, `lifetimes_create.go`, `fleets.go`, `selfserve_door.go`, `system.go`, `docs/site/cli.md`; T9a `e2e/selfserve/job_world_test.go` (`faultStore.PutOnce`), `host/e2e_bridge.go` and `host/e2e_cleanup_bridge.go` (new: the real wake, the holder, the controls), `lease/leasetest/hold.go` (`HoldDeletes`), `offlinecleanup/layout_linux.go` (`SetTrustForE2E`), `host/movespass.go` and `host/movescleanup.go` (`CleanupManual`); `cmd/console/main.go` is not changed. The mirror (`internal/lease/mirror.go`) is not changed. |
| -- Names used across tasks: `offlinecleanup.Retired` (`State`, `Outcome`, `Keys`, `Events`, `PagesPending`, `LocalLeft`, `Root`, `Rev`, `PublishedRev`, `MarkedAt`, `ReserveUncertain`, `InFlight`, `Superseding`, `RootRequestedAt`, `CancelRequestedAt`), `CorruptStub`, `CleanupID`, `InventoryKey`, `InventoryVersion`, `InFlightDelete`, `KeyOutcome` (`State`, `Done`), `Event`, `PendingPage`, `Report`, `FileIdentity`, `LocalPath`, `KeptPath`, `RootCleanupRequest`, `RootCleanupPath`, `RootCleanupAck`, `ValidateRetired`, `ValidateCorruptStub`, `ValidateReport`, `UsableVersionID`, `Identify`, `RootOnlyChain`, `RootOwned`, `SetTrustForE2E`, `InventoryKey.Latest`, `InventoryVersion.Seq`, `Report.InFlight`, `Report.AbandonPath`; `moves.HostKey`, `IsHostKey`, `Active`, `ReadActive`, `PeekActive`, `ReadForAcquire`, `NextActiveBody`, `PutTable`, `TableWrites` (`Unanswered`, `Resend`), `Op`, `Conflicts`, `Reserve`, `Holds`, `Release`, `ErrReserved`, `ErrUnreadable`, `ErrUncertain`, `ErrReservationMismatch`, `RecordAway`, `Away*`, `OfflineOpDir`, `RetiredFile`, `CleanupReportKey`, `OfflineKindCreate`, `OfflineKindPlace`; `lease.CleaningOwner`, `CleaningExpiry`, `IsCleaningOwner`, `MarkCleaning`, `UnmarkCleaning`, `ErrNotOurLock`, `lockBody` (`Cleanups`, `WriteID`), `newWriteID`, `readLockBody`, `putLockUntilAnswered`, `lockAttemptTimeout`, `ErrLockWriteUncertain`, `leasetest.Holder`, `VersionedStore.DeleteVersion`; `activator.errLeaseCleaning`, `cleaningRetryAfter`; `release.PutOnce`, `SingleAttemptPutter`, `ErrNoSingleAttemptPut`; `deleteInventoriedVersion`, `w.cleanupVisit`, `w.supersededBy`, `w.retireKey`, `w.settleInFlight`, `w.kickCleanup`, `cleanupRun`, `cleanupMu`, `w.tableWrites`, `w.armedTable`, `errDisarmed`, `w.rootAgentPage`, `cfg.CleanupSwitch`, `cfg.RootArmed`, `cleanupMode` (`modeDisarmed`, `modeWaiting`, `modeArmed`), `w.mode`, `w.gateWait`, `cfg.CleanupManual`, `agentRunsAsRoot`, `recordRetired`, `createRetired`, `writeRetired`, `markPublished`, `flushPageOutboxFor`, `reapRetired`; `release.ServeRootCleanupRequests`, `RootCleanupOptions`, `RootCleanupBase`, `RootCleanupDir`, `ReadRootCleanupAck`, `LockRootPass`, `RootCleanupGated`, `ArmRootCleanup`, `RootCleanupArmed`, `QuiesceRootCleanup`, `CarlosProcesses`, `CarlosProcess`, `AdoptPasses`, `AgentRunsUnprivileged`, `ErrRootCleanupRefused`, `ErrNotRootOnly`, `ProgramsPath`; `rootCleanupAuthority`, `rootCleanupPass`, `rootStage`, `rootRefusedAfter`, `adoptRootGate`, `enterRootGate`, `cleanupBeforeDropHook`, the files `root-cleanup/gated`, `root-cleanup/quiesced` and `cleanup.abandon`; `registry.WithHostFrozen`, `SetPlacementContext`; `consoleRegistry.SetPlacementContext`; `s.offlineCleanupsOf`, `s.reservePlacement`, `s.registerPlacement`, `s.deregisterPlacement`, `s.settlePlacement`, `s.placing`, `s.residenceSweep`, `s.releaseAllocationKey`, `s.store.Allocation`, `recoveryListedHook`, `s.admission`, `s.admitResidence`, `s.closeAdmission`, `errConsoleDraining`, `errHostUnderCleanup`, `s.tableWrites`; the CLI's `moveArgs.log`, `logMove`, `logCleanup` and `carlos system reservations drain`; the bridge's `E2ERealWake`, `E2ERows`, `(*E2E).Holder`, and `BoxHandle`'s `SetCleanup`, `SetAgentRunsAsRoot`, `Cleanup`, `Retired`, `BeforeDrop`, `RootPass` (`RootPassControls`, `RootPassResult`), `Markers`, `Wake` (`WakeAnswer`) and `DBRows`; the e2e world's `faultStore.PutOnce`, `boxStore`, `cleanupWorld`, `newCleanupWorld`; `leasetest.Holder.HoldDeletes`. |
| +- Built-code changes this plan makes, by task: T1 `moves/offline.go`, `host/movesoffline.go`, `release/programhistory.go` (export); T2a `moves/offline.go` (kinds, epoch, rev, `valid()`), `moves/reserve.go` (new, with `PeekActive`), `moves/tablewrite.go` (new), `release/store.go` (`PutOnce`); T2b `console/offlinereserve.go`, `console/selfserve_pass.go`, `console/lifetimes_store.go`, `console/server.go` (`tableWrites`); T2c `console/lifetimes_recovery.go`, `console/selfserve_pages.go`; T3 `lease/cleaning.go` (new: the mark, the unmark, `lockBody`'s count and write id, and the single-attempt lock writes), `lease/leasetest/hold.go` (new, test support), `lease/lease.go`, `lease/tombstone.go` (the refusal, `readLockBody`, `LiftTombstone` through `putLockUntilAnswered`), `lease/lockstate.go` (the refusals), `host/movesoffline_land.go` (the landing's wait), `activator/backend_exec.go` and `activator/activator.go` (the wake's answer); T4b, T4c `host` (new files only); T4d `host/movespass.go` (`CleanupSwitch`, `RootArmed`, the root agent's page), `host/movescleanup.go` (`armedTable`, the run's mode); T5a `host/movespaging.go`, `console/apiinstancemove_offline.go`; T5b `console/authcli.go`, `move.go`, `docs/site/cli.md`, the copy record; T6a `registry`; T6b `host/agent.go` (the lock and the gate, through `enterRootGate`), `host/adoptrootgate.go` (new), `release/rootcleanup.go` (new, with `RootCleanupArmed`), `release/carlosprocs_linux.go` and `_other.go` (new), `host/adoptlock.go` and `host/movesheldlock.go` (comments), `host/claimcensus.go`; T7a `console/store.go`, `lifetimes_store.go` (`host_key`, `Allocation`), `lifetimes_create.go`, `lifetimes_recovery.go` (the reload); T7b `fleets.go`, `console/store.go` (`claimRawSetAppPlacement`), `console/view.go` (`consoleRegistry`), `registry/registry.go` (`SetPlacementContext`), `registry/remote_dispatch.go` (the deadline), `offlinereserve.go`, `residence_sweep.go`; T7c `console/authcli.go`, `lifetimes_create.go`, `fleets.go`, `selfserve_door.go`, `system.go`, `docs/site/cli.md`; T9a `e2e/selfserve/job_world_test.go` (`faultStore.PutOnce`), `host/e2e_bridge.go` and `host/e2e_cleanup_bridge.go` (new: the real wake, the holder, the controls), `lease/leasetest/hold.go` (`HoldDeletes`), `offlinecleanup/layout_linux.go` and `layout_other.go` (`SetTrustForE2E`), `host/movespass.go` and `host/movescleanup.go` (`CleanupManual`); `cmd/console/main.go` is not changed. The mirror (`internal/lease/mirror.go`) is not changed. |
| +- Names used across tasks: `offlinecleanup.Retired` (`State`, `Outcome`, `Keys`, `Events`, `PagesPending`, `LocalLeft`, `Root`, `Rev`, `PublishedRev`, `MarkedAt`, `ReserveUncertain`, `InFlight`, `Superseding`, `RootRequestedAt`, `CancelRequestedAt`), `CorruptStub`, `CleanupID`, `InventoryKey`, `InventoryVersion`, `InFlightDelete`, `KeyOutcome` (`State`, `Done`), `Event`, `PendingPage`, `Report`, `FileIdentity`, `LocalPath`, `KeptPath`, `RootCleanupRequest`, `RootCleanupPath`, `RootCleanupAck`, `ValidateRetired`, `ValidateCorruptStub`, `ValidateReport`, `UsableVersionID`, `Identify`, `RootOnlyChain`, `RootOnlyDir`, `RootOwned`, `SetTrustForE2E`, `InventoryKey.Latest`, `InventoryVersion.Seq`, `Report.InFlight`, `Report.AbandonPath`; `moves.HostKey`, `IsHostKey`, `Active`, `ReadActive`, `PeekActive`, `ReadForAcquire`, `NextActiveBody`, `PutTable`, `TableWrites` (`Unanswered`, `Resend`), `Op`, `Conflicts`, `Reserve`, `Holds`, `Release`, `ErrReserved`, `ErrUnreadable`, `ErrUncertain`, `ErrReservationMismatch`, `RecordAway`, `Away*`, `OfflineOpDir`, `RetiredFile`, `CleanupReportKey`, `OfflineKindCreate`, `OfflineKindPlace`; `lease.CleaningOwner`, `CleaningExpiry`, `IsCleaningOwner`, `MarkCleaning`, `UnmarkCleaning`, `ErrNotOurLock`, `lockBody` (`Cleanups`, `WriteID`), `newWriteID`, `readLockBody`, `putLockUntilAnswered`, `lockAttemptTimeout`, `ErrLockWriteUncertain`, `leasetest.Holder`, `VersionedStore.DeleteVersion`; `activator.errLeaseCleaning`, `cleaningRetryAfter`; `release.PutOnce`, `SingleAttemptPutter`, `ErrNoSingleAttemptPut`; `deleteInventoriedVersion`, `w.cleanupVisit`, `w.supersededBy`, `w.retireKey`, `w.settleInFlight`, `w.kickCleanup`, `cleanupRun`, `cleanupMu`, `w.tableWrites`, `w.armedTable`, `errDisarmed`, `w.rootAgentPage`, `cfg.CleanupSwitch`, `cfg.RootArmed`, `cleanupMode` (`modeDisarmed`, `modeWaiting`, `modeArmed`), `w.mode`, `w.gateWait`, `cfg.CleanupManual`, `agentRunsAsRoot`, `recordRetired`, `createRetired`, `writeRetired`, `markPublished`, `flushPageOutboxFor`, `reapRetired`; `release.ServeRootCleanupRequests`, `RootCleanupOptions`, `RootCleanupBase`, `RootCleanupDir`, `ReadRootCleanupAck`, `LockRootPass`, `RootCleanupGated`, `ArmRootCleanup`, `RootCleanupArmed`, `QuiesceRootCleanup`, `CarlosProcesses`, `CarlosProcess`, `AdoptPasses`, `AgentRunsUnprivileged`, `ErrRootCleanupRefused`, `ErrNotRootOnly`, `ProgramsPath`; `rootCleanupAuthority`, `rootCleanupPass`, `rootStage`, `rootRefusedAfter`, `adoptRootGate`, `enterRootGate`, `cleanupBeforeDropHook`, the files `root-cleanup/gated`, `root-cleanup/quiesced` and `cleanup.abandon`; `registry.WithHostFrozen`, `SetPlacementContext`; `consoleRegistry.SetPlacementContext`; `s.offlineCleanupsOf`, `s.reservePlacement`, `s.registerPlacement`, `s.deregisterPlacement`, `s.settlePlacement`, `s.placing`, `s.residenceSweep`, `s.releaseAllocationKey`, `s.store.Allocation`, `recoveryListedHook`, `s.admission`, `s.admitResidence`, `s.closeAdmission`, `errConsoleDraining`, `errHostUnderCleanup`, `s.tableWrites`; the CLI's `moveArgs.log`, `logMove`, `logCleanup` and `carlos system reservations drain`; the bridge's `E2ERealWake`, `E2ERows`, `(*E2E).Holder`, and `BoxHandle`'s `SetCleanup`, `SetAgentRunsAsRoot`, `Cleanup`, `Retired`, `BeforeDrop`, `RootPass` (`RootPassControls`, `RootPassResult`), `Markers`, `Wake` (`WakeAnswer`) and `DBRows`; the e2e world's `faultStore.PutOnce`, `boxStore`, `cleanupWorld`, `newCleanupWorld`; `leasetest.Holder.HoldDeletes`. |
| - Review Focus 1 to 5 are pinned by T3+T4b+T4c, T4a, T6b+T6c, T2a-c+T7a-c, T5a+T5b+T8, and end to end by T9b. |
| - **Open questions for Paul** (amadan discussion 363): (1) legacy moves without a hostname reservation: accept the exemption, or reserve around `MoveComplete` on the box (a box-side release and crash path, and longer legacy-cut downtime while root answers)? (2) Cleanup runs 24 hours after **every** offline move, operator moves included, replacing the runbook's manual cleanup; intended (§10 Q4 talks about self-serve)? (3) Is the box switch `CARLOS_OFFLINE_CLEANUP=on` acceptable as the rollout gate? (4) A move of the same host waits while its cleanup holds `host/<hostname>`, which can last until root answers (paged every 10 min, hourly once root's ancestry is refused) or until an unanswered delete is answered or abandoned (paged every 10 min); acceptable? (5) The copy rows: five door sentences (the drain's 503, rev 9: "this console is draining its reservations. try again later"), the wake's 503 body for a replica under cleanup ("instance is being cleaned up after a move. retry shortly", rev 8), nine page subjects and one page body (rev 9 adds "cleanup of <host> waits for its reservation to be released: cleanup is off on this box"), the agent's page "cleanup needs the agent to run as its own user" with its body, which is also root's cancellation reason (rev 9), and the CLI's lines: the waiting-delete line, the drain's lines (rev 9 rewrites them: no rollback wording, a refusal while a self-serve move is in flight), and the `--log` paragraph; rev 10 withdraws rev 9's three `carlos system cleanup-ungate` lines with the verb. (6) On every built box the programs file sits under the agent-writable `/data`, so root never removes it and it stays the runbook's; acceptable, or should the agent remove it itself? (7) Withdrawn by rev 6: the 15-minute `QuietUntil` and the marker-removing repair are gone; question 9 replaces it. (8) Rev 9: the drain, now an operator's tool, refuses while a self-serve move is in flight and then leaves admission open; once it has started, the console takes no new create or placement until it restarts. Acceptable? (9) **The §7.6 amendment:** the cleanup deletes every inventoried version of the retired replica by its version id (oldest first, the one current at `gone` last) instead of writing delete markers, so the source copy of a finished move is not recoverable from the bucket after the cleanup, in exchange for a boundary the server enforces (a late or retried request can remove only the version it names). **Rev 7 adds a consequence:** the mark that keeps every later owner off the replica while a delete may still land has no expiry, so a cleanup that cannot settle a delete (the bucket keeps refusing it, or the box is gone) holds the host's replica marked, and with it every wake, landing and return of that host in this region, until an operator acts on the page: by repairing what refuses the delete, or by abandoning it on the box (`cleanup.abandon`). Accept, or keep markers (then only `deleteInventoriedVersion` changes, and the settlement rules need a fresh review)? (10) **Rev 9, no console rollback across plan 4 (V3), the decision for Paul:** once a plan 4 console has written the reservation table, a console from before this plan is never run against that deployment again. A console defect is fixed by rolling forward, and plan 4 itself is rolled back by disarming cleanup on every box: no new obligations; existing ones finish, are superseded or page; one that needs its reservation released while its box is disarmed pages, and the drain releases it on the box's report. Rev 8's console rollback (the drain as a prerequisite, the clean-exit line, the 20-minute wait) is gone, because review 8 showed that no count or wait it used survives a console crash or is bounded by S3. Accept this as the operating rule? (11) **Rev 9, cleanup needs the agent to run as its own user (V1):** a box whose agent runs as root (the built code allows it, `agentUnitsOption`) never cleans up, pages once per agent start, and its leftovers stay the runbook's; the shipped `carlos-edge.service` runs it as `carlos`. Acceptable? (12) **Rev 10, the gate is for the life of the box (W2):** once cleanup has been armed on a box, root's `gated` and `quiesced` markers stay there for good, and no command removes them. Disarming stops new cleanups but keeps root serving any request still owed, including one published after the disarm; the price is that every later adopt pass on that box needs the root pass lock to publish, so a fault on that lock means a pass that publishes nothing (logged as an ALERT) instead of today's fail-open adoption. Acceptable? |