amadan
plan 4 rev 10: the unmark never reconstructs an earlier lock body (a cleanups counter in the lifted tombstone; the unmark always writes); the ungate verb is gone and the arming markers are permanent
Answers Astra's review of rev 9 (a delayed mark landing on a recurring ETag; ungating stranding a future request).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1 file changed,
+46
−39
docs/superpowers/plans/2026-10-22-self-serve-moves-4-cleanup.md+46 −39
CI failed — run details
CI log
=== gate === sigwho: pid 2 ppid 1 pgid 0 sid 0; child pid 3 .amadan/test-gate.sh gate: 2026-10-24T11:17:42Z on aarch64, 32 cpus, 61385 MB available, go1.26.8 linux/arm64 gate: +0s start launchctx gate: +0s start vet gate: +0s start gofmt gate: +1s start rest gate: +11s start internal_console@0 gate: +11s start internal_console@1 gate: +11s start internal_console@2 gate: +11s start internal_console@3 gate: +11s start internal_console@4 gate: +11s start internal_console@5 gate: +11s start internal_console@6 gate: +11s start internal_console@7 gate: +11s waiting for internal_console's shards before the next split package gate: +173s start root@0 gate: +173s start root@1 gate: +173s start root@2 gate: +173s waiting for root's shards before the next split package amadan: step gate failed: signal: terminated amadan: job failed at step gate — its output is above, not at the tail of this log
diff --git a/docs/superpowers/plans/2026-10-22-self-serve-moves-4-cleanup.md b/docs/superpowers/plans/2026-10-22-self-serve-moves-4-cleanup.md| index c246dd017..757e16b82 100644 |
| --- a/docs/superpowers/plans/2026-10-22-self-serve-moves-4-cleanup.md |
| +++ b/docs/superpowers/plans/2026-10-22-self-serve-moves-4-cleanup.md |
| @@ -2,11 +2,13 @@ |
| > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| -Rev 9, 2026-10-24. Rev 8's review (`after236/selfserve-plan4-review8.md`: items 3, 7 and 8 closed; items 1, 2, 4, 5 and 6 open) is answered as follows, under the controller's rulings V1 to V3. **Item 1 (a root-run agent publishes outside the gate), V1:** cleanup requires an agent that does not run as root. The shipped `carlos-edge.service` (`User=carlos` and its sandbox, `internal/host/units/carlos-edge.service:28-50`) is the supported topology (Global Constraint "Arming and rollout"). The agent never arms with euid 0 (`armOffline` reads `agentRunsAsRoot`, `internal/host/agent.go:2206`) and pages "cleanup needs the agent to run as its own user" once per start instead (Task 4d, `TestRootAgentIsNeverArmed`). Root's stage checks the same every pass before it certifies an arming or touches a path (`release.AgentRunsUnprivileged`: the unit's `User=` as systemd reports it, and no process of the installed binary running `agent` with uid 0), and otherwise answers every request `cancelled` with that reason (Task 6b, `TestRootStepRefusesARootAgent`). The adopt-process census matches by executable (`readlink` and `stat` of `/proc/<pid>/exe` against `<optDir>/carlos` and `carlos.prev`), not by name, so a renamed path of the installed binary is still counted; an unrelated binary running `adopt` is outside support (Self-review notes). **Item 2 (the arming identity can be reused), V2:** each arming writes a fresh random nonce into `gated` (root-owned, `fsync`ed, create-only); `quiesced` records the nonce it certifies, and a certificate whose nonce differs from the current `gated`'s is void and removed; `gated` is removed only by `carlos system cleanup-ungate`, which removes `quiesced` first and then `gated`, so a crash between leaves no certificate (Task 6b, `TestStaleCertificateIsVoid`, `TestUngateRemovesTheCertificateFirst`). **Items 4, 5 and 6 (rollback settlement), V3:** there is no console rollback across the plan 4 boundary. Once a plan 4 console has written `active.json`, a console from before this plan never runs against that deployment again, and a console defect is fixed by rolling forward; plan 4 is rolled back by disarming cleanup on every box (Global Constraints "No console rollback across the plan 4 boundary" and "Rolling plan 4 back is disarming cleanup"). Rev 8's rollback machinery is therefore gone: the drain as a rollback prerequisite, the 20-minute attempt window and `LastTimedOut`, the console's `SIGTERM` handler and its clean-exit line, and the S3 completion-latency assumption with its dev witness. Nothing left in the plan rests on a time bound or on a count that a crash loses. What stays: `carlos system reservations drain` as an operator's tool (Task 7c: it settles stuck `create` and `place` entries, fence before release, releases a `cleanup` entry whose box reports the obligation `releasing` or `ended`, and refuses while a self-serve move is in flight), and single-attempt table writes resent until answered, now to settle one operation's own acquisition (F2), with the registration rule stated: a write's registration is removed only by the goroutine that owns it, after its final answer, under the table-writes mutex, and ownership passes to a resend only after the owner's last attempt has returned from `PutOnce` (Task 2a, `TestRegistrationIsRemovedOnlyByItsOwner`). **The box writer boundary (item 6):** an unarmed box never writes `active.json`. Its reads never bootstrap or normalise (`moves.PeekActive`), and its three table writers (`Reserve`, `Release`, the resend) run only when armed; an obligation that needs a release while unarmed stays `releasing` and pages, and the drain tool releases its entry on its report (Global Constraint "An unarmed box never writes the reservation table", Tasks 2a, 4a, 4d and 7c, `TestUnarmedBoxNeverWritesTheTable`, `TestDrainReleasesACleanupOnlyOnItsReport`). |
| +Rev 10, 2026-10-24. Review 9 (`after236/selfserve-plan4-review9.md`) closed rev 8's items 1, 2, 4, 5 and 6, found two new failures and corrected one witness; they are answered under the controller's rulings W1 to W3. **Finding 1 (a delayed `MarkCleaning` restores a permanent mark after the cleanup ends), W1:** the unmark never reconstructs an earlier lock body. The body of `lock.json` gains one optional field, `cleanups` (`lease.lockBody`: litestream's three fields and `Cleanups uint64 \`json:"cleanups,omitempty"\``). The mark carries the count of the lifted tombstone it replaces, and `UnmarkCleaning` writes this move's lifted tombstone with the count it read plus one. So the body after the unmark differs from the body before the mark, its ETag (a digest of the bytes, for these single-part writes) never recurs, and a mark attempt conditioned on the pre-mark ETag can never land after it. `UnmarkCleaning` **always writes** when it finds our mark or this move's lifted tombstone, conditioned on the ETag it read, even when no mark is there: a mark attempt that may still be travelling then names an ETag that is gone for good. The mark's and the unmark's lock writes are single attempts (`aws.NopRetryer{}` per call), resent identically until answered; a 412 after an unanswered attempt is answered by reading the lock (our mark present: the mark is settled; this move's lifted tombstone with a higher count: the unmark is). Older readers ignore the field (`readLock`, `LockState` and litestream's `readLease` decode into `litestream.Lease` and do not refuse unknown fields), so `Acquire`'s parsing is unchanged; `LiftTombstone` writes through `lockBody`, carrying the count as read, which is byte for byte what it writes today when the count is zero (Global Constraint "No lock body recurs"; Task 3, `TestDelayedMarkCannotLandAfterTheUnmark`, `TestUnmarkSettlesAnUncertainMark`; Task 4a, `TestSupersessionAfterAnUncertainMarkSettlesIt`). **Finding 2 (`cleanup-ungate` strands a request that is not yet published), W2:** the verb is removed. No code removes `gated` or `quiesced`. A fresh arming (a gated pass that finds `gated` absent: the box's first, or one after an operator removed a malformed marker by hand) writes `gated` afresh with a new nonce, which voids the standing `quiesced` and so forces a new census. Disarming leaves both in place, so root keeps serving the requests of obligations that were in flight at the disarm, including one published after it, because root's gate is judged by the markers, not by the switch (Global Constraint "Rolling plan 4 back is disarming cleanup"; Task 6b, `TestDisarmedPublicationIsStillServed`, `TestNothingRemovesTheGateMarkers`). The verb's file, its copy rows and its docs line are gone, and the rollout says the markers last for the life of the box. **The witness correction, W3:** Task 9's disarm witness restarts the console after `drained` and only then asserts that creates succeed, because the drain leaves admission closed until a restart. |
| + |
| +Rev 9's answers that stand (review 9 closed rev 8's items 1, 2, 4, 5 and 6 with them). **Item 1 (a root-run agent publishes outside the gate), V1:** cleanup requires an agent that does not run as root. The shipped `carlos-edge.service` (`User=carlos` and its sandbox, `internal/host/units/carlos-edge.service:28-50`) is the supported topology (Global Constraint "Arming and rollout"). The agent never arms with euid 0 (`armOffline` reads `agentRunsAsRoot`, `internal/host/agent.go:2206`) and pages "cleanup needs the agent to run as its own user" once per start instead (Task 4d, `TestRootAgentIsNeverArmed`). Root's stage checks the same every pass before it certifies an arming or touches a path (`release.AgentRunsUnprivileged`: the unit's `User=` as systemd reports it, and no process of the installed binary running `agent` with uid 0), and otherwise answers every request `cancelled` with that reason (Task 6b, `TestRootStepRefusesARootAgent`). The adopt-process census matches by executable (`readlink` and `stat` of `/proc/<pid>/exe` against `<optDir>/carlos` and `carlos.prev`), not by name, so a renamed path of the installed binary is still counted; an unrelated binary running `adopt` is outside support (Self-review notes). **Item 2 (the arming identity can be reused), V2:** each arming writes a fresh random nonce into `gated` (root-owned, `fsync`ed, create-only); `quiesced` records the nonce it certifies, and a certificate whose nonce differs from the current `gated`'s is void; rev 9 also added a verb, `carlos system cleanup-ungate`, to remove `quiesced` and then `gated`; rev 10 removes it (W2 above), and no code removes either marker (Task 6b, `TestStaleCertificateIsVoid`). **Items 4, 5 and 6 (rollback settlement), V3:** there is no console rollback across the plan 4 boundary. Once a plan 4 console has written `active.json`, a console from before this plan never runs against that deployment again, and a console defect is fixed by rolling forward; plan 4 is rolled back by disarming cleanup on every box (Global Constraints "No console rollback across the plan 4 boundary" and "Rolling plan 4 back is disarming cleanup"). Rev 8's rollback machinery is therefore gone: the drain as a rollback prerequisite, the 20-minute attempt window and `LastTimedOut`, the console's `SIGTERM` handler and its clean-exit line, and the S3 completion-latency assumption with its dev witness. Nothing left in the plan rests on a time bound or on a count that a crash loses. What stays: `carlos system reservations drain` as an operator's tool (Task 7c: it settles stuck `create` and `place` entries, fence before release, releases a `cleanup` entry whose box reports the obligation `releasing` or `ended`, and refuses while a self-serve move is in flight), and single-attempt table writes resent until answered, now to settle one operation's own acquisition (F2), with the registration rule stated: a write's registration is removed only by the goroutine that owns it, after its final answer, under the table-writes mutex, and ownership passes to a resend only after the owner's last attempt has returned from `PutOnce` (Task 2a, `TestRegistrationIsRemovedOnlyByItsOwner`). **The box writer boundary (item 6):** an unarmed box never writes `active.json`. Its reads never bootstrap or normalise (`moves.PeekActive`), and its three table writers (`Reserve`, `Release`, the resend) run only when armed; an obligation that needs a release while unarmed stays `releasing` and pages, and the drain tool releases its entry on its report (Global Constraint "An unarmed box never writes the reservation table", Tasks 2a, 4a, 4d and 7c, `TestUnarmedBoxNeverWritesTheTable`, `TestDrainReleasesACleanupOnlyOnItsReport`). |
| Rev 8's answers that stand: **U1**, the arming boundary (`gated`, the census of root adopt processes, `quiesced`), extended above by the agent check, the executable match and the nonce; the census stands in place of an in-process mutex, because each `carlos adopt` process runs one pass and the four adopt units are separate processes that can overlap (`internal/host/agent.go:4568`, `:4645-4648`). **U2** (closed by review 8): recovery reloads the allocation by its op id under the app and hostname locks, before `admit` and before any reservation or effect, and a terminal row only finishes its owed release (Task 7a, `TestRecoveryReloadsTheAllocationUnderItsLocks`). **U4** (closed): a `cleaning:` holder answers a wake with `errLeaseCleaning` and `Retry-After: 30`, never the mark's expiry (Task 3, `TestCleaningMarkAnswersAFiniteRetry`). **U5** (closed): the listing check and the rollout's `get-bucket-versioning` check are two guarantees, stated apart (Global Constraint "Arming and rollout", Task 1). **U3, in part:** `release.PutOnce` with `aws.NopRetryer{}` per call, `moves.PutTable` resending the identical write until it is answered, and the store door's refusal (`release.ErrNoSingleAttemptPut`). |
| -Rev 7's answers that stand (its findings are review 6's): **Finding 1:** the cleaning mark on `lock.json` has no expiry. It is a state, not a timed lease: written by a conditional PUT over this move's lifted tombstone with a sentinel expiry no clock reaches, refused to every lease acquirer whatever its age, never renewed, and lifted only by the cleanup's own `releasing` once every delete of the run is answered, by supersession or cancellation only when no delete is in flight, or by an operator's abandon after the page (Global Constraint "The mark has no expiry", Tasks 3, 4a, 4c). A crash or an outage of any length lifts nothing, so no wake can take the lease while a delete the cleanup sent may still land. **Finding 2:** on a box where cleanup can run (the switch on, or the root-owned `gated` marker an armed pass left), an adopt pass that cannot take the root pass lock publishes nothing that pass; a box never armed keeps today's fail-open adoption (Task 6b's `adoptRootGate`). **Finding 3:** the drain has an admission barrier: one console-wide flag, checked inside the same critical section in which `BeginAllocation` persists its row, a placement registers and `AdmitSelfServeMove` admits; the drain sets it first, then waits on the allocation rows, the placement registrations and the open self-serve moves themselves, not on table entries (Task 7c). **Finding 4:** a new-protocol acquisition is issued only against a table body that carries `epoch` and `rev`; a body without them is first normalised by a separate conditional write and read again (`moves.ReadForAcquire`, Tasks 2a and 2b). **Finding 5:** `consoleRegistry` (`internal/console/view.go:511`) gains `SetPlacementContext`, and the registry's remote dispatcher (`internal/registry/remote_dispatch.go:270`) runs the update under the deadline it received (Task 7b). **Finding 6 and the `null` id:** condition (4) now requires the key's inventoried latest version (the one current at `gone`) to be current at every deletion; a key whose latest is gone or no longer current is `superseded` and nothing more of it is deleted. The inventory records each version's listing sequence and the key's latest id, so no tie is reversed; `ValidateRetired` refuses a `null` id by `lease.UsableVersionID`'s rule, and a listing that holds one writes no `retired.json` (Tasks 1 and 4c). **Initialisation order:** `LockRootPass` and the request directory check the existing ancestry before root creates any missing directory (Task 6b). Rev 9 keeps all of these. The admission barrier (finding 3) now serves the operator's drain, not a rollback; the gate (finding 2) still publishes nothing on a gated box without the root pass lock; and `moves.ReadForAcquire` (finding 4) still normalises before any acquisition, so from the first plan 4 write on no acquisition is conditioned on a metadata-free ETag. |
| +Rev 7's answers that stand (its findings are review 6's): **Finding 1:** the cleaning mark on `lock.json` has no expiry. It is a state, not a timed lease: written by a conditional PUT over this move's lifted tombstone with a sentinel expiry no clock reaches, refused to every lease acquirer whatever its age, never renewed, and lifted only by the cleanup's own `releasing` once every delete of the run is answered, by supersession or cancellation only when no delete is in flight, or by an operator's abandon after the page (Global Constraint "The mark has no expiry", Tasks 3, 4a, 4c). A crash or an outage of any length lifts nothing, so no wake can take the lease while a delete the cleanup sent may still land. **Finding 2:** on a box where cleanup can run (the switch on, or the root-owned `gated` marker an armed pass left), an adopt pass that cannot take the root pass lock publishes nothing that pass; a box never armed keeps today's fail-open adoption (Task 6b's `adoptRootGate`). **Finding 3:** the drain has an admission barrier: one console-wide flag, checked inside the same critical section in which `BeginAllocation` persists its row, a placement registers and `AdmitSelfServeMove` admits; the drain sets it first, then waits on the allocation rows, the placement registrations and the open self-serve moves themselves, not on table entries (Task 7c). **Finding 4:** a new-protocol acquisition is issued only against a table body that carries `epoch` and `rev`; a body without them is first normalised by a separate conditional write and read again (`moves.ReadForAcquire`, Tasks 2a and 2b). **Finding 5:** `consoleRegistry` (`internal/console/view.go:511`) gains `SetPlacementContext`, and the registry's remote dispatcher (`internal/registry/remote_dispatch.go:270`) runs the update under the deadline it received (Task 7b). **Finding 6 and the `null` id:** condition (4) now requires the key's inventoried latest version (the one current at `gone`) to be current at every deletion; a key whose latest is gone or no longer current is `superseded` and nothing more of it is deleted. The inventory records each version's listing sequence and the key's latest id, so no tie is reversed; `ValidateRetired` refuses a `null` id by `lease.UsableVersionID`'s rule, and a listing that holds one writes no `retired.json` (Tasks 1 and 4c). **Initialisation order:** `LockRootPass` and the request directory check the existing ancestry before root creates any missing directory (Task 6b). Rev 10 keeps all of these. The admission barrier (finding 3) now serves the operator's drain, not a rollback; the gate (finding 2) still publishes nothing on a gated box without the root pass lock; and `moves.ReadForAcquire` (finding 4) still normalises before any acquisition, so from the first plan 4 write on no acquisition is conditioned on a metadata-free ETag. |
| Rev 6's answers that stand: every data deletion names one inventoried version id, oldest first within a key, through one function (`deleteInventoriedVersion`, Task 4b), so a late, retried or outstanding delete can remove only the version it names (R1 to R4, F1, N4; a spec amendment, **Paul's question 9**); a delete that went out without an answer is sent again, and answered, before the mark is given up, because a promotion can keep one of our versions in place by its equal ETag (`lease.MirrorPrefixKeepObserved`, `internal/lease/mirror.go:285`; Task 4c); root's step is a stage of `runAdoptUnderRootRelease`, after release adoption (named recovery runs inside it) and after the held stage, under the adopt-pass lock and a root-only pass lock, with the registry host freeze and the held-stage lock as rev 5 had them (R5, F5; Task 6b); the drain command, `carlos system reservations drain` (R6, N11; Task 7c), which rev 9 keeps as an operator's tool while V3 removes the console rollback it served; `--log` takes `--move <id>` or `--cleanup <id>`, and the console lists a host's cleanup reports across moves from the bucket, so a deleted record still answers (R7, Task 5b); placement registers before it reserves and stays registered through settlement, under one deadline (R8, Task 7b); when root rejects a directory's ancestry it writes nothing there, logs the refusal, and the agent pages hourly after an hour without an answer and never releases (Task 6b). Task 2 is split into 2a to 2c, and the delete primitive has its own task with the dev gate. |
| @@ -18,40 +20,41 @@ Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and |
| **Goal:** Twenty-four hours after a move the source box observed as finished, the source box retires the old region's replica from evidence it recorded at `gone`, under a `cleanup` reservation on its own box and on the hostname and a cleaning mark on the replica's lock that has no expiry: key by key with the conditions re-read at each key, each inventoried version deleted by its version id, oldest first, only while the version that was current at `gone` is still current, and every delete answered before the mark or the reservation is given up. Root removes the old live link only inside its adoption pass, under a lock no other root publisher can bypass (on a box where cleanup can run, a pass that cannot take that lock publishes nothing), through a request it copies, authorises from the bucket and answers in its own directory, and only where no other uid can write the link's parent. The obligation's directory goes seven days after it ended. Nothing times out into a release. Cleanup runs only where the agent runs as its own user, and a box writes the reservation table only while cleanup is armed on it. There is no rollback to a console from before this plan: plan 4 is rolled back by disarming cleanup on every box, and a console defect is fixed by rolling forward. |
| -**Architecture:** At `gone` the sealing source box writes `retired.json` under the move's op directory, for a tracked lifetime only: every version and delete marker (key, version id, ETag) under the normalised hibernate prefix (`lock.json` and `build-state.json` aside), each key's versions in deletion order, and the identities of the root-owned paths whose parent chain is root's alone (any other path is recorded as left). A cleanup goroutine, started by the moves pass and never under its lock, walks every obligation through one transition table: observe the finish, wait 24 hours, reserve the box's key and `host/<hostname>`, mark `lock.json`, delete key by key with the conditions re-read, hand root a request (or skip it when nothing local remains), then persist `releasing`, restore the lifted tombstone, release and end. The mark never expires: only that `releasing`, or an operator's abandon after the page, lifts it. Supersession is checked at every step and ends the obligation through the same `releasing`, after any unanswered delete is answered under the mark and any root request is cancelled and acknowledged. The lifetime create and placement hold `host/<hostname>` in the same table, and so does a move's selection. The table is never deleted. |
| +**Architecture:** At `gone` the sealing source box writes `retired.json` under the move's op directory, for a tracked lifetime only: every version and delete marker (key, version id, ETag) under the normalised hibernate prefix (`lock.json` and `build-state.json` aside), each key's versions in deletion order, and the identities of the root-owned paths whose parent chain is root's alone (any other path is recorded as left). A cleanup goroutine, started by the moves pass and never under its lock, walks every obligation through one transition table: observe the finish, wait 24 hours, reserve the box's key and `host/<hostname>`, mark `lock.json`, delete key by key with the conditions re-read, hand root a request (or skip it when nothing local remains), then persist `releasing`, write this move's lifted tombstone back with its cleanup count raised (so no earlier body of the lock recurs), release and end. The mark never expires: only that `releasing`, or an operator's abandon after the page, lifts it. Supersession is checked at every step and ends the obligation through the same `releasing`, after any unanswered delete is answered under the mark and any root request is cancelled and acknowledged. The lifetime create and placement hold `host/<hostname>` in the same table, and so does a move's selection. The table is never deleted. |
| -**Tech Stack:** Go. `internal/offlinecleanup` (new leaf: the wire types, file identity and the layout check), `internal/moves` (host keys, the table's read and write, `PeekActive`, `PutTable` and `TableWrites`, the conflict relation, `Reserve`/`Release`, `RecordAway`), `internal/lease` (the cleaning mark, and its refusal in `AcquireTTL`, `Tombstone` and `LockState`), `internal/activator` (the wake's answer to a mark), `internal/host` (the dispatcher, the delete primitive, the report, the reaper, root's authority and the agent's root half, the root pass lock's wiring and gate, the box's armed-only table writers, the ungate verb's root half, the landing's refusal of the mark), `internal/release` (root's request handler, the root pass lock, the process census by executable, the agent check, the arming boundary with its nonce and the ungate, `PutOnce`), `internal/registry` (`WithHostFrozen`, `SetPlacementContext`, the remote dispatcher's deadline), `internal/console` (the create, placement, the reservation wrappers, recovery mode, the sweeps, the cleanup door, the admission barrier and the drain), the root package (`move.go`, the CLI's `--log`; `system.go` and `system_reservations.go`, the drain verb and the ungate verb). |
| +**Tech Stack:** Go. `internal/offlinecleanup` (new leaf: the wire types, file identity and the layout check), `internal/moves` (host keys, the table's read and write, `PeekActive`, `PutTable` and `TableWrites`, the conflict relation, `Reserve`/`Release`, `RecordAway`), `internal/lease` (the cleaning mark and its single-attempt lock writes, the lock body's `cleanups` count, and the mark's refusal in `AcquireTTL`, `Tombstone` and `LockState`), `internal/activator` (the wake's answer to a mark), `internal/host` (the dispatcher, the delete primitive, the report, the reaper, root's authority and the agent's root half, the root pass lock's wiring and gate, the box's armed-only table writers, the landing's refusal of the mark), `internal/release` (root's request handler, the root pass lock, the process census by executable, the agent check, the arming boundary with its nonce, `PutOnce`), `internal/registry` (`WithHostFrozen`, `SetPlacementContext`, the remote dispatcher's deadline), `internal/console` (the create, placement, the reservation wrappers, recovery mode, the sweeps, the cleanup door, the admission barrier and the drain), the root package (`move.go`, the CLI's `--log`; `system.go` and `system_reservations.go`, the drain verb). |
| **Spec:** `docs/superpowers/specs/2026-10-21-self-serve-offline-moves-design.md` §7.6, §7.1 (the `cleanup` reservation), §8 (the cleanup pages), §9 (rollout), §11. Stated deviations: placement and the lifetime create hold the hostname key (Paul, 2026-10-22, amadan #343); deletion by version id (§7.6 amendment, Paul's question 9); the rest are in the Self-review notes. |
| ## Global Constraints |
| -- **Arming and rollout.** The box acquires a new cleanup reservation, or writes the reservation table at all, only when it is **armed** (`cfg.CleanupArmed`): `CARLOS_OFFLINE_CLEANUP=on` in its environment (read by `armOffline` when the agent starts, from `/etc/carlos/host.env`, the `EnvironmentFile` of the agent's unit, `internal/host/units/carlos-edge.service:18`; root's adopt units read the same file, `carlos-adopt.service:16`, `carlos-stage.service:30`, `carlos-restart.service:31`, `carlos-config-sync.service:24`, and Task 6b's gate reads the switch from it), on an `OfflineCapable` box whose judge is set (`cfg.Judge != nil`, i.e. host claims off, `internal/host/boxjudge.go:21-26`), with an agent that does not run as root. **Cleanup requires an agent that does not run as root (V1).** The supported topology is the shipped `carlos-edge.service`: `User=carlos`, `Group=carlos`, `NoNewPrivileges=yes`, `ProtectSystem=strict` with `ReadWritePaths=/data /run/carlos` (`internal/host/units/carlos-edge.service:28-37`), so the agent cannot write a root-only directory, and every root publisher of a live link is a root adopt pass, which Task 6b's gate and arming boundary cover. The built code also runs an agent as root (`agentUnitsOption`, `agent.go:2213-2218`), and that agent's own adoption (the hourly pass at `agent.go:1326`, the fast sweep at `:2652`, the serial-triggered converge at `:2852`) publishes live links without the root pass lock and, with host claims off, without any other exclusion (`hostwrite`'s pass-through work, `internal/hostwrite/write.go:52`). So on such a box cleanup never arms: `armOffline` leaves `cfg.CleanupArmed` false when `agentRunsAsRoot()` (`agent.go:2206`) and the agent pages "cleanup needs the agent to run as its own user" once per start (Task 4d); and root's stage checks again, every pass, before it certifies an arming or touches a path (`release.AgentRunsUnprivileged`, Task 6b). `retired.json` is written at `gone` whatever the switch says, and an obligation past `finished` runs to its end when the box is disarmed, without writing the reservation table (Global Constraint "An unarmed box never writes the reservation table"). Order: boxes (they run plan 4 unarmed, so they write no table), then the console (plan 4 code everywhere; Global Constraint "No console rollback across the plan 4 boundary"), then `CARLOS_OFFLINE_CLEANUP=on` on the boxes, each running the shipped agent unit. Before the switch goes on in any deployment, a check on dev confirms version-id deletion on the versioned hibernate bucket (Task 4b's dev witness), and `aws s3api get-bucket-versioning` answers `Enabled` for the hibernate bucket of every region the switch goes on in (tofu sets it, `infra/modules/region-remote-host/main.tf:32-34`). **Two guarantees, kept apart (U5).** (i) The listing check at `gone` (Task 1) establishes that every inventoried version id is usable: a listing holding a `null` id writes no `retired.json`. It does not establish that versioning is enabled: a suspended bucket whose objects all predate the suspension lists only real ids, an empty prefix lists nothing, and both pass. (ii) The rollout's one-off `get-bucket-versioning` check establishes `Enabled`. The box roles cannot read the status (`s3:GetBucketVersioning` is granted only in the inventory statement of `infra/modules/region-host/main.tf:223`, and not at all in `region-remote-host`), so no code checks it. A suspended bucket with only real ids passes (i) and is caught by (ii). **If (ii) was skipped** on a suspended bucket, deletion stays safe: every delete the cleanup sends names a real, inventoried id and removes only that version. What changes is new writes: a later owner's write under the prefix (under its lease, so never while the mark stands) creates or overwrites the key's `null` version in place, which the inventory never holds. The cleanup never deletes it, and a key whose current entry it has become is `superseded` by condition (4). |
| +- **Arming and rollout.** The box acquires a new cleanup reservation, or writes the reservation table at all, only when it is **armed** (`cfg.CleanupArmed`): `CARLOS_OFFLINE_CLEANUP=on` in its environment (read by `armOffline` when the agent starts, from `/etc/carlos/host.env`, the `EnvironmentFile` of the agent's unit, `internal/host/units/carlos-edge.service:18`; root's adopt units read the same file, `carlos-adopt.service:16`, `carlos-stage.service:30`, `carlos-restart.service:31`, `carlos-config-sync.service:24`, and Task 6b's gate reads the switch from it), on an `OfflineCapable` box whose judge is set (`cfg.Judge != nil`, i.e. host claims off, `internal/host/boxjudge.go:21-26`), with an agent that does not run as root. **Cleanup requires an agent that does not run as root (V1).** The supported topology is the shipped `carlos-edge.service`: `User=carlos`, `Group=carlos`, `NoNewPrivileges=yes`, `ProtectSystem=strict` with `ReadWritePaths=/data /run/carlos` (`internal/host/units/carlos-edge.service:28-37`), so the agent cannot write a root-only directory, and every root publisher of a live link is a root adopt pass, which Task 6b's gate and arming boundary cover. The built code also runs an agent as root (`agentUnitsOption`, `agent.go:2213-2218`), and that agent's own adoption (the hourly pass at `agent.go:1326`, the fast sweep at `:2652`, the serial-triggered converge at `:2852`) publishes live links without the root pass lock and, with host claims off, without any other exclusion (`hostwrite`'s pass-through work, `internal/hostwrite/write.go:52`). So on such a box cleanup never arms: `armOffline` leaves `cfg.CleanupArmed` false when `agentRunsAsRoot()` (`agent.go:2206`) and the agent pages "cleanup needs the agent to run as its own user" once per start (Task 4d); and root's stage checks again, every pass, before it certifies an arming or touches a path (`release.AgentRunsUnprivileged`, Task 6b). `retired.json` is written at `gone` whatever the switch says, and an obligation past `finished` runs to its end when the box is disarmed, without writing the reservation table (Global Constraint "An unarmed box never writes the reservation table"). Order: boxes (they run plan 4 unarmed, so they write no table), then the console (plan 4 code everywhere; Global Constraint "No console rollback across the plan 4 boundary"), then `CARLOS_OFFLINE_CLEANUP=on` on the boxes, each running the shipped agent unit. **Root's gate is one-way (W2):** the first armed adopt pass on a box writes `gated`, and from then on, for the life of the box, every adopt pass there is gated whatever the switch says (a pass that cannot take the root pass lock publishes nothing that pass); no code removes `gated` or `quiesced` (Task 6b). Before the switch goes on in any deployment, a check on dev confirms version-id deletion on the versioned hibernate bucket (Task 4b's dev witness), and `aws s3api get-bucket-versioning` answers `Enabled` for the hibernate bucket of every region the switch goes on in (tofu sets it, `infra/modules/region-remote-host/main.tf:32-34`). **Two guarantees, kept apart (U5).** (i) The listing check at `gone` (Task 1) establishes that every inventoried version id is usable: a listing holding a `null` id writes no `retired.json`. It does not establish that versioning is enabled: a suspended bucket whose objects all predate the suspension lists only real ids, an empty prefix lists nothing, and both pass. (ii) The rollout's one-off `get-bucket-versioning` check establishes `Enabled`. The box roles cannot read the status (`s3:GetBucketVersioning` is granted only in the inventory statement of `infra/modules/region-host/main.tf:223`, and not at all in `region-remote-host`), so no code checks it. A suspended bucket with only real ids passes (i) and is caught by (ii). **If (ii) was skipped** on a suspended bucket, deletion stays safe: every delete the cleanup sends names a real, inventoried id and removes only that version. What changes is new writes: a later owner's write under the prefix (under its lease, so never while the mark stands) creates or overwrites the key's `null` version in place, which the inventory never holds. The cleanup never deletes it, and a key whose current entry it has become is `superseded` by condition (4). |
| - **No console rollback across the plan 4 boundary (V3).** Once a plan 4 console has written `control/offline-moves/active.json` in its new form (a body with `epoch` and `rev`, a host key, or a `create` or `place` entry), a console from before this plan is never run against that deployment again. A defect in the plan 4 console is fixed by rolling forward to a corrected build. On its roll the console is replaced, never run beside its predecessor against the same deployment bucket. **Why there is no rollback:** a console from before this plan reads a `create` or `place` entry as a malformed table (`moves/offline.go:584-587`; Task 2a's frozen parser pins it), its writes drop `epoch` and `rev`, and its last release deletes the table (`offlinereserve.go:478`), so it can make current again a body, and with it an ETag, that a late new-protocol attempt names. Rev 8 tried to close that with a drain, a count of unanswered writes, a clean-exit line and a 20-minute wait; review 8 showed the count is lost across a console crash and that no S3 deadline bounds an authenticated conditional write (items 4 and 5), so the plan does not allow the old console back at all, and nothing in it depends on a time bound. **The rollout needs no such rule in the other direction:** the boxes run plan 4 first, unarmed, and write nothing to the table (below), so until the console's roll the old console sees only its own writes; the plan 4 console then reads what it left (a metadata-free body parses, and is normalised before any acquisition, Task 2a). |
| -- **Rolling plan 4 back is disarming cleanup.** Unset `CARLOS_OFFLINE_CLEANUP` on every box and restart its agent. From then on no obligation acquires. One already reserved runs to its end without writing the table: it finishes its keys under the mark, answers any delete in flight, is superseded, or waits for root, which keeps serving it under the `gated` marker; one waiting for root ends when root answers, or when an operator cancels it by creating `<dataDir>/.offline-moves/<host>/<move>/root-cleanup.cancel` (root then acknowledges `cancelled`, Task 6b); a delete that cannot be answered keeps the mark and pages until it is answered or an operator abandons it after its page (Task 4c). A `finished` obligation whose reservation write was uncertain, or whose entry the run's read finds in the table, is given up (`releasing`, `Outcome = cancelled`, Task 4a row 2). An obligation in `releasing` lifts its mark and then, because an unarmed box never writes the table, keeps its entries, stays `releasing` and pages once, "cleanup of <host> waits for its reservation to be released: cleanup is off on this box" (Task 4a row 8). An operator then runs `carlos system reservations drain`, which releases a `cleanup` entry whose box's report shows the obligation `releasing` or `ended` (Task 7c); should the box be armed again, its own release writes once more, harmlessly (a release always writes, and holding nothing changes nothing). The console keeps its own `create` and `place` reservations, which settle themselves by plan 2's rules, Task 7a's recovery and Task 7b's settlement; a disarm changes nothing on the console. The `gated` and `quiesced` markers stay on each box that was armed, so its adopt passes keep failing closed on a root pass lock fault and keep serving the requests left; `carlos system cleanup-ungate` removes both once the box is disarmed and no request waits for root (Task 6b), which the disarm does not need. |
| +- **Rolling plan 4 back is disarming cleanup.** Unset `CARLOS_OFFLINE_CLEANUP` on every box and restart its agent. From then on no obligation acquires. One already reserved runs to its end without writing the table: it finishes its keys under the mark, answers any delete in flight, is superseded, or waits for root, which keeps serving it under the `gated` marker; one waiting for root ends when root answers, or when an operator cancels it by creating `<dataDir>/.offline-moves/<host>/<move>/root-cleanup.cancel` (root then acknowledges `cancelled`, Task 6b); a delete that cannot be answered keeps the mark and pages until it is answered or an operator abandons it after its page (Task 4c). A `finished` obligation whose reservation write was uncertain, or whose entry the run's read finds in the table, is given up (`releasing`, `Outcome = cancelled`, Task 4a row 2). An obligation in `releasing` lifts its mark and then, because an unarmed box never writes the table, keeps its entries, stays `releasing` and pages once, "cleanup of <host> waits for its reservation to be released: cleanup is off on this box" (Task 4a row 8). An operator then runs `carlos system reservations drain`, which releases a `cleanup` entry whose box's report shows the obligation `releasing` or `ended` (Task 7c); should the box be armed again, its own release writes once more, harmlessly (a release always writes, and holding nothing changes nothing). The console keeps its own `create` and `place` reservations, which settle themselves by plan 2's rules, Task 7a's recovery and Task 7b's settlement; a disarm changes nothing on the console. The `gated` and `quiesced` markers stay on each box that was armed, and nothing removes them (W2): its adopt passes keep failing closed on a root pass lock fault and keep serving every request left, including one an obligation publishes after the disarm (it was `publishing`, or paused just before its first drop, when the switch went off), because root's gate is judged by the markers, not by the switch (Task 6b, `TestDisarmedPublicationIsStillServed`). Arming the box again needs no step of its own: the switch goes on, and `gated`, its nonce and the certificate still stand, which is sound because every pass since the first arming was gated. |
| - **An unarmed box never writes the reservation table (V3; review 8, item 6).** The box's only table writers are row 2's `moves.Reserve`, row 8's `moves.Release` and the run's `w.tableWrites.Resend` (Tasks 4a and 4d), and each is reached only through `w.armedTable()`, which answers `errDisarmed` without a request when `!cfg.CleanupArmed`. Every other box read of the table, and root's authority read, is `moves.PeekActive`, which never bootstraps an absent table and never normalises a metadata-free one (Task 2a); `moves.ReadActive`'s bootstrap and `NextActiveBody`'s fresh epoch are reached only from the armed writers. So from a disarmed agent's restart on, its box sends no table write at all. What it sent while armed is settled by the next successful conditional write by anyone (Global Constraint "The table is never deleted"): a late acquisition attempt that lands puts an entry in the table that the unarmed obligation finds on its next run's read and gives up, as above (`TestUnarmedBoxNeverWritesTheTable`). |
| - **Tracked lifetimes only.** A cleanup obligation exists only for a tracked lifetime: the judged observation at `gone` is `LifetimeValid` with `Rec.Incarnation > 0` and a live slot at that incarnation. A record with incarnation 0 and no slot (an untracked lifetime, before the lazy migration; `provision.LifetimeOf`, `internal/provision/lifetime.go:127-130`, still answers it `LifetimeValid`), a migrating record, or any other verdict gets no `retired.json`, and its gone report proceeds exactly as before this plan; nothing is ever deleted for it, and its leftovers stay the runbook's. |
| - **Every data deletion names one inventoried version (spec §7.6 amendment, Paul's question 9).** `retired.json`'s `BucketInventory` records, for every key under the normalised prefix except `lock.json` and `build-state.json`, **every** version id the bucket lists at `gone`, delete markers included, not only the current one, with each entry's position in the listing (`Seq`) and the key's latest version id (`Latest`: the entry the listing marks `IsLatest`, the one current at `gone`). Every id is a real one: a `null` id (an object written while the bucket was unversioned or versioning was suspended, which a later suspended write overwrites in place) fails `lease.UsableVersionID`'s rule (`internal/lease/staging.go:414`), and a listing that holds one writes no `retired.json` (Task 1). The cleanup deletes only through `deleteInventoriedVersion` (Task 4b), which is `VersionedStore.DeleteVersion(ctx, bucket, key, versionID)` (`internal/lease/versioned.go:261-275`: an empty id is `ErrNoVersionID` without a request; success and `NoSuchVersion` are both nil) on `cfg.Versioned(ctx, Bucket, BucketRegion)`. A plain `Delete`, a marker-writing delete and an `If-Match` delete are never used. Within a key, the versions that were not current are deleted first, oldest first (`LastModified` ascending, ties by `Seq` descending: an order fixed at `gone` and recorded), and `Latest` last. **Condition (4) is the latest's:** before each deletion of a key's version, a fresh listing of the key must show `Latest` as its current entry; when `Latest` is absent, or another entry is current (a later version, or a marker the inventory does not hold), the key is `superseded` and nothing more of it is deleted. A version is **done** when a delete of it answers (success or no such version); a version other than `Latest` that the listing no longer shows is done (removed by someone with authority, or by a lifecycle rule for noncurrent versions). **The argument, once:** a delete that names a version id can remove only that version, whoever sends it and whenever it lands, so a late attempt, an SDK retry (`internal/lease/endpoint.go:103` keeps the default retries) or a crash-repeated request can never remove a version a later owner wrote, which carries another id. While `Latest` is current, deleting any other version changes nothing current, so a deletion sent after a check that found `Latest` current exposes no older version unless `Latest` stops being current between that check and the delete's landing. Nothing the platform runs can do that: a write needs the lease, which the mark refuses; a plain delete writes a marker on top, which is then current and hides everything beneath it; a lifecycle rule for noncurrent versions never removes a current one. Only a version-id delete of `Latest` by hand, concurrent with the cleanup, can, and the plan claims nothing about that case. **The one case a version id does not settle:** a later owner that keeps one of our versions instead of writing its own. A promotion does exactly this when the destination's ETag equals the staged one (`lease.MirrorPrefixKeepObserved`, `mirror.go:285`), and the target landing of a move into this region mirrors the same way. Both need the lease, or a lock the landing reads as free, and the mark refuses both for as long as it stands; it stands until every delete is answered. A delete is recorded (`InFlight`) before it is sent and sent again until it answers, **before** the obligation leaves `cleaning` (Task 4c), whatever has happened to the reservation meanwhile; an `InFlight` delete that cannot be answered keeps the mark and the entries the obligation still holds, and pages every 10 minutes, until it is answered or an operator abandons it after the page (Task 4c). Once a version is done, any attempt still in the network lands on nothing, because a version id is never reused. **The trade:** the retired replica is not recoverable from the bucket after the cleanup. It is the source copy of a finished move, and litestream's retention would purge it anyway. In exchange, the boundary is enforced by the server. `lock.json` and `build-state.json` are never inventoried and never deleted. |
| - **The table is never deleted.** `control/offline-moves/active.json` is created once, by the first writer, as `{v: 1, epoch: <16 random bytes, hex>, rev: 1}` with `If-None-Match: *` (`release.Store.Put` with `ifMatch ""`, `internal/release/store.go:794-798`); emptiness is a table with no entries. Every later write is a `Put` with `If-Match` the ETag it read, of a body whose `rev` is the read body's plus one and whose `epoch` is the read body's (a fresh one when the read body has none, as the console before this plan wrote), so no two bodies the table holds are equal and an ETag never comes back. That holds because from the first plan 4 write on every writer of the table writes this protocol (Global Constraint "No console rollback across the plan 4 boundary"). A conditional `Put` that read ETag E can land only while E is current, so once any later write succeeds, every earlier outstanding `Put` that read E can never land: **an uncertain acquisition is settled by the next successful conditional write by anyone.** A release always writes (rev + 1), even when its operation holds nothing, so its own success is the settlement its caller needs. **Acquisitions only against an epoch-bearing body (T4).** A write that adds entries (`moves.Reserve`, and so the box, the create and the placement; the console's `reserveBoxes`) reads through `moves.ReadForAcquire`: a body without `epoch` or `rev` (the old console's, left at the console's roll) is first normalised by a separate `Put` with `If-Match` the read ETag, of the same entries with a fresh `epoch` and `rev` 1, then read again, and only a body that carries both is ever an acquisition's condition. **Single attempts, resent until answered (F2).** Every write to the table goes through `moves.PutTable(ctx, s, tw, body, ifMatch)` (Task 2a), and nothing else writes the key (`TestEveryTableWriteGoesThroughPutTable`). It sends one HTTP request through `release.PutOnce`: `S3Store.PutOnce` passes `func(o *awss3.Options) { o.Retryer = aws.NopRetryer{} }` to `PutObject`, so the SDK's default retryer, which `S3Store.Put` keeps (`internal/release/store.go:787`; nothing in `internal/s3compat` or `rootaws` sets a retry option), never sends a copy the code did not see. An answer (success, or `release.ErrCASConflict`: S3's 412, or its 409 and 404 as `Put` maps them) is returned. A timeout, a transport error or a 5xx is no answer: the identical request (the same bytes, the same condition) is sent again after 1 s, 2 s, 4 s and then every 10 s, until it is answered or `ctx` ends. A 412 after an unanswered attempt is an answer: our attempt landed or another write did, and the caller's next read tells which. So an operation's acquisition and its release each end in an answer the operation saw, or in an uncertainty it persists before it gives up: the box's `ReserveUncertain` (or, after a crash inside `Reserve`, a `finished` state whose next armed visit reserves again and whose unarmed visit looks for its own entry), the create's pending allocation row, and the placement's entry, which the sweep fences and releases once it is unregistered. Its own later release, a successful write, then settles every earlier attempt by the rule above. Nothing in that settlement depends on the sending process still running, on a count, or on time. **The registration rule.** A write not yet answered is registered in `tw` (the console's `s.tableWrites`, the box worker's `w.tableWrites`) with its bytes, its condition and one owner. A registration is removed only by the goroutine that owns it, after that goroutine's final answer, under `tw`'s mutex. The caller owns its write while `PutTable` runs; when `ctx` ends with no answer, it gives the write up under the mutex, after its last attempt has returned from `PutOnce`, and the write is then unowned. `tw.Resend` (the console's sweep and drain, the armed box's run) claims unowned writes under the mutex, sends each claimed one once outside it, and under the mutex removes it on an answer or leaves it unowned again; a write owned by a caller or by another `Resend` is skipped. So no sweep, drain or run removes a registration while an attempt of it is inside `PutOnce`, and `tw.Unanswered()` counts every registered write whoever owns it. `tw` is process-local and nothing needs it after a crash: an attempt still in the network then is settled through its operation's persisted uncertainty, as above. A store without `PutOnce` (the store door's client, `internal/fleetapi/fleetapi.go:196`, whose door writes through a retrying client) gets `release.ErrNoSingleAttemptPut` and nothing is sent, so a box on the store door never reserves; the platform's boxes and consoles use `S3Store` or `DirStore` (`internal/host/agent.go:4419-4426`, `cmd/console/main.go:672-684`). |
| - **The prefix (N1).** `Prefix = activator.HibernatePrefix(cfg.OwnPrefix, host) + "/"` (`HibernatePrefix` is `path.Join`, no trailing slash, `internal/activator/activator.go:147`), computed once at `gone` and persisted; the listing is `ListVersions(bucket, Prefix)`; the excluded keys are exactly `Prefix + "lock.json"` and `Prefix + "build-state.json"`; the lease location is `strings.TrimSuffix(Prefix, "/")`. |
| - **The cleanup id** is `offlinecleanup.CleanupID(box, host, moveID) = "cu_" + hex(sha256(box + "\n" + host + "\n" + moveID))[:16]`: recomputable from the directory and the box, so a corrupt file, root and the agent all name the same obligation. |
| -- **The mark has no expiry (T1).** Owner `lease.CleaningOwner(id) = "cleaning:" + id`, written with `ExpiresAt = lease.CleaningExpiry` (`9999-12-31T00:00:00Z`, a time no clock reaches: `litestream.Lease` needs an expiry and `readLock` refuses a zero one, `internal/lease/tombstone.go:140`). It is a state, not a timed lease: written only by a conditional PUT from the lifted tombstone of this move (`moves.TombstoneOwner(MoveID)`, expired); never renewed (there is no reassert); and refused to every lease acquirer whatever its expiry says. litestream's own `AcquireLease` refuses it on the read that conditions its write (`litestream@v0.5.14/s3/leaser.go:91-101`, then the `If-Match` write at `:114`), and `lease.AcquireTTL` (so `Acquire`, `ReplicateWithLease` and the staging lease), `lease.Tombstone` and `lease.LockState` refuse, or report held, any `cleaning:` owner by its prefix, so even a mark whose expiry was edited by hand is never free (Task 3). It is lifted only (a) by the cleanup's own `releasing`, after every delete of the run is answered; (b) by supersession or a cancellation, and only when no delete is `InFlight`; or (c) by an operator's abandon (`cleanup.abandon`, Task 4c) after the page that an outstanding delete is unsettled. A crash, an outage of any length, a disarm or a lost reservation lifts nothing. **So no wake, landing, return or seal of any owner can take the lease while a delete the cleanup sent may still land**, whatever happens to the cleanup's process; that is what makes an in-place adoption of one of our versions (above) impossible during the cleanup. The mark is removed by a conditional PUT back to exactly that lifted tombstone (`{Generation: as read, ExpiresAt: 1970-01-01T00:00:00Z, Owner: moved:<move id>}`, what `LiftTombstone` writes), so the operator return's leftover set (spec §7.5) is unchanged after any cleanup. The owner alone proves no lease was taken since the mark, because every acquisition rewrites the object. **Its consumers get a finite retry (U4):** a wake refused by a mark answers `Retry-After: 30` and its own text, never the distance to the sentinel (Task 3). |
| +- **The mark has no expiry (T1).** Owner `lease.CleaningOwner(id) = "cleaning:" + id`, written with `ExpiresAt = lease.CleaningExpiry` (`9999-12-31T00:00:00Z`, a time no clock reaches: `litestream.Lease` needs an expiry and `readLock` refuses a zero one, `internal/lease/tombstone.go:140`). It is a state, not a timed lease: written only by a conditional PUT from the lifted tombstone of this move (`moves.TombstoneOwner(MoveID)`, expired); never renewed (there is no reassert); and refused to every lease acquirer whatever its expiry says. litestream's own `AcquireLease` refuses it on the read that conditions its write (`litestream@v0.5.14/s3/leaser.go:91-101`, then the `If-Match` write at `:114`), and `lease.AcquireTTL` (so `Acquire`, `ReplicateWithLease` and the staging lease), `lease.Tombstone` and `lease.LockState` refuse, or report held, any `cleaning:` owner by its prefix, so even a mark whose expiry was edited by hand is never free (Task 3). It is lifted only (a) by the cleanup's own `releasing`, after every delete of the run is answered; (b) by supersession or a cancellation, and only when no delete is `InFlight`; or (c) by an operator's abandon (`cleanup.abandon`, Task 4c) after the page that an outstanding delete is unsettled. A crash, an outage of any length, a disarm or a lost reservation lifts nothing. **So no wake, landing, return or seal of any owner can take the lease while a delete the cleanup sent may still land**, whatever happens to the cleanup's process; that is what makes an in-place adoption of one of our versions (above) impossible during the cleanup. The mark is removed by a conditional PUT of this move's lifted tombstone with its cleanup count raised (`{generation: as read, expires_at: 1970-01-01T00:00:00Z, owner: moved:<move id>, cleanups: as read + 1}`, Global Constraint "No lock body recurs"), never by reconstructing the body the mark replaced. The operator return's leftover set (spec §7.5) is unchanged after any cleanup: the return reads the lock's owner and whether it is a lifted tombstone (`internal/host/movesoffline_return.go:270`, `console/offlinereturn.go:465`), never its count. The owner alone proves no lease was taken since the mark, because every acquisition rewrites the object. **Its consumers get a finite retry (U4):** a wake refused by a mark answers `Retry-After: 30` and its own text, never the distance to the sentinel (Task 3). |
| +- **No lock body recurs (W1; review 9, finding 1).** A conditional write that read ETag E lands whenever E is current, and S3's ETag of a single-part `PutObject` is a digest of its bytes (`leasetest` computes it the same way, `internal/lease/leasetest/s3server.go:216`), so writing a lock body again makes every outstanding write that read it valid again. Rev 9 put the lifted tombstone back exactly, so a mark attempt still in the network after the unmark could land and leave a mark that nothing would ever lift. Now: (i) **the body** is `lease.lockBody`, `{generation, expires_at, owner, cleanups}`, the last optional (`omitempty`, so every body written today is unchanged byte for byte). `readLock` (`internal/lease/tombstone.go:120-148`, `json.Unmarshal` at `:137`), `LockState` (`internal/lease/lockstate.go:72`) and litestream's `readLease` (`litestream@v0.5.14/s3/leaser.go:217-218`, a `json.Decoder` without `DisallowUnknownFields`) decode into `litestream.Lease` and ignore it, so `Acquire`, the wake and every lock reader parse as before, and an acquisition's write drops it (litestream writes its own three fields, `leaser.go:108-112`, `:146-150`). (ii) **The writes:** `MarkCleaning` writes `{generation: as read, expires_at: CleaningExpiry, owner: cleaning:<id>, cleanups: as read}` over this move's lifted tombstone; `UnmarkCleaning` writes `{generation: as read, expires_at: 1970-01-01T00:00:00Z, owner: moved:<move id>, cleanups: as read + 1}` over our mark **or** over this move's lifted tombstone with no mark (the settling write, always made), conditioned on the ETag it read; `LiftTombstone` writes through `lockBody` too, carrying the count as read. (iii) **Why no body recurs:** a lifted tombstone of move X at generation g is first written once, by the lift of X's live tombstone (written by X's seal, `internal/host/movesoffline_seal.go:431`, and lifted at its finish or abort, `:1161`, `:1217`; `Reassert` refuses a lifted tombstone, `tombstone.go:242`, and the seal tombstones only a move in its seal). Every later lifted body of X at g is an unmark's, whose count is one more than the body it read; a mark's body names its cleanup and carries the count of the lifted tombstone it replaced; every other writer of the lock writes a live expiry or deletes the object. So the lifted bodies of X at g carry strictly rising counts, each mark body sits between two of them, and once a later write lands, no body that an attempt of the cleanup was conditioned on is ever current again. (iv) **Settlement:** every mark attempt names the ETag of a lifted tombstone of this move; once an unmark of the cleanup lands, that ETag is gone for good, and every obligation that may have sent a mark reaches `releasing`, whose unmark always writes. So no mark attempt lands after the unmark, and an unmark's own late attempts, which name our mark's ETag or a lifted tombstone's, are retired by the same argument. (v) **Liveness:** the mark's and the unmark's writes are single attempts (`PutObject` with `func(o *awss3.Options) { o.Retryer = aws.NopRetryer{} }`, as `S3Store.PutOnce`; litestream's `S3API` takes the option, `leaser.go:38`), each under `lockAttemptTimeout` (2 s), sent again identically after 1 s, 2 s, 4 s and then every 10 s until answered or the operation's 10 s ends (`ErrLockWriteUncertain`). An answer is a success, a 412 or a 404. A 412 is answered by reading the lock: the mark that finds our mark is settled; the unmark that finds this move's lifted tombstone with a higher count than it read is settled; anything else is `ErrTombstoneChanged`, and the obligation's next visit reads afresh. Nothing persists the uncertainty: the next visit's mark reads the lock first, and a supersession's unmark settles it by its write. A wake that read the lifted tombstone just before a settling unmark loses its conditional write and is answered `Retry-After` of at least 1 s (`internal/activator/backend_exec.go:705-708`), as it would be beside rev 9's unmark of a mark. |
| - **The writers the mark excludes** (each writer's check, by file): the wake (`internal/activator/backend_exec.go:701`, `lease.Acquire` in `ExecBackend.Acquire`, step 1 of `doWake` in `activator.go`, before the restore, the staged-replica promotion `promoteStagedReplica` at `backend_exec.go:3397`, `BeforeExec`'s build-state write and litestream replication, all of which run under that lease); the legacy final checkpoint (`internal/host/movespass.go:321-328`, `lease.ReplicateWithLease`, which acquires); the offline seal's checkpoint (`internal/host/movesoffline_seal.go:417`, the same `Checkpoint`) and its tombstone (`movesoffline_seal.go:431`, `lease.Tombstone`, which never replaces a live lease of another owner, so the seal's carry at `:469` is not reached). Not excluded by the mark and harmless: the legacy cut's request carry (`movespass.go:1594`) and the offline seal's carry write only `build-state.json`, which is not inventoried; the seed rollback and finish (`seedwiring.go:247`, `console/seedfinish.go:254`) only delete, by key (a marker, which condition (4) reads as another current entry), and the finish refuses a held lock (`seedfinish.go:255-259`, which `LockState` answers held for a mark); the target landing (`movesoffline_land.go:611`, `:1373`) belongs to a move into this region, which holds `host/<hostname>`, and its lease step waits on a `cleaning:` owner whatever its expiry (`movesoffline_land.go:612-630`, Task 3); the operator return's start door refuses any lock that is not this move's lifted tombstone (`console/offlinereturn.go:465`). `TestHibernatePrefixSitesAreInventoried` (Task 3) pins this list. |
| - **Reservation keys and the one conflict relation.** Keys are a box id (today's bare label, unchanged), the sentinel `"*"`, or `moves.HostKey(host) = "host/" + host`. Kinds: `move`, `cleanup`, `create`, `place`. An entry is valid with a known kind, a non-empty id and host, a non-zero `reserved_at`, and, for `create` and `place`, a non-empty account and app; an entry under a host key names that host (`Host == strings.TrimPrefix(key, "host/")`). An existing entry of another `(Kind, ID)` conflicts with a requested set of keys when (a) it holds a requested key; or (b) one side holds `"*"` and the other holds a box key; or (c) the requester is the console with `cfg.SelfServeMoves != "offline"` and both are of kind `move` (plan 2's one-move-at-a-time rule, now between moves only). Host keys never meet (b) or (c). Every move selection (self-serve and operator) holds `host/<hostname>` beside its box keys or its sentinel; a cleanup holds `[cfg.BoxID, host/<hostname>]` with one entry; a create holds `[host/<hostname>]`; a placement holds `host/<h>` for every instance of the app, one entry per key with `Host = h` and the placement's account and app. |
| - **A cleanup entry is never reclaimed by another reserver.** Its own box resumes it after any crash (its own retry is accepted); a box that never returns leaves an entry the console pages after 6 hours (operator repair). Plan 2's `Server.cleanupReports` and `cleanupReportEvidence` (`offlinereserve.go:95-104`) are deleted. |
| - **The conditions at every key**, (1) to (3) re-read before a key's first delete and (4) before each of its deletes: (1) `moves.RecordAway(obs, MoveID, Incarnation, From) == AwayYes`: the judged observation is `LifetimeValid` (the judge's verdict that the slot vouches for the record) at `Incarnation`, with no move, `LastMove.ID == MoveID && LastMove.Outcome == "done"`, and `Region != From`; (2) `lock.json`'s owner is `CleaningOwner(ID)` (its expiry is the sentinel; nothing about time is read); (3) the box's registry has no row for the host (`w.offlineRow(host)`, `movesoffline.go:996`); (4) `ListVersions(bucket, key)` filtered to `Key == key` shows the key's inventoried `Latest` as its current entry. A read error in any: the obligation stops for this run. (1), (2) or (3) failing definitely: **supersession** (spec §7.6's "superseded as a whole"; never an indefinite refusal). (4) failing (`Latest` absent, or a later version or a marker the inventory does not hold current): that key `superseded`, nothing more of it sent, and a page. Conditions (1) and (3) are also checked at every other step of the lifecycle (Task 4a's `supersededBy`). |
| - The cleanup writes nothing to the registry, the edges file or the offline view, so plan 3's holding entries (built from held rows and the view) are untouched; root's freeze holds the registry's write lock for an `fstatat` and an `unlinkat`, as held publication's does for its swap. |
| - **Root's contract.** Root removes a path only (i) when the path's parent directory and every ancestor up to `/` is a directory owned by uid 0 with no group or other write bit (`offlinecleanup.RootOnlyChain`; otherwise the path is `kept`, reason "writable layout", because no exclusion against a writer that shares the agent's uid is possible there by construction); (ii) **as a stage of the root adoption pass** (`runAdoptUnderRootRelease`), after release adoption and named recovery and after the held stage, while the pass holds the adopt-pass lock and the root pass lock (`release.LockRootPass`, `<optDir>/root-cleanup/pass.lock` on a root-only chain, which every adopt pass takes in every mode before its first publisher; on a box where cleanup can run, Task 6b's gate, a pass that cannot take it publishes nothing that pass, so there every root publisher of the live link runs under it; and root's step runs only once `quiesced` certifies the current arming's nonce, written at a moment after that arming at which no other root adopt process was alive, so no pass admitted before arming is still publishing, and only in a pass whose check found the agent running as its own user, so no root-run agent publishes beside it, Task 6b's arming boundary and agent check), and then inside `reg.WithHostFrozen(host, …)` (no row for the host) and the host's held-stage lock (`withHeldStageLock`, `movesheldlock.go:68`), in the order adopt pass lock > root pass lock > the registry's write lock > the held-stage lock (`movesheldlock.go:26-31` gains the root pass lock); so no other root publisher (adoption, named recovery, `PointLive`) runs on the box while root's step runs, and the agent's publishers are excluded by the freeze and the held-stage lock (the agent, running as its own user, cannot write a root-only directory at all); (iii) after the authority callback has answered yes for this request in this pass and `root-cleanup.cancel` is absent; (iv) when `fstatat(parent, name, AT_SYMLINK_NOFOLLOW)` on the opened parent shows the recorded identity (device, inode, file type, link target); then `unlinkat(parent, name)`. An entry with another identity is `kept`, never removed. Every step is journalled as intent and then result in root's own directory, which is used only when it passes the same root-only check; when it does not, root writes nothing there (Task 6b). Nothing inside the held-stage lock reads the network or takes a lock. |
| -- Timings: 24 h after `FinishedAt`; per run 60 s (`cleanupRunDeadline`), 10 s per operation, at most 50 keys per obligation; pages every 10 min while root has not answered or has not acknowledged a cancel, and while a delete has waited over 10 min for its answer (`InFlight`), hourly once root has not answered for an hour and its request directory's ancestry is not root-only (`rootRefusedAfter`), hourly for a corrupt obligation and for a record unreadable for over an hour; the console pages a `cleanup` entry older than 6 h and a `create` entry with no allocation row older than `offlineLockGrace` (2 min); a placement runs under one 15 s deadline (`instanceStoreTimeout`), and an unregistered `place` entry older than `offlineLockGrace` is settled by the sweep; the op directory goes 7 d after `EndedAt`; root removes its own directory 30 d after its acknowledgement; a wake refused by a cleaning mark answers `Retry-After: 30` (`cleaningRetryAfter`); an unanswered table write is resent after 1, 2 and 4 s and then every 10 s while its caller waits, and then once per console sweep, drain round or armed box run; an obligation whose release waits on a disarmed box pages once (`release-withheld`); a root agent's refusal to arm pages once per agent start. |
| -- Census: `internal/provision/testdata/plan14-root-reads-self-serve-moves-4.tsv`, loaded by a new `loadRootReadInventory(root+"/internal/provision/testdata/plan14-root-reads-self-serve-moves-4.tsv", &inv, true)` block in `internal/provision/plan14_root_reads_test.go` after plan 3's block (`:1383`) and its seal check and before `compareRootReads`, `rev3=PROPOSED-self-serve-moves-4`, rows at the `authority` class with evidence `E2,J3,J4` for the destructive readers (the box's record, registry, lock and versions reads and the abandon file; root's drop, its own request copy and journal, the gate marker and its nonce, the certificate `quiesced`, the process census's reads of `/proc/<pid>/exe`, `status` and `cmdline`, the agent unit's `User=` through `systemctl show`, `active.json` (through `moves.PeekActive`), record, lock, registry-under-freeze, the layout `lstat` chain, the root pass lock's chain and the `fstatat` of each path); `TestPlan9B8RecordWriteCensus` (`plan9_b8_census_test.go:372`) takes `plan14SelfServeMoves4WriteCensusInventory()` wrapping `plan14SelfServeMoves3WriteCensusInventory()` (`self_serve_moves3_census_test.go:22`) with every new or changed bucket writer as an added or re-pinned unit (`moves.PutTable`, through which every one of the following writes, and the resends of `moves.TableWrites`; `moves.ReadActive`'s bootstrap, `moves.ReadForAcquire`'s normalisation write, `moves.Reserve`, `moves.Release`, `reserveBoxes`, `releaseBoxes`, the drain's settling `Release` and its release of a cleanup entry on its box's report, the cleanup report writer, the cleanup's `DeleteVersion` through `deleteInventoriedVersion`; the placement settle, the place sweep and the drain reach `fenceRecord`'s existing record unit); both green. |
| -- Copy rows (pending Paul's review; each task that emits one appends its row, at the next free `MV-O` id, status pending, to `docs/superpowers/plans/2026-10-06-moves-phase1-copy.md` and its id to `movesCopyIDs` in `approved_moves_copy_test.go`, in the same commit): "the previous instance at this hostname is still being cleaned up. try again later"; "an instance of this app is still being cleaned up. try again later"; "an instance of this app is moving. try again later"; "could not reserve the app's instances. try again"; "this console is draining its reservations. try again later"; the wake's 503 body "instance is being cleaned up after a move. retry shortly" (Task 3; it begins with `instance`, which the scheduler's `deliverVia` reads as a wake refusal, `internal/host/schedule_wiring.go:514`); page subjects "cleanup of <host> waits for root", "cleanup of <host> skipped <key>: another version is current", "cleanup of <host> waits for root to acknowledge its cancellation", "cleanup of <host> cannot read its record", "a cleanup has held <host> for over six hours", "cleanup of <host>: root refused the request directory's ancestry", "cleanup of <host> lost its mark before a delete of <key> was answered", "cleanup of <host>: an outstanding delete is unsettled; the host's replica stays marked", "cleanup of <host> waits for its reservation to be released: cleanup is off on this box", each with the body "run carlos move offline status --host <host> --log --cleanup <cleanup id> to see what it waits for" (the console's "held for over six hours" page names the entry's id the same way); the agent's page "cleanup needs the agent to run as its own user" with the body "carlos agent runs as root on <box>. cleanup stays off on this box until carlos-edge.service runs it as its own user, as shipped." (Task 4d), and root's acknowledgement reason of the same words (Task 6b); the CLI's lines of Tasks 5b, 6b (`cleanup-ungate`) and 7c; the `--log` paragraph and the `reservations drain` and `cleanup-ungate` lines of `docs/site/cli.md`. No tenant names in tests (`shop.example.com`, account `bab`, app `shop`). |
| +- Timings: 24 h after `FinishedAt`; per run 60 s (`cleanupRunDeadline`), 10 s per operation, at most 50 keys per obligation; pages every 10 min while root has not answered or has not acknowledged a cancel, and while a delete has waited over 10 min for its answer (`InFlight`), hourly once root has not answered for an hour and its request directory's ancestry is not root-only (`rootRefusedAfter`), hourly for a corrupt obligation and for a record unreadable for over an hour; the console pages a `cleanup` entry older than 6 h and a `create` entry with no allocation row older than `offlineLockGrace` (2 min); a placement runs under one 15 s deadline (`instanceStoreTimeout`), and an unregistered `place` entry older than `offlineLockGrace` is settled by the sweep; the op directory goes 7 d after `EndedAt`; root removes its own directory 30 d after its acknowledgement; a wake refused by a cleaning mark answers `Retry-After: 30` (`cleaningRetryAfter`); an unanswered table write is resent after 1, 2 and 4 s and then every 10 s while its caller waits, and then once per console sweep, drain round or armed box run; an unanswered lock write of the mark or the unmark is resent on the same schedule, each attempt under 2 s (`lockAttemptTimeout`), within its operation's 10 s; an obligation whose release waits on a disarmed box pages once (`release-withheld`); a root agent's refusal to arm pages once per agent start. |
| +- Census: `internal/provision/testdata/plan14-root-reads-self-serve-moves-4.tsv`, loaded by a new `loadRootReadInventory(root+"/internal/provision/testdata/plan14-root-reads-self-serve-moves-4.tsv", &inv, true)` block in `internal/provision/plan14_root_reads_test.go` after plan 3's block (`:1383`) and its seal check and before `compareRootReads`, `rev3=PROPOSED-self-serve-moves-4`, rows at the `authority` class with evidence `E2,J3,J4` for the destructive readers (the box's record, registry, lock and versions reads and the abandon file; root's drop, its own request copy and journal, the gate marker and its nonce, the certificate `quiesced`, the process census's reads of `/proc/<pid>/exe`, `status` and `cmdline`, the agent unit's `User=` through `systemctl show`, `active.json` (through `moves.PeekActive`), record, lock, registry-under-freeze, the layout `lstat` chain, the root pass lock's chain and the `fstatat` of each path); `TestPlan9B8RecordWriteCensus` (`plan9_b8_census_test.go:372`) takes `plan14SelfServeMoves4WriteCensusInventory()` wrapping `plan14SelfServeMoves3WriteCensusInventory()` (`self_serve_moves3_census_test.go:22`) with every new or changed bucket writer as an added or re-pinned unit (`moves.PutTable`, through which every one of the following writes, and the resends of `moves.TableWrites`; `moves.ReadActive`'s bootstrap, `moves.ReadForAcquire`'s normalisation write, `moves.Reserve`, `moves.Release`, `reserveBoxes`, `releaseBoxes`, the drain's settling `Release` and its release of a cleanup entry on its box's report, the cleanup report writer, the cleanup's `DeleteVersion` through `deleteInventoriedVersion`, the lock writes of `lease.MarkCleaning` and `lease.UnmarkCleaning`, and the re-pinned `lease.LiftTombstone`; the placement settle, the place sweep and the drain reach `fenceRecord`'s existing record unit); both green. |
| +- Copy rows (pending Paul's review; each task that emits one appends its row, at the next free `MV-O` id, status pending, to `docs/superpowers/plans/2026-10-06-moves-phase1-copy.md` and its id to `movesCopyIDs` in `approved_moves_copy_test.go`, in the same commit): "the previous instance at this hostname is still being cleaned up. try again later"; "an instance of this app is still being cleaned up. try again later"; "an instance of this app is moving. try again later"; "could not reserve the app's instances. try again"; "this console is draining its reservations. try again later"; the wake's 503 body "instance is being cleaned up after a move. retry shortly" (Task 3; it begins with `instance`, which the scheduler's `deliverVia` reads as a wake refusal, `internal/host/schedule_wiring.go:514`); page subjects "cleanup of <host> waits for root", "cleanup of <host> skipped <key>: another version is current", "cleanup of <host> waits for root to acknowledge its cancellation", "cleanup of <host> cannot read its record", "a cleanup has held <host> for over six hours", "cleanup of <host>: root refused the request directory's ancestry", "cleanup of <host> lost its mark before a delete of <key> was answered", "cleanup of <host>: an outstanding delete is unsettled; the host's replica stays marked", "cleanup of <host> waits for its reservation to be released: cleanup is off on this box", each with the body "run carlos move offline status --host <host> --log --cleanup <cleanup id> to see what it waits for" (the console's "held for over six hours" page names the entry's id the same way); the agent's page "cleanup needs the agent to run as its own user" with the body "carlos agent runs as root on <box>. cleanup stays off on this box until carlos-edge.service runs it as its own user, as shipped." (Task 4d), and root's acknowledgement reason of the same words (Task 6b); the CLI's lines of Tasks 5b and 7c; the `--log` paragraph and the `reservations drain` line of `docs/site/cli.md` (rev 10 removes `cleanup-ungate` and its lines, W2). No tenant names in tests (`shop.example.com`, account `bab`, app `shop`). |
| - Local tests: touched packages with `-run`; CI runs the full suite and `go test -tags e2e ./internal/e2e/...`. |
| ## Review Focus |
| -1. The F1 and N4 interleavings by version id: a reconcile paused with an old observation mints the source row after the cleanup marked the lock, and a wake through `ExecBackend.Acquire` then tries the lease: it gets `HeldError` naming `cleaning:<id>`, no version is written under the prefix, the next key's condition (3) supersedes, the lifted tombstone is restored, the reservation released, and the wake then succeeds. A delete sent before the supersession and still unanswered is sent again under the mark and answered before `releasing` is written; an earlier attempt of it delivered after the release lands on nothing. Two attempts of one delete (an SDK retry) remove one version. The mark has no expiry: a cleanup that stops with a delete in flight and stays down for 25 hours still holds it, a legacy return's wake is refused with `HeldError` naming `cleaning:<id>`, and the resumed cleanup resends the same version id, gets its answer and only then releases; a delete that never answers keeps the mark and pages every 10 minutes until an operator abandons it. No deletion is sent unless a fresh listing found the key's inventoried latest current, so no run leaves a version older than it current; a key whose latest was removed by hand is superseded, not deleted further. A promotion back into the region that kept an inventoried version by its equal ETag finds it intact after the cleanup ended. A wake refused by the mark answers `Retry-After: 30`, so a client polling a cleanup that ends seconds later is back within half a minute, not 292 years (Tasks 3, 4b and 4c, `TestCleaningMarkAnswersAFiniteRetry`, `TestCleaningMarkIsRefusedWhateverItsAge`, `TestMarkOutlivesACrashWithADeleteInFlight`, `TestUnanswerableDeleteHoldsTheMarkAndPages`, `TestAdmittedReconcileAndWakeCannotWriteUnderTheMark`, `TestUnansweredDeleteIsAnsweredBeforeTheUnmark`, `TestRetriedAttemptsRemoveOnlyTheirVersion`, `TestDeletionNeverExposesAnOlderVersion`, `TestLatestRemovedByHandSupersedesTheKey`, `TestPromotionThatKeptAnInventoriedVersionKeepsIt`). |
| -2. A crash after every write and every side effect of the lifecycle (after `Reserve` before `cleaning`, after `cleaning` before the mark, after the mark before `MarkedAt`, after `InFlight` is written before its delete, after a delete before `Done` moves, after the last key before `publishing`, after `releasing` before the unmark, after the unmark before the release, after the release before `ended`), restart, completion: no first delete of a version is sent unless both entries and the mark are found to be this cleanup's (a resend needs only the mark), an entry an operator removed or a mark replaced turns the obligation `superseded`, an unanswered delete is answered under the mark first (only a mark replaced by hand leaves it `unsettled`), nothing is re-acquired from `cleaning`, and the outcome is the table's (Task 4a, `TestLifecycleResumesAtEveryBoundary`). |
| -3. Root under attack and crash: a shared-uid writer plants an acknowledgement, a request with another id, a changed copy of the request and a cancel in the agent's directory, and replaces `/data/adopt-pass.lock` with a symlink so a second pass runs without that lock; on an armed box a second pass's root pass lock fails for a root-level fault; an unarmed pass whose two locks failed is paused mid-publication while a later pass arms the box (review 7's finding 2); the agent's unit is changed to run it as root, or a root `carlos agent` of the installed binary is running; a `quiesced` with an old nonce, or with rev 8's `(dev, inode)` of the current `gated`, is left beside a fresh `gated` (review 8, item 2); the ungate crashes between its two removals; a uid-0 pass runs `adopt` through a renamed path, a hard link and a deleted image of the installed binary; a publication replaces the live link between root's reference check and its `fstatat`; a second adopt pass starts between root's `fstatat` and its `unlinkat`; a route is upserted during the freeze; the link's parent chain is writable; root's request directory's ancestry is writable; root crashes after each journal line and each syscall: root removes only an entry carrying the recorded identity, the replacement survives, the second pass waits on the root pass lock until the step returns, the pass whose root pass lock failed publishes nothing, root's step does not run until the paused fail-open pass has exited and then finds its replacement link kept, root cancels every request with "cleanup needs the agent to run as its own user" and touches nothing while a root agent can run, a stale certificate is void and the census is taken again, an interrupted ungate leaves no certificate, the census counts each form of the installed binary, root creates no directory under an ancestry that has not passed, the upsert waits for the freeze, the writable layout is never touched, a rejected ancestry gets no write, every intent resolves to removed, kept or absent, root finds its own taken request on restart, and the forged files change nothing but stopping (Tasks 6b and 6c, `TestRootRemovesOnlyTheRecordedIdentity`, `TestNoPublisherRunsBetweenFstatatAndUnlinkat`, `TestGatedPassPublishesNothingWithoutTheRootPassLock`, `TestFirstArmingWaitsForAFailOpenPassToEnd`, `TestRootStepRefusesARootAgent`, `TestStaleCertificateIsVoid`, `TestArmingWritesAFreshNonce`, `TestUngateRemovesTheCertificateFirst`, `TestAdoptPassesCensus`, `TestLockRootPassChecksAncestryBeforeCreating`, `TestWritableLayoutIsNeverRemoved`, `TestRejectedAncestryGetsNoWrite`, `TestRootJournalResolvesEveryIntent`, `TestRootFindsItsOwnRequestsOnRestart`, `TestForgedAgentFilesAreHarmless`). |
| +1. The F1 and N4 interleavings by version id: a reconcile paused with an old observation mints the source row after the cleanup marked the lock, and a wake through `ExecBackend.Acquire` then tries the lease: it gets `HeldError` naming `cleaning:<id>`, no version is written under the prefix, the next key's condition (3) supersedes, the lifted tombstone is written back with a raised cleanup count, the reservation released, and the wake then succeeds. A delete sent before the supersession and still unanswered is sent again under the mark and answered before `releasing` is written; an earlier attempt of it delivered after the release lands on nothing. Two attempts of one delete (an SDK retry) remove one version. The mark has no expiry: a cleanup that stops with a delete in flight and stays down for 25 hours still holds it, a legacy return's wake is refused with `HeldError` naming `cleaning:<id>`, and the resumed cleanup resends the same version id, gets its answer and only then releases; a delete that never answers keeps the mark and pages every 10 minutes until an operator abandons it. No deletion is sent unless a fresh listing found the key's inventoried latest current, so no run leaves a version older than it current; a key whose latest was removed by hand is superseded, not deleted further. A promotion back into the region that kept an inventoried version by its equal ETag finds it intact after the cleanup ended. A wake refused by the mark answers `Retry-After: 30`, so a client polling a cleanup that ends seconds later is back within half a minute, not 292 years. A mark attempt times out and stays in the network while its resend marks; the cleanup ends and unmarks; the late attempt then arrives and gets 412, because the unmark wrote a lifted tombstone with a higher count and a new ETag, and the lock stays lifted. An obligation whose mark went unanswered and whose reservation an operator then removed supersedes, and its unmark writes the lifted tombstone with a higher count although no mark is there, so the late attempt gets 412 too (review 9, finding 1). (Tasks 3, 4a, 4b and 4c, `TestDelayedMarkCannotLandAfterTheUnmark`, `TestUnmarkSettlesAnUncertainMark`, `TestSupersessionAfterAnUncertainMarkSettlesIt`, `TestCleaningMarkAnswersAFiniteRetry`, `TestCleaningMarkIsRefusedWhateverItsAge`, `TestMarkOutlivesACrashWithADeleteInFlight`, `TestUnanswerableDeleteHoldsTheMarkAndPages`, `TestAdmittedReconcileAndWakeCannotWriteUnderTheMark`, `TestUnansweredDeleteIsAnsweredBeforeTheUnmark`, `TestRetriedAttemptsRemoveOnlyTheirVersion`, `TestDeletionNeverExposesAnOlderVersion`, `TestLatestRemovedByHandSupersedesTheKey`, `TestPromotionThatKeptAnInventoriedVersionKeepsIt`). |
| +2. A crash after every write and every side effect of the lifecycle (after `Reserve` before `cleaning`, after `cleaning` before the mark, after a mark attempt that went unanswered, after the mark before `MarkedAt`, after `InFlight` is written before its delete, after a delete before `Done` moves, after the last key before `publishing`, after `releasing` before the unmark, after the unmark before the release, after the release before `ended`), restart, completion: no first delete of a version is sent unless both entries and the mark are found to be this cleanup's (a resend needs only the mark), an entry an operator removed or a mark replaced turns the obligation `superseded`, an unanswered delete is answered under the mark first (only a mark replaced by hand leaves it `unsettled`), nothing is re-acquired from `cleaning`, and the outcome is the table's (Task 4a, `TestLifecycleResumesAtEveryBoundary`). |
| +3. Root under attack and crash: a shared-uid writer plants an acknowledgement, a request with another id, a changed copy of the request and a cancel in the agent's directory, and replaces `/data/adopt-pass.lock` with a symlink so a second pass runs without that lock; on an armed box a second pass's root pass lock fails for a root-level fault; an unarmed pass whose two locks failed is paused mid-publication while a later pass arms the box (review 7's finding 2); the agent's unit is changed to run it as root, or a root `carlos agent` of the installed binary is running; a `quiesced` with an old nonce, or with rev 8's `(dev, inode)` of the current `gated`, is left beside a fresh `gated` (review 8, item 2); the box is disarmed while an obligation is paused immediately before its first drop; a uid-0 pass runs `adopt` through a renamed path, a hard link and a deleted image of the installed binary; a publication replaces the live link between root's reference check and its `fstatat`; a second adopt pass starts between root's `fstatat` and its `unlinkat`; a route is upserted during the freeze; the link's parent chain is writable; root's request directory's ancestry is writable; root crashes after each journal line and each syscall: root removes only an entry carrying the recorded identity, the replacement survives, the second pass waits on the root pass lock until the step returns, the pass whose root pass lock failed publishes nothing, root's step does not run until the paused fail-open pass has exited and then finds its replacement link kept, root cancels every request with "cleanup needs the agent to run as its own user" and touches nothing while a root agent can run, a stale certificate is void and the census is taken again, the request published after the disarm is served under the markers, which nothing removes, the census counts each form of the installed binary, root creates no directory under an ancestry that has not passed, the upsert waits for the freeze, the writable layout is never touched, a rejected ancestry gets no write, every intent resolves to removed, kept or absent, root finds its own taken request on restart, and the forged files change nothing but stopping (Tasks 6b and 6c, `TestRootRemovesOnlyTheRecordedIdentity`, `TestNoPublisherRunsBetweenFstatatAndUnlinkat`, `TestGatedPassPublishesNothingWithoutTheRootPassLock`, `TestFirstArmingWaitsForAFailOpenPassToEnd`, `TestRootStepRefusesARootAgent`, `TestStaleCertificateIsVoid`, `TestArmingWritesAFreshNonce`, `TestDisarmedPublicationIsStillServed`, `TestNothingRemovesTheGateMarkers`, `TestAdoptPassesCensus`, `TestLockRootPassChecksAncestryBeforeCreating`, `TestWritableLayoutIsNeverRemoved`, `TestRejectedAncestryGetsNoWrite`, `TestRootJournalResolvesEveryIntent`, `TestRootFindsItsOwnRequestsOnRestart`, `TestForgedAgentFilesAreHarmless`). |
| 4. Residence changes and cleanup exclude each other through `host/<hostname>`: a create (busy → 409, row void, no reservation written), a crash after `BeginAllocation` and after `Reserve`, a terminal row whose release failed, a create's reservation PUT that lands after its recovery voided and released it (refused by its condition); a placement that is busy (nothing written), whose record PUT commits after the client gave up (fenced before release), that pauses after its reservation for longer than the grace (registered, so never swept; its local writes fail at its deadline) and whose console crashed (the sweep fences and releases); an offline move selection; with the switch off two creates of different hosts and a cleanup beside an operator move all proceed, and the sentinel never blocks a create; the drain settles every stuck `place` and `create` entry, fence before release, refuses while a self-serve move is in flight, releases a `cleanup` entry only when its box reports the obligation `releasing` or `ended`, and answers `drained` only when no cleanup holds an entry; a create paused between `BeginAllocation` and its reservation and a placement registered and paused before its reservation keep the drain `draining`; an admission and the drain's flag are one critical section; a recovery paused between `PendingAllocations` and its locks while the create voids that allocation and releases its key reserves nothing and writes nothing (review 7's finding 3); a table write is one HTTP attempt, resent identically until answered, and its registration is removed only by its owner, never while an attempt of it is inside `PutOnce` (review 8, item 6); an unarmed box writes nothing to the table in any state, never bootstraps or normalises it, and an obligation that reaches `releasing` unarmed keeps its entries until the drain releases them on its report (review 8, item 6); a placement paused past its deadline makes no registry update, local or remote (Tasks 2a to 2c, 7a, 7b and 7c, `TestConflictRelation`, `TestUncertainReservationIsSettledByTheNextWrite`, `TestPreEpochETagCannotSatisfyANewAcquisition`, `TestTableWriteIsOneHTTPAttempt`, `TestUnansweredTableWriteIsResentIdentically`, `TestRecoveryReloadsTheAllocationUnderItsLocks`, `TestRegistrationIsRemovedOnlyByItsOwner`, `TestUnarmedBoxNeverWritesTheTable`, `TestDrainReleasesACleanupOnlyOnItsReport`, `TestDrainRefusesWhileASelfServeMoveIsInFlight`, `TestResidenceChangesAndCleanupExclude`, `TestPausedPlacementIsNeverSwept`, `TestRemoteSetPlacementRunsUnderTheReceivedDeadline`, `TestDrainSettlesBeforeItReleases`, `TestDrainCountsWorkWithoutEntries`, `TestAdmissionAndTheDrainFlagAreOneCriticalSection`). |
| 5. Publication is durable and findable: the report under the move id survives failed writes and is republished until `PublishedRev == Rev`; a refusal repeated every run is one event with a count; a skipped key's page survives an outbox that cannot create it, a pager that refuses it and a restart; a periodic page fires once per tick across restarts; the reaper waits for the report, `pages_pending`, the outbox and root's acknowledgement; an older obligation's report is found by its cleanup id after `LastMove` moved on and after the record was deleted (Tasks 5a, 5b and 8, `TestCleanupReportIsDurableAndBlocksTheReaper`, `TestPendingPagesSurviveARestart`, `TestOlderCleanupIsFoundByItsID`). |
| @@ -136,23 +139,26 @@ Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and |
| ### Task 3: The cleaning mark, its refusals, the wake's answer and the writer inventory |
| **Files:** |
| -- Create: `internal/lease/cleaning.go`, `internal/host/hibernateprefix_inventory_test.go` |
| -- Modify: `internal/lease/lease.go` (`AcquireTTL`, line 171: a `readLock` before `AcquireLease`), `internal/lease/tombstone.go` (`Tombstone`, line 183: the owner check before the expiry test), `internal/lease/lockstate.go` (`LockState`, line 78: `held` for a `cleaning:` owner), `internal/host/movesoffline_land.go` (the lease step, lines 612-630: a `cleaning:` owner waits, before the expiry case), `internal/activator/backend_exec.go` (`ExecBackend.Acquire`, lines 702-710), `internal/activator/activator.go` (`errLeaseCleaning` and `cleaningRetryAfter` beside `errLeaseHeld`, line 1083, and `configRefusedRetryAfter`, line 1120; `serveWakeError`, line 5413; `doWake`'s lease log, line 2244), `internal/host/schedule_wiring_test.go` (one case in `TestDeliverViaHandlerMapsWakeRefusals`), the copy record and `approved_moves_copy_test.go` |
| +- Create: `internal/lease/cleaning.go`, `internal/lease/leasetest/hold.go` (`Holder`, the held request), `internal/host/hibernateprefix_inventory_test.go` |
| +- Modify: `internal/lease/lease.go` (`AcquireTTL`, line 171: a `readLock` before `AcquireLease`), `internal/lease/tombstone.go` (`Tombstone`, line 183: the owner check before the expiry test; `readLock`, line 120, becomes a wrapper over the new `readLockBody`; `LiftTombstone`, line 292, marshals `lockBody`), `internal/lease/export_test.go` (`SetLockAttemptTimeoutForTest`), `internal/lease/lockstate.go` (`LockState`, line 78: `held` for a `cleaning:` owner), `internal/host/movesoffline_land.go` (the lease step, lines 612-630: a `cleaning:` owner waits, before the expiry case), `internal/activator/backend_exec.go` (`ExecBackend.Acquire`, lines 702-710), `internal/activator/activator.go` (`errLeaseCleaning` and `cleaningRetryAfter` beside `errLeaseHeld`, line 1083, and `configRefusedRetryAfter`, line 1120; `serveWakeError`, line 5413; `doWake`'s lease log, line 2244), `internal/host/schedule_wiring_test.go` (one case in `TestDeliverViaHandlerMapsWakeRefusals`), the copy record and `approved_moves_copy_test.go` |
| - Test: `internal/lease/cleaning_test.go` (over the S3 fake the tombstone tests use), `internal/host/movesoffline_land_cleaning_test.go`, `internal/activator/cleaning_wake_test.go` |
| **Interfaces:** |
| - `const CleaningPrefix = "cleaning:"`; `var CleaningExpiry = time.Date(9999, 12, 31, 0, 0, 0, 0, time.UTC)`; `func CleaningOwner(id string) string`; `func IsCleaningOwner(owner string) bool` (the prefix and a non-empty id); `var ErrNotOurLock` (the lock is absent, another owner's, or this move's tombstone still live: nothing written). |
| -- `func MarkCleaning(ctx context.Context, bucket, location, region, moveOwner, cleaningOwner string) error`: `readLock` with its ETag (`tombstoneLeaser`, `tombstone.go:99-148`); owner `cleaningOwner` → nil, nothing written (a resumed mark); owner `moveOwner` and `IsLiftedTombstone` → `PutObject` `IfMatch` `{Generation: cur.Generation, ExpiresAt: CleaningExpiry, Owner: cleaningOwner}`; else `ErrNotOurLock`; a lost condition → `ErrTombstoneChanged`. |
| +- **The lock body (W1).** `type lockBody struct { Generation int64 \`json:"generation"\`; ExpiresAt time.Time \`json:"expires_at"\`; Owner string \`json:"owner,omitempty"\`; Cleanups uint64 \`json:"cleanups,omitempty"\` }`: litestream's `Lease` fields with their tags (`litestream@v0.5.14/leaser.go:31-36`), and the count. `func readLockBody(ctx context.Context, leaser *lss3.Leaser, key string) (body *lockBody, etag string, absent bool, err error)` is today's `readLock` (`tombstone.go:120-148`: bounded by `lockBodyLimit`, only an explicit `NoSuchKey` is absence, a zero expiry or a missing ETag is an error) decoding into `lockBody`; `readLock` becomes a wrapper that converts its answer to `*litestream.Lease`, so `Tombstone` and `Reassert` read exactly as before. `LiftTombstone` reads through `readLockBody` and marshals `&lockBody{Generation: cur.Generation, ExpiresAt: liftedExpiry, Owner: cur.Owner, Cleanups: cur.Cleanups}`; with `Cleanups == 0`, which every lock that exists today has, its bytes are those of the `litestream.Lease` it marshals now. Nothing else in the package changes how a lock is parsed. |
| +- **Single-attempt lock writes (W1).** `var ErrLockWriteUncertain`; `var lockAttemptTimeout = 2 * time.Second` (`export_test.go` shortens it); `func putLockUntilAnswered(ctx context.Context, leaser *lss3.Leaser, key string, body []byte, ifMatch string) (status int, err error)`: one `PutObject` per send, with `IfMatch: aws.String(ifMatch)` and the option `func(o *awss3.Options) { o.Retryer = aws.NopRetryer{} }`, each under `context.WithTimeout(ctx, lockAttemptTimeout)`. An answer is returned: success → 200; `PreconditionFailed` or 412 → 412; `NoSuchKey` or 404 → 404 (`apiCode`, `versioned.go:145`). Anything else (a timeout, a transport error, a 5xx) sends the identical bytes under the identical condition again after 1 s, 2 s, 4 s and then every 10 s; `ctx` ending first → `ErrLockWriteUncertain`. Only `MarkCleaning` and `UnmarkCleaning` call it. |
| +- **The held request (test only).** `leasetest.Holder` (`hold.go`): `func NewHolder(next http.Handler) *Holder`, an `http.Handler` in front of `NewS3Server`'s; `(*Holder).HoldPuts(key string, n int) (release func())`: the next `n` `PUT`s of `key` wait inside the server before `next` sees them, so their clients, whose attempt contexts end first, get no answer, and later ones pass; `release` lets every held request through to `next` in arrival order and returns once `next` has answered each, and `next` applies each condition against whatever is current then, as a request delayed in the network would be; `(*Holder).Requests(key string) []HeldRequest` lists every `PUT` of `key` in arrival order with its `Body`, its `IfMatch` and the `Status` `next` gave it (0 while held). Tasks 3, 4a and 4d use it for `lock.json`. |
| +- `func MarkCleaning(ctx context.Context, bucket, location, region, moveOwner, cleaningOwner string) error`: `readLockBody` (`tombstoneLeaser`, `tombstone.go:99-118`); owner `cleaningOwner` → nil, nothing written (a resumed mark); owner `moveOwner` and `IsLiftedTombstone` → `putLockUntilAnswered` of `{Generation: cur.Generation, ExpiresAt: CleaningExpiry, Owner: cleaningOwner, Cleanups: cur.Cleanups}` with `ifMatch` the ETag read: 200 → nil; 412 or 404 → `readLockBody` again: owner `cleaningOwner` → nil (an earlier attempt of ours landed: the mark is settled), else `ErrTombstoneChanged`; `ErrLockWriteUncertain` → itself; any other lock (absent, another owner, this move's tombstone still live) → `ErrNotOurLock`, nothing written. |
| - **No renewal (T1).** Nothing rewrites a `cleaning:` lock but `UnmarkCleaning`: rev 6's `ReassertCleaning`, `cleanupMarkTTL`, `cleanupMarkMargin` and the 30-minute reassert are gone. |
| -- `func UnmarkCleaning(ctx, bucket, location, region, cleaningOwner, moveOwner string) error`: owner `cleaningOwner` → conditional PUT of `{Generation, ExpiresAt: liftedExpiry, Owner: moveOwner}`; absent or another owner → nil (nothing of ours; the caller logs it); a lost condition → `ErrTombstoneChanged`. |
| +- `func UnmarkCleaning(ctx, bucket, location, region, cleaningOwner, moveOwner string) error`: `readLockBody`; owner `cleaningOwner`, **or** owner `moveOwner` with `IsLiftedTombstone` (no mark here: never sent, still travelling, or lifted by an earlier visit) → **always** `putLockUntilAnswered` of `{Generation: cur.Generation, ExpiresAt: liftedExpiry, Owner: moveOwner, Cleanups: cur.Cleanups + 1}` with `ifMatch` the ETag read (W1: the body after the unmark is one the lock never held, so a mark attempt conditioned on an earlier ETag cannot land after it): 200 → nil; 412 → `readLockBody` again: owner `moveOwner`, lifted, with `Cleanups > cur.Cleanups` → nil (an earlier attempt of ours landed), else `ErrTombstoneChanged` (for instance a late mark attempt of ours landed between the read and the write: the obligation stays `releasing`, and its next visit unmarks that mark); 404 → nil (absent: no conditional write of ours can land on a missing key); `ErrLockWriteUncertain` → itself. Absent, another owner, or this move's tombstone still live → nil and nothing written (the caller logs it): no attempt of ours can land there, because each names the ETag of our mark or of this move's lifted tombstone, and none of those bodies recurs (Global Constraint "No lock body recurs"). |
| - **The refusals, by code path.** `AcquireTTL` (`lease.go:171`; so `Acquire`, `ReplicateWithLease` at `:443`, `ExecBackend.Acquire` at `backend_exec.go:701` and the staging lease): `readLock` first; an `IsCleaningOwner` owner → `fmt.Errorf("acquire lease: %w", &HeldError{Owner, ExpiresAt})`, whatever its expiry, before `AcquireLease` and with nothing written; absent → on to `AcquireLease` as today. The sentinel expiry already makes `AcquireLease` refuse on its own read (`litestream@v0.5.14/s3/leaser.go:96`), so a mark written by `MarkCleaning` is refused on the very read that conditions the write; the pre-read adds the refusal for a mark whose expiry was edited by hand, and costs one `GET` per acquisition. `Tombstone` (`tombstone.go:183`): an `IsCleaningOwner` owner → `heldErr(cur)` before the expiry test. `LiftTombstone` and `Reassert` already refuse an owner that is not `moved:` (`tombstone.go:286`, `:225`). `LockState` (`lockstate.go:78`): `held = lockHeld(expires, now) || IsCleaningOwner(owner)`, so the cut's stop evidence (`movespass.go:647`) and the seed finish (`console/seedfinish.go:255`) read a mark as held. The landing's lease step (`movesoffline_land.go:612-630`): an `IsCleaningOwner` owner → `r.landWait("lease", "a cleanup holds the target's lock")`, before the expiry case. The health renewer (`health.go:134`) renews only its own lease by its ETag and never meets the mark. |
| - **The wake's answer to a mark (U4).** Today `ExecBackend.Acquire` turns every `*lease.HeldError` into `errLeaseHeld{retryAfter: time.Until(he.ExpiresAt)}` (`backend_exec.go:705`), and `serveWakeError` answers `Retry-After: int(retryAfter/time.Second) + 1` (`activator.go:5427`); with the sentinel expiry that duration saturates and the header reads `9223372037` on a 64-bit box (review 7's finding 7). The lease's exclusion is unchanged: the mark is still refused whatever its age. Only what the activator tells its consumers changes. `ExecBackend.Acquire`: an `*lease.HeldError` whose `lease.IsCleaningOwner(he.Owner)` → `&errLeaseCleaning{holder: he.Owner}`, and `he.ExpiresAt` is not read; any other holder as today. `type errLeaseCleaning struct{ holder string }` (`Error()`: `instance replica marked by %q: a cleanup is running`); `const cleaningRetryAfter = 30` (seconds, fixed: a cleanup's run is at most 60 s, `cleanupRunDeadline`, and a client back within half a minute finds a finished cleanup promptly). `serveWakeError` gains, before the `errLeaseHeld` case: `case errors.As(err, &cleaning): w.Header().Set("Retry-After", strconv.Itoa(cleaningRetryAfter)); edge.MarkAnswered(r); http.Error(w, "instance is being cleaned up after a move. retry shortly", http.StatusServiceUnavailable)`. The text is operator-facing, lower case and has no em dash; it begins with `instance`, so the scheduler's `deliverVia` (`internal/host/schedule_wiring.go:514`) reads it as a wake refusal and retries it as one; its copy row is pending Paul. `doWake` (line 2244) logs `activator: %s: a cleanup holds the replica (%q); refusing to wake` for an `errLeaseCleaning`, beside its `LEASE HELD` line for an `errLeaseHeld`. |
| - **The inventory test** (`go/ast` with `go/types` over every non-test file under `./internal/...`): every call of `activator.HibernatePrefix` and `(activator.Config).hibernatePrefix` is in the test's list with its class: `reader`, `lease-gated writer`, `tombstone-gated writer`, `build-state only`, `delete only`, `a move into the region (hostname key)` or `excluded: <reason>`, as the Global Constraint lists them; a new site, or a removed one, fails until the list is edited. |
| -- [ ] **Step 1: Write the failing tests**: `TestMarkOnlyFromThisMovesLiftedTombstone` (lifted tombstone of this move → marked, the body's `ExpiresAt.Equal(lease.CleaningExpiry)` and its generation kept; of another move, live tombstone, an ordinary lease, absent → `errors.Is(err, lease.ErrNotOurLock)`, nothing written); `TestMarkRacesAnAcquire` (an `Acquire` landing between the read and the PUT → the mark's condition fails); `TestCleaningMarkIsRefusedWhateverItsAge` (the mark written, then the clock advanced 25 h, and separately its `ExpiresAt` rewritten by hand to `2000-01-01`: `Acquire` → `errors.As(err, &held) && held.Owner == lease.CleaningOwner(id)` and the lock's ETag unchanged; `AcquireTTL` with the staging TTL and `ReplicateWithLease` → the same; `Tombstone` → `*HeldError`, nothing written; `LiftTombstone` with the move's `mine` → `lifted == false`, nothing written; `LockState` → `held == true`); `TestMarkIsNeverRewritten` (a second `MarkCleaning` by the same owner writes nothing: the ETag is unchanged; the package has no exported function but `UnmarkCleaning` that writes a `cleaning:` lock, checked by `go/ast` over `internal/lease`); `TestUnmarkRestoresTheLiftedTombstone` (`IsLiftedTombstone(owner, expires)` true afterwards, the generation kept); `TestLandingWaitsOnACleaningLockWhateverItsExpiry` (the landing's lease step with a `cleaning:` lock whose expiry was edited to the past → `landWait`, never `ok`); `TestCleaningMarkAnswersAFiniteRetry` (review 7's finding 7, at the response: the mark written by `MarkCleaning` over the lease fake with the sentinel expiry; `_, err := backend.Acquire(ctx, rt)` on an `ExecBackend` over that fake, then `a.serveWakeError(rec, req, rt, err)`: `rec.Code == http.StatusServiceUnavailable`, `rec.Header().Get("Retry-After") == "30"`, and the body's first line `== "instance is being cleaned up after a move. retry shortly"`; the same with the mark's expiry edited by hand to `2000-01-01`; an ordinary lease of another box expiring in 40 s → `Retry-After == "41"` and "instance is waking elsewhere; retry shortly", as today); `TestDeliverViaHandlerMapsWakeRefusals` gains the case `{503, "instance is being cleaned up after a move. retry shortly\n", 0, "instance is being cleaned up"}`; `TestMovesStringsAreRecorded` and `TestMovesCopyRecordIsComplete` green with the row pending; `TestHibernatePrefixSitesAreInventoried`. |
| -- [ ] **Step 2: Run to verify they fail**: `go test ./internal/lease -run 'Cleaning|Mark|Unmark' -count=1`, `go test ./internal/host -run 'HibernatePrefixSites|LandingWaitsOnACleaning|DeliverViaHandlerMapsWakeRefusals' -count=1`, `go test ./internal/activator -run 'CleaningMarkAnswers' -count=1`, `go test . -run 'MovesStringsAreRecorded|MovesCopyRecord' -count=1`. |
| +- [ ] **Step 1: Write the failing tests**: `TestMarkOnlyFromThisMovesLiftedTombstone` (lifted tombstone of this move → marked, the body's `ExpiresAt.Equal(lease.CleaningExpiry)` and its generation kept; of another move, live tombstone, an ordinary lease, absent → `errors.Is(err, lease.ErrNotOurLock)`, nothing written); `TestMarkRacesAnAcquire` (an `Acquire` landing between the read and the PUT → the mark's condition fails); `TestCleaningMarkIsRefusedWhateverItsAge` (the mark written, then the clock advanced 25 h, and separately its `ExpiresAt` rewritten by hand to `2000-01-01`: `Acquire` → `errors.As(err, &held) && held.Owner == lease.CleaningOwner(id)` and the lock's ETag unchanged; `AcquireTTL` with the staging TTL and `ReplicateWithLease` → the same; `Tombstone` → `*HeldError`, nothing written; `LiftTombstone` with the move's `mine` → `lifted == false`, nothing written; `LockState` → `held == true`); `TestMarkIsNeverRewritten` (a second `MarkCleaning` by the same owner writes nothing: the ETag is unchanged; the package has no exported function but `UnmarkCleaning` that writes a `cleaning:` lock, checked by `go/ast` over `internal/lease`); `TestUnmarkWritesALiftedTombstoneThatNeverRecurs` (W1: `IsLiftedTombstone(owner, expires)` true afterwards, the generation kept, `l.Cleanups == before.Cleanups+1`, and the body's bytes and ETag differ from the pre-mark ones; a second `UnmarkCleaning` with no mark there writes again, `l.Cleanups == before.Cleanups+2`; across a mark, its unmark, a resumed unmark, and a mark and unmark under a second cleanup id, the ETags of every version the bucket lists for `lock.json` are distinct; absent, a box's ordinary lease and another move's tombstone → nil and `s.writeCount()` unchanged); `TestDelayedMarkCannotLandAfterTheUnmark` (W1's witness, review 9's interleaving, over `tombFixture` behind a `leasetest.Holder`: this move's lifted tombstone T is current at generation 4, with bytes `tBody` and ETag `tETag`; `release := holder.HoldPuts(lock, 1)`; `MarkCleaning`'s first attempt A1 is held and times out under `lockAttemptTimeout`, and its resend A2 is answered: `err == nil`, the lock's owner is `lease.CleaningOwner(id)` with `cleanups` 0, and `reqs[0].Body` and `reqs[1].Body` are byte-equal with `reqs[0].IfMatch == tETag && reqs[1].IfMatch == tETag`; the cleanup ends: `UnmarkCleaning` → nil, the lock is lifted, at generation 4, with `cleanups == 1`, `!bytes.Equal(body, tBody)` and an ETag `!= tETag`; then `release()`: A1 reaches the store and `holder.Requests(lock)[0].Status == http.StatusPreconditionFailed`, the lock's bytes are still the unmark's, and `lease.Acquire` over the location succeeds; rev 9's unmark, which wrote `tBody` back, fails this test: A1 answers 200 and the lock is left `cleaning:<id>`); `TestUnmarkSettlesAnUncertainMark` (W1: T current; `holder.HoldPuts(lock, 1)` and `MarkCleaning` under a context that ends before the first resend → `errors.Is(err, lease.ErrLockWriteUncertain)`, the lock still `tBody`; `UnmarkCleaning` finds no mark and writes anyway: owner `moved:<move id>`, lifted, `cleanups == 1`; `release()`: the held attempt answers 412 and the lock is unchanged; and separately, the held attempt released by `tombStore.beforeWrite` between `UnmarkCleaning`'s read and its write: the mark lands, the unmark's write answers 412 and its second read finds our mark → `errors.Is(err, lease.ErrTombstoneChanged)`; a second `UnmarkCleaning` → lifted, `cleanups == 1`); `TestLockWritesAreSingleAttemptsResentIdentically` (an endpoint answering 500 to every `PUT` of the lock: each send reaches the server exactly once, where the SDK's default retryer would send three, and every resend's bytes and `If-Match` equal the first's; a 412 after a timed-out attempt with our mark current → nil; `ctx` ending first → `ErrLockWriteUncertain`); `TestOlderReadersIgnoreTheCleanupCount` (a lifted tombstone with `"cleanups": 3`: `LockState` answers its owner and expiry with `held == false`, `lease.IsLiftedTombstone` is true, and `lease.Acquire` takes it, writing generation + 1 with no `cleanups` key; a mark with `"cleanups": 3` is still refused by `Acquire` with a `HeldError` naming `cleaning:<id>`); `TestLiftTombstoneBytesAreUnchanged` (a live tombstone at generation 7 lifted: the bytes written equal `json.Marshal(&litestream.Lease{Generation: 7, ExpiresAt: time.Unix(0, 0).UTC(), Owner: tombMove})`); `TestLandingWaitsOnACleaningLockWhateverItsExpiry` (the landing's lease step with a `cleaning:` lock whose expiry was edited to the past → `landWait`, never `ok`); `TestCleaningMarkAnswersAFiniteRetry` (review 7's finding 7, at the response: the mark written by `MarkCleaning` over the lease fake with the sentinel expiry; `_, err := backend.Acquire(ctx, rt)` on an `ExecBackend` over that fake, then `a.serveWakeError(rec, req, rt, err)`: `rec.Code == http.StatusServiceUnavailable`, `rec.Header().Get("Retry-After") == "30"`, and the body's first line `== "instance is being cleaned up after a move. retry shortly"`; the same with the mark's expiry edited by hand to `2000-01-01`; an ordinary lease of another box expiring in 40 s → `Retry-After == "41"` and "instance is waking elsewhere; retry shortly", as today); `TestDeliverViaHandlerMapsWakeRefusals` gains the case `{503, "instance is being cleaned up after a move. retry shortly\n", 0, "instance is being cleaned up"}`; `TestMovesStringsAreRecorded` and `TestMovesCopyRecordIsComplete` green with the row pending; `TestHibernatePrefixSitesAreInventoried`. |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/lease -run 'Cleaning|Mark|Unmark|LockWrites|OlderReaders|LiftTombstoneBytes' -count=1`, `go test ./internal/host -run 'HibernatePrefixSites|LandingWaitsOnACleaning|DeliverViaHandlerMapsWakeRefusals' -count=1`, `go test ./internal/activator -run 'CleaningMarkAnswers' -count=1`, `go test . -run 'MovesStringsAreRecorded|MovesCopyRecord' -count=1`. |
| - [ ] **Step 3: Implement. Step 4: Run to verify they pass**, plus `go test ./internal/lease -run 'Tombstone|Acquire|LockState' -count=1` (the existing tombstone and lock tests green). |
| -- [ ] **Step 5: Commit** `lease: the cleaning mark, a state with no expiry written only over this move's lifted tombstone and put back exactly, refused by every acquirer whatever its age, and answered to a wake with a 30-second retry; every hibernate-prefix site inventoried`. |
| +- [ ] **Step 5: Commit** `lease: the cleaning mark, a state with no expiry written only over this move's lifted tombstone and lifted back with a raised cleanup count so that no earlier lock body recurs, its lock writes single attempts resent until answered, refused by every acquirer whatever its age, and answered to a wake with a 30-second retry; every hibernate-prefix site inventoried`. |
| ### Task 4a: The persisted lifecycle and its recovery |
| @@ -168,11 +174,11 @@ Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and |
| 1. `open`: observe: `AwayFinishing` → stay (the original move's finishing has precedence; never supersession); `AwayMoved` → supersede; `AwayYes` → then the row check (a row → supersede) → `finished`, `FinishedAt = LastMove.At`. |
| 2. `finished`: `supersededBy` → supersede; `!cfg.CleanupArmed` → **no table write** (V3): with `ReserveUncertain` set, or with an entry of `{cleanup, ID}` in the run's one `moves.PeekActive` (Task 4d), the acquisition that may have landed is given up: `releasing`, `Outcome = cancelled`, event `disarmed` (row 8 then lifts nothing, because nothing was marked, and withholds the release, so its entry, if any, waits for an armed visit or for the drain on this report, Task 7c); else nothing; stay; `now - FinishedAt < 24 h` → stay; else, through `w.armedTable()` (Task 4d), `moves.Reserve(ctx, s, {cleanup, ID}, {cfg.BoxID: e, HostKey(Host): e})` with `e = {cleanup, ID, Host, Account, App, now}`: `ErrReserved` → stay, event `busy`, `ReserveUncertain` cleared (the table changed since any earlier write of ours was read, so that write can never land); `ErrUnreadable` → stay, event `refused/reservation-unreadable`; `ErrUncertain` → stay, `ReserveUncertain = true`, the same event (the next armed visit's own retry settles it; disarmed, the obligation is given up as above); held → `ReservedAt` from the returned entry, `ReserveUncertain` cleared, every inventory key `pending` with `Done = 0`, `State = cleaning`. |
| 3. `cleaning`, `publishing`, `root-requested` (**ownership**, before anything else; rows 3 to 5 run in order within one visit): `moves.PeekActive` (a read that never writes, Task 2a); `!moves.Holds(table, [BoxID, HostKey(Host)], {cleanup, ID})` → supersede, reason `reservation-lost` (an operator removed it; never re-acquired); a read error → stay. |
| - 4. `cleaning` with `MarkedAt` zero: `MarkCleaning(..., TombstoneOwner, CleaningOwner(ID))`: `ErrNotOurLock` → supersede, reason `lock-taken`; another error → stay; marked → `MarkedAt = now`. |
| + 4. `cleaning` with `MarkedAt` zero: `MarkCleaning(..., TombstoneOwner, CleaningOwner(ID))`: `ErrNotOurLock` → supersede, reason `lock-taken`; `ErrLockWriteUncertain` (no attempt answered within the operation) or another error → stay, event `refused/mark-failed`, `MarkedAt` still zero. The next visit runs rows 3 and 4 again: its `MarkCleaning` reads the lock first (our mark → marked; this move's lifted tombstone → the identical write again), and a supersession found first goes to `releasing`, whose unmark's write settles any attempt still in the network (W1); marked → `MarkedAt = now`. |
| 5. `cleaning`: first, with `InFlight` set, `w.settleInFlight` (Task 4c), whatever row 3 found; it answers done, stop (the delete still unanswered: the obligation stays `cleaning`, holding the mark and whatever entries it still holds, and nothing else runs this visit), `abandoned` (an operator's abandon after the page) or `unsettled` (the mark replaced by hand). `abandoned` or `unsettled` → `releasing`, `Outcome = superseded`, and no further key is retired; done with `Superseding` set → `releasing`, `Outcome = superseded`, `Superseding` cleared in the same write. Then for each `pending` key in inventory order within the budget: `w.retireKey` (Task 4c), which answers the key's terminal outcome, a supersession reason, or stop. Every key terminal and `InFlight` nil: `supersededBy` once more → supersede; else `LocalInventory` empty → `releasing`, `Outcome = cleaned`; else `publishing`. |
| 6. `publishing`, `root-requested`, `cancel-pending`: Task 6b's rows; `publishing` and `root-requested` begin with `supersededBy` (a reason → supersede). |
| 7. **Supersede**: from `publishing` or `root-requested` → `cancel-pending` (Task 6b), so the reservation and the mark stay until root has acknowledged; from `cancel-pending` nothing changes; from `cleaning` with `InFlight` set, the obligation stays `cleaning` with `Superseding = <reason>` persisted, holding the mark and whatever entries it still holds, and row 5 only settles until `w.settleInFlight` answers done, `abandoned` or `unsettled`, and then goes to `releasing` (T1 (b): supersession lifts the mark only when no delete is in flight); from any other state → `releasing`, `Outcome = superseded` (from `open` and `finished` too: `Release` and `UnmarkCleaning` change nothing of ours when nothing is held, and a `finished` obligation may hold an entry its crash left). Event `superseded/<reason>`. |
| - 8. `releasing`: `UnmarkCleaning(..., CleaningOwner(ID), TombstoneOwner)` → error → stay (an obligation that never marked, or already unmarked, finds another owner and writes nothing); then **unarmed** (`!cfg.CleanupArmed`, V3): no table write; the obligation stays `releasing`, event `release-withheld`, and the first time, in the same `writeRetired`, the one-shot page `release-withheld` (Task 5a) is owed, "cleanup of <host> waits for its reservation to be released: cleanup is off on this box"; its entries wait for an armed visit or for the drain, which releases them on this obligation's report (Task 7c); **armed**: through `w.armedTable()`, `moves.Release(ctx, s, {cleanup, ID})` → error → stay; → `ended`, `EndedAt = now`. The mark goes before the reservation, so a residence change admitted by the release never meets a mark, and no version is deleted after `releasing` is written, so a release by the drain while the mark still stands only delays a wake by its 30-second retry. |
| + 8. `releasing`: `UnmarkCleaning(..., CleaningOwner(ID), TombstoneOwner)` (Task 3: over our mark and over this move's lifted tombstone alike it writes the lifted tombstone with a raised count, so it writes for an obligation that never marked or already unmarked too, and any mark attempt of ours still in the network can no longer land; over an absent lock or another owner's it writes nothing) → an error (`ErrLockWriteUncertain` and `ErrTombstoneChanged` included) → stay, event `refused/unmark-failed`; then **unarmed** (`!cfg.CleanupArmed`, V3): no table write; the obligation stays `releasing`, event `release-withheld`, and the first time, in the same `writeRetired`, the one-shot page `release-withheld` (Task 5a) is owed, "cleanup of <host> waits for its reservation to be released: cleanup is off on this box"; its entries wait for an armed visit or for the drain, which releases them on this obligation's report (Task 7c); **armed**: through `w.armedTable()`, `moves.Release(ctx, s, {cleanup, ID})` → error → stay; → `ended`, `EndedAt = now`. The mark goes before the reservation, so a residence change admitted by the release never meets a mark, and no version is deleted after `releasing` is written, so a release by the drain while the mark still stands only delays a wake by its 30-second retry. |
| 9. `ended`: Task 5a's publication and Task 8's reaper only. `corrupt`: Task 5a's hourly page only. |
| - **The outcome table** (F7; the only place `Outcome` is chosen; never `cleaned` otherwise): |
| @@ -191,10 +197,10 @@ Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and |
| Root's per-path results go into the report whatever the outcome (`Root`). A key `superseded` or `unsettled` is reported as such whatever the outcome. |
| - **Restart** resumes every state through the same table: `finished` re-runs `Reserve` when armed, whose own retry returns the held entry, and when disarmed only reads (row 2); `cleaning` never reserves, and sends a first delete of a version only after rows 3 and 4 found both entries and the mark this cleanup's (a resend of the `InFlight` delete needs only the mark, Task 4c); an `InFlight` delete found after a crash is answered first, however long the crash lasted: the mark has no expiry, so it still stands. |
| -- [ ] **Step 1: Write the failing tests** against a `DirStore` deployment bucket, the lease's S3 fake for `lock.json`, a real registry, an injected clock and key and settle steps the test controls: `TestRecordAway` (each branch); `TestSupersessionIsCheckedAtEveryStep` (a row appearing and, separately, the record moving on, at: `open`, `finished`, between two keys, with a delete in flight, after the last key and before `publishing`, in `publishing` before the drop, in `root-requested`: each ends `superseded` through `releasing` or `cancel-pending`, never `cleaned`, and the in-flight delete is answered before `releasing`); `TestOutcomeTable` (each row, with root's answers from a fake acknowledgement reader); `TestCleanupSupersedesWhenTheRecordMovesOn` (a second move B to C, a recreation, a deletion, a placement back to `From`, each before the reservation: superseded, the lifted tombstone untouched, nothing held); `TestOriginalFinishingMoveIsNotSupersession`; `TestUnarmedBoxNeverWritesTheTable` (V3; review 8, item 6: an armed worker drives one obligation into each of `finished` with `ReserveUncertain`, `finished` whose `Reserve` attempt the fake delivers only after the restart, `cleaning` with a delete in flight, `publishing`, `root-requested`, `cancel-pending` and `releasing`; the switch goes off and the worker restarts with `cfg.CleanupArmed == false`; the store fake records a mark; every visit and run then completes, and again with `active.json` removed from the fake before the run and with a metadata-free body in it: `len(fake.PutsAfter(mark, moves.OfflineActiveKey)) == 0` in each case; both `finished` ones reach `releasing` with `r.Outcome == "cancelled"`, the late-landed one on the run after its entry appears; the one in `cleaning` answers its delete under the mark and reaches `releasing`; each `releasing` one has lifted its mark where it had one (`lease.IsLiftedTombstone(owner, expires)`), still has its entries (`moves.Holds(table, keys, op)`), and owes `release-withheld` in `r.PagesPending` exactly once across three runs; re-armed, the next visit's `Release` is the box's first table write and the obligation ends); `TestLifecycleResumesAtEveryBoundary` (Review Focus 2); `TestOwnershipLostIsSupersession` (an entry deleted by an operator; the mark replaced by a lease by hand; each → `superseded`, no new delete sent; with a delete in flight and the entry deleted by an operator, the same version id is resent under the mark and answered before `releasing` is written: `r.State == "cleaning"` until `fake.Answered(key, v)`). |
| -- [ ] **Step 2: Run to verify they fail**: `go test ./internal/moves -run 'RecordAway' -count=1`, `go test ./internal/host -run 'SupersessionIsChecked|OutcomeTable|CleanupSupersedes|OriginalFinishing|UnarmedBoxNever|LifecycleResumes|OwnershipLost' -count=1`. |
| +- [ ] **Step 1: Write the failing tests** against a `DirStore` deployment bucket, the lease's S3 fake for `lock.json`, a real registry, an injected clock and key and settle steps the test controls: `TestRecordAway` (each branch); `TestSupersessionIsCheckedAtEveryStep` (a row appearing and, separately, the record moving on, at: `open`, `finished`, between two keys, with a delete in flight, after the last key and before `publishing`, in `publishing` before the drop, in `root-requested`: each ends `superseded` through `releasing` or `cancel-pending`, never `cleaned`, and the in-flight delete is answered before `releasing`); `TestOutcomeTable` (each row, with root's answers from a fake acknowledgement reader); `TestCleanupSupersedesWhenTheRecordMovesOn` (a second move B to C, a recreation, a deletion, a placement back to `From`, each before the reservation: superseded, the lifted tombstone untouched, nothing held); `TestOriginalFinishingMoveIsNotSupersession`; `TestUnarmedBoxNeverWritesTheTable` (V3; review 8, item 6: an armed worker drives one obligation into each of `finished` with `ReserveUncertain`, `finished` whose `Reserve` attempt the fake delivers only after the restart, `cleaning` with a delete in flight, `publishing`, `root-requested`, `cancel-pending` and `releasing`; the switch goes off and the worker restarts with `cfg.CleanupArmed == false`; the store fake records a mark; every visit and run then completes, and again with `active.json` removed from the fake before the run and with a metadata-free body in it: `len(fake.PutsAfter(mark, moves.OfflineActiveKey)) == 0` in each case; both `finished` ones reach `releasing` with `r.Outcome == "cancelled"`, the late-landed one on the run after its entry appears; the one in `cleaning` answers its delete under the mark and reaches `releasing`; each `releasing` one has lifted its mark where it had one (`lease.IsLiftedTombstone(owner, expires)`), still has its entries (`moves.Holds(table, keys, op)`), and owes `release-withheld` in `r.PagesPending` exactly once across three runs; re-armed, the next visit's `Release` is the box's first table write and the obligation ends); `TestLifecycleResumesAtEveryBoundary` (Review Focus 2); `TestSupersessionAfterAnUncertainMarkSettlesIt` (W1, review 9's second case: an obligation in `cleaning` with `MarkedAt` zero; the lease fake holds the next three `PUT`s of `lock.json` (`holder.HoldPuts(lock, 3)`), so the visit's `MarkCleaning` answers `ErrLockWriteUncertain` with the lock still this move's lifted tombstone T at ETag `tETag`: `r.State == "cleaning" && r.MarkedAt.IsZero()`; an operator deletes the box entry; the next visit supersedes (`reservation-lost`) to `releasing`, and its `UnmarkCleaning` writes over T although no mark is there: the lock's `cleanups == 1` and its ETag `!= tETag`; the release and `ended` follow; `release()` then delivers the held attempts: each `Status == http.StatusPreconditionFailed`, the lock's owner is `moved:<move id>` with `lease.IsLiftedTombstone(owner, expires)`, and `ExecBackend.Acquire` over the fake succeeds; with rev 9's unmark, which wrote nothing over a lifted tombstone, the first held attempt lands and the lock is left `cleaning:<id>` for good); `TestOwnershipLostIsSupersession` (an entry deleted by an operator; the mark replaced by a lease by hand; each → `superseded`, no new delete sent; with a delete in flight and the entry deleted by an operator, the same version id is resent under the mark and answered before `releasing` is written: `r.State == "cleaning"` until `fake.Answered(key, v)`). |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/moves -run 'RecordAway' -count=1`, `go test ./internal/host -run 'SupersessionIsChecked|OutcomeTable|CleanupSupersedes|OriginalFinishing|UnarmedBoxNever|LifecycleResumes|SupersessionAfterAnUncertainMark|OwnershipLost' -count=1`. |
| - [ ] **Step 3: Implement. Step 4: Run to verify they pass.** |
| -- [ ] **Step 5: Commit** `moves: the cleanup's persisted lifecycle, supersession checked at every step and one outcome table`. |
| +- [ ] **Step 5: Commit** `moves: the cleanup's persisted lifecycle, supersession checked at every step, one outcome table, and an unmark that always writes, so an uncertain mark is settled`. |
| ### Task 4b: The delete primitive and its dev gate |
| @@ -300,14 +306,14 @@ Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and |
| ### Task 6b: The root-owned request and acknowledgement, as a stage of the adopt pass |
| **Files:** |
| -- Create: `internal/release/rootcleanup.go` (discovery, the copy, the dispatch, the acknowledgement, `LockRootPass`, `RootCleanupGated`, `ArmRootCleanup`, `QuiesceRootCleanup`, `UngateRootCleanup`; portable), `internal/release/carlosprocs_linux.go` and `carlosprocs_other.go` (`CarlosProcesses`, `AdoptPasses`, `AgentRunsUnprivileged`), `internal/host/adoptrootgate.go` (`adoptRootGate`), `internal/host/cleanupungate.go` (`RunSystemCleanupUngate`), `internal/release/rootcleanup_other.go` (answers every request `cancelled` with `Reason: "unsupported platform"`; `LockRootPass` answers `ErrNotRootOnly`), `internal/host/movescleanup_root.go` (the agent's rows, `rootCleanupPass`) |
| -- Modify: `internal/host/agent.go` (`runAdoptUnderRootRelease`: the root pass lock and the gate right after `lockAdoptPass` at line 4655, in every mode, before `ensureUnits` at line 4664; `rootCleanupPass(...)` after `heldStagePass` at line 4932), `internal/host/adoptlock.go` (the lock-order comment at lines 52-54 names the root pass lock between the adopt-pass lock and the restart marks), `internal/host/movesheldlock.go` (the order comment, lines 26-31), `internal/host/claimcensus.go` (`claimOptCensus` gains `<optDir>/root-cleanup` as box state, if its completeness test requires it), `system.go` (`cleanup-ungate` in `systemVerb`'s list, line 53, and in `runSystem`'s switch beside `floor-allows-prev`, line 72, as a root verb; `systemUsage`), `docs/site/cli.md` (the verb line), the copy record and `approved_moves_copy_test.go` |
| -- Test: `internal/release/rootcleanup_test.go`, `internal/release/carlosprocs_linux_test.go`, `internal/host/movescleanup_root_test.go`, `internal/host/adopt_rootcleanup_stage_test.go`, `internal/host/adoptrootgate_test.go`, `internal/host/cleanupungate_test.go` |
| +- Create: `internal/release/rootcleanup.go` (discovery, the copy, the dispatch, the acknowledgement, `LockRootPass`, `RootCleanupGated`, `ArmRootCleanup`, `QuiesceRootCleanup`; portable), `internal/release/carlosprocs_linux.go` and `carlosprocs_other.go` (`CarlosProcesses`, `AdoptPasses`, `AgentRunsUnprivileged`), `internal/host/adoptrootgate.go` (`adoptRootGate`), `internal/release/rootcleanup_other.go` (answers every request `cancelled` with `Reason: "unsupported platform"`; `LockRootPass` answers `ErrNotRootOnly`), `internal/host/movescleanup_root.go` (the agent's rows, `rootCleanupPass`, and `var cleanupBeforeDropHook func()`, nil in production, run just before `writeRootCleanupRequest`, for the W2 witness) |
| +- Modify: `internal/host/agent.go` (`runAdoptUnderRootRelease`: the root pass lock and the gate right after `lockAdoptPass` at line 4655, in every mode, before `ensureUnits` at line 4664; `rootCleanupPass(...)` after `heldStagePass` at line 4932), `internal/host/adoptlock.go` (the lock-order comment at lines 52-54 names the root pass lock between the adopt-pass lock and the restart marks), `internal/host/movesheldlock.go` (the order comment, lines 26-31), `internal/host/claimcensus.go` (`claimOptCensus` gains `<optDir>/root-cleanup` as box state, if its completeness test requires it), the copy record and `approved_moves_copy_test.go` (root's acknowledgement reason) |
| +- Test: `internal/release/rootcleanup_test.go`, `internal/release/carlosprocs_linux_test.go`, `internal/host/movescleanup_root_test.go`, `internal/host/adopt_rootcleanup_stage_test.go`, `internal/host/adoptrootgate_test.go` |
| **Interfaces:** |
| - **The root pass lock (R5, F5; T7).** `func LockRootPass(optDir string) (unlock func(), err error)`, in this order, so no directory is created under an ancestry that has not passed: (1) `offlinecleanup.RootOnlyChain(optDir)` on the **existing** chain from `/` to `optDir`: false → `ErrNotRootOnly`, nothing created; an error → itself; (2) `RootCleanupBase(optDir) = <optDir>/root-cleanup/` created by root `0755` when absent (`os.Mkdir`; `EEXIST` is fine); (3) `RootOnlyChain` on the base (a base another uid made, or a symlink, answers false → `ErrNotRootOnly`); (4) `pass.lock` in it opened with `safefile.OpenFile(O_CREATE|O_RDWR, 0o600)` and `lockFileEx` (blocking `flock`), as `lockExclusive` does (`adoptlock.go:55-68`). `ErrNotRootOnly` is a definite answer about the box, not a failure to lock: only root can make a chain root-only or not, so every pass gets the same answer, and root's step can never run there (a writable `/opt` is a writable layout). **Why the step does not depend on the adopt-pass lock's identity:** that lock lives in the agent-writable `/data` (`adoptlock.go:16`), where a shared-uid writer can replace or plant its name (`safefile` then refuses it, and the pass continues unserialized, `agent.go:4655-4661`); the root pass lock lives on a root-only chain, which no other uid can change, so its acquisition needs no identity check. The adopt-pass lock is still required for the step (it is held, so the sweep keeps its own rule), but nothing relies on its identity. |
| -- **The gate (T2).** In `runAdoptUnderRootRelease`, right after `lockAdoptPass` (`agent.go:4655-4661`) and whatever that answered, in every mode (hourly, `-stage-only`, `-config-only`, `-restart-only`), before `ensureUnits` (`:4664`) and every later stage: `unlockRoot, rootLockErr := release.LockRootPass(release.OptDir())` (deferred when it succeeds); `gated := os.Getenv("CARLOS_OFFLINE_CLEANUP") == "on" || release.RootCleanupGated(release.OptDir())`; then `adoptRootGate(gated, rootLockErr)` answers one of three. **Serve** (`rootLockErr == nil`): the pass runs as today; when `gated`, it arms the box if `<optDir>/root-cleanup/gated` is absent (`ArmRootCleanup`, below), and its last stage is root's (the stage below: the agent check, `QuiesceRootCleanup`, and only then root's step). **Publish nothing** (`gated`, and `rootLockErr` neither nil nor `ErrNotRootOnly`): one `log.Printf("ALERT: adopt: root pass lock %s: %v: publishing nothing this pass", ...)` and `return nil`, so the pass runs none of `ensureUnits`, `recoverRootRestarts` (`:4695`; its resumes call `PointLive` at `rootrollback_resume.go:125` and `roothistory_recovery.go:216`), `rootJudgePrev` (`:4698`), `sweepStagedVersionsPass` (`:4720`), the `-restart-only` branch (`:4758`), config sync and the `-config-only` branch (`:4794`), `capabilityPass` (`:4837`), `linkRootHistory` (`:4888`), `release.Pass` (`:4920`: adoption and rollback, `adopt.go:1751`, `:1870`, `rollback.go:124`, and named recovery inside it) and `heldStagePass` (`:4932`: `materializeheld.go:460`, `:494`); the next pass tries again. **Publish without serving** (not `gated`, or `ErrNotRootOnly`): today's fail-open adoption, logged as the adopt-pass lock's failure is, and no root step. `func RootCleanupGated(optDir string) bool`: `Lstat` of `<optDir>/root-cleanup/gated`: present → true; `ENOENT` → false; any other error → true (fail closed). The marker is removed only by `carlos system cleanup-ungate` (below): it records that cleanup was armed on this box, so a disarmed box, whose obligations may still wait for root (Global Constraint "Rolling plan 4 back is disarming cleanup"), keeps serving them under the same rule. **Why this closes R5:** on a gated box every pass that publishes holds the root pass lock, and root's step runs only as the last stage of a pass holding it, so no root publisher of the live link or the programs file runs while root's step runs; the agent's own publishers are excluded by the freeze and the held-stage lock as before. On a box never gated no root step can run (it needs the lock and the switch or the marker), so fail-open adoption there races nothing. The moment a box first becomes gated is closed by the arming boundary. |
| -- **The arming boundary (U1; review 7's finding 2).** The gate leaves one interleaving: a pass that read the switch off and found no marker, and whose two locks failed (a planted `/data/adopt-pass.lock` and a root-level fault on `LockRootPass`), publishes fail-open; if it is still running when a later pass has armed the box and runs root's step, a publication can land between root's `fstatat` and `unlinkat`. Setting the switch or creating the marker does not stop a pass already past its gate. **How passes run, verified:** `carlos adopt` runs one pass per process and exits (`runAdopt`, `agent.go:4568`); it is started by four root oneshot units, `carlos-adopt.service` (`carlos-adopt.timer`, hourly), `carlos-stage.service` (`carlos-stage.path`, the agent's `/data/stage-requested`), `carlos-config-sync.service` (`carlos-adopt.path`, `/data/config-sync-requested`) and `carlos-restart.service` (`carlos-restart.path`, `/data/restart-requested`), each a separate process, and passes of different units can overlap ("the four units that exec this verb can start in the same second", `agent.go:4645-4648`). Within one unit nothing overlaps: systemd runs one instance of a unit, a start request for a oneshot unit that is still activating joins its job, and a unit is inactive again only once its control group is empty (the adopt units keep the default `KillMode=control-group`), so a restarted pass's predecessor has exited and its kernel operations have completed before the new one runs. An in-process mutex would therefore serialise nothing (no process runs two passes), and none is added; the boundary is a census of processes instead. **The census, by executable (V1).** `func CarlosProcesses(procDir, optDir string) ([]CarlosProcess, error)` (Linux; `type CarlosProcess struct { PID int; Verb string; RealUID, EffectiveUID uint32 }`): every numeric entry of `procDir` (`/proc` in production; the adopt units set neither `ProtectProc=` nor `PrivatePIDs=`, unlike `carlos-relay.service:32` and `carlos-instance-sidecar@.service:117-119`, so a root pass sees every process) **whose executable is the installed binary**: `readlink(<pid>/exe)`, with a trailing ` (deleted)` removed (what a task still running a binary an apply has renamed over reads, `internal/host/rootgeneration_services_linux.go:344-350`), equal to `<optDir>/carlos` or `<optDir>/carlos.prev`; or `stat(<pid>/exe)`, which follows to the task's image, naming the device and inode of either. `Verb` is `argv[1]` of `cmdline` split on NUL; the uids are the first two fields of `status`'s `Uid:` line. `argv[0]` is never read, so a renamed path or a hard link of the installed binary is still counted, and **an unrelated binary that runs `adopt` is outside support** (Self-review notes). An entry that vanishes while it is read (`ENOENT`, `ESRCH`) has exited and is skipped; any other error is returned, and an unknown census is never an empty one. `func AdoptPasses(procDir, optDir string, self int) ([]int, error)`: the processes with `Verb == "adopt"` and `RealUID == 0`, other than `self`. A process of another uid is not counted, so the agent's uid cannot hold the boundary shut. **The agent check (V1).** `func AgentRunsUnprivileged(ctx context.Context, procDir, optDir string, unitUser func(context.Context) (string, error)) (ok bool, reason string, err error)`, with `unitUser` in production `systemctl show -p User --value carlos-edge.service` (systemd's merged value, drop-ins included; root already asks systemd about this unit, `internal/host/claimstop_production.go:52`): a user "" (systemd's default, which is root), `root` or `0` → false, "carlos-edge.service runs the agent as root"; then `CarlosProcesses`: any with `Verb == "agent"` and `RealUID == 0 || EffectiveUID == 0` → false, "a carlos agent runs as root (pid <n>)"; an error from either → the error; else true. Off Linux `CarlosProcesses` and `AgentRunsUnprivileged` answer `ErrNotRootOnly`. **The arming and its nonce (V2; review 8, item 2).** `func ArmRootCleanup(optDir string) error`, called by a serving gated pass when `gated` is absent: a fresh nonce, 16 bytes from `crypto/rand` as 32 hex digits, written as `{"nonce": N, "at": <now>}` to a temp file in `RootCleanupBase`, `fsync`ed, `os.Link`ed to `gated` (create-only: `EEXIST` means another pass armed first, which is fine), the temp removed, the base `fsync`ed; root `0644`. Every creation of `gated` is an arming and has its own nonce. `func QuiesceRootCleanup(optDir string, census func() ([]int, error)) (bool, error)`, called only by a pass holding both locks with `gated` true, after every other stage and after the agent check: (1) `gated`: `Lstat` not a regular file owned by uid 0 with no group or other write bit, or its body (read bounded, 512 bytes) without a nonce of 32 hex digits → false, logged (fail closed: `cleanup-ungate` and a new arming repair it); (2) `<optDir>/root-cleanup/quiesced` present and recording that nonce → true; present with another nonce, or with none (written for an earlier arming, by hand, or by rev 8's form, which recorded the `(dev, inode)` of `gated`) → **void**: removed, the base `fsync`ed, and on to (3); (3) the census: an error → false with it; any pid → false, one `log.Printf("root-cleanup: waiting for adopt pass %v, which may predate arming, to end before the first cleanup step", pids)`; none → `quiesced` written create-only (temp, `fsync`, `os.Link`, base `fsync`) as `{"nonce": N, "at": <now>}` → true. Nothing reads the file identity of `gated`: a filesystem can give a recreated `gated` the inode of the removed one (review 8, item 2), and a nonce is never written twice. **The ungate.** `func UngateRootCleanup(optDir, dataDir string, switchOn bool) error` and the root verb `carlos system cleanup-ungate` (`host.RunSystemCleanupUngate`, run by hand as root on the box): it takes `LockRootPass`, so no adopt pass is between its gate and its last stage; `CARLOS_OFFLINE_CLEANUP=on` in `/etc/carlos/host.env` → refused, "cleanup is still on in /etc/carlos/host.env. unset CARLOS_OFFLINE_CLEANUP and restart the agent first."; a drop under `<dataDir>/.offline-moves/*/*/root-cleanup.json`, or a root `<id>/` with `request.json` and no acknowledgement → refused, "a cleanup request still waits for root: <id>."; else `quiesced` removed and the base `fsync`ed, **then** `gated` removed and the base `fsync`ed, and it prints "cleanup is ungated on this box. adopt passes no longer need the root pass lock." A crash between the two removals leaves `gated` without `quiesced`, so the arming's next certificate is written only after a census; a crash before the first leaves both, and the certificate still certifies the arming it was written for. It is the only code that removes `gated`. **Why it closes the first arming:** a pass that publishes without the root pass lock is one whose gate answered "publish without serving", so it read the switch off and found no `gated` marker; `quiesced` is written for a nonce only after the `gated` carrying that nonce existed, at a moment the census found no other root adopt process of the installed binary alive; every such process alive after that moment was started after it, so after `gated` existed, and reads the marker (`RootCleanupGated` fails closed on any `Lstat` error but `ENOENT`), so it holds the root pass lock or publishes nothing. A `/proc` walk lists every process that exists for the whole walk, and one started during the walk started after `gated` existed. A certificate for an earlier arming certifies nothing about this one, because its nonce differs, and is removed before the census. So once `quiesced` stands for the current nonce, no pass admitted before this arming is still running, and root's step, which requires it, never meets one. A census that keeps finding a process (a pass blocked behind this one's lock counts too) costs only time: the next pass, at the latest the next hour, tries again, and the obligation pages `root-wait` meanwhile. |
| +- **The gate (T2).** In `runAdoptUnderRootRelease`, right after `lockAdoptPass` (`agent.go:4655-4661`) and whatever that answered, in every mode (hourly, `-stage-only`, `-config-only`, `-restart-only`), before `ensureUnits` (`:4664`) and every later stage: `unlockRoot, rootLockErr := release.LockRootPass(release.OptDir())` (deferred when it succeeds); `gated := os.Getenv("CARLOS_OFFLINE_CLEANUP") == "on" || release.RootCleanupGated(release.OptDir())`; then `adoptRootGate(gated, rootLockErr)` answers one of three. **Serve** (`rootLockErr == nil`): the pass runs as today; when `gated`, it arms the box if `<optDir>/root-cleanup/gated` is absent (`ArmRootCleanup`, below), and its last stage is root's (the stage below: the agent check, `QuiesceRootCleanup`, and only then root's step). **Publish nothing** (`gated`, and `rootLockErr` neither nil nor `ErrNotRootOnly`): one `log.Printf("ALERT: adopt: root pass lock %s: %v: publishing nothing this pass", ...)` and `return nil`, so the pass runs none of `ensureUnits`, `recoverRootRestarts` (`:4695`; its resumes call `PointLive` at `rootrollback_resume.go:125` and `roothistory_recovery.go:216`), `rootJudgePrev` (`:4698`), `sweepStagedVersionsPass` (`:4720`), the `-restart-only` branch (`:4758`), config sync and the `-config-only` branch (`:4794`), `capabilityPass` (`:4837`), `linkRootHistory` (`:4888`), `release.Pass` (`:4920`: adoption and rollback, `adopt.go:1751`, `:1870`, `rollback.go:124`, and named recovery inside it) and `heldStagePass` (`:4932`: `materializeheld.go:460`, `:494`); the next pass tries again. **Publish without serving** (not `gated`, or `ErrNotRootOnly`): today's fail-open adoption, logged as the adopt-pass lock's failure is, and no root step. `func RootCleanupGated(optDir string) bool`: `Lstat` of `<optDir>/root-cleanup/gated`: present → true; `ENOENT` → false; any other error → true (fail closed). No code removes the marker (W2; review 9, finding 2): it records that cleanup was armed on this box, so a disarmed box, whose obligations may still wait for root or may still publish a request (Global Constraint "Rolling plan 4 back is disarming cleanup"), keeps serving them under the same rule; the gate is judged by the markers, not by the switch. **Why this closes R5:** on a gated box every pass that publishes holds the root pass lock, and root's step runs only as the last stage of a pass holding it, so no root publisher of the live link or the programs file runs while root's step runs; the agent's own publishers are excluded by the freeze and the held-stage lock as before. On a box never gated no root step can run (it needs the lock and the switch or the marker), so fail-open adoption there races nothing. The moment a box first becomes gated is closed by the arming boundary. |
| +- **The arming boundary (U1; review 7's finding 2).** The gate leaves one interleaving: a pass that read the switch off and found no marker, and whose two locks failed (a planted `/data/adopt-pass.lock` and a root-level fault on `LockRootPass`), publishes fail-open; if it is still running when a later pass has armed the box and runs root's step, a publication can land between root's `fstatat` and `unlinkat`. Setting the switch or creating the marker does not stop a pass already past its gate. **How passes run, verified:** `carlos adopt` runs one pass per process and exits (`runAdopt`, `agent.go:4568`); it is started by four root oneshot units, `carlos-adopt.service` (`carlos-adopt.timer`, hourly), `carlos-stage.service` (`carlos-stage.path`, the agent's `/data/stage-requested`), `carlos-config-sync.service` (`carlos-adopt.path`, `/data/config-sync-requested`) and `carlos-restart.service` (`carlos-restart.path`, `/data/restart-requested`), each a separate process, and passes of different units can overlap ("the four units that exec this verb can start in the same second", `agent.go:4645-4648`). Within one unit nothing overlaps: systemd runs one instance of a unit, a start request for a oneshot unit that is still activating joins its job, and a unit is inactive again only once its control group is empty (the adopt units keep the default `KillMode=control-group`), so a restarted pass's predecessor has exited and its kernel operations have completed before the new one runs. An in-process mutex would therefore serialise nothing (no process runs two passes), and none is added; the boundary is a census of processes instead. **The census, by executable (V1).** `func CarlosProcesses(procDir, optDir string) ([]CarlosProcess, error)` (Linux; `type CarlosProcess struct { PID int; Verb string; RealUID, EffectiveUID uint32 }`): every numeric entry of `procDir` (`/proc` in production; the adopt units set neither `ProtectProc=` nor `PrivatePIDs=`, unlike `carlos-relay.service:32` and `carlos-instance-sidecar@.service:117-119`, so a root pass sees every process) **whose executable is the installed binary**: `readlink(<pid>/exe)`, with a trailing ` (deleted)` removed (what a task still running a binary an apply has renamed over reads, `internal/host/rootgeneration_services_linux.go:344-350`), equal to `<optDir>/carlos` or `<optDir>/carlos.prev`; or `stat(<pid>/exe)`, which follows to the task's image, naming the device and inode of either. `Verb` is `argv[1]` of `cmdline` split on NUL; the uids are the first two fields of `status`'s `Uid:` line. `argv[0]` is never read, so a renamed path or a hard link of the installed binary is still counted, and **an unrelated binary that runs `adopt` is outside support** (Self-review notes). An entry that vanishes while it is read (`ENOENT`, `ESRCH`) has exited and is skipped; any other error is returned, and an unknown census is never an empty one. `func AdoptPasses(procDir, optDir string, self int) ([]int, error)`: the processes with `Verb == "adopt"` and `RealUID == 0`, other than `self`. A process of another uid is not counted, so the agent's uid cannot hold the boundary shut. **The agent check (V1).** `func AgentRunsUnprivileged(ctx context.Context, procDir, optDir string, unitUser func(context.Context) (string, error)) (ok bool, reason string, err error)`, with `unitUser` in production `systemctl show -p User --value carlos-edge.service` (systemd's merged value, drop-ins included; root already asks systemd about this unit, `internal/host/claimstop_production.go:52`): a user "" (systemd's default, which is root), `root` or `0` → false, "carlos-edge.service runs the agent as root"; then `CarlosProcesses`: any with `Verb == "agent"` and `RealUID == 0 || EffectiveUID == 0` → false, "a carlos agent runs as root (pid <n>)"; an error from either → the error; else true. Off Linux `CarlosProcesses` and `AgentRunsUnprivileged` answer `ErrNotRootOnly`. **The arming and its nonce (V2; review 8, item 2).** `func ArmRootCleanup(optDir string) error`, called by a serving gated pass when `gated` is absent: a fresh nonce, 16 bytes from `crypto/rand` as 32 hex digits, written as `{"nonce": N, "at": <now>}` to a temp file in `RootCleanupBase`, `fsync`ed, `os.Link`ed to `gated` (create-only: `EEXIST` means another pass armed first, which is fine), the temp removed, the base `fsync`ed; root `0644`. Every creation of `gated` is an arming and has its own nonce. `gated` is absent only before the box's first arming, or after an operator removed it by hand, which is the one repair for a malformed marker and is made with the switch on, so every pass stays gated meanwhile. So arming again after a disarm, with `gated` still standing, keeps its nonce and its certificate, which is sound because every pass since that arming was gated. `func QuiesceRootCleanup(optDir string, census func() ([]int, error)) (bool, error)`, called only by a pass holding both locks with `gated` true, after every other stage and after the agent check: (1) `gated`: `Lstat` not a regular file owned by uid 0 with no group or other write bit, or its body (read bounded, 512 bytes) without a nonce of 32 hex digits → false, logged (fail closed: an operator repairs it by removing `gated` by hand with the switch on, and the next serving pass arms afresh with a new nonce); (2) `<optDir>/root-cleanup/quiesced` present and recording that nonce → true; present with another nonce, or with none (written for an earlier arming, by hand, or by rev 8's form, which recorded the `(dev, inode)` of `gated`) → **void**: it certifies nothing and is left in place, and on to (3); (3) the census: an error → false with it; any pid → false, one `log.Printf("root-cleanup: waiting for adopt pass %v, which may predate arming, to end before the first cleanup step", pids)`; none → `quiesced` written as `{"nonce": N, "at": <now>}` (temp, `fsync`, `rename` over any void certificate, base `fsync`; two writers are never concurrent, because each holds the root pass lock) → true. Nothing removes `quiesced` (W2): a void one is replaced only by a certificate for the current nonce, after its census. Nothing reads the file identity of `gated`: a filesystem can give a recreated `gated` the inode of the removed one (review 8, item 2), and a nonce is never written twice. **Why it closes the first arming:** a pass that publishes without the root pass lock is one whose gate answered "publish without serving", so it read the switch off and found no `gated` marker; `quiesced` is written for a nonce only after the `gated` carrying that nonce existed, at a moment the census found no other root adopt process of the installed binary alive; every such process alive after that moment was started after it, so after `gated` existed, and reads the marker (`RootCleanupGated` fails closed on any `Lstat` error but `ENOENT`), so it holds the root pass lock or publishes nothing. A `/proc` walk lists every process that exists for the whole walk, and one started during the walk started after `gated` existed. A certificate for an earlier arming certifies nothing about this one, because its nonce differs, and is replaced only after a census for this one. So once `quiesced` stands for the current nonce, no pass admitted before this arming is still running, and root's step, which requires it, never meets one. A census that keeps finding a process (a pass blocked behind this one's lock counts too) costs only time: the next pass, at the latest the next hour, tries again, and the obligation pages `root-wait` meanwhile. |
| - **The stage (S12).** `rootCleanupPass(...)` runs after `heldStagePass` (line 4932), so after `release.Pass`, inside which named recovery runs (`recovery.passOption`, `agent.go:4890`, `namedrecovery_pass.go:15`), and only when `passLockErr == nil && rootLockErr == nil && gated`; nonfatal. It first runs the agent check, `release.AgentRunsUnprivileged(ctx, "/proc", release.OptDir(), unitUser)`: an error → nothing this pass; false → one `log.Printf("ALERT: root-cleanup: %s; cleanup needs the agent to run as its own user", reason)`, no `quiesced` written, and every request this pass would serve (each drop that passes the checks (0) and (1) below and is taken, and each taken request) answered `cancelled` with `Reason: "cleanup needs the agent to run as its own user"`, with nothing touched (no path step, no authority call); true → `QuiesceRootCleanup(release.OptDir(), func() ([]int, error) { return release.AdoptPasses("/proc", release.OptDir(), os.Getpid()) })`, and only when that answers true, `ServeRootCleanupRequests`. The check runs once per pass, before any path is touched. `-config-only` and `-restart-only` return before it (`agent.go:4758-4767`, `:4794-4801`) but have taken the root pass lock, so they wait for a running step and a running step waits for them. Inside one pass the stages run one after another, so no root publisher of this pass runs while root's step runs, and every other adopt pass is blocked on the root pass lock. |
| - **Files.** The agent's drop `<OfflineOpDir>/root-cleanup.json` (create-only) and cancel `<OfflineOpDir>/root-cleanup.cancel` (create-only). Root's directory `release.RootCleanupDir(optDir, id) = <optDir>/root-cleanup/<id>/` holding `request.json` (root's copy), `journal` (Task 6c) and exactly one of `done.json`, `cancelled.json`. Each `<id>/` is created by root `0755` only after `RootOnlyChain` has passed on the base (T7: the existing ancestry is checked before any missing directory is created), and used only when the chain passes on it too. **When it does not (S16), root writes nothing there**, removes nothing, and leaves the drop where it is: it logs `root-cleanup: <id> refused: <dir> is not root-only; repair its ownership, or remove <drop> to give up` to the adopt unit's journal (`log.Printf`, the systemd journal of `carlos-adopt.service`), which is the only record of the refusal. A repaired ancestry lets the next pass take the request. |
| - The wire types (declared in Task 1): `type RootCleanupRequest struct { V int; CleanupID, MoveID, Host, Account, App, Box string; Incarnation uint64; From, To string; Bucket, BucketRegion, Prefix string; Paths []RootCleanupPath; WrittenAt time.Time }`; `type RootCleanupPath struct { Which string; Identity FileIdentity }`; `type RootCleanupAck struct { V int; CleanupID, MoveID, Host, Outcome string (done or cancelled); Removed, Absent []string; Kept []KeptPath; Reason string; At time.Time }`. |
| @@ -319,10 +325,10 @@ Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and |
| - `func rootCleanupPass(ctx context.Context, dataDir, optDir string, reg *registry.Registry, store release.Store, judge *provision.LifetimeJudge, box, region string)`: `ServeRootCleanupRequests` with `Authorize` = Task 6a's callback, `Freeze` = `reg.WithHostFrozen` (a non-nil row is `rowPresent`), `HostLock` = `withHeldStageLock(dataDir, ·, ·)`, `Processes` reading `/proc/*/exe` (`ENOENT`/`ESRCH` skipped as exited, any other error returned). |
| - **The agent's rows** (Task 4a's row 6): `publishing`: `supersededBy` → supersede; the drop absent and `RootCleanupDir` absent → `writeRootCleanupRequest` (create-only; an existing drop with equal bytes is fine) → touch the stage-request marker (`holdStageRequester(dataDir)(Host)`, `moveshold.go:277`) → `RootRequestedAt = now`, `root-requested`, event; either present (a crash after the drop) → `root-requested` with `RootRequestedAt` restored, in this order: the drop's mtime when the drop exists; else the `at` of root's journal `taken` line; else `now`. `root-requested`: `supersededBy` → supersede (to `cancel-pending`); `ReadRootCleanupAck(optDir, ID)` → an acknowledgement whose `CleanupID`, `Host`, `MoveID` match → `Root = ack`, event, `releasing` with the outcome table's answer; a mismatch → page `corrupt-<h>`, stay; none → the drop is read **first** and then `RootCleanupDir`: both absent (an operator removed the drop before root took it; root takes a request into its directory before it removes the drop) → `releasing`, `Outcome = superseded`; else, after `rootRefusedAfter` (1 h) since `RootRequestedAt` with the agent's own `RootOnlyChain` check of `RootCleanupBase` (or of `RootCleanupDir` when it exists) failing → event `root-refused` and the hourly `root-refused-<h>` page, stay (**never a release**; the mark, which has no expiry, and the entries stay until an operator repairs the ancestry or removes the drop); else the `root-wait` pages. `cancel-pending`: the cancel file written create-only and `CancelRequestedAt` set (once); an acknowledgement → `releasing`, `Outcome = superseded`; else the drop is read **first** and then `RootCleanupDir`: both absent → `releasing`, `Outcome = superseded` (root takes a request into its directory before it removes the drop, so a drop already gone means root's directory exists, and no drop was written after `publishing`); else the `cancel-wait` pages. |
| -- [ ] **Step 1: Write the failing tests**: `TestRootCleanupStepRunsInsideThePass` (a pass driven through `runAdoptUnderRootRelease`'s stages with fakes: the step runs after `release.Pass` and `heldStagePass`; with the adopt-pass lock failed, with the root pass lock failed, and on `-config-only` and `-restart-only`, no step; every mode takes the root pass lock before its first publisher); `TestRootPassLockIsRootOnly` (a base directory the test uid owns → `ErrNotRootOnly`, no lock; the lock's chain check runs as any uid against an injected chain check); `TestLockRootPassChecksAncestryBeforeCreating` (T7: an `optDir` whose existing chain fails → `errors.Is(err, release.ErrNotRootOnly)` and `os.Lstat(base)` answers `fs.ErrNotExist` afterwards; a root-only `optDir` without a base → created, checked, locked, in that order in the injected syscall log; a base that is a symlink → `ErrNotRootOnly`, nothing created through it); `TestGatedPassPublishesNothingWithoutTheRootPassLock` (T2: a pass driven through `runAdoptUnderRootRelease`'s stages with a recording fake per stage; the switch on and `LockRootPass` failing with an injected `EIO` → `len(calls) == 0` across `ensureUnits`, `recoverRootRestarts`, `rootJudgePrev`, `sweepStagedVersionsPass`, the restart pass, config sync, `capabilityPass`, `linkRootHistory`, `release.Pass`, `heldStagePass` and the root step, in each of the four modes, and one log line; the switch off with the marker present → the same; the next pass with the lock → every stage in today's order and the step last); `TestUngatedBoxKeepsFailOpenAdoption` (the switch off and no marker, the lock failing → every stage runs as today, no step, no marker created; `ErrNotRootOnly` with the switch on → the same); `TestFirstArmingWaitsForAFailOpenPassToEnd` (U1's witness, review 7's finding 2 interleaving, through `runAdoptUnderRootRelease`'s stages with the production lock functions, an injected `LockRootPass` fault for one pass and an injected census: pass A, with the switch off, no marker, `/data/adopt-pass.lock` replaced by a symlink and `LockRootPass` failing with `EIO`, runs fail-open and is paused inside `release.Pass` after reading its route; the switch goes on and a drop is written; pass B takes both locks, arms the box (`gated` with a nonce), its agent check answers true, and its census answers A's pid: `QuiesceRootCleanup` answers false, `os.Lstat(quiesced)` answers `fs.ErrNotExist`, and root's step is not called, `len(stepCalls) == 0`; A resumes, publishes a new live link for the host and exits, and the census answers no pid; pass C writes `quiesced` and runs root's step, which finds the link A published, not the recorded one: `kept`, "identity changed", and `release.LiveLink(optDir, host)` still names A's target); `TestAdoptPassesCensus` (V1: a fake `/proc` tree whose `exe` entries are symlinks into a fake `optDir`: uid-0 processes running `adopt --stage-only` through `<opt>/carlos`, through `<opt>/carlos.prev`, through an `exe` that reads `<opt>/carlos (deleted)`, and through a hard link of `<opt>/carlos` at another path (`stat` names its inode) are each counted, and so is one whose `argv[0]` reads `x`; a uid-0 `adopt` through `/usr/local/bin/carlos`, a copy with another inode, is not (outside support), nor are `self`, a uid-1001 process through `<opt>/carlos` running `adopt`, a uid-0 `<opt>/carlos agent` and an entry removed during the walk; an unreadable `status` → the error); `TestStaleCertificateIsVoid` (V2's witness, forcing identity reuse: `gated` armed with nonce N1; beside it, in turn, a `quiesced` the test writes with nonce N0, and one in rev 8's form whose `gated_dev` and `gated_ino` equal the current `gated`'s `Lstat` identity and which has no nonce; with the injected census answering a pid, `QuiesceRootCleanup` answers false, `errors.Is(err, fs.ErrNotExist)` for `os.Lstat(quiesced)` afterwards, and `len(stepCalls) == 0`; with the census then answering none, `quiesced` is written with `nonce == N1` and the step runs; a `gated` that is a symlink, is owned by another uid, or holds `"nonce": "zz"` → false and nothing written); `TestArmingWritesAFreshNonce` (two armings with an ungate between: `n1 != n2`, each matching `^[0-9a-f]{32}$`; the injected syscall log of one arming is write temp, `fsync` temp, link `gated`, unlink temp, `fsync` base; two passes arming at once: one link wins, the other's `os.Link` answers `EEXIST`, and both then read the winner's nonce); `TestUngateRemovesTheCertificateFirst` (V2: the syscall log is lock `pass.lock`, unlink `quiesced`, `fsync` base, unlink `gated`, `fsync` base; a crash injected after the first `fsync`: `gated` stands without `quiesced`, and the next armed pass takes the census before its step; refused with the switch on, with a drop present and with a taken request unacknowledged: nothing removed and the refusal printed; the verb run by a non-root uid → refused before any lock); `TestRootStepRefusesARootAgent` (V1: `unitUser` answering "", "root" and "0", and separately `carlos` with a fake `/proc` holding a uid-0 `<opt>/carlos agent`: `len(stepCalls) == 0`, `len(authorizeCalls) == 0`, `os.Lstat(quiesced)` answers `fs.ErrNotExist`, every drop and taken request is answered `cancelled` with `ack.Reason == "cleanup needs the agent to run as its own user"`, and one ALERT line; the agent's next visit reads the acknowledgement and goes to `releasing` with `Outcome == "cancelled"`; `unitUser` failing → nothing answered and nothing written this pass; `carlos` and no root agent → the step runs); `TestArmedPassLeavesTheGatedMarker` (an armed pass holding the lock creates the marker; a later pass with the switch off is gated; a marker `Lstat` error other than `ENOENT` → gated); `TestRootFindsItsOwnRequestsOnRestart` (root takes a request, removes the drop and stops: the next pass serves it from `request.json`); `TestForgedAgentFilesAreHarmless` (a `done.json` planted in the agent's directory is never read; a drop with another id is cancelled under the expected id; a second drop with other bytes after the take is removed and changes nothing; a forged cancel stops root and ends the obligation `cancelled`, nothing else); `TestRejectedAncestryGetsNoWrite` (a `RootCleanupBase` or `<id>/` that fails the chain check: nothing created, written or removed under it, the drop kept, one log line; repaired, the next pass takes it); `TestAgentPagesARefusedAncestryAndNeverReleases` (no answer for 1 h with the chain failing from the agent's view: hourly `root-refused` pages, the obligation stays `root-requested` with both entries and the mark for a simulated week; the drop removed by an operator → `releasing`, `superseded`); `TestAgentReadsOnlyRootsAcknowledgement` (an untrusted acknowledgement → `superseded`, paths kept "writable layout"); `TestRootRequestedAtIsRestored` (a crash after the drop: from its mtime; after the take: from the `taken` line); `TestUnansweredRootRequestPagesEveryTenMinutes`; `TestCancelBeforeAndDuringRootsWork`; `TestCancelPendingResolvesWithoutARequest` (a crash after `publishing` before the drop; root never wrote a directory); `TestUnsupportedPlatformCancels`. |
| -- [ ] **Step 2: Run to verify they fail**: `go test ./internal/release -run 'RootFinds|ForgedAgent|RejectedAncestry|RootPassLock|LockRootPassChecks|CancelBefore|Unsupported' -count=1`, `go test ./internal/release -run 'AdoptPassesCensus|StaleCertificate|ArmingWrites' -count=1`, `go test ./internal/host -run 'RootCleanupStep|GatedPass|UngatedBox|ArmedPassLeaves|FirstArming|UngateRemoves|RootStepRefuses|AgentPages|AgentReadsOnly|RootRequestedAt|Unanswered|CancelPending' -count=1`. |
| +- [ ] **Step 1: Write the failing tests**: `TestRootCleanupStepRunsInsideThePass` (a pass driven through `runAdoptUnderRootRelease`'s stages with fakes: the step runs after `release.Pass` and `heldStagePass`; with the adopt-pass lock failed, with the root pass lock failed, and on `-config-only` and `-restart-only`, no step; every mode takes the root pass lock before its first publisher); `TestRootPassLockIsRootOnly` (a base directory the test uid owns → `ErrNotRootOnly`, no lock; the lock's chain check runs as any uid against an injected chain check); `TestLockRootPassChecksAncestryBeforeCreating` (T7: an `optDir` whose existing chain fails → `errors.Is(err, release.ErrNotRootOnly)` and `os.Lstat(base)` answers `fs.ErrNotExist` afterwards; a root-only `optDir` without a base → created, checked, locked, in that order in the injected syscall log; a base that is a symlink → `ErrNotRootOnly`, nothing created through it); `TestGatedPassPublishesNothingWithoutTheRootPassLock` (T2: a pass driven through `runAdoptUnderRootRelease`'s stages with a recording fake per stage; the switch on and `LockRootPass` failing with an injected `EIO` → `len(calls) == 0` across `ensureUnits`, `recoverRootRestarts`, `rootJudgePrev`, `sweepStagedVersionsPass`, the restart pass, config sync, `capabilityPass`, `linkRootHistory`, `release.Pass`, `heldStagePass` and the root step, in each of the four modes, and one log line; the switch off with the marker present → the same; the next pass with the lock → every stage in today's order and the step last); `TestNeverArmedBoxKeepsFailOpenAdoption` (the switch off and no marker, the lock failing → every stage runs as today, no step, no marker created; `ErrNotRootOnly` with the switch on → the same); `TestFirstArmingWaitsForAFailOpenPassToEnd` (U1's witness, review 7's finding 2 interleaving, through `runAdoptUnderRootRelease`'s stages with the production lock functions, an injected `LockRootPass` fault for one pass and an injected census: pass A, with the switch off, no marker, `/data/adopt-pass.lock` replaced by a symlink and `LockRootPass` failing with `EIO`, runs fail-open and is paused inside `release.Pass` after reading its route; the switch goes on and a drop is written; pass B takes both locks, arms the box (`gated` with a nonce), its agent check answers true, and its census answers A's pid: `QuiesceRootCleanup` answers false, `os.Lstat(quiesced)` answers `fs.ErrNotExist`, and root's step is not called, `len(stepCalls) == 0`; A resumes, publishes a new live link for the host and exits, and the census answers no pid; pass C writes `quiesced` and runs root's step, which finds the link A published, not the recorded one: `kept`, "identity changed", and `release.LiveLink(optDir, host)` still names A's target); `TestAdoptPassesCensus` (V1: a fake `/proc` tree whose `exe` entries are symlinks into a fake `optDir`: uid-0 processes running `adopt --stage-only` through `<opt>/carlos`, through `<opt>/carlos.prev`, through an `exe` that reads `<opt>/carlos (deleted)`, and through a hard link of `<opt>/carlos` at another path (`stat` names its inode) are each counted, and so is one whose `argv[0]` reads `x`; a uid-0 `adopt` through `/usr/local/bin/carlos`, a copy with another inode, is not (outside support), nor are `self`, a uid-1001 process through `<opt>/carlos` running `adopt`, a uid-0 `<opt>/carlos agent` and an entry removed during the walk; an unreadable `status` → the error); `TestStaleCertificateIsVoid` (V2's witness, forcing identity reuse: `gated` armed with nonce N1; beside it, in turn, a `quiesced` the test writes with nonce N0, and one in rev 8's form whose `gated_dev` and `gated_ino` equal the current `gated`'s `Lstat` identity and which has no nonce; with the injected census answering a pid, `QuiesceRootCleanup` answers false, `quiesced` is left byte for byte as the test wrote it (nothing removes it), and `len(stepCalls) == 0`; with the census then answering none, it is replaced by one with `nonce == N1` and the step runs; a `gated` that is a symlink, is owned by another uid, or holds `"nonce": "zz"` → false and nothing written); `TestArmingWritesAFreshNonce` (two armings with `gated` removed by hand between, the only way a box is armed twice: `n1 != n2`, each matching `^[0-9a-f]{32}$`; the injected syscall log of one arming is write temp, `fsync` temp, link `gated`, unlink temp, `fsync` base; two passes arming at once: one link wins, the other's `os.Link` answers `EEXIST`, and both then read the winner's nonce); `TestDisarmedPublicationIsStillServed` (W2's witness, review 9's finding 2 interleaving: an armed box, with `gated` and `quiesced` standing for nonce N; an obligation reaches `publishing` and `cleanupBeforeDropHook` pauses its visit immediately before its first drop; `CARLOS_OFFLINE_CLEANUP` is unset in the test's `host.env` and the agent restarts with `cfg.CleanupArmed == false`, the paused worker discarded; the new worker's visit writes the drop and touches the stage-request marker; root's next pass, driven through `runAdoptUnderRootRelease`'s stages with the switch off: `adoptRootGate` answers serve, because `gated` stands, the agent check answers true, `QuiesceRootCleanup` answers true with no census call (`quiesced` names N), and root takes the request and acknowledges `done`, with `release.LiveLink(optDir, host)` absent afterwards; the agent's next visit reads the acknowledgement: `releasing`, `r.Outcome == "cleaned"`, the mark lifted, the release withheld with `release-withheld` owed, and `len(fake.PutsAfter(mark, moves.OfflineActiveKey)) == 0`; `gated` and `quiesced` are byte for byte what they were before the disarm); `TestNothingRemovesTheGateMarkers` (W2: `go/ast` with `go/types` over the non-test files of `internal/release` and `internal/host`: the marker names are referenced only by `ArmRootCleanup`, `RootCleanupGated` and `QuiesceRootCleanup`, and no `os.Remove`, `os.RemoveAll`, `unix.Unlinkat` or `os.Rename` takes `gated` as its path, or `quiesced` as anything but a rename's target; and in behaviour, an armed box disarmed and driven through ten passes, with and without the root pass lock, and through root's 30-day removal of its `<id>/` directories, keeps both files byte for byte); `TestRootStepRefusesARootAgent` (V1: `unitUser` answering "", "root" and "0", and separately `carlos` with a fake `/proc` holding a uid-0 `<opt>/carlos agent`: `len(stepCalls) == 0`, `len(authorizeCalls) == 0`, `os.Lstat(quiesced)` answers `fs.ErrNotExist`, every drop and taken request is answered `cancelled` with `ack.Reason == "cleanup needs the agent to run as its own user"`, and one ALERT line; the agent's next visit reads the acknowledgement and goes to `releasing` with `Outcome == "cancelled"`; `unitUser` failing → nothing answered and nothing written this pass; `carlos` and no root agent → the step runs); `TestArmedPassLeavesTheGatedMarker` (an armed pass holding the lock creates the marker; a later pass with the switch off is gated; a marker `Lstat` error other than `ENOENT` → gated); `TestRootFindsItsOwnRequestsOnRestart` (root takes a request, removes the drop and stops: the next pass serves it from `request.json`); `TestForgedAgentFilesAreHarmless` (a `done.json` planted in the agent's directory is never read; a drop with another id is cancelled under the expected id; a second drop with other bytes after the take is removed and changes nothing; a forged cancel stops root and ends the obligation `cancelled`, nothing else); `TestRejectedAncestryGetsNoWrite` (a `RootCleanupBase` or `<id>/` that fails the chain check: nothing created, written or removed under it, the drop kept, one log line; repaired, the next pass takes it); `TestAgentPagesARefusedAncestryAndNeverReleases` (no answer for 1 h with the chain failing from the agent's view: hourly `root-refused` pages, the obligation stays `root-requested` with both entries and the mark for a simulated week; the drop removed by an operator → `releasing`, `superseded`); `TestAgentReadsOnlyRootsAcknowledgement` (an untrusted acknowledgement → `superseded`, paths kept "writable layout"); `TestRootRequestedAtIsRestored` (a crash after the drop: from its mtime; after the take: from the `taken` line); `TestUnansweredRootRequestPagesEveryTenMinutes`; `TestCancelBeforeAndDuringRootsWork`; `TestCancelPendingResolvesWithoutARequest` (a crash after `publishing` before the drop; root never wrote a directory); `TestUnsupportedPlatformCancels`. |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/release -run 'RootFinds|ForgedAgent|RejectedAncestry|RootPassLock|LockRootPassChecks|CancelBefore|Unsupported' -count=1`, `go test ./internal/release -run 'AdoptPassesCensus|StaleCertificate|ArmingWrites|NothingRemovesTheGate' -count=1`, `go test ./internal/host -run 'RootCleanupStep|GatedPass|NeverArmedBox|ArmedPassLeaves|FirstArming|DisarmedPublication|RootStepRefuses|AgentPages|AgentReadsOnly|RootRequestedAt|Unanswered|CancelPending' -count=1`. |
| - [ ] **Step 3: Implement** (until Task 6c lands, the per-path step answers kept with reason "not built", so this commit removes nothing). **Step 4: Run to verify they pass**, then `GOOS=darwin go build ./internal/release/ ./internal/host/` and `GOOS=windows go vet ./internal/release/ ./internal/host/`. |
| -- [ ] **Step 5: Commit** `root: a stage of the adopt pass under a root-only pass lock, which an armed box's passes must hold to publish, runs only beside an agent that is not root and first runs once no adopt process of the installed binary that predates this arming's nonce is alive, is ungated only certificate first, takes a cleanup request into its own directory, writes nothing where the ancestry is not root's, and answers in a file the agent trusts only on a root-only chain`. |
| +- [ ] **Step 5: Commit** `root: a stage of the adopt pass under a root-only pass lock, which an armed box's passes must hold to publish, runs only beside an agent that is not root and first runs once no adopt process of the installed binary that predates this arming's nonce is alive, keeps its gate markers for the life of the box, takes a cleanup request into its own directory, writes nothing where the ancestry is not root's, and answers in a file the agent trusts only on a root-only chain`. |
| ### Task 6c: Root's filesystem step, its journal and recovery |
| @@ -415,7 +421,7 @@ Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and |
| - Create: `internal/provision/testdata/plan14-root-reads-self-serve-moves-4.tsv`, `internal/provision/self_serve_moves4_census_test.go`, `docs/superpowers/notes/census-proposal-self-serve-moves-4.md` |
| - Modify: `internal/provision/plan14_root_reads_test.go`, `internal/provision/plan9_b8_census_test.go` (line 372) |
| -- [ ] **Step 1: Write the tests**: `TestCleanupTwentyFourHoursAfterTheObservedFinish` (a forward move; at 24 h every inventoried version is gone, each removed by its version id, oldest first, no marker written by the cleanup, `lock.json` and `build-state.json` remain, the lifted tombstone is back, root removed the live link when its chain is root-only, the programs file is reported left, the entries are gone, the table is still there and empty, the report is under the move id and `carlos move offline status --log` shows it); `TestCleanupSupersededByAPlacementBack` (the placement is refused while the cleanup holds; after it ends, a placement back to the source runs); `TestCreateUnderCleanupRefusedThenAllowed`; `TestRowMintedUnderTheMarkSupersedes`; `TestLateAttemptAfterSupersessionChangesNothing` (a delete's first attempt held past the supersession; the second attempt answered under the mark; the release; a new wake writes; the held attempt is delivered: the instance serves its new data); `TestPromotionBackIntoTheRegionKeepsItsData` (a legacy move back into the source region while a superseded cleanup had a delete in flight: the delete is answered under the mark, the wake waits for the unmark, the promotion keeps equal-ETag versions in place and copies the rest, and the instance's next wake restores the full replica); `TestOperatorReturnAfterSupersededCleanupStillFindsItsLeftovers` (the return's start door accepts the forward's leftover set after a superseded cleanup that had marked and unmarked); `TestLegacyMoveIntoTheSourceRegionIsTolerated` (switch off: a legacy move back to the source region while the cleanup holds: the cleanup supersedes at its next key, answers any delete in flight, the wake waits for the unmark, the promotion restores the staged data and the instance serves it); `TestUntrackedLifetimeMoveFinishes` (an incarnation-0 record with no slot moved by an operator: `gone` reports, the move finishes, no obligation exists); `TestOlderCleanupInTheMoveLog` (two moves of one host, the first's cleanup open: `--log --cleanup <first id>` shows it after the second finished and after the record is deleted); `TestDisarmEndsObligationsWithoutTableWrites` (plan 4's rollback, V3: cleanups on two boxes, one `cleaning` with a delete held by the fake and one `root-requested`; both boxes disarmed and their agents restarted at `mark`: from then on no box request reaches `active.json` (`len(fake.PutsAfterBy(mark, moves.OfflineActiveKey, "box")) == 0`); the delete is answered under the mark, root answers, both obligations reach `releasing` with their marks lifted and their entries held, and each pages "cleanup of <host> waits for its reservation to be released: cleanup is off on this box" once; `carlos system reservations drain` releases both entries on their reports, settles a placement and a create paused between `BeginAllocation` and its reservation, and answers `drained`; a create of each host then succeeds); `TestRootRunAgentNeverCleansUp` (V1: the source box's agent runs with `agentRunsAsRoot` true and the switch on: at 24 h nothing is reserved, marked or deleted, the page "cleanup needs the agent to run as its own user" was sent once, and a request left from an earlier armed run is answered `cancelled` by root with that reason and nothing removed); `TestCleanupHoldsTheMarkUntilItsDeleteIsAnswered` (the cleanup's agent is stopped with a delete in flight for 25 h of the test clock while a legacy move returns the host to the source region: the return's wake waits; the agent restarts, the delete is answered, the mark lifts, and the instance serves the returned data). |
| +- [ ] **Step 1: Write the tests**: `TestCleanupTwentyFourHoursAfterTheObservedFinish` (a forward move; at 24 h every inventoried version is gone, each removed by its version id, oldest first, no marker written by the cleanup, `lock.json` and `build-state.json` remain, the lifted tombstone is back, root removed the live link when its chain is root-only, the programs file is reported left, the entries are gone, the table is still there and empty, the report is under the move id and `carlos move offline status --log` shows it); `TestCleanupSupersededByAPlacementBack` (the placement is refused while the cleanup holds; after it ends, a placement back to the source runs); `TestCreateUnderCleanupRefusedThenAllowed`; `TestRowMintedUnderTheMarkSupersedes`; `TestLateAttemptAfterSupersessionChangesNothing` (a delete's first attempt held past the supersession; the second attempt answered under the mark; the release; a new wake writes; the held attempt is delivered: the instance serves its new data); `TestPromotionBackIntoTheRegionKeepsItsData` (a legacy move back into the source region while a superseded cleanup had a delete in flight: the delete is answered under the mark, the wake waits for the unmark, the promotion keeps equal-ETag versions in place and copies the rest, and the instance's next wake restores the full replica); `TestOperatorReturnAfterSupersededCleanupStillFindsItsLeftovers` (the return's start door accepts the forward's leftover set after a superseded cleanup that had marked and unmarked; the lock now carries `cleanups: 1`, which the return does not read); `TestLegacyMoveIntoTheSourceRegionIsTolerated` (switch off: a legacy move back to the source region while the cleanup holds: the cleanup supersedes at its next key, answers any delete in flight, the wake waits for the unmark, the promotion restores the staged data and the instance serves it); `TestUntrackedLifetimeMoveFinishes` (an incarnation-0 record with no slot moved by an operator: `gone` reports, the move finishes, no obligation exists); `TestOlderCleanupInTheMoveLog` (two moves of one host, the first's cleanup open: `--log --cleanup <first id>` shows it after the second finished and after the record is deleted); `TestDisarmEndsObligationsWithoutTableWrites` (plan 4's rollback, V3: cleanups on two boxes, one `cleaning` with a delete held by the fake and one `root-requested`; both boxes disarmed and their agents restarted at `mark`: from then on no box request reaches `active.json` (`len(fake.PutsAfterBy(mark, moves.OfflineActiveKey, "box")) == 0`); the delete is answered under the mark, root (the switch off, `gated` standing) answers, both obligations reach `releasing` with their marks lifted and their entries held, and each pages "cleanup of <host> waits for its reservation to be released: cleanup is off on this box" once; `carlos system reservations drain` releases both entries on their reports, settles a placement and a create paused between `BeginAllocation` and its reservation, and answers `drained`; a create of each host is then still refused with 503 and "this console is draining its reservations. try again later", because the drain leaves admission closed; the console is restarted (a new `Server` over the same store and bucket, W3), and only then does a create of each host succeed); `TestRootRunAgentNeverCleansUp` (V1: the source box's agent runs with `agentRunsAsRoot` true and the switch on: at 24 h nothing is reserved, marked or deleted, the page "cleanup needs the agent to run as its own user" was sent once, and a request left from an earlier armed run is answered `cancelled` by root with that reason and nothing removed); `TestCleanupHoldsTheMarkUntilItsDeleteIsAnswered` (the cleanup's agent is stopped with a delete in flight for 25 h of the test clock while a legacy move returns the host to the source region: the return's wake waits; the agent restarts, the delete is answered, the mark lifts, and the instance serves the returned data). |
| - [ ] **Step 2: Run them; fix what they find within this plan's scope.** |
| - [ ] **Step 3: Run both census tests and the cross-builds**; write the census proposal with the loader block, the changed envelopes (`internal/host`, `internal/release`, `internal/registry`, `internal/lease`, `internal/moves`, `internal/console`, `internal/offlinecleanup`, the root package) and the units. |
| - [ ] **Step 4: Commit** `self-serve moves: cleanup end to end`; push. |
| @@ -425,12 +431,13 @@ Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and |
| ## Self-review notes |
| - Spec coverage: §7.6 the source replica (T1, T3, T4a, T4b, T4c, T4d), root-owned leftovers with the link and process checks (T6a, T6b, T6c), parked local state (T8), the staged objects of an aborted move (unchanged); §7.1's `cleanup` reservation and the hostname keys (T2a, T2b, T2c, T7a, T7b); §8's cleanup pages and every cleanup key in the move log (T5a, T5b); §9 rollout, and rollback as disarming cleanup with no console rollback (Global Constraints, T4a, T4d, T7c); §11's cleanup witnesses (T4a, T4c, T9). |
| -- **Stated deviations.** (1) Legacy moves hold no reservation. A legacy move into the source region carries a move on the record from its door onward, so condition (1) supersedes the cleanup at its next key, and its completion's wake cannot take the lease while the mark holds; after the unmark the wake's promotion mirrors the staged replica over the prefix (`promoteStagedReplica`, an exact mirror, called during the restore after acquisition, `backend_exec.go:819`), so no deleted key matters; review 4's writer audit confirms this normal-operation argument (completion changes the record; promotion waits for acquisition). A delete sent before the supersession is answered under the mark, before the unmark, so none is outstanding when the promotion keeps an equal-ETag version in place. Holding a key from the box at `MoveComplete` (`movespass.go:1617`) would add a box-side release and crash path and extend a legacy cut's downtime behind root's answer, for no exclusion the mark does not already give. (2) Placement keeps `SetRecordRegion`'s ETag-conditional write and settles an uncertain one by `fenceRecord`, after which no earlier PUT can land; the reservation entry is the persisted intent, so no journal table is added. (3) A cleanup entry is never reclaimed by another reserver (§7.1 allowed a status-based reclaim); the console pages instead. (4) Only the sealing box's local paths are cleaned; other boxes in the source region keep theirs, as today. (5) The create reserves after `BeginAllocation`, not before, so every reservation has an intent row behind it. (6) `build-state.json` is kept with `lock.json`. (7) **§7.6's "plain `DeleteObject`, `VersionId` deletes are never used" is amended (Paul's question 9):** every inventoried version is deleted by its version id, oldest first, the version current at `gone` last and required current before every deletion, so no older version is exposed (short of a version-id delete of that version by hand during the run) and no late request can touch another owner's version. The retired replica is then not recoverable from the bucket. (8) The table is never deleted (plan 2 deleted it when empty). (9) Root removes a leftover only on a root-only chain; the programs file, under the agent-writable `/data` on every built box, is recorded as left at `gone` and stays the runbook's, and so do both paths on a writable-opt box. (10) Root's rename-aside is gone; identity is compared on the opened parent with `fstatat`. (11) §7.6's "a key already carrying a delete marker is done" becomes: a key whose current entry is not its inventoried latest (a later version, or a marker the inventory does not hold) is `superseded` and nothing more of it is deleted, so a foreign marker and everything beneath it are left; a marker the inventory holds as the latest is deleted last, like any latest. (12) Root's step is a stage of the adopt pass, under a new root-only pass lock that every adopt pass takes; held publication's freeze and lock are unchanged. (13) **The cleaning mark has no expiry** (§7.6 has the cleanup hold a lease; here it is a state): a cleanup that cannot get an answer for a delete holds the host's replica marked, and with it every wake, landing and return of the host in this region, until the delete is answered or an operator abandons it after the page (Paul's question 9). (14) **Root's gate:** on a box where cleanup is armed (the root-owned `gated` marker, which only `cleanup-ungate` removes, or the switch on now), an adopt pass that cannot take the root pass lock, for any reason but a chain that is not root-only, publishes nothing that pass; a box never armed keeps today's fail-open adoption; root's step first runs after an arming only once a census of `/proc` finds no other root adopt process of the installed binary alive after it (`quiesced`, for that arming's nonce), and runs only beside an agent that is not root. (15) The bucket's versioning status is not read by code (the box roles lack `s3:GetBucketVersioning`). Two separate guarantees stand in for it: the listing at `gone` refuses a `null` id, so every inventoried id is usable; and the rollout's one-off `get-bucket-versioning` check establishes `Enabled`, which the listing cannot (a suspended bucket with only real ids passes it). Skipping the rollout check leaves deletion safe and lets later writes overwrite a `null` version the inventory never holds. (16) **Table writes are single attempts** (`moves.PutTable`), resent identically until answered, each registration removed only by its owner; the store door's client has no single-attempt write, so a box on the store door never reserves a cleanup. (17) **No console rollback across this plan (V3; §9 describes a rollback):** once a plan 4 console has written the table, a console from before this plan never runs against that deployment; plan 4 is rolled back by disarming cleanup on every box, a console defect by rolling forward, and an unarmed box never writes the table. `carlos system reservations drain` stays as an operator's tool. (18) **Cleanup requires an agent that does not run as root (V1):** the shipped `carlos-edge.service` is the supported topology; a box whose agent runs as root never arms, and root's stage refuses beside it. (19) **The arming identity is a nonce (V2):** `gated` carries a fresh random nonce per arming, `quiesced` certifies one nonce, and `gated` is removed only by `carlos system cleanup-ungate`, certificate first. |
| +- **Stated deviations.** (1) Legacy moves hold no reservation. A legacy move into the source region carries a move on the record from its door onward, so condition (1) supersedes the cleanup at its next key, and its completion's wake cannot take the lease while the mark holds; after the unmark the wake's promotion mirrors the staged replica over the prefix (`promoteStagedReplica`, an exact mirror, called during the restore after acquisition, `backend_exec.go:819`), so no deleted key matters; review 4's writer audit confirms this normal-operation argument (completion changes the record; promotion waits for acquisition). A delete sent before the supersession is answered under the mark, before the unmark, so none is outstanding when the promotion keeps an equal-ETag version in place. Holding a key from the box at `MoveComplete` (`movespass.go:1617`) would add a box-side release and crash path and extend a legacy cut's downtime behind root's answer, for no exclusion the mark does not already give. (2) Placement keeps `SetRecordRegion`'s ETag-conditional write and settles an uncertain one by `fenceRecord`, after which no earlier PUT can land; the reservation entry is the persisted intent, so no journal table is added. (3) A cleanup entry is never reclaimed by another reserver (§7.1 allowed a status-based reclaim); the console pages instead. (4) Only the sealing box's local paths are cleaned; other boxes in the source region keep theirs, as today. (5) The create reserves after `BeginAllocation`, not before, so every reservation has an intent row behind it. (6) `build-state.json` is kept with `lock.json`. (7) **§7.6's "plain `DeleteObject`, `VersionId` deletes are never used" is amended (Paul's question 9):** every inventoried version is deleted by its version id, oldest first, the version current at `gone` last and required current before every deletion, so no older version is exposed (short of a version-id delete of that version by hand during the run) and no late request can touch another owner's version. The retired replica is then not recoverable from the bucket. (8) The table is never deleted (plan 2 deleted it when empty). (9) Root removes a leftover only on a root-only chain; the programs file, under the agent-writable `/data` on every built box, is recorded as left at `gone` and stays the runbook's, and so do both paths on a writable-opt box. (10) Root's rename-aside is gone; identity is compared on the opened parent with `fstatat`. (11) §7.6's "a key already carrying a delete marker is done" becomes: a key whose current entry is not its inventoried latest (a later version, or a marker the inventory does not hold) is `superseded` and nothing more of it is deleted, so a foreign marker and everything beneath it are left; a marker the inventory holds as the latest is deleted last, like any latest. (12) Root's step is a stage of the adopt pass, under a new root-only pass lock that every adopt pass takes; held publication's freeze and lock are unchanged. (13) **The cleaning mark has no expiry** (§7.6 has the cleanup hold a lease; here it is a state): a cleanup that cannot get an answer for a delete holds the host's replica marked, and with it every wake, landing and return of the host in this region, until the delete is answered or an operator abandons it after the page (Paul's question 9). (14) **Root's gate:** on a box where cleanup is armed (the root-owned `gated` marker, which no code removes, or the switch on now), an adopt pass that cannot take the root pass lock, for any reason but a chain that is not root-only, publishes nothing that pass; a box never armed keeps today's fail-open adoption; root's step first runs after an arming only once a census of `/proc` finds no other root adopt process of the installed binary alive after it (`quiesced`, for that arming's nonce), and runs only beside an agent that is not root. (15) The bucket's versioning status is not read by code (the box roles lack `s3:GetBucketVersioning`). Two separate guarantees stand in for it: the listing at `gone` refuses a `null` id, so every inventoried id is usable; and the rollout's one-off `get-bucket-versioning` check establishes `Enabled`, which the listing cannot (a suspended bucket with only real ids passes it). Skipping the rollout check leaves deletion safe and lets later writes overwrite a `null` version the inventory never holds. (16) **Table writes are single attempts** (`moves.PutTable`), resent identically until answered, each registration removed only by its owner; the store door's client has no single-attempt write, so a box on the store door never reserves a cleanup. (17) **No console rollback across this plan (V3; §9 describes a rollback):** once a plan 4 console has written the table, a console from before this plan never runs against that deployment; plan 4 is rolled back by disarming cleanup on every box, a console defect by rolling forward, and an unarmed box never writes the table. `carlos system reservations drain` stays as an operator's tool. (18) **Cleanup requires an agent that does not run as root (V1):** the shipped `carlos-edge.service` is the supported topology; a box whose agent runs as root never arms, and root's stage refuses beside it. (19) **The arming identity is a nonce (V2):** `gated` carries a fresh random nonce per arming, `quiesced` certifies one nonce, and no code removes `gated` or `quiesced` (W2): disarming leaves both, so root keeps serving a request published after the disarm, and an arming after `gated` was removed by hand draws a new nonce, which voids the old certificate and forces a census. The cost: a box once armed keeps its adopt passes gated for its life, so a root pass lock fault there always means a pass that publishes nothing. (20) **No lock body recurs (W1):** `lock.json` gains the optional `cleanups` count, which older readers ignore; the unmark raises it instead of restoring the pre-mark body, and always writes, so a delayed mark attempt can never land after it. |
| - **First arming (U1).** Rev 7 named one residual: a pass admitted fail-open before the box was gated, still publishing when root's step first runs. Rev 8 closes it with the arming boundary (Task 6b): root's step requires `quiesced`, written only at a moment after `gated` existed when no other root adopt process was alive, so every pass that could publish without the root pass lock has exited first; rev 9 binds `quiesced` to the arming's nonce and counts processes by executable (below). The ruling asked for an in-process pass mutex and argued that processes do not overlap because the adopt unit is one service. Verified against the units: `carlos adopt` runs one pass per process, and four separate units start it (`carlos-adopt`, `carlos-stage`, `carlos-config-sync`, `carlos-restart`), whose passes can overlap (`agent.go:4645-4648`). One unit never overlaps itself (a restarted pass's predecessor has exited), but an in-process mutex would serialise nothing, so the boundary is a process census instead. On census-required layouts the outer root-release lock (`rootstate.WithRelease`, `internal/rootstate/release.go:14`) already serialises every pass, and a failure there is fatal; the census matters on census-optional layouts, where that wrapper runs the pass directly (`internal/host/rootreleaselock.go:38-40`). |
| - **The census's and the agent check's boundary (V1).** The supported invocation of `carlos adopt` is the four root units running the shipped binary at its installed path, `/opt/carlos/carlos` (`carlos-adopt.service:17`, `carlos-stage.service:31`, `carlos-config-sync.service:25`, `carlos-restart.service:32`). The census matches a process by its executable, never by its name: `readlink` of `/proc/<pid>/exe` (with ` (deleted)` stripped) against `<optDir>/carlos` and `<optDir>/carlos.prev`, or `stat` of it against their device and inode, so a renamed path, a hard link, or a binary an apply has since renamed over is still counted. An unrelated binary (another inode at another path) running `adopt` as root is outside support: the plan claims nothing about it, as it claims nothing about a root user editing a live link by hand. Likewise the agent check covers the shipped unit (systemd's merged `User=`) and any root process of the installed binary running `agent`; a root process of an unrelated binary that publishes live links is outside support, and so is a change of the agent's topology in the middle of one adopt pass (the check runs once per pass, before any path is touched, and the next pass sees the change). The supported topology, `carlos-edge.service` with `User=carlos` and its sandbox, is stated in the Global Constraint "Arming and rollout" and is a prerequisite of the rollout. |
| -- **The arming nonce (V2).** Rev 8 bound `quiesced` to the `(dev, inode)` of `gated`, which a filesystem may reuse after `gated` is removed (review 8, item 2). Rev 9 binds it to a nonce that each arming draws afresh and writes into `gated`; a certificate for any other nonce is void and removed before a census. The ungate removes `quiesced` before `gated`, so a crash between them leaves no certificate at all; even a certificate left by hand certifies nothing, because its nonce is not the next arming's. |
| +- **The arming nonce (V2).** Rev 8 bound `quiesced` to the `(dev, inode)` of `gated`, which a filesystem may reuse after `gated` is removed (review 8, item 2). Rev 9 binds it to a nonce that each arming draws afresh and writes into `gated`; a certificate for any other nonce is void, and is replaced only after a census for the current nonce. Rev 10 removes rev 9's ungate (W2; review 9, finding 2): its refusals covered existing drops and taken requests, not a request that an obligation still running on the disarmed box would publish later, which root, no longer gated, would never serve. No code removes either marker now, so a box once armed stays gated for its life; an arming happens only when `gated` is absent (the first, or after a hand removal), and a certificate left by hand certifies nothing, because its nonce is not the current one. |
| +- **The lock's cleanup count (W1).** Rev 9 restored the lifted tombstone byte for byte at the unmark, and the bucket's ETags are digests of the bytes, so a mark attempt still in the network after the unmark found its condition true again (review 9, finding 1). The fix keeps the lock's format: one optional field that every existing reader ignores (`readLock`, `LockState`, litestream's `readLease`), written only by the mark (as read) and the unmark (as read + 1), and carried by `LiftTombstone`, whose bytes are unchanged while the count is zero. The ruling's alternative, `last_cleanup` with the cleanup id and a count, would add nothing here: a mark's body already names its cleanup in its owner, and the count alone keeps the lifted bodies apart. A higher generation would also make the body new, but the generation is litestream's, and `AcquireLease` derives the next lease's from it (`leaser.go:103-106`), so the plan leaves it alone. |
| - Plan 2 amendments (built code, changed here): move selections hold `host/<hostname>`; the switch-off rule applies between moves only, so a cleanup neither blocks nor is blocked by an operator move on another box; recovery mode counts `move` entries only; the cleanup reclaim interface is deleted; `releaseBoxes` writes an empty table instead of deleting it; every table write carries `epoch` and `rev`. |
| -- Built-code changes this plan makes, by task: T1 `moves/offline.go`, `host/movesoffline.go`, `release/programhistory.go` (export); T2a `moves/offline.go` (kinds, epoch, rev, `valid()`), `moves/reserve.go` (new, with `PeekActive`), `moves/tablewrite.go` (new), `release/store.go` (`PutOnce`); T2b `console/offlinereserve.go`, `console/selfserve_pass.go`, `console/lifetimes_store.go`, `console/server.go` (`tableWrites`); T2c `console/lifetimes_recovery.go`, `console/selfserve_pages.go`; T3 `lease/cleaning.go` (new), `lease/lease.go`, `lease/tombstone.go`, `lease/lockstate.go` (the refusals), `host/movesoffline_land.go` (the landing's wait), `activator/backend_exec.go` and `activator/activator.go` (the wake's answer); T4b, T4c `host` (new files only); T4d `host/movespass.go` (`CleanupArmed`, the root agent's page), `host/movescleanup.go` (`armedTable`); T5a `host/movespaging.go`, `console/apiinstancemove_offline.go`; T5b `console/authcli.go`, `move.go`, `docs/site/cli.md`, the copy record; T6a `registry`; T6b `host/agent.go` (the lock and the gate), `host/adoptrootgate.go` (new), `release/carlosprocs_linux.go` and `_other.go` (new), `host/cleanupungate.go` (new), `system.go` (`cleanup-ungate`), `host/adoptlock.go` and `host/movesheldlock.go` (comments), `host/claimcensus.go`; T7a `console/store.go`, `lifetimes_store.go` (`host_key`, `Allocation`), `lifetimes_create.go`, `lifetimes_recovery.go` (the reload); T7b `fleets.go`, `console/store.go` (`claimRawSetAppPlacement`), `console/view.go` (`consoleRegistry`), `registry/registry.go` (`SetPlacementContext`), `registry/remote_dispatch.go` (the deadline), `offlinereserve.go`, `residence_sweep.go`; T7c `console/authcli.go`, `lifetimes_create.go`, `fleets.go`, `selfserve_door.go`, `system.go`, `docs/site/cli.md`; `cmd/console/main.go` is not changed. The mirror (`internal/lease/mirror.go`) is not changed. |
| -- Names used across tasks: `offlinecleanup.Retired` (`State`, `Outcome`, `Keys`, `Events`, `PagesPending`, `LocalLeft`, `Root`, `Rev`, `PublishedRev`, `MarkedAt`, `ReserveUncertain`, `InFlight`, `Superseding`, `RootRequestedAt`, `CancelRequestedAt`), `CorruptStub`, `CleanupID`, `InventoryKey`, `InventoryVersion`, `InFlightDelete`, `KeyOutcome` (`State`, `Done`), `Event`, `PendingPage`, `Report`, `FileIdentity`, `LocalPath`, `KeptPath`, `RootCleanupRequest`, `RootCleanupPath`, `RootCleanupAck`, `ValidateRetired`, `ValidateCorruptStub`, `ValidateReport`, `UsableVersionID`, `Identify`, `RootOnlyChain`, `InventoryKey.Latest`, `InventoryVersion.Seq`, `Report.InFlight`, `Report.AbandonPath`; `moves.HostKey`, `IsHostKey`, `Active`, `ReadActive`, `PeekActive`, `ReadForAcquire`, `NextActiveBody`, `PutTable`, `TableWrites` (`Unanswered`, `Resend`), `Op`, `Conflicts`, `Reserve`, `Holds`, `Release`, `ErrReserved`, `ErrUnreadable`, `ErrUncertain`, `ErrReservationMismatch`, `RecordAway`, `Away*`, `OfflineOpDir`, `RetiredFile`, `CleanupReportKey`, `OfflineKindCreate`, `OfflineKindPlace`; `lease.CleaningOwner`, `CleaningExpiry`, `IsCleaningOwner`, `MarkCleaning`, `UnmarkCleaning`, `ErrNotOurLock`, `VersionedStore.DeleteVersion`; `activator.errLeaseCleaning`, `cleaningRetryAfter`; `release.PutOnce`, `SingleAttemptPutter`, `ErrNoSingleAttemptPut`; `deleteInventoriedVersion`, `w.cleanupVisit`, `w.supersededBy`, `w.retireKey`, `w.settleInFlight`, `w.kickCleanup`, `cleanupRun`, `cleanupMu`, `w.tableWrites`, `w.armedTable`, `errDisarmed`, `w.rootAgentPage`, `cfg.CleanupArmed`, `agentRunsAsRoot`, `recordRetired`, `createRetired`, `writeRetired`, `markPublished`, `flushPageOutboxFor`, `reapRetired`; `release.ServeRootCleanupRequests`, `RootCleanupOptions`, `RootCleanupBase`, `RootCleanupDir`, `ReadRootCleanupAck`, `LockRootPass`, `RootCleanupGated`, `ArmRootCleanup`, `QuiesceRootCleanup`, `UngateRootCleanup`, `CarlosProcesses`, `CarlosProcess`, `AdoptPasses`, `AgentRunsUnprivileged`, `ErrRootCleanupRefused`, `ErrNotRootOnly`, `ProgramsPath`; `rootCleanupAuthority`, `rootCleanupPass`, `rootRefusedAfter`, `adoptRootGate`, `host.RunSystemCleanupUngate`, the files `root-cleanup/gated`, `root-cleanup/quiesced` and `cleanup.abandon`; `registry.WithHostFrozen`, `SetPlacementContext`; `consoleRegistry.SetPlacementContext`; `s.offlineCleanupsOf`, `s.reservePlacement`, `s.registerPlacement`, `s.deregisterPlacement`, `s.settlePlacement`, `s.placing`, `s.residenceSweep`, `s.releaseAllocationKey`, `s.store.Allocation`, `recoveryListedHook`, `s.admission`, `s.admitResidence`, `s.closeAdmission`, `errConsoleDraining`, `errHostUnderCleanup`, `s.tableWrites`; the CLI's `moveArgs.log`, `logMove`, `logCleanup`, `carlos system reservations drain` and `carlos system cleanup-ungate`. |
| +- Built-code changes this plan makes, by task: T1 `moves/offline.go`, `host/movesoffline.go`, `release/programhistory.go` (export); T2a `moves/offline.go` (kinds, epoch, rev, `valid()`), `moves/reserve.go` (new, with `PeekActive`), `moves/tablewrite.go` (new), `release/store.go` (`PutOnce`); T2b `console/offlinereserve.go`, `console/selfserve_pass.go`, `console/lifetimes_store.go`, `console/server.go` (`tableWrites`); T2c `console/lifetimes_recovery.go`, `console/selfserve_pages.go`; T3 `lease/cleaning.go` (new: the mark, the unmark, `lockBody`'s count and the single-attempt lock writes), `lease/leasetest/hold.go` (new, test support), `lease/lease.go`, `lease/tombstone.go` (the refusal, `readLockBody`, `LiftTombstone` through `lockBody`), `lease/lockstate.go` (the refusals), `host/movesoffline_land.go` (the landing's wait), `activator/backend_exec.go` and `activator/activator.go` (the wake's answer); T4b, T4c `host` (new files only); T4d `host/movespass.go` (`CleanupArmed`, the root agent's page), `host/movescleanup.go` (`armedTable`); T5a `host/movespaging.go`, `console/apiinstancemove_offline.go`; T5b `console/authcli.go`, `move.go`, `docs/site/cli.md`, the copy record; T6a `registry`; T6b `host/agent.go` (the lock and the gate), `host/adoptrootgate.go` (new), `release/carlosprocs_linux.go` and `_other.go` (new), `host/adoptlock.go` and `host/movesheldlock.go` (comments), `host/claimcensus.go`; T7a `console/store.go`, `lifetimes_store.go` (`host_key`, `Allocation`), `lifetimes_create.go`, `lifetimes_recovery.go` (the reload); T7b `fleets.go`, `console/store.go` (`claimRawSetAppPlacement`), `console/view.go` (`consoleRegistry`), `registry/registry.go` (`SetPlacementContext`), `registry/remote_dispatch.go` (the deadline), `offlinereserve.go`, `residence_sweep.go`; T7c `console/authcli.go`, `lifetimes_create.go`, `fleets.go`, `selfserve_door.go`, `system.go`, `docs/site/cli.md`; `cmd/console/main.go` is not changed. The mirror (`internal/lease/mirror.go`) is not changed. |
| +- Names used across tasks: `offlinecleanup.Retired` (`State`, `Outcome`, `Keys`, `Events`, `PagesPending`, `LocalLeft`, `Root`, `Rev`, `PublishedRev`, `MarkedAt`, `ReserveUncertain`, `InFlight`, `Superseding`, `RootRequestedAt`, `CancelRequestedAt`), `CorruptStub`, `CleanupID`, `InventoryKey`, `InventoryVersion`, `InFlightDelete`, `KeyOutcome` (`State`, `Done`), `Event`, `PendingPage`, `Report`, `FileIdentity`, `LocalPath`, `KeptPath`, `RootCleanupRequest`, `RootCleanupPath`, `RootCleanupAck`, `ValidateRetired`, `ValidateCorruptStub`, `ValidateReport`, `UsableVersionID`, `Identify`, `RootOnlyChain`, `InventoryKey.Latest`, `InventoryVersion.Seq`, `Report.InFlight`, `Report.AbandonPath`; `moves.HostKey`, `IsHostKey`, `Active`, `ReadActive`, `PeekActive`, `ReadForAcquire`, `NextActiveBody`, `PutTable`, `TableWrites` (`Unanswered`, `Resend`), `Op`, `Conflicts`, `Reserve`, `Holds`, `Release`, `ErrReserved`, `ErrUnreadable`, `ErrUncertain`, `ErrReservationMismatch`, `RecordAway`, `Away*`, `OfflineOpDir`, `RetiredFile`, `CleanupReportKey`, `OfflineKindCreate`, `OfflineKindPlace`; `lease.CleaningOwner`, `CleaningExpiry`, `IsCleaningOwner`, `MarkCleaning`, `UnmarkCleaning`, `ErrNotOurLock`, `lockBody` (`Cleanups`), `readLockBody`, `putLockUntilAnswered`, `lockAttemptTimeout`, `ErrLockWriteUncertain`, `leasetest.Holder`, `VersionedStore.DeleteVersion`; `activator.errLeaseCleaning`, `cleaningRetryAfter`; `release.PutOnce`, `SingleAttemptPutter`, `ErrNoSingleAttemptPut`; `deleteInventoriedVersion`, `w.cleanupVisit`, `w.supersededBy`, `w.retireKey`, `w.settleInFlight`, `w.kickCleanup`, `cleanupRun`, `cleanupMu`, `w.tableWrites`, `w.armedTable`, `errDisarmed`, `w.rootAgentPage`, `cfg.CleanupArmed`, `agentRunsAsRoot`, `recordRetired`, `createRetired`, `writeRetired`, `markPublished`, `flushPageOutboxFor`, `reapRetired`; `release.ServeRootCleanupRequests`, `RootCleanupOptions`, `RootCleanupBase`, `RootCleanupDir`, `ReadRootCleanupAck`, `LockRootPass`, `RootCleanupGated`, `ArmRootCleanup`, `QuiesceRootCleanup`, `CarlosProcesses`, `CarlosProcess`, `AdoptPasses`, `AgentRunsUnprivileged`, `ErrRootCleanupRefused`, `ErrNotRootOnly`, `ProgramsPath`; `rootCleanupAuthority`, `rootCleanupPass`, `rootRefusedAfter`, `adoptRootGate`, `cleanupBeforeDropHook`, the files `root-cleanup/gated`, `root-cleanup/quiesced` and `cleanup.abandon`; `registry.WithHostFrozen`, `SetPlacementContext`; `consoleRegistry.SetPlacementContext`; `s.offlineCleanupsOf`, `s.reservePlacement`, `s.registerPlacement`, `s.deregisterPlacement`, `s.settlePlacement`, `s.placing`, `s.residenceSweep`, `s.releaseAllocationKey`, `s.store.Allocation`, `recoveryListedHook`, `s.admission`, `s.admitResidence`, `s.closeAdmission`, `errConsoleDraining`, `errHostUnderCleanup`, `s.tableWrites`; the CLI's `moveArgs.log`, `logMove`, `logCleanup` and `carlos system reservations drain`. |
| - Review Focus 1 to 5 are pinned by T3+T4b+T4c, T4a, T6b+T6c, T2a-c+T7a-c, T5a+T5b+T8. |
| -- **Open questions for Paul** (amadan discussion 363): (1) legacy moves without a hostname reservation: accept the exemption, or reserve around `MoveComplete` on the box (a box-side release and crash path, and longer legacy-cut downtime while root answers)? (2) Cleanup runs 24 hours after **every** offline move, operator moves included, replacing the runbook's manual cleanup; intended (§10 Q4 talks about self-serve)? (3) Is the box switch `CARLOS_OFFLINE_CLEANUP=on` acceptable as the rollout gate? (4) A move of the same host waits while its cleanup holds `host/<hostname>`, which can last until root answers (paged every 10 min, hourly once root's ancestry is refused) or until an unanswered delete is answered or abandoned (paged every 10 min); acceptable? (5) The copy rows: five door sentences (the drain's 503, rev 9: "this console is draining its reservations. try again later"), the wake's 503 body for a replica under cleanup ("instance is being cleaned up after a move. retry shortly", rev 8), nine page subjects and one page body (rev 9 adds "cleanup of <host> waits for its reservation to be released: cleanup is off on this box"), the agent's page "cleanup needs the agent to run as its own user" with its body, which is also root's cancellation reason (rev 9), and the CLI's lines: the waiting-delete line, the drain's lines (rev 9 rewrites them: no rollback wording, a refusal while a self-serve move is in flight), the three lines of `carlos system cleanup-ungate` (rev 9), and the `--log` paragraph. (6) On every built box the programs file sits under the agent-writable `/data`, so root never removes it and it stays the runbook's; acceptable, or should the agent remove it itself? (7) Withdrawn by rev 6: the 15-minute `QuietUntil` and the marker-removing repair are gone; question 9 replaces it. (8) Rev 9: the drain, now an operator's tool, refuses while a self-serve move is in flight and then leaves admission open; once it has started, the console takes no new create or placement until it restarts. Acceptable? (9) **The §7.6 amendment:** the cleanup deletes every inventoried version of the retired replica by its version id (oldest first, the one current at `gone` last) instead of writing delete markers, so the source copy of a finished move is not recoverable from the bucket after the cleanup, in exchange for a boundary the server enforces (a late or retried request can remove only the version it names). **Rev 7 adds a consequence:** the mark that keeps every later owner off the replica while a delete may still land has no expiry, so a cleanup that cannot settle a delete (the bucket keeps refusing it, or the box is gone) holds the host's replica marked, and with it every wake, landing and return of that host in this region, until an operator acts on the page: by repairing what refuses the delete, or by abandoning it on the box (`cleanup.abandon`). Accept, or keep markers (then only `deleteInventoriedVersion` changes, and the settlement rules need a fresh review)? (10) **Rev 9, no console rollback across plan 4 (V3), the decision for Paul:** once a plan 4 console has written the reservation table, a console from before this plan is never run against that deployment again. A console defect is fixed by rolling forward, and plan 4 itself is rolled back by disarming cleanup on every box: no new obligations; existing ones finish, are superseded or page; one that needs its reservation released while its box is disarmed pages, and the drain releases it on the box's report. Rev 8's console rollback (the drain as a prerequisite, the clean-exit line, the 20-minute wait) is gone, because review 8 showed that no count or wait it used survives a console crash or is bounded by S3. Accept this as the operating rule? (11) **Rev 9, cleanup needs the agent to run as its own user (V1):** a box whose agent runs as root (the built code allows it, `agentUnitsOption`) never cleans up, pages once per agent start, and its leftovers stay the runbook's; the shipped `carlos-edge.service` runs it as `carlos`. Acceptable? |
| +- **Open questions for Paul** (amadan discussion 363): (1) legacy moves without a hostname reservation: accept the exemption, or reserve around `MoveComplete` on the box (a box-side release and crash path, and longer legacy-cut downtime while root answers)? (2) Cleanup runs 24 hours after **every** offline move, operator moves included, replacing the runbook's manual cleanup; intended (§10 Q4 talks about self-serve)? (3) Is the box switch `CARLOS_OFFLINE_CLEANUP=on` acceptable as the rollout gate? (4) A move of the same host waits while its cleanup holds `host/<hostname>`, which can last until root answers (paged every 10 min, hourly once root's ancestry is refused) or until an unanswered delete is answered or abandoned (paged every 10 min); acceptable? (5) The copy rows: five door sentences (the drain's 503, rev 9: "this console is draining its reservations. try again later"), the wake's 503 body for a replica under cleanup ("instance is being cleaned up after a move. retry shortly", rev 8), nine page subjects and one page body (rev 9 adds "cleanup of <host> waits for its reservation to be released: cleanup is off on this box"), the agent's page "cleanup needs the agent to run as its own user" with its body, which is also root's cancellation reason (rev 9), and the CLI's lines: the waiting-delete line, the drain's lines (rev 9 rewrites them: no rollback wording, a refusal while a self-serve move is in flight), and the `--log` paragraph; rev 10 withdraws rev 9's three `carlos system cleanup-ungate` lines with the verb. (6) On every built box the programs file sits under the agent-writable `/data`, so root never removes it and it stays the runbook's; acceptable, or should the agent remove it itself? (7) Withdrawn by rev 6: the 15-minute `QuietUntil` and the marker-removing repair are gone; question 9 replaces it. (8) Rev 9: the drain, now an operator's tool, refuses while a self-serve move is in flight and then leaves admission open; once it has started, the console takes no new create or placement until it restarts. Acceptable? (9) **The §7.6 amendment:** the cleanup deletes every inventoried version of the retired replica by its version id (oldest first, the one current at `gone` last) instead of writing delete markers, so the source copy of a finished move is not recoverable from the bucket after the cleanup, in exchange for a boundary the server enforces (a late or retried request can remove only the version it names). **Rev 7 adds a consequence:** the mark that keeps every later owner off the replica while a delete may still land has no expiry, so a cleanup that cannot settle a delete (the bucket keeps refusing it, or the box is gone) holds the host's replica marked, and with it every wake, landing and return of that host in this region, until an operator acts on the page: by repairing what refuses the delete, or by abandoning it on the box (`cleanup.abandon`). Accept, or keep markers (then only `deleteInventoriedVersion` changes, and the settlement rules need a fresh review)? (10) **Rev 9, no console rollback across plan 4 (V3), the decision for Paul:** once a plan 4 console has written the reservation table, a console from before this plan is never run against that deployment again. A console defect is fixed by rolling forward, and plan 4 itself is rolled back by disarming cleanup on every box: no new obligations; existing ones finish, are superseded or page; one that needs its reservation released while its box is disarmed pages, and the drain releases it on the box's report. Rev 8's console rollback (the drain as a prerequisite, the clean-exit line, the 20-minute wait) is gone, because review 8 showed that no count or wait it used survives a console crash or is bounded by S3. Accept this as the operating rule? (11) **Rev 9, cleanup needs the agent to run as its own user (V1):** a box whose agent runs as root (the built code allows it, `agentUnitsOption`) never cleans up, pages once per agent start, and its leftovers stay the runbook's; the shipped `carlos-edge.service` runs it as `carlos`. Acceptable? (12) **Rev 10, the gate is for the life of the box (W2):** once cleanup has been armed on a box, root's `gated` and `quiesced` markers stay there for good, and no command removes them. Disarming stops new cleanups but keeps root serving any request still owed, including one published after the disarm; the price is that every later adopt pass on that box needs the root pass lock to publish, so a fault on that lock means a pass that publishes nothing (logged as an ALERT) instead of today's fail-open adoption. Acceptable? |