amadan
plan 4 rev 8: first arming waits for every pre-arming root pass to end; recovery reloads the allocation under its locks; table writes with one attempt, resent until answered, and a 20-minute settlement wait before a drain completes; a finite retry for cleaning contention; the console gets a clean shutdown
Answers Astra's review of rev 7 (findings 2, 3, 4, 7). Four root units run adopt passes, so a /proc check and a quiesced marker replace an in-process mutex; the drain's settlement rests on S3 refusing a signed request older than 15 minutes (plus 5 for skew), checked on dev before arming.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1 file changed,
+79
−70
docs/superpowers/plans/2026-10-22-self-serve-moves-4-cleanup.md+79 −70
CI failed — run details
CI log
=== gate === sigwho: pid 2 ppid 1 pgid 0 sid 0; child pid 3 .amadan/test-gate.sh gate: 2026-10-24T10:21:37Z on aarch64, 32 cpus, 61797 MB available, go1.26.8 linux/arm64 gate: +0s start launchctx gate: +0s start vet gate: +0s start gofmt gate: +1s start rest gate: +11s start internal_console@0 gate: +11s start internal_console@1 gate: +11s start internal_console@2 gate: +12s start internal_console@3 gate: +12s start internal_console@4 gate: +12s start internal_console@5 gate: +12s start internal_console@6 gate: +12s start internal_console@7 gate: +12s waiting for internal_console's shards before the next split package gate: +175s start root@0 gate: +175s start root@1 gate: +175s start root@2 gate: +175s waiting for root's shards before the next split package amadan: step gate failed: signal: terminated amadan: job failed at step gate — its output is above, not at the tail of this log
diff --git a/docs/superpowers/plans/2026-10-22-self-serve-moves-4-cleanup.md b/docs/superpowers/plans/2026-10-22-self-serve-moves-4-cleanup.md| index 9285197c3..f52bf3430 100644 |
| --- a/docs/superpowers/plans/2026-10-22-self-serve-moves-4-cleanup.md |
| +++ b/docs/superpowers/plans/2026-10-22-self-serve-moves-4-cleanup.md |
| @@ -2,7 +2,9 @@ |
| > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| -Rev 7, 2026-10-24. Rev 6's review (`after236/selfserve-plan4-review6.md`: findings 1 to 6, the `null` version id and the initialisation-order nit; R1 to R4, R7 and the acknowledgement location closed; R5, R6 and R8 open through findings 2 to 5) is answered as follows. **Finding 1:** the cleaning mark on `lock.json` has no expiry. It is a state, not a timed lease: written by a conditional PUT over this move's lifted tombstone with a sentinel expiry no clock reaches, refused to every lease acquirer whatever its age, never renewed, and lifted only by the cleanup's own `releasing` once every delete of the run is answered, by supersession or cancellation only when no delete is in flight, or by an operator's abandon after the page (Global Constraint "The mark has no expiry", Tasks 3, 4a, 4c). A crash or an outage of any length lifts nothing, so no wake can take the lease while a delete the cleanup sent may still land. **Finding 2:** on a box where cleanup can run (the switch on, or the root-owned `gated` marker an armed pass left), an adopt pass that cannot take the root pass lock publishes nothing that pass; a box never armed keeps today's fail-open adoption (Task 6b's `adoptRootGate`). **Finding 3:** the drain has an admission barrier: one console-wide flag, checked inside the same critical section in which `BeginAllocation` persists its row, a placement registers and `AdmitSelfServeMove` admits; the drain sets it first, then waits on the allocation rows, the placement registrations and the open self-serve moves themselves, not on table entries (Task 7c). **Finding 4:** a new-protocol acquisition is issued only against a table body that carries `epoch` and `rev`; a body without them is first normalised by a separate conditional write and read again (`moves.ReadForAcquire`, Tasks 2a and 2b). **Finding 5:** `consoleRegistry` (`internal/console/view.go:511`) gains `SetPlacementContext`, and the registry's remote dispatcher (`internal/registry/remote_dispatch.go:270`) runs the update under the deadline it received (Task 7b). **Finding 6 and the `null` id:** condition (4) now requires the key's inventoried latest version (the one current at `gone`) to be current at every deletion; a key whose latest is gone or no longer current is `superseded` and nothing more of it is deleted. The inventory records each version's listing sequence and the key's latest id, so no tie is reversed; `ValidateRetired` refuses a `null` id by `lease.UsableVersionID`'s rule, and a listing that holds one writes no `retired.json` (Tasks 1 and 4c). **Initialisation order:** `LockRootPass` and the request directory check the existing ancestry before root creates any missing directory (Task 6b). |
| +Rev 8, 2026-10-24. Rev 7's review (`after236/selfserve-plan4-review7.md`: findings 1, 5 and 6, the `null`-id nit and the initialisation-order nit closed; findings 2, 3, 4 and 7 and the versioning qualification open) is answered as follows. **Finding 2 (first arming):** root's step runs only after an arming boundary that spans processes. In a pass holding both locks, once the `gated` marker exists, the pass takes a census of the box's root `carlos adopt` processes (`release.AdoptPasses`, read from `/proc`); only when no other one is alive does it write `<optDir>/root-cleanup/quiesced`, bound to that marker's identity, and root's step runs only while `quiesced` is bound to the current marker (Task 6b's `QuiesceRootCleanup`). Every adopt process alive after that moment started after the marker existed, so it publishes under the root pass lock or not at all; a fail-open pass that began before arming keeps root's step waiting until it has exited (`TestFirstArmingWaitsForAFailOpenPassToEnd`). The ruling's in-process pass mutex is not added: each `carlos adopt` process runs exactly one pass, and the four adopt units are separate processes that can overlap (`internal/host/agent.go:4645-4648`); one unit never overlaps itself, which is why the census is enough. **Finding 3:** recovery reloads the allocation's row by its op id under the app and hostname locks, before `admit` and before any reservation or effect; a row found terminal only finishes its owed release (Task 7a, `TestRecoveryReloadsTheAllocationUnderItsLocks`). **Finding 4:** every write to `active.json` (bootstrap, normalisation, acquisition, release, the drain's settling write) goes through `moves.PutTable`: one HTTP attempt with the SDK's retries disabled (`awss3.Options.Retryer = aws.NopRetryer{}` per call, `release.PutOnce`), resent as the identical conditional write until it is answered (success or 412). The console counts its unanswered table writes. The drain answers `drained` only when that count is zero, 20 minutes have passed since its last attempt that ended without an answer (S3 refuses a signed request more than 15 minutes old, which Task 7c's dev witness pins), and every other condition holds. The new console logs a clean exit only with no table write unanswered; a rollback whose new console did not exit cleanly after `drained` rolls forward instead (Global Constraint "Rollback", Tasks 2a and 7c, `TestDrainWaitsOnDelayedNormalizationAndAcquisition`). **Finding 7:** `ExecBackend.Acquire` maps a `cleaning:` holder to `errLeaseCleaning`, which `serveWakeError` answers with `Retry-After: 30` and its own text, never with the mark's expiry (Task 3, `TestCleaningMarkAnswersAFiniteRetry`). **Versioning:** the listing check and the rollout's `get-bucket-versioning` check are two guarantees, stated apart, with what follows if the second was skipped (Global Constraint "Arming and rollout", Task 1). |
| + |
| +Rev 7's answers that stand (its findings are review 6's): **Finding 1:** the cleaning mark on `lock.json` has no expiry. It is a state, not a timed lease: written by a conditional PUT over this move's lifted tombstone with a sentinel expiry no clock reaches, refused to every lease acquirer whatever its age, never renewed, and lifted only by the cleanup's own `releasing` once every delete of the run is answered, by supersession or cancellation only when no delete is in flight, or by an operator's abandon after the page (Global Constraint "The mark has no expiry", Tasks 3, 4a, 4c). A crash or an outage of any length lifts nothing, so no wake can take the lease while a delete the cleanup sent may still land. **Finding 2:** on a box where cleanup can run (the switch on, or the root-owned `gated` marker an armed pass left), an adopt pass that cannot take the root pass lock publishes nothing that pass; a box never armed keeps today's fail-open adoption (Task 6b's `adoptRootGate`). **Finding 3:** the drain has an admission barrier: one console-wide flag, checked inside the same critical section in which `BeginAllocation` persists its row, a placement registers and `AdmitSelfServeMove` admits; the drain sets it first, then waits on the allocation rows, the placement registrations and the open self-serve moves themselves, not on table entries (Task 7c). **Finding 4:** a new-protocol acquisition is issued only against a table body that carries `epoch` and `rev`; a body without them is first normalised by a separate conditional write and read again (`moves.ReadForAcquire`, Tasks 2a and 2b). **Finding 5:** `consoleRegistry` (`internal/console/view.go:511`) gains `SetPlacementContext`, and the registry's remote dispatcher (`internal/registry/remote_dispatch.go:270`) runs the update under the deadline it received (Task 7b). **Finding 6 and the `null` id:** condition (4) now requires the key's inventoried latest version (the one current at `gone`) to be current at every deletion; a key whose latest is gone or no longer current is `superseded` and nothing more of it is deleted. The inventory records each version's listing sequence and the key's latest id, so no tie is reversed; `ValidateRetired` refuses a `null` id by `lease.UsableVersionID`'s rule, and a listing that holds one writes no `retired.json` (Tasks 1 and 4c). **Initialisation order:** `LockRootPass` and the request directory check the existing ancestry before root creates any missing directory (Task 6b). Rev 8 keeps all of these; rev 7's answers to findings 2 and 4 are extended by rev 8's above, not replaced: the gate still publishes nothing on a gated box without the root pass lock, and `moves.ReadForAcquire` still normalises before any acquisition. |
| Rev 6's answers that stand: every data deletion names one inventoried version id, oldest first within a key, through one function (`deleteInventoriedVersion`, Task 4b), so a late, retried or outstanding delete can remove only the version it names (R1 to R4, F1, N4; a spec amendment, **Paul's question 9**); a delete that went out without an answer is sent again, and answered, before the mark is given up, because a promotion can keep one of our versions in place by its equal ETag (`lease.MirrorPrefixKeepObserved`, `internal/lease/mirror.go:285`; Task 4c); root's step is a stage of `runAdoptUnderRootRelease`, after release adoption (named recovery runs inside it) and after the held stage, under the adopt-pass lock and a root-only pass lock, with the registry host freeze and the held-stage lock as rev 5 had them (R5, F5; Task 6b); rollback is a procedure with a drain command, `carlos system reservations drain`, run on the new console before it stops (R6, N11; Task 7c); `--log` takes `--move <id>` or `--cleanup <id>`, and the console lists a host's cleanup reports across moves from the bucket, so a deleted record still answers (R7, Task 5b); placement registers before it reserves and stays registered through settlement, under one deadline (R8, Task 7b); when root rejects a directory's ancestry it writes nothing there, logs the refusal, and the agent pages hourly after an hour without an answer and never releases (Task 6b). Task 2 is split into 2a to 2c, and the delete primitive has its own task with the dev gate. |
| @@ -12,46 +14,46 @@ Rev 4's closed answers stand: one state enum and one transition table (F3); a cl |
| Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and allocation recovery (`internal/console/lifetimes_create.go`, `lifetimes_recovery.go`, `lifetimes_store.go`), `fenceRecord` (`lifetimes_create.go:551`), the judge (`provision.LifetimeJudge.Observe`), the reservation table (`moves.OfflineActive`, `ParseOfflineActive`, `OfflineActiveBody`, `OfflineSentinel`; the console's `reserveBoxes`/`releaseBoxes` and three-way reclaim in `internal/console/offlinereserve.go`), the lease's tombstone (`internal/lease/tombstone.go`), the versioned store (`lease.VersionedStore`, `internal/lease/versioned.go`), the offline source steps (`internal/host/movesoffline.go`), held publication's freeze and lock (`registry.WithHoldsFrozen`, `internal/host/movesheldlock.go`, `heldStageAuthority` in `internal/host/heldstage.go`), the page outbox (`internal/host/movespaging.go`), the offline CLI (`move.go`) and the operator's `carlos system` verbs (`system.go`, `system_lifetimes.go`). |
| -**Goal:** Twenty-four hours after a move the source box observed as finished, the source box retires the old region's replica from evidence it recorded at `gone`, under a `cleanup` reservation on its own box and on the hostname and a cleaning mark on the replica's lock that has no expiry: key by key with the conditions re-read at each key, each inventoried version deleted by its version id, oldest first, only while the version that was current at `gone` is still current, and every delete answered before the mark or the reservation is given up. Root removes the old live link only inside its adoption pass, under a lock no other root publisher can bypass (on a box where cleanup can run, a pass that cannot take that lock publishes nothing), through a request it copies, authorises from the bucket and answers in its own directory, and only where no other uid can write the link's parent. The obligation's directory goes seven days after it ended. Nothing times out into a release. A rollback to the console before this plan drains the new reservation kinds first, behind an admission barrier. |
| +**Goal:** Twenty-four hours after a move the source box observed as finished, the source box retires the old region's replica from evidence it recorded at `gone`, under a `cleanup` reservation on its own box and on the hostname and a cleaning mark on the replica's lock that has no expiry: key by key with the conditions re-read at each key, each inventoried version deleted by its version id, oldest first, only while the version that was current at `gone` is still current, and every delete answered before the mark or the reservation is given up. Root removes the old live link only inside its adoption pass, under a lock no other root publisher can bypass (on a box where cleanup can run, a pass that cannot take that lock publishes nothing), through a request it copies, authorises from the bucket and answers in its own directory, and only where no other uid can write the link's parent. The obligation's directory goes seven days after it ended. Nothing times out into a release. A rollback to the console before this plan drains the new reservation kinds first, behind an admission barrier, and starts the old console only after the new one has exited with every table write answered and no unanswered attempt still able to land. |
| **Architecture:** At `gone` the sealing source box writes `retired.json` under the move's op directory, for a tracked lifetime only: every version and delete marker (key, version id, ETag) under the normalised hibernate prefix (`lock.json` and `build-state.json` aside), each key's versions in deletion order, and the identities of the root-owned paths whose parent chain is root's alone (any other path is recorded as left). A cleanup goroutine, started by the moves pass and never under its lock, walks every obligation through one transition table: observe the finish, wait 24 hours, reserve the box's key and `host/<hostname>`, mark `lock.json`, delete key by key with the conditions re-read, hand root a request (or skip it when nothing local remains), then persist `releasing`, restore the lifted tombstone, release and end. The mark never expires: only that `releasing`, or an operator's abandon after the page, lifts it. Supersession is checked at every step and ends the obligation through the same `releasing`, after any unanswered delete is answered under the mark and any root request is cancelled and acknowledged. The lifetime create and placement hold `host/<hostname>` in the same table, and so does a move's selection. The table is never deleted. |
| -**Tech Stack:** Go. `internal/offlinecleanup` (new leaf: the wire types, file identity and the layout check), `internal/moves` (host keys, the table's read and write, the conflict relation, `Reserve`/`Release`, `RecordAway`), `internal/lease` (the cleaning mark, and its refusal in `AcquireTTL`, `Tombstone` and `LockState`), `internal/host` (the dispatcher, the delete primitive, the report, the reaper, root's authority and the agent's root half, the root pass lock's wiring and gate, the landing's refusal of the mark), `internal/release` (root's request handler, the root pass lock), `internal/registry` (`WithHostFrozen`, `SetPlacementContext`, the remote dispatcher's deadline), `internal/console` (the create, placement, the reservation wrappers, recovery mode, the sweeps, the cleanup door, the admission barrier and the drain), the root package (`move.go`, the CLI's `--log`; `system.go` and `system_reservations.go`, the drain verb). |
| +**Tech Stack:** Go. `internal/offlinecleanup` (new leaf: the wire types, file identity and the layout check), `internal/moves` (host keys, the table's read and write, `PutTable` and `TableWrites`, the conflict relation, `Reserve`/`Release`, `RecordAway`), `internal/lease` (the cleaning mark, and its refusal in `AcquireTTL`, `Tombstone` and `LockState`), `internal/activator` (the wake's answer to a mark), `internal/host` (the dispatcher, the delete primitive, the report, the reaper, root's authority and the agent's root half, the root pass lock's wiring and gate, the landing's refusal of the mark), `internal/release` (root's request handler, the root pass lock, the adopt-process census and the arming boundary, `PutOnce`), `internal/registry` (`WithHostFrozen`, `SetPlacementContext`, the remote dispatcher's deadline), `internal/console` (the create, placement, the reservation wrappers, recovery mode, the sweeps, the cleanup door, the admission barrier and the drain), the root package (`move.go`, the CLI's `--log`; `system.go` and `system_reservations.go`, the drain verb), `cmd/console` (the clean exit). |
| **Spec:** `docs/superpowers/specs/2026-10-21-self-serve-offline-moves-design.md` §7.6, §7.1 (the `cleanup` reservation), §8 (the cleanup pages), §9 (rollout), §11. Stated deviations: placement and the lifetime create hold the hostname key (Paul, 2026-10-22, amadan #343); deletion by version id (§7.6 amendment, Paul's question 9); the rest are in the Self-review notes. |
| ## Global Constraints |
| -- **Arming and rollout.** The box acquires a new cleanup reservation only with `CARLOS_OFFLINE_CLEANUP=on` in its environment (read by `armOffline` when the agent starts, from `/etc/carlos/host.env`, the `EnvironmentFile` of the agent's unit, `internal/host/units/carlos-edge.service:18`; root's adopt units read the same file, `carlos-adopt.service:16`, `carlos-stage.service:30`, `carlos-restart.service:31`, `carlos-config-sync.service:24`, and Task 6b's gate reads the switch from it), on an `OfflineCapable` box whose judge is set (`cfg.Judge != nil`, i.e. host claims off, `internal/host/boxjudge.go:21-26`); `retired.json` is written at `gone` whatever the switch says, and an obligation past `finished` runs to its end if the switch is turned off. Order: boxes, then consoles (plan 4 code everywhere), then `CARLOS_OFFLINE_CLEANUP=on` on the boxes. Before the switch goes on in any deployment, a check on dev confirms version-id deletion on the versioned hibernate bucket (Task 4b's dev witness), and `aws s3api get-bucket-versioning` answers `Enabled` for the hibernate bucket of every region the switch goes on in (tofu sets it, `infra/modules/region-remote-host/main.tf:32-34`; the box roles cannot read the status, so the code checks the listing instead, Task 1). |
| +- **Arming and rollout.** The box acquires a new cleanup reservation only with `CARLOS_OFFLINE_CLEANUP=on` in its environment (read by `armOffline` when the agent starts, from `/etc/carlos/host.env`, the `EnvironmentFile` of the agent's unit, `internal/host/units/carlos-edge.service:18`; root's adopt units read the same file, `carlos-adopt.service:16`, `carlos-stage.service:30`, `carlos-restart.service:31`, `carlos-config-sync.service:24`, and Task 6b's gate reads the switch from it), on an `OfflineCapable` box whose judge is set (`cfg.Judge != nil`, i.e. host claims off, `internal/host/boxjudge.go:21-26`); `retired.json` is written at `gone` whatever the switch says, and an obligation past `finished` runs to its end if the switch is turned off. Order: boxes, then consoles (plan 4 code everywhere), then `CARLOS_OFFLINE_CLEANUP=on` on the boxes. Before the switch goes on in any deployment, a check on dev confirms version-id deletion on the versioned hibernate bucket (Task 4b's dev witness) and S3's refusal of a signed request delivered 16 minutes late (Task 7c's dev witness), and `aws s3api get-bucket-versioning` answers `Enabled` for the hibernate bucket of every region the switch goes on in (tofu sets it, `infra/modules/region-remote-host/main.tf:32-34`). **Two guarantees, kept apart (U5).** (i) The listing check at `gone` (Task 1) establishes that every inventoried version id is usable: a listing holding a `null` id writes no `retired.json`. It does not establish that versioning is enabled: a suspended bucket whose objects all predate the suspension lists only real ids, an empty prefix lists nothing, and both pass. (ii) The rollout's one-off `get-bucket-versioning` check establishes `Enabled`. The box roles cannot read the status (`s3:GetBucketVersioning` is granted only in the inventory statement of `infra/modules/region-host/main.tf:223`, and not at all in `region-remote-host`), so no code checks it. A suspended bucket with only real ids passes (i) and is caught by (ii). **If (ii) was skipped** on a suspended bucket, deletion stays safe: every delete the cleanup sends names a real, inventoried id and removes only that version. What changes is new writes: a later owner's write under the prefix (under its lease, so never while the mark stands) creates or overwrites the key's `null` version in place, which the inventory never holds. The cleanup never deletes it, and a key whose current entry it has become is `superseded` by condition (4). |
| - **Rollback**, in this order and no other: |
| - 1. Unset `CARLOS_OFFLINE_CLEANUP` on every box and restart its agent, so no box acquires again. Wait until no `cleanup` entry is left in `control/offline-moves/active.json`: an obligation already reserved runs to its end (completion or supersession); one waiting for root ends when root answers, or when an operator cancels it by creating `<dataDir>/.offline-moves/<host>/<move>/root-cleanup.cancel` (root then acknowledges `cancelled`, Task 6b); a `finished` obligation whose reservation write was uncertain releases it while unarmed (Task 4a, row 2); an obligation whose delete has not been answered keeps its entries and the mark until the delete is answered or an operator abandons it after its page (Task 4c), and the wait includes it. Step 1 leaves Task 6b's `gated` marker on each box that was armed, so its adopt passes keep failing closed on a root pass lock fault and keep serving the requests left; an operator may remove `<optDir>/root-cleanup/gated` once no drop and no unacknowledged request is left on the box, which the rollback does not need. |
| - 2. Run `carlos system reservations drain` against the **new** console, before it stops (Task 7c). Its first call sets the console's admission barrier: one flag, checked inside the same critical section in which a create's `BeginAllocation` persists its row, a placement registers in `s.placing` and a self-serve request's `AdmitSelfServeMove` admits, so every admission either committed before the flag (and the drain's reads see it) or is refused after it; there is no check-then-admit gap. The drain then waits on the work itself, not on table entries: every allocation row `PendingAllocations` answers (pending, or terminal with its key not yet released) is driven by recovery to its release, every registered placement settles itself, every unregistered `place` entry is fenced (`fenceRecord`, through `s.settlePlacement`) **before** it is released, and every open self-serve move must end (a self-serve move's `move` entries hold its host key, which the old console's own retry reads as a mismatch, `offlinereserve.go:177-183`; an operator move's own retry tolerates it). It answers `drained` only when, after a settling write of its own and a fresh read, no `cleanup`, `create` or `place` entry, no pending allocation row, no registered placement and no open self-serve move is left. |
| - 3. Stop the new console. |
| - 4. Only then start the old console. |
| - Why this is safe: the old console reads the table that is left (only `move` entries; its `valid()` takes `move` and `cleanup`, `moves/offline.go:584-587`, and reads `create` or `place` as a malformed table). The old console's writes drop `epoch` and `rev`, and its last release deletes the table (`offlinereserve.go:478`), so a metadata-free body, and its ETag, can come back: **the epoch protects only among new writers.** The hazard is a new-protocol acquisition outstanding with such an ETag as its condition, and none can exist: every new-protocol acquisition is issued only against a body that carries `epoch` and `rev` (a body without them is first normalised by a separate conditional write and read again, `moves.ReadForAcquire`, Task 2a), and an old writer never writes a body with an `epoch`, so the ETag an outstanding acquisition names is never current again once any later write lands, old or new. Boxes no longer acquire (step 1), the drain's barrier and settling write end the console's own acquisitions (step 2), and the new console is stopped (step 3). A new-protocol release that lands late writes the table it read with its own entries removed and nothing added, so it is harmless. The old create and placement take no hostname key, which is safe once no cleanup holds or can acquire one (step 1). |
| + 1. Unset `CARLOS_OFFLINE_CLEANUP` on every box and restart its agent, so no box acquires again. Wait until no `cleanup` entry is left in `control/offline-moves/active.json`: an obligation already reserved runs to its end (completion or supersession); one waiting for root ends when root answers, or when an operator cancels it by creating `<dataDir>/.offline-moves/<host>/<move>/root-cleanup.cancel` (root then acknowledges `cancelled`, Task 6b); a `finished` obligation whose reservation write was uncertain releases it while unarmed (Task 4a, row 2); an obligation whose delete has not been answered keeps its entries and the mark until the delete is answered or an operator abandons it after its page (Task 4c), and the wait includes it. Step 1 leaves Task 6b's `gated` and `quiesced` markers on each box that was armed, so its adopt passes keep failing closed on a root pass lock fault and keep serving the requests left; an operator may remove `<optDir>/root-cleanup/gated` and `quiesced` once no drop and no unacknowledged request is left on the box, which the rollback does not need (a `quiesced` left behind is ignored, because it is bound to the identity of the `gated` it was written for). Note the time of the last agent restart of this step: a box sends a normalisation or a bootstrap only when it acquires, which it stops doing at that restart, and step 4 waits 20 minutes past it. |
| + 2. Run `carlos system reservations drain` against the **new** console, before it stops (Task 7c). Its first call sets the console's admission barrier: one flag, checked inside the same critical section in which a create's `BeginAllocation` persists its row, a placement registers in `s.placing` and a self-serve request's `AdmitSelfServeMove` admits, so every admission either committed before the flag (and the drain's reads see it) or is refused after it; there is no check-then-admit gap. The drain then waits on the work itself, not on table entries: every allocation row `PendingAllocations` answers (pending, or terminal with its key not yet released) is driven by recovery to its release, every registered placement settles itself, every unregistered `place` entry is fenced (`fenceRecord`, through `s.settlePlacement`) **before** it is released, and every open self-serve move must end (a self-serve move's `move` entries hold its host key, which the old console's own retry reads as a mismatch, `offlinereserve.go:177-183`; an operator move's own retry tolerates it). It also waits on the console's own table writes (U3): every write this console issued to `active.json` must be answered (`s.tableWrites.Unanswered() == 0`, success or 412, by its own resends), and 20 minutes (`tableAttemptWindow`) must have passed since its last attempt that ended without an answer (`s.tableWrites.LastTimedOut()`; none, no wait). It answers `drained` only when, after a settling write of its own and a fresh read, no `cleanup`, `create` or `place` entry, no pending allocation row, no registered placement, no open self-serve move and no unanswered table write is left, and the attempt window has passed. |
| + 3. Stop the new console (`systemctl stop`). On `SIGTERM` it refuses new table writes, waits up to 30 s for the ones in progress to be answered, and logs either `reservations: exit clean: no table write unanswered; last unanswered attempt <time or none>` or `reservations: exit unclean: <n> table writes unanswered` (Task 7c). **If the clean line is not in its journal after the `drained` answer (it crashed, it was killed, or a write was still unanswered), do not roll back to the old console: roll forward.** Start the new console again, run the drain again until `drained` (its first call sets the barrier again), and stop it again; repeat until it exits clean. |
| + 4. Only then, and no sooner than 20 minutes after the later of the last unanswered attempt the clean exit line names and the last agent restart of step 1, start the old console. |
| + Why this is safe (U3). The old console reads the table that is left (only `move` entries; its `valid()` takes `move` and `cleanup`, `moves/offline.go:584-587`, and reads `create` or `place` as a malformed table). Its writes drop `epoch` and `rev`, and its last release deletes the table (`offlinereserve.go:478`), so a metadata-free body, and with it its ETag, can be current again, and so can an absent table. A conditional write lands only while its condition is current, so the hazard is a new-protocol write that can still land after the old console starts: a late normalisation (conditioned on a metadata-free ETag) or bootstrap (create-only) attempt would restore the epoch-bearing body it wrote, and with it the ETag a late acquisition attempt names (review 7's finding 4). **Nothing new-protocol is outstanding once the new console has reported `drained` and exited clean:** every new-protocol write is one HTTP attempt, so no SDK retry can send a second copy later (`moves.PutTable`, Global Constraint "The table is never deleted"); each write is resent identically until it is answered, and the console counts the ones that are not; an attempt that ended without an answer may still reach S3, but S3 refuses a signed request more than 15 minutes after its signing time, so it can land only within that window, which the drain and step 4 wait out (20 minutes, with five of clock skew). The console's own writes are all answered before `drained` and again before the clean exit line (step 3); after the barrier no admission starts a new one. A box sends a normalisation or a bootstrap only when it acquires, which ends at its step-1 restart, 20 minutes before step 4; a box's later releases are conditioned on epoch-bearing ETags, which only a late normalisation or bootstrap could make current again. So no new-protocol attempt can land once the old console runs, and the epoch protocol's own guarantee (no body twice, so no ETag twice) covers every write before it. A console that did not exit clean may have died with an attempt in flight that no count recorded; rolling forward keeps the new protocol the only writer until a clean exit settles it. The old create and placement take no hostname key, which is safe once no cleanup holds or can acquire one (step 1). |
| - **Tracked lifetimes only.** A cleanup obligation exists only for a tracked lifetime: the judged observation at `gone` is `LifetimeValid` with `Rec.Incarnation > 0` and a live slot at that incarnation. A record with incarnation 0 and no slot (an untracked lifetime, before the lazy migration; `provision.LifetimeOf`, `internal/provision/lifetime.go:127-130`, still answers it `LifetimeValid`), a migrating record, or any other verdict gets no `retired.json`, and its gone report proceeds exactly as before this plan; nothing is ever deleted for it, and its leftovers stay the runbook's. |
| - **Every data deletion names one inventoried version (spec §7.6 amendment, Paul's question 9).** `retired.json`'s `BucketInventory` records, for every key under the normalised prefix except `lock.json` and `build-state.json`, **every** version id the bucket lists at `gone`, delete markers included, not only the current one, with each entry's position in the listing (`Seq`) and the key's latest version id (`Latest`: the entry the listing marks `IsLatest`, the one current at `gone`). Every id is a real one: a `null` id (an object written while the bucket was unversioned or versioning was suspended, which a later suspended write overwrites in place) fails `lease.UsableVersionID`'s rule (`internal/lease/staging.go:414`), and a listing that holds one writes no `retired.json` (Task 1). The cleanup deletes only through `deleteInventoriedVersion` (Task 4b), which is `VersionedStore.DeleteVersion(ctx, bucket, key, versionID)` (`internal/lease/versioned.go:261-275`: an empty id is `ErrNoVersionID` without a request; success and `NoSuchVersion` are both nil) on `cfg.Versioned(ctx, Bucket, BucketRegion)`. A plain `Delete`, a marker-writing delete and an `If-Match` delete are never used. Within a key, the versions that were not current are deleted first, oldest first (`LastModified` ascending, ties by `Seq` descending: an order fixed at `gone` and recorded), and `Latest` last. **Condition (4) is the latest's:** before each deletion of a key's version, a fresh listing of the key must show `Latest` as its current entry; when `Latest` is absent, or another entry is current (a later version, or a marker the inventory does not hold), the key is `superseded` and nothing more of it is deleted. A version is **done** when a delete of it answers (success or no such version); a version other than `Latest` that the listing no longer shows is done (removed by someone with authority, or by a lifecycle rule for noncurrent versions). **The argument, once:** a delete that names a version id can remove only that version, whoever sends it and whenever it lands, so a late attempt, an SDK retry (`internal/lease/endpoint.go:103` keeps the default retries) or a crash-repeated request can never remove a version a later owner wrote, which carries another id. While `Latest` is current, deleting any other version changes nothing current, so a deletion sent after a check that found `Latest` current exposes no older version unless `Latest` stops being current between that check and the delete's landing. Nothing the platform runs can do that: a write needs the lease, which the mark refuses; a plain delete writes a marker on top, which is then current and hides everything beneath it; a lifecycle rule for noncurrent versions never removes a current one. Only a version-id delete of `Latest` by hand, concurrent with the cleanup, can, and the plan claims nothing about that case. **The one case a version id does not settle:** a later owner that keeps one of our versions instead of writing its own. A promotion does exactly this when the destination's ETag equals the staged one (`lease.MirrorPrefixKeepObserved`, `mirror.go:285`), and the target landing of a move into this region mirrors the same way. Both need the lease, or a lock the landing reads as free, and the mark refuses both for as long as it stands; it stands until every delete is answered. A delete is recorded (`InFlight`) before it is sent and sent again until it answers, **before** the obligation leaves `cleaning` (Task 4c), whatever has happened to the reservation meanwhile; an `InFlight` delete that cannot be answered keeps the mark and the entries the obligation still holds, and pages every 10 minutes, until it is answered or an operator abandons it after the page (Task 4c). Once a version is done, any attempt still in the network lands on nothing, because a version id is never reused. **The trade:** the retired replica is not recoverable from the bucket after the cleanup. It is the source copy of a finished move, and litestream's retention would purge it anyway. In exchange, the boundary is enforced by the server. `lock.json` and `build-state.json` are never inventoried and never deleted. |
| -- **The table is never deleted.** `control/offline-moves/active.json` is created once, by the first writer, as `{v: 1, epoch: <16 random bytes, hex>, rev: 1}` with `If-None-Match: *` (`release.Store.Put` with `ifMatch ""`, `internal/release/store.go:794-798`); emptiness is a table with no entries. Every later write is a `Put` with `If-Match` the ETag it read, of a body whose `rev` is the read body's plus one and whose `epoch` is the read body's (a fresh one when the read body has none, as an older console's writes do), so no two bodies the table ever holds are equal and an ETag never comes back. A conditional `Put` that read ETag E can land only while E is current, so once any later write succeeds, every earlier outstanding `Put` that read E can never land: an uncertain acquisition is settled by the next successful conditional write by anyone. A release always writes (rev + 1), even when its operation holds nothing, so its own success is the settlement its caller needs. **Acquisitions only against an epoch-bearing body (T4).** A write that adds entries (`moves.Reserve`, and so the box, the create and the placement; the console's `reserveBoxes`) reads through `moves.ReadForAcquire`: a body without `epoch` or `rev` (an older console's) is first normalised by a separate `Put` with `If-Match` the read ETag, of the same entries with a fresh `epoch` and `rev` 1, then read again, and only a body that carries both is ever an acquisition's condition. An old writer never writes a body with an `epoch`, so the ETag an acquisition names is never current again after any later write, old or new. Among new writers every write has this property; against old writers acquisitions have it through the normalisation, and the rollback procedure covers the rest. |
| +- **The table is never deleted.** `control/offline-moves/active.json` is created once, by the first writer, as `{v: 1, epoch: <16 random bytes, hex>, rev: 1}` with `If-None-Match: *` (`release.Store.Put` with `ifMatch ""`, `internal/release/store.go:794-798`); emptiness is a table with no entries. Every later write is a `Put` with `If-Match` the ETag it read, of a body whose `rev` is the read body's plus one and whose `epoch` is the read body's (a fresh one when the read body has none, as an older console's writes do), so no two bodies the table ever holds are equal and an ETag never comes back. A conditional `Put` that read ETag E can land only while E is current, so once any later write succeeds, every earlier outstanding `Put` that read E can never land: an uncertain acquisition is settled by the next successful conditional write by anyone. A release always writes (rev + 1), even when its operation holds nothing, so its own success is the settlement its caller needs. **Acquisitions only against an epoch-bearing body (T4).** A write that adds entries (`moves.Reserve`, and so the box, the create and the placement; the console's `reserveBoxes`) reads through `moves.ReadForAcquire`: a body without `epoch` or `rev` (an older console's) is first normalised by a separate `Put` with `If-Match` the read ETag, of the same entries with a fresh `epoch` and `rev` 1, then read again, and only a body that carries both is ever an acquisition's condition. Among new writers no body, and so no ETag, is ever current twice. Against an old writer that is not enough on its own, because a late new-protocol normalisation or bootstrap attempt can restore an epoch-bearing body after an old writer made its condition current again (review 7's finding 4), so: **single attempts (U3).** Every write to the table goes through `moves.PutTable(ctx, s, tw, body, ifMatch)` (Task 2a), and nothing else writes the key (`TestEveryTableWriteGoesThroughPutTable`). It sends one HTTP request through `release.PutOnce`: `S3Store.PutOnce` passes `func(o *awss3.Options) { o.Retryer = aws.NopRetryer{} }` to `PutObject`, so the SDK's default retryer, which `S3Store.Put` keeps (`internal/release/store.go:787`, nothing in `internal/s3compat` or `rootaws` sets a retry option), never sends a second copy. An answer (success, or `release.ErrCASConflict`: S3's 412, or its 409 and 404 as `Put` maps them) is returned. A timeout, a transport error or a 5xx is no answer: the identical request (the same bytes, the same condition) is sent again after 1 s, 2 s, 4 s and then every 10 s, until it is answered or `ctx` ends. A 412 after an unanswered attempt is an answer: our attempt landed or another write did, and the next read tells which. A write not yet answered is registered in `tw` (the console's `s.tableWrites`, the box worker's `w.tableWrites`) with its bytes, its condition and the time of its last unanswered attempt, and the next sweep (console) or run (box) resends it before anything else; the drain reads `tw.Unanswered()` and `tw.LastTimedOut()`. A store without `PutOnce` (the store door's client, `internal/fleetapi/fleetapi.go:196`, whose door writes through a retrying client) gets `release.ErrNoSingleAttemptPut` and nothing is sent, so a box on the store door never reserves; the platform's boxes and consoles use `S3Store` or `DirStore` (`internal/host/agent.go:4419-4426`, `cmd/console/main.go:672-684`). |
| - **The prefix (N1).** `Prefix = activator.HibernatePrefix(cfg.OwnPrefix, host) + "/"` (`HibernatePrefix` is `path.Join`, no trailing slash, `internal/activator/activator.go:147`), computed once at `gone` and persisted; the listing is `ListVersions(bucket, Prefix)`; the excluded keys are exactly `Prefix + "lock.json"` and `Prefix + "build-state.json"`; the lease location is `strings.TrimSuffix(Prefix, "/")`. |
| - **The cleanup id** is `offlinecleanup.CleanupID(box, host, moveID) = "cu_" + hex(sha256(box + "\n" + host + "\n" + moveID))[:16]`: recomputable from the directory and the box, so a corrupt file, root and the agent all name the same obligation. |
| -- **The mark has no expiry (T1).** Owner `lease.CleaningOwner(id) = "cleaning:" + id`, written with `ExpiresAt = lease.CleaningExpiry` (`9999-12-31T00:00:00Z`, a time no clock reaches: `litestream.Lease` needs an expiry and `readLock` refuses a zero one, `internal/lease/tombstone.go:140`). It is a state, not a timed lease: written only by a conditional PUT from the lifted tombstone of this move (`moves.TombstoneOwner(MoveID)`, expired); never renewed (there is no reassert); and refused to every lease acquirer whatever its expiry says. litestream's own `AcquireLease` refuses it on the read that conditions its write (`litestream@v0.5.14/s3/leaser.go:91-101`, then the `If-Match` write at `:114`), and `lease.AcquireTTL` (so `Acquire`, `ReplicateWithLease` and the staging lease), `lease.Tombstone` and `lease.LockState` refuse, or report held, any `cleaning:` owner by its prefix, so even a mark whose expiry was edited by hand is never free (Task 3). It is lifted only (a) by the cleanup's own `releasing`, after every delete of the run is answered; (b) by supersession or a cancellation, and only when no delete is `InFlight`; or (c) by an operator's abandon (`cleanup.abandon`, Task 4c) after the page that an outstanding delete is unsettled. A crash, an outage of any length, a disarm or a lost reservation lifts nothing. **So no wake, landing, return or seal of any owner can take the lease while a delete the cleanup sent may still land**, whatever happens to the cleanup's process; that is what makes an in-place adoption of one of our versions (above) impossible during the cleanup. The mark is removed by a conditional PUT back to exactly that lifted tombstone (`{Generation: as read, ExpiresAt: 1970-01-01T00:00:00Z, Owner: moved:<move id>}`, what `LiftTombstone` writes), so the operator return's leftover set (spec §7.5) is unchanged after any cleanup. The owner alone proves no lease was taken since the mark, because every acquisition rewrites the object. |
| +- **The mark has no expiry (T1).** Owner `lease.CleaningOwner(id) = "cleaning:" + id`, written with `ExpiresAt = lease.CleaningExpiry` (`9999-12-31T00:00:00Z`, a time no clock reaches: `litestream.Lease` needs an expiry and `readLock` refuses a zero one, `internal/lease/tombstone.go:140`). It is a state, not a timed lease: written only by a conditional PUT from the lifted tombstone of this move (`moves.TombstoneOwner(MoveID)`, expired); never renewed (there is no reassert); and refused to every lease acquirer whatever its expiry says. litestream's own `AcquireLease` refuses it on the read that conditions its write (`litestream@v0.5.14/s3/leaser.go:91-101`, then the `If-Match` write at `:114`), and `lease.AcquireTTL` (so `Acquire`, `ReplicateWithLease` and the staging lease), `lease.Tombstone` and `lease.LockState` refuse, or report held, any `cleaning:` owner by its prefix, so even a mark whose expiry was edited by hand is never free (Task 3). It is lifted only (a) by the cleanup's own `releasing`, after every delete of the run is answered; (b) by supersession or a cancellation, and only when no delete is `InFlight`; or (c) by an operator's abandon (`cleanup.abandon`, Task 4c) after the page that an outstanding delete is unsettled. A crash, an outage of any length, a disarm or a lost reservation lifts nothing. **So no wake, landing, return or seal of any owner can take the lease while a delete the cleanup sent may still land**, whatever happens to the cleanup's process; that is what makes an in-place adoption of one of our versions (above) impossible during the cleanup. The mark is removed by a conditional PUT back to exactly that lifted tombstone (`{Generation: as read, ExpiresAt: 1970-01-01T00:00:00Z, Owner: moved:<move id>}`, what `LiftTombstone` writes), so the operator return's leftover set (spec §7.5) is unchanged after any cleanup. The owner alone proves no lease was taken since the mark, because every acquisition rewrites the object. **Its consumers get a finite retry (U4):** a wake refused by a mark answers `Retry-After: 30` and its own text, never the distance to the sentinel (Task 3). |
| - **The writers the mark excludes** (each writer's check, by file): the wake (`internal/activator/backend_exec.go:701`, `lease.Acquire` in `ExecBackend.Acquire`, step 1 of `doWake` in `activator.go`, before the restore, the staged-replica promotion `promoteStagedReplica` at `backend_exec.go:3397`, `BeforeExec`'s build-state write and litestream replication, all of which run under that lease); the legacy final checkpoint (`internal/host/movespass.go:321-328`, `lease.ReplicateWithLease`, which acquires); the offline seal's checkpoint (`internal/host/movesoffline_seal.go:417`, the same `Checkpoint`) and its tombstone (`movesoffline_seal.go:431`, `lease.Tombstone`, which never replaces a live lease of another owner, so the seal's carry at `:469` is not reached). Not excluded by the mark and harmless: the legacy cut's request carry (`movespass.go:1594`) and the offline seal's carry write only `build-state.json`, which is not inventoried; the seed rollback and finish (`seedwiring.go:247`, `console/seedfinish.go:254`) only delete, by key (a marker, which condition (4) reads as another current entry), and the finish refuses a held lock (`seedfinish.go:255-259`, which `LockState` answers held for a mark); the target landing (`movesoffline_land.go:611`, `:1373`) belongs to a move into this region, which holds `host/<hostname>`, and its lease step waits on a `cleaning:` owner whatever its expiry (`movesoffline_land.go:612-630`, Task 3); the operator return's start door refuses any lock that is not this move's lifted tombstone (`console/offlinereturn.go:465`). `TestHibernatePrefixSitesAreInventoried` (Task 3) pins this list. |
| - **Reservation keys and the one conflict relation.** Keys are a box id (today's bare label, unchanged), the sentinel `"*"`, or `moves.HostKey(host) = "host/" + host`. Kinds: `move`, `cleanup`, `create`, `place`. An entry is valid with a known kind, a non-empty id and host, a non-zero `reserved_at`, and, for `create` and `place`, a non-empty account and app; an entry under a host key names that host (`Host == strings.TrimPrefix(key, "host/")`). An existing entry of another `(Kind, ID)` conflicts with a requested set of keys when (a) it holds a requested key; or (b) one side holds `"*"` and the other holds a box key; or (c) the requester is the console with `cfg.SelfServeMoves != "offline"` and both are of kind `move` (plan 2's one-move-at-a-time rule, now between moves only). Host keys never meet (b) or (c). Every move selection (self-serve and operator) holds `host/<hostname>` beside its box keys or its sentinel; a cleanup holds `[cfg.BoxID, host/<hostname>]` with one entry; a create holds `[host/<hostname>]`; a placement holds `host/<h>` for every instance of the app, one entry per key with `Host = h` and the placement's account and app. |
| - **A cleanup entry is never reclaimed by another reserver.** Its own box resumes it after any crash (its own retry is accepted); a box that never returns leaves an entry the console pages after 6 hours (operator repair). Plan 2's `Server.cleanupReports` and `cleanupReportEvidence` (`offlinereserve.go:95-104`) are deleted. |
| - **The conditions at every key**, (1) to (3) re-read before a key's first delete and (4) before each of its deletes: (1) `moves.RecordAway(obs, MoveID, Incarnation, From) == AwayYes`: the judged observation is `LifetimeValid` (the judge's verdict that the slot vouches for the record) at `Incarnation`, with no move, `LastMove.ID == MoveID && LastMove.Outcome == "done"`, and `Region != From`; (2) `lock.json`'s owner is `CleaningOwner(ID)` (its expiry is the sentinel; nothing about time is read); (3) the box's registry has no row for the host (`w.offlineRow(host)`, `movesoffline.go:996`); (4) `ListVersions(bucket, key)` filtered to `Key == key` shows the key's inventoried `Latest` as its current entry. A read error in any: the obligation stops for this run. (1), (2) or (3) failing definitely: **supersession** (spec §7.6's "superseded as a whole"; never an indefinite refusal). (4) failing (`Latest` absent, or a later version or a marker the inventory does not hold current): that key `superseded`, nothing more of it sent, and a page. Conditions (1) and (3) are also checked at every other step of the lifecycle (Task 4a's `supersededBy`). |
| - The cleanup writes nothing to the registry, the edges file or the offline view, so plan 3's holding entries (built from held rows and the view) are untouched; root's freeze holds the registry's write lock for an `fstatat` and an `unlinkat`, as held publication's does for its swap. |
| -- **Root's contract.** Root removes a path only (i) when the path's parent directory and every ancestor up to `/` is a directory owned by uid 0 with no group or other write bit (`offlinecleanup.RootOnlyChain`; otherwise the path is `kept`, reason "writable layout", because no exclusion against a writer that shares the agent's uid is possible there by construction); (ii) **as a stage of the root adoption pass** (`runAdoptUnderRootRelease`), after release adoption and named recovery and after the held stage, while the pass holds the adopt-pass lock and the root pass lock (`release.LockRootPass`, `<optDir>/root-cleanup/pass.lock` on a root-only chain, which every adopt pass takes in every mode before its first publisher; on a box where cleanup can run, Task 6b's gate, a pass that cannot take it publishes nothing that pass, so there every root publisher of the live link runs under it), and then inside `reg.WithHostFrozen(host, …)` (no row for the host) and the host's held-stage lock (`withHeldStageLock`, `movesheldlock.go:68`), in the order adopt pass lock > root pass lock > the registry's write lock > the held-stage lock (`movesheldlock.go:26-31` gains the root pass lock); so no other root publisher (adoption, named recovery, `PointLive`) runs on the box while root's step runs, and the agent's publishers are excluded by the freeze and the held-stage lock; (iii) after the authority callback has answered yes for this request in this pass and `root-cleanup.cancel` is absent; (iv) when `fstatat(parent, name, AT_SYMLINK_NOFOLLOW)` on the opened parent shows the recorded identity (device, inode, file type, link target); then `unlinkat(parent, name)`. An entry with another identity is `kept`, never removed. Every step is journalled as intent and then result in root's own directory, which is used only when it passes the same root-only check; when it does not, root writes nothing there (Task 6b). Nothing inside the held-stage lock reads the network or takes a lock. |
| -- Timings: 24 h after `FinishedAt`; per run 60 s (`cleanupRunDeadline`), 10 s per operation, at most 50 keys per obligation; pages every 10 min while root has not answered or has not acknowledged a cancel, and while a delete has waited over 10 min for its answer (`InFlight`), hourly once root has not answered for an hour and its request directory's ancestry is not root-only (`rootRefusedAfter`), hourly for a corrupt obligation and for a record unreadable for over an hour; the console pages a `cleanup` entry older than 6 h and a `create` entry with no allocation row older than `offlineLockGrace` (2 min); a placement runs under one 15 s deadline (`instanceStoreTimeout`), and an unregistered `place` entry older than `offlineLockGrace` is settled by the sweep; the op directory goes 7 d after `EndedAt`; root removes its own directory 30 d after its acknowledgement. |
| -- Census: `internal/provision/testdata/plan14-root-reads-self-serve-moves-4.tsv`, loaded by a new `loadRootReadInventory(root+"/internal/provision/testdata/plan14-root-reads-self-serve-moves-4.tsv", &inv, true)` block in `internal/provision/plan14_root_reads_test.go` after plan 3's block (`:1383`) and its seal check and before `compareRootReads`, `rev3=PROPOSED-self-serve-moves-4`, rows at the `authority` class with evidence `E2,J3,J4` for the destructive readers (the box's record, registry, lock and versions reads and the abandon file; root's drop, its own request copy and journal, the gate marker, `active.json`, record, lock, registry-under-freeze, the layout `lstat` chain, the root pass lock's chain and the `fstatat` of each path); `TestPlan9B8RecordWriteCensus` (`plan9_b8_census_test.go:372`) takes `plan14SelfServeMoves4WriteCensusInventory()` wrapping `plan14SelfServeMoves3WriteCensusInventory()` (`self_serve_moves3_census_test.go:22`) with every new or changed bucket writer as an added or re-pinned unit (`moves.ReadActive`'s bootstrap, `moves.ReadForAcquire`'s normalisation write, `moves.Reserve`, `moves.Release`, `reserveBoxes`, `releaseBoxes`, the drain's settling `Release`, the cleanup report writer, the cleanup's `DeleteVersion` through `deleteInventoriedVersion`; the placement settle, the place sweep and the drain reach `fenceRecord`'s existing record unit); both green. |
| -- Copy rows (pending Paul's review; each task that emits one appends its row, at the next free `MV-O` id, status pending, to `docs/superpowers/plans/2026-10-06-moves-phase1-copy.md` and its id to `movesCopyIDs` in `approved_moves_copy_test.go`, in the same commit): "the previous instance at this hostname is still being cleaned up. try again later"; "an instance of this app is still being cleaned up. try again later"; "an instance of this app is moving. try again later"; "could not reserve the app's instances. try again"; "this console is being rolled back. try again in a few minutes"; page subjects "cleanup of <host> waits for root", "cleanup of <host> skipped <key>: another version is current", "cleanup of <host> waits for root to acknowledge its cancellation", "cleanup of <host> cannot read its record", "a cleanup has held <host> for over six hours", "cleanup of <host>: root refused the request directory's ancestry", "cleanup of <host> lost its mark before a delete of <key> was answered", "cleanup of <host>: an outstanding delete is unsettled; the host's replica stays marked", each with the body "run carlos move offline status --host <host> --log --cleanup <cleanup id> to see what it waits for" (the console's "held for over six hours" page names the entry's id the same way); the CLI's lines of Tasks 5b and 7c; the `--log` paragraph and the `reservations drain` line of `docs/site/cli.md`. No tenant names in tests (`shop.example.com`, account `bab`, app `shop`). |
| +- **Root's contract.** Root removes a path only (i) when the path's parent directory and every ancestor up to `/` is a directory owned by uid 0 with no group or other write bit (`offlinecleanup.RootOnlyChain`; otherwise the path is `kept`, reason "writable layout", because no exclusion against a writer that shares the agent's uid is possible there by construction); (ii) **as a stage of the root adoption pass** (`runAdoptUnderRootRelease`), after release adoption and named recovery and after the held stage, while the pass holds the adopt-pass lock and the root pass lock (`release.LockRootPass`, `<optDir>/root-cleanup/pass.lock` on a root-only chain, which every adopt pass takes in every mode before its first publisher; on a box where cleanup can run, Task 6b's gate, a pass that cannot take it publishes nothing that pass, so there every root publisher of the live link runs under it; and root's step runs only once `quiesced` records a moment, after the box was gated, at which no other root adopt process was alive, so no pass admitted before arming is still publishing, Task 6b's arming boundary), and then inside `reg.WithHostFrozen(host, …)` (no row for the host) and the host's held-stage lock (`withHeldStageLock`, `movesheldlock.go:68`), in the order adopt pass lock > root pass lock > the registry's write lock > the held-stage lock (`movesheldlock.go:26-31` gains the root pass lock); so no other root publisher (adoption, named recovery, `PointLive`) runs on the box while root's step runs, and the agent's publishers are excluded by the freeze and the held-stage lock; (iii) after the authority callback has answered yes for this request in this pass and `root-cleanup.cancel` is absent; (iv) when `fstatat(parent, name, AT_SYMLINK_NOFOLLOW)` on the opened parent shows the recorded identity (device, inode, file type, link target); then `unlinkat(parent, name)`. An entry with another identity is `kept`, never removed. Every step is journalled as intent and then result in root's own directory, which is used only when it passes the same root-only check; when it does not, root writes nothing there (Task 6b). Nothing inside the held-stage lock reads the network or takes a lock. |
| +- Timings: 24 h after `FinishedAt`; per run 60 s (`cleanupRunDeadline`), 10 s per operation, at most 50 keys per obligation; pages every 10 min while root has not answered or has not acknowledged a cancel, and while a delete has waited over 10 min for its answer (`InFlight`), hourly once root has not answered for an hour and its request directory's ancestry is not root-only (`rootRefusedAfter`), hourly for a corrupt obligation and for a record unreadable for over an hour; the console pages a `cleanup` entry older than 6 h and a `create` entry with no allocation row older than `offlineLockGrace` (2 min); a placement runs under one 15 s deadline (`instanceStoreTimeout`), and an unregistered `place` entry older than `offlineLockGrace` is settled by the sweep; the op directory goes 7 d after `EndedAt`; root removes its own directory 30 d after its acknowledgement; a wake refused by a cleaning mark answers `Retry-After: 30` (`cleaningRetryAfter`); an unanswered table write is resent after 1, 2 and 4 s and then every 10 s, and the drain and the rollback wait 20 min (`tableAttemptWindow`) after the last unanswered attempt; the console's exit waits up to 30 s for its table writes. |
| +- Census: `internal/provision/testdata/plan14-root-reads-self-serve-moves-4.tsv`, loaded by a new `loadRootReadInventory(root+"/internal/provision/testdata/plan14-root-reads-self-serve-moves-4.tsv", &inv, true)` block in `internal/provision/plan14_root_reads_test.go` after plan 3's block (`:1383`) and its seal check and before `compareRootReads`, `rev3=PROPOSED-self-serve-moves-4`, rows at the `authority` class with evidence `E2,J3,J4` for the destructive readers (the box's record, registry, lock and versions reads and the abandon file; root's drop, its own request copy and journal, the gate marker, `active.json`, record, lock, registry-under-freeze, the layout `lstat` chain, the root pass lock's chain and the `fstatat` of each path); `TestPlan9B8RecordWriteCensus` (`plan9_b8_census_test.go:372`) takes `plan14SelfServeMoves4WriteCensusInventory()` wrapping `plan14SelfServeMoves3WriteCensusInventory()` (`self_serve_moves3_census_test.go:22`) with every new or changed bucket writer as an added or re-pinned unit (`moves.PutTable`, through which every one of the following writes, and the resends of `moves.TableWrites`; `moves.ReadActive`'s bootstrap, `moves.ReadForAcquire`'s normalisation write, `moves.Reserve`, `moves.Release`, `reserveBoxes`, `releaseBoxes`, the drain's settling `Release`, the cleanup report writer, the cleanup's `DeleteVersion` through `deleteInventoriedVersion`; the placement settle, the place sweep and the drain reach `fenceRecord`'s existing record unit); both green. |
| +- Copy rows (pending Paul's review; each task that emits one appends its row, at the next free `MV-O` id, status pending, to `docs/superpowers/plans/2026-10-06-moves-phase1-copy.md` and its id to `movesCopyIDs` in `approved_moves_copy_test.go`, in the same commit): "the previous instance at this hostname is still being cleaned up. try again later"; "an instance of this app is still being cleaned up. try again later"; "an instance of this app is moving. try again later"; "could not reserve the app's instances. try again"; "this console is being rolled back. try again in a few minutes"; the wake's 503 body "instance is being cleaned up after a move. retry shortly" (Task 3; it begins with `instance`, which the scheduler's `deliverVia` reads as a wake refusal, `internal/host/schedule_wiring.go:514`); page subjects "cleanup of <host> waits for root", "cleanup of <host> skipped <key>: another version is current", "cleanup of <host> waits for root to acknowledge its cancellation", "cleanup of <host> cannot read its record", "a cleanup has held <host> for over six hours", "cleanup of <host>: root refused the request directory's ancestry", "cleanup of <host> lost its mark before a delete of <key> was answered", "cleanup of <host>: an outstanding delete is unsettled; the host's replica stays marked", each with the body "run carlos move offline status --host <host> --log --cleanup <cleanup id> to see what it waits for" (the console's "held for over six hours" page names the entry's id the same way); the CLI's lines of Tasks 5b and 7c; the `--log` paragraph and the `reservations drain` line of `docs/site/cli.md`. No tenant names in tests (`shop.example.com`, account `bab`, app `shop`). |
| - Local tests: touched packages with `-run`; CI runs the full suite and `go test -tags e2e ./internal/e2e/...`. |
| ## Review Focus |
| -1. The F1 and N4 interleavings by version id: a reconcile paused with an old observation mints the source row after the cleanup marked the lock, and a wake through `ExecBackend.Acquire` then tries the lease: it gets `HeldError` naming `cleaning:<id>`, no version is written under the prefix, the next key's condition (3) supersedes, the lifted tombstone is restored, the reservation released, and the wake then succeeds. A delete sent before the supersession and still unanswered is sent again under the mark and answered before `releasing` is written; an earlier attempt of it delivered after the release lands on nothing. Two attempts of one delete (an SDK retry) remove one version. The mark has no expiry: a cleanup that stops with a delete in flight and stays down for 25 hours still holds it, a legacy return's wake is refused with `HeldError` naming `cleaning:<id>`, and the resumed cleanup resends the same version id, gets its answer and only then releases; a delete that never answers keeps the mark and pages every 10 minutes until an operator abandons it. No deletion is sent unless a fresh listing found the key's inventoried latest current, so no run leaves a version older than it current; a key whose latest was removed by hand is superseded, not deleted further. A promotion back into the region that kept an inventoried version by its equal ETag finds it intact after the cleanup ended (Tasks 3, 4b and 4c, `TestCleaningMarkIsRefusedWhateverItsAge`, `TestMarkOutlivesACrashWithADeleteInFlight`, `TestUnanswerableDeleteHoldsTheMarkAndPages`, `TestAdmittedReconcileAndWakeCannotWriteUnderTheMark`, `TestUnansweredDeleteIsAnsweredBeforeTheUnmark`, `TestRetriedAttemptsRemoveOnlyTheirVersion`, `TestDeletionNeverExposesAnOlderVersion`, `TestLatestRemovedByHandSupersedesTheKey`, `TestPromotionThatKeptAnInventoriedVersionKeepsIt`). |
| +1. The F1 and N4 interleavings by version id: a reconcile paused with an old observation mints the source row after the cleanup marked the lock, and a wake through `ExecBackend.Acquire` then tries the lease: it gets `HeldError` naming `cleaning:<id>`, no version is written under the prefix, the next key's condition (3) supersedes, the lifted tombstone is restored, the reservation released, and the wake then succeeds. A delete sent before the supersession and still unanswered is sent again under the mark and answered before `releasing` is written; an earlier attempt of it delivered after the release lands on nothing. Two attempts of one delete (an SDK retry) remove one version. The mark has no expiry: a cleanup that stops with a delete in flight and stays down for 25 hours still holds it, a legacy return's wake is refused with `HeldError` naming `cleaning:<id>`, and the resumed cleanup resends the same version id, gets its answer and only then releases; a delete that never answers keeps the mark and pages every 10 minutes until an operator abandons it. No deletion is sent unless a fresh listing found the key's inventoried latest current, so no run leaves a version older than it current; a key whose latest was removed by hand is superseded, not deleted further. A promotion back into the region that kept an inventoried version by its equal ETag finds it intact after the cleanup ended. A wake refused by the mark answers `Retry-After: 30`, so a client polling a cleanup that ends seconds later is back within half a minute, not 292 years (Tasks 3, 4b and 4c, `TestCleaningMarkAnswersAFiniteRetry`, `TestCleaningMarkIsRefusedWhateverItsAge`, `TestMarkOutlivesACrashWithADeleteInFlight`, `TestUnanswerableDeleteHoldsTheMarkAndPages`, `TestAdmittedReconcileAndWakeCannotWriteUnderTheMark`, `TestUnansweredDeleteIsAnsweredBeforeTheUnmark`, `TestRetriedAttemptsRemoveOnlyTheirVersion`, `TestDeletionNeverExposesAnOlderVersion`, `TestLatestRemovedByHandSupersedesTheKey`, `TestPromotionThatKeptAnInventoriedVersionKeepsIt`). |
| 2. A crash after every write and every side effect of the lifecycle (after `Reserve` before `cleaning`, after `cleaning` before the mark, after the mark before `MarkedAt`, after `InFlight` is written before its delete, after a delete before `Done` moves, after the last key before `publishing`, after `releasing` before the unmark, after the unmark before the release, after the release before `ended`), restart, completion: no first delete of a version is sent unless both entries and the mark are found to be this cleanup's (a resend needs only the mark), an entry an operator removed or a mark replaced turns the obligation `superseded`, an unanswered delete is answered under the mark first (only a mark replaced by hand leaves it `unsettled`), nothing is re-acquired from `cleaning`, and the outcome is the table's (Task 4a, `TestLifecycleResumesAtEveryBoundary`). |
| -3. Root under attack and crash: a shared-uid writer plants an acknowledgement, a request with another id, a changed copy of the request and a cancel in the agent's directory, and replaces `/data/adopt-pass.lock` with a symlink so a second pass runs without that lock; on an armed box a second pass's root pass lock fails for a root-level fault; a publication replaces the live link between root's reference check and its `fstatat`; a second adopt pass starts between root's `fstatat` and its `unlinkat`; a route is upserted during the freeze; the link's parent chain is writable; root's request directory's ancestry is writable; root crashes after each journal line and each syscall: root removes only an entry carrying the recorded identity, the replacement survives, the second pass waits on the root pass lock until the step returns, the pass whose root pass lock failed publishes nothing, root creates no directory under an ancestry that has not passed, the upsert waits for the freeze, the writable layout is never touched, a rejected ancestry gets no write, every intent resolves to removed, kept or absent, root finds its own taken request on restart, and the forged files change nothing but stopping (Tasks 6b and 6c, `TestRootRemovesOnlyTheRecordedIdentity`, `TestNoPublisherRunsBetweenFstatatAndUnlinkat`, `TestGatedPassPublishesNothingWithoutTheRootPassLock`, `TestLockRootPassChecksAncestryBeforeCreating`, `TestWritableLayoutIsNeverRemoved`, `TestRejectedAncestryGetsNoWrite`, `TestRootJournalResolvesEveryIntent`, `TestRootFindsItsOwnRequestsOnRestart`, `TestForgedAgentFilesAreHarmless`). |
| -4. Residence changes and cleanup exclude each other through `host/<hostname>`: a create (busy → 409, row void, no reservation written), a crash after `BeginAllocation` and after `Reserve`, a terminal row whose release failed, a create's reservation PUT that lands after its recovery voided and released it (refused by its condition); a placement that is busy (nothing written), whose record PUT commits after the client gave up (fenced before release), that pauses after its reservation for longer than the grace (registered, so never swept; its local writes fail at its deadline) and whose console crashed (the sweep fences and releases); an offline move selection; with the switch off two creates of different hosts and a cleanup beside an operator move all proceed, and the sentinel never blocks a create; the drain settles every `place` and `create` entry and refuses to finish while a self-serve move or a cleanup holds an entry; a create paused between `BeginAllocation` and its reservation and a placement registered and paused before its reservation keep the drain `draining`; an admission and the drain's flag are one critical section; no acquisition ever names a pre-epoch ETag, so an old writer that recreates one cannot revive an outstanding acquisition; a placement paused past its deadline makes no registry update, local or remote (Tasks 2a to 2c, 7a, 7b and 7c, `TestConflictRelation`, `TestUncertainReservationIsSettledByTheNextWrite`, `TestPreEpochETagCannotSatisfyANewAcquisition`, `TestResidenceChangesAndCleanupExclude`, `TestPausedPlacementIsNeverSwept`, `TestRemoteSetPlacementRunsUnderTheReceivedDeadline`, `TestDrainSettlesBeforeItReleases`, `TestDrainCountsWorkWithoutEntries`, `TestAdmissionAndTheDrainFlagAreOneCriticalSection`). |
| +3. Root under attack and crash: a shared-uid writer plants an acknowledgement, a request with another id, a changed copy of the request and a cancel in the agent's directory, and replaces `/data/adopt-pass.lock` with a symlink so a second pass runs without that lock; on an armed box a second pass's root pass lock fails for a root-level fault; an unarmed pass whose two locks failed is paused mid-publication while a later pass arms the box (review 7's finding 2); a publication replaces the live link between root's reference check and its `fstatat`; a second adopt pass starts between root's `fstatat` and its `unlinkat`; a route is upserted during the freeze; the link's parent chain is writable; root's request directory's ancestry is writable; root crashes after each journal line and each syscall: root removes only an entry carrying the recorded identity, the replacement survives, the second pass waits on the root pass lock until the step returns, the pass whose root pass lock failed publishes nothing, root's step does not run until the paused fail-open pass has exited and then finds its replacement link kept, root creates no directory under an ancestry that has not passed, the upsert waits for the freeze, the writable layout is never touched, a rejected ancestry gets no write, every intent resolves to removed, kept or absent, root finds its own taken request on restart, and the forged files change nothing but stopping (Tasks 6b and 6c, `TestRootRemovesOnlyTheRecordedIdentity`, `TestNoPublisherRunsBetweenFstatatAndUnlinkat`, `TestGatedPassPublishesNothingWithoutTheRootPassLock`, `TestFirstArmingWaitsForAFailOpenPassToEnd`, `TestLockRootPassChecksAncestryBeforeCreating`, `TestWritableLayoutIsNeverRemoved`, `TestRejectedAncestryGetsNoWrite`, `TestRootJournalResolvesEveryIntent`, `TestRootFindsItsOwnRequestsOnRestart`, `TestForgedAgentFilesAreHarmless`). |
| +4. Residence changes and cleanup exclude each other through `host/<hostname>`: a create (busy → 409, row void, no reservation written), a crash after `BeginAllocation` and after `Reserve`, a terminal row whose release failed, a create's reservation PUT that lands after its recovery voided and released it (refused by its condition); a placement that is busy (nothing written), whose record PUT commits after the client gave up (fenced before release), that pauses after its reservation for longer than the grace (registered, so never swept; its local writes fail at its deadline) and whose console crashed (the sweep fences and releases); an offline move selection; with the switch off two creates of different hosts and a cleanup beside an operator move all proceed, and the sentinel never blocks a create; the drain settles every `place` and `create` entry and refuses to finish while a self-serve move or a cleanup holds an entry; a create paused between `BeginAllocation` and its reservation and a placement registered and paused before its reservation keep the drain `draining`; an admission and the drain's flag are one critical section; a recovery paused between `PendingAllocations` and its locks while the create voids that allocation and releases its key reserves nothing and writes nothing (review 7's finding 3); a table write is one HTTP attempt, resent identically until answered; a normalisation and an acquisition whose first attempts are each held in the network keep the drain `draining` until each write is answered and 20 minutes have passed since each held attempt, and once the old console recreates the pre-epoch body, the held attempts delivered then are refused, so the table never regains the epoch body or the `place` entry (review 7's finding 4); a new console that exits without the clean line is rolled forward; a placement paused past its deadline makes no registry update, local or remote (Tasks 2a to 2c, 7a, 7b and 7c, `TestConflictRelation`, `TestUncertainReservationIsSettledByTheNextWrite`, `TestPreEpochETagCannotSatisfyANewAcquisition`, `TestTableWriteIsOneHTTPAttempt`, `TestUnansweredTableWriteIsResentIdentically`, `TestRecoveryReloadsTheAllocationUnderItsLocks`, `TestDrainWaitsOnDelayedNormalizationAndAcquisition`, `TestCleanExitNeedsEveryTableWriteAnswered`, `TestResidenceChangesAndCleanupExclude`, `TestPausedPlacementIsNeverSwept`, `TestRemoteSetPlacementRunsUnderTheReceivedDeadline`, `TestDrainSettlesBeforeItReleases`, `TestDrainCountsWorkWithoutEntries`, `TestAdmissionAndTheDrainFlagAreOneCriticalSection`). |
| 5. Publication is durable and findable: the report under the move id survives failed writes and is republished until `PublishedRev == Rev`; a refusal repeated every run is one event with a count; a skipped key's page survives an outbox that cannot create it, a pager that refuses it and a restart; a periodic page fires once per tick across restarts; the reaper waits for the report, `pages_pending`, the outbox and root's acknowledgement; an older obligation's report is found by its cleanup id after `LastMove` moved on and after the record was deleted (Tasks 5a, 5b and 8, `TestCleanupReportIsDurableAndBlocksTheReaper`, `TestPendingPagesSurviveARestart`, `TestOlderCleanupIsFoundByItsID`). |
| --- |
| @@ -72,42 +74,45 @@ Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and |
| - `ValidateRetired(r Retired) error`: `V == 1`; `ID == CleanupID(Box, Host, MoveID)`; `MoveID` a plain segment; `Host` passes the host shape; `Account`, `App`, `Box` non-empty; `Incarnation > 0`; `From != To`; `State` in the set and not `corrupt`; `Outcome` "" before `releasing` and in the set after; `Superseding` "" or a supersession reason, and set only in `cleaning` with `InFlight` set; `WrittenAt` non-zero; `Prefix` ends with `/`; inventory keys unique, under `Prefix`, neither excluded key; every key has at least one version; every version id passes `offlinecleanup.UsableVersionID` (non-empty and not `null`: `lease.UsableVersionID`'s rule, `internal/lease/staging.go:414`, restated here because this package imports the standard library only, and pinned to it by `TestUsableVersionIDMatchesTheLease`) and is unique within the key; `Seq` unique across the inventory; exactly one version has `Latest` set, it is the last, and `InventoryKey.Latest` is its id; the versions before it in the recorded order (`LastModified` ascending, ties by `Seq` descending); `ETag` non-empty exactly for the versions that are not markers; `Which` values unique across `LocalInventory` and `LocalLeft`, and known; every `Keys` entry names an inventory key with `Done` in `0..len(Versions)`; `InFlight`, when set, names an inventoried version of a key whose `Keys` state is `pending`. |
| - `type CorruptStub struct { Corrupt bool; ID, Host string; FoundAt time.Time }` (`{corrupt: true, id, host, found_at}`); `ValidateCorruptStub(c, box, host, move string) error`: `Corrupt`, `Host == host`, `ID == CleanupID(box, host, move)`, `FoundAt` non-zero. |
| - `func OfflineOpDir(dataDir, host, op string) string` (`<dataDir>/.offline-moves/<host>/<op>`, "" on a bad name, the checks `OfflineParkDir` applies); `func RetiredFile(dataDir, host, op string) string`; `func CleanupReportKey(host, op, box string) string` (`OfflineReportPrefix(host, op) + "cleanup/" + box + ".json"`). |
| -- **The evidence at `gone`** (`recordRetired`): the record is in `offline-finishing` carrying this move, `cfg.Judge != nil`, and `cfg.Region == m.From` (else nothing is written and the box logs once per day). **Tracked (N5):** `obs := cfg.Judge.Observe(ctx, s, rec.Account, rec.App, host)`; `LifetimeUnreadable` → an error (the step retries next pass, as any read in it does); `obs.Verdict == LifetimeValid && obs.Rec.Incarnation > 0 && obs.Slot != nil && obs.Slot.Incarnation == obs.Rec.Incarnation` → tracked; any other answer (incarnation 0 with no slot, a migrating record, stale, corrupt) → untracked: no `retired.json`, logged once per day, and `recordRetired` returns nil so the gone report is written as before this plan. Then: `OpenAt = *Move.OpenedAt`; `FinalTXID` from the seal at `moves.OfflineSealKey(host, id)` validated by `moves.ValidateOfflineSeal`; `TombstoneOwner = moves.TombstoneOwner(id)`; `Bucket/BucketRegion = cfg.OwnBucket/OwnBucketRegion`; `Prefix` as the Global Constraint; **`BucketInventory`** = `cfg.Versioned(ctx, Bucket, BucketRegion).ListVersions(ctx, Bucket, Prefix)` (any page error fails the whole call, `versioned.go:45-50`), keeping every entry, object version or delete marker, with `strings.HasPrefix(Key, Prefix)` and without the two excluded keys, each with `Seq` its index in the answer, grouped by key. **A `null` id writes nothing:** an entry whose version id `lease.UsableVersionID` refuses (an object written while the bucket was unversioned, or while versioning was suspended, which a later suspended write overwrites in place, so its id does not name one version) → no `retired.json`, logged once per day, and `recordRetired` returns nil, so the gone report proceeds as for an untracked lifetime. The listing is the versioning check: a bucket whose versioning is not enabled lists its objects as `null`, and the box roles cannot read the status itself (`s3:GetBucketVersioning` is granted only in the region host's inventory statement, `infra/modules/region-host/main.tf:223`, and not at all in `infra/modules/region-remote-host/main.tf`). Each key's versions in deletion order: the entries that are not `IsLatest` by `LastModified` ascending, ties by `Seq` descending (the listing is newest first with ties unordered, `versioned.go:46-47`, `:132-141`; the order is recorded, so nothing later depends on how a tie was listed), then the one `IsLatest` entry (`Latest`), whose id is also the key's `InventoryKey.Latest`; a key whose listing has no `IsLatest` entry, or more than one, fails the call (the step retries next pass); for each of `release.LiveLink(cfg.OptDir, host)` and `release.ProgramsPath(cfg.DataDir, host)`: absent → nothing; present and `RootOnlyChain(filepath.Dir(path))` → `LocalInventory`; present otherwise → `LocalLeft` with reason "writable layout" (on every built box the programs file is here: the agent's unit has `ReadWritePaths=/data`); `State = open`, `Rev = 1`. |
| +- **The evidence at `gone`** (`recordRetired`): the record is in `offline-finishing` carrying this move, `cfg.Judge != nil`, and `cfg.Region == m.From` (else nothing is written and the box logs once per day). **Tracked (N5):** `obs := cfg.Judge.Observe(ctx, s, rec.Account, rec.App, host)`; `LifetimeUnreadable` → an error (the step retries next pass, as any read in it does); `obs.Verdict == LifetimeValid && obs.Rec.Incarnation > 0 && obs.Slot != nil && obs.Slot.Incarnation == obs.Rec.Incarnation` → tracked; any other answer (incarnation 0 with no slot, a migrating record, stale, corrupt) → untracked: no `retired.json`, logged once per day, and `recordRetired` returns nil so the gone report is written as before this plan. Then: `OpenAt = *Move.OpenedAt`; `FinalTXID` from the seal at `moves.OfflineSealKey(host, id)` validated by `moves.ValidateOfflineSeal`; `TombstoneOwner = moves.TombstoneOwner(id)`; `Bucket/BucketRegion = cfg.OwnBucket/OwnBucketRegion`; `Prefix` as the Global Constraint; **`BucketInventory`** = `cfg.Versioned(ctx, Bucket, BucketRegion).ListVersions(ctx, Bucket, Prefix)` (any page error fails the whole call, `versioned.go:45-50`), keeping every entry, object version or delete marker, with `strings.HasPrefix(Key, Prefix)` and without the two excluded keys, each with `Seq` its index in the answer, grouped by key. **A `null` id writes nothing:** an entry whose version id `lease.UsableVersionID` refuses (an object written while the bucket was unversioned, or while versioning was suspended, which a later suspended write overwrites in place, so its id does not name one version) → no `retired.json`, logged once per day, and `recordRetired` returns nil, so the gone report proceeds as for an untracked lifetime. This check establishes one guarantee, that every inventoried id is usable, and claims nothing about the bucket's versioning status (U5): a bucket never versioned lists its objects as `null` and is refused here, but a suspended bucket whose objects all predate the suspension lists only real ids and passes, and so does an empty prefix. `Enabled` is the rollout's separate one-off check (Global Constraint "Arming and rollout", guarantee (ii)), because the box roles cannot read the status (`s3:GetBucketVersioning` is granted only in the region host's inventory statement, `infra/modules/region-host/main.tf:223`, and not at all in `infra/modules/region-remote-host/main.tf`). Each key's versions in deletion order: the entries that are not `IsLatest` by `LastModified` ascending, ties by `Seq` descending (the listing is newest first with ties unordered, `versioned.go:46-47`, `:132-141`; the order is recorded, so nothing later depends on how a tie was listed), then the one `IsLatest` entry (`Latest`), whose id is also the key's `InventoryKey.Latest`; a key whose listing has no `IsLatest` entry, or more than one, fails the call (the step retries next pass); for each of `release.LiveLink(cfg.OptDir, host)` and `release.ProgramsPath(cfg.DataDir, host)`: absent → nothing; present and `RootOnlyChain(filepath.Dir(path))` → `LocalInventory`; present otherwise → `LocalLeft` with reason "writable layout" (on every built box the programs file is here: the agent's unit has `ReadWritePaths=/data`); `State = open`, `Rev = 1`. |
| - Writes: `createRetired` is create-only (temp file, `fsync`, `os.Link` into place, directory `fsync`; an existing file is never replaced); `writeRetired` (temp, `fsync`, rename, directory `fsync`) increments `Rev` on every write except `markPublished` (Task 5a), which sets `PublishedRev` alone. |
| - **Corrupt (N6):** a file that is a valid `CorruptStub` stays as it is, never rewritten. A file that is neither a valid `Retired` nor a valid stub is moved aside create-only (`os.Link` then remove): to `retired.json.corrupt-1` when that name is free, else to the next free `retired.json.corrupt-<n>`, so the first damaged file is never overwritten; then the stub is written with `FoundAt` = `retired.json.corrupt-1`'s mtime. A stub keeps the obligation listed as `corrupt` and pages hourly from `FoundAt` (Task 5a) until an operator repairs it (a valid `retired.json` restored from `.corrupt-1`, or the directory removed after the entries and the mark are released by hand); it is never reaped and never released, so any entry or mark it held stays until the repair. |
| -- [ ] **Step 1: Write the failing tests**: `TestRetiredJSONWrittenOnceAtGoneBySealer` (fields; a key with three versions and a key whose current entry is a marker each recorded whole, in deletion order with the latest last and each version's ETag; `FinalTXID` from the seal; a non-sealer source box writes nothing; a second gone pass does not rewrite; an identity error writes nothing and no gone report); `TestUntrackedLifetimeWritesNoRetiredAndReportsGone` (incarnation 0 with no slot, a migrating record, a stale verdict: no file, the gone report written; an unreadable lifetime: no file and no gone report this pass, both next pass); `TestRetiredPrefixIsNormalisedOnce` (`cfg.OwnPrefix = "instances"` → `Prefix == "instances/shop.example.com/"`; the fake versioned store was listed with exactly that prefix; `instances/shop.example.com/lock.json` and `.../build-state.json` excluded with all their versions; `instances/shop.example.co` keys not listed; `ValidateRetired` passes on what gone wrote); `TestInventoryOrderIsOldestFirstLatestLast` (versions with equal `LastModified`: ordered by `Seq` descending, recorded, and the latest last and named whatever the tie order: `k.Latest == k.Versions[len(k.Versions)-1].VersionID && k.Versions[len(k.Versions)-1].Latest`; a listing with two `IsLatest` entries for one key → an error and no file); `TestNullVersionIDWritesNoRetired` (a listing with a `null` id under the prefix: `recordRetired` returns nil, no file exists, the gone report is written; `ValidateRetired` of an otherwise valid inventory with one `null` id → an error naming it); `TestUsableVersionIDMatchesTheLease` (external test package: for `""`, `"null"`, `"Null"`, `"3HL4kqtJlcpXroDTDmJ+rmSpXd3dIbrHY"`, `offlinecleanup.UsableVersionID(v) == lease.UsableVersionID(v)`); `TestWritablePathsAreLeftNotRequested` (a path under a directory the test uid owns → `LocalLeft`, "writable layout"; a path under a root-only chain the test cannot build is covered by injecting `RootOnlyChain`); `TestCleanupIDIsDeterministic`; `TestValidateRetired` (each rule); `TestCorruptStubIsValidAndKeepsTheFirstDamage` (damaged → `.corrupt-1` and a stub that validates; a second scan changes nothing; a damaged stub → `.corrupt-2`, `FoundAt` still `.corrupt-1`'s mtime); `TestIdentifyUsesLstat` (a dangling link has an identity; a regular file; absent); `TestRootOnlyChain` (`/usr` true on a normal system; a temp directory false; a symlink component false). |
| -- [ ] **Step 2: Run to verify they fail**: `go test ./internal/offlinecleanup -count=1`, `go test ./internal/host -run 'RetiredJSON|Untracked|RetiredPrefix|InventoryOrder|NullVersionID|WritablePaths|CorruptStub' -count=1`. |
| +- [ ] **Step 1: Write the failing tests**: `TestRetiredJSONWrittenOnceAtGoneBySealer` (fields; a key with three versions and a key whose current entry is a marker each recorded whole, in deletion order with the latest last and each version's ETag; `FinalTXID` from the seal; a non-sealer source box writes nothing; a second gone pass does not rewrite; an identity error writes nothing and no gone report); `TestUntrackedLifetimeWritesNoRetiredAndReportsGone` (incarnation 0 with no slot, a migrating record, a stale verdict: no file, the gone report written; an unreadable lifetime: no file and no gone report this pass, both next pass); `TestRetiredPrefixIsNormalisedOnce` (`cfg.OwnPrefix = "instances"` → `Prefix == "instances/shop.example.com/"`; the fake versioned store was listed with exactly that prefix; `instances/shop.example.com/lock.json` and `.../build-state.json` excluded with all their versions; `instances/shop.example.co` keys not listed; `ValidateRetired` passes on what gone wrote); `TestInventoryOrderIsOldestFirstLatestLast` (versions with equal `LastModified`: ordered by `Seq` descending, recorded, and the latest last and named whatever the tie order: `k.Latest == k.Versions[len(k.Versions)-1].VersionID && k.Versions[len(k.Versions)-1].Latest`; a listing with two `IsLatest` entries for one key → an error and no file); `TestNullVersionIDWritesNoRetired` (a listing with a `null` id under the prefix: `recordRetired` returns nil, no file exists, the gone report is written; `ValidateRetired` of an otherwise valid inventory with one `null` id → an error naming it); `TestRealIDsOnASuspendedBucketStillWriteRetired` (U5's limit, pinned: a fake bucket marked suspended whose listing under the prefix holds only real ids → `retired.json` is written and validates; the listing check is about ids, and the status is the rollout's check); `TestUsableVersionIDMatchesTheLease` (external test package: for `""`, `"null"`, `"Null"`, `"3HL4kqtJlcpXroDTDmJ+rmSpXd3dIbrHY"`, `offlinecleanup.UsableVersionID(v) == lease.UsableVersionID(v)`); `TestWritablePathsAreLeftNotRequested` (a path under a directory the test uid owns → `LocalLeft`, "writable layout"; a path under a root-only chain the test cannot build is covered by injecting `RootOnlyChain`); `TestCleanupIDIsDeterministic`; `TestValidateRetired` (each rule); `TestCorruptStubIsValidAndKeepsTheFirstDamage` (damaged → `.corrupt-1` and a stub that validates; a second scan changes nothing; a damaged stub → `.corrupt-2`, `FoundAt` still `.corrupt-1`'s mtime); `TestIdentifyUsesLstat` (a dangling link has an identity; a regular file; absent); `TestRootOnlyChain` (`/usr` true on a normal system; a temp directory false; a symlink component false). |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/offlinecleanup -count=1`, `go test ./internal/host -run 'RetiredJSON|Untracked|RetiredPrefix|InventoryOrder|NullVersionID|RealIDsOnASuspended|WritablePaths|CorruptStub' -count=1`. |
| - [ ] **Step 3: Implement. Step 4: Run to verify they pass.** |
| - [ ] **Step 5: Commit** `moves: the sealing source box records retired.json at gone for a tracked lifetime, with every real version id in a recorded deletion order and the key's latest named, a normalised prefix and a deterministic cleanup id`. |
| ### Task 2a: The table's kinds, epoch and rev, and its parser |
| **Files:** |
| -- Create: `internal/moves/reserve.go` (`ReadActive`, `ReadForAcquire`, `NextActiveBody`, `Release`, `Holds`; Task 2b adds the rest) |
| +- Create: `internal/moves/reserve.go` (`ReadActive`, `ReadForAcquire`, `NextActiveBody`, `Release`, `Holds`; Task 2b adds the rest), `internal/moves/tablewrite.go` (`PutTable`, `TableWrites`, `ErrTableClosed`), `internal/moves/tablewrite_audit_test.go` |
| +- Modify: `internal/release/store.go` (`SingleAttemptPutter`, `PutOnce`, `ErrNoSingleAttemptPut`; `(*S3Store).PutOnce`, `(*DirStore).PutOnce`, `prefixStore.PutOnce`) |
| - Modify: `internal/moves/offline.go` (`OfflineKindCreate = "create"`, `OfflineKindPlace = "place"`; `OfflineActive` gains `Epoch string \`json:"epoch,omitempty"\`` and `Rev uint64 \`json:"rev,omitempty"\``; `OfflineReservation.valid()` and `ParseOfflineActive` take the Global Constraint's entry rules; `HostKey`, `IsHostKey`) |
| -- Test: `internal/moves/reserve_test.go`, `internal/moves/offline_compat_test.go` |
| +- Test: `internal/moves/reserve_test.go`, `internal/moves/offline_compat_test.go`, `internal/moves/tablewrite_test.go`, `internal/release/putonce_test.go` |
| **Interfaces:** |
| +- **One attempt per send (U3).** `type SingleAttemptPutter interface { PutOnce(ctx context.Context, key string, body []byte, ifMatch string) (string, error) }`; `var ErrNoSingleAttemptPut`; `func PutOnce(ctx context.Context, s Store, key string, body []byte, ifMatch string) (string, error)`: `s` a `SingleAttemptPutter` → its `PutOnce`; else `ErrNoSingleAttemptPut`, nothing sent. `(*S3Store).PutOnce` is `Put`'s body (`store.go:787-818`, the same condition and the same error mapping) with `s.Client.PutObject(ctx, in, func(o *awss3.Options) { o.Retryer = aws.NopRetryer{} })`: `aws.NopRetryer` (`aws-sdk-go-v2/aws/retryer.go`) answers `MaxAttempts() == 1` and retries nothing, and a per-call `Retryer` replaces the client's default retryer for that call. `(*DirStore).PutOnce` is `Put` (a local write has no retries). `prefixStore.PutOnce` rebases the key and forwards to the inner store's `PutOnce`, or answers `ErrNoSingleAttemptPut`. `Put` itself is unchanged for every other caller. |
| +- `type TableWrites struct` (mutex-guarded; the zero value is usable): `func (tw *TableWrites) Unanswered() int`, `func (tw *TableWrites) LastTimedOut() time.Time` (the time of the last attempt, of any write, that ended without an answer; zero when none), `func (tw *TableWrites) Resend(ctx context.Context, s release.Store)` (each registered write sent again, identically, once; an answer removes it), `func (tw *TableWrites) Close()` (from then on `PutTable` answers `ErrTableClosed` and sends nothing). `func PutTable(ctx context.Context, s release.Store, tw *TableWrites, body []byte, ifMatch string) (etag string, err error)`: registers the write, then `release.PutOnce(ctx, s, OfflineActiveKey, body, ifMatch)`; success → the ETag, the write removed; `release.ErrCASConflict` → that error, removed (an answer: our attempt landed or another did, and the caller's next read tells which); `ErrNoSingleAttemptPut` → that error, removed (nothing was sent); any other error (a timeout, a transport error, a 5xx) → `LastTimedOut` set to that attempt's send time, the same request sent again after 1 s, 2 s, 4 s and then every 10 s; `ctx` ending first → `ErrUncertain` wrapped, the write left registered for `Resend`. **Below and in later tasks, `ReadActive`, `ReadForAcquire`, `Reserve` and `Release` take `tw` after `s` and are written without it; every call passes its process's one `TableWrites`: the console's `s.tableWrites`, the box worker's `w.tableWrites`.** Every write they make is one `PutTable`. |
| - `func HostKey(host string) string`; `func IsHostKey(key string) bool` (`strings.HasPrefix(key, "host/")`; a box label never contains `/`, `controlname.ValidLabel`). |
| -- `type Active struct { Body OfflineActive; Table map[string]OfflineReservation; ETag string }`; `func ReadActive(ctx context.Context, s release.Store) (Active, error)`: `Get`; absent → the bootstrap `Put` of `{v: 1, epoch: newEpoch(), rev: 1}` with `ifMatch ""` (create-only), then `Get` again whatever the `Put` answered (a lost create means another writer created it; an uncertain one can only ever create an empty table, so it is harmless); a read error or `ParseOfflineActive` failing → `ErrUnreadable`. `func NextActiveBody(prev OfflineActive, table map[string]OfflineReservation) OfflineActive`: `OfflineActiveBody(table)` with `Epoch = prev.Epoch` (a new one when empty) and `Rev = prev.Rev + 1`. Readers that only read (`SelfServeEvidence`, `selfserve_pages.go:143`) keep treating an absent object as an empty table. `func ReadForAcquire(ctx context.Context, s release.Store) (Active, error)` (T4): `ReadActive`; a body with `Epoch == ""` or `Rev == 0` (an older console wrote it) → the normalisation write, one `Put` with `If-Match` the read ETag of `NextActiveBody(body, table)` (the same entries, a fresh epoch, `rev` 1), whatever it answers, then `ReadActive` again; at most three rounds; a body still without both → `ErrUnreadable` wrapping "the table could not be normalised". Only acquisitions call it (`Reserve`, Task 2b, and the console's `reserveBoxes`); a release adds nothing and reads through `ReadActive`. |
| -- `type Op struct{ Kind, ID string }`; `var ErrUnreadable`; `func Holds(table map[string]OfflineReservation, keys []string, mine Op) bool`; `func Release(ctx context.Context, s release.Store, mine Op) error`: `ReadActive`; remove `mine`'s entries; **always** one `Put` with `If-Match` of `NextActiveBody`, even when `mine` held nothing (the settling write); a CAS conflict re-reads, three rounds as `releaseBoxes`; a malformed table is an error and is left as it is. A release that read no entry and wrote successfully proves that no earlier `Put` of `mine` can land (the Global Constraint), which a read alone never does. `Release` accepts an `Op` of any kind, so the drain's settling write (Task 7c) is a `Release` of an op that holds nothing. |
| +- `type Active struct { Body OfflineActive; Table map[string]OfflineReservation; ETag string }`; `func ReadActive(ctx context.Context, s release.Store) (Active, error)`: `Get`; absent → the bootstrap `PutTable` of `{v: 1, epoch: newEpoch(), rev: 1}` with `ifMatch ""` (create-only), then `Get` again whatever the `Put` answered (a lost create means another writer created it; an uncertain one can only ever create an empty table, so it is harmless); a read error or `ParseOfflineActive` failing → `ErrUnreadable`. `func NextActiveBody(prev OfflineActive, table map[string]OfflineReservation) OfflineActive`: `OfflineActiveBody(table)` with `Epoch = prev.Epoch` (a new one when empty) and `Rev = prev.Rev + 1`. Readers that only read (`SelfServeEvidence`, `selfserve_pages.go:143`) keep treating an absent object as an empty table. `func ReadForAcquire(ctx context.Context, s release.Store) (Active, error)` (T4): `ReadActive`; a body with `Epoch == ""` or `Rev == 0` (an older console wrote it) → the normalisation write, one `PutTable` with `If-Match` the read ETag of `NextActiveBody(body, table)` (the same entries, a fresh epoch, `rev` 1), whatever it answers, then `ReadActive` again; at most three rounds; a body still without both → `ErrUnreadable` wrapping "the table could not be normalised". Only acquisitions call it (`Reserve`, Task 2b, and the console's `reserveBoxes`); a release adds nothing and reads through `ReadActive`. |
| +- `type Op struct{ Kind, ID string }`; `var ErrUnreadable`; `func Holds(table map[string]OfflineReservation, keys []string, mine Op) bool`; `func Release(ctx context.Context, s release.Store, mine Op) error`: `ReadActive`; remove `mine`'s entries; **always** one `PutTable` with `If-Match` of `NextActiveBody`, even when `mine` held nothing (the settling write); a CAS conflict re-reads, three rounds as `releaseBoxes`; a malformed table is an error and is left as it is. A release that read no entry and wrote successfully proves that no earlier `Put` of `mine` can land (the Global Constraint), which a read alone never does. `Release` accepts an `Op` of any kind, so the drain's settling write (Task 7c) is a `Release` of an op that holds nothing. |
| - **The old parser, frozen:** `offline_compat_test.go` carries `parsePlan3OfflineActive`, a verbatim copy of `ParseOfflineActive` and `valid()` as built at `0840937f` (`offline.go:584-587`, `:599-622`), and is the witness of what a console before this plan reads. |
| -- [ ] **Step 1: Write the failing tests**: `TestEntryShapeRules` (a `place` or `create` entry without account or app, a host-key entry naming another host → malformed; each valid shape parses); `TestOldParserReadsTheNewTable` (bodies written by `NextActiveBody` with `move` and `cleanup` entries, host keys and `epoch`/`rev` parse with `parsePlan3OfflineActive`; a body with a `create` or a `place` entry is malformed to it, which the rollback procedure's step 2 covers); `TestActiveTableIsNeverDeleted` (releasing the last entry writes an empty table; no `Delete` or `DeleteIfMatch` reaches the key); `TestBootstrapIsCreateOnlyAndEmpty` (two first writers: one bootstrap lands, both then write through `If-Match`); `TestRevAndEpochNeverRepeatABody` (alternating writes of one entry: every ETag distinct; a body without `epoch`, as plan 3 writes, gets one); `TestReleaseAlwaysWrites` (an op that holds nothing still writes rev + 1); `TestAcquisitionNormalizesAPreEpochTableFirst` (a body as plan 3 writes it, `{v: 1, reservations: {...}}` at ETag `E0`: the fake store's log is `Get`, `Put(ifMatch E0)` of the same entries with an epoch and `rev == 1`, `Get`, and only then the acquisition's `Put` with `ifMatch == E1 && E1 != E0`; a normalisation `Put` that loses its condition is followed by a fresh `Get`, never by an acquisition against `E0`); `TestPreEpochETagCannotSatisfyANewAcquisition` (review 6's finding 4: a metadata-free table holding operator move A at ETag `E`; a placement's acquisition `Put` is held by the fake, and `held.IfMatch != E`; the placement's settlement and the drain write; plan 3's writer, simulated by `parsePlan3OfflineActive` and a re-marshal through plan 3's struct, recreates the A-only body, so `E` is current again; the held `Put` is delivered: `errors.Is(err, release.ErrCASConflict)`, and the table holds no `place` entry). |
| -- [ ] **Step 2: Run to verify they fail**: `go test ./internal/moves -run 'EntryShape|OldParser|ActiveTable|Bootstrap|RevAndEpoch|ReleaseAlways|NormalizesAPreEpoch|PreEpochETag' -count=1`. |
| +- [ ] **Step 1: Write the failing tests**: `TestEntryShapeRules` (a `place` or `create` entry without account or app, a host-key entry naming another host → malformed; each valid shape parses); `TestOldParserReadsTheNewTable` (bodies written by `NextActiveBody` with `move` and `cleanup` entries, host keys and `epoch`/`rev` parse with `parsePlan3OfflineActive`; a body with a `create` or a `place` entry is malformed to it, which the rollback procedure's step 2 covers); `TestActiveTableIsNeverDeleted` (releasing the last entry writes an empty table; no `Delete` or `DeleteIfMatch` reaches the key); `TestBootstrapIsCreateOnlyAndEmpty` (two first writers: one bootstrap lands, both then write through `If-Match`); `TestRevAndEpochNeverRepeatABody` (alternating writes of one entry: every ETag distinct; a body without `epoch`, as plan 3 writes, gets one); `TestReleaseAlwaysWrites` (an op that holds nothing still writes rev + 1); `TestAcquisitionNormalizesAPreEpochTableFirst` (a body as plan 3 writes it, `{v: 1, reservations: {...}}` at ETag `E0`: the fake store's log is `Get`, `Put(ifMatch E0)` of the same entries with an epoch and `rev == 1`, `Get`, and only then the acquisition's `Put` with `ifMatch == E1 && E1 != E0`; a normalisation `Put` that loses its condition is followed by a fresh `Get`, never by an acquisition against `E0`); `TestPreEpochETagCannotSatisfyANewAcquisition` (review 6's finding 4: a metadata-free table holding operator move A at ETag `E`; a placement's acquisition `Put` is held by the fake, and `held.IfMatch != E`; the placement's settlement and the drain write; plan 3's writer, simulated by `parsePlan3OfflineActive` and a re-marshal through plan 3's struct, recreates the A-only body, so `E` is current again; the held `Put` is delivered: `errors.Is(err, release.ErrCASConflict)`, and the table holds no `place` entry); `TestTableWriteIsOneHTTPAttempt` (an `httptest` S3 endpoint behind `S3Store`, answering 500 to every `PutObject`: `PutOnce` returns the error after exactly one request reached the server, where `Put` sends the SDK's default three; a 412 → `release.ErrCASConflict` after one request); `TestUnansweredTableWriteIsResentIdentically` (a fake whose first attempt answers a timeout: the second request's bytes and `If-Match` equal the first's byte for byte; a 412 after it → `errors.Is(err, release.ErrCASConflict)` and `tw.Unanswered() == 0`; `ctx` ending before any answer → `errors.Is(err, moves.ErrUncertain)`, `tw.Unanswered() == 1`, `tw.LastTimedOut()` equal to the first attempt's send time, and a later `tw.Resend` that is answered leaves `tw.Unanswered() == 0`; after `tw.Close()`, `errors.Is(err, moves.ErrTableClosed)` and the fake saw no request); `TestStoreWithoutPutOnceWritesNothing` (a `fleetapi` client: `ReadForAcquire` and `Release` answer `release.ErrNoSingleAttemptPut` and the door saw no `PUT`); `TestEveryTableWriteGoesThroughPutTable` (`go/ast` with `go/types` over the non-test files under `./internal/...`: no `Put`, `PutFrom`, `PutOnce`, `Delete` or `DeleteIfMatch` with `moves.OfflineActiveKey` as its key outside `moves.PutTable`'s one `release.PutOnce`). |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/moves -run 'EntryShape|OldParser|ActiveTable|Bootstrap|RevAndEpoch|ReleaseAlways|NormalizesAPreEpoch|PreEpochETag|TableWrite|UnansweredTableWrite|WithoutPutOnce|EveryTableWrite' -count=1`, `go test ./internal/release -run 'TableWriteIsOne|PutOnce' -count=1`. |
| - [ ] **Step 3: Implement. Step 4: Run to verify they pass**, plus `go test ./internal/moves -run 'Offline' -count=1` (plan 2's parser tests green). |
| -- [ ] **Step 5: Commit** `moves: reservation kinds for creates and placements, and a table that is never deleted, never repeats a body and is normalised before any acquisition`. |
| +- [ ] **Step 5: Commit** `moves: reservation kinds for creates and placements, and a table that is never deleted, never repeats a body, is normalised before any acquisition and is written one HTTP attempt at a time, resent until answered`. |
| ### Task 2b: One conflict relation, `Reserve`, and recovery mode counting moves only |
| **Files:** |
| -- Modify: `internal/moves/reserve.go` (`Conflicts`, `Reserve`, the errors), `internal/console/lifetimes_store.go` (`AllocationState(opID) (state string, found bool, err error)`), `internal/console/offlinereserve.go` (`reserveBoxes` reads through `moves.ReadForAcquire` and writes `moves.NextActiveBody` with `If-Match`; its steps (4) and (5) become one call to `moves.Conflicts`; the wanted keys add `moves.HostKey(r.Host)`; `releaseBoxes` wraps `moves.Release`, so the `DeleteIfMatch` at line 478 goes; `offlineReclaimable`: `cleanup` → `reclaimHeld` always, `create` and `place` below; `cleanupReportEvidence` and `Server.cleanupReports` deleted), `internal/console/selfserve_pass.go` (`SelfServeEvidence`, lines 41-53, counts only entries of kind `move`) |
| +- Modify: `internal/moves/reserve.go` (`Conflicts`, `Reserve`, the errors), `internal/console/server.go` (`Server` gains `tableWrites moves.TableWrites`), `internal/console/lifetimes_store.go` (`AllocationState(opID) (state string, found bool, err error)`), `internal/console/offlinereserve.go` (`reserveBoxes` reads through `moves.ReadForAcquire` and writes `moves.NextActiveBody` through `moves.PutTable` with `If-Match`, so its line 241 `Put` and `releaseBoxes`' line 487 `Put` go; its steps (4) and (5) become one call to `moves.Conflicts`; the wanted keys add `moves.HostKey(r.Host)`; `releaseBoxes` wraps `moves.Release`, so the `DeleteIfMatch` at line 478 goes; `offlineReclaimable`: `cleanup` → `reclaimHeld` always, `create` and `place` below; `cleanupReportEvidence` and `Server.cleanupReports` deleted), `internal/console/selfserve_pass.go` (`SelfServeEvidence`, lines 41-53, counts only entries of kind `move`) |
| - Test: `internal/moves/reserve_conflict_test.go`, `internal/console/offlinereserve_hostkey_test.go` |
| **Interfaces:** |
| - `func Conflicts(table map[string]OfflineReservation, keys []string, mine Op, movesExclusive bool) map[Op]map[string]OfflineReservation` = the Global Constraint's relation, `movesExclusive` applying rule (c) only when `mine.Kind == move`. |
| -- `var ErrReservationMismatch`, `ErrUncertain`; `type ErrReserved struct { Key string; Holder OfflineReservation }`; `func Reserve(ctx context.Context, s release.Store, mine Op, want map[string]OfflineReservation) (map[string]OfflineReservation, error)` for the box, the create and the placement (N10: the caller names the entry for each key; every entry carries `mine` and is valid, else an error and nothing written): `ReadForAcquire` (so the condition of its `Put` is always an epoch-bearing ETag); own entries on a key outside `want` → `ErrReservationMismatch`; own entries covering `want` → the held entries returned, nothing written; any conflict → `ErrReserved` naming the oldest conflicting entry (no reclaim on these paths); else one `Put` with `If-Match` the read ETag of `NextActiveBody` with the missing keys added (at the held entries' `ReservedAt` when partial); a CAS conflict re-reads once (through `ReadForAcquire` again); a second loss → `ErrReserved` from the last read; a `Put` error that is not a CAS conflict → `ErrUncertain` wrapped: the caller keeps its state as if it might hold the keys, and the next successful write by anyone, its own retry or its release included, settles it. |
| +- `var ErrReservationMismatch`, `ErrUncertain`; `type ErrReserved struct { Key string; Holder OfflineReservation }`; `func Reserve(ctx context.Context, s release.Store, mine Op, want map[string]OfflineReservation) (map[string]OfflineReservation, error)` for the box, the create and the placement (N10: the caller names the entry for each key; every entry carries `mine` and is valid, else an error and nothing written): `ReadForAcquire` (so the condition of its `Put` is always an epoch-bearing ETag); own entries on a key outside `want` → `ErrReservationMismatch`; own entries covering `want` → the held entries returned, nothing written; any conflict → `ErrReserved` naming the oldest conflicting entry (no reclaim on these paths); else one `Put` with `If-Match` the read ETag of `NextActiveBody` with the missing keys added (at the held entries' `ReservedAt` when partial); a CAS conflict re-reads once (through `ReadForAcquire` again); a second loss → `ErrReserved` from the last read; `PutTable` answering `ErrUncertain` (its context ended with the write unanswered; the write stays in `tw` and is resent until answered) → `ErrUncertain` wrapped: the caller keeps its state as if it might hold the keys, and the next successful write by anyone, its own retry or its release included, settles it. |
| - The console's `reserveBoxes`: `want` = an entry `r` for each of `sel.reservationKeys()` and for `moves.HostKey(r.Host)`; the own retry is unchanged (an own reservation made before this plan, without the host key, is partial and completed in the same write; an operator operation holding the sentinel covers every box key and has its host key completed); conflicts = `moves.Conflicts(table, keys, mine, s.cfg.SelfServeMoves != "offline")`; reclaim as today for `move`; `create`: `s.store.AllocationState(opID)` found and terminal → `reclaimYes`, pending → `reclaimHeld`, not found or an error → `reclaimUnreadable`; `place`: `reclaimHeld` until Task 7b replaces it with `s.settlePlacement`. |
| - [ ] **Step 1: Write the failing tests**: `TestConflictRelation` (every pair of kinds over box keys, host keys and the sentinel, with and without `movesExclusive`: two creates of different hosts never conflict; a create and a sentinel move of another host do not; a create and a move of the same host do; a cleanup and an operator move on disjoint boxes do not, with the switch off as on; two moves on disjoint boxes conflict only with the switch off); `TestUncertainReservationIsSettledByTheNextWrite` (review 4's interleaving: create C's `Put` is held by the fake store and answers a timeout; a cleanup acquires; C's recovery reads no entry of C, voids and calls `Release(C)`, which writes; the cleanup releases; C's held `Put` is then delivered and fails its condition; the table never holds C); `TestReserveReturnsTheHeldEntry`; `TestReserveCompletesAPartialOwnReservation`; `TestReserveUncertainPutIsNotAnAnswer` (a store that commits and then errors → `ErrUncertain`; the own retry returns the held entries); `TestMoveSelectionsHoldTheHostKey` (a reservation written by plan 2's code, without the host key, is completed by its own retry; a cleanup on the host key makes the move busy); `TestReleaseIsTheConsolesAlgorithm` (plan 2's `releaseBoxes` tests green through the wrapper, with "deleted" read as "empty"); `TestConsoleNeverDeletesTheTable`; `TestCleanupEntriesAreNeverReclaimed`; `TestCreateEntryReclaimOnTheAllocationRow`; `TestRecoveryModeCountsMoveEntriesOnly` (cleanup, create and place entries alone, and an empty table, leave `SelfServeEvidence` false). |
| @@ -122,19 +127,19 @@ Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and |
| - Test: `internal/console/residence_sweep_test.go` |
| **Interfaces:** |
| -- `s.residenceSweep(ctx, now)`: one `moves.ReadActive`; a `cleanup` entry older than 6 h → `s.selfServePage(ctx, id, ssPageCleanupHeld, "a cleanup has held <host> for over six hours", body, now)` with the body naming the entry's id; a `create` entry older than `offlineLockGrace` with no allocation row → `ssPageCreateOrphan` (same body form). A read error is logged and the sweep waits for the next iteration. Task 7b adds the placement settle. |
| +- `s.residenceSweep(ctx, now)`: first `s.tableWrites.Resend(ctx, s.cfg.Releases)` (U3: a write whose caller gave up is resent here every iteration until it is answered); then one `moves.ReadActive`; a `cleanup` entry older than 6 h → `s.selfServePage(ctx, id, ssPageCleanupHeld, "a cleanup has held <host> for over six hours", body, now)` with the body naming the entry's id; a `create` entry older than `offlineLockGrace` with no allocation row → `ssPageCreateOrphan` (same body form). A read error is logged and the sweep waits for the next iteration. Task 7b adds the placement settle. |
| -- [ ] **Step 1: Write the failing tests**: `TestResidenceSweepPagesOldEntries` (a 6 h 1 m cleanup entry pages once, keyed by its id; a 5 h one does not; a create entry with no row past the grace pages; one with a pending row does not); `TestResidenceSweepRunsInEveryRecoveryIteration`; `TestResidenceSweepReadErrorChangesNothing`. |
| +- [ ] **Step 1: Write the failing tests**: `TestResidenceSweepPagesOldEntries` (a 6 h 1 m cleanup entry pages once, keyed by its id; a 5 h one does not; a create entry with no row past the grace pages; one with a pending row does not); `TestResidenceSweepRunsInEveryRecoveryIteration`; `TestResidenceSweepReadErrorChangesNothing`; `TestResidenceSweepResendsUnansweredTableWrites` (a write left registered by an `ErrUncertain` create: the next iteration sends the identical request, and its answer leaves `s.tableWrites.Unanswered() == 0`). |
| - [ ] **Step 2: Run to verify they fail**: `go test ./internal/console -run 'ResidenceSweep' -count=1`. |
| - [ ] **Step 3: Implement. Step 4: Run to verify they pass.** |
| -- [ ] **Step 5: Commit** `console: the residence sweep pages a cleanup held over six hours and a create entry without its allocation row`. |
| +- [ ] **Step 5: Commit** `console: the residence sweep resends unanswered table writes and pages a cleanup held over six hours and a create entry without its allocation row`. |
| -### Task 3: The cleaning mark, its refusals and the writer inventory |
| +### Task 3: The cleaning mark, its refusals, the wake's answer and the writer inventory |
| **Files:** |
| - Create: `internal/lease/cleaning.go`, `internal/host/hibernateprefix_inventory_test.go` |
| -- Modify: `internal/lease/lease.go` (`AcquireTTL`, line 171: a `readLock` before `AcquireLease`), `internal/lease/tombstone.go` (`Tombstone`, line 183: the owner check before the expiry test), `internal/lease/lockstate.go` (`LockState`, line 78: `held` for a `cleaning:` owner), `internal/host/movesoffline_land.go` (the lease step, lines 612-630: a `cleaning:` owner waits, before the expiry case) |
| -- Test: `internal/lease/cleaning_test.go` (over the S3 fake the tombstone tests use), `internal/host/movesoffline_land_cleaning_test.go` |
| +- Modify: `internal/lease/lease.go` (`AcquireTTL`, line 171: a `readLock` before `AcquireLease`), `internal/lease/tombstone.go` (`Tombstone`, line 183: the owner check before the expiry test), `internal/lease/lockstate.go` (`LockState`, line 78: `held` for a `cleaning:` owner), `internal/host/movesoffline_land.go` (the lease step, lines 612-630: a `cleaning:` owner waits, before the expiry case), `internal/activator/backend_exec.go` (`ExecBackend.Acquire`, lines 702-710), `internal/activator/activator.go` (`errLeaseCleaning` and `cleaningRetryAfter` beside `errLeaseHeld`, line 1083, and `configRefusedRetryAfter`, line 1120; `serveWakeError`, line 5413; `doWake`'s lease log, line 2244), `internal/host/schedule_wiring_test.go` (one case in `TestDeliverViaHandlerMapsWakeRefusals`), the copy record and `approved_moves_copy_test.go` |
| +- Test: `internal/lease/cleaning_test.go` (over the S3 fake the tombstone tests use), `internal/host/movesoffline_land_cleaning_test.go`, `internal/activator/cleaning_wake_test.go` |
| **Interfaces:** |
| - `const CleaningPrefix = "cleaning:"`; `var CleaningExpiry = time.Date(9999, 12, 31, 0, 0, 0, 0, time.UTC)`; `func CleaningOwner(id string) string`; `func IsCleaningOwner(owner string) bool` (the prefix and a non-empty id); `var ErrNotOurLock` (the lock is absent, another owner's, or this move's tombstone still live: nothing written). |
| @@ -142,12 +147,13 @@ Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and |
| - **No renewal (T1).** Nothing rewrites a `cleaning:` lock but `UnmarkCleaning`: rev 6's `ReassertCleaning`, `cleanupMarkTTL`, `cleanupMarkMargin` and the 30-minute reassert are gone. |
| - `func UnmarkCleaning(ctx, bucket, location, region, cleaningOwner, moveOwner string) error`: owner `cleaningOwner` → conditional PUT of `{Generation, ExpiresAt: liftedExpiry, Owner: moveOwner}`; absent or another owner → nil (nothing of ours; the caller logs it); a lost condition → `ErrTombstoneChanged`. |
| - **The refusals, by code path.** `AcquireTTL` (`lease.go:171`; so `Acquire`, `ReplicateWithLease` at `:443`, `ExecBackend.Acquire` at `backend_exec.go:701` and the staging lease): `readLock` first; an `IsCleaningOwner` owner → `fmt.Errorf("acquire lease: %w", &HeldError{Owner, ExpiresAt})`, whatever its expiry, before `AcquireLease` and with nothing written; absent → on to `AcquireLease` as today. The sentinel expiry already makes `AcquireLease` refuse on its own read (`litestream@v0.5.14/s3/leaser.go:96`), so a mark written by `MarkCleaning` is refused on the very read that conditions the write; the pre-read adds the refusal for a mark whose expiry was edited by hand, and costs one `GET` per acquisition. `Tombstone` (`tombstone.go:183`): an `IsCleaningOwner` owner → `heldErr(cur)` before the expiry test. `LiftTombstone` and `Reassert` already refuse an owner that is not `moved:` (`tombstone.go:286`, `:225`). `LockState` (`lockstate.go:78`): `held = lockHeld(expires, now) || IsCleaningOwner(owner)`, so the cut's stop evidence (`movespass.go:647`) and the seed finish (`console/seedfinish.go:255`) read a mark as held. The landing's lease step (`movesoffline_land.go:612-630`): an `IsCleaningOwner` owner → `r.landWait("lease", "a cleanup holds the target's lock")`, before the expiry case. The health renewer (`health.go:134`) renews only its own lease by its ETag and never meets the mark. |
| +- **The wake's answer to a mark (U4).** Today `ExecBackend.Acquire` turns every `*lease.HeldError` into `errLeaseHeld{retryAfter: time.Until(he.ExpiresAt)}` (`backend_exec.go:705`), and `serveWakeError` answers `Retry-After: int(retryAfter/time.Second) + 1` (`activator.go:5427`); with the sentinel expiry that duration saturates and the header reads `9223372037` on a 64-bit box (review 7's finding 7). The lease's exclusion is unchanged: the mark is still refused whatever its age. Only what the activator tells its consumers changes. `ExecBackend.Acquire`: an `*lease.HeldError` whose `lease.IsCleaningOwner(he.Owner)` → `&errLeaseCleaning{holder: he.Owner}`, and `he.ExpiresAt` is not read; any other holder as today. `type errLeaseCleaning struct{ holder string }` (`Error()`: `instance replica marked by %q: a cleanup is running`); `const cleaningRetryAfter = 30` (seconds, fixed: a cleanup's run is at most 60 s, `cleanupRunDeadline`, and a client back within half a minute finds a finished cleanup promptly). `serveWakeError` gains, before the `errLeaseHeld` case: `case errors.As(err, &cleaning): w.Header().Set("Retry-After", strconv.Itoa(cleaningRetryAfter)); edge.MarkAnswered(r); http.Error(w, "instance is being cleaned up after a move. retry shortly", http.StatusServiceUnavailable)`. The text is operator-facing, lower case and has no em dash; it begins with `instance`, so the scheduler's `deliverVia` (`internal/host/schedule_wiring.go:514`) reads it as a wake refusal and retries it as one; its copy row is pending Paul. `doWake` (line 2244) logs `activator: %s: a cleanup holds the replica (%q); refusing to wake` for an `errLeaseCleaning`, beside its `LEASE HELD` line for an `errLeaseHeld`. |
| - **The inventory test** (`go/ast` with `go/types` over every non-test file under `./internal/...`): every call of `activator.HibernatePrefix` and `(activator.Config).hibernatePrefix` is in the test's list with its class: `reader`, `lease-gated writer`, `tombstone-gated writer`, `build-state only`, `delete only`, `a move into the region (hostname key)` or `excluded: <reason>`, as the Global Constraint lists them; a new site, or a removed one, fails until the list is edited. |
| -- [ ] **Step 1: Write the failing tests**: `TestMarkOnlyFromThisMovesLiftedTombstone` (lifted tombstone of this move → marked, the body's `ExpiresAt.Equal(lease.CleaningExpiry)` and its generation kept; of another move, live tombstone, an ordinary lease, absent → `errors.Is(err, lease.ErrNotOurLock)`, nothing written); `TestMarkRacesAnAcquire` (an `Acquire` landing between the read and the PUT → the mark's condition fails); `TestCleaningMarkIsRefusedWhateverItsAge` (the mark written, then the clock advanced 25 h, and separately its `ExpiresAt` rewritten by hand to `2000-01-01`: `Acquire` → `errors.As(err, &held) && held.Owner == lease.CleaningOwner(id)` and the lock's ETag unchanged; `AcquireTTL` with the staging TTL and `ReplicateWithLease` → the same; `Tombstone` → `*HeldError`, nothing written; `LiftTombstone` with the move's `mine` → `lifted == false`, nothing written; `LockState` → `held == true`); `TestMarkIsNeverRewritten` (a second `MarkCleaning` by the same owner writes nothing: the ETag is unchanged; the package has no exported function but `UnmarkCleaning` that writes a `cleaning:` lock, checked by `go/ast` over `internal/lease`); `TestUnmarkRestoresTheLiftedTombstone` (`IsLiftedTombstone(owner, expires)` true afterwards, the generation kept); `TestLandingWaitsOnACleaningLockWhateverItsExpiry` (the landing's lease step with a `cleaning:` lock whose expiry was edited to the past → `landWait`, never `ok`); `TestHibernatePrefixSitesAreInventoried`. |
| -- [ ] **Step 2: Run to verify they fail**: `go test ./internal/lease -run 'Cleaning|Mark|Unmark' -count=1`, `go test ./internal/host -run 'HibernatePrefixSites|LandingWaitsOnACleaning' -count=1`. |
| +- [ ] **Step 1: Write the failing tests**: `TestMarkOnlyFromThisMovesLiftedTombstone` (lifted tombstone of this move → marked, the body's `ExpiresAt.Equal(lease.CleaningExpiry)` and its generation kept; of another move, live tombstone, an ordinary lease, absent → `errors.Is(err, lease.ErrNotOurLock)`, nothing written); `TestMarkRacesAnAcquire` (an `Acquire` landing between the read and the PUT → the mark's condition fails); `TestCleaningMarkIsRefusedWhateverItsAge` (the mark written, then the clock advanced 25 h, and separately its `ExpiresAt` rewritten by hand to `2000-01-01`: `Acquire` → `errors.As(err, &held) && held.Owner == lease.CleaningOwner(id)` and the lock's ETag unchanged; `AcquireTTL` with the staging TTL and `ReplicateWithLease` → the same; `Tombstone` → `*HeldError`, nothing written; `LiftTombstone` with the move's `mine` → `lifted == false`, nothing written; `LockState` → `held == true`); `TestMarkIsNeverRewritten` (a second `MarkCleaning` by the same owner writes nothing: the ETag is unchanged; the package has no exported function but `UnmarkCleaning` that writes a `cleaning:` lock, checked by `go/ast` over `internal/lease`); `TestUnmarkRestoresTheLiftedTombstone` (`IsLiftedTombstone(owner, expires)` true afterwards, the generation kept); `TestLandingWaitsOnACleaningLockWhateverItsExpiry` (the landing's lease step with a `cleaning:` lock whose expiry was edited to the past → `landWait`, never `ok`); `TestCleaningMarkAnswersAFiniteRetry` (review 7's finding 7, at the response: the mark written by `MarkCleaning` over the lease fake with the sentinel expiry; `_, err := backend.Acquire(ctx, rt)` on an `ExecBackend` over that fake, then `a.serveWakeError(rec, req, rt, err)`: `rec.Code == http.StatusServiceUnavailable`, `rec.Header().Get("Retry-After") == "30"`, and the body's first line `== "instance is being cleaned up after a move. retry shortly"`; the same with the mark's expiry edited by hand to `2000-01-01`; an ordinary lease of another box expiring in 40 s → `Retry-After == "41"` and "instance is waking elsewhere; retry shortly", as today); `TestDeliverViaHandlerMapsWakeRefusals` gains the case `{503, "instance is being cleaned up after a move. retry shortly\n", 0, "instance is being cleaned up"}`; `TestMovesStringsAreRecorded` and `TestMovesCopyRecordIsComplete` green with the row pending; `TestHibernatePrefixSitesAreInventoried`. |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/lease -run 'Cleaning|Mark|Unmark' -count=1`, `go test ./internal/host -run 'HibernatePrefixSites|LandingWaitsOnACleaning|DeliverViaHandlerMapsWakeRefusals' -count=1`, `go test ./internal/activator -run 'CleaningMarkAnswers' -count=1`, `go test . -run 'MovesStringsAreRecorded|MovesCopyRecord' -count=1`. |
| - [ ] **Step 3: Implement. Step 4: Run to verify they pass**, plus `go test ./internal/lease -run 'Tombstone|Acquire|LockState' -count=1` (the existing tombstone and lock tests green). |
| -- [ ] **Step 5: Commit** `lease: the cleaning mark, a state with no expiry written only over this move's lifted tombstone and put back exactly, refused by every acquirer whatever its age; every hibernate-prefix site inventoried`. |
| +- [ ] **Step 5: Commit** `lease: the cleaning mark, a state with no expiry written only over this move's lifted tombstone and put back exactly, refused by every acquirer whatever its age, and answered to a wake with a 30-second retry; every hibernate-prefix site inventoried`. |
| ### Task 4a: The persisted lifecycle and its recovery |
| @@ -225,16 +231,16 @@ Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and |
| ### Task 4d: Scheduling and renewal, outside the pass lock |
| **Files:** |
| -- Modify: `internal/host/movespass.go` (`movesWorker` gains `cleanupMu sync.Mutex`; `moveCfg` gains `CleanupArmed bool`, set in `armOffline` from `CARLOS_OFFLINE_CLEANUP`; `movesPass` ends with `w.kickCleanup(ctx, s)`), `internal/host/movescleanup.go` (`kickCleanup`, `cleanupRun`, the reassert) |
| +- Modify: `internal/host/movespass.go` (`movesWorker` gains `cleanupMu sync.Mutex` and `tableWrites moves.TableWrites`; `moveCfg` gains `CleanupArmed bool`, set in `armOffline` from `CARLOS_OFFLINE_CLEANUP`; `movesPass` ends with `w.kickCleanup(ctx, s)`), `internal/host/movescleanup.go` (`kickCleanup`, `cleanupRun`, the reassert) |
| - Test: `internal/host/movescleanup_run_test.go` |
| **Interfaces:** |
| - `func (w *movesWorker) kickCleanup(ctx context.Context, s release.Store)`: nothing unless `cfg.OfflineCapable && cfg.Judge != nil`; `w.cleanupMu.TryLock()` false → return; else a goroutine runs `w.cleanupRun(runCtx, s)` with `runCtx, cancel := context.WithTimeout(ctx, cleanupRunDeadline)` and unlocks when it returns. **Locks:** the goroutine holds `cleanupMu` only, never `w.pass` and never `w.mu` across I/O; `movesPass` holds `w.pass` while it only try-locks `cleanupMu`, so neither waits on the other and `offlineReassert` (`movesoffline.go:795`, which takes `w.pass`) is never behind a cleanup. |
| -- `cleanupRun`: lists `<dataDir>/.offline-moves/*/*/retired.json` and the corrupt stubs, starts after the path recorded in `<dataDir>/.offline-moves/.cleanup-cursor`, calls `cleanupVisit` on each obligation in path order until the deadline (each read, list, delete, lock call and report write under `context.WithTimeout(runCtx, 10*time.Second)`), at most 50 keys per obligation, records the cursor after each obligation, then runs publication (Task 5a) and the reaper (Task 8). One `cfg.Versioned` store per bucket per run. |
| +- `cleanupRun`: first `w.tableWrites.Resend(ctx, s)` (U3: a table write a visit left unanswered is resent, identically, before anything else); then lists `<dataDir>/.offline-moves/*/*/retired.json` and the corrupt stubs, starts after the path recorded in `<dataDir>/.offline-moves/.cleanup-cursor`, calls `cleanupVisit` on each obligation in path order until the deadline (each read, list, delete, lock call and report write under `context.WithTimeout(runCtx, 10*time.Second)`), at most 50 keys per obligation, records the cursor after each obligation, then runs publication (Task 5a) and the reaper (Task 8). One `cfg.Versioned` store per bucket per run. |
| - **No renewal (T1).** The mark has no expiry, so no visit rewrites it: from `MarkCleaning` to `UnmarkCleaning` the lock object is not written by the cleanup, whether it waits on root, on a cancel acknowledgement or on an unanswered delete, and however long the box is down between runs. A foreign owner found by a visit (condition (2), `settleInFlight`, Task 6a's authority) is a hand edit and is handled where it is read. |
| -- [ ] **Step 1: Write the failing tests**: `TestCleanupRunsTwentyFourHoursAfterTheObservedFinish` (at 23 h 59 m nothing; at 24 h reserved, marked, deleted); `TestCleanupUnarmedLeavesObligationsFinished`; `TestCleanupRunsOutsideThePassLock` (a versioned fake that blocks: `offlineReassert` returns while the run is blocked; the run stops at its deadline; the cursor moves on; a second kick while one runs starts nothing); `TestMarkIsNeverRewrittenWhileWaiting` (root unanswered for 3 h, and a delete unanswered across runs and a 25 h stop: the lock's ETag is the one `MarkCleaning` left until `UnmarkCleaning`, `fake.LockPuts(location) == 2`); `TestRunBudgetAndCursor` (120 keys: 50 per run; three obligations and a deadline: each visited in turn across runs). |
| -- [ ] **Step 2: Run to verify they fail**: `go test ./internal/host -run 'CleanupRuns|CleanupUnarmed|MarkIsNeverRewritten|RunBudget' -count=1`. |
| +- [ ] **Step 1: Write the failing tests**: `TestCleanupRunsTwentyFourHoursAfterTheObservedFinish` (at 23 h 59 m nothing; at 24 h reserved, marked, deleted); `TestCleanupUnarmedLeavesObligationsFinished`; `TestCleanupRunsOutsideThePassLock` (a versioned fake that blocks: `offlineReassert` returns while the run is blocked; the run stops at its deadline; the cursor moves on; a second kick while one runs starts nothing); `TestMarkIsNeverRewrittenWhileWaiting` (root unanswered for 3 h, and a delete unanswered across runs and a 25 h stop: the lock's ETag is the one `MarkCleaning` left until `UnmarkCleaning`, `fake.LockPuts(location) == 2`); `TestRunBudgetAndCursor` (120 keys: 50 per run; three obligations and a deadline: each visited in turn across runs); `TestCleanupRunResendsAnUnansweredTableWrite` (a `Reserve` whose attempts all time out leaves `ReserveUncertain` and one registered write; the next run's first request is that write's identical bytes and condition, before any listing). |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/host -run 'CleanupRuns|CleanupUnarmed|MarkIsNeverRewritten|RunBudget|CleanupRunResends' -count=1`. |
| - [ ] **Step 3: Implement. Step 4: Run to verify they pass.** |
| - [ ] **Step 5: Commit** `moves: the cleanup runs in its own goroutine under its own lock, with a run budget, a cursor and a mark it never rewrites`. |
| @@ -292,14 +298,15 @@ Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and |
| ### Task 6b: The root-owned request and acknowledgement, as a stage of the adopt pass |
| **Files:** |
| -- Create: `internal/release/rootcleanup.go` (discovery, the copy, the dispatch, the acknowledgement, `LockRootPass`, `RootCleanupGated`; portable), `internal/host/adoptrootgate.go` (`adoptRootGate`), `internal/release/rootcleanup_other.go` (answers every request `cancelled` with `Reason: "unsupported platform"`; `LockRootPass` answers `ErrNotRootOnly`), `internal/host/movescleanup_root.go` (the agent's rows, `rootCleanupPass`) |
| +- Create: `internal/release/rootcleanup.go` (discovery, the copy, the dispatch, the acknowledgement, `LockRootPass`, `RootCleanupGated`, `QuiesceRootCleanup`; portable), `internal/release/adoptpasses_linux.go` and `adoptpasses_other.go` (`AdoptPasses`), `internal/host/adoptrootgate.go` (`adoptRootGate`), `internal/release/rootcleanup_other.go` (answers every request `cancelled` with `Reason: "unsupported platform"`; `LockRootPass` answers `ErrNotRootOnly`), `internal/host/movescleanup_root.go` (the agent's rows, `rootCleanupPass`) |
| - Modify: `internal/host/agent.go` (`runAdoptUnderRootRelease`: the root pass lock and the gate right after `lockAdoptPass` at line 4655, in every mode, before `ensureUnits` at line 4664; `rootCleanupPass(...)` after `heldStagePass` at line 4932), `internal/host/adoptlock.go` (the lock-order comment at lines 52-54 names the root pass lock between the adopt-pass lock and the restart marks), `internal/host/movesheldlock.go` (the order comment, lines 26-31), `internal/host/claimcensus.go` (`claimOptCensus` gains `<optDir>/root-cleanup` as box state, if its completeness test requires it) |
| -- Test: `internal/release/rootcleanup_test.go`, `internal/host/movescleanup_root_test.go`, `internal/host/adopt_rootcleanup_stage_test.go`, `internal/host/adoptrootgate_test.go` |
| +- Test: `internal/release/rootcleanup_test.go`, `internal/release/adoptpasses_linux_test.go`, `internal/host/movescleanup_root_test.go`, `internal/host/adopt_rootcleanup_stage_test.go`, `internal/host/adoptrootgate_test.go` |
| **Interfaces:** |
| - **The root pass lock (R5, F5; T7).** `func LockRootPass(optDir string) (unlock func(), err error)`, in this order, so no directory is created under an ancestry that has not passed: (1) `offlinecleanup.RootOnlyChain(optDir)` on the **existing** chain from `/` to `optDir`: false → `ErrNotRootOnly`, nothing created; an error → itself; (2) `RootCleanupBase(optDir) = <optDir>/root-cleanup/` created by root `0755` when absent (`os.Mkdir`; `EEXIST` is fine); (3) `RootOnlyChain` on the base (a base another uid made, or a symlink, answers false → `ErrNotRootOnly`); (4) `pass.lock` in it opened with `safefile.OpenFile(O_CREATE|O_RDWR, 0o600)` and `lockFileEx` (blocking `flock`), as `lockExclusive` does (`adoptlock.go:55-68`). `ErrNotRootOnly` is a definite answer about the box, not a failure to lock: only root can make a chain root-only or not, so every pass gets the same answer, and root's step can never run there (a writable `/opt` is a writable layout). **Why the step does not depend on the adopt-pass lock's identity:** that lock lives in the agent-writable `/data` (`adoptlock.go:16`), where a shared-uid writer can replace or plant its name (`safefile` then refuses it, and the pass continues unserialized, `agent.go:4655-4661`); the root pass lock lives on a root-only chain, which no other uid can change, so its acquisition needs no identity check. The adopt-pass lock is still required for the step (it is held, so the sweep keeps its own rule), but nothing relies on its identity. |
| -- **The gate (T2).** In `runAdoptUnderRootRelease`, right after `lockAdoptPass` (`agent.go:4655-4661`) and whatever that answered, in every mode (hourly, `-stage-only`, `-config-only`, `-restart-only`), before `ensureUnits` (`:4664`) and every later stage: `unlockRoot, rootLockErr := release.LockRootPass(release.OptDir())` (deferred when it succeeds); `gated := os.Getenv("CARLOS_OFFLINE_CLEANUP") == "on" || release.RootCleanupGated(release.OptDir())`; then `adoptRootGate(gated, rootLockErr)` answers one of three. **Serve** (`rootLockErr == nil`): the pass runs as today; when `gated`, it creates `<optDir>/root-cleanup/gated` (create-only, root `0644`) if absent and runs root's step as its last stage. **Publish nothing** (`gated`, and `rootLockErr` neither nil nor `ErrNotRootOnly`): one `log.Printf("ALERT: adopt: root pass lock %s: %v: publishing nothing this pass", ...)` and `return nil`, so the pass runs none of `ensureUnits`, `recoverRootRestarts` (`:4695`; its resumes call `PointLive` at `rootrollback_resume.go:125` and `roothistory_recovery.go:216`), `rootJudgePrev` (`:4698`), `sweepStagedVersionsPass` (`:4720`), the `-restart-only` branch (`:4758`), config sync and the `-config-only` branch (`:4794`), `capabilityPass` (`:4837`), `linkRootHistory` (`:4888`), `release.Pass` (`:4920`: adoption and rollback, `adopt.go:1751`, `:1870`, `rollback.go:124`, and named recovery inside it) and `heldStagePass` (`:4932`: `materializeheld.go:460`, `:494`); the next pass tries again. **Publish without serving** (not `gated`, or `ErrNotRootOnly`): today's fail-open adoption, logged as the adopt-pass lock's failure is, and no root step. `func RootCleanupGated(optDir string) bool`: `Lstat` of `<optDir>/root-cleanup/gated`: present → true; `ENOENT` → false; any other error → true (fail closed). The marker is never removed by code: it records that cleanup was armed on this box, so a box disarmed for a rollback, whose obligations may still wait for root (Global Constraint, rollback step 1), keeps serving them under the same rule. **Why this closes R5:** on a gated box every pass that publishes holds the root pass lock, and root's step runs only as the last stage of a pass holding it, so no root publisher of the live link or the programs file runs while root's step runs; the agent's own publishers are excluded by the freeze and the held-stage lock as before. On a box never gated no root step can run (it needs the lock and the switch or the marker), so fail-open adoption there races nothing. The one case left is in the Self-review notes. |
| -- **The stage (S12).** `rootCleanupPass(...)` runs after `heldStagePass` (line 4932), so after `release.Pass`, inside which named recovery runs (`recovery.passOption`, `agent.go:4890`, `namedrecovery_pass.go:15`), and only when `passLockErr == nil && rootLockErr == nil && gated`; nonfatal. `-config-only` and `-restart-only` return before it (`agent.go:4758-4767`, `:4794-4801`) but have taken the root pass lock, so they wait for a running step and a running step waits for them. Inside one pass the stages run one after another, so no root publisher of this pass runs while root's step runs, and every other adopt pass is blocked on the root pass lock. |
| +- **The gate (T2).** In `runAdoptUnderRootRelease`, right after `lockAdoptPass` (`agent.go:4655-4661`) and whatever that answered, in every mode (hourly, `-stage-only`, `-config-only`, `-restart-only`), before `ensureUnits` (`:4664`) and every later stage: `unlockRoot, rootLockErr := release.LockRootPass(release.OptDir())` (deferred when it succeeds); `gated := os.Getenv("CARLOS_OFFLINE_CLEANUP") == "on" || release.RootCleanupGated(release.OptDir())`; then `adoptRootGate(gated, rootLockErr)` answers one of three. **Serve** (`rootLockErr == nil`): the pass runs as today; when `gated`, it creates `<optDir>/root-cleanup/gated` (create-only, root `0644`) if absent, and as its last stage runs `QuiesceRootCleanup` and then, only when that answers true, root's step (the arming boundary below). **Publish nothing** (`gated`, and `rootLockErr` neither nil nor `ErrNotRootOnly`): one `log.Printf("ALERT: adopt: root pass lock %s: %v: publishing nothing this pass", ...)` and `return nil`, so the pass runs none of `ensureUnits`, `recoverRootRestarts` (`:4695`; its resumes call `PointLive` at `rootrollback_resume.go:125` and `roothistory_recovery.go:216`), `rootJudgePrev` (`:4698`), `sweepStagedVersionsPass` (`:4720`), the `-restart-only` branch (`:4758`), config sync and the `-config-only` branch (`:4794`), `capabilityPass` (`:4837`), `linkRootHistory` (`:4888`), `release.Pass` (`:4920`: adoption and rollback, `adopt.go:1751`, `:1870`, `rollback.go:124`, and named recovery inside it) and `heldStagePass` (`:4932`: `materializeheld.go:460`, `:494`); the next pass tries again. **Publish without serving** (not `gated`, or `ErrNotRootOnly`): today's fail-open adoption, logged as the adopt-pass lock's failure is, and no root step. `func RootCleanupGated(optDir string) bool`: `Lstat` of `<optDir>/root-cleanup/gated`: present → true; `ENOENT` → false; any other error → true (fail closed). The marker is never removed by code: it records that cleanup was armed on this box, so a box disarmed for a rollback, whose obligations may still wait for root (Global Constraint, rollback step 1), keeps serving them under the same rule. **Why this closes R5:** on a gated box every pass that publishes holds the root pass lock, and root's step runs only as the last stage of a pass holding it, so no root publisher of the live link or the programs file runs while root's step runs; the agent's own publishers are excluded by the freeze and the held-stage lock as before. On a box never gated no root step can run (it needs the lock and the switch or the marker), so fail-open adoption there races nothing. The moment a box first becomes gated is closed by the arming boundary. |
| +- **The arming boundary (U1; review 7's finding 2).** The gate leaves one interleaving: a pass that read the switch off and found no marker, and whose two locks failed (a planted `/data/adopt-pass.lock` and a root-level fault on `LockRootPass`), publishes fail-open; if it is still running when a later pass has armed the box and runs root's step, a publication can land between root's `fstatat` and `unlinkat`. Setting the switch or creating the marker does not stop a pass already past its gate. **How passes run, verified:** `carlos adopt` runs one pass per process and exits (`runAdopt`, `agent.go:4568`); it is started by four root oneshot units, `carlos-adopt.service` (`carlos-adopt.timer`, hourly), `carlos-stage.service` (`carlos-stage.path`, the agent's `/data/stage-requested`), `carlos-config-sync.service` (`carlos-adopt.path`, `/data/config-sync-requested`) and `carlos-restart.service` (`carlos-restart.path`, `/data/restart-requested`), each a separate process, and passes of different units can overlap ("the four units that exec this verb can start in the same second", `agent.go:4645-4648`). Within one unit nothing overlaps: systemd runs one instance of a unit, a start request for a oneshot unit that is still activating joins its job, and a unit is inactive again only once its control group is empty (the adopt units keep the default `KillMode=control-group`), so a restarted pass's predecessor has exited and its kernel operations have completed before the new one runs. An in-process mutex would therefore serialise nothing (no process runs two passes), and none is added; the boundary is a census of processes instead. `func AdoptPasses(procDir string, self int) ([]int, error)` (Linux): every numeric entry of `procDir` (`/proc` in production; the adopt units set neither `ProtectProc=` nor `PrivatePIDs=`, unlike `carlos-relay.service:32` and `carlos-instance-sidecar@.service:117-119`, so a root pass sees every process) whose `status` has real uid 0 and whose `cmdline`, split on NUL, has `filepath.Base(argv[0])` equal to `carlos` or beginning `carlos.` and `argv[1] == "adopt"`, other than `self`; an entry that vanishes while it is read (`ENOENT`, `ESRCH`) has exited and is skipped; any other error is returned, and an unknown census is never an empty one. A process of another uid cannot be counted, so the agent's uid cannot hold the boundary shut. Off Linux it answers `ErrNotRootOnly`. `func QuiesceRootCleanup(optDir string, census func() ([]int, error)) (bool, error)`, called only by a pass holding both locks with `gated` true, after every other stage: (1) `Lstat` of `<optDir>/root-cleanup/gated`: not a regular file owned by uid 0 → false; (2) `<optDir>/root-cleanup/quiesced` present and recording that marker's `(dev, inode)` → true; present and recording another identity (the operator removed `gated` and a later pass made a new one) → removed, and on to (3); (3) the census: an error → false with it; any pid → false, one `log.Printf("root-cleanup: waiting for adopt pass %v, which may predate arming, to end before the first cleanup step", pids)`; none → `quiesced` written create-only (temp, `fsync`, `os.Link`, directory `fsync`) as `{"gated_dev": D, "gated_ino": I, "at": <now>}`, → true. **Why it closes the first arming:** a pass that publishes without the root pass lock is one whose gate answered "publish without serving", so it read the switch off and found no `gated` marker; `quiesced` is written only after `gated` existed, at a moment the census found no other root adopt process alive; every adopt process alive after that moment was started after it, so after `gated` existed, and reads the marker (`RootCleanupGated` fails closed on any `Lstat` error but `ENOENT`), so it holds the root pass lock or publishes nothing. A `/proc` walk lists every process that exists for the whole walk, and one started during the walk started after `gated` existed. So once `quiesced` stands, no pass admitted before arming is still running, and root's step, which requires it, never meets one. A census that keeps finding a process (a pass blocked behind this one's lock counts too) costs only time: the next pass, at the latest the next hour, tries again, and the obligation pages `root-wait` meanwhile. |
| +- **The stage (S12).** `rootCleanupPass(...)` runs after `heldStagePass` (line 4932), so after `release.Pass`, inside which named recovery runs (`recovery.passOption`, `agent.go:4890`, `namedrecovery_pass.go:15`), and only when `passLockErr == nil && rootLockErr == nil && gated` and `QuiesceRootCleanup(release.OptDir(), func() ([]int, error) { return release.AdoptPasses("/proc", os.Getpid()) })` answered true; nonfatal. `-config-only` and `-restart-only` return before it (`agent.go:4758-4767`, `:4794-4801`) but have taken the root pass lock, so they wait for a running step and a running step waits for them. Inside one pass the stages run one after another, so no root publisher of this pass runs while root's step runs, and every other adopt pass is blocked on the root pass lock. |
| - **Files.** The agent's drop `<OfflineOpDir>/root-cleanup.json` (create-only) and cancel `<OfflineOpDir>/root-cleanup.cancel` (create-only). Root's directory `release.RootCleanupDir(optDir, id) = <optDir>/root-cleanup/<id>/` holding `request.json` (root's copy), `journal` (Task 6c) and exactly one of `done.json`, `cancelled.json`. Each `<id>/` is created by root `0755` only after `RootOnlyChain` has passed on the base (T7: the existing ancestry is checked before any missing directory is created), and used only when the chain passes on it too. **When it does not (S16), root writes nothing there**, removes nothing, and leaves the drop where it is: it logs `root-cleanup: <id> refused: <dir> is not root-only; repair its ownership, or remove <drop> to give up` to the adopt unit's journal (`log.Printf`, the systemd journal of `carlos-adopt.service`), which is the only record of the refusal. A repaired ancestry lets the next pass take the request. |
| - The wire types (declared in Task 1): `type RootCleanupRequest struct { V int; CleanupID, MoveID, Host, Account, App, Box string; Incarnation uint64; From, To string; Bucket, BucketRegion, Prefix string; Paths []RootCleanupPath; WrittenAt time.Time }`; `type RootCleanupPath struct { Which string; Identity FileIdentity }`; `type RootCleanupAck struct { V int; CleanupID, MoveID, Host, Outcome string (done or cancelled); Removed, Absent []string; Kept []KeptPath; Reason string; At time.Time }`. |
| - `type RootCleanupOptions struct { DataDir, OptDir, Box, Region string; Authorize func(ctx context.Context, req offlinecleanup.RootCleanupRequest) error; Freeze func(host string, fn func(rowPresent bool) error) error; HostLock func(host string, fn func() error) error; Processes func() ([]string, error); Now func() time.Time }`; `var ErrRootCleanupRefused`, `ErrNotRootOnly`; `func ServeRootCleanupRequests(ctx context.Context, o RootCleanupOptions) error`. |
| @@ -310,10 +317,10 @@ Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and |
| - `func rootCleanupPass(ctx context.Context, dataDir, optDir string, reg *registry.Registry, store release.Store, judge *provision.LifetimeJudge, box, region string)`: `ServeRootCleanupRequests` with `Authorize` = Task 6a's callback, `Freeze` = `reg.WithHostFrozen` (a non-nil row is `rowPresent`), `HostLock` = `withHeldStageLock(dataDir, ·, ·)`, `Processes` reading `/proc/*/exe` (`ENOENT`/`ESRCH` skipped as exited, any other error returned). |
| - **The agent's rows** (Task 4a's row 6): `publishing`: `supersededBy` → supersede; the drop absent and `RootCleanupDir` absent → `writeRootCleanupRequest` (create-only; an existing drop with equal bytes is fine) → touch the stage-request marker (`holdStageRequester(dataDir)(Host)`, `moveshold.go:277`) → `RootRequestedAt = now`, `root-requested`, event; either present (a crash after the drop) → `root-requested` with `RootRequestedAt` restored, in this order: the drop's mtime when the drop exists; else the `at` of root's journal `taken` line; else `now`. `root-requested`: `supersededBy` → supersede (to `cancel-pending`); `ReadRootCleanupAck(optDir, ID)` → an acknowledgement whose `CleanupID`, `Host`, `MoveID` match → `Root = ack`, event, `releasing` with the outcome table's answer; a mismatch → page `corrupt-<h>`, stay; none → the drop is read **first** and then `RootCleanupDir`: both absent (an operator removed the drop before root took it; root takes a request into its directory before it removes the drop) → `releasing`, `Outcome = superseded`; else, after `rootRefusedAfter` (1 h) since `RootRequestedAt` with the agent's own `RootOnlyChain` check of `RootCleanupBase` (or of `RootCleanupDir` when it exists) failing → event `root-refused` and the hourly `root-refused-<h>` page, stay (**never a release**; the mark, which has no expiry, and the entries stay until an operator repairs the ancestry or removes the drop); else the `root-wait` pages. `cancel-pending`: the cancel file written create-only and `CancelRequestedAt` set (once); an acknowledgement → `releasing`, `Outcome = superseded`; else the drop is read **first** and then `RootCleanupDir`: both absent → `releasing`, `Outcome = superseded` (root takes a request into its directory before it removes the drop, so a drop already gone means root's directory exists, and no drop was written after `publishing`); else the `cancel-wait` pages. |
| -- [ ] **Step 1: Write the failing tests**: `TestRootCleanupStepRunsInsideThePass` (a pass driven through `runAdoptUnderRootRelease`'s stages with fakes: the step runs after `release.Pass` and `heldStagePass`; with the adopt-pass lock failed, with the root pass lock failed, and on `-config-only` and `-restart-only`, no step; every mode takes the root pass lock before its first publisher); `TestRootPassLockIsRootOnly` (a base directory the test uid owns → `ErrNotRootOnly`, no lock; the lock's chain check runs as any uid against an injected chain check); `TestLockRootPassChecksAncestryBeforeCreating` (T7: an `optDir` whose existing chain fails → `errors.Is(err, release.ErrNotRootOnly)` and `os.Lstat(base)` answers `fs.ErrNotExist` afterwards; a root-only `optDir` without a base → created, checked, locked, in that order in the injected syscall log; a base that is a symlink → `ErrNotRootOnly`, nothing created through it); `TestGatedPassPublishesNothingWithoutTheRootPassLock` (T2: a pass driven through `runAdoptUnderRootRelease`'s stages with a recording fake per stage; the switch on and `LockRootPass` failing with an injected `EIO` → `len(calls) == 0` across `ensureUnits`, `recoverRootRestarts`, `rootJudgePrev`, `sweepStagedVersionsPass`, the restart pass, config sync, `capabilityPass`, `linkRootHistory`, `release.Pass`, `heldStagePass` and the root step, in each of the four modes, and one log line; the switch off with the marker present → the same; the next pass with the lock → every stage in today's order and the step last); `TestUngatedBoxKeepsFailOpenAdoption` (the switch off and no marker, the lock failing → every stage runs as today, no step, no marker created; `ErrNotRootOnly` with the switch on → the same); `TestArmedPassLeavesTheGatedMarker` (an armed pass holding the lock creates the marker; a later pass with the switch off is gated; a marker `Lstat` error other than `ENOENT` → gated); `TestRootFindsItsOwnRequestsOnRestart` (root takes a request, removes the drop and stops: the next pass serves it from `request.json`); `TestForgedAgentFilesAreHarmless` (a `done.json` planted in the agent's directory is never read; a drop with another id is cancelled under the expected id; a second drop with other bytes after the take is removed and changes nothing; a forged cancel stops root and ends the obligation `cancelled`, nothing else); `TestRejectedAncestryGetsNoWrite` (a `RootCleanupBase` or `<id>/` that fails the chain check: nothing created, written or removed under it, the drop kept, one log line; repaired, the next pass takes it); `TestAgentPagesARefusedAncestryAndNeverReleases` (no answer for 1 h with the chain failing from the agent's view: hourly `root-refused` pages, the obligation stays `root-requested` with both entries and the mark for a simulated week; the drop removed by an operator → `releasing`, `superseded`); `TestAgentReadsOnlyRootsAcknowledgement` (an untrusted acknowledgement → `superseded`, paths kept "writable layout"); `TestRootRequestedAtIsRestored` (a crash after the drop: from its mtime; after the take: from the `taken` line); `TestUnansweredRootRequestPagesEveryTenMinutes`; `TestCancelBeforeAndDuringRootsWork`; `TestCancelPendingResolvesWithoutARequest` (a crash after `publishing` before the drop; root never wrote a directory); `TestUnsupportedPlatformCancels`. |
| -- [ ] **Step 2: Run to verify they fail**: `go test ./internal/release -run 'RootFinds|ForgedAgent|RejectedAncestry|RootPassLock|LockRootPassChecks|CancelBefore|Unsupported' -count=1`, `go test ./internal/host -run 'RootCleanupStep|GatedPass|UngatedBox|ArmedPassLeaves|AgentPages|AgentReadsOnly|RootRequestedAt|Unanswered|CancelPending' -count=1`. |
| +- [ ] **Step 1: Write the failing tests**: `TestRootCleanupStepRunsInsideThePass` (a pass driven through `runAdoptUnderRootRelease`'s stages with fakes: the step runs after `release.Pass` and `heldStagePass`; with the adopt-pass lock failed, with the root pass lock failed, and on `-config-only` and `-restart-only`, no step; every mode takes the root pass lock before its first publisher); `TestRootPassLockIsRootOnly` (a base directory the test uid owns → `ErrNotRootOnly`, no lock; the lock's chain check runs as any uid against an injected chain check); `TestLockRootPassChecksAncestryBeforeCreating` (T7: an `optDir` whose existing chain fails → `errors.Is(err, release.ErrNotRootOnly)` and `os.Lstat(base)` answers `fs.ErrNotExist` afterwards; a root-only `optDir` without a base → created, checked, locked, in that order in the injected syscall log; a base that is a symlink → `ErrNotRootOnly`, nothing created through it); `TestGatedPassPublishesNothingWithoutTheRootPassLock` (T2: a pass driven through `runAdoptUnderRootRelease`'s stages with a recording fake per stage; the switch on and `LockRootPass` failing with an injected `EIO` → `len(calls) == 0` across `ensureUnits`, `recoverRootRestarts`, `rootJudgePrev`, `sweepStagedVersionsPass`, the restart pass, config sync, `capabilityPass`, `linkRootHistory`, `release.Pass`, `heldStagePass` and the root step, in each of the four modes, and one log line; the switch off with the marker present → the same; the next pass with the lock → every stage in today's order and the step last); `TestUngatedBoxKeepsFailOpenAdoption` (the switch off and no marker, the lock failing → every stage runs as today, no step, no marker created; `ErrNotRootOnly` with the switch on → the same); `TestFirstArmingWaitsForAFailOpenPassToEnd` (U1's witness, review 7's finding 2 interleaving, through `runAdoptUnderRootRelease`'s stages with the production lock functions, an injected `LockRootPass` fault for one pass and an injected census: pass A, with the switch off, no marker, `/data/adopt-pass.lock` replaced by a symlink and `LockRootPass` failing with `EIO`, runs fail-open and is paused inside `release.Pass` after reading its route; the switch goes on and a drop is written; pass B takes both locks, creates `gated`, and its census answers A's pid: `QuiesceRootCleanup` answers false, `os.Lstat(quiesced)` answers `fs.ErrNotExist`, and root's step is not called, `len(stepCalls) == 0`; A resumes, publishes a new live link for the host and exits, and the census answers no pid; pass C writes `quiesced` and runs root's step, which finds the link A published, not the recorded one: `kept`, "identity changed", and `release.LiveLink(optDir, host)` still names A's target); `TestAdoptPassesCensus` (a fake `/proc` tree: a uid-0 `carlos adopt --stage-only` and a uid-0 `carlos.prev adopt` counted; `self`, a uid-1001 process whose cmdline reads `carlos adopt`, a uid-0 `carlos agent` and an entry removed during the walk not counted; an unreadable `status` → the error); `TestQuiescedIsBoundToTheGatedMarker` (a `quiesced` written for one `gated`, then `gated` removed and created again: the next call removes the stale `quiesced` and takes the census again; a `gated` that is a symlink → false); `TestArmedPassLeavesTheGatedMarker` (an armed pass holding the lock creates the marker; a later pass with the switch off is gated; a marker `Lstat` error other than `ENOENT` → gated); `TestRootFindsItsOwnRequestsOnRestart` (root takes a request, removes the drop and stops: the next pass serves it from `request.json`); `TestForgedAgentFilesAreHarmless` (a `done.json` planted in the agent's directory is never read; a drop with another id is cancelled under the expected id; a second drop with other bytes after the take is removed and changes nothing; a forged cancel stops root and ends the obligation `cancelled`, nothing else); `TestRejectedAncestryGetsNoWrite` (a `RootCleanupBase` or `<id>/` that fails the chain check: nothing created, written or removed under it, the drop kept, one log line; repaired, the next pass takes it); `TestAgentPagesARefusedAncestryAndNeverReleases` (no answer for 1 h with the chain failing from the agent's view: hourly `root-refused` pages, the obligation stays `root-requested` with both entries and the mark for a simulated week; the drop removed by an operator → `releasing`, `superseded`); `TestAgentReadsOnlyRootsAcknowledgement` (an untrusted acknowledgement → `superseded`, paths kept "writable layout"); `TestRootRequestedAtIsRestored` (a crash after the drop: from its mtime; after the take: from the `taken` line); `TestUnansweredRootRequestPagesEveryTenMinutes`; `TestCancelBeforeAndDuringRootsWork`; `TestCancelPendingResolvesWithoutARequest` (a crash after `publishing` before the drop; root never wrote a directory); `TestUnsupportedPlatformCancels`. |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/release -run 'RootFinds|ForgedAgent|RejectedAncestry|RootPassLock|LockRootPassChecks|CancelBefore|Unsupported' -count=1`, `go test ./internal/release -run 'AdoptPassesCensus|QuiescedIsBound' -count=1`, `go test ./internal/host -run 'RootCleanupStep|GatedPass|UngatedBox|ArmedPassLeaves|FirstArming|AgentPages|AgentReadsOnly|RootRequestedAt|Unanswered|CancelPending' -count=1`. |
| - [ ] **Step 3: Implement** (until Task 6c lands, the per-path step answers kept with reason "not built", so this commit removes nothing). **Step 4: Run to verify they pass**, then `GOOS=darwin go build ./internal/release/ ./internal/host/` and `GOOS=windows go vet ./internal/release/ ./internal/host/`. |
| -- [ ] **Step 5: Commit** `root: a stage of the adopt pass under a root-only pass lock, which an armed box's passes must hold to publish, takes a cleanup request into its own directory, writes nothing where the ancestry is not root's, and answers in a file the agent trusts only on a root-only chain`. |
| +- [ ] **Step 5: Commit** `root: a stage of the adopt pass under a root-only pass lock, which an armed box's passes must hold to publish, first runs once no adopt process that predates arming is alive, takes a cleanup request into its own directory, writes nothing where the ancestry is not root's, and answers in a file the agent trusts only on a root-only chain`. |
| ### Task 6c: Root's filesystem step, its journal and recovery |
| @@ -335,18 +342,19 @@ Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and |
| ### Task 7a: The create holds the hostname key |
| **Files:** |
| -- Modify: `internal/console/store.go` (`ALTER TABLE host_allocations ADD COLUMN host_key INTEGER NOT NULL DEFAULT 0`), `internal/console/lifetimes_store.go` (`BeginAllocation` inserts `host_key = 1`; `ClearAllocationHostKey(opID)`; `PendingAllocations` selects `state = 'pending' OR host_key = 1`; `Allocation.HostKey bool`), `internal/console/lifetimes_create.go` (the reservation between `BeginAllocation` and `allocSlotLocked`, line 103), `internal/console/lifetimes_recovery.go` (`recoverAllocationLocked`, line 152) |
| +- Modify: `internal/console/store.go` (`ALTER TABLE host_allocations ADD COLUMN host_key INTEGER NOT NULL DEFAULT 0`), `internal/console/lifetimes_store.go` (`BeginAllocation` inserts `host_key = 1`; `ClearAllocationHostKey(opID)`; `PendingAllocations` selects `state = 'pending' OR host_key = 1`; `Allocation.HostKey bool`, scanned in `allocations`, line 149; `Allocation(opID) (Allocation, bool, error)`, one row by its op id with the same columns), `internal/console/lifetimes_create.go` (the reservation between `BeginAllocation` and `allocSlotLocked`, line 103), `internal/console/lifetimes_recovery.go` (`recoverLifetimes`' allocation loop, lines 73-84, and `recoverAllocationLocked`, line 152), `internal/console/export_test.go` (`recoveryListedHook`) |
| - Test: `internal/console/lifetimes_hostkey_test.go` |
| **Interfaces:** |
| - **The create**, after `BeginAllocation` (the row is the intent; `admit`'s `PendingForHost` is unchanged): `moves.Reserve(ctx, s.cfg.Releases, {create, a.OpID}, {HostKey(host): {create, a.OpID, host, acct, app, now}})`: held → step 4 as today; `ErrReserved` by a `cleanup` → `settleAllocation(a, lifetimeVoid, errHostUnderCleanup)` (409 "the previous instance at this hostname is still being cleaned up. try again later"); by another holder → void, `errLifetimeSettling`; `ErrUnreadable`, `ErrReservationMismatch` or uncertain → `errLifetimePending` (recovery owns the row). After every terminal state the create path or recovery writes, `s.releaseAllocationKey(a)` = `moves.Release(ctx, s.cfg.Releases, {create, OpID})` (a write, so its success settles an uncertain reservation `Put`) then `ClearAllocationHostKey(OpID)`; a failure leaves `host_key = 1`. |
| - **"Busy writes no reservation", qualified (F2):** step 2 of the create (`planCreateLocked`, `lifetimes_create.go:74`, `:162`) can retire an orphan slot (`retireSlotOrphanLocked`, `:215`) or clean a stale body (`retireStaleLocked`, `:205`) before `BeginAllocation` and before any reservation. That is the existing create protocol's own work, outside the reservation, and safe beside a cleanup: it acts only on a lifetime its slot no longer vouches for (an orphan slot has no record; a stale body is not the slot's), which a cleanup's condition (1) already judges `AwayMoved`, so that cleanup supersedes; and neither write mints a replica version or a registry row (plan 1's rules). A busy create therefore leaves nothing written by the reservation or by step 4 onward. |
| -- **Recovery** (`recoverAllocationLocked`): first branch, `a.State != pending && a.HostKey` → `releaseAllocationKey`, return; every branch that writes the slot, the stale body or the record first calls the same `moves.Reserve` (its own retry returns the held entry, or writes the entry again under a fresh ETag): `ErrReserved` with no slot of its own → void and release; `ErrReserved` with its own slot → stays pending, logged hourly; unreadable or uncertain → pending. A read that finds no entry of the create is never taken as settlement; only the release's successful write is. |
| +- **Recovery reloads its row (U2; review 7's finding 3).** Today `recoverLifetimes` lists the allocations (`s.store.PendingAllocations()`, `lifetimes_recovery.go:73`) before it takes any lock, and hands each listed value to `recoverAllocationLocked` under `withAppHost` (`:79`), which never reads the row again, so a create that voided the allocation between the listing and the locks leaves recovery acting on a stale `pending`. From this task the listed `Allocation` is only a name: `recoverAllocationLocked(ctx, listed)` begins, under the app and hostname locks and before `s.admit` (`:154`) and before any reservation or effect, with `a, found, err := s.store.Allocation(listed.OpID)`; an error → stays pending, logged hourly; not found → logged hourly, nothing done; `a.State != pending` → a terminal allocation only finishes its owed release: `a.HostKey` → `s.releaseAllocationKey(a)`, else nothing; return. Only a row still `pending` goes on, and every later step uses the reloaded `a`, never `listed`. `recoveryListedHook` (nil in production) runs between the listing and the first `withAppHost`, for the witness. The drain's step (1) (Task 7c) goes through the same function, so it reloads too. |
| +- **Recovery** (`recoverAllocationLocked`, after the reload): every branch that writes the slot, the stale body or the record first calls the same `moves.Reserve` (its own retry returns the held entry, or writes the entry again under a fresh ETag): `ErrReserved` with no slot of its own → void and release; `ErrReserved` with its own slot → stays pending, logged hourly; unreadable or uncertain → pending. A read that finds no entry of the create is never taken as settlement; only the release's successful write is. |
| -- [ ] **Step 1: Write the failing tests**: `TestCreateReservesTheHostnameKeyAfterBeginAllocation` (busy by cleanup → 409 cleanup copy, the row void, no slot, no record, no entry; busy by a move → settling; a crash after `BeginAllocation` and before `Reserve` → recovery reserves and continues, or voids when busy; a crash after `Reserve` → recovery continues under its own retry; a crash after the terminal state and before the release → the sweep releases and clears; `LastIncarnation` ignores the void row); `TestCreateUncertainReservationIsSettledByItsRelease` (review 4's F2 interleaving through the real create and recovery: the create's held `Put` delivered after the release fails its condition; `host_key` cleared only after the release's write); `TestPendingAllocationHoldsTheKeyUntilRecoveryEnds`; `TestCreateStepTwoIsOutsideTheReservation` (an orphan slot retired before a busy answer; a cleanup of that host, if any, supersedes); `TestHolderChecksPreserved` (today's create refusals byte for byte). |
| -- [ ] **Step 2: Run to verify they fail**: `go test ./internal/console -run 'HostnameKeyAfter|CreateUncertain|PendingAllocation|CreateStepTwo|HolderChecks' -count=1`. |
| +- [ ] **Step 1: Write the failing tests**: `TestCreateReservesTheHostnameKeyAfterBeginAllocation` (busy by cleanup → 409 cleanup copy, the row void, no slot, no record, no entry; busy by a move → settling; a crash after `BeginAllocation` and before `Reserve` → recovery reserves and continues, or voids when busy; a crash after `Reserve` → recovery continues under its own retry; a crash after the terminal state and before the release → the sweep releases and clears; `LastIncarnation` ignores the void row); `TestCreateUncertainReservationIsSettledByItsRelease` (review 4's F2 interleaving through the real create and recovery: the create's held `Put` delivered after the release fails its condition; `host_key` cleared only after the release's write); `TestPendingAllocationHoldsTheKeyUntilRecoveryEnds`; `TestCreateStepTwoIsOutsideTheReservation` (an orphan slot retired before a busy answer; a cleanup of that host, if any, supersedes); `TestHolderChecksPreserved` (today's create refusals byte for byte); `TestRecoveryReloadsTheAllocationUnderItsLocks` (U2's witness, review 7's finding 3 interleaving: a cleanup holds `host/shop.example.com`; a create persists allocation A; recovery lists A pending and `recoveryListedHook` pauses it before its locks; the create meets the cleanup's entry, voids A, releases (`host_key` cleared); the cleanup ends; the drain's round would answer `drained`; recovery resumes: no `PutTable` reaches the fake store after the create's release (`len(fake.PutsAfter(mark)) == 0`), `moves.ReadActive` holds no `create` entry, `provision.ReadHostIncarnation` answers the slot as before, and `s.store.Allocation(A)` is still `void`; and, separately, with the create's release failed so A is `void` with `host_key = 1`: the resumed recovery sends exactly one `Release` of `{create, A}`, clears the key, and writes nothing else). |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/console -run 'HostnameKeyAfter|CreateUncertain|PendingAllocation|CreateStepTwo|HolderChecks|RecoveryReloads' -count=1`. |
| - [ ] **Step 3: Implement. Step 4: Run to verify they pass**, plus `go test ./internal/console -run 'Lifetime|Allocation' -count=1`. |
| -- [ ] **Step 5: Commit** `console: the lifetime create holds its hostname in the reservation table, with the allocation row as its intent and a release that settles it`. |
| +- [ ] **Step 5: Commit** `console: the lifetime create holds its hostname in the reservation table, with the allocation row as its intent and a release that settles it; recovery reloads the row under its locks`. |
| ### Task 7b: Placement registers, then holds every instance's hostname key, under one deadline |
| @@ -369,19 +377,20 @@ Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and |
| ### Task 7c: The rollback drain: `carlos system reservations drain` |
| **Files:** |
| -- Create: `internal/console/reservations_drain.go` (`handleAPIReservationsDrain`, `s.admission`, `s.admitResidence`, `errConsoleDraining`), `system_reservations.go` (the CLI verb, in `system_lifetimes.go`'s form) |
| -- Modify: `internal/console/authcli.go` (`s.cliRoute("POST /api/cli/system/reservations/drain", svcrole.Set{}, s.handleAPIReservationsDrain)`, beside line 1024), `internal/console/lifetimes_create.go` (`BeginAllocation`, line 102, runs inside `s.admitResidence`), `internal/console/fleets.go` (the placement's `s.registerPlacement`, Task 7b, runs inside it), `internal/console/selfserve_door.go` (`AdmitSelfServeMove`, line 347, runs inside it), `system.go` (`reservations` in `systemVerb`'s list, line 53, `runSystem`'s switch and `systemUsage`), `docs/site/cli.md` (the verb line), the copy record and `approved_moves_copy_test.go` |
| -- Test: `internal/console/reservations_drain_test.go`, `system_reservations_test.go` |
| +- Create: `internal/console/reservations_drain.go` (`handleAPIReservationsDrain`, `s.admission`, `s.admitResidence`, `errConsoleDraining`, `tableAttemptWindow`, `(*Server).CloseTableWrites`), `system_reservations.go` (the CLI verb, in `system_lifetimes.go`'s form) |
| +- Modify: `internal/console/authcli.go` (`s.cliRoute("POST /api/cli/system/reservations/drain", svcrole.Set{}, s.handleAPIReservationsDrain)`, beside line 1024), `internal/console/lifetimes_create.go` (`BeginAllocation`, line 102, runs inside `s.admitResidence`), `internal/console/fleets.go` (the placement's `s.registerPlacement`, Task 7b, runs inside it), `internal/console/selfserve_door.go` (`AdmitSelfServeMove`, line 347, runs inside it), `system.go` (`reservations` in `systemVerb`'s list, line 53, `runSystem`'s switch and `systemUsage`), `cmd/console/main.go` (the `SIGTERM` handler around `httpSrv.Serve`, line 624, which today ends the process with `log.Fatal`), `docs/site/cli.md` (the verb line), the copy record and `approved_moves_copy_test.go` |
| +- Test: `internal/console/reservations_drain_test.go`, `internal/console/reservations_drain_window_test.go`, `system_reservations_test.go`, `cmd/console/main_test.go` |
| **Interfaces:** |
| - **The admission barrier (T3).** `s.admission` is `struct { mu sync.Mutex; draining bool }`, and `func (s *Server) admitResidence(fn func() error) error` takes `mu`, answers `errConsoleDraining` without running `fn` when `draining` is set, else runs `fn` and returns its error, then releases `mu`. The three admitters persist their admission **inside** it, so the check and the admit are one critical section: the create's `s.store.BeginAllocation(a)` (`lifetimes_create.go:102`, under the app lock as today), the placement's `s.registerPlacement(id)` (Task 7b, before its reservation), and the self-serve door's `s.store.AdmitSelfServeMove(row)` (`selfserve_door.go:347`). `errConsoleDraining` answers 503 "this console is being rolled back. try again in a few minutes"; nothing is written, because no row, registration or admission exists yet. `mu` is the innermost console lock: it is taken under the app lock where the create holds one, nothing is taken under it, and `fn` is one local SQLite write or one map insert. The drain sets `draining` with `mu` held, so every admission either committed before it, and the drain's reads below see its row or registration, or is refused. |
| -- **The door** (operator only: `s.moveOperator`, a 404 otherwise, and a 404 without `s.cfg.Releases`, as `handleAPILifetimesCutover`, `lifetimes_cutover.go:526-531`). Idempotent; each call is one round. The first call sets `s.admission.draining` for the rest of this process (a restart clears it, which is the way to abandon a rollback). The round waits on the work itself, not on table entries: (1) every row `s.store.PendingAllocations()` answers (pending, or terminal with `host_key = 1`, Task 7a), whether or not the table holds its entry: `recoverAllocationLocked` under the app lock (a create still running holds that lock, so recovery runs after it returns), which reserves, completes or voids, and releases the key; a row still pending afterwards is counted; (2) the ids registered in `s.placing`: counted, untouched (each settles itself, fence before release, then deregisters, Task 7b); (3) every `place` operation in the table whose id is not registered: settled now as the sweep settles it (fence every host key's record, then `moves.Release`), **never released before its fences answered**; (4) a `create` operation in the table with no allocation row: counted (the sweep pages it); (5) the counts: `cleanup` operations in the table, open self-serve moves (`s.store.OpenSelfServeMoves()`, `selfserve_store.go:562`), pending rows and `create` entries, registrations and `place` entries; (6) all zero → the settling write, `moves.Release(ctx, s.cfg.Releases, {Kind: "drain", ID: "drain"})` (it holds nothing and writes rev + 1), then the table, the allocation rows, `s.placing` and the open self-serve moves read again: still zero → `drained`; else `draining`. The answer: `{state: "draining" | "drained", cleanups, creates, placements, self_serve_moves int, rev uint64}`. |
| -- **The CLI** `carlos system reservations drain [--console <origin>] [--no-wait]`: `requireDeployAPISession`, then the door every 5 s (`waitLifetimesCutover`'s loop and retry rules: a failed poll or a 5xx is asked again, up to 5 in a row or 60 s; a refusal is final), printing each line that differs from the last: "draining: <n> cleanups, <n> creates, <n> placements and <n> self-serve moves still hold reservations." and, at the end, "drained: no new kind of reservation is held. stop this console now, then start the old one." Copy rows, pending. The command never stops the console itself. |
| +- **The door** (operator only: `s.moveOperator`, a 404 otherwise, and a 404 without `s.cfg.Releases`, as `handleAPILifetimesCutover`, `lifetimes_cutover.go:526-531`). Idempotent; each call is one round. The first call sets `s.admission.draining` for the rest of this process (a restart clears it, which is the way to abandon a rollback). The round waits on the work itself, not on table entries: (1) every row `s.store.PendingAllocations()` answers (pending, or terminal with `host_key = 1`, Task 7a), whether or not the table holds its entry: `recoverAllocationLocked` under the app lock (a create still running holds that lock, so recovery runs after it returns), which reserves, completes or voids, and releases the key; a row still pending afterwards is counted; (2) the ids registered in `s.placing`: counted, untouched (each settles itself, fence before release, then deregisters, Task 7b); (3) every `place` operation in the table whose id is not registered: settled now as the sweep settles it (fence every host key's record, then `moves.Release`), **never released before its fences answered**; (4) a `create` operation in the table with no allocation row: counted (the sweep pages it); (5) the counts: `cleanup` operations in the table, open self-serve moves (`s.store.OpenSelfServeMoves()`, `selfserve_store.go:562`), pending rows and `create` entries, registrations and `place` entries; (5a) the console's table writes (U3): `s.tableWrites.Resend(ctx, s.cfg.Releases)`, then `unanswered := s.tableWrites.Unanswered()` and `landUntil := s.tableWrites.LastTimedOut().Add(tableAttemptWindow)` (`tableAttemptWindow = 20 * time.Minute`: S3 refuses a request whose signing time is more than 15 minutes from its own clock, plus five minutes for skew; zero when no attempt ever went unanswered); (6) all zero, `unanswered == 0` and `!now.Before(landUntil)` → the settling write, `moves.Release(ctx, s.cfg.Releases, {Kind: "drain", ID: "drain"})` (it holds nothing and writes rev + 1, one `PutTable`), which must itself be answered (`ErrUncertain` → `draining`), then the table, the allocation rows, `s.placing`, the open self-serve moves and `s.tableWrites` read again: still zero → `drained`; else `draining`. The answer: `{state: "draining" | "drained", cleanups, creates, placements, self_serve_moves, unanswered_writes int, land_until int64 (Unix seconds, 0 when past), rev uint64}`. Every write of this round, the settlement's releases and fences included, goes through `PutTable` and so is counted. |
| +- **The clean exit (U3).** `cmd/console/main.go` serves under `signal.NotifyContext(ctx, syscall.SIGTERM, os.Interrupt)`; on the signal it calls `srv.CloseTableWrites(ctx)` with a 30 s deadline and then exits 0. `CloseTableWrites`: `s.tableWrites.Close()` (a `PutTable` started after it answers `moves.ErrTableClosed` and sends nothing; the doors answer it as an unreadable table, 502 or 503 as today), then `Resend` until `Unanswered() == 0` or the deadline; then one line, `reservations: exit clean: no table write unanswered; last unanswered attempt <RFC 3339 or none>` when zero, else `reservations: exit unclean: <n> table writes unanswered`. Only the clean line is a clean exit for the rollback (Global Constraint, step 3); a crash or a `SIGKILL` writes neither, and the procedure then rolls forward. The handler stops nothing else, so a stop outside a rollback behaves as today apart from the line and the up-to-30-second wait. |
| +- **The CLI** `carlos system reservations drain [--console <origin>] [--no-wait]`: `requireDeployAPISession`, then the door every 5 s (`waitLifetimesCutover`'s loop and retry rules: a failed poll or a 5xx is asked again, up to 5 in a row or 60 s; a refusal is final), printing each line that differs from the last: "draining: <n> cleanups, <n> creates, <n> placements and <n> self-serve moves still hold reservations.", "draining: <n> reservation writes wait for an answer.", "draining: a reservation write went unanswered. it could still land until <time>." and, at the end, "drained: no new kind of reservation is held. stop this console now. start the old one only once this console's journal says it exited clean, and not before <time>." (`<time>` the answer's `land_until`, or now when 0) Copy rows, pending. The command never stops the console itself. |
| -- [ ] **Step 1: Write the failing tests**: `TestDrainSettlesBeforeItReleases` (Review Focus 4: a `place` entry whose record PUT the fake store commits late: the fence runs and makes it fail before the release; the entry goes only after the fence answered; a fence failure keeps it and the answer stays `draining`); `TestDrainRefusesNewWork` (after the first call a create, a placement and a self-serve request answer 503 and write nothing; a restarted console takes them again); `TestDrainWaitsForInFlightWork` (a registered placement, a pending allocation, an open self-serve move and a `cleanup` entry each keep the answer `draining` with its count); `TestDrainCountsWorkWithoutEntries` (review 6's finding 3: a create the fake store pauses after `BeginAllocation` and before its `Reserve`: the round answers `draining` with `creates == 1` while the table holds no `create` entry; resumed, the create reserves, completes and releases, and the next round answers `drained`; a placement registered and paused before its `Reserve`: `placements == 1` until it settles); `TestAdmissionAndTheDrainFlagAreOneCriticalSection` (a create whose `BeginAllocation` the fake store blocks inside `admitResidence`: the drain's first call blocks on `mu` until it returns, then counts the row; a create that starts after the flag gets `errors.Is(err, errConsoleDraining)` and `PendingAllocations` is unchanged; the same for a placement's registration, `len(s.placing)` unchanged, and a self-serve `AdmitSelfServeMove`, `OpenSelfServeMoves` unchanged); `TestDrainEndsWithASettlingWrite` (`drained` only after the console's own write and a re-read; a create `Put` held by the fake store from before the drain is delivered after it and fails its condition); `TestDrainIsOperatorOnly`; `TestSystemReservationsDrainCLI` (the lines, the wait and the retries). |
| -- [ ] **Step 2: Run to verify they fail**: `go test ./internal/console -run 'Drain|AdmissionAndTheDrain' -count=1`, `go test . -run 'SystemReservationsDrain|MovesStringsAreRecorded|MovesCopyRecord' -count=1`. |
| +- [ ] **Step 1: Write the failing tests**: `TestDrainSettlesBeforeItReleases` (Review Focus 4: a `place` entry whose record PUT the fake store commits late: the fence runs and makes it fail before the release; the entry goes only after the fence answered; a fence failure keeps it and the answer stays `draining`); `TestDrainRefusesNewWork` (after the first call a create, a placement and a self-serve request answer 503 and write nothing; a restarted console takes them again); `TestDrainWaitsForInFlightWork` (a registered placement, a pending allocation, an open self-serve move and a `cleanup` entry each keep the answer `draining` with its count); `TestDrainCountsWorkWithoutEntries` (review 6's finding 3: a create the fake store pauses after `BeginAllocation` and before its `Reserve`: the round answers `draining` with `creates == 1` while the table holds no `create` entry; resumed, the create reserves, completes and releases, and the next round answers `drained`; a placement registered and paused before its `Reserve`: `placements == 1` until it settles); `TestAdmissionAndTheDrainFlagAreOneCriticalSection` (a create whose `BeginAllocation` the fake store blocks inside `admitResidence`: the drain's first call blocks on `mu` until it returns, then counts the row; a create that starts after the flag gets `errors.Is(err, errConsoleDraining)` and `PendingAllocations` is unchanged; the same for a placement's registration, `len(s.placing)` unchanged, and a self-serve `AdmitSelfServeMove`, `OpenSelfServeMoves` unchanged); `TestDrainEndsWithASettlingWrite` (`drained` only after the console's own write and a re-read; a create `Put` held by the fake store from before the drain is delivered after it and fails its condition); `TestDrainIsOperatorOnly`; `TestSystemReservationsDrainCLI` (the lines, the wait and the retries); `TestDrainWaitsOnUnansweredTableWrites` (a normalisation and, separately, an acquisition whose every attempt the fake answers with a timeout: the round answers `draining` with `unanswered_writes == 1`; the fake starts answering, the next round's resend is answered, `unanswered_writes == 0`); `TestDrainWaitsOnDelayedNormalizationAndAcquisition` (U3's regression witness, review 7's finding 4 interleaving, over a fake S3 store that holds an attempt in its network and models S3's 15-minute validity: a held attempt delivered more than 15 minutes after its send time is refused, `RequestTimeTooSkewed`, and changes nothing. The table is plan 3's A-only body at `E0`, metadata-free. A placement's `ReadForAcquire` sends normalisation N: its first attempt N1 is held and the client sees a timeout; the resend lands `E1`. The placement's acquisition P against `E1`: its first attempt P1 is held; the resend lands. The placement settles and releases; the round answers `draining` with `land_until == P1.sent + 20 min` and `unanswered_writes == 0`; run again with only N1 held, `land_until == N1.sent + 20 min`; at `+5 min` still `draining`. At `+20 min`: `drained`. The console exits through `CloseTableWrites` with the clean line. Plan 3's frozen writer then recreates the A-only body, so `E0` is current again; the fake delivers N1 and then P1: both refused as expired, `fake.Current(OfflineActiveKey)` is still the old console's body, `parsePlan3OfflineActive` reads it, and it holds no `place` entry. A control run asserts the hazard is real: the old console's recreation and the delivery of N1 and P1 at `+5 min`, without waiting for `drained`, restore `E1` and the `place` entry); `TestCleanExitNeedsEveryTableWriteAnswered` (`CloseTableWrites` with one write unanswered for the whole deadline → the unclean line naming 1; with it answered within the deadline → the clean line naming the last unanswered attempt's time; a `PutTable` after `Close` → `moves.ErrTableClosed`, and the fake saw no request; `cmd/console`'s handler, driven by a `SIGTERM` to a test-built binary over a `DirStore`, exits 0 and prints the clean line). **Dev witness** (recorded in the commit message, before any arming, beside Task 4b's): on the dev deployment bucket under a test key, a conditional `PutObject` captured after signing by a test `http.RoundTripper` and replayed unchanged 16 minutes later → refused (`403`, `RequestTimeTooSkewed` expected), and the object is unchanged; the same capture replayed at once → `200`. The test key is removed afterwards. |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/console -run 'Drain|AdmissionAndTheDrain|CleanExit' -count=1`, `go test . -run 'SystemReservationsDrain|MovesStringsAreRecorded|MovesCopyRecord' -count=1`, `go test ./cmd/console -run 'CleanExit' -count=1`. |
| - [ ] **Step 3: Implement. Step 4: Run to verify they pass.** |
| -- [ ] **Step 5: Commit** `console: carlos system reservations drain closes admission in one critical section, then settles every allocation, placement and create reservation before a rollback`. |
| +- [ ] **Step 5: Commit** `console: carlos system reservations drain closes admission in one critical section, settles every allocation, placement and create reservation, and waits until every table write is answered and no unanswered attempt can still land; the console logs a clean exit only with none unanswered`. |
| ### Task 8: The reaper |
| @@ -400,11 +409,11 @@ Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and |
| ### Task 9: End to end and the census |
| **Files:** |
| -- Test: `internal/e2e/selfserve/cleanup_test.go` (plan 2's bridge `host.E2E`, a clock the test advances, a root pass the test drives through the adopt pass's stages with both locks held, a hibernate bucket fake with S3's versioned semantics and a hook that holds a delete) |
| +- Test: `internal/e2e/selfserve/cleanup_test.go` (plan 2's bridge `host.E2E`, a clock the test advances, a root pass the test drives through the adopt pass's stages with both locks held and an empty adopt-process census, a hibernate bucket fake with S3's versioned semantics and a hook that holds a delete) |
| - Create: `internal/provision/testdata/plan14-root-reads-self-serve-moves-4.tsv`, `internal/provision/self_serve_moves4_census_test.go`, `docs/superpowers/notes/census-proposal-self-serve-moves-4.md` |
| - Modify: `internal/provision/plan14_root_reads_test.go`, `internal/provision/plan9_b8_census_test.go` (line 372) |
| -- [ ] **Step 1: Write the tests**: `TestCleanupTwentyFourHoursAfterTheObservedFinish` (a forward move; at 24 h every inventoried version is gone, each removed by its version id, oldest first, no marker written by the cleanup, `lock.json` and `build-state.json` remain, the lifted tombstone is back, root removed the live link when its chain is root-only, the programs file is reported left, the entries are gone, the table is still there and empty, the report is under the move id and `carlos move offline status --log` shows it); `TestCleanupSupersededByAPlacementBack` (the placement is refused while the cleanup holds; after it ends, a placement back to the source runs); `TestCreateUnderCleanupRefusedThenAllowed`; `TestRowMintedUnderTheMarkSupersedes`; `TestLateAttemptAfterSupersessionChangesNothing` (a delete's first attempt held past the supersession; the second attempt answered under the mark; the release; a new wake writes; the held attempt is delivered: the instance serves its new data); `TestPromotionBackIntoTheRegionKeepsItsData` (a legacy move back into the source region while a superseded cleanup had a delete in flight: the delete is answered under the mark, the wake waits for the unmark, the promotion keeps equal-ETag versions in place and copies the rest, and the instance's next wake restores the full replica); `TestOperatorReturnAfterSupersededCleanupStillFindsItsLeftovers` (the return's start door accepts the forward's leftover set after a superseded cleanup that had marked and unmarked); `TestLegacyMoveIntoTheSourceRegionIsTolerated` (switch off: a legacy move back to the source region while the cleanup holds: the cleanup supersedes at its next key, answers any delete in flight, the wake waits for the unmark, the promotion restores the staged data and the instance serves it); `TestUntrackedLifetimeMoveFinishes` (an incarnation-0 record with no slot moved by an operator: `gone` reports, the move finishes, no obligation exists); `TestOlderCleanupInTheMoveLog` (two moves of one host, the first's cleanup open: `--log --cleanup <first id>` shows it after the second finished and after the record is deleted); `TestRollbackDrain` (the procedure's steps 1 and 2 against a live cleanup, a placement and a create paused between `BeginAllocation` and its reservation: the drain answers `drained` only after the cleanup ended and both were settled; the old console's parser then reads the table); `TestCleanupHoldsTheMarkUntilItsDeleteIsAnswered` (the cleanup's agent is stopped with a delete in flight for 25 h of the test clock while a legacy move returns the host to the source region: the return's wake waits; the agent restarts, the delete is answered, the mark lifts, and the instance serves the returned data). |
| +- [ ] **Step 1: Write the tests**: `TestCleanupTwentyFourHoursAfterTheObservedFinish` (a forward move; at 24 h every inventoried version is gone, each removed by its version id, oldest first, no marker written by the cleanup, `lock.json` and `build-state.json` remain, the lifted tombstone is back, root removed the live link when its chain is root-only, the programs file is reported left, the entries are gone, the table is still there and empty, the report is under the move id and `carlos move offline status --log` shows it); `TestCleanupSupersededByAPlacementBack` (the placement is refused while the cleanup holds; after it ends, a placement back to the source runs); `TestCreateUnderCleanupRefusedThenAllowed`; `TestRowMintedUnderTheMarkSupersedes`; `TestLateAttemptAfterSupersessionChangesNothing` (a delete's first attempt held past the supersession; the second attempt answered under the mark; the release; a new wake writes; the held attempt is delivered: the instance serves its new data); `TestPromotionBackIntoTheRegionKeepsItsData` (a legacy move back into the source region while a superseded cleanup had a delete in flight: the delete is answered under the mark, the wake waits for the unmark, the promotion keeps equal-ETag versions in place and copies the rest, and the instance's next wake restores the full replica); `TestOperatorReturnAfterSupersededCleanupStillFindsItsLeftovers` (the return's start door accepts the forward's leftover set after a superseded cleanup that had marked and unmarked); `TestLegacyMoveIntoTheSourceRegionIsTolerated` (switch off: a legacy move back to the source region while the cleanup holds: the cleanup supersedes at its next key, answers any delete in flight, the wake waits for the unmark, the promotion restores the staged data and the instance serves it); `TestUntrackedLifetimeMoveFinishes` (an incarnation-0 record with no slot moved by an operator: `gone` reports, the move finishes, no obligation exists); `TestOlderCleanupInTheMoveLog` (two moves of one host, the first's cleanup open: `--log --cleanup <first id>` shows it after the second finished and after the record is deleted); `TestRollbackDrain` (the procedure's steps 1 and 2 against a live cleanup, a placement and a create paused between `BeginAllocation` and its reservation: the drain answers `drained` only after the cleanup ended and both were settled; with one table attempt held by the fake, only 20 minutes after it; the new console's exit line is the clean one; the old console's parser then reads the table); `TestCleanupHoldsTheMarkUntilItsDeleteIsAnswered` (the cleanup's agent is stopped with a delete in flight for 25 h of the test clock while a legacy move returns the host to the source region: the return's wake waits; the agent restarts, the delete is answered, the mark lifts, and the instance serves the returned data). |
| - [ ] **Step 2: Run them; fix what they find within this plan's scope.** |
| - [ ] **Step 3: Run both census tests and the cross-builds**; write the census proposal with the loader block, the changed envelopes (`internal/host`, `internal/release`, `internal/registry`, `internal/lease`, `internal/moves`, `internal/console`, `internal/offlinecleanup`, the root package) and the units. |
| - [ ] **Step 4: Commit** `self-serve moves: cleanup end to end`; push. |
| @@ -414,10 +423,10 @@ Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and |
| ## Self-review notes |
| - Spec coverage: §7.6 the source replica (T1, T3, T4a, T4b, T4c, T4d), root-owned leftovers with the link and process checks (T6a, T6b, T6c), parked local state (T8), the staged objects of an aborted move (unchanged); §7.1's `cleanup` reservation and the hostname keys (T2a, T2b, T2c, T7a, T7b); §8's cleanup pages and every cleanup key in the move log (T5a, T5b); §9 rollout and rollback (Global Constraints, T7c); §11's cleanup witnesses (T4a, T4c, T9). |
| -- **Stated deviations.** (1) Legacy moves hold no reservation. A legacy move into the source region carries a move on the record from its door onward, so condition (1) supersedes the cleanup at its next key, and its completion's wake cannot take the lease while the mark holds; after the unmark the wake's promotion mirrors the staged replica over the prefix (`promoteStagedReplica`, an exact mirror, called during the restore after acquisition, `backend_exec.go:819`), so no deleted key matters; review 4's writer audit confirms this normal-operation argument (completion changes the record; promotion waits for acquisition). A delete sent before the supersession is answered under the mark, before the unmark, so none is outstanding when the promotion keeps an equal-ETag version in place. Holding a key from the box at `MoveComplete` (`movespass.go:1617`) would add a box-side release and crash path and extend a legacy cut's downtime behind root's answer, for no exclusion the mark does not already give. (2) Placement keeps `SetRecordRegion`'s ETag-conditional write and settles an uncertain one by `fenceRecord`, after which no earlier PUT can land; the reservation entry is the persisted intent, so no journal table is added. (3) A cleanup entry is never reclaimed by another reserver (§7.1 allowed a status-based reclaim); the console pages instead. (4) Only the sealing box's local paths are cleaned; other boxes in the source region keep theirs, as today. (5) The create reserves after `BeginAllocation`, not before, so every reservation has an intent row behind it. (6) `build-state.json` is kept with `lock.json`. (7) **§7.6's "plain `DeleteObject`, `VersionId` deletes are never used" is amended (Paul's question 9):** every inventoried version is deleted by its version id, oldest first, the version current at `gone` last and required current before every deletion, so no older version is exposed (short of a version-id delete of that version by hand during the run) and no late request can touch another owner's version. The retired replica is then not recoverable from the bucket. (8) The table is never deleted (plan 2 deleted it when empty). (9) Root removes a leftover only on a root-only chain; the programs file, under the agent-writable `/data` on every built box, is recorded as left at `gone` and stays the runbook's, and so do both paths on a writable-opt box. (10) Root's rename-aside is gone; identity is compared on the opened parent with `fstatat`. (11) §7.6's "a key already carrying a delete marker is done" becomes: a key whose current entry is not its inventoried latest (a later version, or a marker the inventory does not hold) is `superseded` and nothing more of it is deleted, so a foreign marker and everything beneath it are left; a marker the inventory holds as the latest is deleted last, like any latest. (12) Root's step is a stage of the adopt pass, under a new root-only pass lock that every adopt pass takes; held publication's freeze and lock are unchanged. (13) **The cleaning mark has no expiry** (§7.6 has the cleanup hold a lease; here it is a state): a cleanup that cannot get an answer for a delete holds the host's replica marked, and with it every wake, landing and return of the host in this region, until the delete is answered or an operator abandons it after the page (Paul's question 9). (14) **Root's gate:** on a box where cleanup was ever armed (the root-owned `gated` marker, or the switch on now), an adopt pass that cannot take the root pass lock, for any reason but a chain that is not root-only, publishes nothing that pass; a box never armed keeps today's fail-open adoption. (15) The bucket's versioning status is not read (the box roles lack `s3:GetBucketVersioning`); a `null` id in the listing, which is how an unversioned or suspended bucket shows, writes no `retired.json`, and the rollout checks the status by hand. |
| -- **The one residual in root's exclusion.** On a gated box there is none: every pass that publishes holds the root pass lock. On a box that was never gated no root step can run. What is left is the moment a box first becomes gated: a pass that read the switch off and found no marker, whose adopt-pass lock failed (a shared-uid writer's symlink at `/data/adopt-pass.lock`) and whose root pass lock failed for a root-level fault (which no other uid can cause), publishes fail-open; if it is still running when a later pass, after the switch went on and the agent's first cleanup dropped a request, runs root's step, the two overlap. Three independent conditions, one of them a fault only root can cause, at one moment in a box's life. Failing closed on every box, armed or not, would remove even this, at the cost of today's fail-open adoption on boxes that never run a cleanup; the plan follows the ruling and names the case here. |
| +- **Stated deviations.** (1) Legacy moves hold no reservation. A legacy move into the source region carries a move on the record from its door onward, so condition (1) supersedes the cleanup at its next key, and its completion's wake cannot take the lease while the mark holds; after the unmark the wake's promotion mirrors the staged replica over the prefix (`promoteStagedReplica`, an exact mirror, called during the restore after acquisition, `backend_exec.go:819`), so no deleted key matters; review 4's writer audit confirms this normal-operation argument (completion changes the record; promotion waits for acquisition). A delete sent before the supersession is answered under the mark, before the unmark, so none is outstanding when the promotion keeps an equal-ETag version in place. Holding a key from the box at `MoveComplete` (`movespass.go:1617`) would add a box-side release and crash path and extend a legacy cut's downtime behind root's answer, for no exclusion the mark does not already give. (2) Placement keeps `SetRecordRegion`'s ETag-conditional write and settles an uncertain one by `fenceRecord`, after which no earlier PUT can land; the reservation entry is the persisted intent, so no journal table is added. (3) A cleanup entry is never reclaimed by another reserver (§7.1 allowed a status-based reclaim); the console pages instead. (4) Only the sealing box's local paths are cleaned; other boxes in the source region keep theirs, as today. (5) The create reserves after `BeginAllocation`, not before, so every reservation has an intent row behind it. (6) `build-state.json` is kept with `lock.json`. (7) **§7.6's "plain `DeleteObject`, `VersionId` deletes are never used" is amended (Paul's question 9):** every inventoried version is deleted by its version id, oldest first, the version current at `gone` last and required current before every deletion, so no older version is exposed (short of a version-id delete of that version by hand during the run) and no late request can touch another owner's version. The retired replica is then not recoverable from the bucket. (8) The table is never deleted (plan 2 deleted it when empty). (9) Root removes a leftover only on a root-only chain; the programs file, under the agent-writable `/data` on every built box, is recorded as left at `gone` and stays the runbook's, and so do both paths on a writable-opt box. (10) Root's rename-aside is gone; identity is compared on the opened parent with `fstatat`. (11) §7.6's "a key already carrying a delete marker is done" becomes: a key whose current entry is not its inventoried latest (a later version, or a marker the inventory does not hold) is `superseded` and nothing more of it is deleted, so a foreign marker and everything beneath it are left; a marker the inventory holds as the latest is deleted last, like any latest. (12) Root's step is a stage of the adopt pass, under a new root-only pass lock that every adopt pass takes; held publication's freeze and lock are unchanged. (13) **The cleaning mark has no expiry** (§7.6 has the cleanup hold a lease; here it is a state): a cleanup that cannot get an answer for a delete holds the host's replica marked, and with it every wake, landing and return of the host in this region, until the delete is answered or an operator abandons it after the page (Paul's question 9). (14) **Root's gate:** on a box where cleanup was ever armed (the root-owned `gated` marker, or the switch on now), an adopt pass that cannot take the root pass lock, for any reason but a chain that is not root-only, publishes nothing that pass; a box never armed keeps today's fail-open adoption; root's step first runs on a box only once a census of `/proc` finds no other root adopt process alive after the box was gated (`quiesced`). (15) The bucket's versioning status is not read by code (the box roles lack `s3:GetBucketVersioning`). Two separate guarantees stand in for it: the listing at `gone` refuses a `null` id, so every inventoried id is usable; and the rollout's one-off `get-bucket-versioning` check establishes `Enabled`, which the listing cannot (a suspended bucket with only real ids passes it). Skipping the rollout check leaves deletion safe and lets later writes overwrite a `null` version the inventory never holds. (16) **Table writes are single attempts** (`moves.PutTable`), resent identically until answered; the store door's client has no single-attempt write, so a box on the store door never reserves a cleanup. A rollback needs the new console's clean exit line and waits 20 minutes after the last unanswered table attempt; without the clean line it rolls forward. |
| +- **First arming (U1).** Rev 7 named one residual: a pass admitted fail-open before the box was gated, still publishing when root's step first runs. Rev 8 closes it with the arming boundary (Task 6b): root's step requires `quiesced`, written only at a moment after `gated` existed when no other root adopt process was alive, so every pass that could publish without the root pass lock has exited first. The ruling asked for an in-process pass mutex and argued that processes do not overlap because the adopt unit is one service. Verified against the units: `carlos adopt` runs one pass per process, and four separate units start it (`carlos-adopt`, `carlos-stage`, `carlos-config-sync`, `carlos-restart`), whose passes can overlap (`agent.go:4645-4648`). One unit never overlaps itself (a restarted pass's predecessor has exited), but an in-process mutex would serialise nothing, so the boundary is a process census instead. On census-required layouts the outer root-release lock (`rootstate.WithRelease`, `internal/rootstate/release.go:14`) already serialises every pass, and a failure there is fatal; the census matters on census-optional layouts, where that wrapper runs the pass directly (`internal/host/rootreleaselock.go:38-40`). |
| - Plan 2 amendments (built code, changed here): move selections hold `host/<hostname>`; the switch-off rule applies between moves only, so a cleanup neither blocks nor is blocked by an operator move on another box; recovery mode counts `move` entries only; the cleanup reclaim interface is deleted; `releaseBoxes` writes an empty table instead of deleting it; every table write carries `epoch` and `rev`. |
| -- Built-code changes this plan makes, by task: T1 `moves/offline.go`, `host/movesoffline.go`, `release/programhistory.go` (export); T2a `moves/offline.go` (kinds, epoch, rev, `valid()`); T2b `console/offlinereserve.go`, `console/selfserve_pass.go`, `console/lifetimes_store.go`; T2c `console/lifetimes_recovery.go`, `console/selfserve_pages.go`; T3 `lease/cleaning.go` (new), `lease/lease.go`, `lease/tombstone.go`, `lease/lockstate.go` (the refusals), `host/movesoffline_land.go` (the landing's wait); T4b, T4c `host` (new files only); T4d `host/movespass.go`; T5a `host/movespaging.go`, `console/apiinstancemove_offline.go`; T5b `console/authcli.go`, `move.go`, `docs/site/cli.md`, the copy record; T6a `registry`; T6b `host/agent.go` (the lock and the gate), `host/adoptrootgate.go` (new), `host/adoptlock.go` and `host/movesheldlock.go` (comments), `host/claimcensus.go`; T7a `console/store.go`, `lifetimes_store.go`, `lifetimes_create.go`, `lifetimes_recovery.go`; T7b `fleets.go`, `console/store.go` (`claimRawSetAppPlacement`), `console/view.go` (`consoleRegistry`), `registry/registry.go` (`SetPlacementContext`), `registry/remote_dispatch.go` (the deadline), `offlinereserve.go`, `residence_sweep.go`; T7c `console/authcli.go`, `lifetimes_create.go`, `fleets.go`, `selfserve_door.go`, `system.go`, `docs/site/cli.md`. The mirror (`internal/lease/mirror.go`) is not changed. |
| -- Names used across tasks: `offlinecleanup.Retired` (`State`, `Outcome`, `Keys`, `Events`, `PagesPending`, `LocalLeft`, `Root`, `Rev`, `PublishedRev`, `MarkedAt`, `ReserveUncertain`, `InFlight`, `Superseding`, `RootRequestedAt`, `CancelRequestedAt`), `CorruptStub`, `CleanupID`, `InventoryKey`, `InventoryVersion`, `InFlightDelete`, `KeyOutcome` (`State`, `Done`), `Event`, `PendingPage`, `Report`, `FileIdentity`, `LocalPath`, `KeptPath`, `RootCleanupRequest`, `RootCleanupPath`, `RootCleanupAck`, `ValidateRetired`, `ValidateCorruptStub`, `ValidateReport`, `UsableVersionID`, `Identify`, `RootOnlyChain`, `InventoryKey.Latest`, `InventoryVersion.Seq`, `Report.InFlight`, `Report.AbandonPath`; `moves.HostKey`, `IsHostKey`, `Active`, `ReadActive`, `ReadForAcquire`, `NextActiveBody`, `Op`, `Conflicts`, `Reserve`, `Holds`, `Release`, `ErrReserved`, `ErrUnreadable`, `ErrUncertain`, `ErrReservationMismatch`, `RecordAway`, `Away*`, `OfflineOpDir`, `RetiredFile`, `CleanupReportKey`, `OfflineKindCreate`, `OfflineKindPlace`; `lease.CleaningOwner`, `CleaningExpiry`, `IsCleaningOwner`, `MarkCleaning`, `UnmarkCleaning`, `ErrNotOurLock`, `VersionedStore.DeleteVersion`; `deleteInventoriedVersion`, `w.cleanupVisit`, `w.supersededBy`, `w.retireKey`, `w.settleInFlight`, `w.kickCleanup`, `cleanupRun`, `cleanupMu`, `cfg.CleanupArmed`, `recordRetired`, `createRetired`, `writeRetired`, `markPublished`, `flushPageOutboxFor`, `reapRetired`; `release.ServeRootCleanupRequests`, `RootCleanupOptions`, `RootCleanupBase`, `RootCleanupDir`, `ReadRootCleanupAck`, `LockRootPass`, `RootCleanupGated`, `ErrRootCleanupRefused`, `ErrNotRootOnly`, `ProgramsPath`; `rootCleanupAuthority`, `rootCleanupPass`, `rootRefusedAfter`, `adoptRootGate`, the files `root-cleanup/gated` and `cleanup.abandon`; `registry.WithHostFrozen`, `SetPlacementContext`; `consoleRegistry.SetPlacementContext`; `s.offlineCleanupsOf`, `s.reservePlacement`, `s.registerPlacement`, `s.deregisterPlacement`, `s.settlePlacement`, `s.placing`, `s.residenceSweep`, `s.releaseAllocationKey`, `s.admission`, `s.admitResidence`, `errConsoleDraining`, `errHostUnderCleanup`; the CLI's `moveArgs.log`, `logMove`, `logCleanup`, and `carlos system reservations drain`. |
| +- Built-code changes this plan makes, by task: T1 `moves/offline.go`, `host/movesoffline.go`, `release/programhistory.go` (export); T2a `moves/offline.go` (kinds, epoch, rev, `valid()`), `moves/tablewrite.go` (new), `release/store.go` (`PutOnce`); T2b `console/offlinereserve.go`, `console/selfserve_pass.go`, `console/lifetimes_store.go`, `console/server.go` (`tableWrites`); T2c `console/lifetimes_recovery.go`, `console/selfserve_pages.go`; T3 `lease/cleaning.go` (new), `lease/lease.go`, `lease/tombstone.go`, `lease/lockstate.go` (the refusals), `host/movesoffline_land.go` (the landing's wait), `activator/backend_exec.go` and `activator/activator.go` (the wake's answer); T4b, T4c `host` (new files only); T4d `host/movespass.go`; T5a `host/movespaging.go`, `console/apiinstancemove_offline.go`; T5b `console/authcli.go`, `move.go`, `docs/site/cli.md`, the copy record; T6a `registry`; T6b `host/agent.go` (the lock and the gate), `host/adoptrootgate.go` (new), `release/adoptpasses_linux.go` and `_other.go` (new), `host/adoptlock.go` and `host/movesheldlock.go` (comments), `host/claimcensus.go`; T7a `console/store.go`, `lifetimes_store.go` (`host_key`, `Allocation`), `lifetimes_create.go`, `lifetimes_recovery.go` (the reload); T7b `fleets.go`, `console/store.go` (`claimRawSetAppPlacement`), `console/view.go` (`consoleRegistry`), `registry/registry.go` (`SetPlacementContext`), `registry/remote_dispatch.go` (the deadline), `offlinereserve.go`, `residence_sweep.go`; T7c `console/authcli.go`, `lifetimes_create.go`, `fleets.go`, `selfserve_door.go`, `system.go`, `cmd/console/main.go` (the `SIGTERM` handler), `docs/site/cli.md`. The mirror (`internal/lease/mirror.go`) is not changed. |
| +- Names used across tasks: `offlinecleanup.Retired` (`State`, `Outcome`, `Keys`, `Events`, `PagesPending`, `LocalLeft`, `Root`, `Rev`, `PublishedRev`, `MarkedAt`, `ReserveUncertain`, `InFlight`, `Superseding`, `RootRequestedAt`, `CancelRequestedAt`), `CorruptStub`, `CleanupID`, `InventoryKey`, `InventoryVersion`, `InFlightDelete`, `KeyOutcome` (`State`, `Done`), `Event`, `PendingPage`, `Report`, `FileIdentity`, `LocalPath`, `KeptPath`, `RootCleanupRequest`, `RootCleanupPath`, `RootCleanupAck`, `ValidateRetired`, `ValidateCorruptStub`, `ValidateReport`, `UsableVersionID`, `Identify`, `RootOnlyChain`, `InventoryKey.Latest`, `InventoryVersion.Seq`, `Report.InFlight`, `Report.AbandonPath`; `moves.HostKey`, `IsHostKey`, `Active`, `ReadActive`, `ReadForAcquire`, `NextActiveBody`, `PutTable`, `TableWrites` (`Unanswered`, `LastTimedOut`, `Resend`, `Close`), `ErrTableClosed`, `Op`, `Conflicts`, `Reserve`, `Holds`, `Release`, `ErrReserved`, `ErrUnreadable`, `ErrUncertain`, `ErrReservationMismatch`, `RecordAway`, `Away*`, `OfflineOpDir`, `RetiredFile`, `CleanupReportKey`, `OfflineKindCreate`, `OfflineKindPlace`; `lease.CleaningOwner`, `CleaningExpiry`, `IsCleaningOwner`, `MarkCleaning`, `UnmarkCleaning`, `ErrNotOurLock`, `VersionedStore.DeleteVersion`; `activator.errLeaseCleaning`, `cleaningRetryAfter`; `release.PutOnce`, `SingleAttemptPutter`, `ErrNoSingleAttemptPut`; `deleteInventoriedVersion`, `w.cleanupVisit`, `w.supersededBy`, `w.retireKey`, `w.settleInFlight`, `w.kickCleanup`, `cleanupRun`, `cleanupMu`, `w.tableWrites`, `cfg.CleanupArmed`, `recordRetired`, `createRetired`, `writeRetired`, `markPublished`, `flushPageOutboxFor`, `reapRetired`; `release.ServeRootCleanupRequests`, `RootCleanupOptions`, `RootCleanupBase`, `RootCleanupDir`, `ReadRootCleanupAck`, `LockRootPass`, `RootCleanupGated`, `QuiesceRootCleanup`, `AdoptPasses`, `ErrRootCleanupRefused`, `ErrNotRootOnly`, `ProgramsPath`; `rootCleanupAuthority`, `rootCleanupPass`, `rootRefusedAfter`, `adoptRootGate`, the files `root-cleanup/gated`, `root-cleanup/quiesced` and `cleanup.abandon`; `registry.WithHostFrozen`, `SetPlacementContext`; `consoleRegistry.SetPlacementContext`; `s.offlineCleanupsOf`, `s.reservePlacement`, `s.registerPlacement`, `s.deregisterPlacement`, `s.settlePlacement`, `s.placing`, `s.residenceSweep`, `s.releaseAllocationKey`, `s.store.Allocation`, `recoveryListedHook`, `s.admission`, `s.admitResidence`, `errConsoleDraining`, `errHostUnderCleanup`, `s.tableWrites`, `tableAttemptWindow`, `(*Server).CloseTableWrites`; the CLI's `moveArgs.log`, `logMove`, `logCleanup`, and `carlos system reservations drain`. |
| - Review Focus 1 to 5 are pinned by T3+T4b+T4c, T4a, T6b+T6c, T2a-c+T7a-c, T5a+T5b+T8. |
| -- **Open questions for Paul** (amadan discussion 363): (1) legacy moves without a hostname reservation: accept the exemption, or reserve around `MoveComplete` on the box (a box-side release and crash path, and longer legacy-cut downtime while root answers)? (2) Cleanup runs 24 hours after **every** offline move, operator moves included, replacing the runbook's manual cleanup; intended (§10 Q4 talks about self-serve)? (3) Is the box switch `CARLOS_OFFLINE_CLEANUP=on` acceptable as the rollout gate? (4) A move of the same host waits while its cleanup holds `host/<hostname>`, which can last until root answers (paged every 10 min, hourly once root's ancestry is refused) or until an unanswered delete is answered or abandoned (paged every 10 min); acceptable? (5) The copy rows: five door sentences, eight page subjects and one page body (now also the CLI's lines, among them the waiting-delete line, the drain's lines and the `--log` paragraph). (6) On every built box the programs file sits under the agent-writable `/data`, so root never removes it and it stays the runbook's; acceptable, or should the agent remove it itself? (7) Withdrawn by rev 6: the 15-minute `QuietUntil` and the marker-removing repair are gone; question 9 replaces it. (8) Rollback needs no self-serve move in flight; the drain now refuses to finish until none is. Acceptable? (9) **The §7.6 amendment:** the cleanup deletes every inventoried version of the retired replica by its version id (oldest first, the one current at `gone` last) instead of writing delete markers, so the source copy of a finished move is not recoverable from the bucket after the cleanup, in exchange for a boundary the server enforces (a late or retried request can remove only the version it names). **Rev 7 adds a consequence:** the mark that keeps every later owner off the replica while a delete may still land has no expiry, so a cleanup that cannot settle a delete (the bucket keeps refusing it, or the box is gone) holds the host's replica marked, and with it every wake, landing and return of that host in this region, until an operator acts on the page: by repairing what refuses the delete, or by abandoning it on the box (`cleanup.abandon`). Accept, or keep markers (then only `deleteInventoriedVersion` changes, and the settlement rules need a fresh review)? |
| +- **Open questions for Paul** (amadan discussion 363): (1) legacy moves without a hostname reservation: accept the exemption, or reserve around `MoveComplete` on the box (a box-side release and crash path, and longer legacy-cut downtime while root answers)? (2) Cleanup runs 24 hours after **every** offline move, operator moves included, replacing the runbook's manual cleanup; intended (§10 Q4 talks about self-serve)? (3) Is the box switch `CARLOS_OFFLINE_CLEANUP=on` acceptable as the rollout gate? (4) A move of the same host waits while its cleanup holds `host/<hostname>`, which can last until root answers (paged every 10 min, hourly once root's ancestry is refused) or until an unanswered delete is answered or abandoned (paged every 10 min); acceptable? (5) The copy rows: five door sentences, the wake's 503 body for a replica under cleanup ("instance is being cleaned up after a move. retry shortly", rev 8), eight page subjects and one page body (now also the CLI's lines, among them the waiting-delete line, the drain's lines, two of them new in rev 8 for unanswered reservation writes, and the `--log` paragraph). (6) On every built box the programs file sits under the agent-writable `/data`, so root never removes it and it stays the runbook's; acceptable, or should the agent remove it itself? (7) Withdrawn by rev 6: the 15-minute `QuietUntil` and the marker-removing repair are gone; question 9 replaces it. (8) Rollback needs no self-serve move in flight; the drain now refuses to finish until none is. Acceptable? (9) **The §7.6 amendment:** the cleanup deletes every inventoried version of the retired replica by its version id (oldest first, the one current at `gone` last) instead of writing delete markers, so the source copy of a finished move is not recoverable from the bucket after the cleanup, in exchange for a boundary the server enforces (a late or retried request can remove only the version it names). **Rev 7 adds a consequence:** the mark that keeps every later owner off the replica while a delete may still land has no expiry, so a cleanup that cannot settle a delete (the bucket keeps refusing it, or the box is gone) holds the host's replica marked, and with it every wake, landing and return of that host in this region, until an operator acts on the page: by repairing what refuses the delete, or by abandoning it on the box (`cleanup.abandon`). Accept, or keep markers (then only `deleteInventoriedVersion` changes, and the settlement rules need a fresh review)? (10) **Rev 8, rollback timing:** a rollback now needs the new console to log a clean exit after `drained`, and waits 20 minutes after the last reservation write that went unanswered (usually none, so no wait) and after the last agent restart of step 1; a console that crashed or was killed after `drained` is rolled forward (restarted, drained, stopped again), never rolled back. Acceptable as an operating rule? |