amadan
plan 4 rev 7: the cleaning mark never expires while a delete is in flight; an armed adopt pass publishes nothing without the root pass lock; a drain barrier shared with every admitter; acquisitions only against epoch-bearing bodies; the inventoried latest version must be current
Answers Astra's review of rev 6 (findings 1-6, the null version id, the init order). The mark is a state (a sentinel expiry litestream refuses), lifted by the cleanup after every delete is answered, by supersession only with nothing in flight, or by an operator's abandon file after the page.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1 file changed,
+110
−103
docs/superpowers/plans/2026-10-22-self-serve-moves-4-cleanup.md+110 −103
CI failed — run details
CI log
=== gate === sigwho: pid 2 ppid 1 pgid 0 sid 0; child pid 3 .amadan/test-gate.sh gate: 2026-10-24T09:56:34Z on aarch64, 32 cpus, 61449 MB available, go1.26.8 linux/arm64 gate: +0s start launchctx gate: +0s start vet gate: +0s start gofmt gate: +0s start rest gate: +12s start internal_console@0 gate: +12s start internal_console@1 gate: +12s start internal_console@2 gate: +12s start internal_console@3 gate: +12s start internal_console@4 gate: +12s start internal_console@5 gate: +12s start internal_console@6 gate: +12s start internal_console@7 gate: +12s waiting for internal_console's shards before the next split package gate: +172s start root@0 gate: +172s start root@1 gate: +172s start root@2 gate: +172s waiting for root's shards before the next split package amadan: step gate failed: signal: terminated amadan: job failed at step gate — its output is above, not at the tail of this log
diff --git a/docs/superpowers/plans/2026-10-22-self-serve-moves-4-cleanup.md b/docs/superpowers/plans/2026-10-22-self-serve-moves-4-cleanup.md| index 0363c1e05..9285197c3 100644 |
| --- a/docs/superpowers/plans/2026-10-22-self-serve-moves-4-cleanup.md |
| +++ b/docs/superpowers/plans/2026-10-22-self-serve-moves-4-cleanup.md |
| @@ -2,54 +2,56 @@ |
| > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| -Rev 6, 2026-10-24. Rev 5's review (`after236/selfserve-plan4-review5.md`: R1 to R8 and the acknowledgement-location contradiction; F2 to F4, F6 to F13, N1 to N3 and N5 to N10 closed; F1, F5 and N4 open; N11 partial) is answered by removing the machinery that kept growing a settlement protocol and by putting root's step where no other root publisher can run. **R1 to R4, F1, N4:** the ETag-conditional delete, `QuietUntil` and the repair pass are gone. `retired.json` records **every** version id under the normalised prefix at `gone`, delete markers included, and the cleanup deletes each one by its version id, oldest first within a key, through one function (`deleteInventoriedVersion`, Task 4b). A late, retried or outstanding delete can then only remove the version it names, which is ours, never a version a later owner wrote. This is a spec amendment (design §7.6 says markers and never `VersionId` deletes) and is **Paul's question 9**; the plan stays buildable either way because only Task 4b's function would change. One rule is added beyond the ruling, because of a built-code fact: a promotion keeps a destination object whose ETag equals the staged one instead of rewriting it (`lease.MirrorPrefixKeepObserved`, `internal/lease/mirror.go:285`), so a later owner can adopt one of our versions in place. A delete that went out without an answer is therefore sent again, and answered, before the mark or the reservation is given up, and only while both are still this cleanup's (Task 4c). **R5, F5:** root's step is a stage of `runAdoptUnderRootRelease`, after release adoption (named recovery runs inside it) and after the held stage, and it runs only while the pass holds a new root-only pass lock that every adopt pass takes in every mode (Task 6b). The registry host freeze and the held-stage lock stay as rev 5 had them. **R6, N11:** rollback is a procedure with a drain command, `carlos system reservations drain`, run on the new console before it stops (Global Constraint "Arming and rollout", Task 7c). **R7:** `--log` takes `--move <id>` or `--cleanup <id>`, and the console lists a host's cleanup reports across moves from the bucket, so a deleted record still answers (Task 5b). **R8:** placement registers before it reserves and stays registered through settlement; one deadline covers the whole operation, and the two local writes take it (Task 7b). **The acknowledgement location:** when root rejects a directory's ancestry it writes nothing there; it logs the refusal, and the agent pages hourly after an hour without an answer and never releases (Task 6b). Task 2 is split into 2a to 2c, the repair task is gone, and the delete primitive has its own task with the dev gate. |
| +Rev 7, 2026-10-24. Rev 6's review (`after236/selfserve-plan4-review6.md`: findings 1 to 6, the `null` version id and the initialisation-order nit; R1 to R4, R7 and the acknowledgement location closed; R5, R6 and R8 open through findings 2 to 5) is answered as follows. **Finding 1:** the cleaning mark on `lock.json` has no expiry. It is a state, not a timed lease: written by a conditional PUT over this move's lifted tombstone with a sentinel expiry no clock reaches, refused to every lease acquirer whatever its age, never renewed, and lifted only by the cleanup's own `releasing` once every delete of the run is answered, by supersession or cancellation only when no delete is in flight, or by an operator's abandon after the page (Global Constraint "The mark has no expiry", Tasks 3, 4a, 4c). A crash or an outage of any length lifts nothing, so no wake can take the lease while a delete the cleanup sent may still land. **Finding 2:** on a box where cleanup can run (the switch on, or the root-owned `gated` marker an armed pass left), an adopt pass that cannot take the root pass lock publishes nothing that pass; a box never armed keeps today's fail-open adoption (Task 6b's `adoptRootGate`). **Finding 3:** the drain has an admission barrier: one console-wide flag, checked inside the same critical section in which `BeginAllocation` persists its row, a placement registers and `AdmitSelfServeMove` admits; the drain sets it first, then waits on the allocation rows, the placement registrations and the open self-serve moves themselves, not on table entries (Task 7c). **Finding 4:** a new-protocol acquisition is issued only against a table body that carries `epoch` and `rev`; a body without them is first normalised by a separate conditional write and read again (`moves.ReadForAcquire`, Tasks 2a and 2b). **Finding 5:** `consoleRegistry` (`internal/console/view.go:511`) gains `SetPlacementContext`, and the registry's remote dispatcher (`internal/registry/remote_dispatch.go:270`) runs the update under the deadline it received (Task 7b). **Finding 6 and the `null` id:** condition (4) now requires the key's inventoried latest version (the one current at `gone`) to be current at every deletion; a key whose latest is gone or no longer current is `superseded` and nothing more of it is deleted. The inventory records each version's listing sequence and the key's latest id, so no tie is reversed; `ValidateRetired` refuses a `null` id by `lease.UsableVersionID`'s rule, and a listing that holds one writes no `retired.json` (Tasks 1 and 4c). **Initialisation order:** `LockRootPass` and the request directory check the existing ancestry before root creates any missing directory (Task 6b). |
| + |
| +Rev 6's answers that stand: every data deletion names one inventoried version id, oldest first within a key, through one function (`deleteInventoriedVersion`, Task 4b), so a late, retried or outstanding delete can remove only the version it names (R1 to R4, F1, N4; a spec amendment, **Paul's question 9**); a delete that went out without an answer is sent again, and answered, before the mark is given up, because a promotion can keep one of our versions in place by its equal ETag (`lease.MirrorPrefixKeepObserved`, `internal/lease/mirror.go:285`; Task 4c); root's step is a stage of `runAdoptUnderRootRelease`, after release adoption (named recovery runs inside it) and after the held stage, under the adopt-pass lock and a root-only pass lock, with the registry host freeze and the held-stage lock as rev 5 had them (R5, F5; Task 6b); rollback is a procedure with a drain command, `carlos system reservations drain`, run on the new console before it stops (R6, N11; Task 7c); `--log` takes `--move <id>` or `--cleanup <id>`, and the console lists a host's cleanup reports across moves from the bucket, so a deleted record still answers (R7, Task 5b); placement registers before it reserves and stays registered through settlement, under one deadline (R8, Task 7b); when root rejects a directory's ancestry it writes nothing there, logs the refusal, and the agent pages hourly after an hour without an answer and never releases (Task 6b). Task 2 is split into 2a to 2c, and the delete primitive has its own task with the dev gate. |
| Rev 5's answers that stand: the reservation table is never deleted and every write carries a rising `rev` and the table's `epoch` (F2); no rename-aside, and root removes by `fstatat` and `unlinkat` on the opened parent, only on a root-only chain, journalling intent and result in its own directory, which it also walks on restart (F5, F6); supersession is checked at every step, with one outcome table (F7); an untracked lifetime gets no `retired.json` (N5); the corrupt stub, the activator defaults, one placement entry per host key, the restored `RootRequestedAt` and the persisted one-shot page (N6, N9, N10, F3, F12, N8, F13); the CLI's `--log` (N7, extended by R7). |
| -Rev 4's closed answers stand: one state enum and one transition table (F3); a cleanup entry is never reclaimed by another reserver (F4); the programs file has no reference check and the live link keeps §7.6's link and process checks, with unreadable evidence retried (F8); root authorises through a host-level callback, so `internal/release` imports only the leaf `offlinecleanup` (F9); the reaper removes only directories whose obligation has ended (F10); the cleanup runs under its own `cleanupMu`, outside `w.pass` (F11); the prefix is normalised once (N1); `releasing` is persisted before the unmark and the release, and `cleaning` never re-acquires (N2); placement keeps `SetRecordRegion`'s conditional write, and `fenceRecord` settles an uncertain one (N3). Reviews: rev 1 (`after236/selfserve-plan4-review.md`), rev 2 (`selfserve-plan4-review2.md`), rev 3 (`selfserve-plan4-review3.md`), rev 4 (`selfserve-plan4-review4.md`), rev 5 (`selfserve-plan4-review5.md`). |
| +Rev 4's closed answers stand: one state enum and one transition table (F3); a cleanup entry is never reclaimed by another reserver (F4); the programs file has no reference check and the live link keeps §7.6's link and process checks, with unreadable evidence retried (F8); root authorises through a host-level callback, so `internal/release` imports only the leaf `offlinecleanup` (F9); the reaper removes only directories whose obligation has ended (F10); the cleanup runs under its own `cleanupMu`, outside `w.pass` (F11); the prefix is normalised once (N1); `releasing` is persisted before the unmark and the release, and `cleaning` never re-acquires (N2); placement keeps `SetRecordRegion`'s conditional write, and `fenceRecord` settles an uncertain one (N3). Reviews: rev 1 (`after236/selfserve-plan4-review.md`), rev 2 (`selfserve-plan4-review2.md`), rev 3 (`selfserve-plan4-review3.md`), rev 4 (`selfserve-plan4-review4.md`), rev 5 (`selfserve-plan4-review5.md`), rev 6 (`selfserve-plan4-review6.md`). |
| Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and allocation recovery (`internal/console/lifetimes_create.go`, `lifetimes_recovery.go`, `lifetimes_store.go`), `fenceRecord` (`lifetimes_create.go:551`), the judge (`provision.LifetimeJudge.Observe`), the reservation table (`moves.OfflineActive`, `ParseOfflineActive`, `OfflineActiveBody`, `OfflineSentinel`; the console's `reserveBoxes`/`releaseBoxes` and three-way reclaim in `internal/console/offlinereserve.go`), the lease's tombstone (`internal/lease/tombstone.go`), the versioned store (`lease.VersionedStore`, `internal/lease/versioned.go`), the offline source steps (`internal/host/movesoffline.go`), held publication's freeze and lock (`registry.WithHoldsFrozen`, `internal/host/movesheldlock.go`, `heldStageAuthority` in `internal/host/heldstage.go`), the page outbox (`internal/host/movespaging.go`), the offline CLI (`move.go`) and the operator's `carlos system` verbs (`system.go`, `system_lifetimes.go`). |
| -**Goal:** Twenty-four hours after a move the source box observed as finished, the source box retires the old region's replica from evidence it recorded at `gone`, under a `cleanup` reservation on its own box and on the hostname and a cleaning mark on the replica's lock: key by key with the conditions re-read at each key, each inventoried version deleted by its version id, oldest first, and every delete answered before the mark or the reservation is given up. Root removes the old live link only inside its adoption pass, under a lock no other root publisher can bypass, through a request it copies, authorises from the bucket and answers in its own directory, and only where no other uid can write the link's parent. The obligation's directory goes seven days after it ended. Nothing times out into a release. A rollback to the console before this plan drains the new reservation kinds first. |
| +**Goal:** Twenty-four hours after a move the source box observed as finished, the source box retires the old region's replica from evidence it recorded at `gone`, under a `cleanup` reservation on its own box and on the hostname and a cleaning mark on the replica's lock that has no expiry: key by key with the conditions re-read at each key, each inventoried version deleted by its version id, oldest first, only while the version that was current at `gone` is still current, and every delete answered before the mark or the reservation is given up. Root removes the old live link only inside its adoption pass, under a lock no other root publisher can bypass (on a box where cleanup can run, a pass that cannot take that lock publishes nothing), through a request it copies, authorises from the bucket and answers in its own directory, and only where no other uid can write the link's parent. The obligation's directory goes seven days after it ended. Nothing times out into a release. A rollback to the console before this plan drains the new reservation kinds first, behind an admission barrier. |
| -**Architecture:** At `gone` the sealing source box writes `retired.json` under the move's op directory, for a tracked lifetime only: every version and delete marker (key, version id, ETag) under the normalised hibernate prefix (`lock.json` and `build-state.json` aside), each key's versions in deletion order, and the identities of the root-owned paths whose parent chain is root's alone (any other path is recorded as left). A cleanup goroutine, started by the moves pass and never under its lock, walks every obligation through one transition table: observe the finish, wait 24 hours, reserve the box's key and `host/<hostname>`, mark `lock.json`, delete key by key with the conditions re-read, hand root a request (or skip it when nothing local remains), then persist `releasing`, restore the lifted tombstone, release and end. Supersession is checked at every step and ends the obligation through the same `releasing`, after any unanswered delete is answered under the mark and any root request is cancelled and acknowledged. The lifetime create and placement hold `host/<hostname>` in the same table, and so does a move's selection. The table is never deleted. |
| +**Architecture:** At `gone` the sealing source box writes `retired.json` under the move's op directory, for a tracked lifetime only: every version and delete marker (key, version id, ETag) under the normalised hibernate prefix (`lock.json` and `build-state.json` aside), each key's versions in deletion order, and the identities of the root-owned paths whose parent chain is root's alone (any other path is recorded as left). A cleanup goroutine, started by the moves pass and never under its lock, walks every obligation through one transition table: observe the finish, wait 24 hours, reserve the box's key and `host/<hostname>`, mark `lock.json`, delete key by key with the conditions re-read, hand root a request (or skip it when nothing local remains), then persist `releasing`, restore the lifted tombstone, release and end. The mark never expires: only that `releasing`, or an operator's abandon after the page, lifts it. Supersession is checked at every step and ends the obligation through the same `releasing`, after any unanswered delete is answered under the mark and any root request is cancelled and acknowledged. The lifetime create and placement hold `host/<hostname>` in the same table, and so does a move's selection. The table is never deleted. |
| -**Tech Stack:** Go. `internal/offlinecleanup` (new leaf: the wire types, file identity and the layout check), `internal/moves` (host keys, the table's read and write, the conflict relation, `Reserve`/`Release`, `RecordAway`), `internal/lease` (the cleaning mark), `internal/host` (the dispatcher, the delete primitive, the report, the reaper, root's authority and the agent's root half, the root pass lock's wiring), `internal/release` (root's request handler, the root pass lock), `internal/registry` (`WithHostFrozen`, `SetPlacementContext`), `internal/console` (the create, placement, the reservation wrappers, recovery mode, the sweeps, the cleanup door, the drain), the root package (`move.go`, the CLI's `--log`; `system.go` and `system_reservations.go`, the drain verb). |
| +**Tech Stack:** Go. `internal/offlinecleanup` (new leaf: the wire types, file identity and the layout check), `internal/moves` (host keys, the table's read and write, the conflict relation, `Reserve`/`Release`, `RecordAway`), `internal/lease` (the cleaning mark, and its refusal in `AcquireTTL`, `Tombstone` and `LockState`), `internal/host` (the dispatcher, the delete primitive, the report, the reaper, root's authority and the agent's root half, the root pass lock's wiring and gate, the landing's refusal of the mark), `internal/release` (root's request handler, the root pass lock), `internal/registry` (`WithHostFrozen`, `SetPlacementContext`, the remote dispatcher's deadline), `internal/console` (the create, placement, the reservation wrappers, recovery mode, the sweeps, the cleanup door, the admission barrier and the drain), the root package (`move.go`, the CLI's `--log`; `system.go` and `system_reservations.go`, the drain verb). |
| **Spec:** `docs/superpowers/specs/2026-10-21-self-serve-offline-moves-design.md` §7.6, §7.1 (the `cleanup` reservation), §8 (the cleanup pages), §9 (rollout), §11. Stated deviations: placement and the lifetime create hold the hostname key (Paul, 2026-10-22, amadan #343); deletion by version id (§7.6 amendment, Paul's question 9); the rest are in the Self-review notes. |
| ## Global Constraints |
| -- **Arming and rollout.** The box acquires a new cleanup reservation only with `CARLOS_OFFLINE_CLEANUP=on` in its environment (read by `armOffline` when the agent starts), on an `OfflineCapable` box whose judge is set (`cfg.Judge != nil`, i.e. host claims off, `internal/host/boxjudge.go:21-26`); `retired.json` is written at `gone` whatever the switch says, and an obligation past `finished` runs to its end if the switch is turned off. Order: boxes, then consoles (plan 4 code everywhere), then `CARLOS_OFFLINE_CLEANUP=on` on the boxes. Before the switch goes on in any deployment, a check on dev confirms version-id deletion on the versioned hibernate bucket (Task 4b's dev witness). |
| +- **Arming and rollout.** The box acquires a new cleanup reservation only with `CARLOS_OFFLINE_CLEANUP=on` in its environment (read by `armOffline` when the agent starts, from `/etc/carlos/host.env`, the `EnvironmentFile` of the agent's unit, `internal/host/units/carlos-edge.service:18`; root's adopt units read the same file, `carlos-adopt.service:16`, `carlos-stage.service:30`, `carlos-restart.service:31`, `carlos-config-sync.service:24`, and Task 6b's gate reads the switch from it), on an `OfflineCapable` box whose judge is set (`cfg.Judge != nil`, i.e. host claims off, `internal/host/boxjudge.go:21-26`); `retired.json` is written at `gone` whatever the switch says, and an obligation past `finished` runs to its end if the switch is turned off. Order: boxes, then consoles (plan 4 code everywhere), then `CARLOS_OFFLINE_CLEANUP=on` on the boxes. Before the switch goes on in any deployment, a check on dev confirms version-id deletion on the versioned hibernate bucket (Task 4b's dev witness), and `aws s3api get-bucket-versioning` answers `Enabled` for the hibernate bucket of every region the switch goes on in (tofu sets it, `infra/modules/region-remote-host/main.tf:32-34`; the box roles cannot read the status, so the code checks the listing instead, Task 1). |
| - **Rollback**, in this order and no other: |
| - 1. Unset `CARLOS_OFFLINE_CLEANUP` on every box and restart its agent, so no box acquires again. Wait until no `cleanup` entry is left in `control/offline-moves/active.json`: an obligation already reserved runs to its end (completion or supersession); one waiting for root ends when root answers, or when an operator cancels it by creating `<dataDir>/.offline-moves/<host>/<move>/root-cleanup.cancel` (root then acknowledges `cancelled`, Task 6b); a `finished` obligation whose reservation write was uncertain releases it while unarmed (Task 4a, row 2). |
| - 2. Run `carlos system reservations drain` against the **new** console, before it stops (Task 7c). From its first call the console refuses new creates, placements and self-serve move requests for the rest of its process. It settles every `place` entry by the per-host fence (`fenceRecord`, through `s.settlePlacement`) **before** it releases the entry, runs every `create` entry's allocation recovery to its release, and answers `drained` only when, after a settling write of its own, no `cleanup`, `create` or `place` entry and no open self-serve move is left. It refuses to finish while a self-serve move is in flight: a self-serve move's `move` entries hold its host key, which the old console's own retry reads as a mismatch (`offlinereserve.go:177-183`; an operator move's own retry tolerates it). |
| + 1. Unset `CARLOS_OFFLINE_CLEANUP` on every box and restart its agent, so no box acquires again. Wait until no `cleanup` entry is left in `control/offline-moves/active.json`: an obligation already reserved runs to its end (completion or supersession); one waiting for root ends when root answers, or when an operator cancels it by creating `<dataDir>/.offline-moves/<host>/<move>/root-cleanup.cancel` (root then acknowledges `cancelled`, Task 6b); a `finished` obligation whose reservation write was uncertain releases it while unarmed (Task 4a, row 2); an obligation whose delete has not been answered keeps its entries and the mark until the delete is answered or an operator abandons it after its page (Task 4c), and the wait includes it. Step 1 leaves Task 6b's `gated` marker on each box that was armed, so its adopt passes keep failing closed on a root pass lock fault and keep serving the requests left; an operator may remove `<optDir>/root-cleanup/gated` once no drop and no unacknowledged request is left on the box, which the rollback does not need. |
| + 2. Run `carlos system reservations drain` against the **new** console, before it stops (Task 7c). Its first call sets the console's admission barrier: one flag, checked inside the same critical section in which a create's `BeginAllocation` persists its row, a placement registers in `s.placing` and a self-serve request's `AdmitSelfServeMove` admits, so every admission either committed before the flag (and the drain's reads see it) or is refused after it; there is no check-then-admit gap. The drain then waits on the work itself, not on table entries: every allocation row `PendingAllocations` answers (pending, or terminal with its key not yet released) is driven by recovery to its release, every registered placement settles itself, every unregistered `place` entry is fenced (`fenceRecord`, through `s.settlePlacement`) **before** it is released, and every open self-serve move must end (a self-serve move's `move` entries hold its host key, which the old console's own retry reads as a mismatch, `offlinereserve.go:177-183`; an operator move's own retry tolerates it). It answers `drained` only when, after a settling write of its own and a fresh read, no `cleanup`, `create` or `place` entry, no pending allocation row, no registered placement and no open self-serve move is left. |
| 3. Stop the new console. |
| 4. Only then start the old console. |
| - Why this is safe: the old console reads the table that is left (only `move` entries; its `valid()` takes `move` and `cleanup`, `moves/offline.go:584-587`, and reads `create` or `place` as a malformed table). The old console's writes drop `epoch` and `rev`, and its last release deletes the table (`offlinereserve.go:478`), so an earlier ETag can come back: **the epoch protects only among new writers.** The hazard is a new-protocol acquisition still outstanding when such an ETag returns, and none can be: boxes no longer acquire (step 1), the new console is stopped (step 3) after its drain's settling write (step 2), and a conditional PUT that read an ETag before that write can never land after it. A new-protocol release that lands late writes the table it read with nothing added, so it is harmless. The old create and placement take no hostname key, which is safe once no cleanup holds or can acquire one (step 1). |
| + Why this is safe: the old console reads the table that is left (only `move` entries; its `valid()` takes `move` and `cleanup`, `moves/offline.go:584-587`, and reads `create` or `place` as a malformed table). The old console's writes drop `epoch` and `rev`, and its last release deletes the table (`offlinereserve.go:478`), so a metadata-free body, and its ETag, can come back: **the epoch protects only among new writers.** The hazard is a new-protocol acquisition outstanding with such an ETag as its condition, and none can exist: every new-protocol acquisition is issued only against a body that carries `epoch` and `rev` (a body without them is first normalised by a separate conditional write and read again, `moves.ReadForAcquire`, Task 2a), and an old writer never writes a body with an `epoch`, so the ETag an outstanding acquisition names is never current again once any later write lands, old or new. Boxes no longer acquire (step 1), the drain's barrier and settling write end the console's own acquisitions (step 2), and the new console is stopped (step 3). A new-protocol release that lands late writes the table it read with its own entries removed and nothing added, so it is harmless. The old create and placement take no hostname key, which is safe once no cleanup holds or can acquire one (step 1). |
| - **Tracked lifetimes only.** A cleanup obligation exists only for a tracked lifetime: the judged observation at `gone` is `LifetimeValid` with `Rec.Incarnation > 0` and a live slot at that incarnation. A record with incarnation 0 and no slot (an untracked lifetime, before the lazy migration; `provision.LifetimeOf`, `internal/provision/lifetime.go:127-130`, still answers it `LifetimeValid`), a migrating record, or any other verdict gets no `retired.json`, and its gone report proceeds exactly as before this plan; nothing is ever deleted for it, and its leftovers stay the runbook's. |
| -- **Every data deletion names one inventoried version (spec §7.6 amendment, Paul's question 9).** `retired.json`'s `BucketInventory` records, for every key under the normalised prefix except `lock.json` and `build-state.json`, **every** version id the bucket lists at `gone`, delete markers included, not only the current one. The cleanup deletes only through `deleteInventoriedVersion` (Task 4b), which is `VersionedStore.DeleteVersion(ctx, bucket, key, versionID)` (`internal/lease/versioned.go:261-275`: an empty id sends nothing; success and `NoSuchVersion` are both nil) on `cfg.Versioned(ctx, Bucket, BucketRegion)`. A plain `Delete`, a marker-writing delete and an `If-Match` delete are never used. Within a key, the versions are deleted oldest first, and the version that was current at `gone` last. A version is **done** when a delete of it answers (success or no such version); a version the listing no longer shows is done (someone with authority removed it). **The argument, once:** a delete that names a version id can remove only that version, whoever sends it and whenever it lands, so a late attempt, an SDK retry (`internal/lease/endpoint.go:103` keeps the default retries) or a crash-repeated request can never remove a version a later owner wrote, which carries another id. Deleting a version that is not current changes no current version. Versions are only ever added on top, and the inventory holds every version that existed at `gone`, so by the time the version that was current at `gone` is deleted every older one is gone: no version older than the inventoried current one is ever exposed. That is the risk §7.6 avoids by writing markers, and this order removes it. **The one case a version id does not settle:** a later owner that keeps one of our versions instead of writing its own. A promotion does exactly this when the destination's ETag equals the staged one (`lease.MirrorPrefixKeepObserved`, `mirror.go:285`), and the target landing of a move into this region mirrors the same way. Both need the lease or the hostname key, which the mark and the reservation exclude. So a delete sent without an answer is recorded (`InFlight`) before it is sent and sent again until it answers, **before** the obligation leaves `cleaning` (Task 4c), and only while rows 3 and 4 find both entries and the mark this cleanup's. When either is lost first, nothing more is sent; the key is `unsettled` and paged. Once a version is done, any attempt still in the network lands on nothing, because a version id is never reused. **The trade:** the retired replica is not recoverable from the bucket after the cleanup. It is the source copy of a finished move, and litestream's retention would purge it anyway. In exchange, the boundary is enforced by the server. `lock.json` and `build-state.json` are never inventoried and never deleted. |
| -- **The table is never deleted.** `control/offline-moves/active.json` is created once, by the first writer, as `{v: 1, epoch: <16 random bytes, hex>, rev: 1}` with `If-None-Match: *` (`release.Store.Put` with `ifMatch ""`, `internal/release/store.go:794-798`); emptiness is a table with no entries. Every later write is a `Put` with `If-Match` the ETag it read, of a body whose `rev` is the read body's plus one and whose `epoch` is the read body's (a fresh one when the read body has none, as an older console's writes do), so no two bodies the table ever holds are equal and an ETag never comes back. A conditional `Put` that read ETag E can land only while E is current, so once any later write succeeds, every earlier outstanding `Put` that read E can never land: an uncertain acquisition is settled by the next successful conditional write by anyone. A release always writes (rev + 1), even when its operation holds nothing, so its own success is the settlement its caller needs. This holds among new writers only; the rollback procedure covers the old ones. |
| +- **Every data deletion names one inventoried version (spec §7.6 amendment, Paul's question 9).** `retired.json`'s `BucketInventory` records, for every key under the normalised prefix except `lock.json` and `build-state.json`, **every** version id the bucket lists at `gone`, delete markers included, not only the current one, with each entry's position in the listing (`Seq`) and the key's latest version id (`Latest`: the entry the listing marks `IsLatest`, the one current at `gone`). Every id is a real one: a `null` id (an object written while the bucket was unversioned or versioning was suspended, which a later suspended write overwrites in place) fails `lease.UsableVersionID`'s rule (`internal/lease/staging.go:414`), and a listing that holds one writes no `retired.json` (Task 1). The cleanup deletes only through `deleteInventoriedVersion` (Task 4b), which is `VersionedStore.DeleteVersion(ctx, bucket, key, versionID)` (`internal/lease/versioned.go:261-275`: an empty id is `ErrNoVersionID` without a request; success and `NoSuchVersion` are both nil) on `cfg.Versioned(ctx, Bucket, BucketRegion)`. A plain `Delete`, a marker-writing delete and an `If-Match` delete are never used. Within a key, the versions that were not current are deleted first, oldest first (`LastModified` ascending, ties by `Seq` descending: an order fixed at `gone` and recorded), and `Latest` last. **Condition (4) is the latest's:** before each deletion of a key's version, a fresh listing of the key must show `Latest` as its current entry; when `Latest` is absent, or another entry is current (a later version, or a marker the inventory does not hold), the key is `superseded` and nothing more of it is deleted. A version is **done** when a delete of it answers (success or no such version); a version other than `Latest` that the listing no longer shows is done (removed by someone with authority, or by a lifecycle rule for noncurrent versions). **The argument, once:** a delete that names a version id can remove only that version, whoever sends it and whenever it lands, so a late attempt, an SDK retry (`internal/lease/endpoint.go:103` keeps the default retries) or a crash-repeated request can never remove a version a later owner wrote, which carries another id. While `Latest` is current, deleting any other version changes nothing current, so a deletion sent after a check that found `Latest` current exposes no older version unless `Latest` stops being current between that check and the delete's landing. Nothing the platform runs can do that: a write needs the lease, which the mark refuses; a plain delete writes a marker on top, which is then current and hides everything beneath it; a lifecycle rule for noncurrent versions never removes a current one. Only a version-id delete of `Latest` by hand, concurrent with the cleanup, can, and the plan claims nothing about that case. **The one case a version id does not settle:** a later owner that keeps one of our versions instead of writing its own. A promotion does exactly this when the destination's ETag equals the staged one (`lease.MirrorPrefixKeepObserved`, `mirror.go:285`), and the target landing of a move into this region mirrors the same way. Both need the lease, or a lock the landing reads as free, and the mark refuses both for as long as it stands; it stands until every delete is answered. A delete is recorded (`InFlight`) before it is sent and sent again until it answers, **before** the obligation leaves `cleaning` (Task 4c), whatever has happened to the reservation meanwhile; an `InFlight` delete that cannot be answered keeps the mark and the entries the obligation still holds, and pages every 10 minutes, until it is answered or an operator abandons it after the page (Task 4c). Once a version is done, any attempt still in the network lands on nothing, because a version id is never reused. **The trade:** the retired replica is not recoverable from the bucket after the cleanup. It is the source copy of a finished move, and litestream's retention would purge it anyway. In exchange, the boundary is enforced by the server. `lock.json` and `build-state.json` are never inventoried and never deleted. |
| +- **The table is never deleted.** `control/offline-moves/active.json` is created once, by the first writer, as `{v: 1, epoch: <16 random bytes, hex>, rev: 1}` with `If-None-Match: *` (`release.Store.Put` with `ifMatch ""`, `internal/release/store.go:794-798`); emptiness is a table with no entries. Every later write is a `Put` with `If-Match` the ETag it read, of a body whose `rev` is the read body's plus one and whose `epoch` is the read body's (a fresh one when the read body has none, as an older console's writes do), so no two bodies the table ever holds are equal and an ETag never comes back. A conditional `Put` that read ETag E can land only while E is current, so once any later write succeeds, every earlier outstanding `Put` that read E can never land: an uncertain acquisition is settled by the next successful conditional write by anyone. A release always writes (rev + 1), even when its operation holds nothing, so its own success is the settlement its caller needs. **Acquisitions only against an epoch-bearing body (T4).** A write that adds entries (`moves.Reserve`, and so the box, the create and the placement; the console's `reserveBoxes`) reads through `moves.ReadForAcquire`: a body without `epoch` or `rev` (an older console's) is first normalised by a separate `Put` with `If-Match` the read ETag, of the same entries with a fresh `epoch` and `rev` 1, then read again, and only a body that carries both is ever an acquisition's condition. An old writer never writes a body with an `epoch`, so the ETag an acquisition names is never current again after any later write, old or new. Among new writers every write has this property; against old writers acquisitions have it through the normalisation, and the rollback procedure covers the rest. |
| - **The prefix (N1).** `Prefix = activator.HibernatePrefix(cfg.OwnPrefix, host) + "/"` (`HibernatePrefix` is `path.Join`, no trailing slash, `internal/activator/activator.go:147`), computed once at `gone` and persisted; the listing is `ListVersions(bucket, Prefix)`; the excluded keys are exactly `Prefix + "lock.json"` and `Prefix + "build-state.json"`; the lease location is `strings.TrimSuffix(Prefix, "/")`. |
| - **The cleanup id** is `offlinecleanup.CleanupID(box, host, moveID) = "cu_" + hex(sha256(box + "\n" + host + "\n" + moveID))[:16]`: recomputable from the directory and the box, so a corrupt file, root and the agent all name the same obligation. |
| -- **The mark.** Owner `lease.CleaningOwner(id) = "cleaning:" + id`; TTL `cleanupMarkTTL = 1 h`; reasserted whenever less than 30 min remain, in every state from the mark to the unmark; condition (2) requires at least `cleanupMarkMargin = 15 min` left, a bound that keeps the mark from lapsing inside a run. The mark excludes new wakes and replications for the whole run, so no version the run deletes can become a later owner's during it (a late attempt is safe by its version id, above). Written only by a conditional PUT from the lifted tombstone of this move (`moves.TombstoneOwner(MoveID)`, expired); removed by a conditional PUT back to exactly that lifted tombstone (`{Generation: as read, ExpiresAt: 1970-01-01T00:00:00Z, Owner: moved:<move id>}`, what `LiftTombstone` writes), so the operator return's leftover set (spec §7.5) is unchanged after any cleanup. An expired mark whose bytes are unchanged still proves no lease was taken since, because every acquisition rewrites the object. |
| -- **The writers the mark excludes** (each writer's check, by file): the wake (`internal/activator/backend_exec.go:701`, `lease.Acquire` in `ExecBackend.Acquire`, step 1 of `doWake` in `activator.go`, before the restore, the staged-replica promotion `promoteStagedReplica` at `backend_exec.go:3397`, `BeforeExec`'s build-state write and litestream replication, all of which run under that lease); the legacy final checkpoint (`internal/host/movespass.go:321-328`, `lease.ReplicateWithLease`, which acquires); the offline seal's checkpoint (`internal/host/movesoffline_seal.go:417`, the same `Checkpoint`) and its tombstone (`movesoffline_seal.go:431`, `lease.Tombstone`, which never replaces a live lease of another owner, so the seal's carry at `:469` is not reached). Not excluded by the mark and harmless: the legacy cut's request carry (`movespass.go:1594`) and the offline seal's carry write only `build-state.json`, which is not inventoried; the seed rollback and finish (`seedwiring.go:247`, `console/seedfinish.go:254`) only delete; the target landing (`movesoffline_land.go:611`, `:1373`) belongs to a move into this region, which holds `host/<hostname>`. `TestHibernatePrefixSitesAreInventoried` (Task 3) pins this list. |
| +- **The mark has no expiry (T1).** Owner `lease.CleaningOwner(id) = "cleaning:" + id`, written with `ExpiresAt = lease.CleaningExpiry` (`9999-12-31T00:00:00Z`, a time no clock reaches: `litestream.Lease` needs an expiry and `readLock` refuses a zero one, `internal/lease/tombstone.go:140`). It is a state, not a timed lease: written only by a conditional PUT from the lifted tombstone of this move (`moves.TombstoneOwner(MoveID)`, expired); never renewed (there is no reassert); and refused to every lease acquirer whatever its expiry says. litestream's own `AcquireLease` refuses it on the read that conditions its write (`litestream@v0.5.14/s3/leaser.go:91-101`, then the `If-Match` write at `:114`), and `lease.AcquireTTL` (so `Acquire`, `ReplicateWithLease` and the staging lease), `lease.Tombstone` and `lease.LockState` refuse, or report held, any `cleaning:` owner by its prefix, so even a mark whose expiry was edited by hand is never free (Task 3). It is lifted only (a) by the cleanup's own `releasing`, after every delete of the run is answered; (b) by supersession or a cancellation, and only when no delete is `InFlight`; or (c) by an operator's abandon (`cleanup.abandon`, Task 4c) after the page that an outstanding delete is unsettled. A crash, an outage of any length, a disarm or a lost reservation lifts nothing. **So no wake, landing, return or seal of any owner can take the lease while a delete the cleanup sent may still land**, whatever happens to the cleanup's process; that is what makes an in-place adoption of one of our versions (above) impossible during the cleanup. The mark is removed by a conditional PUT back to exactly that lifted tombstone (`{Generation: as read, ExpiresAt: 1970-01-01T00:00:00Z, Owner: moved:<move id>}`, what `LiftTombstone` writes), so the operator return's leftover set (spec §7.5) is unchanged after any cleanup. The owner alone proves no lease was taken since the mark, because every acquisition rewrites the object. |
| +- **The writers the mark excludes** (each writer's check, by file): the wake (`internal/activator/backend_exec.go:701`, `lease.Acquire` in `ExecBackend.Acquire`, step 1 of `doWake` in `activator.go`, before the restore, the staged-replica promotion `promoteStagedReplica` at `backend_exec.go:3397`, `BeforeExec`'s build-state write and litestream replication, all of which run under that lease); the legacy final checkpoint (`internal/host/movespass.go:321-328`, `lease.ReplicateWithLease`, which acquires); the offline seal's checkpoint (`internal/host/movesoffline_seal.go:417`, the same `Checkpoint`) and its tombstone (`movesoffline_seal.go:431`, `lease.Tombstone`, which never replaces a live lease of another owner, so the seal's carry at `:469` is not reached). Not excluded by the mark and harmless: the legacy cut's request carry (`movespass.go:1594`) and the offline seal's carry write only `build-state.json`, which is not inventoried; the seed rollback and finish (`seedwiring.go:247`, `console/seedfinish.go:254`) only delete, by key (a marker, which condition (4) reads as another current entry), and the finish refuses a held lock (`seedfinish.go:255-259`, which `LockState` answers held for a mark); the target landing (`movesoffline_land.go:611`, `:1373`) belongs to a move into this region, which holds `host/<hostname>`, and its lease step waits on a `cleaning:` owner whatever its expiry (`movesoffline_land.go:612-630`, Task 3); the operator return's start door refuses any lock that is not this move's lifted tombstone (`console/offlinereturn.go:465`). `TestHibernatePrefixSitesAreInventoried` (Task 3) pins this list. |
| - **Reservation keys and the one conflict relation.** Keys are a box id (today's bare label, unchanged), the sentinel `"*"`, or `moves.HostKey(host) = "host/" + host`. Kinds: `move`, `cleanup`, `create`, `place`. An entry is valid with a known kind, a non-empty id and host, a non-zero `reserved_at`, and, for `create` and `place`, a non-empty account and app; an entry under a host key names that host (`Host == strings.TrimPrefix(key, "host/")`). An existing entry of another `(Kind, ID)` conflicts with a requested set of keys when (a) it holds a requested key; or (b) one side holds `"*"` and the other holds a box key; or (c) the requester is the console with `cfg.SelfServeMoves != "offline"` and both are of kind `move` (plan 2's one-move-at-a-time rule, now between moves only). Host keys never meet (b) or (c). Every move selection (self-serve and operator) holds `host/<hostname>` beside its box keys or its sentinel; a cleanup holds `[cfg.BoxID, host/<hostname>]` with one entry; a create holds `[host/<hostname>]`; a placement holds `host/<h>` for every instance of the app, one entry per key with `Host = h` and the placement's account and app. |
| - **A cleanup entry is never reclaimed by another reserver.** Its own box resumes it after any crash (its own retry is accepted); a box that never returns leaves an entry the console pages after 6 hours (operator repair). Plan 2's `Server.cleanupReports` and `cleanupReportEvidence` (`offlinereserve.go:95-104`) are deleted. |
| -- **The conditions at every key**, re-read before a key's first delete: (1) `moves.RecordAway(obs, MoveID, Incarnation, From) == AwayYes`: the judged observation is `LifetimeValid` (the judge's verdict that the slot vouches for the record) at `Incarnation`, with no move, `LastMove.ID == MoveID && LastMove.Outcome == "done"`, and `Region != From`; (2) `lock.json`'s owner is `CleaningOwner(ID)` with at least `cleanupMarkMargin` left after any reassert; (3) the box's registry has no row for the host (`w.offlineRow(host)`, `movesoffline.go:996`); (4) `ListVersions(bucket, key)` filtered to `Key == key` shows as current either one of the key's inventoried versions or a delete marker the inventory does not hold (someone deleted the key; the versions beneath it are ours). A read error in any: the obligation stops for this run. (1), (2) or (3) failing definitely: **supersession** (spec §7.6's "superseded as a whole"; never an indefinite refusal). (4) failing, an object version the inventory does not hold being current: that key `superseded`, never touched, and a page. Conditions (1) and (3) are also checked at every other step of the lifecycle (Task 4a's `supersededBy`). |
| +- **The conditions at every key**, (1) to (3) re-read before a key's first delete and (4) before each of its deletes: (1) `moves.RecordAway(obs, MoveID, Incarnation, From) == AwayYes`: the judged observation is `LifetimeValid` (the judge's verdict that the slot vouches for the record) at `Incarnation`, with no move, `LastMove.ID == MoveID && LastMove.Outcome == "done"`, and `Region != From`; (2) `lock.json`'s owner is `CleaningOwner(ID)` (its expiry is the sentinel; nothing about time is read); (3) the box's registry has no row for the host (`w.offlineRow(host)`, `movesoffline.go:996`); (4) `ListVersions(bucket, key)` filtered to `Key == key` shows the key's inventoried `Latest` as its current entry. A read error in any: the obligation stops for this run. (1), (2) or (3) failing definitely: **supersession** (spec §7.6's "superseded as a whole"; never an indefinite refusal). (4) failing (`Latest` absent, or a later version or a marker the inventory does not hold current): that key `superseded`, nothing more of it sent, and a page. Conditions (1) and (3) are also checked at every other step of the lifecycle (Task 4a's `supersededBy`). |
| - The cleanup writes nothing to the registry, the edges file or the offline view, so plan 3's holding entries (built from held rows and the view) are untouched; root's freeze holds the registry's write lock for an `fstatat` and an `unlinkat`, as held publication's does for its swap. |
| -- **Root's contract.** Root removes a path only (i) when the path's parent directory and every ancestor up to `/` is a directory owned by uid 0 with no group or other write bit (`offlinecleanup.RootOnlyChain`; otherwise the path is `kept`, reason "writable layout", because no exclusion against a writer that shares the agent's uid is possible there by construction); (ii) **as a stage of the root adoption pass** (`runAdoptUnderRootRelease`), after release adoption and named recovery and after the held stage, while the pass holds the adopt-pass lock and the root pass lock (`release.LockRootPass`, `<optDir>/root-cleanup/pass.lock` on a root-only chain, which every adopt pass takes in every mode before its first publisher), and then inside `reg.WithHostFrozen(host, …)` (no row for the host) and the host's held-stage lock (`withHeldStageLock`, `movesheldlock.go:68`), in the order adopt pass lock > root pass lock > the registry's write lock > the held-stage lock (`movesheldlock.go:26-31` gains the root pass lock); so no other root publisher (adoption, named recovery, `PointLive`) runs on the box while root's step runs, and the agent's publishers are excluded by the freeze and the held-stage lock; (iii) after the authority callback has answered yes for this request in this pass and `root-cleanup.cancel` is absent; (iv) when `fstatat(parent, name, AT_SYMLINK_NOFOLLOW)` on the opened parent shows the recorded identity (device, inode, file type, link target); then `unlinkat(parent, name)`. An entry with another identity is `kept`, never removed. Every step is journalled as intent and then result in root's own directory, which is used only when it passes the same root-only check; when it does not, root writes nothing there (Task 6b). Nothing inside the held-stage lock reads the network or takes a lock. |
| -- Timings: 24 h after `FinishedAt`; per run 60 s (`cleanupRunDeadline`), 10 s per operation, at most 50 keys per obligation; pages every 10 min while root has not answered or has not acknowledged a cancel, hourly once root has not answered for an hour and its request directory's ancestry is not root-only (`rootRefusedAfter`), hourly for a corrupt obligation and for a record unreadable for over an hour; the console pages a `cleanup` entry older than 6 h and a `create` entry with no allocation row older than `offlineLockGrace` (2 min); a placement runs under one 15 s deadline (`instanceStoreTimeout`), and an unregistered `place` entry older than `offlineLockGrace` is settled by the sweep; the op directory goes 7 d after `EndedAt`; root removes its own directory 30 d after its acknowledgement. |
| -- Census: `internal/provision/testdata/plan14-root-reads-self-serve-moves-4.tsv`, loaded by a new `loadRootReadInventory(root+"/internal/provision/testdata/plan14-root-reads-self-serve-moves-4.tsv", &inv, true)` block in `internal/provision/plan14_root_reads_test.go` after plan 3's block (`:1383`) and its seal check and before `compareRootReads`, `rev3=PROPOSED-self-serve-moves-4`, rows at the `authority` class with evidence `E2,J3,J4` for the destructive readers (the box's record, registry, lock and versions reads; root's drop, its own request copy and journal, `active.json`, record, lock, registry-under-freeze, the layout `lstat` chain, the root pass lock's chain and the `fstatat` of each path); `TestPlan9B8RecordWriteCensus` (`plan9_b8_census_test.go:372`) takes `plan14SelfServeMoves4WriteCensusInventory()` wrapping `plan14SelfServeMoves3WriteCensusInventory()` (`self_serve_moves3_census_test.go:22`) with every new or changed bucket writer as an added or re-pinned unit (`moves.ReadActive`'s bootstrap, `moves.Reserve`, `moves.Release`, `reserveBoxes`, `releaseBoxes`, the drain's settling `Release`, the cleanup report writer, the cleanup's `DeleteVersion` through `deleteInventoriedVersion`; the placement settle, the place sweep and the drain reach `fenceRecord`'s existing record unit); both green. |
| -- Copy rows (pending Paul's review; each task that emits one appends its row, at the next free `MV-O` id, status pending, to `docs/superpowers/plans/2026-10-06-moves-phase1-copy.md` and its id to `movesCopyIDs` in `approved_moves_copy_test.go`, in the same commit): "the previous instance at this hostname is still being cleaned up. try again later"; "an instance of this app is still being cleaned up. try again later"; "an instance of this app is moving. try again later"; "could not reserve the app's instances. try again"; "this console is being rolled back. try again in a few minutes"; page subjects "cleanup of <host> waits for root", "cleanup of <host> skipped <key>: another version is current", "cleanup of <host> waits for root to acknowledge its cancellation", "cleanup of <host> cannot read its record", "a cleanup has held <host> for over six hours", "cleanup of <host>: root refused the request directory's ancestry", "cleanup of <host> lost its mark before a delete of <key> was answered", each with the body "run carlos move offline status --host <host> --log --cleanup <cleanup id> to see what it waits for" (the console's "held for over six hours" page names the entry's id the same way); the CLI's lines of Tasks 5b and 7c; the `--log` paragraph and the `reservations drain` line of `docs/site/cli.md`. No tenant names in tests (`shop.example.com`, account `bab`, app `shop`). |
| +- **Root's contract.** Root removes a path only (i) when the path's parent directory and every ancestor up to `/` is a directory owned by uid 0 with no group or other write bit (`offlinecleanup.RootOnlyChain`; otherwise the path is `kept`, reason "writable layout", because no exclusion against a writer that shares the agent's uid is possible there by construction); (ii) **as a stage of the root adoption pass** (`runAdoptUnderRootRelease`), after release adoption and named recovery and after the held stage, while the pass holds the adopt-pass lock and the root pass lock (`release.LockRootPass`, `<optDir>/root-cleanup/pass.lock` on a root-only chain, which every adopt pass takes in every mode before its first publisher; on a box where cleanup can run, Task 6b's gate, a pass that cannot take it publishes nothing that pass, so there every root publisher of the live link runs under it), and then inside `reg.WithHostFrozen(host, …)` (no row for the host) and the host's held-stage lock (`withHeldStageLock`, `movesheldlock.go:68`), in the order adopt pass lock > root pass lock > the registry's write lock > the held-stage lock (`movesheldlock.go:26-31` gains the root pass lock); so no other root publisher (adoption, named recovery, `PointLive`) runs on the box while root's step runs, and the agent's publishers are excluded by the freeze and the held-stage lock; (iii) after the authority callback has answered yes for this request in this pass and `root-cleanup.cancel` is absent; (iv) when `fstatat(parent, name, AT_SYMLINK_NOFOLLOW)` on the opened parent shows the recorded identity (device, inode, file type, link target); then `unlinkat(parent, name)`. An entry with another identity is `kept`, never removed. Every step is journalled as intent and then result in root's own directory, which is used only when it passes the same root-only check; when it does not, root writes nothing there (Task 6b). Nothing inside the held-stage lock reads the network or takes a lock. |
| +- Timings: 24 h after `FinishedAt`; per run 60 s (`cleanupRunDeadline`), 10 s per operation, at most 50 keys per obligation; pages every 10 min while root has not answered or has not acknowledged a cancel, and while a delete has waited over 10 min for its answer (`InFlight`), hourly once root has not answered for an hour and its request directory's ancestry is not root-only (`rootRefusedAfter`), hourly for a corrupt obligation and for a record unreadable for over an hour; the console pages a `cleanup` entry older than 6 h and a `create` entry with no allocation row older than `offlineLockGrace` (2 min); a placement runs under one 15 s deadline (`instanceStoreTimeout`), and an unregistered `place` entry older than `offlineLockGrace` is settled by the sweep; the op directory goes 7 d after `EndedAt`; root removes its own directory 30 d after its acknowledgement. |
| +- Census: `internal/provision/testdata/plan14-root-reads-self-serve-moves-4.tsv`, loaded by a new `loadRootReadInventory(root+"/internal/provision/testdata/plan14-root-reads-self-serve-moves-4.tsv", &inv, true)` block in `internal/provision/plan14_root_reads_test.go` after plan 3's block (`:1383`) and its seal check and before `compareRootReads`, `rev3=PROPOSED-self-serve-moves-4`, rows at the `authority` class with evidence `E2,J3,J4` for the destructive readers (the box's record, registry, lock and versions reads and the abandon file; root's drop, its own request copy and journal, the gate marker, `active.json`, record, lock, registry-under-freeze, the layout `lstat` chain, the root pass lock's chain and the `fstatat` of each path); `TestPlan9B8RecordWriteCensus` (`plan9_b8_census_test.go:372`) takes `plan14SelfServeMoves4WriteCensusInventory()` wrapping `plan14SelfServeMoves3WriteCensusInventory()` (`self_serve_moves3_census_test.go:22`) with every new or changed bucket writer as an added or re-pinned unit (`moves.ReadActive`'s bootstrap, `moves.ReadForAcquire`'s normalisation write, `moves.Reserve`, `moves.Release`, `reserveBoxes`, `releaseBoxes`, the drain's settling `Release`, the cleanup report writer, the cleanup's `DeleteVersion` through `deleteInventoriedVersion`; the placement settle, the place sweep and the drain reach `fenceRecord`'s existing record unit); both green. |
| +- Copy rows (pending Paul's review; each task that emits one appends its row, at the next free `MV-O` id, status pending, to `docs/superpowers/plans/2026-10-06-moves-phase1-copy.md` and its id to `movesCopyIDs` in `approved_moves_copy_test.go`, in the same commit): "the previous instance at this hostname is still being cleaned up. try again later"; "an instance of this app is still being cleaned up. try again later"; "an instance of this app is moving. try again later"; "could not reserve the app's instances. try again"; "this console is being rolled back. try again in a few minutes"; page subjects "cleanup of <host> waits for root", "cleanup of <host> skipped <key>: another version is current", "cleanup of <host> waits for root to acknowledge its cancellation", "cleanup of <host> cannot read its record", "a cleanup has held <host> for over six hours", "cleanup of <host>: root refused the request directory's ancestry", "cleanup of <host> lost its mark before a delete of <key> was answered", "cleanup of <host>: an outstanding delete is unsettled; the host's replica stays marked", each with the body "run carlos move offline status --host <host> --log --cleanup <cleanup id> to see what it waits for" (the console's "held for over six hours" page names the entry's id the same way); the CLI's lines of Tasks 5b and 7c; the `--log` paragraph and the `reservations drain` line of `docs/site/cli.md`. No tenant names in tests (`shop.example.com`, account `bab`, app `shop`). |
| - Local tests: touched packages with `-run`; CI runs the full suite and `go test -tags e2e ./internal/e2e/...`. |
| ## Review Focus |
| -1. The F1 and N4 interleavings by version id: a reconcile paused with an old observation mints the source row after the cleanup marked the lock, and a wake through `ExecBackend.Acquire` then tries the lease: it gets `HeldError` naming `cleaning:<id>`, no version is written under the prefix, the next key's condition (3) supersedes, the lifted tombstone is restored, the reservation released, and the wake then succeeds. A delete sent before the supersession and still unanswered is sent again under the mark and answered before `releasing` is written; an earlier attempt of it delivered after the release lands on nothing. Two attempts of one delete (an SDK retry) remove one version. No partial run ever leaves a version older than the inventoried current one current. A promotion back into the region that kept an inventoried version by its equal ETag finds it intact after the cleanup ended (Tasks 4b and 4c, `TestAdmittedReconcileAndWakeCannotWriteUnderTheMark`, `TestUnansweredDeleteIsAnsweredBeforeTheUnmark`, `TestRetriedAttemptsRemoveOnlyTheirVersion`, `TestDeletionNeverExposesAnOlderVersion`, `TestPromotionThatKeptAnInventoriedVersionKeepsIt`). |
| -2. A crash after every write and every side effect of the lifecycle (after `Reserve` before `cleaning`, after `cleaning` before the mark, after the mark before `MarkedAt`, after `InFlight` is written before its delete, after a delete before `Done` moves, after the last key before `publishing`, after `releasing` before the unmark, after the unmark before the release, after the release before `ended`), restart, completion: no version is deleted unless both entries and the mark are found to be this cleanup's, an entry an operator removed or a mark replaced turns the obligation `superseded` with any unanswered delete recorded `unsettled`, nothing is re-acquired from `cleaning`, and the outcome is the table's (Task 4a, `TestLifecycleResumesAtEveryBoundary`). |
| -3. Root under attack and crash: a shared-uid writer plants an acknowledgement, a request with another id, a changed copy of the request and a cancel in the agent's directory, and replaces `/data/adopt-pass.lock` with a symlink so a second pass runs without that lock; a publication replaces the live link between root's reference check and its `fstatat`; a second adopt pass starts between root's `fstatat` and its `unlinkat`; a route is upserted during the freeze; the link's parent chain is writable; root's request directory's ancestry is writable; root crashes after each journal line and each syscall: root removes only an entry carrying the recorded identity, the replacement survives, the second pass waits on the root pass lock until the step returns, the upsert waits for the freeze, the writable layout is never touched, a rejected ancestry gets no write, every intent resolves to removed, kept or absent, root finds its own taken request on restart, and the forged files change nothing but stopping (Tasks 6b and 6c, `TestRootRemovesOnlyTheRecordedIdentity`, `TestNoPublisherRunsBetweenFstatatAndUnlinkat`, `TestWritableLayoutIsNeverRemoved`, `TestRejectedAncestryGetsNoWrite`, `TestRootJournalResolvesEveryIntent`, `TestRootFindsItsOwnRequestsOnRestart`, `TestForgedAgentFilesAreHarmless`). |
| -4. Residence changes and cleanup exclude each other through `host/<hostname>`: a create (busy → 409, row void, no reservation written), a crash after `BeginAllocation` and after `Reserve`, a terminal row whose release failed, a create's reservation PUT that lands after its recovery voided and released it (refused by its condition); a placement that is busy (nothing written), whose record PUT commits after the client gave up (fenced before release), that pauses after its reservation for longer than the grace (registered, so never swept; its local writes fail at its deadline) and whose console crashed (the sweep fences and releases); an offline move selection; with the switch off two creates of different hosts and a cleanup beside an operator move all proceed, and the sentinel never blocks a create; the drain settles every `place` and `create` entry and refuses to finish while a self-serve move or a cleanup holds an entry (Tasks 2a to 2c, 7a, 7b and 7c, `TestConflictRelation`, `TestUncertainReservationIsSettledByTheNextWrite`, `TestResidenceChangesAndCleanupExclude`, `TestPausedPlacementIsNeverSwept`, `TestDrainSettlesBeforeItReleases`). |
| +1. The F1 and N4 interleavings by version id: a reconcile paused with an old observation mints the source row after the cleanup marked the lock, and a wake through `ExecBackend.Acquire` then tries the lease: it gets `HeldError` naming `cleaning:<id>`, no version is written under the prefix, the next key's condition (3) supersedes, the lifted tombstone is restored, the reservation released, and the wake then succeeds. A delete sent before the supersession and still unanswered is sent again under the mark and answered before `releasing` is written; an earlier attempt of it delivered after the release lands on nothing. Two attempts of one delete (an SDK retry) remove one version. The mark has no expiry: a cleanup that stops with a delete in flight and stays down for 25 hours still holds it, a legacy return's wake is refused with `HeldError` naming `cleaning:<id>`, and the resumed cleanup resends the same version id, gets its answer and only then releases; a delete that never answers keeps the mark and pages every 10 minutes until an operator abandons it. No deletion is sent unless a fresh listing found the key's inventoried latest current, so no run leaves a version older than it current; a key whose latest was removed by hand is superseded, not deleted further. A promotion back into the region that kept an inventoried version by its equal ETag finds it intact after the cleanup ended (Tasks 3, 4b and 4c, `TestCleaningMarkIsRefusedWhateverItsAge`, `TestMarkOutlivesACrashWithADeleteInFlight`, `TestUnanswerableDeleteHoldsTheMarkAndPages`, `TestAdmittedReconcileAndWakeCannotWriteUnderTheMark`, `TestUnansweredDeleteIsAnsweredBeforeTheUnmark`, `TestRetriedAttemptsRemoveOnlyTheirVersion`, `TestDeletionNeverExposesAnOlderVersion`, `TestLatestRemovedByHandSupersedesTheKey`, `TestPromotionThatKeptAnInventoriedVersionKeepsIt`). |
| +2. A crash after every write and every side effect of the lifecycle (after `Reserve` before `cleaning`, after `cleaning` before the mark, after the mark before `MarkedAt`, after `InFlight` is written before its delete, after a delete before `Done` moves, after the last key before `publishing`, after `releasing` before the unmark, after the unmark before the release, after the release before `ended`), restart, completion: no first delete of a version is sent unless both entries and the mark are found to be this cleanup's (a resend needs only the mark), an entry an operator removed or a mark replaced turns the obligation `superseded`, an unanswered delete is answered under the mark first (only a mark replaced by hand leaves it `unsettled`), nothing is re-acquired from `cleaning`, and the outcome is the table's (Task 4a, `TestLifecycleResumesAtEveryBoundary`). |
| +3. Root under attack and crash: a shared-uid writer plants an acknowledgement, a request with another id, a changed copy of the request and a cancel in the agent's directory, and replaces `/data/adopt-pass.lock` with a symlink so a second pass runs without that lock; on an armed box a second pass's root pass lock fails for a root-level fault; a publication replaces the live link between root's reference check and its `fstatat`; a second adopt pass starts between root's `fstatat` and its `unlinkat`; a route is upserted during the freeze; the link's parent chain is writable; root's request directory's ancestry is writable; root crashes after each journal line and each syscall: root removes only an entry carrying the recorded identity, the replacement survives, the second pass waits on the root pass lock until the step returns, the pass whose root pass lock failed publishes nothing, root creates no directory under an ancestry that has not passed, the upsert waits for the freeze, the writable layout is never touched, a rejected ancestry gets no write, every intent resolves to removed, kept or absent, root finds its own taken request on restart, and the forged files change nothing but stopping (Tasks 6b and 6c, `TestRootRemovesOnlyTheRecordedIdentity`, `TestNoPublisherRunsBetweenFstatatAndUnlinkat`, `TestGatedPassPublishesNothingWithoutTheRootPassLock`, `TestLockRootPassChecksAncestryBeforeCreating`, `TestWritableLayoutIsNeverRemoved`, `TestRejectedAncestryGetsNoWrite`, `TestRootJournalResolvesEveryIntent`, `TestRootFindsItsOwnRequestsOnRestart`, `TestForgedAgentFilesAreHarmless`). |
| +4. Residence changes and cleanup exclude each other through `host/<hostname>`: a create (busy → 409, row void, no reservation written), a crash after `BeginAllocation` and after `Reserve`, a terminal row whose release failed, a create's reservation PUT that lands after its recovery voided and released it (refused by its condition); a placement that is busy (nothing written), whose record PUT commits after the client gave up (fenced before release), that pauses after its reservation for longer than the grace (registered, so never swept; its local writes fail at its deadline) and whose console crashed (the sweep fences and releases); an offline move selection; with the switch off two creates of different hosts and a cleanup beside an operator move all proceed, and the sentinel never blocks a create; the drain settles every `place` and `create` entry and refuses to finish while a self-serve move or a cleanup holds an entry; a create paused between `BeginAllocation` and its reservation and a placement registered and paused before its reservation keep the drain `draining`; an admission and the drain's flag are one critical section; no acquisition ever names a pre-epoch ETag, so an old writer that recreates one cannot revive an outstanding acquisition; a placement paused past its deadline makes no registry update, local or remote (Tasks 2a to 2c, 7a, 7b and 7c, `TestConflictRelation`, `TestUncertainReservationIsSettledByTheNextWrite`, `TestPreEpochETagCannotSatisfyANewAcquisition`, `TestResidenceChangesAndCleanupExclude`, `TestPausedPlacementIsNeverSwept`, `TestRemoteSetPlacementRunsUnderTheReceivedDeadline`, `TestDrainSettlesBeforeItReleases`, `TestDrainCountsWorkWithoutEntries`, `TestAdmissionAndTheDrainFlagAreOneCriticalSection`). |
| 5. Publication is durable and findable: the report under the move id survives failed writes and is republished until `PublishedRev == Rev`; a refusal repeated every run is one event with a count; a skipped key's page survives an outbox that cannot create it, a pager that refuses it and a restart; a periodic page fires once per tick across restarts; the reaper waits for the report, `pages_pending`, the outbox and root's acknowledgement; an older obligation's report is found by its cleanup id after `LastMove` moved on and after the record was deleted (Tasks 5a, 5b and 8, `TestCleanupReportIsDurableAndBlocksTheReaper`, `TestPendingPagesSurviveARestart`, `TestOlderCleanupIsFoundByItsID`). |
| --- |
| @@ -63,49 +65,49 @@ Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and |
| **Interfaces:** |
| - `func CleanupID(box, host, moveID string) string` as the Global Constraint; `const MarkPrefix = "cleaning:"`. |
| -- `type Retired struct { V int; ID, MoveID, Host, Account, App, Box string; Incarnation uint64; From, To string; OpenAt time.Time; FinalTXID uint64; TombstoneOwner string; Bucket, BucketRegion, Prefix string; BucketInventory []InventoryKey; LocalInventory []LocalPath; LocalLeft []KeptPath; WrittenAt time.Time; State, Outcome string; FinishedAt, ReservedAt, MarkedAt, RootRequestedAt, CancelRequestedAt, EndedAt time.Time; ReserveUncertain bool; InFlight *InFlightDelete; Keys map[string]KeyOutcome; Events []Event; PagesPending []PendingPage; Root *RootCleanupAck; Rev, PublishedRev uint64 }`, every field tagged snake_case, times `omitzero`, `V = 1`. |
| +- `type Retired struct { V int; ID, MoveID, Host, Account, App, Box string; Incarnation uint64; From, To string; OpenAt time.Time; FinalTXID uint64; TombstoneOwner string; Bucket, BucketRegion, Prefix string; BucketInventory []InventoryKey; LocalInventory []LocalPath; LocalLeft []KeptPath; WrittenAt time.Time; State, Outcome, Superseding string; FinishedAt, ReservedAt, MarkedAt, RootRequestedAt, CancelRequestedAt, EndedAt time.Time; ReserveUncertain bool; InFlight *InFlightDelete; Keys map[string]KeyOutcome; Events []Event; PagesPending []PendingPage; Root *RootCleanupAck; Rev, PublishedRev uint64 }`, every field tagged snake_case, times `omitzero`, `V = 1`. |
| - **States** (`State`, the only lifecycle field): `open`, `finished`, `cleaning`, `publishing`, `root-requested`, `cancel-pending`, `releasing`, `ended`, `corrupt`. `Outcome` is "" until `releasing`, then `cleaned`, `superseded` or `cancelled` (Task 4a's outcome table). |
| -- `type InventoryKey struct { Key string; Versions []InventoryVersion }` with `Versions` in deletion order; `type InventoryVersion struct { VersionID, ETag string; Size int64; DeleteMarker, Latest bool; LastModified time.Time }` (`ETag` unquoted, as `lease.ObjectVersion` carries it, `internal/lease/versioned.go:25-33`; "" for a marker); `type InFlightDelete struct { Key, VersionID string; At time.Time }`; `type KeyOutcome struct { State string (pending, deleted, superseded, unsettled); Done int (versions, in deletion order, known done); At time.Time }`; `type Event`, `type PendingPage` (Task 5a); `type FileIdentity struct { Dev, Inode uint64; Mode string (symlink or regular); LinkTarget string }`; `type LocalPath struct { Which string (live-link or programs-file); Identity FileIdentity }` (root derives the path itself; the agent's file names none); `type KeptPath struct { Which, Reason string }`. |
| +- `type InventoryKey struct { Key, Latest string; Versions []InventoryVersion }` with `Versions` in deletion order and `Latest` the version id of the last (the one current at `gone`); `type InventoryVersion struct { Seq int; VersionID, ETag string; Size int64; DeleteMarker, Latest bool; LastModified time.Time }` (`Seq` the entry's index in `ListVersions`' answer for the whole prefix; `ETag` unquoted, as `lease.ObjectVersion` carries it, `internal/lease/versioned.go:25-33`; "" for a marker); `type InFlightDelete struct { Key, VersionID string; At time.Time }` (`At` the first send, kept on every resend); `type KeyOutcome struct { State string (pending, deleted, superseded, unsettled); Done int (versions, in deletion order, known done); At time.Time }`; `type Event`, `type PendingPage` (Task 5a); `type FileIdentity struct { Dev, Inode uint64; Mode string (symlink or regular); LinkTarget string }`; `type LocalPath struct { Which string (live-link or programs-file); Identity FileIdentity }` (root derives the path itself; the agent's file names none); `type KeptPath struct { Which, Reason string }`. |
| - `func Identify(path string) (FileIdentity, bool, error)`: `Lstat` (never follows), `Readlink` for a symlink; absent → `false`; any other error fails the step. `func RootOnlyChain(dir string) (bool, error)`: `Lstat` of every component from `/` to `dir`: each a directory (a symlink component answers false), owned by uid 0, `mode & 0o022 == 0`. Off Linux both answer absent and false (the cleanup is Linux-only, as the box is). |
| -- `ValidateRetired(r Retired) error`: `V == 1`; `ID == CleanupID(Box, Host, MoveID)`; `MoveID` a plain segment; `Host` passes the host shape; `Account`, `App`, `Box` non-empty; `Incarnation > 0`; `From != To`; `State` in the set and not `corrupt`; `Outcome` "" before `releasing` and in the set after; `WrittenAt` non-zero; `Prefix` ends with `/`; inventory keys unique, under `Prefix`, neither excluded key; every key has at least one version, version ids non-empty and unique within the key, exactly one `Latest` and it is the last, `ETag` non-empty exactly for the versions that are not markers; `Which` values unique across `LocalInventory` and `LocalLeft`, and known; every `Keys` entry names an inventory key with `Done` in `0..len(Versions)`; `InFlight`, when set, names an inventoried version of a key whose `Keys` state is `pending`. |
| +- `ValidateRetired(r Retired) error`: `V == 1`; `ID == CleanupID(Box, Host, MoveID)`; `MoveID` a plain segment; `Host` passes the host shape; `Account`, `App`, `Box` non-empty; `Incarnation > 0`; `From != To`; `State` in the set and not `corrupt`; `Outcome` "" before `releasing` and in the set after; `Superseding` "" or a supersession reason, and set only in `cleaning` with `InFlight` set; `WrittenAt` non-zero; `Prefix` ends with `/`; inventory keys unique, under `Prefix`, neither excluded key; every key has at least one version; every version id passes `offlinecleanup.UsableVersionID` (non-empty and not `null`: `lease.UsableVersionID`'s rule, `internal/lease/staging.go:414`, restated here because this package imports the standard library only, and pinned to it by `TestUsableVersionIDMatchesTheLease`) and is unique within the key; `Seq` unique across the inventory; exactly one version has `Latest` set, it is the last, and `InventoryKey.Latest` is its id; the versions before it in the recorded order (`LastModified` ascending, ties by `Seq` descending); `ETag` non-empty exactly for the versions that are not markers; `Which` values unique across `LocalInventory` and `LocalLeft`, and known; every `Keys` entry names an inventory key with `Done` in `0..len(Versions)`; `InFlight`, when set, names an inventoried version of a key whose `Keys` state is `pending`. |
| - `type CorruptStub struct { Corrupt bool; ID, Host string; FoundAt time.Time }` (`{corrupt: true, id, host, found_at}`); `ValidateCorruptStub(c, box, host, move string) error`: `Corrupt`, `Host == host`, `ID == CleanupID(box, host, move)`, `FoundAt` non-zero. |
| - `func OfflineOpDir(dataDir, host, op string) string` (`<dataDir>/.offline-moves/<host>/<op>`, "" on a bad name, the checks `OfflineParkDir` applies); `func RetiredFile(dataDir, host, op string) string`; `func CleanupReportKey(host, op, box string) string` (`OfflineReportPrefix(host, op) + "cleanup/" + box + ".json"`). |
| -- **The evidence at `gone`** (`recordRetired`): the record is in `offline-finishing` carrying this move, `cfg.Judge != nil`, and `cfg.Region == m.From` (else nothing is written and the box logs once per day). **Tracked (N5):** `obs := cfg.Judge.Observe(ctx, s, rec.Account, rec.App, host)`; `LifetimeUnreadable` → an error (the step retries next pass, as any read in it does); `obs.Verdict == LifetimeValid && obs.Rec.Incarnation > 0 && obs.Slot != nil && obs.Slot.Incarnation == obs.Rec.Incarnation` → tracked; any other answer (incarnation 0 with no slot, a migrating record, stale, corrupt) → untracked: no `retired.json`, logged once per day, and `recordRetired` returns nil so the gone report is written as before this plan. Then: `OpenAt = *Move.OpenedAt`; `FinalTXID` from the seal at `moves.OfflineSealKey(host, id)` validated by `moves.ValidateOfflineSeal`; `TombstoneOwner = moves.TombstoneOwner(id)`; `Bucket/BucketRegion = cfg.OwnBucket/OwnBucketRegion`; `Prefix` as the Global Constraint; **`BucketInventory`** = `cfg.Versioned(ctx, Bucket, BucketRegion).ListVersions(ctx, Bucket, Prefix)` (any page error fails the whole call, `versioned.go:45-50`), keeping every entry, object version or delete marker, with `strings.HasPrefix(Key, Prefix)` and without the two excluded keys, grouped by key; each key's versions in deletion order: the entries that are not `IsLatest` in the reverse of the listing's order (the listing is newest first with ties unordered, so this is oldest first; a tie's order does not matter, because deleting a version that is not current changes no current version), then the one `IsLatest` entry (`Latest`); for each of `release.LiveLink(cfg.OptDir, host)` and `release.ProgramsPath(cfg.DataDir, host)`: absent → nothing; present and `RootOnlyChain(filepath.Dir(path))` → `LocalInventory`; present otherwise → `LocalLeft` with reason "writable layout" (on every built box the programs file is here: the agent's unit has `ReadWritePaths=/data`); `State = open`, `Rev = 1`. |
| +- **The evidence at `gone`** (`recordRetired`): the record is in `offline-finishing` carrying this move, `cfg.Judge != nil`, and `cfg.Region == m.From` (else nothing is written and the box logs once per day). **Tracked (N5):** `obs := cfg.Judge.Observe(ctx, s, rec.Account, rec.App, host)`; `LifetimeUnreadable` → an error (the step retries next pass, as any read in it does); `obs.Verdict == LifetimeValid && obs.Rec.Incarnation > 0 && obs.Slot != nil && obs.Slot.Incarnation == obs.Rec.Incarnation` → tracked; any other answer (incarnation 0 with no slot, a migrating record, stale, corrupt) → untracked: no `retired.json`, logged once per day, and `recordRetired` returns nil so the gone report is written as before this plan. Then: `OpenAt = *Move.OpenedAt`; `FinalTXID` from the seal at `moves.OfflineSealKey(host, id)` validated by `moves.ValidateOfflineSeal`; `TombstoneOwner = moves.TombstoneOwner(id)`; `Bucket/BucketRegion = cfg.OwnBucket/OwnBucketRegion`; `Prefix` as the Global Constraint; **`BucketInventory`** = `cfg.Versioned(ctx, Bucket, BucketRegion).ListVersions(ctx, Bucket, Prefix)` (any page error fails the whole call, `versioned.go:45-50`), keeping every entry, object version or delete marker, with `strings.HasPrefix(Key, Prefix)` and without the two excluded keys, each with `Seq` its index in the answer, grouped by key. **A `null` id writes nothing:** an entry whose version id `lease.UsableVersionID` refuses (an object written while the bucket was unversioned, or while versioning was suspended, which a later suspended write overwrites in place, so its id does not name one version) → no `retired.json`, logged once per day, and `recordRetired` returns nil, so the gone report proceeds as for an untracked lifetime. The listing is the versioning check: a bucket whose versioning is not enabled lists its objects as `null`, and the box roles cannot read the status itself (`s3:GetBucketVersioning` is granted only in the region host's inventory statement, `infra/modules/region-host/main.tf:223`, and not at all in `infra/modules/region-remote-host/main.tf`). Each key's versions in deletion order: the entries that are not `IsLatest` by `LastModified` ascending, ties by `Seq` descending (the listing is newest first with ties unordered, `versioned.go:46-47`, `:132-141`; the order is recorded, so nothing later depends on how a tie was listed), then the one `IsLatest` entry (`Latest`), whose id is also the key's `InventoryKey.Latest`; a key whose listing has no `IsLatest` entry, or more than one, fails the call (the step retries next pass); for each of `release.LiveLink(cfg.OptDir, host)` and `release.ProgramsPath(cfg.DataDir, host)`: absent → nothing; present and `RootOnlyChain(filepath.Dir(path))` → `LocalInventory`; present otherwise → `LocalLeft` with reason "writable layout" (on every built box the programs file is here: the agent's unit has `ReadWritePaths=/data`); `State = open`, `Rev = 1`. |
| - Writes: `createRetired` is create-only (temp file, `fsync`, `os.Link` into place, directory `fsync`; an existing file is never replaced); `writeRetired` (temp, `fsync`, rename, directory `fsync`) increments `Rev` on every write except `markPublished` (Task 5a), which sets `PublishedRev` alone. |
| - **Corrupt (N6):** a file that is a valid `CorruptStub` stays as it is, never rewritten. A file that is neither a valid `Retired` nor a valid stub is moved aside create-only (`os.Link` then remove): to `retired.json.corrupt-1` when that name is free, else to the next free `retired.json.corrupt-<n>`, so the first damaged file is never overwritten; then the stub is written with `FoundAt` = `retired.json.corrupt-1`'s mtime. A stub keeps the obligation listed as `corrupt` and pages hourly from `FoundAt` (Task 5a) until an operator repairs it (a valid `retired.json` restored from `.corrupt-1`, or the directory removed after the entries and the mark are released by hand); it is never reaped and never released, so any entry or mark it held stays until the repair. |
| -- [ ] **Step 1: Write the failing tests**: `TestRetiredJSONWrittenOnceAtGoneBySealer` (fields; a key with three versions and a key whose current entry is a marker each recorded whole, in deletion order with the latest last and each version's ETag; `FinalTXID` from the seal; a non-sealer source box writes nothing; a second gone pass does not rewrite; an identity error writes nothing and no gone report); `TestUntrackedLifetimeWritesNoRetiredAndReportsGone` (incarnation 0 with no slot, a migrating record, a stale verdict: no file, the gone report written; an unreadable lifetime: no file and no gone report this pass, both next pass); `TestRetiredPrefixIsNormalisedOnce` (`cfg.OwnPrefix = "instances"` → `Prefix == "instances/shop.example.com/"`; the fake versioned store was listed with exactly that prefix; `instances/shop.example.com/lock.json` and `.../build-state.json` excluded with all their versions; `instances/shop.example.co` keys not listed; `ValidateRetired` passes on what gone wrote); `TestInventoryOrderIsOldestFirstLatestLast` (versions with equal `LastModified`: the latest is last whatever the tie order); `TestWritablePathsAreLeftNotRequested` (a path under a directory the test uid owns → `LocalLeft`, "writable layout"; a path under a root-only chain the test cannot build is covered by injecting `RootOnlyChain`); `TestCleanupIDIsDeterministic`; `TestValidateRetired` (each rule); `TestCorruptStubIsValidAndKeepsTheFirstDamage` (damaged → `.corrupt-1` and a stub that validates; a second scan changes nothing; a damaged stub → `.corrupt-2`, `FoundAt` still `.corrupt-1`'s mtime); `TestIdentifyUsesLstat` (a dangling link has an identity; a regular file; absent); `TestRootOnlyChain` (`/usr` true on a normal system; a temp directory false; a symlink component false). |
| -- [ ] **Step 2: Run to verify they fail**: `go test ./internal/offlinecleanup -count=1`, `go test ./internal/host -run 'RetiredJSON|Untracked|RetiredPrefix|InventoryOrder|WritablePaths|CorruptStub' -count=1`. |
| +- [ ] **Step 1: Write the failing tests**: `TestRetiredJSONWrittenOnceAtGoneBySealer` (fields; a key with three versions and a key whose current entry is a marker each recorded whole, in deletion order with the latest last and each version's ETag; `FinalTXID` from the seal; a non-sealer source box writes nothing; a second gone pass does not rewrite; an identity error writes nothing and no gone report); `TestUntrackedLifetimeWritesNoRetiredAndReportsGone` (incarnation 0 with no slot, a migrating record, a stale verdict: no file, the gone report written; an unreadable lifetime: no file and no gone report this pass, both next pass); `TestRetiredPrefixIsNormalisedOnce` (`cfg.OwnPrefix = "instances"` → `Prefix == "instances/shop.example.com/"`; the fake versioned store was listed with exactly that prefix; `instances/shop.example.com/lock.json` and `.../build-state.json` excluded with all their versions; `instances/shop.example.co` keys not listed; `ValidateRetired` passes on what gone wrote); `TestInventoryOrderIsOldestFirstLatestLast` (versions with equal `LastModified`: ordered by `Seq` descending, recorded, and the latest last and named whatever the tie order: `k.Latest == k.Versions[len(k.Versions)-1].VersionID && k.Versions[len(k.Versions)-1].Latest`; a listing with two `IsLatest` entries for one key → an error and no file); `TestNullVersionIDWritesNoRetired` (a listing with a `null` id under the prefix: `recordRetired` returns nil, no file exists, the gone report is written; `ValidateRetired` of an otherwise valid inventory with one `null` id → an error naming it); `TestUsableVersionIDMatchesTheLease` (external test package: for `""`, `"null"`, `"Null"`, `"3HL4kqtJlcpXroDTDmJ+rmSpXd3dIbrHY"`, `offlinecleanup.UsableVersionID(v) == lease.UsableVersionID(v)`); `TestWritablePathsAreLeftNotRequested` (a path under a directory the test uid owns → `LocalLeft`, "writable layout"; a path under a root-only chain the test cannot build is covered by injecting `RootOnlyChain`); `TestCleanupIDIsDeterministic`; `TestValidateRetired` (each rule); `TestCorruptStubIsValidAndKeepsTheFirstDamage` (damaged → `.corrupt-1` and a stub that validates; a second scan changes nothing; a damaged stub → `.corrupt-2`, `FoundAt` still `.corrupt-1`'s mtime); `TestIdentifyUsesLstat` (a dangling link has an identity; a regular file; absent); `TestRootOnlyChain` (`/usr` true on a normal system; a temp directory false; a symlink component false). |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/offlinecleanup -count=1`, `go test ./internal/host -run 'RetiredJSON|Untracked|RetiredPrefix|InventoryOrder|NullVersionID|WritablePaths|CorruptStub' -count=1`. |
| - [ ] **Step 3: Implement. Step 4: Run to verify they pass.** |
| -- [ ] **Step 5: Commit** `moves: the sealing source box records retired.json at gone for a tracked lifetime, with every version id in deletion order, a normalised prefix and a deterministic cleanup id`. |
| +- [ ] **Step 5: Commit** `moves: the sealing source box records retired.json at gone for a tracked lifetime, with every real version id in a recorded deletion order and the key's latest named, a normalised prefix and a deterministic cleanup id`. |
| ### Task 2a: The table's kinds, epoch and rev, and its parser |
| **Files:** |
| -- Create: `internal/moves/reserve.go` (`ReadActive`, `NextActiveBody`, `Release`, `Holds`; Task 2b adds the rest) |
| +- Create: `internal/moves/reserve.go` (`ReadActive`, `ReadForAcquire`, `NextActiveBody`, `Release`, `Holds`; Task 2b adds the rest) |
| - Modify: `internal/moves/offline.go` (`OfflineKindCreate = "create"`, `OfflineKindPlace = "place"`; `OfflineActive` gains `Epoch string \`json:"epoch,omitempty"\`` and `Rev uint64 \`json:"rev,omitempty"\``; `OfflineReservation.valid()` and `ParseOfflineActive` take the Global Constraint's entry rules; `HostKey`, `IsHostKey`) |
| - Test: `internal/moves/reserve_test.go`, `internal/moves/offline_compat_test.go` |
| **Interfaces:** |
| - `func HostKey(host string) string`; `func IsHostKey(key string) bool` (`strings.HasPrefix(key, "host/")`; a box label never contains `/`, `controlname.ValidLabel`). |
| -- `type Active struct { Body OfflineActive; Table map[string]OfflineReservation; ETag string }`; `func ReadActive(ctx context.Context, s release.Store) (Active, error)`: `Get`; absent → the bootstrap `Put` of `{v: 1, epoch: newEpoch(), rev: 1}` with `ifMatch ""` (create-only), then `Get` again whatever the `Put` answered (a lost create means another writer created it; an uncertain one can only ever create an empty table, so it is harmless); a read error or `ParseOfflineActive` failing → `ErrUnreadable`. `func NextActiveBody(prev OfflineActive, table map[string]OfflineReservation) OfflineActive`: `OfflineActiveBody(table)` with `Epoch = prev.Epoch` (a new one when empty) and `Rev = prev.Rev + 1`. Readers that only read (`SelfServeEvidence`, `selfserve_pages.go:143`) keep treating an absent object as an empty table. |
| +- `type Active struct { Body OfflineActive; Table map[string]OfflineReservation; ETag string }`; `func ReadActive(ctx context.Context, s release.Store) (Active, error)`: `Get`; absent → the bootstrap `Put` of `{v: 1, epoch: newEpoch(), rev: 1}` with `ifMatch ""` (create-only), then `Get` again whatever the `Put` answered (a lost create means another writer created it; an uncertain one can only ever create an empty table, so it is harmless); a read error or `ParseOfflineActive` failing → `ErrUnreadable`. `func NextActiveBody(prev OfflineActive, table map[string]OfflineReservation) OfflineActive`: `OfflineActiveBody(table)` with `Epoch = prev.Epoch` (a new one when empty) and `Rev = prev.Rev + 1`. Readers that only read (`SelfServeEvidence`, `selfserve_pages.go:143`) keep treating an absent object as an empty table. `func ReadForAcquire(ctx context.Context, s release.Store) (Active, error)` (T4): `ReadActive`; a body with `Epoch == ""` or `Rev == 0` (an older console wrote it) → the normalisation write, one `Put` with `If-Match` the read ETag of `NextActiveBody(body, table)` (the same entries, a fresh epoch, `rev` 1), whatever it answers, then `ReadActive` again; at most three rounds; a body still without both → `ErrUnreadable` wrapping "the table could not be normalised". Only acquisitions call it (`Reserve`, Task 2b, and the console's `reserveBoxes`); a release adds nothing and reads through `ReadActive`. |
| - `type Op struct{ Kind, ID string }`; `var ErrUnreadable`; `func Holds(table map[string]OfflineReservation, keys []string, mine Op) bool`; `func Release(ctx context.Context, s release.Store, mine Op) error`: `ReadActive`; remove `mine`'s entries; **always** one `Put` with `If-Match` of `NextActiveBody`, even when `mine` held nothing (the settling write); a CAS conflict re-reads, three rounds as `releaseBoxes`; a malformed table is an error and is left as it is. A release that read no entry and wrote successfully proves that no earlier `Put` of `mine` can land (the Global Constraint), which a read alone never does. `Release` accepts an `Op` of any kind, so the drain's settling write (Task 7c) is a `Release` of an op that holds nothing. |
| - **The old parser, frozen:** `offline_compat_test.go` carries `parsePlan3OfflineActive`, a verbatim copy of `ParseOfflineActive` and `valid()` as built at `0840937f` (`offline.go:584-587`, `:599-622`), and is the witness of what a console before this plan reads. |
| -- [ ] **Step 1: Write the failing tests**: `TestEntryShapeRules` (a `place` or `create` entry without account or app, a host-key entry naming another host → malformed; each valid shape parses); `TestOldParserReadsTheNewTable` (bodies written by `NextActiveBody` with `move` and `cleanup` entries, host keys and `epoch`/`rev` parse with `parsePlan3OfflineActive`; a body with a `create` or a `place` entry is malformed to it, which the rollback procedure's step 2 covers); `TestActiveTableIsNeverDeleted` (releasing the last entry writes an empty table; no `Delete` or `DeleteIfMatch` reaches the key); `TestBootstrapIsCreateOnlyAndEmpty` (two first writers: one bootstrap lands, both then write through `If-Match`); `TestRevAndEpochNeverRepeatABody` (alternating writes of one entry: every ETag distinct; a body without `epoch`, as plan 3 writes, gets one); `TestReleaseAlwaysWrites` (an op that holds nothing still writes rev + 1). |
| -- [ ] **Step 2: Run to verify they fail**: `go test ./internal/moves -run 'EntryShape|OldParser|ActiveTable|Bootstrap|RevAndEpoch|ReleaseAlways' -count=1`. |
| +- [ ] **Step 1: Write the failing tests**: `TestEntryShapeRules` (a `place` or `create` entry without account or app, a host-key entry naming another host → malformed; each valid shape parses); `TestOldParserReadsTheNewTable` (bodies written by `NextActiveBody` with `move` and `cleanup` entries, host keys and `epoch`/`rev` parse with `parsePlan3OfflineActive`; a body with a `create` or a `place` entry is malformed to it, which the rollback procedure's step 2 covers); `TestActiveTableIsNeverDeleted` (releasing the last entry writes an empty table; no `Delete` or `DeleteIfMatch` reaches the key); `TestBootstrapIsCreateOnlyAndEmpty` (two first writers: one bootstrap lands, both then write through `If-Match`); `TestRevAndEpochNeverRepeatABody` (alternating writes of one entry: every ETag distinct; a body without `epoch`, as plan 3 writes, gets one); `TestReleaseAlwaysWrites` (an op that holds nothing still writes rev + 1); `TestAcquisitionNormalizesAPreEpochTableFirst` (a body as plan 3 writes it, `{v: 1, reservations: {...}}` at ETag `E0`: the fake store's log is `Get`, `Put(ifMatch E0)` of the same entries with an epoch and `rev == 1`, `Get`, and only then the acquisition's `Put` with `ifMatch == E1 && E1 != E0`; a normalisation `Put` that loses its condition is followed by a fresh `Get`, never by an acquisition against `E0`); `TestPreEpochETagCannotSatisfyANewAcquisition` (review 6's finding 4: a metadata-free table holding operator move A at ETag `E`; a placement's acquisition `Put` is held by the fake, and `held.IfMatch != E`; the placement's settlement and the drain write; plan 3's writer, simulated by `parsePlan3OfflineActive` and a re-marshal through plan 3's struct, recreates the A-only body, so `E` is current again; the held `Put` is delivered: `errors.Is(err, release.ErrCASConflict)`, and the table holds no `place` entry). |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/moves -run 'EntryShape|OldParser|ActiveTable|Bootstrap|RevAndEpoch|ReleaseAlways|NormalizesAPreEpoch|PreEpochETag' -count=1`. |
| - [ ] **Step 3: Implement. Step 4: Run to verify they pass**, plus `go test ./internal/moves -run 'Offline' -count=1` (plan 2's parser tests green). |
| -- [ ] **Step 5: Commit** `moves: reservation kinds for creates and placements, and a table that is never deleted and never repeats a body`. |
| +- [ ] **Step 5: Commit** `moves: reservation kinds for creates and placements, and a table that is never deleted, never repeats a body and is normalised before any acquisition`. |
| ### Task 2b: One conflict relation, `Reserve`, and recovery mode counting moves only |
| **Files:** |
| -- Modify: `internal/moves/reserve.go` (`Conflicts`, `Reserve`, the errors), `internal/console/lifetimes_store.go` (`AllocationState(opID) (state string, found bool, err error)`), `internal/console/offlinereserve.go` (`reserveBoxes` reads through `moves.ReadActive` and writes `moves.NextActiveBody` with `If-Match`; its steps (4) and (5) become one call to `moves.Conflicts`; the wanted keys add `moves.HostKey(r.Host)`; `releaseBoxes` wraps `moves.Release`, so the `DeleteIfMatch` at line 478 goes; `offlineReclaimable`: `cleanup` → `reclaimHeld` always, `create` and `place` below; `cleanupReportEvidence` and `Server.cleanupReports` deleted), `internal/console/selfserve_pass.go` (`SelfServeEvidence`, lines 41-53, counts only entries of kind `move`) |
| +- Modify: `internal/moves/reserve.go` (`Conflicts`, `Reserve`, the errors), `internal/console/lifetimes_store.go` (`AllocationState(opID) (state string, found bool, err error)`), `internal/console/offlinereserve.go` (`reserveBoxes` reads through `moves.ReadForAcquire` and writes `moves.NextActiveBody` with `If-Match`; its steps (4) and (5) become one call to `moves.Conflicts`; the wanted keys add `moves.HostKey(r.Host)`; `releaseBoxes` wraps `moves.Release`, so the `DeleteIfMatch` at line 478 goes; `offlineReclaimable`: `cleanup` → `reclaimHeld` always, `create` and `place` below; `cleanupReportEvidence` and `Server.cleanupReports` deleted), `internal/console/selfserve_pass.go` (`SelfServeEvidence`, lines 41-53, counts only entries of kind `move`) |
| - Test: `internal/moves/reserve_conflict_test.go`, `internal/console/offlinereserve_hostkey_test.go` |
| **Interfaces:** |
| - `func Conflicts(table map[string]OfflineReservation, keys []string, mine Op, movesExclusive bool) map[Op]map[string]OfflineReservation` = the Global Constraint's relation, `movesExclusive` applying rule (c) only when `mine.Kind == move`. |
| -- `var ErrReservationMismatch`, `ErrUncertain`; `type ErrReserved struct { Key string; Holder OfflineReservation }`; `func Reserve(ctx context.Context, s release.Store, mine Op, want map[string]OfflineReservation) (map[string]OfflineReservation, error)` for the box, the create and the placement (N10: the caller names the entry for each key; every entry carries `mine` and is valid, else an error and nothing written): `ReadActive`; own entries on a key outside `want` → `ErrReservationMismatch`; own entries covering `want` → the held entries returned, nothing written; any conflict → `ErrReserved` naming the oldest conflicting entry (no reclaim on these paths); else one `Put` with `If-Match` the read ETag of `NextActiveBody` with the missing keys added (at the held entries' `ReservedAt` when partial); a CAS conflict re-reads once; a second loss → `ErrReserved` from the last read; a `Put` error that is not a CAS conflict → `ErrUncertain` wrapped: the caller keeps its state as if it might hold the keys, and the next successful write by anyone, its own retry or its release included, settles it. |
| +- `var ErrReservationMismatch`, `ErrUncertain`; `type ErrReserved struct { Key string; Holder OfflineReservation }`; `func Reserve(ctx context.Context, s release.Store, mine Op, want map[string]OfflineReservation) (map[string]OfflineReservation, error)` for the box, the create and the placement (N10: the caller names the entry for each key; every entry carries `mine` and is valid, else an error and nothing written): `ReadForAcquire` (so the condition of its `Put` is always an epoch-bearing ETag); own entries on a key outside `want` → `ErrReservationMismatch`; own entries covering `want` → the held entries returned, nothing written; any conflict → `ErrReserved` naming the oldest conflicting entry (no reclaim on these paths); else one `Put` with `If-Match` the read ETag of `NextActiveBody` with the missing keys added (at the held entries' `ReservedAt` when partial); a CAS conflict re-reads once (through `ReadForAcquire` again); a second loss → `ErrReserved` from the last read; a `Put` error that is not a CAS conflict → `ErrUncertain` wrapped: the caller keeps its state as if it might hold the keys, and the next successful write by anyone, its own retry or its release included, settles it. |
| - The console's `reserveBoxes`: `want` = an entry `r` for each of `sel.reservationKeys()` and for `moves.HostKey(r.Host)`; the own retry is unchanged (an own reservation made before this plan, without the host key, is partial and completed in the same write; an operator operation holding the sentinel covers every box key and has its host key completed); conflicts = `moves.Conflicts(table, keys, mine, s.cfg.SelfServeMoves != "offline")`; reclaim as today for `move`; `create`: `s.store.AllocationState(opID)` found and terminal → `reclaimYes`, pending → `reclaimHeld`, not found or an error → `reclaimUnreadable`; `place`: `reclaimHeld` until Task 7b replaces it with `s.settlePlacement`. |
| - [ ] **Step 1: Write the failing tests**: `TestConflictRelation` (every pair of kinds over box keys, host keys and the sentinel, with and without `movesExclusive`: two creates of different hosts never conflict; a create and a sentinel move of another host do not; a create and a move of the same host do; a cleanup and an operator move on disjoint boxes do not, with the switch off as on; two moves on disjoint boxes conflict only with the switch off); `TestUncertainReservationIsSettledByTheNextWrite` (review 4's interleaving: create C's `Put` is held by the fake store and answers a timeout; a cleanup acquires; C's recovery reads no entry of C, voids and calls `Release(C)`, which writes; the cleanup releases; C's held `Put` is then delivered and fails its condition; the table never holds C); `TestReserveReturnsTheHeldEntry`; `TestReserveCompletesAPartialOwnReservation`; `TestReserveUncertainPutIsNotAnAnswer` (a store that commits and then errors → `ErrUncertain`; the own retry returns the held entries); `TestMoveSelectionsHoldTheHostKey` (a reservation written by plan 2's code, without the host key, is completed by its own retry; a cleanup on the host key makes the move busy); `TestReleaseIsTheConsolesAlgorithm` (plan 2's `releaseBoxes` tests green through the wrapper, with "deleted" read as "empty"); `TestConsoleNeverDeletesTheTable`; `TestCleanupEntriesAreNeverReclaimed`; `TestCreateEntryReclaimOnTheAllocationRow`; `TestRecoveryModeCountsMoveEntriesOnly` (cleanup, create and place entries alone, and an empty table, leave `SelfServeEvidence` false). |
| @@ -127,23 +129,25 @@ Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and |
| - [ ] **Step 3: Implement. Step 4: Run to verify they pass.** |
| - [ ] **Step 5: Commit** `console: the residence sweep pages a cleanup held over six hours and a create entry without its allocation row`. |
| -### Task 3: The cleaning mark and the writer inventory |
| +### Task 3: The cleaning mark, its refusals and the writer inventory |
| **Files:** |
| - Create: `internal/lease/cleaning.go`, `internal/host/hibernateprefix_inventory_test.go` |
| -- Test: `internal/lease/cleaning_test.go` (over the S3 fake the tombstone tests use) |
| +- Modify: `internal/lease/lease.go` (`AcquireTTL`, line 171: a `readLock` before `AcquireLease`), `internal/lease/tombstone.go` (`Tombstone`, line 183: the owner check before the expiry test), `internal/lease/lockstate.go` (`LockState`, line 78: `held` for a `cleaning:` owner), `internal/host/movesoffline_land.go` (the lease step, lines 612-630: a `cleaning:` owner waits, before the expiry case) |
| +- Test: `internal/lease/cleaning_test.go` (over the S3 fake the tombstone tests use), `internal/host/movesoffline_land_cleaning_test.go` |
| **Interfaces:** |
| -- `const CleaningPrefix = "cleaning:"`; `func CleaningOwner(id string) string`; `var ErrNotOurLock` (the lock is absent, another owner's, or this move's tombstone still live: nothing written). |
| -- `func MarkCleaning(ctx context.Context, bucket, location, region, moveOwner, cleaningOwner string, ttl time.Duration, now time.Time) (expires time.Time, err error)`: `readLock` with its ETag (`tombstoneLeaser`, `tombstone.go:99-148`); owner `cleaningOwner` → `ReassertCleaning`; owner `moveOwner` and `IsLiftedTombstone` → `PutObject` `IfMatch` `{Generation: cur.Generation, ExpiresAt: now+ttl, Owner: cleaningOwner}`; else `ErrNotOurLock`; a lost condition → `ErrTombstoneChanged`; `ttl` in `(0, MaxTombstoneTTL]`. |
| -- `func ReassertCleaning(ctx, bucket, location, region, cleaningOwner string, ttl time.Duration, now time.Time) (time.Time, error)`: owner `cleaningOwner`, live or expired → conditional rewrite of the expiry; else `ErrNotOurLock`. |
| +- `const CleaningPrefix = "cleaning:"`; `var CleaningExpiry = time.Date(9999, 12, 31, 0, 0, 0, 0, time.UTC)`; `func CleaningOwner(id string) string`; `func IsCleaningOwner(owner string) bool` (the prefix and a non-empty id); `var ErrNotOurLock` (the lock is absent, another owner's, or this move's tombstone still live: nothing written). |
| +- `func MarkCleaning(ctx context.Context, bucket, location, region, moveOwner, cleaningOwner string) error`: `readLock` with its ETag (`tombstoneLeaser`, `tombstone.go:99-148`); owner `cleaningOwner` → nil, nothing written (a resumed mark); owner `moveOwner` and `IsLiftedTombstone` → `PutObject` `IfMatch` `{Generation: cur.Generation, ExpiresAt: CleaningExpiry, Owner: cleaningOwner}`; else `ErrNotOurLock`; a lost condition → `ErrTombstoneChanged`. |
| +- **No renewal (T1).** Nothing rewrites a `cleaning:` lock but `UnmarkCleaning`: rev 6's `ReassertCleaning`, `cleanupMarkTTL`, `cleanupMarkMargin` and the 30-minute reassert are gone. |
| - `func UnmarkCleaning(ctx, bucket, location, region, cleaningOwner, moveOwner string) error`: owner `cleaningOwner` → conditional PUT of `{Generation, ExpiresAt: liftedExpiry, Owner: moveOwner}`; absent or another owner → nil (nothing of ours; the caller logs it); a lost condition → `ErrTombstoneChanged`. |
| +- **The refusals, by code path.** `AcquireTTL` (`lease.go:171`; so `Acquire`, `ReplicateWithLease` at `:443`, `ExecBackend.Acquire` at `backend_exec.go:701` and the staging lease): `readLock` first; an `IsCleaningOwner` owner → `fmt.Errorf("acquire lease: %w", &HeldError{Owner, ExpiresAt})`, whatever its expiry, before `AcquireLease` and with nothing written; absent → on to `AcquireLease` as today. The sentinel expiry already makes `AcquireLease` refuse on its own read (`litestream@v0.5.14/s3/leaser.go:96`), so a mark written by `MarkCleaning` is refused on the very read that conditions the write; the pre-read adds the refusal for a mark whose expiry was edited by hand, and costs one `GET` per acquisition. `Tombstone` (`tombstone.go:183`): an `IsCleaningOwner` owner → `heldErr(cur)` before the expiry test. `LiftTombstone` and `Reassert` already refuse an owner that is not `moved:` (`tombstone.go:286`, `:225`). `LockState` (`lockstate.go:78`): `held = lockHeld(expires, now) || IsCleaningOwner(owner)`, so the cut's stop evidence (`movespass.go:647`) and the seed finish (`console/seedfinish.go:255`) read a mark as held. The landing's lease step (`movesoffline_land.go:612-630`): an `IsCleaningOwner` owner → `r.landWait("lease", "a cleanup holds the target's lock")`, before the expiry case. The health renewer (`health.go:134`) renews only its own lease by its ETag and never meets the mark. |
| - **The inventory test** (`go/ast` with `go/types` over every non-test file under `./internal/...`): every call of `activator.HibernatePrefix` and `(activator.Config).hibernatePrefix` is in the test's list with its class: `reader`, `lease-gated writer`, `tombstone-gated writer`, `build-state only`, `delete only`, `a move into the region (hostname key)` or `excluded: <reason>`, as the Global Constraint lists them; a new site, or a removed one, fails until the list is edited. |
| -- [ ] **Step 1: Write the failing tests**: `TestMarkOnlyFromThisMovesLiftedTombstone` (lifted tombstone of this move → marked; of another move, live tombstone, an ordinary lease, absent → `ErrNotOurLock`, nothing written); `TestMarkRacesAnAcquire` (an `Acquire` landing between the read and the PUT → the mark's condition fails); `TestAcquireReplicateAndTombstoneRefuseTheMark` (`Acquire` → `*HeldError` with owner `cleaning:<id>`; `ReplicateWithLease` → the same; `Tombstone` → error, nothing written; `LiftTombstone` with the move's `mine` → `HeldError`); `TestReassertKeepsAnUnchangedExpiredMark` (expired and unchanged → extended; taken and released, so absent → `ErrNotOurLock`); `TestUnmarkRestoresTheLiftedTombstone` (`IsLiftedTombstone(owner, expires)` true afterwards, the generation kept); `TestHibernatePrefixSitesAreInventoried`. |
| -- [ ] **Step 2: Run to verify they fail**: `go test ./internal/lease -run 'Cleaning|Mark|Unmark' -count=1`, `go test ./internal/host -run 'HibernatePrefixSites' -count=1`. |
| -- [ ] **Step 3: Implement. Step 4: Run to verify they pass.** |
| -- [ ] **Step 5: Commit** `lease: the cleaning mark, a live lease written only over this move's lifted tombstone and put back exactly; every hibernate-prefix site inventoried`. |
| +- [ ] **Step 1: Write the failing tests**: `TestMarkOnlyFromThisMovesLiftedTombstone` (lifted tombstone of this move → marked, the body's `ExpiresAt.Equal(lease.CleaningExpiry)` and its generation kept; of another move, live tombstone, an ordinary lease, absent → `errors.Is(err, lease.ErrNotOurLock)`, nothing written); `TestMarkRacesAnAcquire` (an `Acquire` landing between the read and the PUT → the mark's condition fails); `TestCleaningMarkIsRefusedWhateverItsAge` (the mark written, then the clock advanced 25 h, and separately its `ExpiresAt` rewritten by hand to `2000-01-01`: `Acquire` → `errors.As(err, &held) && held.Owner == lease.CleaningOwner(id)` and the lock's ETag unchanged; `AcquireTTL` with the staging TTL and `ReplicateWithLease` → the same; `Tombstone` → `*HeldError`, nothing written; `LiftTombstone` with the move's `mine` → `lifted == false`, nothing written; `LockState` → `held == true`); `TestMarkIsNeverRewritten` (a second `MarkCleaning` by the same owner writes nothing: the ETag is unchanged; the package has no exported function but `UnmarkCleaning` that writes a `cleaning:` lock, checked by `go/ast` over `internal/lease`); `TestUnmarkRestoresTheLiftedTombstone` (`IsLiftedTombstone(owner, expires)` true afterwards, the generation kept); `TestLandingWaitsOnACleaningLockWhateverItsExpiry` (the landing's lease step with a `cleaning:` lock whose expiry was edited to the past → `landWait`, never `ok`); `TestHibernatePrefixSitesAreInventoried`. |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/lease -run 'Cleaning|Mark|Unmark' -count=1`, `go test ./internal/host -run 'HibernatePrefixSites|LandingWaitsOnACleaning' -count=1`. |
| +- [ ] **Step 3: Implement. Step 4: Run to verify they pass**, plus `go test ./internal/lease -run 'Tombstone|Acquire|LockState' -count=1` (the existing tombstone and lock tests green). |
| +- [ ] **Step 5: Commit** `lease: the cleaning mark, a state with no expiry written only over this move's lifted tombstone and put back exactly, refused by every acquirer whatever its age; every hibernate-prefix site inventoried`. |
| ### Task 4a: The persisted lifecycle and its recovery |
| @@ -159,10 +163,10 @@ Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and |
| 1. `open`: observe: `AwayFinishing` → stay (the original move's finishing has precedence; never supersession); `AwayMoved` → supersede; `AwayYes` → then the row check (a row → supersede) → `finished`, `FinishedAt = LastMove.At`. |
| 2. `finished`: `supersededBy` → supersede; `!cfg.CleanupArmed` → with `ReserveUncertain` set, `moves.Release(ctx, s, {cleanup, ID})` (success clears it, event `released-unarmed`), else nothing; stay; `now - FinishedAt < 24 h` → stay; else `moves.Reserve(ctx, s, {cleanup, ID}, {cfg.BoxID: e, HostKey(Host): e})` with `e = {cleanup, ID, Host, Account, App, now}`: `ErrReserved` → stay, event `busy`, `ReserveUncertain` cleared (the table changed since any earlier write of ours was read, so that write can never land); `ErrUnreadable` → stay, event `refused/reservation-unreadable`; `ErrUncertain` → stay, `ReserveUncertain = true`, the same event (the next visit's own retry settles it, or, unarmed, its release); held → `ReservedAt` from the returned entry, `ReserveUncertain` cleared, every inventory key `pending` with `Done = 0`, `State = cleaning`. |
| 3. `cleaning`, `publishing`, `root-requested` (**ownership**, before anything else; rows 3 to 5 run in order within one visit): read `active.json`; `!moves.Holds(table, [BoxID, HostKey(Host)], {cleanup, ID})` → supersede, reason `reservation-lost` (an operator removed it; never re-acquired); a read error → stay. |
| - 4. `cleaning` with `MarkedAt` zero: `MarkCleaning(..., TombstoneOwner, CleaningOwner(ID), cleanupMarkTTL, now)`: `ErrNotOurLock` → supersede, reason `lock-taken`; another error → stay; marked → `MarkedAt = now`. |
| - 5. `cleaning`: first, with `InFlight` set, `w.settleInFlight` (Task 4c); it answers done, stop, or `unsettled` (the mark is no longer ours). Then for each `pending` key in inventory order within the budget: `w.retireKey` (Task 4c), which answers the key's terminal outcome, a supersession reason, or stop. Every key terminal and `InFlight` nil: `supersededBy` once more → supersede; else `LocalInventory` empty → `releasing`, `Outcome = cleaned`; else `publishing`. |
| + 4. `cleaning` with `MarkedAt` zero: `MarkCleaning(..., TombstoneOwner, CleaningOwner(ID))`: `ErrNotOurLock` → supersede, reason `lock-taken`; another error → stay; marked → `MarkedAt = now`. |
| + 5. `cleaning`: first, with `InFlight` set, `w.settleInFlight` (Task 4c), whatever row 3 found; it answers done, stop (the delete still unanswered: the obligation stays `cleaning`, holding the mark and whatever entries it still holds, and nothing else runs this visit), `abandoned` (an operator's abandon after the page) or `unsettled` (the mark replaced by hand). `abandoned` or `unsettled` → `releasing`, `Outcome = superseded`, and no further key is retired; done with `Superseding` set → `releasing`, `Outcome = superseded`, `Superseding` cleared in the same write. Then for each `pending` key in inventory order within the budget: `w.retireKey` (Task 4c), which answers the key's terminal outcome, a supersession reason, or stop. Every key terminal and `InFlight` nil: `supersededBy` once more → supersede; else `LocalInventory` empty → `releasing`, `Outcome = cleaned`; else `publishing`. |
| 6. `publishing`, `root-requested`, `cancel-pending`: Task 6b's rows; `publishing` and `root-requested` begin with `supersededBy` (a reason → supersede). |
| - 7. **Supersede**: from `publishing` or `root-requested` → `cancel-pending` (Task 6b), so the reservation and the mark stay until root has acknowledged; from `cancel-pending` nothing changes; from `cleaning` with `InFlight` set, the obligation stays `cleaning` until `w.settleInFlight` answers done or `unsettled` (with the reason kept in the event), and only then goes on; from any other state → `releasing`, `Outcome = superseded` (from `open` and `finished` too: `Release` and `UnmarkCleaning` change nothing of ours when nothing is held, and a `finished` obligation may hold an entry its crash left). Event `superseded/<reason>`. |
| + 7. **Supersede**: from `publishing` or `root-requested` → `cancel-pending` (Task 6b), so the reservation and the mark stay until root has acknowledged; from `cancel-pending` nothing changes; from `cleaning` with `InFlight` set, the obligation stays `cleaning` with `Superseding = <reason>` persisted, holding the mark and whatever entries it still holds, and row 5 only settles until `w.settleInFlight` answers done, `abandoned` or `unsettled`, and then goes to `releasing` (T1 (b): supersession lifts the mark only when no delete is in flight); from any other state → `releasing`, `Outcome = superseded` (from `open` and `finished` too: `Release` and `UnmarkCleaning` change nothing of ours when nothing is held, and a `finished` obligation may hold an entry its crash left). Event `superseded/<reason>`. |
| 8. `releasing`: `UnmarkCleaning(..., CleaningOwner(ID), TombstoneOwner)` → error → stay; `moves.Release(ctx, s, {cleanup, ID})` → error → stay; → `ended`, `EndedAt = now`. The mark goes before the reservation, so a residence change admitted by the release never meets a mark, and no version is deleted after `releasing` is written. |
| 9. `ended`: Task 5a's publication and Task 8's reaper only. `corrupt`: Task 5a's hourly page only. |
| - **The outcome table** (F7; the only place `Outcome` is chosen; never `cleaned` otherwise): |
| @@ -176,11 +180,12 @@ Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and |
| | in `root-requested`, a trusted `cancelled` acknowledgement the agent did not ask for (authority refused, malformed request, unsupported platform, a forged cancel) | `cancelled` | |
| | supersession from `open`, `finished` or `cleaning` (`record-moved`, `row-present`, `lock-taken`, `reservation-lost`) | `superseded` | |
| | `cancel-pending` resolved (any acknowledgement, or no request was ever taken) | `superseded` | |
| + | in `cleaning`, an operator's abandon of an unanswered delete (`cleanup.abandon`, Task 4c), the key `unsettled` | `superseded` | |
| Root's per-path results go into the report whatever the outcome (`Root`). A key `superseded` or `unsettled` is reported as such whatever the outcome. |
| -- **Restart** resumes every state through the same table: `finished` re-runs `Reserve`, whose own retry returns the held entry; `cleaning` never reserves, and deletes only after rows 3 and 4 found both entries and the mark this cleanup's; an `InFlight` delete found after a crash is answered first. |
| +- **Restart** resumes every state through the same table: `finished` re-runs `Reserve`, whose own retry returns the held entry; `cleaning` never reserves, and sends a first delete of a version only after rows 3 and 4 found both entries and the mark this cleanup's (a resend of the `InFlight` delete needs only the mark, Task 4c); an `InFlight` delete found after a crash is answered first, however long the crash lasted: the mark has no expiry, so it still stands. |
| -- [ ] **Step 1: Write the failing tests** against a `DirStore` deployment bucket, the lease's S3 fake for `lock.json`, a real registry, an injected clock and key and settle steps the test controls: `TestRecordAway` (each branch); `TestSupersessionIsCheckedAtEveryStep` (a row appearing and, separately, the record moving on, at: `open`, `finished`, between two keys, with a delete in flight, after the last key and before `publishing`, in `publishing` before the drop, in `root-requested`: each ends `superseded` through `releasing` or `cancel-pending`, never `cleaned`, and the in-flight delete is answered before `releasing`); `TestOutcomeTable` (each row, with root's answers from a fake acknowledgement reader); `TestCleanupSupersedesWhenTheRecordMovesOn` (a second move B to C, a recreation, a deletion, a placement back to `From`, each before the reservation: superseded, the lifted tombstone untouched, nothing held); `TestOriginalFinishingMoveIsNotSupersession`; `TestUnarmedUncertainReservationIsReleased` (an uncertain `Reserve`, then the switch off and a restart: the next visit releases, the table holds no entry of the cleanup, the obligation stays `finished`); `TestLifecycleResumesAtEveryBoundary` (Review Focus 2); `TestOwnershipLostIsSupersession` (an entry deleted by an operator; the mark replaced by a lease; each → `superseded`, no delete sent). |
| +- [ ] **Step 1: Write the failing tests** against a `DirStore` deployment bucket, the lease's S3 fake for `lock.json`, a real registry, an injected clock and key and settle steps the test controls: `TestRecordAway` (each branch); `TestSupersessionIsCheckedAtEveryStep` (a row appearing and, separately, the record moving on, at: `open`, `finished`, between two keys, with a delete in flight, after the last key and before `publishing`, in `publishing` before the drop, in `root-requested`: each ends `superseded` through `releasing` or `cancel-pending`, never `cleaned`, and the in-flight delete is answered before `releasing`); `TestOutcomeTable` (each row, with root's answers from a fake acknowledgement reader); `TestCleanupSupersedesWhenTheRecordMovesOn` (a second move B to C, a recreation, a deletion, a placement back to `From`, each before the reservation: superseded, the lifted tombstone untouched, nothing held); `TestOriginalFinishingMoveIsNotSupersession`; `TestUnarmedUncertainReservationIsReleased` (an uncertain `Reserve`, then the switch off and a restart: the next visit releases, the table holds no entry of the cleanup, the obligation stays `finished`); `TestLifecycleResumesAtEveryBoundary` (Review Focus 2); `TestOwnershipLostIsSupersession` (an entry deleted by an operator; the mark replaced by a lease by hand; each → `superseded`, no new delete sent; with a delete in flight and the entry deleted by an operator, the same version id is resent under the mark and answered before `releasing` is written: `r.State == "cleaning"` until `fake.Answered(key, v)`). |
| - [ ] **Step 2: Run to verify they fail**: `go test ./internal/moves -run 'RecordAway' -count=1`, `go test ./internal/host -run 'SupersessionIsChecked|OutcomeTable|CleanupSupersedes|OriginalFinishing|UnarmedUncertain|LifecycleResumes|OwnershipLost' -count=1`. |
| - [ ] **Step 3: Implement. Step 4: Run to verify they pass.** |
| - [ ] **Step 5: Commit** `moves: the cleanup's persisted lifecycle, supersession checked at every step and one outcome table`. |
| @@ -189,7 +194,7 @@ Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and |
| **Files:** |
| - Create: `internal/host/movescleanup_delete.go` (`deleteInventoriedVersion`), `internal/host/movescleanup_delete_audit_test.go` |
| -- Modify: the versioned test fake the host tests use (it gains `DeleteVersion` with S3's versioned semantics, an attempt counter, and a hook that holds a request and delivers it later, after the caller has been answered by a later attempt) |
| +- Modify: the versioned test fake the host tests use (it gains `DeleteVersion` with S3's versioned semantics, an attempt counter, and a hook that holds a request and delivers it later, after the caller has been answered by a later attempt; and the accessors the tests of Tasks 4c, 4d and 9 read: `DeleteAttemptsOf(key)`, `DeleteAttemptsOfVersion(key, versionID)`, `Answered(key, versionID)`, `Current(key)` and `VersionsWrittenUnder(prefix)`), the lease's S3 fake (it gains `LockPuts(location)`, the count of writes to a lock) |
| - Test: `internal/host/movescleanup_delete_test.go` |
| **Interfaces:** |
| @@ -198,7 +203,7 @@ Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and |
| - [ ] **Step 1: Write the failing tests**: `TestDeleteNamesTheVersion` (the fake saw a version id on every delete, never a bare key and never `If-Match`; an empty version id sends nothing); `TestDeletingAVersionThatIsNotCurrentChangesNothingCurrent` (two versions: deleting the older leaves the newer current, no marker written); `TestDeletingTheLastVersionLeavesNoObject` (the key lists nothing; no marker); `TestDeletingAMarkerByItsID` (an inventoried marker removed, the version beneath it, also inventoried and already deleted, stays gone); `TestAGoneVersionIsDone` (`NoSuchVersion` → nil); `TestCleanupDeletesOnlyThroughThePrimitive`. |
| - [ ] **Step 2: Run to verify they fail**: `go test ./internal/host -run 'DeleteNames|DeletingAVersion|DeletingTheLast|DeletingAMarker|AGoneVersion|DeletesOnlyThrough' -count=1`. |
| -- [ ] **Step 3: Implement. Step 4: Run to verify they pass.** **Dev witness** (recorded in the commit message, before any arming): on the dev hibernate bucket, a test key under a test prefix with versions v1 and v2: `DeleteObject` with `VersionId` v1 → v2 still current and no marker; with v2 → the key lists no version and no marker; v2 again → `NoSuchVersion` (the primitive answers nil); a write v3, then a `DeleteObject` of v2 → v3 current. The test key is removed by its version ids afterwards. |
| +- [ ] **Step 3: Implement. Step 4: Run to verify they pass.** **Dev witness** (recorded in the commit message, before any arming): `aws s3api get-bucket-versioning` on the dev hibernate bucket answers `Enabled`; on it, a test key under a test prefix with versions v1 and v2: `DeleteObject` with `VersionId` v1 → v2 still current and no marker; with v2 → the key lists no version and no marker; v2 again → `NoSuchVersion` (the primitive answers nil); a write v3, then a `DeleteObject` of v2 → v3 current. The test key is removed by its version ids afterwards. |
| - [ ] **Step 5: Commit** `moves: the cleanup's one delete names an inventoried version id; dev witness of versioned deletion`. |
| ### Task 4c: Retiring a key, oldest version first, and answering every delete |
| @@ -208,14 +213,14 @@ Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and |
| - Test: `internal/host/movescleanup_retire_test.go` |
| **Interfaces:** |
| -- `func (w *movesWorker) retireKey(ctx context.Context, s release.Store, r *offlinecleanup.Retired, k offlinecleanup.InventoryKey, vs lease.VersionedStore, now time.Time) (outcome string, superseded string, err error)`, in order: `supersededBy` (conditions (1) and (3)); the mark (condition (2): Task 4d reasserts first; then `lease.LockState` at the location: owner not ours, or under `cleanupMarkMargin` left → `lock-taken`); condition (4): `vs.ListVersions(Bucket, key)` filtered to `Key == key`: the current entry an object version the inventory does not hold → `superseded` (nothing sent; the skipped page, Task 5a); otherwise, for each version `i` from `Keys[key].Done` in deletion order: not in the listing → done (`Done = i + 1`); in it → one `writeRetired` with `InFlight = {key, versionID, now}` and `Done = i`, then `deleteInventoriedVersion`: nil → `Done = i + 1`, `InFlight = nil` (written with the next `InFlight` or the key's last write); an error → event `refused/delete-unanswered`, `InFlight` stays set, stop this obligation for this run. Every version done → `deleted`. A read error → event (`versions-unreadable`, `lock-unreadable`, `registry-unreadable`, `record-unreadable`), stop for this run. |
| -- `func (w *movesWorker) settleInFlight(ctx context.Context, s release.Store, r *offlinecleanup.Retired, vs lease.VersionedStore, now time.Time) (state string, err error)`: runs only after rows 3 and 4 of this visit found both entries and the mark this cleanup's, and re-checks the mark's owner (`lease.LockState`) just before sending: ours → `deleteInventoriedVersion` for `InFlight` again (the same version id, so any earlier attempt and this one remove one version): nil → that key's `Done` moves past it, `InFlight = nil`, answer done; an error → stop. Not ours, or row 3 or 4 superseded this visit → nothing is sent: the key `unsettled`, `InFlight = nil`, a `PendingPage` `unsettled-<i>` ("cleanup of <host> lost its mark before a delete of <key> was answered") in the same write, answer `unsettled`. |
| -- **Why the order is safe** (the Global Constraint's argument, applied): a key's latest inventoried version is sent only after every older one is done, and a version that is not current can be removed without changing what is current; so at every point of a run, and after any crash, the current entry of every key is one of: the inventoried current version, a later owner's version, a marker someone else wrote, or nothing. |
| +- `func (w *movesWorker) retireKey(ctx context.Context, s release.Store, r *offlinecleanup.Retired, k offlinecleanup.InventoryKey, vs lease.VersionedStore, now time.Time) (outcome string, superseded string, err error)`, in order: `supersededBy` (conditions (1) and (3)); the mark (condition (2): `lease.LockState` at the location: owner not `CleaningOwner(ID)` → `lock-taken`); then for each version `i` from `Keys[key].Done` in deletion order: **condition (4), read again before every deletion** (T6): `vs.ListVersions(Bucket, key)` filtered to `Key == key`; its current entry is not `k.Latest` (`Latest` absent from the listing, or a later version or a marker the inventory does not hold is current) → the key `superseded`, nothing sent, the skipped page (Task 5a), next key; version `i` not in the listing → done (`Done = i + 1`; only a version other than `Latest` can be absent here, because an absent `Latest` failed (4)); in it → one `writeRetired` with `InFlight = {key, versionID, now}` and `Done = i`, then `deleteInventoriedVersion`: nil → `Done = i + 1`, `InFlight = nil` (written with the next `InFlight` or the key's last write); an error → event `refused/delete-unanswered`, `InFlight` stays set, stop this obligation for this run. Every version done → `deleted`. A read error → event (`versions-unreadable`, `lock-unreadable`, `registry-unreadable`, `record-unreadable`), stop for this run. |
| +- `func (w *movesWorker) settleInFlight(ctx context.Context, s release.Store, r *offlinecleanup.Retired, vs lease.VersionedStore, now time.Time) (state string, err error)` (T1): runs first in every `cleaning` visit with `InFlight` set, whether or not row 3 found the entries this visit (an operator may have removed one; the mark is what excludes every writer that could adopt the version, Global Constraint), and checks two things just before sending. (i) `<OfflineOpDir>/cleanup.abandon` present (`Lstat`) and `now - InFlight.At >= 10 min` (the first `inflight` page is owed): nothing sent; the key `unsettled`, `InFlight = nil`, event `abandoned` (`Key`) in the same write; answer `abandoned`. Present earlier: logged, ignored. (ii) The mark's owner (`lease.LockState`) not `CleaningOwner(ID)` (only a hand edit can do this: nothing the platform runs rewrites a `cleaning:` lock): nothing sent; the key `unsettled`, `InFlight = nil`, a `PendingPage` `unsettled-<i>` ("cleanup of <host> lost its mark before a delete of <key> was answered") in the same write; answer `unsettled`. A read error in either → stop. Otherwise `deleteInventoriedVersion` for `InFlight` again, the same version id, so any earlier attempt and this one remove one version; condition (4) is not applied to a resend, because the earlier attempt may land whatever this one does, and only an answer lets the mark go. Nil → that key's `Done` moves past it, `InFlight = nil`, answer done; an error → event `refused/delete-unanswered` (deduped), stop. While `InFlight` stays set, Task 5a's `inflight-<n>` page fires every 10 minutes from `InFlight.At`: "cleanup of <host>: an outstanding delete is unsettled; the host's replica stays marked". **The documented command** for (i), named in the log (Task 5b) and in `docs/site/cli.md`: on the source box, create the file the log names (`touch /data/.offline-moves/<host>/<move>/cleanup.abandon`, as the agent's user). It is for an operator who has read the page and accepts that the outstanding attempt may still land after the mark is lifted; a forged abandon needs the agent's uid, which already holds the agent's bucket credentials, so it adds nothing an attacker with that uid lacks. |
| +- **Why the order is safe** (the Global Constraint's argument, applied): every deletion is sent only after a fresh listing found the key's `Latest` current, and while `Latest` is current no deletion of another version changes what is current; `Latest` itself is sent last. So at every point of a run, and after any crash, the current entry of every key is `Latest`, a later owner's version, a marker someone else wrote, or nothing; the one exception the plan names is a version-id delete of `Latest` by hand that lands between a check and our delete. |
| -- [ ] **Step 1: Write the failing tests**: `TestFourConditionsAtEveryKey` (each failing between key 1 and key 2: stop on a read error, supersede on (1), (2), (3); (4) with a later object version current → `superseded`, nothing sent; with a foreign marker current → our versions beneath it deleted, the marker kept); `TestDeletionNeverExposesAnOlderVersion` (a key with three versions; a crash and an error after each delete: the current entry is never v1 or v2 while v3 is present); `TestUnansweredDeleteIsAnsweredBeforeTheUnmark` (Review Focus 1: the fake holds a delete and times it out; a row appears; the obligation stays `cleaning` and sends the same version id again under the mark; only after the answer does it write `releasing`, unmark and release; the held first attempt is delivered after a wake wrote a new version: nothing changes); `TestRetriedAttemptsRemoveOnlyTheirVersion` (two attempts of one delete, the first delivered after the second was answered and after a later owner wrote v4: v4 current); `TestLostMarkLeavesTheDeleteUnsettled` (the mark replaced while a delete is in flight: nothing more is sent, the key `unsettled`, the page owed in `pages_pending`, the obligation `superseded`); `TestPromotionThatKeptAnInventoriedVersionKeepsIt` (a cleanup superseded with a delete in flight, answered under the mark; after the release a promotion through `lease.MirrorPrefix` finds an equal ETag and keeps the inventoried version in place: no delete of the cleanup reaches it, and it is current after the run ends); `TestAdmittedReconcileAndWakeCannotWriteUnderTheMark` (Review Focus 1, with `ExecBackend.Acquire` over the fake); `TestLockAndBuildStateNeverDeleted`. |
| -- [ ] **Step 2: Run to verify they fail**: `go test ./internal/host -run 'FourConditions|DeletionNeverExposes|UnansweredDelete|RetriedAttempts|LostMark|PromotionThatKept|AdmittedReconcile|LockAndBuildState' -count=1`. |
| +- [ ] **Step 1: Write the failing tests**: `TestFourConditionsAtEveryKey` (each failing between key 1 and key 2: stop on a read error, supersede on (1), (2), (3); (4) with a later object version current → `superseded`, nothing sent; with a foreign marker current → `superseded`, nothing sent, the marker and every version beneath it kept: `fake.DeleteAttemptsOf(key) == 0`); `TestLatestRemovedByHandSupersedesTheKey` (review 6's finding 6: v1 and v2 sharing a `LastModified`, v3 latest; the recorded order deletes one of v1, v2 first; then v3 is deleted by hand by its version id; the next check finds `Latest` absent → the key `superseded` and the cleanup sends nothing more for it: `fake.DeleteAttemptsOf(key) == 1`; the skipped page owed); `TestDeletionNeverExposesAnOlderVersion` (a key with three versions, two of them tied: a crash and an error after each delete: `fake.Current(key).VersionID == k.Latest` until the latest's own delete, after which the key lists nothing); `TestUnansweredDeleteIsAnsweredBeforeTheUnmark` (Review Focus 1: the fake holds a delete and times it out; a row appears; the obligation stays `cleaning` and sends the same version id again under the mark; only after the answer does it write `releasing`, unmark and release; the held first attempt is delivered after a wake wrote a new version: nothing changes); `TestRetriedAttemptsRemoveOnlyTheirVersion` (two attempts of one delete, the first delivered after the second was answered and after a later owner wrote v4: v4 current); `TestMarkOutlivesACrashWithADeleteInFlight` (T1's witness: `InFlight = v` written and the fake holds the delete; the cleanup's process stops, its worker discarded; the clock advances 25 h; a legacy move back to `From` completes and its wake runs `ExecBackend.Acquire` over the fake: `errors.As(err, &held) && held.Owner == lease.CleaningOwner(r.ID)` and `fake.VersionsWrittenUnder(prefix) == 0`; a new worker resumes the obligation: its first visit resends `v` (`fake.DeleteAttemptsOfVersion(key, v) == 2`), the answer moves `Done` past `v`, then `releasing`, the lifted tombstone is back (`lease.IsLiftedTombstone(owner, expires)`), the entries are released, and the wake then acquires); `TestUnanswerableDeleteHoldsTheMarkAndPages` (every resend errors for 2 h, and an operator removes the box entry after 30 min: the obligation stays `cleaning`, the lock's owner is `cleaning:<id>` at every visit, `releasing` is never written, and the pages `inflight-1` to `inflight-12` each fire once); `TestAbandonAfterThePageReleasesTheMark` (the abandon file present before `inflight-1` is owed: ignored, nothing changes; after it: the key `unsettled`, event `abandoned`, then `releasing`, the unmark, the release, `Outcome == "superseded"`); `TestForeignMarkLeavesTheDeleteUnsettled` (the mark replaced by hand while a delete is in flight: nothing more is sent, the key `unsettled`, the page owed in `pages_pending`, the obligation `superseded`); `TestPromotionThatKeptAnInventoriedVersionKeepsIt` (a cleanup superseded with a delete in flight, answered under the mark; after the release a promotion through `lease.MirrorPrefix` finds an equal ETag and keeps the inventoried version in place: no delete of the cleanup reaches it, and it is current after the run ends); `TestAdmittedReconcileAndWakeCannotWriteUnderTheMark` (Review Focus 1, with `ExecBackend.Acquire` over the fake); `TestLockAndBuildStateNeverDeleted`. |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/host -run 'FourConditions|LatestRemovedByHand|DeletionNeverExposes|UnansweredDelete|RetriedAttempts|MarkOutlives|UnanswerableDelete|AbandonAfter|ForeignMark|PromotionThatKept|AdmittedReconcile|LockAndBuildState' -count=1`. |
| - [ ] **Step 3: Implement. Step 4: Run to verify they pass.** |
| -- [ ] **Step 5: Commit** `moves: a key is retired by its inventoried versions oldest first, and every delete is answered under the mark before the cleanup gives it up`. |
| +- [ ] **Step 5: Commit** `moves: a key is retired by its inventoried versions oldest first while its latest is current, and every delete is answered under the mark before the cleanup gives it up`. |
| ### Task 4d: Scheduling and renewal, outside the pass lock |
| @@ -226,12 +231,12 @@ Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and |
| **Interfaces:** |
| - `func (w *movesWorker) kickCleanup(ctx context.Context, s release.Store)`: nothing unless `cfg.OfflineCapable && cfg.Judge != nil`; `w.cleanupMu.TryLock()` false → return; else a goroutine runs `w.cleanupRun(runCtx, s)` with `runCtx, cancel := context.WithTimeout(ctx, cleanupRunDeadline)` and unlocks when it returns. **Locks:** the goroutine holds `cleanupMu` only, never `w.pass` and never `w.mu` across I/O; `movesPass` holds `w.pass` while it only try-locks `cleanupMu`, so neither waits on the other and `offlineReassert` (`movesoffline.go:795`, which takes `w.pass`) is never behind a cleanup. |
| - `cleanupRun`: lists `<dataDir>/.offline-moves/*/*/retired.json` and the corrupt stubs, starts after the path recorded in `<dataDir>/.offline-moves/.cleanup-cursor`, calls `cleanupVisit` on each obligation in path order until the deadline (each read, list, delete, lock call and report write under `context.WithTimeout(runCtx, 10*time.Second)`), at most 50 keys per obligation, records the cursor after each obligation, then runs publication (Task 5a) and the reaper (Task 8). One `cfg.Versioned` store per bucket per run. |
| -- **Renewal:** in `cleaning` (after `MarkedAt`), `publishing`, `root-requested`, `cancel-pending` and `releasing` before the unmark, every visit reasserts the mark when under 30 min remain (`ReassertCleaning`; `ErrNotOurLock` → supersede, reason `lock-taken`, which in `cancel-pending` and `releasing` changes nothing), so the mark never lapses while root, a cancel acknowledgement or an unanswered delete is waited for. |
| +- **No renewal (T1).** The mark has no expiry, so no visit rewrites it: from `MarkCleaning` to `UnmarkCleaning` the lock object is not written by the cleanup, whether it waits on root, on a cancel acknowledgement or on an unanswered delete, and however long the box is down between runs. A foreign owner found by a visit (condition (2), `settleInFlight`, Task 6a's authority) is a hand edit and is handled where it is read. |
| -- [ ] **Step 1: Write the failing tests**: `TestCleanupRunsTwentyFourHoursAfterTheObservedFinish` (at 23 h 59 m nothing; at 24 h reserved, marked, deleted); `TestCleanupUnarmedLeavesObligationsFinished`; `TestCleanupRunsOutsideThePassLock` (a versioned fake that blocks: `offlineReassert` returns while the run is blocked; the run stops at its deadline; the cursor moves on; a second kick while one runs starts nothing); `TestMarkIsReassertedWhileWaiting` (root unanswered for 3 h, and a delete unanswered across runs: the mark never under 30 min); `TestRunBudgetAndCursor` (120 keys: 50 per run; three obligations and a deadline: each visited in turn across runs). |
| -- [ ] **Step 2: Run to verify they fail**: `go test ./internal/host -run 'CleanupRuns|CleanupUnarmed|MarkIsReasserted|RunBudget' -count=1`. |
| +- [ ] **Step 1: Write the failing tests**: `TestCleanupRunsTwentyFourHoursAfterTheObservedFinish` (at 23 h 59 m nothing; at 24 h reserved, marked, deleted); `TestCleanupUnarmedLeavesObligationsFinished`; `TestCleanupRunsOutsideThePassLock` (a versioned fake that blocks: `offlineReassert` returns while the run is blocked; the run stops at its deadline; the cursor moves on; a second kick while one runs starts nothing); `TestMarkIsNeverRewrittenWhileWaiting` (root unanswered for 3 h, and a delete unanswered across runs and a 25 h stop: the lock's ETag is the one `MarkCleaning` left until `UnmarkCleaning`, `fake.LockPuts(location) == 2`); `TestRunBudgetAndCursor` (120 keys: 50 per run; three obligations and a deadline: each visited in turn across runs). |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/host -run 'CleanupRuns|CleanupUnarmed|MarkIsNeverRewritten|RunBudget' -count=1`. |
| - [ ] **Step 3: Implement. Step 4: Run to verify they pass.** |
| -- [ ] **Step 5: Commit** `moves: the cleanup runs in its own goroutine under its own lock, with a run budget, a cursor and a mark that never lapses while it waits`. |
| +- [ ] **Step 5: Commit** `moves: the cleanup runs in its own goroutine under its own lock, with a run budget, a cursor and a mark it never rewrites`. |
| ### Task 5a: Publication: the report under the move, its events, the pages |
| @@ -241,12 +246,12 @@ Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and |
| - Test: `internal/host/movescleanup_report_test.go`, `internal/console/offline_cleanup_evidence_test.go` |
| **Interfaces:** |
| -- `type Event struct { Seq uint64; Kind, Key, Detail string; At, LastAt time.Time; Count int }`. Kinds and their stable details: `finished`; `busy` (the holder's kind); `reserved`; `released-unarmed`; `marked`; `refused` (`record-unreadable`, `registry-unreadable`, `lock-unreadable`, `versions-unreadable`, `delete-unanswered`, `reservation-unreadable`, `mark-failed`, `unmark-failed`, `release-failed`, `report-unwritten`); `skipped` (`Key`); `unsettled` (`Key`); `superseded` (`record-moved`, `row-present`, `lock-taken`, `reservation-lost`); `root-requested`; `root-refused` (root has not answered and its request directory's ancestry is not root-only); `cancel-requested`; `root-done`; `root-kept` (`Key` the path's `Which`, `Detail` root's reason); `root-cancelled`; `ended` (the outcome). An event equal in `Kind`, `Key` and `Detail` to the last one increments its `Count` and `LastAt` instead of appending, so every distinct refusal is kept and a repeated one is counted (the raw error goes to the box log only). |
| -- `type Report struct { V int; CleanupID, MoveID, Host, Account, App, Box, State, Outcome string; Keys map[string]KeyOutcome; Events []Event; Left []KeptPath; Root *RootCleanupAck; Rev uint64; At time.Time }` at `moves.CleanupReportKey(Host, MoveID, Box)`, a plain replacing `Put`; `ValidateReport(r Report, key string) error` (the id recomputes, the key is the one its fields name). `Account` and `App` let the console's door keep a report to the app it was asked about (Task 5b). |
| +- `type Event struct { Seq uint64; Kind, Key, Detail string; At, LastAt time.Time; Count int }`. Kinds and their stable details: `finished`; `busy` (the holder's kind); `reserved`; `released-unarmed`; `marked`; `refused` (`record-unreadable`, `registry-unreadable`, `lock-unreadable`, `versions-unreadable`, `delete-unanswered`, `reservation-unreadable`, `mark-failed`, `unmark-failed`, `release-failed`, `report-unwritten`); `skipped` (`Key`); `unsettled` (`Key`); `abandoned` (`Key`); `superseded` (`record-moved`, `row-present`, `lock-taken`, `reservation-lost`); `root-requested`; `root-refused` (root has not answered and its request directory's ancestry is not root-only); `cancel-requested`; `root-done`; `root-kept` (`Key` the path's `Which`, `Detail` root's reason); `root-cancelled`; `ended` (the outcome). An event equal in `Kind`, `Key` and `Detail` to the last one increments its `Count` and `LastAt` instead of appending, so every distinct refusal is kept and a repeated one is counted (the raw error goes to the box log only). |
| +- `type Report struct { V int; CleanupID, MoveID, Host, Account, App, Box, State, Outcome string; Keys map[string]KeyOutcome; InFlight *InFlightDelete; AbandonPath string; Events []Event; Left []KeptPath; Root *RootCleanupAck; Rev uint64; At time.Time }` (`InFlight` and `AbandonPath`, the box's own `<OfflineOpDir>/cleanup.abandon`, set only while a delete waits for its answer) at `moves.CleanupReportKey(Host, MoveID, Box)`, a plain replacing `Put`; `ValidateReport(r Report, key string) error` (the id recomputes, the key is the one its fields name). `Account` and `App` let the console's door keep a report to the app it was asked about (Task 5b). |
| - After each obligation's transitions in a run: `PublishedRev < Rev` → write the report from the file → on success `markPublished` (`PublishedRev = Rev`); a failure → event `refused/report-unwritten` (deduped; the event itself bumps `Rev`, so the next run publishes it too). The bucket report is never removed by this plan, so it outlives the op directory and the record. |
| -- **Pages** through `w.firePage(ctx, Host, ID, rule, alerting.LevelWarning, subject, body)` (one page per `(host, op, rule)` in the outbox, `movespaging.go:232`), the body naming `--cleanup <ID>`. Periodic rules, fired from persisted times, so a lost one is followed by the next tick's: `root-wait-<n>` with `n = floor((now - RootRequestedAt) / 10 min)` for `n >= 1`, until `root-refused` applies; `root-refused-<h>` with `h = floor((now - RootRequestedAt) / 1 h)` once `h >= 1` and the agent's own `RootOnlyChain` check of root's request directory fails (Task 6b); `cancel-wait-<n>` from `CancelRequestedAt`; `corrupt-<h>` with `h = floor((now - the stub's FoundAt) / 1 h)`; `unreadable-<h>` while the record has been unreadable for over an hour (from that event's `At`). **One-shot rules (N8):** `skipped-<i>` for inventory index `i` is written as a `PendingPage{Rule, Subject, At}` into `PagesPending` in the same `writeRetired` that makes key `i` `superseded`, and `unsettled-<i>` in the one that makes it `unsettled`, so no state has the key terminal without its page owed. After the transitions, each pending page is fired: it leaves `PagesPending` when `firePage` answers sent, paged before, unwired (no pager on this box, logged as every moves page is), or pending with its entry on disk (`readPageEntry(pageEntryPath(...))` succeeds); it stays on undelivered and on pending held only in memory (`firePage`'s index, `movespaging.go:280-292`, which a restart loses). Each run then calls `w.flushPageOutboxFor(ctx, Host, ID)`, whatever `cfg.Enabled` says, so the outbox's other entries keep plan 6's rule (with the switch off they wait on disk). |
| +- **Pages** through `w.firePage(ctx, Host, ID, rule, alerting.LevelWarning, subject, body)` (one page per `(host, op, rule)` in the outbox, `movespaging.go:232`), the body naming `--cleanup <ID>`. Periodic rules, fired from persisted times, so a lost one is followed by the next tick's: `root-wait-<n>` with `n = floor((now - RootRequestedAt) / 10 min)` for `n >= 1`, until `root-refused` applies; `root-refused-<h>` with `h = floor((now - RootRequestedAt) / 1 h)` once `h >= 1` and the agent's own `RootOnlyChain` check of root's request directory fails (Task 6b); `cancel-wait-<n>` from `CancelRequestedAt`; `corrupt-<h>` with `h = floor((now - the stub's FoundAt) / 1 h)`; `inflight-<n>` with `n = floor((now - InFlight.At) / 10 min)` for `n >= 1` while `InFlight` is set, subject "cleanup of <host>: an outstanding delete is unsettled; the host's replica stays marked"; `unreadable-<h>` while the record has been unreadable for over an hour (from that event's `At`). **One-shot rules (N8):** `skipped-<i>` for inventory index `i` is written as a `PendingPage{Rule, Subject, At}` into `PagesPending` in the same `writeRetired` that makes key `i` `superseded`, and `unsettled-<i>` in the one that makes it `unsettled`, so no state has the key terminal without its page owed. After the transitions, each pending page is fired: it leaves `PagesPending` when `firePage` answers sent, paged before, unwired (no pager on this box, logged as every moves page is), or pending with its entry on disk (`readPageEntry(pageEntryPath(...))` succeeds); it stays on undelivered and on pending held only in memory (`firePage`'s index, `movespaging.go:280-292`, which a restart loses). Each run then calls `w.flushPageOutboxFor(ctx, Host, ID)`, whatever `cfg.Enabled` says, so the outbox's other entries keep plan 6's rule (with the switch off they wait on disk). |
| -- [ ] **Step 1: Write the failing tests**: `TestCleanupReportIsDurableAndBlocksTheReaper` (Review Focus 5, with Task 8's reaper once it exists; until then the report half: a failing `Put` for three runs, then success: one report with every event and `PublishedRev == Rev`); `TestRefusalsAreStableDedupedEvents` (a read error every run for an hour is one event with its count; a different detail appends); `TestPendingPagesSurviveARestart` (a skipped key and an unsettled key; the outbox cannot create the entry and the pager refuses; restart: each page is fired again from `PagesPending` and leaves it only once on disk or sent); `TestPagesFireOncePerTickAcrossRestarts` (root unanswered for 35 min across a restart at 15 min: pages `root-wait-1`, `-2`, `-3`, each once; with the ancestry refused, `root-refused-1` at 1 h and `-2` at 2 h, and no `root-wait` after 1 h); `TestPageFlushIsScopedToTheCleanup` (`cfg.Enabled` false: the cleanup's undelivered page is retried, another move's entry is not touched); `TestReadOfflineEvidenceSkipsCleanupReports`. |
| +- [ ] **Step 1: Write the failing tests**: `TestCleanupReportIsDurableAndBlocksTheReaper` (Review Focus 5, with Task 8's reaper once it exists; until then the report half: a failing `Put` for three runs, then success: one report with every event and `PublishedRev == Rev`); `TestRefusalsAreStableDedupedEvents` (a read error every run for an hour is one event with its count; a different detail appends); `TestPendingPagesSurviveARestart` (a skipped key and an unsettled key; the outbox cannot create the entry and the pager refuses; restart: each page is fired again from `PagesPending` and leaves it only once on disk or sent); `TestPagesFireOncePerTickAcrossRestarts` (root unanswered for 35 min across a restart at 15 min: pages `root-wait-1`, `-2`, `-3`, each once; with the ancestry refused, `root-refused-1` at 1 h and `-2` at 2 h, and no `root-wait` after 1 h; a delete unanswered for 25 min across a restart at 12 min: `inflight-1` and `inflight-2`, each once, and the report carries `InFlight` and `AbandonPath`); `TestPageFlushIsScopedToTheCleanup` (`cfg.Enabled` false: the cleanup's undelivered page is retried, another move's entry is not touched); `TestReadOfflineEvidenceSkipsCleanupReports`. |
| - [ ] **Step 2: Run to verify they fail**: `go test ./internal/host -run 'CleanupReport|RefusalsAre|PendingPages|PagesFire|PageFlush' -count=1`, `go test ./internal/console -run 'SkipsCleanup' -count=1`. |
| - [ ] **Step 3: Implement. Step 4: Run to verify they pass.** |
| - [ ] **Step 5: Commit** `moves: the cleanup's report under the move id, republished until it lands, with stable de-duplicated events, and pages owed in retired.json until the outbox has them`. |
| @@ -259,11 +264,11 @@ Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and |
| - Test: `internal/console/offline_cleanups_test.go`, `move_test.go` |
| **Interfaces:** |
| -- **The door** `GET /api/cli/apps/{account}/{app}/instances/{host}/move/offline/cleanups[?move=<id>|?cleanup=<id>]`, `svcrole.Set{}`, operator only (`s.moveOperator`, a 404 otherwise, as `offlineGate`): the instance gate for the account and app and the host shape as `moveGate` checks them, and **no record read**, so a host whose record was deleted, or whose `LastMove` has moved on, still answers. `offlineCleanupsOf(ctx, acct, app, host, move, cleanup)` lists `control/offline-moves/<host>/` once and keeps the keys of the shape `<host>/<op>/cleanup/<box>.json`; with `move` set, only that `op`; it reads each kept key, keeps a report that decodes, passes `offlinecleanup.ValidateReport` for its key and names this `Account` and `App`, and, with `cleanup` set, only a report whose `CleanupID` matches. The answer is `{cleanup: [offlineCleanupView]}`, newest `At` first, with `offlineCleanupView{CleanupID, MoveID, Box, State, Outcome, Keys []{Key, State}, Events []{Kind, Key, Detail, At, LastAt, Count}, Left []{Which, Reason}, Root *{Outcome, Removed, Absent []string; Kept []{Which, Reason}}, At}` (Unix seconds, as the status view's other times). A listing error is a 502; a report that fails a read is logged and left out. A malformed `move` (not a plain segment) or `cleanup` (not `cu_` and 16 hex) is a 400. Operator moves and legacy moves get it as self-serve moves do. The status door (`handleAPIInstanceOfflineStatus`, line 1950) is unchanged. |
| -- **The CLI** (`parseMoveArgs`, `move.go:138`, its flag switch at `:166-176`): `case "offline"` with `a.offline == "status"` adds `fs.BoolVar(&a.log, "log", false, "show the move's log, with the source box's cleanup")`, `fs.StringVar(&a.logMove, "move", "", "with --log: the cleanup of this move")` and `fs.StringVar(&a.logCleanup, "cleanup", "", "with --log: the cleanup with this id")`; `--move` and `--cleanup` only with `--log` and not together (a usage error otherwise). `runStatus` (`move.go:1507`) polls `w.m.path + "?log=1"` when `--log` is set, as today, and then asks the cleanups door once with the filter, in both its `state == "none"` branch and the in-flight one; a 404 from the status door (the record is gone) with `--log` still asks the cleanups door and says its lines. Per report: "Cleanup <id> of <host> on <box> for move <move> is <state>." or, ended, "Cleanup <id> of <host> on <box> for move <move> ended <outcome>."; per key not `deleted`: "Object <key> is <state>."; per left path: "<box> left <which> in place: <reason>." (the existing MV form); root's answer: "Root answered <outcome>: <n> removed, <n> already gone, <n> kept."; per event: "At <time>: <kind> <key> <detail>, <count> times until <time>." (count 1: "At <time>: <kind> <key> <detail>."); per self-serve event the status door returns: "At <time>: <phase> <reason> <detail>."; no report: "No cleanup has reported for <host> yet." (with a filter: "No cleanup of <host> matches <id>."). Without a filter every cleanup of the host is said, newest first. Times as the CLI says them today (`2006-01-02 15:04 UTC`). The decoded view's field names are kept apart from the tree's, as `offlineView`'s are, for the copy gate (`move.go:1092-1100`); event kinds, keys, details and reasons are box diagnostics, not copy. The exit code is the status door's, unchanged by the log. |
| -- **Docs:** `docs/site/cli.md` gains `carlos move offline status --host <host> --log [--move <id> | --cleanup <id>]` in the verb list and one paragraph: what the log adds (the move's events and, a day after a move finished, the source box's cleanup of the old region's copy, object by object, what it left and why), and that `--cleanup` finds a cleanup a page names, even after a later move or a deleted instance. Copy rows, pending. |
| +- **The door** `GET /api/cli/apps/{account}/{app}/instances/{host}/move/offline/cleanups[?move=<id>|?cleanup=<id>]`, `svcrole.Set{}`, operator only (`s.moveOperator`, a 404 otherwise, as `offlineGate`): the instance gate for the account and app and the host shape as `moveGate` checks them, and **no record read**, so a host whose record was deleted, or whose `LastMove` has moved on, still answers. `offlineCleanupsOf(ctx, acct, app, host, move, cleanup)` lists `control/offline-moves/<host>/` once and keeps the keys of the shape `<host>/<op>/cleanup/<box>.json`; with `move` set, only that `op`; it reads each kept key, keeps a report that decodes, passes `offlinecleanup.ValidateReport` for its key and names this `Account` and `App`, and, with `cleanup` set, only a report whose `CleanupID` matches. The answer is `{cleanup: [offlineCleanupView]}`, newest `At` first, with `offlineCleanupView{CleanupID, MoveID, Box, State, Outcome, Keys []{Key, State}, InFlight *{Key, AbandonPath string; Since int64}, Events []{Kind, Key, Detail, At, LastAt, Count}, Left []{Which, Reason}, Root *{Outcome, Removed, Absent []string; Kept []{Which, Reason}}, At}` (Unix seconds, as the status view's other times). A listing error is a 502; a report that fails a read is logged and left out. A malformed `move` (not a plain segment) or `cleanup` (not `cu_` and 16 hex) is a 400. Operator moves and legacy moves get it as self-serve moves do. The status door (`handleAPIInstanceOfflineStatus`, line 1950) is unchanged. |
| +- **The CLI** (`parseMoveArgs`, `move.go:138`, its flag switch at `:166-176`): `case "offline"` with `a.offline == "status"` adds `fs.BoolVar(&a.log, "log", false, "show the move's log, with the source box's cleanup")`, `fs.StringVar(&a.logMove, "move", "", "with --log: the cleanup of this move")` and `fs.StringVar(&a.logCleanup, "cleanup", "", "with --log: the cleanup with this id")`; `--move` and `--cleanup` only with `--log` and not together (a usage error otherwise). `runStatus` (`move.go:1507`) polls `w.m.path + "?log=1"` when `--log` is set, as today, and then asks the cleanups door once with the filter, in both its `state == "none"` branch and the in-flight one; a 404 from the status door (the record is gone) with `--log` still asks the cleanups door and says its lines. Per report: "Cleanup <id> of <host> on <box> for move <move> is <state>." or, ended, "Cleanup <id> of <host> on <box> for move <move> ended <outcome>."; per key not `deleted`: "Object <key> is <state>."; per left path: "<box> left <which> in place: <reason>." (the existing MV form); a delete waiting for its answer: "A delete of <key> has waited for its answer since <time>. the replica stays marked until it has one. to give it up, create <path> on <box>."; root's answer: "Root answered <outcome>: <n> removed, <n> already gone, <n> kept."; per event: "At <time>: <kind> <key> <detail>, <count> times until <time>." (count 1: "At <time>: <kind> <key> <detail>."); per self-serve event the status door returns: "At <time>: <phase> <reason> <detail>."; no report: "No cleanup has reported for <host> yet." (with a filter: "No cleanup of <host> matches <id>."). Without a filter every cleanup of the host is said, newest first. Times as the CLI says them today (`2006-01-02 15:04 UTC`). The decoded view's field names are kept apart from the tree's, as `offlineView`'s are, for the copy gate (`move.go:1092-1100`); event kinds, keys, details and reasons are box diagnostics, not copy. The exit code is the status door's, unchanged by the log. |
| +- **Docs:** `docs/site/cli.md` gains `carlos move offline status --host <host> --log [--move <id> | --cleanup <id>]` in the verb list and one paragraph: what the log adds (the move's events and, a day after a move finished, the source box's cleanup of the old region's copy, object by object, what it left and why), and that `--cleanup` finds a cleanup a page names, even after a later move or a deleted instance; and what to do about a delete that waits for its answer (the replica stays marked; creating the file the log names gives it up, accepting that the outstanding attempt may still land). Copy rows, pending. |
| -- [ ] **Step 1: Write the failing tests**: `TestOfflineCleanupsDoorNeedsNoRecord` (an operator move, no self-serve row, and then the record deleted: the door still answers the report); `TestOlderCleanupIsFoundByItsID` (Review Focus 5: move A's cleanup is open, move B finishes and has its own report; `?cleanup=<A's id>` answers A's report only; no filter answers both, newest first; `?move=<A>` answers A's); `TestOfflineCleanupsDoorFilters` (a report at the wrong key, with another id, or naming another app is left out; a malformed filter → 400; a listing error → 502; a member's token → 404); `TestMoveOfflineStatusLogFlags` (`--log` accepted only on `offline status`; `--move` and `--cleanup` only with `--log` and not together; the requests carry `log=1` and the filter); `TestMoveOfflineStatusLogRendering` (state `none`, in flight, and a 404 status door; each line above; no report; no match); `TestMovesStringsAreRecorded` and `TestMovesCopyRecordIsComplete` green with the new rows pending. |
| +- [ ] **Step 1: Write the failing tests**: `TestOfflineCleanupsDoorNeedsNoRecord` (an operator move, no self-serve row, and then the record deleted: the door still answers the report); `TestOlderCleanupIsFoundByItsID` (Review Focus 5: move A's cleanup is open, move B finishes and has its own report; `?cleanup=<A's id>` answers A's report only; no filter answers both, newest first; `?move=<A>` answers A's); `TestOfflineCleanupsDoorFilters` (a report at the wrong key, with another id, or naming another app is left out; a malformed filter → 400; a listing error → 502; a member's token → 404); `TestMoveOfflineStatusLogFlags` (`--log` accepted only on `offline status`; `--move` and `--cleanup` only with `--log` and not together; the requests carry `log=1` and the filter); `TestMoveOfflineStatusLogRendering` (state `none`, in flight, and a 404 status door; each line above, the waiting-delete line with the report's `AbandonPath`; no report; no match); `TestMovesStringsAreRecorded` and `TestMovesCopyRecordIsComplete` green with the new rows pending. |
| - [ ] **Step 2: Run to verify they fail**: `go test ./internal/console -run 'OfflineCleanups|OlderCleanup' -count=1`, `go test . -run 'MoveOfflineStatusLog|MovesStringsAreRecorded|MovesCopyRecord' -count=1`. |
| - [ ] **Step 3: Implement. Step 4: Run to verify they pass.** |
| - [ ] **Step 5: Commit** `move: carlos move offline status --log shows a host's cleanup reports, by move or by cleanup id, for every kind of move and after the record is gone`. |
| @@ -277,7 +282,7 @@ Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and |
| **Interfaces:** |
| - `func (r *Registry) WithHostFrozen(host string, fn func(row *Route) error) error`: `BEGIN IMMEDIATE`, `SELECT routeCols FROM routes WHERE host = ?` on the same connection, `fn`, `ROLLBACK`, as `WithHoldsFrozen` (`registry.go:1626-1660`); remote: method `WithHostFrozen`, the callback carrying the row. |
| -- `func rootCleanupAuthority(store release.Store, judge *provision.LifetimeJudge, box, region string, hib func() (*activator.Config, error)) func(context.Context, offlinecleanup.RootCleanupRequest) error`, with `judge` the root pass's `rootJudge` (`agent.go`, `newBoxJudge`), `box` `BoxLabel()` (the same label the agent uses as `cfg.BoxID`, `movespass.go:342-346`), `region` root's `CARLOS_REGION` and `hib` `activator.ConfigFromEnv` (N9: the activator's own reading and defaults, `activator.go:114-133`): (a) `active.json` parses and both `table[box]` and `table[moves.HostKey(Host)]` are `{cleanup, CleanupID, Host, Account, App}`; (b) `moves.RecordAway(judge.Observe(ctx, store, Account, App, Host), MoveID, Incarnation, From) == AwayYes`; (c) `hib()` is not nil (nil: refused, "no hibernation bucket on this box"), `req.Bucket == cfg.Bucket`, `req.BucketRegion == cfg.Region` (both "" when the AWS chain resolves the region, `activator.go:99`), `req.Prefix == activator.HibernatePrefix(cfg.Prefix, Host) + "/"` (an empty `cfg.Prefix` is valid and gives `<host>/`), and `lease.LockState(ctx, cfg.Bucket, strings.TrimSuffix(req.Prefix, "/"), cfg.Region)` names owner `lease.CleaningOwner(CleanupID)` (live or expired: the owner alone proves no lease was taken since the mark). (a), (b) or (c) answering no → `ErrRootCleanupRefused` with the reason; a read error → itself (the request waits for the next pass). |
| +- `func rootCleanupAuthority(store release.Store, judge *provision.LifetimeJudge, box, region string, hib func() (*activator.Config, error)) func(context.Context, offlinecleanup.RootCleanupRequest) error`, with `judge` the root pass's `rootJudge` (`agent.go`, `newBoxJudge`), `box` `BoxLabel()` (the same label the agent uses as `cfg.BoxID`, `movespass.go:342-346`), `region` root's `CARLOS_REGION` and `hib` `activator.ConfigFromEnv` (N9: the activator's own reading and defaults, `activator.go:114-133`): (a) `active.json` parses and both `table[box]` and `table[moves.HostKey(Host)]` are `{cleanup, CleanupID, Host, Account, App}`; (b) `moves.RecordAway(judge.Observe(ctx, store, Account, App, Host), MoveID, Incarnation, From) == AwayYes`; (c) `hib()` is not nil (nil: refused, "no hibernation bucket on this box"), `req.Bucket == cfg.Bucket`, `req.BucketRegion == cfg.Region` (both "" when the AWS chain resolves the region, `activator.go:99`), `req.Prefix == activator.HibernatePrefix(cfg.Prefix, Host) + "/"` (an empty `cfg.Prefix` is valid and gives `<host>/`), and `lease.LockState(ctx, cfg.Bucket, strings.TrimSuffix(req.Prefix, "/"), cfg.Region)` names owner `lease.CleaningOwner(CleanupID)` (the mark has no expiry; the owner alone proves no lease was taken since the mark). (a), (b) or (c) answering no → `ErrRootCleanupRefused` with the reason; a read error → itself (the request waits for the next pass). |
| - [ ] **Step 1: Write the failing tests**: `TestWithHostFrozenLocalAndRemote` (an `Upsert` issued during `fn` returns after it; the row given is the row at the freeze); `TestRootAuthorityComesFromTheBucket` (no entries; only the host key; another account or app; the record home again; the mark not ours; the request's bucket, region or prefix not root's → refused; a read error → the error); `TestRootAuthorityUsesTheActivatorDefaults` (`CARLOS_HIBERNATE_REGION` unset with a request region "" → authorised; `CARLOS_HIBERNATE_PREFIX` unset → the prefix `<host>/` authorised; the bucket unset → refused). |
| - [ ] **Step 2: Run to verify they fail**: `go test ./internal/registry -run 'HostFrozen' -count=1`, `go test ./internal/host -run 'RootAuthority' -count=1`. |
| @@ -287,27 +292,28 @@ Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and |
| ### Task 6b: The root-owned request and acknowledgement, as a stage of the adopt pass |
| **Files:** |
| -- Create: `internal/release/rootcleanup.go` (discovery, the copy, the dispatch, the acknowledgement, `LockRootPass`; portable), `internal/release/rootcleanup_other.go` (answers every request `cancelled` with `Reason: "unsupported platform"`; `LockRootPass` answers `ErrNotRootOnly`), `internal/host/movescleanup_root.go` (the agent's rows, `rootCleanupPass`) |
| -- Modify: `internal/host/agent.go` (`runAdoptUnderRootRelease`: the root pass lock right after `lockAdoptPass` at line 4655, in every mode; `rootCleanupPass(...)` after `heldStagePass` at line 4932), `internal/host/adoptlock.go` (the lock-order comment at lines 52-54 names the root pass lock between the adopt-pass lock and the restart marks), `internal/host/movesheldlock.go` (the order comment, lines 26-31), `internal/host/claimcensus.go` (`claimOptCensus` gains `<optDir>/root-cleanup` as box state, if its completeness test requires it) |
| -- Test: `internal/release/rootcleanup_test.go`, `internal/host/movescleanup_root_test.go`, `internal/host/adopt_rootcleanup_stage_test.go` |
| +- Create: `internal/release/rootcleanup.go` (discovery, the copy, the dispatch, the acknowledgement, `LockRootPass`, `RootCleanupGated`; portable), `internal/host/adoptrootgate.go` (`adoptRootGate`), `internal/release/rootcleanup_other.go` (answers every request `cancelled` with `Reason: "unsupported platform"`; `LockRootPass` answers `ErrNotRootOnly`), `internal/host/movescleanup_root.go` (the agent's rows, `rootCleanupPass`) |
| +- Modify: `internal/host/agent.go` (`runAdoptUnderRootRelease`: the root pass lock and the gate right after `lockAdoptPass` at line 4655, in every mode, before `ensureUnits` at line 4664; `rootCleanupPass(...)` after `heldStagePass` at line 4932), `internal/host/adoptlock.go` (the lock-order comment at lines 52-54 names the root pass lock between the adopt-pass lock and the restart marks), `internal/host/movesheldlock.go` (the order comment, lines 26-31), `internal/host/claimcensus.go` (`claimOptCensus` gains `<optDir>/root-cleanup` as box state, if its completeness test requires it) |
| +- Test: `internal/release/rootcleanup_test.go`, `internal/host/movescleanup_root_test.go`, `internal/host/adopt_rootcleanup_stage_test.go`, `internal/host/adoptrootgate_test.go` |
| **Interfaces:** |
| -- **The root pass lock (R5, F5).** `func LockRootPass(optDir string) (unlock func(), err error)`: `RootCleanupBase(optDir) = <optDir>/root-cleanup/` is created by root `0755` when absent; `offlinecleanup.RootOnlyChain` on it false → `ErrNotRootOnly`, no lock (root's step then removes nothing on this box: a writable `/opt` is a writable layout); else `pass.lock` in it opened with `safefile.OpenFile(O_CREATE|O_RDWR, 0o600)` and `lockFileEx` (blocking `flock`), as `lockExclusive` does. In `runAdoptUnderRootRelease`, right after `lockAdoptPass` (line 4655) and whatever that answered, in every mode (hourly, `-stage-only`, `-config-only`, `-restart-only`), before `recoverRootRestarts` and every other publisher: `unlockRoot, rootLockErr := release.LockRootPass(release.OptDir())`, deferred, logged and never fatal, as the adopt-pass lock's failure is. So a pass whose adopt-pass lock failed (a symlink planted at `/data/adopt-pass.lock`, which `safefile` refuses) still waits on the root pass lock before it can publish. **Why the step does not depend on the adopt-pass lock's identity:** that lock lives in the agent-writable `/data` (`adoptlock.go:16`), where a shared-uid writer can replace or plant its name; the root pass lock lives on a root-only chain, which no other uid can change, so its acquisition is unchanged and needs no identity check. The adopt-pass lock is still required for the step (it is held, so the sweep keeps its own rule), but nothing relies on its identity. The one case left is a pass whose root pass lock fails for a root-level fault (an I/O error on a root-only directory, which no other uid can cause): that pass runs no root step, but it is not excluded from another pass's step; see the Self-review notes. |
| -- **The stage (S12).** `rootCleanupPass(...)` runs after `heldStagePass` (line 4932), so after `release.Pass`, inside which named recovery runs (`recovery.passOption`, `agent.go:4890`, `namedrecovery_pass.go:15`), and only when `passLockErr == nil && rootLockErr == nil`; nonfatal. `-config-only` and `-restart-only` return before it (`agent.go:4758-4767`, `:4794-4801`) but have taken the root pass lock, so they wait for a running step and a running step waits for them. Inside one pass the stages run one after another, so no root publisher of this pass runs while root's step runs, and every other adopt pass is blocked on the root pass lock. |
| -- **Files.** The agent's drop `<OfflineOpDir>/root-cleanup.json` (create-only) and cancel `<OfflineOpDir>/root-cleanup.cancel` (create-only). Root's directory `release.RootCleanupDir(optDir, id) = <optDir>/root-cleanup/<id>/` holding `request.json` (root's copy), `journal` (Task 6c) and exactly one of `done.json`, `cancelled.json`. Each `<id>/` is created by root `0755`; root uses `RootCleanupBase` and `<id>/` only when `offlinecleanup.RootOnlyChain` passes on them. **When it does not (S16), root writes nothing there**, removes nothing, and leaves the drop where it is: it logs `root-cleanup: <id> refused: <dir> is not root-only; repair its ownership, or remove <drop> to give up` to the adopt unit's journal (`log.Printf`, the systemd journal of `carlos-adopt.service`), which is the only record of the refusal. A repaired ancestry lets the next pass take the request. |
| +- **The root pass lock (R5, F5; T7).** `func LockRootPass(optDir string) (unlock func(), err error)`, in this order, so no directory is created under an ancestry that has not passed: (1) `offlinecleanup.RootOnlyChain(optDir)` on the **existing** chain from `/` to `optDir`: false → `ErrNotRootOnly`, nothing created; an error → itself; (2) `RootCleanupBase(optDir) = <optDir>/root-cleanup/` created by root `0755` when absent (`os.Mkdir`; `EEXIST` is fine); (3) `RootOnlyChain` on the base (a base another uid made, or a symlink, answers false → `ErrNotRootOnly`); (4) `pass.lock` in it opened with `safefile.OpenFile(O_CREATE|O_RDWR, 0o600)` and `lockFileEx` (blocking `flock`), as `lockExclusive` does (`adoptlock.go:55-68`). `ErrNotRootOnly` is a definite answer about the box, not a failure to lock: only root can make a chain root-only or not, so every pass gets the same answer, and root's step can never run there (a writable `/opt` is a writable layout). **Why the step does not depend on the adopt-pass lock's identity:** that lock lives in the agent-writable `/data` (`adoptlock.go:16`), where a shared-uid writer can replace or plant its name (`safefile` then refuses it, and the pass continues unserialized, `agent.go:4655-4661`); the root pass lock lives on a root-only chain, which no other uid can change, so its acquisition needs no identity check. The adopt-pass lock is still required for the step (it is held, so the sweep keeps its own rule), but nothing relies on its identity. |
| +- **The gate (T2).** In `runAdoptUnderRootRelease`, right after `lockAdoptPass` (`agent.go:4655-4661`) and whatever that answered, in every mode (hourly, `-stage-only`, `-config-only`, `-restart-only`), before `ensureUnits` (`:4664`) and every later stage: `unlockRoot, rootLockErr := release.LockRootPass(release.OptDir())` (deferred when it succeeds); `gated := os.Getenv("CARLOS_OFFLINE_CLEANUP") == "on" || release.RootCleanupGated(release.OptDir())`; then `adoptRootGate(gated, rootLockErr)` answers one of three. **Serve** (`rootLockErr == nil`): the pass runs as today; when `gated`, it creates `<optDir>/root-cleanup/gated` (create-only, root `0644`) if absent and runs root's step as its last stage. **Publish nothing** (`gated`, and `rootLockErr` neither nil nor `ErrNotRootOnly`): one `log.Printf("ALERT: adopt: root pass lock %s: %v: publishing nothing this pass", ...)` and `return nil`, so the pass runs none of `ensureUnits`, `recoverRootRestarts` (`:4695`; its resumes call `PointLive` at `rootrollback_resume.go:125` and `roothistory_recovery.go:216`), `rootJudgePrev` (`:4698`), `sweepStagedVersionsPass` (`:4720`), the `-restart-only` branch (`:4758`), config sync and the `-config-only` branch (`:4794`), `capabilityPass` (`:4837`), `linkRootHistory` (`:4888`), `release.Pass` (`:4920`: adoption and rollback, `adopt.go:1751`, `:1870`, `rollback.go:124`, and named recovery inside it) and `heldStagePass` (`:4932`: `materializeheld.go:460`, `:494`); the next pass tries again. **Publish without serving** (not `gated`, or `ErrNotRootOnly`): today's fail-open adoption, logged as the adopt-pass lock's failure is, and no root step. `func RootCleanupGated(optDir string) bool`: `Lstat` of `<optDir>/root-cleanup/gated`: present → true; `ENOENT` → false; any other error → true (fail closed). The marker is never removed by code: it records that cleanup was armed on this box, so a box disarmed for a rollback, whose obligations may still wait for root (Global Constraint, rollback step 1), keeps serving them under the same rule. **Why this closes R5:** on a gated box every pass that publishes holds the root pass lock, and root's step runs only as the last stage of a pass holding it, so no root publisher of the live link or the programs file runs while root's step runs; the agent's own publishers are excluded by the freeze and the held-stage lock as before. On a box never gated no root step can run (it needs the lock and the switch or the marker), so fail-open adoption there races nothing. The one case left is in the Self-review notes. |
| +- **The stage (S12).** `rootCleanupPass(...)` runs after `heldStagePass` (line 4932), so after `release.Pass`, inside which named recovery runs (`recovery.passOption`, `agent.go:4890`, `namedrecovery_pass.go:15`), and only when `passLockErr == nil && rootLockErr == nil && gated`; nonfatal. `-config-only` and `-restart-only` return before it (`agent.go:4758-4767`, `:4794-4801`) but have taken the root pass lock, so they wait for a running step and a running step waits for them. Inside one pass the stages run one after another, so no root publisher of this pass runs while root's step runs, and every other adopt pass is blocked on the root pass lock. |
| +- **Files.** The agent's drop `<OfflineOpDir>/root-cleanup.json` (create-only) and cancel `<OfflineOpDir>/root-cleanup.cancel` (create-only). Root's directory `release.RootCleanupDir(optDir, id) = <optDir>/root-cleanup/<id>/` holding `request.json` (root's copy), `journal` (Task 6c) and exactly one of `done.json`, `cancelled.json`. Each `<id>/` is created by root `0755` only after `RootOnlyChain` has passed on the base (T7: the existing ancestry is checked before any missing directory is created), and used only when the chain passes on it too. **When it does not (S16), root writes nothing there**, removes nothing, and leaves the drop where it is: it logs `root-cleanup: <id> refused: <dir> is not root-only; repair its ownership, or remove <drop> to give up` to the adopt unit's journal (`log.Printf`, the systemd journal of `carlos-adopt.service`), which is the only record of the refusal. A repaired ancestry lets the next pass take the request. |
| - The wire types (declared in Task 1): `type RootCleanupRequest struct { V int; CleanupID, MoveID, Host, Account, App, Box string; Incarnation uint64; From, To string; Bucket, BucketRegion, Prefix string; Paths []RootCleanupPath; WrittenAt time.Time }`; `type RootCleanupPath struct { Which string; Identity FileIdentity }`; `type RootCleanupAck struct { V int; CleanupID, MoveID, Host, Outcome string (done or cancelled); Removed, Absent []string; Kept []KeptPath; Reason string; At time.Time }`. |
| - `type RootCleanupOptions struct { DataDir, OptDir, Box, Region string; Authorize func(ctx context.Context, req offlinecleanup.RootCleanupRequest) error; Freeze func(host string, fn func(rowPresent bool) error) error; HostLock func(host string, fn func() error) error; Processes func() ([]string, error); Now func() time.Time }`; `var ErrRootCleanupRefused`, `ErrNotRootOnly`; `func ServeRootCleanupRequests(ctx context.Context, o RootCleanupOptions) error`. |
| - **Discovery (F6):** each pass walks, with `safefile` (no symlinks), (i) the drops `<DataDir>/.offline-moves/<host>/<move>/root-cleanup.json` (host passing `ValidHostShape`, move a plain segment; `id = CleanupID(o.Box, host, move)`) and (ii) root's own `<OptDir>/root-cleanup/<id>/` directories that hold `request.json` and no acknowledgement, so a request root took and then crashed on is served from its own copy whether or not a drop remains. |
| -- **Root, per drop:** (0) `RootOnlyChain` on `RootCleanupBase` and, when it exists, `RootCleanupDir(id)`: false → the S16 refusal above, next drop. (1) Read bounded (4096 bytes) and validate: `Host`, `MoveID` equal the path; `Box == o.Box`; `CleanupID == id`; `From == o.Region`; `Which` values known and unique. Invalid → `cancelled` under `id` with `Reason: "malformed request"`, the drop removed. (2) Copy: `request.json` create-only (temp, `fsync`, `os.Link`, directory `fsync`) with the drop's bytes; the journal's first line `taken <at>` (`fsync`); then the drop is removed. A `request.json` already present with other bytes: the drop is removed and logged, root's copy stands (a later drop never changes or cancels a taken request). From here root reads only its copy. |
| +- **Root, per drop:** (0) `RootOnlyChain` on `RootCleanupBase` and, when it exists, `RootCleanupDir(id)`, before anything is created: false → the S16 refusal above, next drop. (1) Read bounded (4096 bytes) and validate: `Host`, `MoveID` equal the path; `Box == o.Box`; `CleanupID == id`; `From == o.Region`; `Which` values known and unique. Invalid → `cancelled` under `id` with `Reason: "malformed request"`, the drop removed. (2) Copy: `request.json` create-only (temp, `fsync`, `os.Link`, directory `fsync`) with the drop's bytes; the journal's first line `taken <at>` (`fsync`); then the drop is removed. A `request.json` already present with other bytes: the drop is removed and logged, root's copy stands (a later drop never changes or cancels a taken request). From here root reads only its copy. |
| - **Root, per taken request** (from (2) or from discovery (ii)): Task 6c's journal recovery; `o.Authorize(ctx, req)`: `ErrRootCleanupRefused` → `cancelled` with its reason, nothing touched; another error → the next pass; each path not yet terminal through Task 6c's step; every path terminal → `cancelled.json` when the authority refused or any path was kept for "cancelled", else `done.json` (temp, `fsync`, rename, directory `fsync`). `<optDir>/root-cleanup/<id>/` older than 30 days past its acknowledgement is removed by root. |
| - `func ReadRootCleanupAck(optDir, id string) (ack *offlinecleanup.RootCleanupAck, trusted bool, err error)`: `trusted` only when `RootOnlyChain(RootCleanupDir(optDir, id))` passes and the file, opened with `safefile` (no symlinks), is owned by uid 0 with no group or other write bit; an acknowledgement the agent cannot trust is advisory (Task 4a's outcome table: `superseded`, every requested path recorded kept "writable layout"). |
| - `func rootCleanupPass(ctx context.Context, dataDir, optDir string, reg *registry.Registry, store release.Store, judge *provision.LifetimeJudge, box, region string)`: `ServeRootCleanupRequests` with `Authorize` = Task 6a's callback, `Freeze` = `reg.WithHostFrozen` (a non-nil row is `rowPresent`), `HostLock` = `withHeldStageLock(dataDir, ·, ·)`, `Processes` reading `/proc/*/exe` (`ENOENT`/`ESRCH` skipped as exited, any other error returned). |
| -- **The agent's rows** (Task 4a's row 6): `publishing`: `supersededBy` → supersede; the drop absent and `RootCleanupDir` absent → `writeRootCleanupRequest` (create-only; an existing drop with equal bytes is fine) → touch the stage-request marker (`holdStageRequester(dataDir)(Host)`, `moveshold.go:277`) → `RootRequestedAt = now`, `root-requested`, event; either present (a crash after the drop) → `root-requested` with `RootRequestedAt` restored, in this order: the drop's mtime when the drop exists; else the `at` of root's journal `taken` line; else `now`. `root-requested`: `supersededBy` → supersede (to `cancel-pending`); `ReadRootCleanupAck(optDir, ID)` → an acknowledgement whose `CleanupID`, `Host`, `MoveID` match → `Root = ack`, event, `releasing` with the outcome table's answer; a mismatch → page `corrupt-<h>`, stay; none → the drop is read **first** and then `RootCleanupDir`: both absent (an operator removed the drop before root took it; root takes a request into its directory before it removes the drop) → `releasing`, `Outcome = superseded`; else, after `rootRefusedAfter` (1 h) since `RootRequestedAt` with the agent's own `RootOnlyChain` check of `RootCleanupBase` (or of `RootCleanupDir` when it exists) failing → event `root-refused` and the hourly `root-refused-<h>` page, stay (**never a release**; the mark is reasserted and the entries stay until an operator repairs the ancestry or removes the drop); else the `root-wait` pages. `cancel-pending`: the cancel file written create-only and `CancelRequestedAt` set (once); an acknowledgement → `releasing`, `Outcome = superseded`; else the drop is read **first** and then `RootCleanupDir`: both absent → `releasing`, `Outcome = superseded` (root takes a request into its directory before it removes the drop, so a drop already gone means root's directory exists, and no drop was written after `publishing`); else the `cancel-wait` pages. |
| +- **The agent's rows** (Task 4a's row 6): `publishing`: `supersededBy` → supersede; the drop absent and `RootCleanupDir` absent → `writeRootCleanupRequest` (create-only; an existing drop with equal bytes is fine) → touch the stage-request marker (`holdStageRequester(dataDir)(Host)`, `moveshold.go:277`) → `RootRequestedAt = now`, `root-requested`, event; either present (a crash after the drop) → `root-requested` with `RootRequestedAt` restored, in this order: the drop's mtime when the drop exists; else the `at` of root's journal `taken` line; else `now`. `root-requested`: `supersededBy` → supersede (to `cancel-pending`); `ReadRootCleanupAck(optDir, ID)` → an acknowledgement whose `CleanupID`, `Host`, `MoveID` match → `Root = ack`, event, `releasing` with the outcome table's answer; a mismatch → page `corrupt-<h>`, stay; none → the drop is read **first** and then `RootCleanupDir`: both absent (an operator removed the drop before root took it; root takes a request into its directory before it removes the drop) → `releasing`, `Outcome = superseded`; else, after `rootRefusedAfter` (1 h) since `RootRequestedAt` with the agent's own `RootOnlyChain` check of `RootCleanupBase` (or of `RootCleanupDir` when it exists) failing → event `root-refused` and the hourly `root-refused-<h>` page, stay (**never a release**; the mark, which has no expiry, and the entries stay until an operator repairs the ancestry or removes the drop); else the `root-wait` pages. `cancel-pending`: the cancel file written create-only and `CancelRequestedAt` set (once); an acknowledgement → `releasing`, `Outcome = superseded`; else the drop is read **first** and then `RootCleanupDir`: both absent → `releasing`, `Outcome = superseded` (root takes a request into its directory before it removes the drop, so a drop already gone means root's directory exists, and no drop was written after `publishing`); else the `cancel-wait` pages. |
| -- [ ] **Step 1: Write the failing tests**: `TestRootCleanupStepRunsInsideThePass` (a pass driven through `runAdoptUnderRootRelease`'s stages with fakes: the step runs after `release.Pass` and `heldStagePass`; with the adopt-pass lock failed, with the root pass lock failed, and on `-config-only` and `-restart-only`, no step; every mode takes the root pass lock before its first publisher); `TestRootPassLockIsRootOnly` (a base directory the test uid owns → `ErrNotRootOnly`, no lock; the lock's chain check runs as any uid against an injected chain check); `TestRootFindsItsOwnRequestsOnRestart` (root takes a request, removes the drop and stops: the next pass serves it from `request.json`); `TestForgedAgentFilesAreHarmless` (a `done.json` planted in the agent's directory is never read; a drop with another id is cancelled under the expected id; a second drop with other bytes after the take is removed and changes nothing; a forged cancel stops root and ends the obligation `cancelled`, nothing else); `TestRejectedAncestryGetsNoWrite` (a `RootCleanupBase` or `<id>/` that fails the chain check: nothing created, written or removed under it, the drop kept, one log line; repaired, the next pass takes it); `TestAgentPagesARefusedAncestryAndNeverReleases` (no answer for 1 h with the chain failing from the agent's view: hourly `root-refused` pages, the obligation stays `root-requested` with both entries and the mark for a simulated week; the drop removed by an operator → `releasing`, `superseded`); `TestAgentReadsOnlyRootsAcknowledgement` (an untrusted acknowledgement → `superseded`, paths kept "writable layout"); `TestRootRequestedAtIsRestored` (a crash after the drop: from its mtime; after the take: from the `taken` line); `TestUnansweredRootRequestPagesEveryTenMinutes`; `TestCancelBeforeAndDuringRootsWork`; `TestCancelPendingResolvesWithoutARequest` (a crash after `publishing` before the drop; root never wrote a directory); `TestUnsupportedPlatformCancels`. |
| -- [ ] **Step 2: Run to verify they fail**: `go test ./internal/release -run 'RootFinds|ForgedAgent|RejectedAncestry|RootPassLock|CancelBefore|Unsupported' -count=1`, `go test ./internal/host -run 'RootCleanupStep|AgentPages|AgentReadsOnly|RootRequestedAt|Unanswered|CancelPending' -count=1`. |
| +- [ ] **Step 1: Write the failing tests**: `TestRootCleanupStepRunsInsideThePass` (a pass driven through `runAdoptUnderRootRelease`'s stages with fakes: the step runs after `release.Pass` and `heldStagePass`; with the adopt-pass lock failed, with the root pass lock failed, and on `-config-only` and `-restart-only`, no step; every mode takes the root pass lock before its first publisher); `TestRootPassLockIsRootOnly` (a base directory the test uid owns → `ErrNotRootOnly`, no lock; the lock's chain check runs as any uid against an injected chain check); `TestLockRootPassChecksAncestryBeforeCreating` (T7: an `optDir` whose existing chain fails → `errors.Is(err, release.ErrNotRootOnly)` and `os.Lstat(base)` answers `fs.ErrNotExist` afterwards; a root-only `optDir` without a base → created, checked, locked, in that order in the injected syscall log; a base that is a symlink → `ErrNotRootOnly`, nothing created through it); `TestGatedPassPublishesNothingWithoutTheRootPassLock` (T2: a pass driven through `runAdoptUnderRootRelease`'s stages with a recording fake per stage; the switch on and `LockRootPass` failing with an injected `EIO` → `len(calls) == 0` across `ensureUnits`, `recoverRootRestarts`, `rootJudgePrev`, `sweepStagedVersionsPass`, the restart pass, config sync, `capabilityPass`, `linkRootHistory`, `release.Pass`, `heldStagePass` and the root step, in each of the four modes, and one log line; the switch off with the marker present → the same; the next pass with the lock → every stage in today's order and the step last); `TestUngatedBoxKeepsFailOpenAdoption` (the switch off and no marker, the lock failing → every stage runs as today, no step, no marker created; `ErrNotRootOnly` with the switch on → the same); `TestArmedPassLeavesTheGatedMarker` (an armed pass holding the lock creates the marker; a later pass with the switch off is gated; a marker `Lstat` error other than `ENOENT` → gated); `TestRootFindsItsOwnRequestsOnRestart` (root takes a request, removes the drop and stops: the next pass serves it from `request.json`); `TestForgedAgentFilesAreHarmless` (a `done.json` planted in the agent's directory is never read; a drop with another id is cancelled under the expected id; a second drop with other bytes after the take is removed and changes nothing; a forged cancel stops root and ends the obligation `cancelled`, nothing else); `TestRejectedAncestryGetsNoWrite` (a `RootCleanupBase` or `<id>/` that fails the chain check: nothing created, written or removed under it, the drop kept, one log line; repaired, the next pass takes it); `TestAgentPagesARefusedAncestryAndNeverReleases` (no answer for 1 h with the chain failing from the agent's view: hourly `root-refused` pages, the obligation stays `root-requested` with both entries and the mark for a simulated week; the drop removed by an operator → `releasing`, `superseded`); `TestAgentReadsOnlyRootsAcknowledgement` (an untrusted acknowledgement → `superseded`, paths kept "writable layout"); `TestRootRequestedAtIsRestored` (a crash after the drop: from its mtime; after the take: from the `taken` line); `TestUnansweredRootRequestPagesEveryTenMinutes`; `TestCancelBeforeAndDuringRootsWork`; `TestCancelPendingResolvesWithoutARequest` (a crash after `publishing` before the drop; root never wrote a directory); `TestUnsupportedPlatformCancels`. |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/release -run 'RootFinds|ForgedAgent|RejectedAncestry|RootPassLock|LockRootPassChecks|CancelBefore|Unsupported' -count=1`, `go test ./internal/host -run 'RootCleanupStep|GatedPass|UngatedBox|ArmedPassLeaves|AgentPages|AgentReadsOnly|RootRequestedAt|Unanswered|CancelPending' -count=1`. |
| - [ ] **Step 3: Implement** (until Task 6c lands, the per-path step answers kept with reason "not built", so this commit removes nothing). **Step 4: Run to verify they pass**, then `GOOS=darwin go build ./internal/release/ ./internal/host/` and `GOOS=windows go vet ./internal/release/ ./internal/host/`. |
| -- [ ] **Step 5: Commit** `root: a stage of the adopt pass under a root-only pass lock takes a cleanup request into its own directory, writes nothing where the ancestry is not root's, and answers in a file the agent trusts only on a root-only chain`. |
| +- [ ] **Step 5: Commit** `root: a stage of the adopt pass under a root-only pass lock, which an armed box's passes must hold to publish, takes a cleanup request into its own directory, writes nothing where the ancestry is not root's, and answers in a file the agent trusts only on a root-only chain`. |
| ### Task 6c: Root's filesystem step, its journal and recovery |
| @@ -319,9 +325,9 @@ Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and |
| - **The journal** `<RootCleanupDir>/journal`: append-only JSON lines, each `fsync`ed before the next syscall: `{at, step: "taken"}`; per path `{at, which, step: "intent", identity}` then exactly one of `{at, which, step: "removed"}`, `{at, which, step: "kept", reason}`, `{at, which, step: "absent"}`. A path's result is its last result line; the acknowledgement is built from results alone, so no outcome is unknown. |
| - **Recovery (F6),** per path whose last line is `intent`: `fstatat` the name in the opened parent again: absent → journal `absent` (removed by us before the crash or by anyone: recorded as absent, never as removed); present with the recorded identity → the step is retried from the start below; present with another identity → journal `kept`, "identity changed". Nothing was ever renamed, so nothing needs restoring after a cancel or a refusal. |
| - **The step,** per path not yet terminal (`live-link` first; root derives the path: `LiveLink(OptDir, host)`, `ProgramsPath(DataDir, host)`): for the live link only, outside the freeze: the build directory it targets; another link under `<OptDir>/live/` resolving to the same build → `kept`, "the build is still referenced"; `o.Processes()` naming an executable under the build → `kept`, "a process runs the build"; `Processes` failing → the next pass (unknown is never unreferenced). The programs file has no reference check: its schema (`{schema, programs}`, Go main-package paths, `programhistory.go:28-34`) belongs to this host alone and no reader resolves another host through it. Then `o.Authorize` again (the same two answers). Then `RootOnlyChain(parent)` false → `kept`, "writable layout". Then `o.Freeze(host, func(rowPresent) { rowPresent → kept, "the instance has a row on this box"; o.HostLock(host, func() { root-cleanup.cancel present (Lstat) → kept, "cancelled"; open the parent (O_DIRECTORY|O_NOFOLLOW) and fstat it: not uid 0 or group/other writable → kept, "writable layout"; journal intent; fstatat(parent, name, AT_SYMLINK_NOFOLLOW): ENOENT → journal absent; identity (dev, inode, file type, readlinkat target) differs → journal kept, "identity changed"; equal → unlinkat(parent, name, 0): ENOENT → journal absent; success → fsync the parent, journal removed }) })`. Between `fstatat` and `unlinkat` only root can change an entry of a root-only directory, and every root writer of these names is excluded: the root adoption publishers by the pass (this step is a stage of it, and every other adopt pass waits on the root pass lock), held publication by the held-stage lock, and anything that needs a row by the freeze. |
| -- **The writers of these names** (`TestRootOnlyLiveLinkWritersAreExcluded`, `go/ast` with `go/types` over non-test files under `./internal/...`): every call of `release.PointLive`, `release.RecordProgram` and `(*activator.ExecBackend).ensureLiveLink` is listed with what keeps it off the name during root's step: `root pass lock` (reached only from `runAdoptUnderRootRelease`, which holds the root pass lock for its whole run and runs root's step as a later stage: `adopt.go:1751`, `:1870`, `rollback.go:124`, `rootrollback_resume.go:125`, `roothistory_recovery.go:216`, and named recovery's `namedrecovery.go:120` and `namedrecovery_active.go:191` through `recovery.passOption` inside `release.Pass`, `agent.go:4890`, `namedrecovery_pass.go:15`), `held-stage lock` (held publication: `materializeheld.go:460`, `:494`), `needs a row` (the freeze), `agent process` (the agent cannot write a root-only directory: `ensureLiveLink`, the agent's own adopt pass at `agent.go:1319`) or `e2e only`; the test also proves each `root pass lock` site's caller chain reaches only `runAdoptUnderRootRelease`. A new site, or a removed one, fails until the list is edited, and a site the implementer cannot place in a class is a question for the reviewer, not a guess. |
| +- **The writers of these names** (`TestRootOnlyLiveLinkWritersAreExcluded`, `go/ast` with `go/types` over non-test files under `./internal/...`): every call of `release.PointLive`, `release.RecordProgram` and `(*activator.ExecBackend).ensureLiveLink` is listed with what keeps it off the name during root's step: `root pass lock` (reached only from `runAdoptUnderRootRelease`'s stages after `adoptRootGate`, which on a gated box runs them only while the pass holds the root pass lock for its whole run, and runs root's step as a later stage: `adopt.go:1751`, `:1870`, `rollback.go:124`, `rootrollback_resume.go:125`, `roothistory_recovery.go:216`, and named recovery's `namedrecovery.go:120` and `namedrecovery_active.go:191` through `recovery.passOption` inside `release.Pass`, `agent.go:4890`, `namedrecovery_pass.go:15`), `held-stage lock` (held publication: `materializeheld.go:460`, `:494`), `needs a row` (the freeze), `agent process` (the agent cannot write a root-only directory: `ensureLiveLink`, the agent's own adopt pass at `agent.go:1319`) or `e2e only`; the test also proves each `root pass lock` site's caller chain reaches only `runAdoptUnderRootRelease`. A new site, or a removed one, fails until the list is edited, and a site the implementer cannot place in a class is a question for the reviewer, not a guess. |
| -- [ ] **Step 1: Write the failing tests**: `TestRootRemovesOnlyTheRecordedIdentity` (Review Focus 3: a publication replaces the link between the reference check and the `fstatat` → kept, the replacement at its name; the recorded entry → removed); `TestNoPublisherRunsBetweenFstatatAndUnlinkat` (the S12 witness: a hook between `fstatat` and `unlinkat` starts a second adopt pass through the production lock functions, with `/data/adopt-pass.lock` replaced by a symlink so its adopt-pass lock fails, and a held publication of the same host: the second pass blocks in `LockRootPass` and the held publication in the held-stage lock until the step returns; the recorded entry is removed, and the second pass's publication lands after it); `TestWritableLayoutIsNeverRemoved` (a parent the test uid owns; an ancestor writable by group: kept "writable layout", nothing unlinked, no intent line); `TestRootRowCheckIsAtomicWithTheUnlink` (an `Upsert` issued during the freeze returns after it; a row present at the freeze → kept); `TestRootJournalResolvesEveryIntent` (a crash after each journal line and after the `unlinkat`: each restart ends in removed, kept or absent; a crash after the unlink and before `removed` → `absent`); `TestReferencedBuildIsKept` (another host's link to the build; a process; `Processes` failing → left); `TestProgramsFileHasNoReferenceCheck`; `TestRootOnlyLiveLinkWritersAreExcluded`. The Linux tests that need a root-owned parent run when the test runs as root (the root runner, `root-runner.py`) and are skipped otherwise; the identity, journal and lock-order logic run as any uid against an injected chain check. |
| +- [ ] **Step 1: Write the failing tests**: `TestRootRemovesOnlyTheRecordedIdentity` (Review Focus 3: a publication replaces the link between the reference check and the `fstatat` → kept, the replacement at its name; the recorded entry → removed); `TestNoPublisherRunsBetweenFstatatAndUnlinkat` (the S12 witness: a hook between `fstatat` and `unlinkat` starts a second adopt pass through the production lock functions, with `/data/adopt-pass.lock` replaced by a symlink so its adopt-pass lock fails, and a held publication of the same host: the second pass blocks in `LockRootPass` and the held publication in the held-stage lock until the step returns; the recorded entry is removed, and the second pass's publication lands after it; and, separately on the gated box, a second pass whose `LockRootPass` fails with an injected fault while the step runs: it publishes nothing, `len(calls) == 0`, and the recorded entry is removed with no replacement at its name); `TestWritableLayoutIsNeverRemoved` (a parent the test uid owns; an ancestor writable by group: kept "writable layout", nothing unlinked, no intent line); `TestRootRowCheckIsAtomicWithTheUnlink` (an `Upsert` issued during the freeze returns after it; a row present at the freeze → kept); `TestRootJournalResolvesEveryIntent` (a crash after each journal line and after the `unlinkat`: each restart ends in removed, kept or absent; a crash after the unlink and before `removed` → `absent`); `TestReferencedBuildIsKept` (another host's link to the build; a process; `Processes` failing → left); `TestProgramsFileHasNoReferenceCheck`; `TestRootOnlyLiveLinkWritersAreExcluded`. The Linux tests that need a root-owned parent run when the test runs as root (the root runner, `root-runner.py`) and are skipped otherwise; the identity, journal and lock-order logic run as any uid against an injected chain check. |
| - [ ] **Step 2: Run to verify they fail**: `go test ./internal/release -run 'RootRemoves|WritableLayout|RootRow|RootJournal|ReferencedBuild|ProgramsFile' -count=1`, `go test ./internal/host -run 'LiveLinkWriters|NoPublisherRuns' -count=1`. |
| - [ ] **Step 3: Implement. Step 4: Run to verify they pass**, then the cross-builds of Task 6b. |
| - [ ] **Step 5: Commit** `root: removes a recorded leftover by fstatat and unlinkat on its opened parent, inside the adopt pass, under the registry freeze and the held-stage lock, never on a writable layout, journalling intent and result`. |
| @@ -346,35 +352,36 @@ Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and |
| **Files:** |
| - Create: `internal/console/placereserve.go` |
| -- Modify: `internal/console/fleets.go` (`placeAppTarget`, line 568), `internal/console/store.go` (`claimRawSetAppPlacement`, line 1692, takes the context: `s.db.ExecContext(ctx, …)`; its adapter `SetAppPlacementContext`, line 2725, passes `claimContext`), `internal/registry/registry.go` (`SetPlacementContext(ctx, host, target)`, line 2369's body with `r.sql.ExecContext(ctx, …)` and `r.remote.call(ctx, …)`, `remote.go:138`; `SetPlacement` calls it with `context.Background()`), `internal/console/offlinereserve.go` (`place` reclaim → `s.settlePlacement`), `internal/console/residence_sweep.go` (the placement settle) |
| -- Test: `internal/console/placereserve_test.go`, `internal/console/residence_audit_test.go`, `internal/registry/setplacement_context_test.go` |
| +- Modify: `internal/console/fleets.go` (`placeAppTarget`, line 568), `internal/console/store.go` (`claimRawSetAppPlacement`, line 1692, takes the context: `s.db.ExecContext(ctx, …)`; its adapter `SetAppPlacementContext`, line 2725, passes `claimContext`), `internal/registry/registry.go` (`SetPlacementContext(ctx, host, target)`, line 2369's body with `r.sql.ExecContext(ctx, …)` and `r.remote.call(ctx, remoteRequest{Method: "SetPlacement", …})` where line 2371 passes `nil` today, `remote.go:138`; `SetPlacement` calls it with `context.Background()`), `internal/registry/remote_dispatch.go` (T5: line 270's `case "SetPlacement": err = r.SetPlacement(q.Host, q.Target)` becomes `err = r.SetPlacementContext(ctx, q.Host, q.Target)`, the `ctx` the worker built from the request's `Deadline`, `remote.go:287-291`; the wire method name is unchanged, so an older client's request, which carries no deadline, runs under the worker's context as today), `internal/console/view.go` (T5: `consoleRegistry`, line 516: `SetPlacement(string, string) error` becomes `SetPlacementContext(context.Context, string, string) error`; `fleets.go:669` is its only caller), `internal/console/offlinereserve.go` (`place` reclaim → `s.settlePlacement`), `internal/console/residence_sweep.go` (the placement settle) |
| +- Test: `internal/console/placereserve_test.go`, `internal/console/residence_audit_test.go`, `internal/registry/setplacement_context_test.go`, `internal/registry/remote_setplacement_test.go` |
| **Interfaces:** |
| -- **The placement** (`placeAppTarget`, claims-off branch only; the host-claim branch returns before this as today), after the claims branch and before `SetAppPlacementContext`: `ctx, cancel := context.WithTimeout(ctx, instanceStoreTimeout)` (15 s), **the one operation deadline**, covering the listing, the reservation, the app placement, the routes and the records; the app's record keys are listed (the listing at line 678 moves up); then **`id := newPlaceID(); s.registerPlacement(id)`** (`s.placing`, a mutex-guarded `map[string]time.Time`) **before** `s.reservePlacement(ctx, acct, app, hosts, id)`: one `moves.Reserve(ctx, s.cfg.Releases, {place, id}, want)` with `want[HostKey(h)] = {place, id, h, acct.Sqid, app, now}` for every listed host (N10: one valid entry per key, its `Host` that key's host); `ErrReserved` by a `cleanup` → 409 "an instance of this app is still being cleaned up. try again later"; by another → 409 "an instance of this app is moving. try again later"; unreadable or uncertain → 502 "could not reserve the app's instances. try again" (an uncertain write is settled by the settle's release); nothing else is written on any refusal. Then app placement, routes and records exactly as today, with the two local writes under the deadline: **`s.store.SetAppPlacementContext(ctx, …)`** (which now reaches SQLite through `ExecContext`) and **`reg.SetPlacementContext(ctx, rt.Host, target)`** in the routes loop (today `reg.SetPlacement`, which takes no context). A caller paused past the deadline therefore cannot make either local write, nor `SetRecordRegion`, after it. |
| +- **The placement** (`placeAppTarget`, claims-off branch only; the host-claim branch returns before this as today), after the claims branch and before `SetAppPlacementContext`: `ctx, cancel := context.WithTimeout(ctx, instanceStoreTimeout)` (15 s), **the one operation deadline**, covering the listing, the reservation, the app placement, the routes and the records; the app's record keys are listed (the listing at line 678 moves up); then **`id := newPlaceID(); s.registerPlacement(id)`** (`s.placing`, a mutex-guarded `map[string]time.Time`; Task 7c runs the registration inside the admission barrier) **before** `s.reservePlacement(ctx, acct, app, hosts, id)`: one `moves.Reserve(ctx, s.cfg.Releases, {place, id}, want)` with `want[HostKey(h)] = {place, id, h, acct.Sqid, app, now}` for every listed host (N10: one valid entry per key, its `Host` that key's host); `ErrReserved` by a `cleanup` → 409 "an instance of this app is still being cleaned up. try again later"; by another → 409 "an instance of this app is moving. try again later"; unreadable or uncertain → 502 "could not reserve the app's instances. try again" (an uncertain write is settled by the settle's release); nothing else is written on any refusal. Then app placement, routes and records exactly as today, with the two local writes under the deadline: **`s.store.SetAppPlacementContext(ctx, …)`** (which now reaches SQLite through `ExecContext`) and **`reg.SetPlacementContext(ctx, rt.Host, target)`** in the routes loop (today `reg.SetPlacement`, `fleets.go:669`, which takes no context), through `consoleRegistry`, which gains the method, to a `*registry.Registry` that is either local (`ExecContext`) or remote (the deadline travels in `remoteRequest.Deadline`, and the worker's dispatcher now runs the update under it). A caller paused past the deadline therefore cannot make either local write, nor `SetRecordRegion`, after it. |
| - **Settlement.** `defer s.settlePlacement(id, uncertain)` from the registration on, on every return: under its own `context.WithTimeout(context.Background(), instanceStoreTimeout)`, each host whose `SetRecordRegion` returned an error other than `nil`, `os.ErrNotExist` or a seed conflict is fenced (`s.lockAppHost`, `s.admit(host, "")`, `s.fenceRecord(ctx, acct, app, host, fenceOwner{})`, as `offlineMoveReclaimable` fences); every fence answered → `moves.Release(ctx, s.cfg.Releases, {place, id})` (a write, so it also settles an uncertain reservation); a fence or the release failing → the entry stays for the sweep; then, and only then, `s.deregisterPlacement(id)`. The registration therefore covers the whole of acquisition, the writes, the fence and the release. A record created after the listing came through the create, which holds its own key. |
| - **The sweep** (in `residenceSweep`) and `offlineReclaimable` for `place`: an operation is abandoned only when its id is **not registered** in `s.placing` **and** every entry of it is older than `offlineLockGrace`; then the record of every host key it holds is fenced (each entry's host, account and app), and `moves.Release` follows; any failure → kept, logged hourly. A registered id is never swept, however old: its own deadline ends its writes and its own settle releases it. |
| - **The audit** `TestEveryResidenceChangeHoldsTheHostnameKey` (`go/ast` over non-test files under `./internal/...`): every call of `provision.SetRecordRegion`, `provision.OfflineFlip`, `provision.OfflineAbort`, `provision.MoveComplete`, `provision.MoveFlip`, `provision.MoveRollBack` and `(*Server).createRecordWithLifetimeLocked` is listed with the reservation that covers it (placement, a move's selection, the create) or its reviewed exemption: `MoveComplete` (`internal/host/movespass.go:1617`, the legacy cut's last act on the source box: tolerated, Self-review notes), `MoveFlip` and `MoveRollBack` (Phase 1, host claims on, where no box judge and so no cleanup exists). |
| -- [ ] **Step 1: Write the failing tests**: `TestPlacementReservesEveryInstanceFirst` (busy → app placement, routes and records unchanged; the table written holds one valid entry per host key, and plan 3's frozen parser reads it as malformed, which the rollback procedure covers); `TestPlacementRegistersBeforeItReserves` (the id is in `s.placing` when the fake store sees the reservation `Put`, and leaves it only after the settle's release); `TestPausedPlacementIsNeverSwept` (review 5's R8 interleaving: the caller pauses after `Reserve` past the grace; the sweep runs and leaves the entry; another placement of the app is refused busy; the caller resumes past its deadline: `SetAppPlacementContext` and `SetPlacementContext` fail with the context's error, nothing is written, and its settle releases); `TestPlacementFencesAnUncertainWriteBeforeRelease` (a store that commits the record PUT after the client's context is cancelled: the fence makes it fail its condition, the entry released after the fence); `TestPlacementCrashIsSettledByTheSweep` (a new process with an empty `s.placing`); `TestSetPlacementContextHonoursTheDeadline` (registry, local and remote); `TestResidenceChangesAndCleanupExclude` (Review Focus 4, with the real create, the real placement door and an offline move selection against a box cleanup's entries); `TestEveryResidenceChangeHoldsTheHostnameKey`; `TestHolderChecksPreserved` (today's placement refusals byte for byte). |
| -- [ ] **Step 2: Run to verify they fail**: `go test ./internal/console -run 'Placement|PausedPlacement|ResidenceChanges|EveryResidence|HolderChecks' -count=1`, `go test ./internal/registry -run 'SetPlacementContext' -count=1`. |
| +- [ ] **Step 1: Write the failing tests**: `TestPlacementReservesEveryInstanceFirst` (busy → app placement, routes and records unchanged; the table written holds one valid entry per host key, and plan 3's frozen parser reads it as malformed, which the rollback procedure covers); `TestPlacementRegistersBeforeItReserves` (the id is in `s.placing` when the fake store sees the reservation `Put`, and leaves it only after the settle's release); `TestPausedPlacementIsNeverSwept` (review 5's R8 interleaving: the caller pauses after `Reserve` past the grace; the sweep runs and leaves the entry; another placement of the app is refused busy; the caller resumes past its deadline: `SetAppPlacementContext` and `SetPlacementContext` fail with the context's error, nothing is written, and its settle releases); `TestPlacementFencesAnUncertainWriteBeforeRelease` (a store that commits the record PUT after the client's context is cancelled: the fence makes it fail its condition, the entry released after the fence); `TestPlacementCrashIsSettledByTheSweep` (a new process with an empty `s.placing`); `TestSetPlacementContextHonoursTheDeadline` (local: a context already past its deadline → `errors.Is(err, context.DeadlineExceeded)` and `reg.Get(host).Region` unchanged); `TestRemoteSetPlacementRunsUnderTheReceivedDeadline` (review 6's finding 5: over the remote pipe, a test-only worker hook (`remoteDispatchHook`, set in `export_test.go`) pauses the dispatch of a `SetPlacement` request past the deadline it carries; the call answers `errors.Is(err, context.DeadlineExceeded)`, and a fresh `Get` shows `Region` unchanged, so the update did not run under `context.Background()`; a request without a deadline, as an older client sends, still updates); `TestConsoleRegistryHasSetPlacementContext` (`var _ consoleRegistry = (*registry.Registry)(nil)` compiles with the new method; `placeAppTarget` called with a context past its deadline writes no route: `got.Region == before.Region` for every instance, local and remote); `TestResidenceChangesAndCleanupExclude` (Review Focus 4, with the real create, the real placement door and an offline move selection against a box cleanup's entries); `TestEveryResidenceChangeHoldsTheHostnameKey`; `TestHolderChecksPreserved` (today's placement refusals byte for byte). |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/console -run 'Placement|PausedPlacement|ResidenceChanges|EveryResidence|HolderChecks|ConsoleRegistryHas' -count=1`, `go test ./internal/registry -run 'SetPlacementContext|RemoteSetPlacement' -count=1`. |
| - [ ] **Step 3: Implement. Step 4: Run to verify they pass**, plus `go test ./internal/console -run 'Place' -count=1`. |
| -- [ ] **Step 5: Commit** `console: placement registers before it reserves every instance's hostname, writes under one deadline, fences an uncertain record write before it releases, and a sweep settles only an unregistered one`. |
| +- [ ] **Step 5: Commit** `console: placement registers before it reserves every instance's hostname, writes under one deadline that reaches the registry's remote worker, fences an uncertain record write before it releases, and a sweep settles only an unregistered one`. |
| ### Task 7c: The rollback drain: `carlos system reservations drain` |
| **Files:** |
| -- Create: `internal/console/reservations_drain.go` (`handleAPIReservationsDrain`, `s.draining`), `system_reservations.go` (the CLI verb, in `system_lifetimes.go`'s form) |
| -- Modify: `internal/console/authcli.go` (`s.cliRoute("POST /api/cli/system/reservations/drain", svcrole.Set{}, s.handleAPIReservationsDrain)`, beside line 1024), `internal/console/lifetimes_create.go` (a draining console refuses a create before `BeginAllocation`, line 102), `internal/console/fleets.go` (a draining console refuses a placement before its registration), `internal/console/selfserve_door.go` (a draining console refuses a self-serve move request before `AdmitSelfServeMove`, line 347), `system.go` (`reservations` in `systemVerb`'s list, line 53, `runSystem`'s switch and `systemUsage`), `docs/site/cli.md` (the verb line), the copy record and `approved_moves_copy_test.go` |
| +- Create: `internal/console/reservations_drain.go` (`handleAPIReservationsDrain`, `s.admission`, `s.admitResidence`, `errConsoleDraining`), `system_reservations.go` (the CLI verb, in `system_lifetimes.go`'s form) |
| +- Modify: `internal/console/authcli.go` (`s.cliRoute("POST /api/cli/system/reservations/drain", svcrole.Set{}, s.handleAPIReservationsDrain)`, beside line 1024), `internal/console/lifetimes_create.go` (`BeginAllocation`, line 102, runs inside `s.admitResidence`), `internal/console/fleets.go` (the placement's `s.registerPlacement`, Task 7b, runs inside it), `internal/console/selfserve_door.go` (`AdmitSelfServeMove`, line 347, runs inside it), `system.go` (`reservations` in `systemVerb`'s list, line 53, `runSystem`'s switch and `systemUsage`), `docs/site/cli.md` (the verb line), the copy record and `approved_moves_copy_test.go` |
| - Test: `internal/console/reservations_drain_test.go`, `system_reservations_test.go` |
| **Interfaces:** |
| -- **The door** (operator only: `s.moveOperator`, a 404 otherwise, and a 404 without `s.cfg.Releases`, as `handleAPILifetimesCutover`, `lifetimes_cutover.go:526-531`). Idempotent; each call is one round. The first call sets `s.draining` (an `atomic.Bool` for the rest of this process; a restart clears it, which is the way to abandon a rollback): from then a create, a placement and a self-serve move request answer 503 "this console is being rolled back. try again in a few minutes", before anything is written. The round: (1) `moves.ReadActive`; (2) every `place` operation: registered in `s.placing` → counted in flight, untouched; else settled now as the sweep settles it (fence every host key's record, then `moves.Release`), **never released before its fences answered**; (3) every `create` operation: its allocation row through `recoverAllocationLocked` under the app lock (a terminal or voided row releases its key; a pending one stays and is counted); (4) the counts: `cleanup` operations in the table, open self-serve moves (`s.store.OpenSelfServeMoves()`, `selfserve_store.go:562`), `create` and `place` operations left; (5) all four zero → the settling write, `moves.Release(ctx, s.cfg.Releases, {Kind: "drain", ID: "drain"})` (it holds nothing and writes rev + 1), then one more `moves.ReadActive`: still zero → `drained`; else `draining`. The answer: `{state: "draining" | "drained", cleanups, creates, placements, self_serve_moves int, rev uint64}`. |
| +- **The admission barrier (T3).** `s.admission` is `struct { mu sync.Mutex; draining bool }`, and `func (s *Server) admitResidence(fn func() error) error` takes `mu`, answers `errConsoleDraining` without running `fn` when `draining` is set, else runs `fn` and returns its error, then releases `mu`. The three admitters persist their admission **inside** it, so the check and the admit are one critical section: the create's `s.store.BeginAllocation(a)` (`lifetimes_create.go:102`, under the app lock as today), the placement's `s.registerPlacement(id)` (Task 7b, before its reservation), and the self-serve door's `s.store.AdmitSelfServeMove(row)` (`selfserve_door.go:347`). `errConsoleDraining` answers 503 "this console is being rolled back. try again in a few minutes"; nothing is written, because no row, registration or admission exists yet. `mu` is the innermost console lock: it is taken under the app lock where the create holds one, nothing is taken under it, and `fn` is one local SQLite write or one map insert. The drain sets `draining` with `mu` held, so every admission either committed before it, and the drain's reads below see its row or registration, or is refused. |
| +- **The door** (operator only: `s.moveOperator`, a 404 otherwise, and a 404 without `s.cfg.Releases`, as `handleAPILifetimesCutover`, `lifetimes_cutover.go:526-531`). Idempotent; each call is one round. The first call sets `s.admission.draining` for the rest of this process (a restart clears it, which is the way to abandon a rollback). The round waits on the work itself, not on table entries: (1) every row `s.store.PendingAllocations()` answers (pending, or terminal with `host_key = 1`, Task 7a), whether or not the table holds its entry: `recoverAllocationLocked` under the app lock (a create still running holds that lock, so recovery runs after it returns), which reserves, completes or voids, and releases the key; a row still pending afterwards is counted; (2) the ids registered in `s.placing`: counted, untouched (each settles itself, fence before release, then deregisters, Task 7b); (3) every `place` operation in the table whose id is not registered: settled now as the sweep settles it (fence every host key's record, then `moves.Release`), **never released before its fences answered**; (4) a `create` operation in the table with no allocation row: counted (the sweep pages it); (5) the counts: `cleanup` operations in the table, open self-serve moves (`s.store.OpenSelfServeMoves()`, `selfserve_store.go:562`), pending rows and `create` entries, registrations and `place` entries; (6) all zero → the settling write, `moves.Release(ctx, s.cfg.Releases, {Kind: "drain", ID: "drain"})` (it holds nothing and writes rev + 1), then the table, the allocation rows, `s.placing` and the open self-serve moves read again: still zero → `drained`; else `draining`. The answer: `{state: "draining" | "drained", cleanups, creates, placements, self_serve_moves int, rev uint64}`. |
| - **The CLI** `carlos system reservations drain [--console <origin>] [--no-wait]`: `requireDeployAPISession`, then the door every 5 s (`waitLifetimesCutover`'s loop and retry rules: a failed poll or a 5xx is asked again, up to 5 in a row or 60 s; a refusal is final), printing each line that differs from the last: "draining: <n> cleanups, <n> creates, <n> placements and <n> self-serve moves still hold reservations." and, at the end, "drained: no new kind of reservation is held. stop this console now, then start the old one." Copy rows, pending. The command never stops the console itself. |
| -- [ ] **Step 1: Write the failing tests**: `TestDrainSettlesBeforeItReleases` (Review Focus 4: a `place` entry whose record PUT the fake store commits late: the fence runs and makes it fail before the release; the entry goes only after the fence answered; a fence failure keeps it and the answer stays `draining`); `TestDrainRefusesNewWork` (after the first call a create, a placement and a self-serve request answer 503 and write nothing; a restarted console takes them again); `TestDrainWaitsForInFlightWork` (a registered placement, a pending allocation, an open self-serve move and a `cleanup` entry each keep the answer `draining` with its count); `TestDrainEndsWithASettlingWrite` (`drained` only after the console's own write and a re-read; a create `Put` held by the fake store from before the drain is delivered after it and fails its condition); `TestDrainIsOperatorOnly`; `TestSystemReservationsDrainCLI` (the lines, the wait and the retries). |
| -- [ ] **Step 2: Run to verify they fail**: `go test ./internal/console -run 'Drain' -count=1`, `go test . -run 'SystemReservationsDrain|MovesStringsAreRecorded|MovesCopyRecord' -count=1`. |
| +- [ ] **Step 1: Write the failing tests**: `TestDrainSettlesBeforeItReleases` (Review Focus 4: a `place` entry whose record PUT the fake store commits late: the fence runs and makes it fail before the release; the entry goes only after the fence answered; a fence failure keeps it and the answer stays `draining`); `TestDrainRefusesNewWork` (after the first call a create, a placement and a self-serve request answer 503 and write nothing; a restarted console takes them again); `TestDrainWaitsForInFlightWork` (a registered placement, a pending allocation, an open self-serve move and a `cleanup` entry each keep the answer `draining` with its count); `TestDrainCountsWorkWithoutEntries` (review 6's finding 3: a create the fake store pauses after `BeginAllocation` and before its `Reserve`: the round answers `draining` with `creates == 1` while the table holds no `create` entry; resumed, the create reserves, completes and releases, and the next round answers `drained`; a placement registered and paused before its `Reserve`: `placements == 1` until it settles); `TestAdmissionAndTheDrainFlagAreOneCriticalSection` (a create whose `BeginAllocation` the fake store blocks inside `admitResidence`: the drain's first call blocks on `mu` until it returns, then counts the row; a create that starts after the flag gets `errors.Is(err, errConsoleDraining)` and `PendingAllocations` is unchanged; the same for a placement's registration, `len(s.placing)` unchanged, and a self-serve `AdmitSelfServeMove`, `OpenSelfServeMoves` unchanged); `TestDrainEndsWithASettlingWrite` (`drained` only after the console's own write and a re-read; a create `Put` held by the fake store from before the drain is delivered after it and fails its condition); `TestDrainIsOperatorOnly`; `TestSystemReservationsDrainCLI` (the lines, the wait and the retries). |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/console -run 'Drain|AdmissionAndTheDrain' -count=1`, `go test . -run 'SystemReservationsDrain|MovesStringsAreRecorded|MovesCopyRecord' -count=1`. |
| - [ ] **Step 3: Implement. Step 4: Run to verify they pass.** |
| -- [ ] **Step 5: Commit** `console: carlos system reservations drain settles every placement and create reservation and refuses new ones before a rollback`. |
| +- [ ] **Step 5: Commit** `console: carlos system reservations drain closes admission in one critical section, then settles every allocation, placement and create reservation before a rollback`. |
| ### Task 8: The reaper |
| @@ -397,7 +404,7 @@ Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and |
| - Create: `internal/provision/testdata/plan14-root-reads-self-serve-moves-4.tsv`, `internal/provision/self_serve_moves4_census_test.go`, `docs/superpowers/notes/census-proposal-self-serve-moves-4.md` |
| - Modify: `internal/provision/plan14_root_reads_test.go`, `internal/provision/plan9_b8_census_test.go` (line 372) |
| -- [ ] **Step 1: Write the tests**: `TestCleanupTwentyFourHoursAfterTheObservedFinish` (a forward move; at 24 h every inventoried version is gone, each removed by its version id, oldest first, no marker written by the cleanup, `lock.json` and `build-state.json` remain, the lifted tombstone is back, root removed the live link when its chain is root-only, the programs file is reported left, the entries are gone, the table is still there and empty, the report is under the move id and `carlos move offline status --log` shows it); `TestCleanupSupersededByAPlacementBack` (the placement is refused while the cleanup holds; after it ends, a placement back to the source runs); `TestCreateUnderCleanupRefusedThenAllowed`; `TestRowMintedUnderTheMarkSupersedes`; `TestLateAttemptAfterSupersessionChangesNothing` (a delete's first attempt held past the supersession; the second attempt answered under the mark; the release; a new wake writes; the held attempt is delivered: the instance serves its new data); `TestPromotionBackIntoTheRegionKeepsItsData` (a legacy move back into the source region while a superseded cleanup had a delete in flight: the delete is answered under the mark, the wake waits for the unmark, the promotion keeps equal-ETag versions in place and copies the rest, and the instance's next wake restores the full replica); `TestOperatorReturnAfterSupersededCleanupStillFindsItsLeftovers` (the return's start door accepts the forward's leftover set after a superseded cleanup that had marked and unmarked); `TestLegacyMoveIntoTheSourceRegionIsTolerated` (switch off: a legacy move back to the source region while the cleanup holds: the cleanup supersedes at its next key, answers any delete in flight, the wake waits for the unmark, the promotion restores the staged data and the instance serves it); `TestUntrackedLifetimeMoveFinishes` (an incarnation-0 record with no slot moved by an operator: `gone` reports, the move finishes, no obligation exists); `TestOlderCleanupInTheMoveLog` (two moves of one host, the first's cleanup open: `--log --cleanup <first id>` shows it after the second finished and after the record is deleted); `TestRollbackDrain` (the procedure's steps 1 and 2 against a live cleanup, a placement and a create: the drain answers `drained` only after the cleanup ended and both were settled; the old console's parser then reads the table). |
| +- [ ] **Step 1: Write the tests**: `TestCleanupTwentyFourHoursAfterTheObservedFinish` (a forward move; at 24 h every inventoried version is gone, each removed by its version id, oldest first, no marker written by the cleanup, `lock.json` and `build-state.json` remain, the lifted tombstone is back, root removed the live link when its chain is root-only, the programs file is reported left, the entries are gone, the table is still there and empty, the report is under the move id and `carlos move offline status --log` shows it); `TestCleanupSupersededByAPlacementBack` (the placement is refused while the cleanup holds; after it ends, a placement back to the source runs); `TestCreateUnderCleanupRefusedThenAllowed`; `TestRowMintedUnderTheMarkSupersedes`; `TestLateAttemptAfterSupersessionChangesNothing` (a delete's first attempt held past the supersession; the second attempt answered under the mark; the release; a new wake writes; the held attempt is delivered: the instance serves its new data); `TestPromotionBackIntoTheRegionKeepsItsData` (a legacy move back into the source region while a superseded cleanup had a delete in flight: the delete is answered under the mark, the wake waits for the unmark, the promotion keeps equal-ETag versions in place and copies the rest, and the instance's next wake restores the full replica); `TestOperatorReturnAfterSupersededCleanupStillFindsItsLeftovers` (the return's start door accepts the forward's leftover set after a superseded cleanup that had marked and unmarked); `TestLegacyMoveIntoTheSourceRegionIsTolerated` (switch off: a legacy move back to the source region while the cleanup holds: the cleanup supersedes at its next key, answers any delete in flight, the wake waits for the unmark, the promotion restores the staged data and the instance serves it); `TestUntrackedLifetimeMoveFinishes` (an incarnation-0 record with no slot moved by an operator: `gone` reports, the move finishes, no obligation exists); `TestOlderCleanupInTheMoveLog` (two moves of one host, the first's cleanup open: `--log --cleanup <first id>` shows it after the second finished and after the record is deleted); `TestRollbackDrain` (the procedure's steps 1 and 2 against a live cleanup, a placement and a create paused between `BeginAllocation` and its reservation: the drain answers `drained` only after the cleanup ended and both were settled; the old console's parser then reads the table); `TestCleanupHoldsTheMarkUntilItsDeleteIsAnswered` (the cleanup's agent is stopped with a delete in flight for 25 h of the test clock while a legacy move returns the host to the source region: the return's wake waits; the agent restarts, the delete is answered, the mark lifts, and the instance serves the returned data). |
| - [ ] **Step 2: Run them; fix what they find within this plan's scope.** |
| - [ ] **Step 3: Run both census tests and the cross-builds**; write the census proposal with the loader block, the changed envelopes (`internal/host`, `internal/release`, `internal/registry`, `internal/lease`, `internal/moves`, `internal/console`, `internal/offlinecleanup`, the root package) and the units. |
| - [ ] **Step 4: Commit** `self-serve moves: cleanup end to end`; push. |
| @@ -407,10 +414,10 @@ Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and |
| ## Self-review notes |
| - Spec coverage: §7.6 the source replica (T1, T3, T4a, T4b, T4c, T4d), root-owned leftovers with the link and process checks (T6a, T6b, T6c), parked local state (T8), the staged objects of an aborted move (unchanged); §7.1's `cleanup` reservation and the hostname keys (T2a, T2b, T2c, T7a, T7b); §8's cleanup pages and every cleanup key in the move log (T5a, T5b); §9 rollout and rollback (Global Constraints, T7c); §11's cleanup witnesses (T4a, T4c, T9). |
| -- **Stated deviations.** (1) Legacy moves hold no reservation. A legacy move into the source region carries a move on the record from its door onward, so condition (1) supersedes the cleanup at its next key, and its completion's wake cannot take the lease while the mark holds; after the unmark the wake's promotion mirrors the staged replica over the prefix (`promoteStagedReplica`, an exact mirror, called during the restore after acquisition, `backend_exec.go:819`), so no deleted key matters; review 4's writer audit confirms this normal-operation argument (completion changes the record; promotion waits for acquisition). A delete sent before the supersession is answered under the mark, before the unmark, so none is outstanding when the promotion keeps an equal-ETag version in place. Holding a key from the box at `MoveComplete` (`movespass.go:1617`) would add a box-side release and crash path and extend a legacy cut's downtime behind root's answer, for no exclusion the mark does not already give. (2) Placement keeps `SetRecordRegion`'s ETag-conditional write and settles an uncertain one by `fenceRecord`, after which no earlier PUT can land; the reservation entry is the persisted intent, so no journal table is added. (3) A cleanup entry is never reclaimed by another reserver (§7.1 allowed a status-based reclaim); the console pages instead. (4) Only the sealing box's local paths are cleaned; other boxes in the source region keep theirs, as today. (5) The create reserves after `BeginAllocation`, not before, so every reservation has an intent row behind it. (6) `build-state.json` is kept with `lock.json`. (7) **§7.6's "plain `DeleteObject`, `VersionId` deletes are never used" is amended (Paul's question 9):** every inventoried version is deleted by its version id, oldest first, the version current at `gone` last, so no older version is ever exposed and no late request can touch another owner's version. The retired replica is then not recoverable from the bucket. (8) The table is never deleted (plan 2 deleted it when empty). (9) Root removes a leftover only on a root-only chain; the programs file, under the agent-writable `/data` on every built box, is recorded as left at `gone` and stays the runbook's, and so do both paths on a writable-opt box. (10) Root's rename-aside is gone; identity is compared on the opened parent with `fstatat`. (11) §7.6's "a key already carrying a delete marker is done" becomes: a key whose current entry is a marker the inventory does not hold has its inventoried versions beneath that marker deleted (nothing current changes), and the marker is left. (12) Root's step is a stage of the adopt pass, under a new root-only pass lock that every adopt pass takes; held publication's freeze and lock are unchanged. |
| -- **The one residual in root's exclusion.** A pass whose root pass lock fails for a root-level fault (an I/O error on a root-only directory, which no other uid can cause) runs no root step, but it still publishes, as a pass whose adopt-pass lock fails does today, so it is not excluded from another pass's step. Making that failure fatal would remove the residual at the cost of a box that stops converging on that fault; the plan keeps today's never-fatal rule and names the case here for the reviewer. |
| +- **Stated deviations.** (1) Legacy moves hold no reservation. A legacy move into the source region carries a move on the record from its door onward, so condition (1) supersedes the cleanup at its next key, and its completion's wake cannot take the lease while the mark holds; after the unmark the wake's promotion mirrors the staged replica over the prefix (`promoteStagedReplica`, an exact mirror, called during the restore after acquisition, `backend_exec.go:819`), so no deleted key matters; review 4's writer audit confirms this normal-operation argument (completion changes the record; promotion waits for acquisition). A delete sent before the supersession is answered under the mark, before the unmark, so none is outstanding when the promotion keeps an equal-ETag version in place. Holding a key from the box at `MoveComplete` (`movespass.go:1617`) would add a box-side release and crash path and extend a legacy cut's downtime behind root's answer, for no exclusion the mark does not already give. (2) Placement keeps `SetRecordRegion`'s ETag-conditional write and settles an uncertain one by `fenceRecord`, after which no earlier PUT can land; the reservation entry is the persisted intent, so no journal table is added. (3) A cleanup entry is never reclaimed by another reserver (§7.1 allowed a status-based reclaim); the console pages instead. (4) Only the sealing box's local paths are cleaned; other boxes in the source region keep theirs, as today. (5) The create reserves after `BeginAllocation`, not before, so every reservation has an intent row behind it. (6) `build-state.json` is kept with `lock.json`. (7) **§7.6's "plain `DeleteObject`, `VersionId` deletes are never used" is amended (Paul's question 9):** every inventoried version is deleted by its version id, oldest first, the version current at `gone` last and required current before every deletion, so no older version is exposed (short of a version-id delete of that version by hand during the run) and no late request can touch another owner's version. The retired replica is then not recoverable from the bucket. (8) The table is never deleted (plan 2 deleted it when empty). (9) Root removes a leftover only on a root-only chain; the programs file, under the agent-writable `/data` on every built box, is recorded as left at `gone` and stays the runbook's, and so do both paths on a writable-opt box. (10) Root's rename-aside is gone; identity is compared on the opened parent with `fstatat`. (11) §7.6's "a key already carrying a delete marker is done" becomes: a key whose current entry is not its inventoried latest (a later version, or a marker the inventory does not hold) is `superseded` and nothing more of it is deleted, so a foreign marker and everything beneath it are left; a marker the inventory holds as the latest is deleted last, like any latest. (12) Root's step is a stage of the adopt pass, under a new root-only pass lock that every adopt pass takes; held publication's freeze and lock are unchanged. (13) **The cleaning mark has no expiry** (§7.6 has the cleanup hold a lease; here it is a state): a cleanup that cannot get an answer for a delete holds the host's replica marked, and with it every wake, landing and return of the host in this region, until the delete is answered or an operator abandons it after the page (Paul's question 9). (14) **Root's gate:** on a box where cleanup was ever armed (the root-owned `gated` marker, or the switch on now), an adopt pass that cannot take the root pass lock, for any reason but a chain that is not root-only, publishes nothing that pass; a box never armed keeps today's fail-open adoption. (15) The bucket's versioning status is not read (the box roles lack `s3:GetBucketVersioning`); a `null` id in the listing, which is how an unversioned or suspended bucket shows, writes no `retired.json`, and the rollout checks the status by hand. |
| +- **The one residual in root's exclusion.** On a gated box there is none: every pass that publishes holds the root pass lock. On a box that was never gated no root step can run. What is left is the moment a box first becomes gated: a pass that read the switch off and found no marker, whose adopt-pass lock failed (a shared-uid writer's symlink at `/data/adopt-pass.lock`) and whose root pass lock failed for a root-level fault (which no other uid can cause), publishes fail-open; if it is still running when a later pass, after the switch went on and the agent's first cleanup dropped a request, runs root's step, the two overlap. Three independent conditions, one of them a fault only root can cause, at one moment in a box's life. Failing closed on every box, armed or not, would remove even this, at the cost of today's fail-open adoption on boxes that never run a cleanup; the plan follows the ruling and names the case here. |
| - Plan 2 amendments (built code, changed here): move selections hold `host/<hostname>`; the switch-off rule applies between moves only, so a cleanup neither blocks nor is blocked by an operator move on another box; recovery mode counts `move` entries only; the cleanup reclaim interface is deleted; `releaseBoxes` writes an empty table instead of deleting it; every table write carries `epoch` and `rev`. |
| -- Built-code changes this plan makes, by task: T1 `moves/offline.go`, `host/movesoffline.go`, `release/programhistory.go` (export); T2a `moves/offline.go` (kinds, epoch, rev, `valid()`); T2b `console/offlinereserve.go`, `console/selfserve_pass.go`, `console/lifetimes_store.go`; T2c `console/lifetimes_recovery.go`, `console/selfserve_pages.go`; T3 `lease` (new file only); T4b, T4c `host` (new files only); T4d `host/movespass.go`; T5a `host/movespaging.go`, `console/apiinstancemove_offline.go`; T5b `console/authcli.go`, `move.go`, `docs/site/cli.md`, the copy record; T6a `registry`; T6b `host/agent.go`, `host/adoptlock.go` and `host/movesheldlock.go` (comments), `host/claimcensus.go`; T7a `console/store.go`, `lifetimes_store.go`, `lifetimes_create.go`, `lifetimes_recovery.go`; T7b `fleets.go`, `console/store.go` (`claimRawSetAppPlacement`), `registry/registry.go` (`SetPlacementContext`), `offlinereserve.go`, `residence_sweep.go`; T7c `console/authcli.go`, `lifetimes_create.go`, `fleets.go`, `selfserve_door.go`, `system.go`, `docs/site/cli.md`. The mirror (`internal/lease/mirror.go`) is not changed. |
| -- Names used across tasks: `offlinecleanup.Retired` (`State`, `Outcome`, `Keys`, `Events`, `PagesPending`, `LocalLeft`, `Root`, `Rev`, `PublishedRev`, `MarkedAt`, `ReserveUncertain`, `InFlight`, `RootRequestedAt`, `CancelRequestedAt`), `CorruptStub`, `CleanupID`, `InventoryKey`, `InventoryVersion`, `InFlightDelete`, `KeyOutcome` (`State`, `Done`), `Event`, `PendingPage`, `Report`, `FileIdentity`, `LocalPath`, `KeptPath`, `RootCleanupRequest`, `RootCleanupPath`, `RootCleanupAck`, `ValidateRetired`, `ValidateCorruptStub`, `ValidateReport`, `Identify`, `RootOnlyChain`; `moves.HostKey`, `IsHostKey`, `Active`, `ReadActive`, `NextActiveBody`, `Op`, `Conflicts`, `Reserve`, `Holds`, `Release`, `ErrReserved`, `ErrUnreadable`, `ErrUncertain`, `ErrReservationMismatch`, `RecordAway`, `Away*`, `OfflineOpDir`, `RetiredFile`, `CleanupReportKey`, `OfflineKindCreate`, `OfflineKindPlace`; `lease.CleaningOwner`, `MarkCleaning`, `ReassertCleaning`, `UnmarkCleaning`, `ErrNotOurLock`, `VersionedStore.DeleteVersion`; `deleteInventoriedVersion`, `w.cleanupVisit`, `w.supersededBy`, `w.retireKey`, `w.settleInFlight`, `w.kickCleanup`, `cleanupRun`, `cleanupMu`, `cfg.CleanupArmed`, `recordRetired`, `createRetired`, `writeRetired`, `markPublished`, `flushPageOutboxFor`, `reapRetired`; `release.ServeRootCleanupRequests`, `RootCleanupOptions`, `RootCleanupBase`, `RootCleanupDir`, `ReadRootCleanupAck`, `LockRootPass`, `ErrRootCleanupRefused`, `ErrNotRootOnly`, `ProgramsPath`; `rootCleanupAuthority`, `rootCleanupPass`, `rootRefusedAfter`; `registry.WithHostFrozen`, `SetPlacementContext`; `s.offlineCleanupsOf`, `s.reservePlacement`, `s.registerPlacement`, `s.deregisterPlacement`, `s.settlePlacement`, `s.placing`, `s.residenceSweep`, `s.releaseAllocationKey`, `s.draining`, `errHostUnderCleanup`; the CLI's `moveArgs.log`, `logMove`, `logCleanup`, and `carlos system reservations drain`. |
| -- Review Focus 1 to 5 are pinned by T4b+T4c, T4a, T6b+T6c, T2a-c+T7a-c, T5a+T5b+T8. |
| -- **Open questions for Paul** (amadan discussion 363): (1) legacy moves without a hostname reservation: accept the exemption, or reserve around `MoveComplete` on the box (a box-side release and crash path, and longer legacy-cut downtime while root answers)? (2) Cleanup runs 24 hours after **every** offline move, operator moves included, replacing the runbook's manual cleanup; intended (§10 Q4 talks about self-serve)? (3) Is the box switch `CARLOS_OFFLINE_CLEANUP=on` acceptable as the rollout gate? (4) A move of the same host waits while its cleanup holds `host/<hostname>`, which can last until root answers (paged every 10 min, hourly once root's ancestry is refused); acceptable? (5) The copy rows: five door sentences, seven page subjects and one page body (now also the CLI's lines, the drain's lines and the `--log` paragraph). (6) On every built box the programs file sits under the agent-writable `/data`, so root never removes it and it stays the runbook's; acceptable, or should the agent remove it itself? (7) Withdrawn by rev 6: the 15-minute `QuietUntil` and the marker-removing repair are gone; question 9 replaces it. (8) Rollback needs no self-serve move in flight; the drain now refuses to finish until none is. Acceptable? (9) **The §7.6 amendment:** the cleanup deletes every inventoried version of the retired replica by its version id (oldest first, the one current at `gone` last) instead of writing delete markers, so the source copy of a finished move is not recoverable from the bucket after the cleanup, in exchange for a boundary the server enforces (a late or retried request can remove only the version it names). Accept, or keep markers (then only `deleteInventoriedVersion` changes, and the settlement rules need a fresh review)? |
| +- Built-code changes this plan makes, by task: T1 `moves/offline.go`, `host/movesoffline.go`, `release/programhistory.go` (export); T2a `moves/offline.go` (kinds, epoch, rev, `valid()`); T2b `console/offlinereserve.go`, `console/selfserve_pass.go`, `console/lifetimes_store.go`; T2c `console/lifetimes_recovery.go`, `console/selfserve_pages.go`; T3 `lease/cleaning.go` (new), `lease/lease.go`, `lease/tombstone.go`, `lease/lockstate.go` (the refusals), `host/movesoffline_land.go` (the landing's wait); T4b, T4c `host` (new files only); T4d `host/movespass.go`; T5a `host/movespaging.go`, `console/apiinstancemove_offline.go`; T5b `console/authcli.go`, `move.go`, `docs/site/cli.md`, the copy record; T6a `registry`; T6b `host/agent.go` (the lock and the gate), `host/adoptrootgate.go` (new), `host/adoptlock.go` and `host/movesheldlock.go` (comments), `host/claimcensus.go`; T7a `console/store.go`, `lifetimes_store.go`, `lifetimes_create.go`, `lifetimes_recovery.go`; T7b `fleets.go`, `console/store.go` (`claimRawSetAppPlacement`), `console/view.go` (`consoleRegistry`), `registry/registry.go` (`SetPlacementContext`), `registry/remote_dispatch.go` (the deadline), `offlinereserve.go`, `residence_sweep.go`; T7c `console/authcli.go`, `lifetimes_create.go`, `fleets.go`, `selfserve_door.go`, `system.go`, `docs/site/cli.md`. The mirror (`internal/lease/mirror.go`) is not changed. |
| +- Names used across tasks: `offlinecleanup.Retired` (`State`, `Outcome`, `Keys`, `Events`, `PagesPending`, `LocalLeft`, `Root`, `Rev`, `PublishedRev`, `MarkedAt`, `ReserveUncertain`, `InFlight`, `Superseding`, `RootRequestedAt`, `CancelRequestedAt`), `CorruptStub`, `CleanupID`, `InventoryKey`, `InventoryVersion`, `InFlightDelete`, `KeyOutcome` (`State`, `Done`), `Event`, `PendingPage`, `Report`, `FileIdentity`, `LocalPath`, `KeptPath`, `RootCleanupRequest`, `RootCleanupPath`, `RootCleanupAck`, `ValidateRetired`, `ValidateCorruptStub`, `ValidateReport`, `UsableVersionID`, `Identify`, `RootOnlyChain`, `InventoryKey.Latest`, `InventoryVersion.Seq`, `Report.InFlight`, `Report.AbandonPath`; `moves.HostKey`, `IsHostKey`, `Active`, `ReadActive`, `ReadForAcquire`, `NextActiveBody`, `Op`, `Conflicts`, `Reserve`, `Holds`, `Release`, `ErrReserved`, `ErrUnreadable`, `ErrUncertain`, `ErrReservationMismatch`, `RecordAway`, `Away*`, `OfflineOpDir`, `RetiredFile`, `CleanupReportKey`, `OfflineKindCreate`, `OfflineKindPlace`; `lease.CleaningOwner`, `CleaningExpiry`, `IsCleaningOwner`, `MarkCleaning`, `UnmarkCleaning`, `ErrNotOurLock`, `VersionedStore.DeleteVersion`; `deleteInventoriedVersion`, `w.cleanupVisit`, `w.supersededBy`, `w.retireKey`, `w.settleInFlight`, `w.kickCleanup`, `cleanupRun`, `cleanupMu`, `cfg.CleanupArmed`, `recordRetired`, `createRetired`, `writeRetired`, `markPublished`, `flushPageOutboxFor`, `reapRetired`; `release.ServeRootCleanupRequests`, `RootCleanupOptions`, `RootCleanupBase`, `RootCleanupDir`, `ReadRootCleanupAck`, `LockRootPass`, `RootCleanupGated`, `ErrRootCleanupRefused`, `ErrNotRootOnly`, `ProgramsPath`; `rootCleanupAuthority`, `rootCleanupPass`, `rootRefusedAfter`, `adoptRootGate`, the files `root-cleanup/gated` and `cleanup.abandon`; `registry.WithHostFrozen`, `SetPlacementContext`; `consoleRegistry.SetPlacementContext`; `s.offlineCleanupsOf`, `s.reservePlacement`, `s.registerPlacement`, `s.deregisterPlacement`, `s.settlePlacement`, `s.placing`, `s.residenceSweep`, `s.releaseAllocationKey`, `s.admission`, `s.admitResidence`, `errConsoleDraining`, `errHostUnderCleanup`; the CLI's `moveArgs.log`, `logMove`, `logCleanup`, and `carlos system reservations drain`. |
| +- Review Focus 1 to 5 are pinned by T3+T4b+T4c, T4a, T6b+T6c, T2a-c+T7a-c, T5a+T5b+T8. |
| +- **Open questions for Paul** (amadan discussion 363): (1) legacy moves without a hostname reservation: accept the exemption, or reserve around `MoveComplete` on the box (a box-side release and crash path, and longer legacy-cut downtime while root answers)? (2) Cleanup runs 24 hours after **every** offline move, operator moves included, replacing the runbook's manual cleanup; intended (§10 Q4 talks about self-serve)? (3) Is the box switch `CARLOS_OFFLINE_CLEANUP=on` acceptable as the rollout gate? (4) A move of the same host waits while its cleanup holds `host/<hostname>`, which can last until root answers (paged every 10 min, hourly once root's ancestry is refused) or until an unanswered delete is answered or abandoned (paged every 10 min); acceptable? (5) The copy rows: five door sentences, eight page subjects and one page body (now also the CLI's lines, among them the waiting-delete line, the drain's lines and the `--log` paragraph). (6) On every built box the programs file sits under the agent-writable `/data`, so root never removes it and it stays the runbook's; acceptable, or should the agent remove it itself? (7) Withdrawn by rev 6: the 15-minute `QuietUntil` and the marker-removing repair are gone; question 9 replaces it. (8) Rollback needs no self-serve move in flight; the drain now refuses to finish until none is. Acceptable? (9) **The §7.6 amendment:** the cleanup deletes every inventoried version of the retired replica by its version id (oldest first, the one current at `gone` last) instead of writing delete markers, so the source copy of a finished move is not recoverable from the bucket after the cleanup, in exchange for a boundary the server enforces (a late or retried request can remove only the version it names). **Rev 7 adds a consequence:** the mark that keeps every later owner off the replica while a delete may still land has no expiry, so a cleanup that cannot settle a delete (the bucket keeps refusing it, or the box is gone) holds the host's replica marked, and with it every wake, landing and return of that host in this region, until an operator acts on the page: by repairing what refuses the delete, or by abandoning it on the box (`cleanup.abandon`). Accept, or keep markers (then only `deleteInventoriedVersion` changes, and the settlement rules need a fresh review)? |