amadan
plan 4 rev 6: deletes by version id from a full-version inventory; root's step inside the adopt pass under a root-only pass lock; rollback as a drain procedure; log lookup by id; placement registers before acquiring
Answers Astra's review of rev 5 (R1-R8). The ETag-conditional delete, the quiet period and the repair pass are gone: a version id is unique, so a late delete can only remove a version that was ours; each delete is resent until answered while the mark and the reservations are held, else the key is unsettled and paged (a promotion can adopt an equal-ETag object in place). Amends design §7.6 (Paul's question 9).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1 file changed,
+187
−117
docs/superpowers/plans/2026-10-22-self-serve-moves-4-cleanup.md+187 −117
CI failed — run details
CI log
=== gate === sigwho: pid 2 ppid 1 pgid 0 sid 0; child pid 3 .amadan/test-gate.sh gate: 2026-10-24T09:21:34Z on aarch64, 32 cpus, 61320 MB available, go1.26.8 linux/arm64 gate: +0s start launchctx gate: +0s start vet gate: +0s start gofmt gate: +1s start rest gate: +13s start internal_console@0 gate: +13s start internal_console@1 gate: +13s start internal_console@2 gate: +13s start internal_console@3 gate: +13s start internal_console@4 gate: +13s start internal_console@5 gate: +13s start internal_console@6 gate: +13s start internal_console@7 gate: +13s waiting for internal_console's shards before the next split package gate: +175s start root@0 gate: +175s start root@1 gate: +175s start root@2 gate: +175s waiting for root's shards before the next split package amadan: step gate failed: signal: terminated amadan: job failed at step gate — its output is above, not at the tail of this log
diff --git a/docs/superpowers/plans/2026-10-22-self-serve-moves-4-cleanup.md b/docs/superpowers/plans/2026-10-22-self-serve-moves-4-cleanup.md| index b8b107232..0363c1e05 100644 |
| --- a/docs/superpowers/plans/2026-10-22-self-serve-moves-4-cleanup.md |
| +++ b/docs/superpowers/plans/2026-10-22-self-serve-moves-4-cleanup.md |
| @@ -2,95 +2,130 @@ |
| > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| -Rev 5, 2026-10-24. Rev 4's review (`after236/selfserve-plan4-review4.md`: F3, F4, F8 to F11 and N1 to N3 closed; F1, F2, F5 to F7, F12, F13 open; N4 to N11 new) is answered by making each destructive step conditional at the server, or by removing it. **N4, F1:** every bucket deletion is `DeleteIfMatch(bucket, key, etag)` on the ETag of the version the inventory recorded: an S3 `If-Match` on the `DeleteObjectInput`, which the mirror does not send today (`internal/lease/mirror.go:787-800` deletes unconditionally; `release.S3Store.DeleteIfMatch`, `internal/release/conditional_delete.go:86-97`, does, for the deployment bucket only). A late or outstanding delete can then only put a marker over the inventoried version, never over a newer one, so the mark is lifted and the reservation released without settling outstanding requests. The one case the condition cannot tell apart, a later version with byte-identical content (equal ETag), is bounded by a 15-minute quiet after any run whose deletes were not all answered, and then repaired: after the quiet, and once more on the obligation's final pass, the box re-lists every key such a run deleted and removes, by its version id, a delete marker that a late cleanup delete put over a later owner's byte-identical version, so the hazard heals instead of merely being bounded. **F2:** `active.json` is never deleted. Every write carries a rising `rev` and the table's `epoch`, so no body repeats and no stale conditional PUT can match a later ETag. Every reservation write is `If-Match`, only the empty bootstrap is create-only, and a release always writes, so its success settles any earlier PUT of the same operation. **F5, F6:** the rename-aside protocol is removed. Root opens the parent, compares the recorded identity by `fstatat`, unlinks by `unlinkat` if it is equal and keeps the entry otherwise, under the registry host freeze and the held-stage lock, journalling intent and then result in a root-owned directory that root also walks on restart. A path whose parent chain a non-root uid can write is never removed by root ("writable layout"). The agent's unit writes `/data` (`internal/host/agent.go:1285`, `:4335`), so the programs file is recorded at `gone` as left in place, not requested. **F7:** supersession is checked at every lifecycle step, and root's `kept` and `cancelled` answers map to `superseded` and `cancelled` in one outcome table. **N5:** an untracked lifetime (incarnation 0 with no slot) gets no `retired.json`, and its gone report proceeds as before this plan. **N6, N9, N10, F3's timestamp, F12, N8, F13:** a corrupt stub has its own valid shape, and the first damaged file is kept as `retired.json.corrupt-1`; root reads the hibernate settings with the activator's own defaults; a placement writes one entry per host key; `RootRequestedAt` is restored from the drop's mtime or root's `taken` line; a one-shot page is persisted in `pages_pending` in the same write that makes its key terminal. **N7:** `carlos move offline status --host H --log` shows the cleanup reports and their events, independent of a self-serve row. **N11:** rollback stops the new console and removes any `create` or `place` entry left in the table by hand before the old console starts. Tasks 4, 6 and 7 are split, the CLI has its own task, and the end-to-end task is last. |
| +Rev 6, 2026-10-24. Rev 5's review (`after236/selfserve-plan4-review5.md`: R1 to R8 and the acknowledgement-location contradiction; F2 to F4, F6 to F13, N1 to N3 and N5 to N10 closed; F1, F5 and N4 open; N11 partial) is answered by removing the machinery that kept growing a settlement protocol and by putting root's step where no other root publisher can run. **R1 to R4, F1, N4:** the ETag-conditional delete, `QuietUntil` and the repair pass are gone. `retired.json` records **every** version id under the normalised prefix at `gone`, delete markers included, and the cleanup deletes each one by its version id, oldest first within a key, through one function (`deleteInventoriedVersion`, Task 4b). A late, retried or outstanding delete can then only remove the version it names, which is ours, never a version a later owner wrote. This is a spec amendment (design §7.6 says markers and never `VersionId` deletes) and is **Paul's question 9**; the plan stays buildable either way because only Task 4b's function would change. One rule is added beyond the ruling, because of a built-code fact: a promotion keeps a destination object whose ETag equals the staged one instead of rewriting it (`lease.MirrorPrefixKeepObserved`, `internal/lease/mirror.go:285`), so a later owner can adopt one of our versions in place. A delete that went out without an answer is therefore sent again, and answered, before the mark or the reservation is given up, and only while both are still this cleanup's (Task 4c). **R5, F5:** root's step is a stage of `runAdoptUnderRootRelease`, after release adoption (named recovery runs inside it) and after the held stage, and it runs only while the pass holds a new root-only pass lock that every adopt pass takes in every mode (Task 6b). The registry host freeze and the held-stage lock stay as rev 5 had them. **R6, N11:** rollback is a procedure with a drain command, `carlos system reservations drain`, run on the new console before it stops (Global Constraint "Arming and rollout", Task 7c). **R7:** `--log` takes `--move <id>` or `--cleanup <id>`, and the console lists a host's cleanup reports across moves from the bucket, so a deleted record still answers (Task 5b). **R8:** placement registers before it reserves and stays registered through settlement; one deadline covers the whole operation, and the two local writes take it (Task 7b). **The acknowledgement location:** when root rejects a directory's ancestry it writes nothing there; it logs the refusal, and the agent pages hourly after an hour without an answer and never releases (Task 6b). Task 2 is split into 2a to 2c, the repair task is gone, and the delete primitive has its own task with the dev gate. |
| -Rev 4's closed answers stand: one state enum and one transition table (F3); a cleanup entry is never reclaimed by another reserver (F4); the programs file has no reference check and the live link keeps §7.6's link and process checks, with unreadable evidence retried (F8); root authorises through a host-level callback, so `internal/release` imports only the leaf `offlinecleanup` (F9); the reaper removes only directories whose obligation has ended (F10); the cleanup runs under its own `cleanupMu`, outside `w.pass` (F11); the prefix is normalised once (N1); `releasing` is persisted before the unmark and the release, and `cleaning` never re-acquires (N2); placement keeps `SetRecordRegion`'s conditional write, and `fenceRecord` settles an uncertain one (N3). Reviews: rev 1 (`after236/selfserve-plan4-review.md`), rev 2 (`selfserve-plan4-review2.md`), rev 3 (`selfserve-plan4-review3.md`), rev 4 (`selfserve-plan4-review4.md`). |
| +Rev 5's answers that stand: the reservation table is never deleted and every write carries a rising `rev` and the table's `epoch` (F2); no rename-aside, and root removes by `fstatat` and `unlinkat` on the opened parent, only on a root-only chain, journalling intent and result in its own directory, which it also walks on restart (F5, F6); supersession is checked at every step, with one outcome table (F7); an untracked lifetime gets no `retired.json` (N5); the corrupt stub, the activator defaults, one placement entry per host key, the restored `RootRequestedAt` and the persisted one-shot page (N6, N9, N10, F3, F12, N8, F13); the CLI's `--log` (N7, extended by R7). |
| -Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and allocation recovery (`internal/console/lifetimes_create.go`, `lifetimes_recovery.go`, `lifetimes_store.go`), `fenceRecord` (`lifetimes_create.go:551`), the judge (`provision.LifetimeJudge.Observe`), the reservation table (`moves.OfflineActive`, `ParseOfflineActive`, `OfflineActiveBody`, `OfflineSentinel`; the console's `reserveBoxes`/`releaseBoxes` and three-way reclaim in `internal/console/offlinereserve.go`), the lease's tombstone (`internal/lease/tombstone.go`), the offline source steps (`internal/host/movesoffline.go`), held publication's freeze and lock (`registry.WithHoldsFrozen`, `internal/host/movesheldlock.go`, `heldStageAuthority` in `internal/host/heldstage.go`), the page outbox (`internal/host/movespaging.go`) and the offline CLI (`move.go`). |
| +Rev 4's closed answers stand: one state enum and one transition table (F3); a cleanup entry is never reclaimed by another reserver (F4); the programs file has no reference check and the live link keeps §7.6's link and process checks, with unreadable evidence retried (F8); root authorises through a host-level callback, so `internal/release` imports only the leaf `offlinecleanup` (F9); the reaper removes only directories whose obligation has ended (F10); the cleanup runs under its own `cleanupMu`, outside `w.pass` (F11); the prefix is normalised once (N1); `releasing` is persisted before the unmark and the release, and `cleaning` never re-acquires (N2); placement keeps `SetRecordRegion`'s conditional write, and `fenceRecord` settles an uncertain one (N3). Reviews: rev 1 (`after236/selfserve-plan4-review.md`), rev 2 (`selfserve-plan4-review2.md`), rev 3 (`selfserve-plan4-review3.md`), rev 4 (`selfserve-plan4-review4.md`), rev 5 (`selfserve-plan4-review5.md`). |
| -**Goal:** Twenty-four hours after a move the source box observed as finished, the source box retires the old region's replica objects from evidence it recorded at `gone`, under a `cleanup` reservation on its own box and on the hostname and a cleaning mark on the replica's lock, key by key with the four conditions re-read at each deletion, and each deletion conditional at the server on the inventoried version's ETag. Root removes the old live link only through a request it copies, authorises from the bucket and answers in its own directory, and only where no other uid can write the link's parent. The obligation's directory goes seven days after it ended. Nothing times out into a release. |
| +Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and allocation recovery (`internal/console/lifetimes_create.go`, `lifetimes_recovery.go`, `lifetimes_store.go`), `fenceRecord` (`lifetimes_create.go:551`), the judge (`provision.LifetimeJudge.Observe`), the reservation table (`moves.OfflineActive`, `ParseOfflineActive`, `OfflineActiveBody`, `OfflineSentinel`; the console's `reserveBoxes`/`releaseBoxes` and three-way reclaim in `internal/console/offlinereserve.go`), the lease's tombstone (`internal/lease/tombstone.go`), the versioned store (`lease.VersionedStore`, `internal/lease/versioned.go`), the offline source steps (`internal/host/movesoffline.go`), held publication's freeze and lock (`registry.WithHoldsFrozen`, `internal/host/movesheldlock.go`, `heldStageAuthority` in `internal/host/heldstage.go`), the page outbox (`internal/host/movespaging.go`), the offline CLI (`move.go`) and the operator's `carlos system` verbs (`system.go`, `system_lifetimes.go`). |
| -**Architecture:** At `gone` the sealing source box writes `retired.json` under the move's op directory, for a tracked lifetime only: the exact current versions (key, version id, ETag) under the normalised hibernate prefix (`lock.json` and `build-state.json` aside), and the identities of the root-owned paths whose parent chain is root's alone (any other path is recorded as left). A cleanup goroutine, started by the moves pass and never under its lock, walks every obligation through one transition table: observe the finish, wait 24 hours, reserve the box's key and `host/<hostname>`, mark `lock.json`, delete key by key with the conditions re-read and the delete conditional on the ETag, hand root a request (or skip it when nothing local remains), then persist `releasing`, restore the lifted tombstone, release and end. Supersession is checked at every step and ends the obligation through the same `releasing`, after any root request is cancelled and acknowledged. The lifetime create and placement hold `host/<hostname>` in the same table, and so does a move's selection. The table is never deleted. |
| +**Goal:** Twenty-four hours after a move the source box observed as finished, the source box retires the old region's replica from evidence it recorded at `gone`, under a `cleanup` reservation on its own box and on the hostname and a cleaning mark on the replica's lock: key by key with the conditions re-read at each key, each inventoried version deleted by its version id, oldest first, and every delete answered before the mark or the reservation is given up. Root removes the old live link only inside its adoption pass, under a lock no other root publisher can bypass, through a request it copies, authorises from the bucket and answers in its own directory, and only where no other uid can write the link's parent. The obligation's directory goes seven days after it ended. Nothing times out into a release. A rollback to the console before this plan drains the new reservation kinds first. |
| -**Tech Stack:** Go. `internal/offlinecleanup` (new leaf: the wire types, file identity and the layout check), `internal/moves` (host keys, the table's read and write, the conflict relation, `Reserve`/`Release`, `RecordAway`), `internal/lease` (the cleaning mark, the mirror's conditional delete), `internal/host` (the dispatcher, the report, the reaper, root's authority and the agent's root half), `internal/release` (root's request handler), `internal/registry` (`WithHostFrozen`), `internal/console` (the create, placement, the reservation wrappers, recovery mode, the sweeps, the status door's cleanup log), the root package (`move.go`, the CLI's `--log`). |
| +**Architecture:** At `gone` the sealing source box writes `retired.json` under the move's op directory, for a tracked lifetime only: every version and delete marker (key, version id, ETag) under the normalised hibernate prefix (`lock.json` and `build-state.json` aside), each key's versions in deletion order, and the identities of the root-owned paths whose parent chain is root's alone (any other path is recorded as left). A cleanup goroutine, started by the moves pass and never under its lock, walks every obligation through one transition table: observe the finish, wait 24 hours, reserve the box's key and `host/<hostname>`, mark `lock.json`, delete key by key with the conditions re-read, hand root a request (or skip it when nothing local remains), then persist `releasing`, restore the lifted tombstone, release and end. Supersession is checked at every step and ends the obligation through the same `releasing`, after any unanswered delete is answered under the mark and any root request is cancelled and acknowledged. The lifetime create and placement hold `host/<hostname>` in the same table, and so does a move's selection. The table is never deleted. |
| -**Spec:** `docs/superpowers/specs/2026-10-21-self-serve-offline-moves-design.md` §7.6, §7.1 (the `cleanup` reservation), §8 (the cleanup pages), §9 (rollout), §11. Stated deviations: placement and the lifetime create hold the hostname key (Paul, 2026-10-22, amadan #343); the rest are in the Self-review notes. |
| +**Tech Stack:** Go. `internal/offlinecleanup` (new leaf: the wire types, file identity and the layout check), `internal/moves` (host keys, the table's read and write, the conflict relation, `Reserve`/`Release`, `RecordAway`), `internal/lease` (the cleaning mark), `internal/host` (the dispatcher, the delete primitive, the report, the reaper, root's authority and the agent's root half, the root pass lock's wiring), `internal/release` (root's request handler, the root pass lock), `internal/registry` (`WithHostFrozen`, `SetPlacementContext`), `internal/console` (the create, placement, the reservation wrappers, recovery mode, the sweeps, the cleanup door, the drain), the root package (`move.go`, the CLI's `--log`; `system.go` and `system_reservations.go`, the drain verb). |
| + |
| +**Spec:** `docs/superpowers/specs/2026-10-21-self-serve-offline-moves-design.md` §7.6, §7.1 (the `cleanup` reservation), §8 (the cleanup pages), §9 (rollout), §11. Stated deviations: placement and the lifetime create hold the hostname key (Paul, 2026-10-22, amadan #343); deletion by version id (§7.6 amendment, Paul's question 9); the rest are in the Self-review notes. |
| ## Global Constraints |
| -- **Arming and rollout.** The box acquires a new cleanup reservation only with `CARLOS_OFFLINE_CLEANUP=on` in its environment, on an `OfflineCapable` box whose judge is set (`cfg.Judge != nil`, i.e. host claims off, `internal/host/boxjudge.go:21-26`); `retired.json` is written at `gone` whatever the switch says, and an obligation past `finished` runs to its end if the switch is turned off. Order: boxes, then consoles (plan 4 code everywhere), then `CARLOS_OFFLINE_CLEANUP=on` on the boxes. Before the switch goes on in any deployment, a check on dev confirms that a `DeleteObject` with a stale `If-Match` on the versioned hibernate bucket answers 412 and writes no delete marker (Task 4b's dev witness). **Rollback:** unset the box switch; stop the new console (only consoles write `create` and `place` entries, so none can be added after it stops; `cleanup` entries are accepted by the console before this plan, whose `valid()` takes `move` and `cleanup`, `moves/offline.go:584-587`, and `create` and `place` are the kinds it reads as a malformed table); read `control/offline-moves/active.json`; any `create` or `place` entry left is a stopped console's and is removed by hand before the old console starts: `aws s3api get-object --profile <profile> --bucket <deployment bucket> --key control/offline-moves/active.json active.json` (note the ETag), delete the entries from `reservations`, add 1 to `rev`, keep `epoch`, then `aws s3api put-object --profile <profile> --bucket <deployment bucket> --key control/offline-moves/active.json --body active.json --if-match <ETag>` (never `delete-object`); for a removed `create` entry the old console's recovery continues its allocation row as in plan 3; for a removed `place` entry the app's records are read and the placement is run again on the old console when they disagree. A self-serve move's `move` entries now include its host key, which the old console's own retry reads as a mismatch (`offlinereserve.go:177-183`), so roll back only when no self-serve move holds an entry. Only then start the old console. |
| +- **Arming and rollout.** The box acquires a new cleanup reservation only with `CARLOS_OFFLINE_CLEANUP=on` in its environment (read by `armOffline` when the agent starts), on an `OfflineCapable` box whose judge is set (`cfg.Judge != nil`, i.e. host claims off, `internal/host/boxjudge.go:21-26`); `retired.json` is written at `gone` whatever the switch says, and an obligation past `finished` runs to its end if the switch is turned off. Order: boxes, then consoles (plan 4 code everywhere), then `CARLOS_OFFLINE_CLEANUP=on` on the boxes. Before the switch goes on in any deployment, a check on dev confirms version-id deletion on the versioned hibernate bucket (Task 4b's dev witness). |
| +- **Rollback**, in this order and no other: |
| + 1. Unset `CARLOS_OFFLINE_CLEANUP` on every box and restart its agent, so no box acquires again. Wait until no `cleanup` entry is left in `control/offline-moves/active.json`: an obligation already reserved runs to its end (completion or supersession); one waiting for root ends when root answers, or when an operator cancels it by creating `<dataDir>/.offline-moves/<host>/<move>/root-cleanup.cancel` (root then acknowledges `cancelled`, Task 6b); a `finished` obligation whose reservation write was uncertain releases it while unarmed (Task 4a, row 2). |
| + 2. Run `carlos system reservations drain` against the **new** console, before it stops (Task 7c). From its first call the console refuses new creates, placements and self-serve move requests for the rest of its process. It settles every `place` entry by the per-host fence (`fenceRecord`, through `s.settlePlacement`) **before** it releases the entry, runs every `create` entry's allocation recovery to its release, and answers `drained` only when, after a settling write of its own, no `cleanup`, `create` or `place` entry and no open self-serve move is left. It refuses to finish while a self-serve move is in flight: a self-serve move's `move` entries hold its host key, which the old console's own retry reads as a mismatch (`offlinereserve.go:177-183`; an operator move's own retry tolerates it). |
| + 3. Stop the new console. |
| + 4. Only then start the old console. |
| + Why this is safe: the old console reads the table that is left (only `move` entries; its `valid()` takes `move` and `cleanup`, `moves/offline.go:584-587`, and reads `create` or `place` as a malformed table). The old console's writes drop `epoch` and `rev`, and its last release deletes the table (`offlinereserve.go:478`), so an earlier ETag can come back: **the epoch protects only among new writers.** The hazard is a new-protocol acquisition still outstanding when such an ETag returns, and none can be: boxes no longer acquire (step 1), the new console is stopped (step 3) after its drain's settling write (step 2), and a conditional PUT that read an ETag before that write can never land after it. A new-protocol release that lands late writes the table it read with nothing added, so it is harmless. The old create and placement take no hostname key, which is safe once no cleanup holds or can acquire one (step 1). |
| - **Tracked lifetimes only.** A cleanup obligation exists only for a tracked lifetime: the judged observation at `gone` is `LifetimeValid` with `Rec.Incarnation > 0` and a live slot at that incarnation. A record with incarnation 0 and no slot (an untracked lifetime, before the lazy migration; `provision.LifetimeOf`, `internal/provision/lifetime.go:127-130`, still answers it `LifetimeValid`), a migrating record, or any other verdict gets no `retired.json`, and its gone report proceeds exactly as before this plan; nothing is ever deleted for it, and its leftovers stay the runbook's. |
| -- **Every data deletion is conditional at the server.** A key is deleted only by `lease.ConditionalMirrorDeleter.DeleteIfMatch(ctx, bucket, key, etag)` on `cfg.PlainMirror` (`lease.NewS3Mirror(ctx, region, region)` for the source bucket's region), with the inventory's ETag for that key: S3 writes a delete marker only while the key's current version carries that ETag and answers 412 otherwise. `VersionId` deletes are never used on a data version (the one exception is the repair's removal of a delete marker, below), a plain `Delete` is never a fallback, and `lock.json` and `build-state.json` are never inventoried and never deleted. **The argument, once:** a delete issued under the conditions and still outstanding after them (a client timeout, a crash, a supersession, the unmark, the release) can land only on the version the inventory recorded; any later writer's version of that key carries another ETag unless its bytes are identical, so the late delete fails and the newer version stays current. That is why the mark may be lifted and the reservation released without settling outstanding requests. For the byte-identical case (a move back into the region, or a legacy promotion, copying the same LTX files back under the same keys), the bound is `QuietUntil`: set before the first delete of each run to the run's end plus `cleanupUncertainQuiet` (15 min), cleared at the end of a run in which every delete was answered, and `releasing` waits for it with the mark and the reservation held. It is a bound on how long the service can hold a request, not a proof, so a **repair** follows it: the keys a run with an unanswered delete issued deletes for are persisted (`RepairKeys`), and once the obligation has ended (so `QuietUntil` has passed and any later owner may have written) and again on its final pass before the reaper, each such key's versions are listed again. When the current version is a delete marker newer than `RepairSince` (the start of the first such run), the version directly beneath it is not the inventoried one, is newer than it and carries the inventoried ETag (the only version a late conditional delete of this cleanup could have landed on), the box removes **that marker** by its version id (`VersionedStore.DeleteVersion`, `internal/lease/versioned.go:261-275`) and records the key `restored`; when the newest version that is not a marker is the inventoried one, or any other shape is current, the key is `done` and nothing is touched. This marker removal is the one `VersionId` delete the plan allows, and it is safe: deleting a delete marker changes the current version only while that marker is current, and then the version directly beneath it becomes current; versions are only ever added on top, so that version is the later owner's newest, never one older than what was current before the late delete landed; a marker that is no longer current when the delete arrives (the owner wrote or deleted again after the listing) is removed without changing anything. The residue: an owner that itself deleted a byte-identical copy of an inventoried file after `RepairSince` gets that copy back once (litestream's retention removes it again on its next enforcement). The cost is one `ListVersions` (`ListObjectVersions` with the key as prefix) per repair key per repair pass, two passes, and only for obligations that had a run with an unanswered delete. |
| -- **The table is never deleted.** `control/offline-moves/active.json` is created once, by the first writer, as `{v: 1, epoch: <16 random bytes, hex>, rev: 1}` with `If-None-Match: *` (`release.Store.Put` with `ifMatch ""`, `internal/release/store.go:794-798`); emptiness is a table with no entries. Every later write is a `Put` with `If-Match` the ETag it read, of a body whose `rev` is the read body's plus one and whose `epoch` is the read body's (a fresh one when the read body has none, as an older console's writes do), so no two bodies the table ever holds are equal and an ETag never comes back. A conditional `Put` that read ETag E can land only while E is current, so once any later write succeeds, every earlier outstanding `Put` that read E can never land: an uncertain acquisition is settled by the next successful conditional write by anyone. A release always writes (rev + 1), even when its operation holds nothing, so its own success is the settlement its caller needs. |
| +- **Every data deletion names one inventoried version (spec §7.6 amendment, Paul's question 9).** `retired.json`'s `BucketInventory` records, for every key under the normalised prefix except `lock.json` and `build-state.json`, **every** version id the bucket lists at `gone`, delete markers included, not only the current one. The cleanup deletes only through `deleteInventoriedVersion` (Task 4b), which is `VersionedStore.DeleteVersion(ctx, bucket, key, versionID)` (`internal/lease/versioned.go:261-275`: an empty id sends nothing; success and `NoSuchVersion` are both nil) on `cfg.Versioned(ctx, Bucket, BucketRegion)`. A plain `Delete`, a marker-writing delete and an `If-Match` delete are never used. Within a key, the versions are deleted oldest first, and the version that was current at `gone` last. A version is **done** when a delete of it answers (success or no such version); a version the listing no longer shows is done (someone with authority removed it). **The argument, once:** a delete that names a version id can remove only that version, whoever sends it and whenever it lands, so a late attempt, an SDK retry (`internal/lease/endpoint.go:103` keeps the default retries) or a crash-repeated request can never remove a version a later owner wrote, which carries another id. Deleting a version that is not current changes no current version. Versions are only ever added on top, and the inventory holds every version that existed at `gone`, so by the time the version that was current at `gone` is deleted every older one is gone: no version older than the inventoried current one is ever exposed. That is the risk §7.6 avoids by writing markers, and this order removes it. **The one case a version id does not settle:** a later owner that keeps one of our versions instead of writing its own. A promotion does exactly this when the destination's ETag equals the staged one (`lease.MirrorPrefixKeepObserved`, `mirror.go:285`), and the target landing of a move into this region mirrors the same way. Both need the lease or the hostname key, which the mark and the reservation exclude. So a delete sent without an answer is recorded (`InFlight`) before it is sent and sent again until it answers, **before** the obligation leaves `cleaning` (Task 4c), and only while rows 3 and 4 find both entries and the mark this cleanup's. When either is lost first, nothing more is sent; the key is `unsettled` and paged. Once a version is done, any attempt still in the network lands on nothing, because a version id is never reused. **The trade:** the retired replica is not recoverable from the bucket after the cleanup. It is the source copy of a finished move, and litestream's retention would purge it anyway. In exchange, the boundary is enforced by the server. `lock.json` and `build-state.json` are never inventoried and never deleted. |
| +- **The table is never deleted.** `control/offline-moves/active.json` is created once, by the first writer, as `{v: 1, epoch: <16 random bytes, hex>, rev: 1}` with `If-None-Match: *` (`release.Store.Put` with `ifMatch ""`, `internal/release/store.go:794-798`); emptiness is a table with no entries. Every later write is a `Put` with `If-Match` the ETag it read, of a body whose `rev` is the read body's plus one and whose `epoch` is the read body's (a fresh one when the read body has none, as an older console's writes do), so no two bodies the table ever holds are equal and an ETag never comes back. A conditional `Put` that read ETag E can land only while E is current, so once any later write succeeds, every earlier outstanding `Put` that read E can never land: an uncertain acquisition is settled by the next successful conditional write by anyone. A release always writes (rev + 1), even when its operation holds nothing, so its own success is the settlement its caller needs. This holds among new writers only; the rollback procedure covers the old ones. |
| - **The prefix (N1).** `Prefix = activator.HibernatePrefix(cfg.OwnPrefix, host) + "/"` (`HibernatePrefix` is `path.Join`, no trailing slash, `internal/activator/activator.go:147`), computed once at `gone` and persisted; the listing is `ListVersions(bucket, Prefix)`; the excluded keys are exactly `Prefix + "lock.json"` and `Prefix + "build-state.json"`; the lease location is `strings.TrimSuffix(Prefix, "/")`. |
| - **The cleanup id** is `offlinecleanup.CleanupID(box, host, moveID) = "cu_" + hex(sha256(box + "\n" + host + "\n" + moveID))[:16]`: recomputable from the directory and the box, so a corrupt file, root and the agent all name the same obligation. |
| -- **The mark.** Owner `lease.CleaningOwner(id) = "cleaning:" + id`; TTL `cleanupMarkTTL = 1 h`; reasserted whenever less than 30 min remain, in every state from the mark to the unmark; condition (2) requires at least `cleanupMarkMargin = 15 min` left, a bound that keeps the mark from lapsing inside a run (the safety of a late delete is the condition on its ETag, above). Written only by a conditional PUT from the lifted tombstone of this move (`moves.TombstoneOwner(MoveID)`, expired); removed by a conditional PUT back to exactly that lifted tombstone (`{Generation: as read, ExpiresAt: 1970-01-01T00:00:00Z, Owner: moved:<move id>}`, what `LiftTombstone` writes), so the operator return's leftover set (spec §7.5) is unchanged after any cleanup. An expired mark whose bytes are unchanged still proves no lease was taken since, because every acquisition rewrites the object. |
| +- **The mark.** Owner `lease.CleaningOwner(id) = "cleaning:" + id`; TTL `cleanupMarkTTL = 1 h`; reasserted whenever less than 30 min remain, in every state from the mark to the unmark; condition (2) requires at least `cleanupMarkMargin = 15 min` left, a bound that keeps the mark from lapsing inside a run. The mark excludes new wakes and replications for the whole run, so no version the run deletes can become a later owner's during it (a late attempt is safe by its version id, above). Written only by a conditional PUT from the lifted tombstone of this move (`moves.TombstoneOwner(MoveID)`, expired); removed by a conditional PUT back to exactly that lifted tombstone (`{Generation: as read, ExpiresAt: 1970-01-01T00:00:00Z, Owner: moved:<move id>}`, what `LiftTombstone` writes), so the operator return's leftover set (spec §7.5) is unchanged after any cleanup. An expired mark whose bytes are unchanged still proves no lease was taken since, because every acquisition rewrites the object. |
| - **The writers the mark excludes** (each writer's check, by file): the wake (`internal/activator/backend_exec.go:701`, `lease.Acquire` in `ExecBackend.Acquire`, step 1 of `doWake` in `activator.go`, before the restore, the staged-replica promotion `promoteStagedReplica` at `backend_exec.go:3397`, `BeforeExec`'s build-state write and litestream replication, all of which run under that lease); the legacy final checkpoint (`internal/host/movespass.go:321-328`, `lease.ReplicateWithLease`, which acquires); the offline seal's checkpoint (`internal/host/movesoffline_seal.go:417`, the same `Checkpoint`) and its tombstone (`movesoffline_seal.go:431`, `lease.Tombstone`, which never replaces a live lease of another owner, so the seal's carry at `:469` is not reached). Not excluded by the mark and harmless: the legacy cut's request carry (`movespass.go:1594`) and the offline seal's carry write only `build-state.json`, which is not inventoried; the seed rollback and finish (`seedwiring.go:247`, `console/seedfinish.go:254`) only delete; the target landing (`movesoffline_land.go:611`, `:1373`) belongs to a move into this region, which holds `host/<hostname>`. `TestHibernatePrefixSitesAreInventoried` (Task 3) pins this list. |
| - **Reservation keys and the one conflict relation.** Keys are a box id (today's bare label, unchanged), the sentinel `"*"`, or `moves.HostKey(host) = "host/" + host`. Kinds: `move`, `cleanup`, `create`, `place`. An entry is valid with a known kind, a non-empty id and host, a non-zero `reserved_at`, and, for `create` and `place`, a non-empty account and app; an entry under a host key names that host (`Host == strings.TrimPrefix(key, "host/")`). An existing entry of another `(Kind, ID)` conflicts with a requested set of keys when (a) it holds a requested key; or (b) one side holds `"*"` and the other holds a box key; or (c) the requester is the console with `cfg.SelfServeMoves != "offline"` and both are of kind `move` (plan 2's one-move-at-a-time rule, now between moves only). Host keys never meet (b) or (c). Every move selection (self-serve and operator) holds `host/<hostname>` beside its box keys or its sentinel; a cleanup holds `[cfg.BoxID, host/<hostname>]` with one entry; a create holds `[host/<hostname>]`; a placement holds `host/<h>` for every instance of the app, one entry per key with `Host = h` and the placement's account and app. |
| - **A cleanup entry is never reclaimed by another reserver.** Its own box resumes it after any crash (its own retry is accepted); a box that never returns leaves an entry the console pages after 6 hours (operator repair). Plan 2's `Server.cleanupReports` and `cleanupReportEvidence` (`offlinereserve.go:95-104`) are deleted. |
| -- **The four conditions at every deletion**, re-read before each `DeleteIfMatch`: (1) `moves.RecordAway(obs, MoveID, Incarnation, From) == AwayYes`: the judged observation is `LifetimeValid` (the judge's verdict that the slot vouches for the record) at `Incarnation`, with no move, `LastMove.ID == MoveID && LastMove.Outcome == "done"`, and `Region != From`; (2) `lock.json`'s owner is `CleaningOwner(ID)` with at least `cleanupMarkMargin` left after any reassert; (3) the box's registry has no row for the host (`w.offlineRow(host)`, `movesoffline.go:996`); (4) `ListVersions(bucket, key)` filtered to `Key == key` shows the inventory's `VersionID` as `IsLatest` and not a delete marker. A read error in any: the obligation stops for this run. (1), (2) or (3) failing definitely: **supersession** (spec §7.6's "superseded as a whole"; never an indefinite refusal). (4): another current version → that key `superseded` and a page; a delete marker current → `marked`; no version and no marker → `absent`. Conditions (1) and (3) are also checked at every other step of the lifecycle (Task 4a's `supersededBy`). |
| +- **The conditions at every key**, re-read before a key's first delete: (1) `moves.RecordAway(obs, MoveID, Incarnation, From) == AwayYes`: the judged observation is `LifetimeValid` (the judge's verdict that the slot vouches for the record) at `Incarnation`, with no move, `LastMove.ID == MoveID && LastMove.Outcome == "done"`, and `Region != From`; (2) `lock.json`'s owner is `CleaningOwner(ID)` with at least `cleanupMarkMargin` left after any reassert; (3) the box's registry has no row for the host (`w.offlineRow(host)`, `movesoffline.go:996`); (4) `ListVersions(bucket, key)` filtered to `Key == key` shows as current either one of the key's inventoried versions or a delete marker the inventory does not hold (someone deleted the key; the versions beneath it are ours). A read error in any: the obligation stops for this run. (1), (2) or (3) failing definitely: **supersession** (spec §7.6's "superseded as a whole"; never an indefinite refusal). (4) failing, an object version the inventory does not hold being current: that key `superseded`, never touched, and a page. Conditions (1) and (3) are also checked at every other step of the lifecycle (Task 4a's `supersededBy`). |
| - The cleanup writes nothing to the registry, the edges file or the offline view, so plan 3's holding entries (built from held rows and the view) are untouched; root's freeze holds the registry's write lock for an `fstatat` and an `unlinkat`, as held publication's does for its swap. |
| -- **Root's contract.** Root removes a path only (i) when the path's parent directory and every ancestor up to `/` is a directory owned by uid 0 with no group or other write bit (`offlinecleanup.RootOnlyChain`; otherwise the path is `kept`, reason "writable layout", because no exclusion against a writer that shares the agent's uid is possible there by construction); (ii) inside `reg.WithHostFrozen(host, …)` (no row for the host) and then the host's held-stage lock (`withHeldStageLock`, `movesheldlock.go:68`), in the order adopt pass lock > the registry's write lock > the held-stage lock (`movesheldlock.go:26-31`); (iii) after the authority callback has answered yes for this request in this pass and `root-cleanup.cancel` is absent; (iv) when `fstatat(parent, name, AT_SYMLINK_NOFOLLOW)` on the opened parent shows the recorded identity (device, inode, file type, link target); then `unlinkat(parent, name)`. An entry with another identity is `kept`, never removed. Every step is journalled as intent and then result in root's own directory, which is used only when it passes the same root-only check. Nothing inside the held-stage lock reads the network or takes a lock. |
| -- Timings: 24 h after `FinishedAt`; per run 60 s (`cleanupRunDeadline`), 10 s per operation, at most 50 keys per obligation; `cleanupUncertainQuiet` 15 min; pages every 10 min while root has not answered or has not acknowledged a cancel, hourly for a corrupt obligation and for a record unreadable for over an hour; the console pages a `cleanup` entry older than 6 h and a `create` entry with no allocation row older than `offlineLockGrace` (2 min); a crashed placement is settled after `offlineLockGrace`; the op directory goes 7 d after `EndedAt`; root removes its own directory 30 d after its acknowledgement. |
| -- Census: `internal/provision/testdata/plan14-root-reads-self-serve-moves-4.tsv`, loaded by a new `loadRootReadInventory(root+"/internal/provision/testdata/plan14-root-reads-self-serve-moves-4.tsv", &inv, true)` block in `internal/provision/plan14_root_reads_test.go` after plan 3's block (`:1383`) and its seal check and before `compareRootReads`, `rev3=PROPOSED-self-serve-moves-4`, rows at the `authority` class with evidence `E2,J3,J4` for the destructive readers (the box's record, registry, lock and versions reads; root's drop, its own request copy and journal, `active.json`, record, lock, registry-under-freeze, the layout `lstat` chain and the `fstatat` of each path); `TestPlan9B8RecordWriteCensus` (`plan9_b8_census_test.go:372`) takes `plan14SelfServeMoves4WriteCensusInventory()` wrapping `plan14SelfServeMoves3WriteCensusInventory()` (`self_serve_moves3_census_test.go:22`) with every new or changed bucket writer as an added or re-pinned unit (`moves.ReadActive`'s bootstrap, `moves.Reserve`, `moves.Release`, `reserveBoxes`, `releaseBoxes`, the cleanup report writer, the cleanup's `DeleteIfMatch`, the repair's `DeleteVersion` of a marker; the placement settle and the place sweep reach `fenceRecord`'s existing record unit); both green. |
| -- Copy rows (pending Paul's review; each task that emits one appends its row, at the next free `MV-O` id, status pending, to `docs/superpowers/plans/2026-10-06-moves-phase1-copy.md` and its id to `movesCopyIDs` in `approved_moves_copy_test.go`, in the same commit): "the previous instance at this hostname is still being cleaned up. try again later"; "an instance of this app is still being cleaned up. try again later"; "an instance of this app is moving. try again later"; "could not reserve the app's instances. try again"; page subjects "cleanup of <host> waits for root", "cleanup of <host> skipped <key>: another version is current", "cleanup of <host> waits for root to acknowledge its cancellation", "cleanup of <host> cannot read its record", "a cleanup has held <host> for over six hours", each with the body "run carlos move offline status --host <host> --log to see what it waits for"; the CLI's lines of Task 5b; the `--log` paragraph of `docs/site/cli.md`. No tenant names in tests (`shop.example.com`, account `bab`, app `shop`). |
| +- **Root's contract.** Root removes a path only (i) when the path's parent directory and every ancestor up to `/` is a directory owned by uid 0 with no group or other write bit (`offlinecleanup.RootOnlyChain`; otherwise the path is `kept`, reason "writable layout", because no exclusion against a writer that shares the agent's uid is possible there by construction); (ii) **as a stage of the root adoption pass** (`runAdoptUnderRootRelease`), after release adoption and named recovery and after the held stage, while the pass holds the adopt-pass lock and the root pass lock (`release.LockRootPass`, `<optDir>/root-cleanup/pass.lock` on a root-only chain, which every adopt pass takes in every mode before its first publisher), and then inside `reg.WithHostFrozen(host, …)` (no row for the host) and the host's held-stage lock (`withHeldStageLock`, `movesheldlock.go:68`), in the order adopt pass lock > root pass lock > the registry's write lock > the held-stage lock (`movesheldlock.go:26-31` gains the root pass lock); so no other root publisher (adoption, named recovery, `PointLive`) runs on the box while root's step runs, and the agent's publishers are excluded by the freeze and the held-stage lock; (iii) after the authority callback has answered yes for this request in this pass and `root-cleanup.cancel` is absent; (iv) when `fstatat(parent, name, AT_SYMLINK_NOFOLLOW)` on the opened parent shows the recorded identity (device, inode, file type, link target); then `unlinkat(parent, name)`. An entry with another identity is `kept`, never removed. Every step is journalled as intent and then result in root's own directory, which is used only when it passes the same root-only check; when it does not, root writes nothing there (Task 6b). Nothing inside the held-stage lock reads the network or takes a lock. |
| +- Timings: 24 h after `FinishedAt`; per run 60 s (`cleanupRunDeadline`), 10 s per operation, at most 50 keys per obligation; pages every 10 min while root has not answered or has not acknowledged a cancel, hourly once root has not answered for an hour and its request directory's ancestry is not root-only (`rootRefusedAfter`), hourly for a corrupt obligation and for a record unreadable for over an hour; the console pages a `cleanup` entry older than 6 h and a `create` entry with no allocation row older than `offlineLockGrace` (2 min); a placement runs under one 15 s deadline (`instanceStoreTimeout`), and an unregistered `place` entry older than `offlineLockGrace` is settled by the sweep; the op directory goes 7 d after `EndedAt`; root removes its own directory 30 d after its acknowledgement. |
| +- Census: `internal/provision/testdata/plan14-root-reads-self-serve-moves-4.tsv`, loaded by a new `loadRootReadInventory(root+"/internal/provision/testdata/plan14-root-reads-self-serve-moves-4.tsv", &inv, true)` block in `internal/provision/plan14_root_reads_test.go` after plan 3's block (`:1383`) and its seal check and before `compareRootReads`, `rev3=PROPOSED-self-serve-moves-4`, rows at the `authority` class with evidence `E2,J3,J4` for the destructive readers (the box's record, registry, lock and versions reads; root's drop, its own request copy and journal, `active.json`, record, lock, registry-under-freeze, the layout `lstat` chain, the root pass lock's chain and the `fstatat` of each path); `TestPlan9B8RecordWriteCensus` (`plan9_b8_census_test.go:372`) takes `plan14SelfServeMoves4WriteCensusInventory()` wrapping `plan14SelfServeMoves3WriteCensusInventory()` (`self_serve_moves3_census_test.go:22`) with every new or changed bucket writer as an added or re-pinned unit (`moves.ReadActive`'s bootstrap, `moves.Reserve`, `moves.Release`, `reserveBoxes`, `releaseBoxes`, the drain's settling `Release`, the cleanup report writer, the cleanup's `DeleteVersion` through `deleteInventoriedVersion`; the placement settle, the place sweep and the drain reach `fenceRecord`'s existing record unit); both green. |
| +- Copy rows (pending Paul's review; each task that emits one appends its row, at the next free `MV-O` id, status pending, to `docs/superpowers/plans/2026-10-06-moves-phase1-copy.md` and its id to `movesCopyIDs` in `approved_moves_copy_test.go`, in the same commit): "the previous instance at this hostname is still being cleaned up. try again later"; "an instance of this app is still being cleaned up. try again later"; "an instance of this app is moving. try again later"; "could not reserve the app's instances. try again"; "this console is being rolled back. try again in a few minutes"; page subjects "cleanup of <host> waits for root", "cleanup of <host> skipped <key>: another version is current", "cleanup of <host> waits for root to acknowledge its cancellation", "cleanup of <host> cannot read its record", "a cleanup has held <host> for over six hours", "cleanup of <host>: root refused the request directory's ancestry", "cleanup of <host> lost its mark before a delete of <key> was answered", each with the body "run carlos move offline status --host <host> --log --cleanup <cleanup id> to see what it waits for" (the console's "held for over six hours" page names the entry's id the same way); the CLI's lines of Tasks 5b and 7c; the `--log` paragraph and the `reservations drain` line of `docs/site/cli.md`. No tenant names in tests (`shop.example.com`, account `bab`, app `shop`). |
| - Local tests: touched packages with `-run`; CI runs the full suite and `go test -tags e2e ./internal/e2e/...`. |
| ## Review Focus |
| -1. The F1 and N4 interleavings: a reconcile paused with an old observation mints the source row after the cleanup marked the lock, and a wake through `ExecBackend.Acquire` then tries the lease: it gets `HeldError` naming `cleaning:<id>`, no version is written under the prefix, the next key's condition (3) supersedes, the lifted tombstone is restored, the reservation released, and the wake then succeeds. A delete issued before the supersession and held by the service lands after the wake wrote a new version of its key: the service refuses it (412), the new version stays current; with byte-identical bytes the release waits out `QuietUntil` first, and a delete that lands even later on a byte-identical re-promoted version is repaired on the next pass with that newer version current again (Task 4b, `TestAdmittedReconcileAndWakeCannotWriteUnderTheMark`, `TestOutstandingDeleteCannotHideANewerVersion`, `TestByteIdenticalRewriteWaitsOutTheQuiet`, `TestLateDeleteOnAByteIdenticalVersionIsRepaired`). |
| -2. A crash after every write and every side effect of the lifecycle (after `Reserve` before `cleaning`, after `cleaning` before the mark, after the mark before `MarkedAt`, after `QuietUntil` before a delete, after a delete before its key outcome, after the last key before `publishing`, after `releasing` before the unmark, after the unmark before the release, after the release before `ended`), restart, completion: no key is deleted unless both entries and the mark are found to be this cleanup's, an entry an operator removed or a mark replaced turns the obligation `superseded`, nothing is re-acquired from `cleaning`, and the outcome is the table's (Task 4a, `TestLifecycleResumesAtEveryBoundary`). |
| -3. Root under attack and crash: a shared-uid writer plants an acknowledgement, a request with another id, a changed copy of the request and a cancel in the agent's directory; a publication replaces the live link between root's reference check and its `fstatat`; a route is upserted during the freeze; the link's parent chain is writable; root crashes after each journal line and each syscall: root removes only an entry carrying the recorded identity, the replacement survives, the upsert waits for the freeze, the writable layout is never touched, every intent resolves to removed, kept or absent, root finds its own taken request on restart, and the forged files change nothing but stopping (Tasks 6b and 6c, `TestRootRemovesOnlyTheRecordedIdentity`, `TestWritableLayoutIsNeverRemoved`, `TestRootJournalResolvesEveryIntent`, `TestRootFindsItsOwnRequestsOnRestart`, `TestForgedAgentFilesAreHarmless`). |
| -4. Residence changes and cleanup exclude each other through `host/<hostname>`: a create (busy → 409, row void, no reservation written), a crash after `BeginAllocation` and after `Reserve`, a terminal row whose release failed, a create's reservation PUT that lands after its recovery voided and released it (refused by its condition); a placement that is busy (nothing written), whose record PUT commits after the client gave up (fenced before release) and whose console crashed (the sweep fences and releases); an offline move selection; with the switch off two creates of different hosts and a cleanup beside an operator move all proceed, and the sentinel never blocks a create (Tasks 2, 7a and 7b, `TestConflictRelation`, `TestUncertainReservationIsSettledByTheNextWrite`, `TestResidenceChangesAndCleanupExclude`). |
| -5. Publication is durable: the report under the move id survives failed writes and is republished until `PublishedRev == Rev`; a refusal repeated every run is one event with a count; a skipped key's page survives an outbox that cannot create it, a pager that refuses it and a restart; a periodic page fires once per tick across restarts; the reaper waits for the report, `pages_pending`, the outbox and root's acknowledgement (Tasks 5a and 8, `TestCleanupReportIsDurableAndBlocksTheReaper`, `TestPendingPagesSurviveARestart`). |
| +1. The F1 and N4 interleavings by version id: a reconcile paused with an old observation mints the source row after the cleanup marked the lock, and a wake through `ExecBackend.Acquire` then tries the lease: it gets `HeldError` naming `cleaning:<id>`, no version is written under the prefix, the next key's condition (3) supersedes, the lifted tombstone is restored, the reservation released, and the wake then succeeds. A delete sent before the supersession and still unanswered is sent again under the mark and answered before `releasing` is written; an earlier attempt of it delivered after the release lands on nothing. Two attempts of one delete (an SDK retry) remove one version. No partial run ever leaves a version older than the inventoried current one current. A promotion back into the region that kept an inventoried version by its equal ETag finds it intact after the cleanup ended (Tasks 4b and 4c, `TestAdmittedReconcileAndWakeCannotWriteUnderTheMark`, `TestUnansweredDeleteIsAnsweredBeforeTheUnmark`, `TestRetriedAttemptsRemoveOnlyTheirVersion`, `TestDeletionNeverExposesAnOlderVersion`, `TestPromotionThatKeptAnInventoriedVersionKeepsIt`). |
| +2. A crash after every write and every side effect of the lifecycle (after `Reserve` before `cleaning`, after `cleaning` before the mark, after the mark before `MarkedAt`, after `InFlight` is written before its delete, after a delete before `Done` moves, after the last key before `publishing`, after `releasing` before the unmark, after the unmark before the release, after the release before `ended`), restart, completion: no version is deleted unless both entries and the mark are found to be this cleanup's, an entry an operator removed or a mark replaced turns the obligation `superseded` with any unanswered delete recorded `unsettled`, nothing is re-acquired from `cleaning`, and the outcome is the table's (Task 4a, `TestLifecycleResumesAtEveryBoundary`). |
| +3. Root under attack and crash: a shared-uid writer plants an acknowledgement, a request with another id, a changed copy of the request and a cancel in the agent's directory, and replaces `/data/adopt-pass.lock` with a symlink so a second pass runs without that lock; a publication replaces the live link between root's reference check and its `fstatat`; a second adopt pass starts between root's `fstatat` and its `unlinkat`; a route is upserted during the freeze; the link's parent chain is writable; root's request directory's ancestry is writable; root crashes after each journal line and each syscall: root removes only an entry carrying the recorded identity, the replacement survives, the second pass waits on the root pass lock until the step returns, the upsert waits for the freeze, the writable layout is never touched, a rejected ancestry gets no write, every intent resolves to removed, kept or absent, root finds its own taken request on restart, and the forged files change nothing but stopping (Tasks 6b and 6c, `TestRootRemovesOnlyTheRecordedIdentity`, `TestNoPublisherRunsBetweenFstatatAndUnlinkat`, `TestWritableLayoutIsNeverRemoved`, `TestRejectedAncestryGetsNoWrite`, `TestRootJournalResolvesEveryIntent`, `TestRootFindsItsOwnRequestsOnRestart`, `TestForgedAgentFilesAreHarmless`). |
| +4. Residence changes and cleanup exclude each other through `host/<hostname>`: a create (busy → 409, row void, no reservation written), a crash after `BeginAllocation` and after `Reserve`, a terminal row whose release failed, a create's reservation PUT that lands after its recovery voided and released it (refused by its condition); a placement that is busy (nothing written), whose record PUT commits after the client gave up (fenced before release), that pauses after its reservation for longer than the grace (registered, so never swept; its local writes fail at its deadline) and whose console crashed (the sweep fences and releases); an offline move selection; with the switch off two creates of different hosts and a cleanup beside an operator move all proceed, and the sentinel never blocks a create; the drain settles every `place` and `create` entry and refuses to finish while a self-serve move or a cleanup holds an entry (Tasks 2a to 2c, 7a, 7b and 7c, `TestConflictRelation`, `TestUncertainReservationIsSettledByTheNextWrite`, `TestResidenceChangesAndCleanupExclude`, `TestPausedPlacementIsNeverSwept`, `TestDrainSettlesBeforeItReleases`). |
| +5. Publication is durable and findable: the report under the move id survives failed writes and is republished until `PublishedRev == Rev`; a refusal repeated every run is one event with a count; a skipped key's page survives an outbox that cannot create it, a pager that refuses it and a restart; a periodic page fires once per tick across restarts; the reaper waits for the report, `pages_pending`, the outbox and root's acknowledgement; an older obligation's report is found by its cleanup id after `LastMove` moved on and after the record was deleted (Tasks 5a, 5b and 8, `TestCleanupReportIsDurableAndBlocksTheReaper`, `TestPendingPagesSurviveARestart`, `TestOlderCleanupIsFoundByItsID`). |
| --- |
| ### Task 1: The obligation and its evidence at `gone` |
| **Files:** |
| -- Create: `internal/offlinecleanup/wire.go` (standard library only; every wire type of this plan, `Retired`, `CorruptStub`, `InventoryObject`, `KeyOutcome`, `Event`, `PendingPage`, `Report`, `FileIdentity`, `LocalPath`, `KeptPath`, `RootCleanupRequest`, `RootCleanupPath`, `RootCleanupAck`, with Tasks 5a and 6b giving the last ones their semantics), `internal/offlinecleanup/layout_linux.go` and `layout_other.go` (`Identify`, `RootOnlyChain`), `internal/host/movescleanup_retired.go` |
| +- Create: `internal/offlinecleanup/wire.go` (standard library only; every wire type of this plan, `Retired`, `CorruptStub`, `InventoryKey`, `InventoryVersion`, `InFlightDelete`, `KeyOutcome`, `Event`, `PendingPage`, `Report`, `FileIdentity`, `LocalPath`, `KeptPath`, `RootCleanupRequest`, `RootCleanupPath`, `RootCleanupAck`, with Tasks 5a and 6b giving the last ones their semantics), `internal/offlinecleanup/layout_linux.go` and `layout_other.go` (`Identify`, `RootOnlyChain`), `internal/host/movescleanup_retired.go` |
| - Modify: `internal/moves/offline.go` (`OfflineOpDir`, `RetiredFile`, `CleanupReportKey`), `internal/host/movesoffline.go` (`offlineSourceGone`, line 1493: with `rt == nil` and this box the move's sealer by `cfg.offlineRoles(m)`, `w.recordRetired(ctx, s, rec)` runs before the gone report; an error writes no gone report this pass; "untracked" is not an error), `internal/release/programhistory.go` (export `ProgramsPath(bakeDir, host)`, today's `programPath`) |
| - Test: `internal/offlinecleanup/wire_test.go`, `internal/offlinecleanup/layout_linux_test.go`, `internal/host/movescleanup_retired_test.go` |
| **Interfaces:** |
| - `func CleanupID(box, host, moveID string) string` as the Global Constraint; `const MarkPrefix = "cleaning:"`. |
| -- `type Retired struct { V int; ID, MoveID, Host, Account, App, Box string; Incarnation uint64; From, To string; OpenAt time.Time; FinalTXID uint64; TombstoneOwner string; Bucket, BucketRegion, Prefix string; BucketInventory []InventoryObject; LocalInventory []LocalPath; LocalLeft []KeptPath; WrittenAt time.Time; State, Outcome string; FinishedAt, ReservedAt, MarkedAt, QuietUntil, RepairSince, RootRequestedAt, CancelRequestedAt, EndedAt time.Time; RunKeys, RepairKeys []string; RepairPasses int; Keys map[string]KeyOutcome; Events []Event; PagesPending []PendingPage; Root *RootCleanupAck; Rev, PublishedRev uint64 }`, every field tagged snake_case, times `omitzero`, `V = 1`. |
| +- `type Retired struct { V int; ID, MoveID, Host, Account, App, Box string; Incarnation uint64; From, To string; OpenAt time.Time; FinalTXID uint64; TombstoneOwner string; Bucket, BucketRegion, Prefix string; BucketInventory []InventoryKey; LocalInventory []LocalPath; LocalLeft []KeptPath; WrittenAt time.Time; State, Outcome string; FinishedAt, ReservedAt, MarkedAt, RootRequestedAt, CancelRequestedAt, EndedAt time.Time; ReserveUncertain bool; InFlight *InFlightDelete; Keys map[string]KeyOutcome; Events []Event; PagesPending []PendingPage; Root *RootCleanupAck; Rev, PublishedRev uint64 }`, every field tagged snake_case, times `omitzero`, `V = 1`. |
| - **States** (`State`, the only lifecycle field): `open`, `finished`, `cleaning`, `publishing`, `root-requested`, `cancel-pending`, `releasing`, `ended`, `corrupt`. `Outcome` is "" until `releasing`, then `cleaned`, `superseded` or `cancelled` (Task 4a's outcome table). |
| -- `type InventoryObject struct { Key, VersionID, ETag string; Size int64 }` (`ETag` unquoted, as `lease.ObjectVersion` carries it, `internal/lease/versioned.go:25-33`); `type KeyOutcome struct { State string (pending, deleted, marked, superseded, absent); VersionID string; At time.Time }`; `type Event`, `type PendingPage` (Task 5a); `type FileIdentity struct { Dev, Inode uint64; Mode string (symlink or regular); LinkTarget string }`; `type LocalPath struct { Which string (live-link or programs-file); Identity FileIdentity }` (root derives the path itself; the agent's file names none); `type KeptPath struct { Which, Reason string }`. |
| +- `type InventoryKey struct { Key string; Versions []InventoryVersion }` with `Versions` in deletion order; `type InventoryVersion struct { VersionID, ETag string; Size int64; DeleteMarker, Latest bool; LastModified time.Time }` (`ETag` unquoted, as `lease.ObjectVersion` carries it, `internal/lease/versioned.go:25-33`; "" for a marker); `type InFlightDelete struct { Key, VersionID string; At time.Time }`; `type KeyOutcome struct { State string (pending, deleted, superseded, unsettled); Done int (versions, in deletion order, known done); At time.Time }`; `type Event`, `type PendingPage` (Task 5a); `type FileIdentity struct { Dev, Inode uint64; Mode string (symlink or regular); LinkTarget string }`; `type LocalPath struct { Which string (live-link or programs-file); Identity FileIdentity }` (root derives the path itself; the agent's file names none); `type KeptPath struct { Which, Reason string }`. |
| - `func Identify(path string) (FileIdentity, bool, error)`: `Lstat` (never follows), `Readlink` for a symlink; absent → `false`; any other error fails the step. `func RootOnlyChain(dir string) (bool, error)`: `Lstat` of every component from `/` to `dir`: each a directory (a symlink component answers false), owned by uid 0, `mode & 0o022 == 0`. Off Linux both answer absent and false (the cleanup is Linux-only, as the box is). |
| -- `ValidateRetired(r Retired) error`: `V == 1`; `ID == CleanupID(Box, Host, MoveID)`; `MoveID` a plain segment; `Host` passes the host shape; `Account`, `App`, `Box` non-empty; `Incarnation > 0`; `From != To`; `State` in the set and not `corrupt`; `Outcome` "" before `releasing` and in the set after; `WrittenAt` non-zero; `Prefix` ends with `/`; inventory keys unique, under `Prefix`, neither excluded key; `ETag` and `VersionID` non-empty for each; `Which` values unique across `LocalInventory` and `LocalLeft`, and known; every `Keys` entry, `RunKeys` and `RepairKeys` entry names an inventory key; `RepairPasses` in 0 to 2. |
| +- `ValidateRetired(r Retired) error`: `V == 1`; `ID == CleanupID(Box, Host, MoveID)`; `MoveID` a plain segment; `Host` passes the host shape; `Account`, `App`, `Box` non-empty; `Incarnation > 0`; `From != To`; `State` in the set and not `corrupt`; `Outcome` "" before `releasing` and in the set after; `WrittenAt` non-zero; `Prefix` ends with `/`; inventory keys unique, under `Prefix`, neither excluded key; every key has at least one version, version ids non-empty and unique within the key, exactly one `Latest` and it is the last, `ETag` non-empty exactly for the versions that are not markers; `Which` values unique across `LocalInventory` and `LocalLeft`, and known; every `Keys` entry names an inventory key with `Done` in `0..len(Versions)`; `InFlight`, when set, names an inventoried version of a key whose `Keys` state is `pending`. |
| - `type CorruptStub struct { Corrupt bool; ID, Host string; FoundAt time.Time }` (`{corrupt: true, id, host, found_at}`); `ValidateCorruptStub(c, box, host, move string) error`: `Corrupt`, `Host == host`, `ID == CleanupID(box, host, move)`, `FoundAt` non-zero. |
| - `func OfflineOpDir(dataDir, host, op string) string` (`<dataDir>/.offline-moves/<host>/<op>`, "" on a bad name, the checks `OfflineParkDir` applies); `func RetiredFile(dataDir, host, op string) string`; `func CleanupReportKey(host, op, box string) string` (`OfflineReportPrefix(host, op) + "cleanup/" + box + ".json"`). |
| -- **The evidence at `gone`** (`recordRetired`): the record is in `offline-finishing` carrying this move, `cfg.Judge != nil`, and `cfg.Region == m.From` (else nothing is written and the box logs once per day). **Tracked (N5):** `obs := cfg.Judge.Observe(ctx, s, rec.Account, rec.App, host)`; `LifetimeUnreadable` → an error (the step retries next pass, as any read in it does); `obs.Verdict == LifetimeValid && obs.Rec.Incarnation > 0 && obs.Slot != nil && obs.Slot.Incarnation == obs.Rec.Incarnation` → tracked; any other answer (incarnation 0 with no slot, a migrating record, stale, corrupt) → untracked: no `retired.json`, logged once per day, and `recordRetired` returns nil so the gone report is written as before this plan. Then: `OpenAt = *Move.OpenedAt`; `FinalTXID` from the seal at `moves.OfflineSealKey(host, id)` validated by `moves.ValidateOfflineSeal`; `TombstoneOwner = moves.TombstoneOwner(id)`; `Bucket/BucketRegion = cfg.OwnBucket/OwnBucketRegion`; `Prefix` as the Global Constraint; `BucketInventory` = `cfg.Versioned(ctx, Bucket, BucketRegion).ListVersions(ctx, Bucket, Prefix)` keeping `IsLatest && !DeleteMarker && strings.HasPrefix(Key, Prefix)` without the two excluded keys, with each version's id and ETag; for each of `release.LiveLink(cfg.OptDir, host)` and `release.ProgramsPath(cfg.DataDir, host)`: absent → nothing; present and `RootOnlyChain(filepath.Dir(path))` → `LocalInventory`; present otherwise → `LocalLeft` with reason "writable layout" (on every built box the programs file is here: the agent's unit has `ReadWritePaths=/data`); `State = open`, `Rev = 1`. |
| +- **The evidence at `gone`** (`recordRetired`): the record is in `offline-finishing` carrying this move, `cfg.Judge != nil`, and `cfg.Region == m.From` (else nothing is written and the box logs once per day). **Tracked (N5):** `obs := cfg.Judge.Observe(ctx, s, rec.Account, rec.App, host)`; `LifetimeUnreadable` → an error (the step retries next pass, as any read in it does); `obs.Verdict == LifetimeValid && obs.Rec.Incarnation > 0 && obs.Slot != nil && obs.Slot.Incarnation == obs.Rec.Incarnation` → tracked; any other answer (incarnation 0 with no slot, a migrating record, stale, corrupt) → untracked: no `retired.json`, logged once per day, and `recordRetired` returns nil so the gone report is written as before this plan. Then: `OpenAt = *Move.OpenedAt`; `FinalTXID` from the seal at `moves.OfflineSealKey(host, id)` validated by `moves.ValidateOfflineSeal`; `TombstoneOwner = moves.TombstoneOwner(id)`; `Bucket/BucketRegion = cfg.OwnBucket/OwnBucketRegion`; `Prefix` as the Global Constraint; **`BucketInventory`** = `cfg.Versioned(ctx, Bucket, BucketRegion).ListVersions(ctx, Bucket, Prefix)` (any page error fails the whole call, `versioned.go:45-50`), keeping every entry, object version or delete marker, with `strings.HasPrefix(Key, Prefix)` and without the two excluded keys, grouped by key; each key's versions in deletion order: the entries that are not `IsLatest` in the reverse of the listing's order (the listing is newest first with ties unordered, so this is oldest first; a tie's order does not matter, because deleting a version that is not current changes no current version), then the one `IsLatest` entry (`Latest`); for each of `release.LiveLink(cfg.OptDir, host)` and `release.ProgramsPath(cfg.DataDir, host)`: absent → nothing; present and `RootOnlyChain(filepath.Dir(path))` → `LocalInventory`; present otherwise → `LocalLeft` with reason "writable layout" (on every built box the programs file is here: the agent's unit has `ReadWritePaths=/data`); `State = open`, `Rev = 1`. |
| - Writes: `createRetired` is create-only (temp file, `fsync`, `os.Link` into place, directory `fsync`; an existing file is never replaced); `writeRetired` (temp, `fsync`, rename, directory `fsync`) increments `Rev` on every write except `markPublished` (Task 5a), which sets `PublishedRev` alone. |
| - **Corrupt (N6):** a file that is a valid `CorruptStub` stays as it is, never rewritten. A file that is neither a valid `Retired` nor a valid stub is moved aside create-only (`os.Link` then remove): to `retired.json.corrupt-1` when that name is free, else to the next free `retired.json.corrupt-<n>`, so the first damaged file is never overwritten; then the stub is written with `FoundAt` = `retired.json.corrupt-1`'s mtime. A stub keeps the obligation listed as `corrupt` and pages hourly from `FoundAt` (Task 5a) until an operator repairs it (a valid `retired.json` restored from `.corrupt-1`, or the directory removed after the entries and the mark are released by hand); it is never reaped and never released, so any entry or mark it held stays until the repair. |
| -- [ ] **Step 1: Write the failing tests**: `TestRetiredJSONWrittenOnceAtGoneBySealer` (fields, each inventory object with its ETag; `FinalTXID` from the seal; a non-sealer source box writes nothing; a second gone pass does not rewrite; an identity error writes nothing and no gone report); `TestUntrackedLifetimeWritesNoRetiredAndReportsGone` (incarnation 0 with no slot, a migrating record, a stale verdict: no file, the gone report written; an unreadable lifetime: no file and no gone report this pass, both next pass); `TestRetiredPrefixIsNormalisedOnce` (`cfg.OwnPrefix = "instances"` → `Prefix == "instances/shop.example.com/"`; the fake versioned store was listed with exactly that prefix; `instances/shop.example.com/lock.json` and `.../build-state.json` excluded; `instances/shop.example.co` keys not listed; `ValidateRetired` passes on what gone wrote); `TestWritablePathsAreLeftNotRequested` (a path under a directory the test uid owns → `LocalLeft`, "writable layout"; a path under a root-only chain the test cannot build is covered by injecting `RootOnlyChain`); `TestCleanupIDIsDeterministic`; `TestValidateRetired` (each rule); `TestCorruptStubIsValidAndKeepsTheFirstDamage` (damaged → `.corrupt-1` and a stub that validates; a second scan changes nothing; a damaged stub → `.corrupt-2`, `FoundAt` still `.corrupt-1`'s mtime); `TestIdentifyUsesLstat` (a dangling link has an identity; a regular file; absent); `TestRootOnlyChain` (`/usr` true on a normal system; a temp directory false; a symlink component false). |
| -- [ ] **Step 2: Run to verify they fail**: `go test ./internal/offlinecleanup -count=1`, `go test ./internal/host -run 'RetiredJSON|Untracked|RetiredPrefix|WritablePaths|CorruptStub' -count=1`. |
| +- [ ] **Step 1: Write the failing tests**: `TestRetiredJSONWrittenOnceAtGoneBySealer` (fields; a key with three versions and a key whose current entry is a marker each recorded whole, in deletion order with the latest last and each version's ETag; `FinalTXID` from the seal; a non-sealer source box writes nothing; a second gone pass does not rewrite; an identity error writes nothing and no gone report); `TestUntrackedLifetimeWritesNoRetiredAndReportsGone` (incarnation 0 with no slot, a migrating record, a stale verdict: no file, the gone report written; an unreadable lifetime: no file and no gone report this pass, both next pass); `TestRetiredPrefixIsNormalisedOnce` (`cfg.OwnPrefix = "instances"` → `Prefix == "instances/shop.example.com/"`; the fake versioned store was listed with exactly that prefix; `instances/shop.example.com/lock.json` and `.../build-state.json` excluded with all their versions; `instances/shop.example.co` keys not listed; `ValidateRetired` passes on what gone wrote); `TestInventoryOrderIsOldestFirstLatestLast` (versions with equal `LastModified`: the latest is last whatever the tie order); `TestWritablePathsAreLeftNotRequested` (a path under a directory the test uid owns → `LocalLeft`, "writable layout"; a path under a root-only chain the test cannot build is covered by injecting `RootOnlyChain`); `TestCleanupIDIsDeterministic`; `TestValidateRetired` (each rule); `TestCorruptStubIsValidAndKeepsTheFirstDamage` (damaged → `.corrupt-1` and a stub that validates; a second scan changes nothing; a damaged stub → `.corrupt-2`, `FoundAt` still `.corrupt-1`'s mtime); `TestIdentifyUsesLstat` (a dangling link has an identity; a regular file; absent); `TestRootOnlyChain` (`/usr` true on a normal system; a temp directory false; a symlink component false). |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/offlinecleanup -count=1`, `go test ./internal/host -run 'RetiredJSON|Untracked|RetiredPrefix|InventoryOrder|WritablePaths|CorruptStub' -count=1`. |
| - [ ] **Step 3: Implement. Step 4: Run to verify they pass.** |
| -- [ ] **Step 5: Commit** `moves: the sealing source box records retired.json at gone for a tracked lifetime, with ETags, a normalised prefix and a deterministic cleanup id`. |
| +- [ ] **Step 5: Commit** `moves: the sealing source box records retired.json at gone for a tracked lifetime, with every version id in deletion order, a normalised prefix and a deterministic cleanup id`. |
| -### Task 2: Hostname keys, a table that is never deleted, and one conflict relation |
| +### Task 2a: The table's kinds, epoch and rev, and its parser |
| **Files:** |
| -- Create: `internal/moves/reserve.go` |
| -- Modify: `internal/moves/offline.go` (`OfflineKindCreate = "create"`, `OfflineKindPlace = "place"`; `OfflineActive` gains `Epoch string \`json:"epoch,omitempty"\`` and `Rev uint64 \`json:"rev,omitempty"\``; `OfflineReservation.valid()` and `ParseOfflineActive` take the Global Constraint's entry rules; `HostKey`, `IsHostKey`), `internal/console/lifetimes_store.go` (`AllocationState(opID) (state string, found bool, err error)`), `internal/console/offlinereserve.go` (`reserveBoxes` reads through `moves.ReadActive` and writes `moves.NextActiveBody` with `If-Match`; its steps (4) and (5) become one call to `moves.Conflicts`; the wanted keys add `moves.HostKey(r.Host)`; `releaseBoxes` wraps `moves.Release`, so the `DeleteIfMatch` at line 478 goes; `offlineReclaimable`: `cleanup` → `reclaimHeld` always, `create` and `place` below; `cleanupReportEvidence` and `Server.cleanupReports` deleted), `internal/console/selfserve_pass.go` (`SelfServeEvidence`, lines 41-53, counts only entries of kind `move`), `internal/console/lifetimes_recovery.go` (each iteration also runs `s.residenceSweep(ctx, now)`), `internal/console/selfserve_pages.go` (two page conditions) |
| -- Test: `internal/moves/reserve_test.go`, `internal/console/offlinereserve_hostkey_test.go` |
| +- Create: `internal/moves/reserve.go` (`ReadActive`, `NextActiveBody`, `Release`, `Holds`; Task 2b adds the rest) |
| +- Modify: `internal/moves/offline.go` (`OfflineKindCreate = "create"`, `OfflineKindPlace = "place"`; `OfflineActive` gains `Epoch string \`json:"epoch,omitempty"\`` and `Rev uint64 \`json:"rev,omitempty"\``; `OfflineReservation.valid()` and `ParseOfflineActive` take the Global Constraint's entry rules; `HostKey`, `IsHostKey`) |
| +- Test: `internal/moves/reserve_test.go`, `internal/moves/offline_compat_test.go` |
| **Interfaces:** |
| - `func HostKey(host string) string`; `func IsHostKey(key string) bool` (`strings.HasPrefix(key, "host/")`; a box label never contains `/`, `controlname.ValidLabel`). |
| - `type Active struct { Body OfflineActive; Table map[string]OfflineReservation; ETag string }`; `func ReadActive(ctx context.Context, s release.Store) (Active, error)`: `Get`; absent → the bootstrap `Put` of `{v: 1, epoch: newEpoch(), rev: 1}` with `ifMatch ""` (create-only), then `Get` again whatever the `Put` answered (a lost create means another writer created it; an uncertain one can only ever create an empty table, so it is harmless); a read error or `ParseOfflineActive` failing → `ErrUnreadable`. `func NextActiveBody(prev OfflineActive, table map[string]OfflineReservation) OfflineActive`: `OfflineActiveBody(table)` with `Epoch = prev.Epoch` (a new one when empty) and `Rev = prev.Rev + 1`. Readers that only read (`SelfServeEvidence`, `selfserve_pages.go:143`) keep treating an absent object as an empty table. |
| -- `type Op struct{ Kind, ID string }`; `func Conflicts(table map[string]OfflineReservation, keys []string, mine Op, movesExclusive bool) map[Op]map[string]OfflineReservation` = the Global Constraint's relation, `movesExclusive` applying rule (c) only when `mine.Kind == move`. |
| -- `var ErrUnreadable`, `ErrReservationMismatch`, `ErrUncertain`; `type ErrReserved struct { Key string; Holder OfflineReservation }`; `func Reserve(ctx context.Context, s release.Store, mine Op, want map[string]OfflineReservation) (map[string]OfflineReservation, error)` for the box, the create and the placement (N10: the caller names the entry for each key; every entry carries `mine` and is valid, else an error and nothing written): `ReadActive`; own entries on a key outside `want` → `ErrReservationMismatch`; own entries covering `want` → the held entries returned, nothing written; any conflict → `ErrReserved` naming the oldest conflicting entry (no reclaim on these paths); else one `Put` with `If-Match` the read ETag of `NextActiveBody` with the missing keys added (at the held entries' `ReservedAt` when partial); a CAS conflict re-reads once; a second loss → `ErrReserved` from the last read; a `Put` error that is not a CAS conflict → `ErrUncertain` wrapped: the caller keeps its state as if it might hold the keys, and the next successful write by anyone, its own retry or its release included, settles it. |
| -- `func Holds(table map[string]OfflineReservation, keys []string, mine Op) bool`; `func Release(ctx context.Context, s release.Store, mine Op) error`: `ReadActive`; remove `mine`'s entries; **always** one `Put` with `If-Match` of `NextActiveBody`, even when `mine` held nothing (the settling write); a CAS conflict re-reads, three rounds as `releaseBoxes`; a malformed table is an error and is left as it is. A release that read no entry and wrote successfully proves that no earlier `Put` of `mine` can land (the Global Constraint), which a read alone never does. |
| +- `type Op struct{ Kind, ID string }`; `var ErrUnreadable`; `func Holds(table map[string]OfflineReservation, keys []string, mine Op) bool`; `func Release(ctx context.Context, s release.Store, mine Op) error`: `ReadActive`; remove `mine`'s entries; **always** one `Put` with `If-Match` of `NextActiveBody`, even when `mine` held nothing (the settling write); a CAS conflict re-reads, three rounds as `releaseBoxes`; a malformed table is an error and is left as it is. A release that read no entry and wrote successfully proves that no earlier `Put` of `mine` can land (the Global Constraint), which a read alone never does. `Release` accepts an `Op` of any kind, so the drain's settling write (Task 7c) is a `Release` of an op that holds nothing. |
| +- **The old parser, frozen:** `offline_compat_test.go` carries `parsePlan3OfflineActive`, a verbatim copy of `ParseOfflineActive` and `valid()` as built at `0840937f` (`offline.go:584-587`, `:599-622`), and is the witness of what a console before this plan reads. |
| + |
| +- [ ] **Step 1: Write the failing tests**: `TestEntryShapeRules` (a `place` or `create` entry without account or app, a host-key entry naming another host → malformed; each valid shape parses); `TestOldParserReadsTheNewTable` (bodies written by `NextActiveBody` with `move` and `cleanup` entries, host keys and `epoch`/`rev` parse with `parsePlan3OfflineActive`; a body with a `create` or a `place` entry is malformed to it, which the rollback procedure's step 2 covers); `TestActiveTableIsNeverDeleted` (releasing the last entry writes an empty table; no `Delete` or `DeleteIfMatch` reaches the key); `TestBootstrapIsCreateOnlyAndEmpty` (two first writers: one bootstrap lands, both then write through `If-Match`); `TestRevAndEpochNeverRepeatABody` (alternating writes of one entry: every ETag distinct; a body without `epoch`, as plan 3 writes, gets one); `TestReleaseAlwaysWrites` (an op that holds nothing still writes rev + 1). |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/moves -run 'EntryShape|OldParser|ActiveTable|Bootstrap|RevAndEpoch|ReleaseAlways' -count=1`. |
| +- [ ] **Step 3: Implement. Step 4: Run to verify they pass**, plus `go test ./internal/moves -run 'Offline' -count=1` (plan 2's parser tests green). |
| +- [ ] **Step 5: Commit** `moves: reservation kinds for creates and placements, and a table that is never deleted and never repeats a body`. |
| + |
| +### Task 2b: One conflict relation, `Reserve`, and recovery mode counting moves only |
| + |
| +**Files:** |
| +- Modify: `internal/moves/reserve.go` (`Conflicts`, `Reserve`, the errors), `internal/console/lifetimes_store.go` (`AllocationState(opID) (state string, found bool, err error)`), `internal/console/offlinereserve.go` (`reserveBoxes` reads through `moves.ReadActive` and writes `moves.NextActiveBody` with `If-Match`; its steps (4) and (5) become one call to `moves.Conflicts`; the wanted keys add `moves.HostKey(r.Host)`; `releaseBoxes` wraps `moves.Release`, so the `DeleteIfMatch` at line 478 goes; `offlineReclaimable`: `cleanup` → `reclaimHeld` always, `create` and `place` below; `cleanupReportEvidence` and `Server.cleanupReports` deleted), `internal/console/selfserve_pass.go` (`SelfServeEvidence`, lines 41-53, counts only entries of kind `move`) |
| +- Test: `internal/moves/reserve_conflict_test.go`, `internal/console/offlinereserve_hostkey_test.go` |
| + |
| +**Interfaces:** |
| +- `func Conflicts(table map[string]OfflineReservation, keys []string, mine Op, movesExclusive bool) map[Op]map[string]OfflineReservation` = the Global Constraint's relation, `movesExclusive` applying rule (c) only when `mine.Kind == move`. |
| +- `var ErrReservationMismatch`, `ErrUncertain`; `type ErrReserved struct { Key string; Holder OfflineReservation }`; `func Reserve(ctx context.Context, s release.Store, mine Op, want map[string]OfflineReservation) (map[string]OfflineReservation, error)` for the box, the create and the placement (N10: the caller names the entry for each key; every entry carries `mine` and is valid, else an error and nothing written): `ReadActive`; own entries on a key outside `want` → `ErrReservationMismatch`; own entries covering `want` → the held entries returned, nothing written; any conflict → `ErrReserved` naming the oldest conflicting entry (no reclaim on these paths); else one `Put` with `If-Match` the read ETag of `NextActiveBody` with the missing keys added (at the held entries' `ReservedAt` when partial); a CAS conflict re-reads once; a second loss → `ErrReserved` from the last read; a `Put` error that is not a CAS conflict → `ErrUncertain` wrapped: the caller keeps its state as if it might hold the keys, and the next successful write by anyone, its own retry or its release included, settles it. |
| - The console's `reserveBoxes`: `want` = an entry `r` for each of `sel.reservationKeys()` and for `moves.HostKey(r.Host)`; the own retry is unchanged (an own reservation made before this plan, without the host key, is partial and completed in the same write; an operator operation holding the sentinel covers every box key and has its host key completed); conflicts = `moves.Conflicts(table, keys, mine, s.cfg.SelfServeMoves != "offline")`; reclaim as today for `move`; `create`: `s.store.AllocationState(opID)` found and terminal → `reclaimYes`, pending → `reclaimHeld`, not found or an error → `reclaimUnreadable`; `place`: `reclaimHeld` until Task 7b replaces it with `s.settlePlacement`. |
| -- `s.residenceSweep(ctx, now)` (in each lifetime recovery iteration, `lifetimeRecoveryIteration`): a `cleanup` entry older than 6 h → `s.selfServePage(ctx, id, ssPageCleanupHeld, "a cleanup has held <host> for over six hours", body, now)`; a `create` entry older than `offlineLockGrace` with no allocation row → `ssPageCreateOrphan` (same body); Task 7b adds the placement settle. |
| -- [ ] **Step 1: Write the failing tests**: `TestConflictRelation` (every pair of kinds over box keys, host keys and the sentinel, with and without `movesExclusive`: two creates of different hosts never conflict; a create and a sentinel move of another host do not; a create and a move of the same host do; a cleanup and an operator move on disjoint boxes do not, with the switch off as on; two moves on disjoint boxes conflict only with the switch off); `TestEntryShapeRules` (a `place` or `create` entry without account or app, a host-key entry naming another host → malformed; a fixture parsed by plan 3's `ParseOfflineActive` accepts `move` and `cleanup` entries and ignores `epoch` and `rev`); `TestActiveTableIsNeverDeleted` (releasing the last entry writes an empty table; the console's `releaseBoxes` likewise; no `Delete` or `DeleteIfMatch` reaches the key); `TestBootstrapIsCreateOnlyAndEmpty` (two first writers: one bootstrap lands, both reserve through `If-Match`); `TestRevAndEpochNeverRepeatABody` (alternating reserves and releases of one entry: every ETag distinct; a body without `epoch`, as plan 3 writes, gets one); `TestUncertainReservationIsSettledByTheNextWrite` (review 4's interleaving: create C's `Put` is held by the fake store and answers a timeout; a cleanup acquires; C's recovery reads no entry of C, voids and calls `Release(C)`, which writes; the cleanup releases; C's held `Put` is then delivered and fails its condition; the table never holds C); `TestReleaseAlwaysWrites`; `TestReserveReturnsTheHeldEntry`; `TestReserveCompletesAPartialOwnReservation`; `TestReserveUncertainPutIsNotAnAnswer` (a store that commits and then errors → `ErrUncertain`; the own retry returns the held entries); `TestMoveSelectionsHoldTheHostKey` (a reservation written by plan 2's code, without the host key, is completed by its own retry; a cleanup on the host key makes the move busy); `TestReleaseIsTheConsolesAlgorithm` (plan 2's `releaseBoxes` tests green through the wrapper, with "deleted" read as "empty"); `TestCleanupEntriesAreNeverReclaimed`; `TestCreateEntryReclaimOnTheAllocationRow`; `TestRecoveryModeCountsMoveEntriesOnly` (cleanup, create and place entries alone, and an empty table, leave `SelfServeEvidence` false); `TestResidenceSweepPagesOldEntries`. |
| -- [ ] **Step 2: Run to verify they fail**: `go test ./internal/moves -run 'Conflict|EntryShape|ActiveTable|Bootstrap|RevAndEpoch|Uncertain|Reserve|Release' -count=1`, `go test ./internal/console -run 'HostKey|Reserve|Reclaim|Release|RecoveryMode|ResidenceSweep|OwnRetry|Sentinel|NeverDeleted' -count=1`. |
| +- [ ] **Step 1: Write the failing tests**: `TestConflictRelation` (every pair of kinds over box keys, host keys and the sentinel, with and without `movesExclusive`: two creates of different hosts never conflict; a create and a sentinel move of another host do not; a create and a move of the same host do; a cleanup and an operator move on disjoint boxes do not, with the switch off as on; two moves on disjoint boxes conflict only with the switch off); `TestUncertainReservationIsSettledByTheNextWrite` (review 4's interleaving: create C's `Put` is held by the fake store and answers a timeout; a cleanup acquires; C's recovery reads no entry of C, voids and calls `Release(C)`, which writes; the cleanup releases; C's held `Put` is then delivered and fails its condition; the table never holds C); `TestReserveReturnsTheHeldEntry`; `TestReserveCompletesAPartialOwnReservation`; `TestReserveUncertainPutIsNotAnAnswer` (a store that commits and then errors → `ErrUncertain`; the own retry returns the held entries); `TestMoveSelectionsHoldTheHostKey` (a reservation written by plan 2's code, without the host key, is completed by its own retry; a cleanup on the host key makes the move busy); `TestReleaseIsTheConsolesAlgorithm` (plan 2's `releaseBoxes` tests green through the wrapper, with "deleted" read as "empty"); `TestConsoleNeverDeletesTheTable`; `TestCleanupEntriesAreNeverReclaimed`; `TestCreateEntryReclaimOnTheAllocationRow`; `TestRecoveryModeCountsMoveEntriesOnly` (cleanup, create and place entries alone, and an empty table, leave `SelfServeEvidence` false). |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/moves -run 'Conflict|Uncertain|Reserve' -count=1`, `go test ./internal/console -run 'HostKey|Reserve|Reclaim|Release|RecoveryMode|OwnRetry|Sentinel|NeverDeletes' -count=1`. |
| +- [ ] **Step 3: Implement. Step 4: Run to verify they pass.** |
| +- [ ] **Step 5: Commit** `moves: one conflict relation shared by the console, the box, creates and placements; recovery mode counts moves only`. |
| + |
| +### Task 2c: The console's residence sweep |
| + |
| +**Files:** |
| +- Modify: `internal/console/lifetimes_recovery.go` (each `lifetimeRecoveryIteration`, line 41, also runs `s.residenceSweep(ctx, now)`), `internal/console/selfserve_pages.go` (the two page conditions), `internal/console/residence_sweep.go` (new: `residenceSweep`) |
| +- Test: `internal/console/residence_sweep_test.go` |
| + |
| +**Interfaces:** |
| +- `s.residenceSweep(ctx, now)`: one `moves.ReadActive`; a `cleanup` entry older than 6 h → `s.selfServePage(ctx, id, ssPageCleanupHeld, "a cleanup has held <host> for over six hours", body, now)` with the body naming the entry's id; a `create` entry older than `offlineLockGrace` with no allocation row → `ssPageCreateOrphan` (same body form). A read error is logged and the sweep waits for the next iteration. Task 7b adds the placement settle. |
| + |
| +- [ ] **Step 1: Write the failing tests**: `TestResidenceSweepPagesOldEntries` (a 6 h 1 m cleanup entry pages once, keyed by its id; a 5 h one does not; a create entry with no row past the grace pages; one with a pending row does not); `TestResidenceSweepRunsInEveryRecoveryIteration`; `TestResidenceSweepReadErrorChangesNothing`. |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/console -run 'ResidenceSweep' -count=1`. |
| - [ ] **Step 3: Implement. Step 4: Run to verify they pass.** |
| -- [ ] **Step 5: Commit** `moves: hostname keys in a reservation table that is never deleted and never repeats a body, with one conflict relation shared by the console, the box, creates and placements`. |
| +- [ ] **Step 5: Commit** `console: the residence sweep pages a cleanup held over six hours and a create entry without its allocation row`. |
| ### Task 3: The cleaning mark and the writer inventory |
| @@ -118,18 +153,18 @@ Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and |
| **Interfaces:** |
| - `type Away int` (`AwayYes`, `AwayFinishing`, `AwayMoved`, `AwayUnreadable`); `func RecordAway(obs provision.Observation, moveID string, incarnation uint64, from string) Away`: `LifetimeUnreadable`/`LifetimeCorrupt` → `AwayUnreadable`; absent, not `LifetimeValid`, `Rec.Retired` or `Rec.Incarnation != incarnation` → `AwayMoved`; `Rec.Move != nil` → `AwayFinishing` when `Move.ID == moveID`, else `AwayMoved`; `LastMove` nil, another id or not `done` → `AwayMoved`; `Rec.Region == from` → `AwayMoved`; else `AwayYes`. Root's authority (Task 6a) calls the same function. |
| -- `func (w *movesWorker) cleanupVisit(ctx context.Context, s release.Store, path string, now time.Time) error`: one obligation through the table below, each state change one `writeRetired`; Task 4c's runner calls it; the tests call it directly. The `cleaning` row's per-key step is `w.retireKey` (Task 4b); until 4b lands it is a field the tests set. |
| +- `func (w *movesWorker) cleanupVisit(ctx context.Context, s release.Store, path string, now time.Time) error`: one obligation through the table below, each state change one `writeRetired`; Task 4d's runner calls it; the tests call it directly. The `cleaning` row's per-key step is `w.retireKey` and its settle step is `w.settleInFlight` (Task 4c); until 4c lands they are fields the tests set. |
| - `func (w *movesWorker) supersededBy(ctx context.Context, s release.Store, r *offlinecleanup.Retired) (reason string, err error)` (F7: the one supersession check): observe (`moves.RecordAway(cfg.Judge.Observe(ctx, s, Account, App, Host), MoveID, Incarnation, From)`): `AwayUnreadable` → an error; `AwayMoved`, or `AwayFinishing` in any state but `open` → `record-moved`; then `w.offlineRow(Host)`: an error → an error; a row → `row-present`; else "". |
| - **The transition table** (first matching row; an error from a check → stay, event `refused/<what>`, next obligation): |
| 1. `open`: observe: `AwayFinishing` → stay (the original move's finishing has precedence; never supersession); `AwayMoved` → supersede; `AwayYes` → then the row check (a row → supersede) → `finished`, `FinishedAt = LastMove.At`. |
| - 2. `finished`: `supersededBy` → supersede; `now - FinishedAt < 24 h` or `!cfg.CleanupArmed` → stay; else `moves.Reserve(ctx, s, {cleanup, ID}, {cfg.BoxID: e, HostKey(Host): e})` with `e = {cleanup, ID, Host, Account, App, now}`: `ErrReserved` → stay, event `busy`; `ErrUnreadable` or uncertain → stay, event `refused/reservation-unreadable` (the next visit's own retry settles it); held → `ReservedAt` from the returned entry, every inventory key `pending`, `State = cleaning`. |
| + 2. `finished`: `supersededBy` → supersede; `!cfg.CleanupArmed` → with `ReserveUncertain` set, `moves.Release(ctx, s, {cleanup, ID})` (success clears it, event `released-unarmed`), else nothing; stay; `now - FinishedAt < 24 h` → stay; else `moves.Reserve(ctx, s, {cleanup, ID}, {cfg.BoxID: e, HostKey(Host): e})` with `e = {cleanup, ID, Host, Account, App, now}`: `ErrReserved` → stay, event `busy`, `ReserveUncertain` cleared (the table changed since any earlier write of ours was read, so that write can never land); `ErrUnreadable` → stay, event `refused/reservation-unreadable`; `ErrUncertain` → stay, `ReserveUncertain = true`, the same event (the next visit's own retry settles it, or, unarmed, its release); held → `ReservedAt` from the returned entry, `ReserveUncertain` cleared, every inventory key `pending` with `Done = 0`, `State = cleaning`. |
| 3. `cleaning`, `publishing`, `root-requested` (**ownership**, before anything else; rows 3 to 5 run in order within one visit): read `active.json`; `!moves.Holds(table, [BoxID, HostKey(Host)], {cleanup, ID})` → supersede, reason `reservation-lost` (an operator removed it; never re-acquired); a read error → stay. |
| 4. `cleaning` with `MarkedAt` zero: `MarkCleaning(..., TombstoneOwner, CleaningOwner(ID), cleanupMarkTTL, now)`: `ErrNotOurLock` → supersede, reason `lock-taken`; another error → stay; marked → `MarkedAt = now`. |
| - 5. `cleaning`, for each `pending` key in inventory order within the budget: `w.retireKey` (Task 4b), which answers the key's terminal outcome, a supersession reason, or stop. Every key terminal: `supersededBy` once more → supersede; else `LocalInventory` empty → `releasing`, `Outcome = cleaned`; else `publishing`. |
| + 5. `cleaning`: first, with `InFlight` set, `w.settleInFlight` (Task 4c); it answers done, stop, or `unsettled` (the mark is no longer ours). Then for each `pending` key in inventory order within the budget: `w.retireKey` (Task 4c), which answers the key's terminal outcome, a supersession reason, or stop. Every key terminal and `InFlight` nil: `supersededBy` once more → supersede; else `LocalInventory` empty → `releasing`, `Outcome = cleaned`; else `publishing`. |
| 6. `publishing`, `root-requested`, `cancel-pending`: Task 6b's rows; `publishing` and `root-requested` begin with `supersededBy` (a reason → supersede). |
| - 7. **Supersede**: from `publishing` or `root-requested` → `cancel-pending` (Task 6b), so the reservation and the mark stay until root has acknowledged; from `cancel-pending` nothing changes; from any other state → `releasing`, `Outcome = superseded` (from `open` and `finished` too: `Release` and `UnmarkCleaning` change nothing of ours when nothing is held, and a `finished` obligation may hold an entry its crash left). Event `superseded/<reason>`. |
| - 8. `releasing`: `now < QuietUntil` → stay (Task 4c keeps the mark); `UnmarkCleaning(..., CleaningOwner(ID), TombstoneOwner)` → error → stay; `moves.Release(ctx, s, {cleanup, ID})` → error → stay; → `ended`, `EndedAt = now`. The mark goes before the reservation, so a residence change admitted by the release never meets a mark, and no key is deleted after `releasing` is written. |
| - 9. `ended`: with `RepairKeys` non-empty and `RepairPasses == 0`, Task 4b's repair pass (then `RepairPasses = 1`); then Task 5a's publication and Task 8's reaper only. `corrupt`: Task 5a's hourly page only. |
| + 7. **Supersede**: from `publishing` or `root-requested` → `cancel-pending` (Task 6b), so the reservation and the mark stay until root has acknowledged; from `cancel-pending` nothing changes; from `cleaning` with `InFlight` set, the obligation stays `cleaning` until `w.settleInFlight` answers done or `unsettled` (with the reason kept in the event), and only then goes on; from any other state → `releasing`, `Outcome = superseded` (from `open` and `finished` too: `Release` and `UnmarkCleaning` change nothing of ours when nothing is held, and a `finished` obligation may hold an entry its crash left). Event `superseded/<reason>`. |
| + 8. `releasing`: `UnmarkCleaning(..., CleaningOwner(ID), TombstoneOwner)` → error → stay; `moves.Release(ctx, s, {cleanup, ID})` → error → stay; → `ended`, `EndedAt = now`. The mark goes before the reservation, so a residence change admitted by the release never meets a mark, and no version is deleted after `releasing` is written. |
| + 9. `ended`: Task 5a's publication and Task 8's reaper only. `corrupt`: Task 5a's hourly page only. |
| - **The outcome table** (F7; the only place `Outcome` is chosen; never `cleaned` otherwise): |
| | How the obligation reached `releasing` | `Outcome` | |
| @@ -142,44 +177,58 @@ Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and |
| | supersession from `open`, `finished` or `cleaning` (`record-moved`, `row-present`, `lock-taken`, `reservation-lost`) | `superseded` | |
| | `cancel-pending` resolved (any acknowledgement, or no request was ever taken) | `superseded` | |
| - Root's per-path results go into the report whatever the outcome (`Root`). |
| -- **Restart** resumes every state through the same table: `finished` re-runs `Reserve`, whose own retry returns the held entry; `cleaning` never reserves, and deletes only after rows 3 and 4 found both entries and the mark this cleanup's. |
| + Root's per-path results go into the report whatever the outcome (`Root`). A key `superseded` or `unsettled` is reported as such whatever the outcome. |
| +- **Restart** resumes every state through the same table: `finished` re-runs `Reserve`, whose own retry returns the held entry; `cleaning` never reserves, and deletes only after rows 3 and 4 found both entries and the mark this cleanup's; an `InFlight` delete found after a crash is answered first. |
| -- [ ] **Step 1: Write the failing tests** against a `DirStore` deployment bucket, the lease's S3 fake for `lock.json`, a real registry, an injected clock and a key step the test controls: `TestRecordAway` (each branch); `TestSupersessionIsCheckedAtEveryStep` (a row appearing and, separately, the record moving on, at: `open`, `finished`, between two keys, after the last key and before `publishing`, in `publishing` before the drop, in `root-requested`: each ends `superseded` through `releasing` or `cancel-pending`, never `cleaned`); `TestOutcomeTable` (each row, with root's answers from a fake acknowledgement reader); `TestCleanupSupersedesWhenTheRecordMovesOn` (a second move B to C, a recreation, a deletion, a placement back to `From`, each before the reservation: superseded, the lifted tombstone untouched, nothing held); `TestOriginalFinishingMoveIsNotSupersession`; `TestLifecycleResumesAtEveryBoundary` (Review Focus 2); `TestOwnershipLostIsSupersession` (an entry deleted by an operator; the mark replaced by a lease; each → `superseded`, no delete). |
| -- [ ] **Step 2: Run to verify they fail**: `go test ./internal/moves -run 'RecordAway' -count=1`, `go test ./internal/host -run 'SupersessionIsChecked|OutcomeTable|CleanupSupersedes|OriginalFinishing|LifecycleResumes|OwnershipLost' -count=1`. |
| +- [ ] **Step 1: Write the failing tests** against a `DirStore` deployment bucket, the lease's S3 fake for `lock.json`, a real registry, an injected clock and key and settle steps the test controls: `TestRecordAway` (each branch); `TestSupersessionIsCheckedAtEveryStep` (a row appearing and, separately, the record moving on, at: `open`, `finished`, between two keys, with a delete in flight, after the last key and before `publishing`, in `publishing` before the drop, in `root-requested`: each ends `superseded` through `releasing` or `cancel-pending`, never `cleaned`, and the in-flight delete is answered before `releasing`); `TestOutcomeTable` (each row, with root's answers from a fake acknowledgement reader); `TestCleanupSupersedesWhenTheRecordMovesOn` (a second move B to C, a recreation, a deletion, a placement back to `From`, each before the reservation: superseded, the lifted tombstone untouched, nothing held); `TestOriginalFinishingMoveIsNotSupersession`; `TestUnarmedUncertainReservationIsReleased` (an uncertain `Reserve`, then the switch off and a restart: the next visit releases, the table holds no entry of the cleanup, the obligation stays `finished`); `TestLifecycleResumesAtEveryBoundary` (Review Focus 2); `TestOwnershipLostIsSupersession` (an entry deleted by an operator; the mark replaced by a lease; each → `superseded`, no delete sent). |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/moves -run 'RecordAway' -count=1`, `go test ./internal/host -run 'SupersessionIsChecked|OutcomeTable|CleanupSupersedes|OriginalFinishing|UnarmedUncertain|LifecycleResumes|OwnershipLost' -count=1`. |
| - [ ] **Step 3: Implement. Step 4: Run to verify they pass.** |
| - [ ] **Step 5: Commit** `moves: the cleanup's persisted lifecycle, supersession checked at every step and one outcome table`. |
| -### Task 4b: The conditional deletion |
| +### Task 4b: The delete primitive and its dev gate |
| **Files:** |
| -- Create: `internal/host/movescleanup_delete.go` (`retireKey`) |
| -- Modify: `internal/lease/mirror.go` (`ConditionalMirrorDeleter`, `ErrDeleteConditionFailed`, `(*s3Mirror).DeleteIfMatch`), the versioned test fake the host tests use (it gains `DeleteIfMatch` with S3's semantics and a hook to hold a request and deliver it later) |
| -- Test: `internal/lease/mirror_deleteifmatch_test.go`, `internal/host/movescleanup_delete_test.go` |
| +- Create: `internal/host/movescleanup_delete.go` (`deleteInventoriedVersion`), `internal/host/movescleanup_delete_audit_test.go` |
| +- Modify: the versioned test fake the host tests use (it gains `DeleteVersion` with S3's versioned semantics, an attempt counter, and a hook that holds a request and delivers it later, after the caller has been answered by a later attempt) |
| +- Test: `internal/host/movescleanup_delete_test.go` |
| **Interfaces:** |
| -- `type ConditionalMirrorDeleter interface { DeleteIfMatch(ctx context.Context, bucket, key, etag string) error }`; `var ErrDeleteConditionFailed`. `(*s3Mirror).DeleteIfMatch`: an empty or malformed `etag` → an error without a request; `m.clientFor(bucket, m.src).DeleteObject(ctx, &awss3.DeleteObjectInput{Bucket: &bucket, Key: &key, IfMatch: aws.String("\"" + normalizeETag(etag) + "\"")})` (the one-line change against `Delete` at `mirror.go:787-800`); `PreconditionFailed` or `ConditionalRequestConflict` (412, 409, by code and by HTTP status, as `release.s3CASConflict`, `store.go:742-756`) → `ErrDeleteConditionFailed`; `NoSuchKey`/404 → `ErrMirrorNotFound`; anything else returned as it is. The cleanup asserts the interface on `cfg.PlainMirror`'s store; a store without it → event `refused/delete-unsupported`, stop, nothing deleted (never a fallback to `Delete`). |
| -- `func (w *movesWorker) retireKey(ctx context.Context, s release.Store, r *offlinecleanup.Retired, obj offlinecleanup.InventoryObject, mirror lease.ConditionalMirrorDeleter, vs lease.VersionedStore, now time.Time) (outcome string, superseded string, err error)`, in order: `supersededBy` (conditions (1) and (3)); the mark (condition (2): Task 4c reasserts first; then `lease.LockState` at the location: owner not ours, or under `cleanupMarkMargin` left → `lock-taken`); condition (4): `vs.ListVersions(Bucket, key)` filtered to `Key == key`: a marker current → `marked`; another version current → `superseded` (the skipped page, Task 5a); none → `absent`; the inventory's version current → `mirror.DeleteIfMatch(Bucket, key, obj.ETag)`: nil → `deleted` (the condition held at the server); `ErrDeleteConditionFailed` or `ErrMirrorNotFound` → list the key again and classify as above (the inventory's version still current, after a 409 race → stays `pending` for the next run); any other error → event `refused/delete-uncertain`, `QuietUntil` stays set (below), stop this obligation for this run. `Keys[key]` is written before the next key; a crash between the delete and the write finds a marker current next run → `marked`, the same end. A read error → event (`versions-unreadable`, `lock-unreadable`, `registry-unreadable`, `record-unreadable`), stop for this run. |
| -- **`QuietUntil`** (the byte-identical bound): before the first `DeleteIfMatch` of a run, one `writeRetired` sets `QuietUntil = runStart + cleanupRunDeadline + cleanupUncertainQuiet` and `RunKeys` to the pending keys this run may delete (at most 50; a crash after it leaves both set); at the end of a run in which every delete issued was answered (nil, 412, 409 or 404), both are cleared in the run's last state write; a run with any other answer leaves `QuietUntil` at that delete's issue time plus `cleanupUncertainQuiet`, if later, moves `RunKeys` into `RepairKeys` and sets `RepairSince` (once) to the run's start. A visit that finds `RunKeys` set from a crashed run does the same move first. Task 4a's row 8 waits for `QuietUntil`. |
| -- `func (w *movesWorker) repairKeys(ctx context.Context, r *offlinecleanup.Retired, vs lease.VersionedStore, now time.Time) error` (the **repair**, the Global Constraint's rule): per key of `RepairKeys`, `vs.ListVersions(Bucket, key)` filtered to `Key == key` (the current entry first, the rest newest first, `versioned.go:45-50`): the current entry a delete marker with `LastModified >= RepairSince`, the next entry not a marker, its `VersionID` not the inventory's, its `LastModified` after the inventoried version's and its `ETag` the inventory's → `vs.DeleteVersion(Bucket, key, marker.VersionID)` → event `restored` for the key; anything else → `done`, nothing touched. A listing or delete error → event `refused/repair-unreadable`, the pass stops and runs again next visit (`RepairPasses` unchanged). Called from Task 4a's row 9 (the first pass after `ended`) and from Task 8's reaper (the final pass, which then clears `RepairKeys`). |
| +- `func deleteInventoriedVersion(ctx context.Context, vs lease.VersionedStore, bucket, key string, v offlinecleanup.InventoryVersion) error`: `vs.DeleteVersion(ctx, bucket, key, v.VersionID)`, nothing else. Nil means the version is done (removed, or already gone: `DeleteVersion` maps `NoSuchVersion` and 404 to nil, `versioned.go:269-272`); any error means unanswered. **The cleanup's only bucket delete.** Should Paul answer question 9 no, this function is the one place that changes (to a marker-writing delete of the current version), and the plan's settlement rules would then need the review they had in rev 5; nothing else in this plan depends on how a version is retired. |
| +- **The audit** `TestCleanupDeletesOnlyThroughThePrimitive` (`go/ast` with `go/types` over the non-test files `internal/host/movescleanup*.go` and `movesreap.go`): no call of `Delete`, `DeleteIfMatch`, `DeleteObject` or `DeleteVersion` on any store except the one `DeleteVersion` inside `deleteInventoriedVersion`; and `deleteInventoriedVersion` has exactly one caller set (Task 4c's `retireKey` and `settleInFlight`). |
| -- [ ] **Step 1: Write the failing tests**: `TestMirrorDeleteIfMatchSendsIfMatch` (`s3Mirror` against an `httptest` S3 stub: the DELETE carries `If-Match: "<etag>"`; a 412 → `ErrDeleteConditionFailed`; a 409 → the same; a 404 → `ErrMirrorNotFound`; an empty ETag sends nothing); `TestDeletesAreConditionalOnTheInventoriedETag` (every delete the fake saw names the inventory's ETag; none is a plain `Delete` and none carries a version id); `TestUnconditionalDeleteIsNeverAFallback`; `TestFourConditionsAtEveryKey` (each failing between key 1 and key 2: stop on a read error, supersede on (1), (2), (3), the per-key outcomes of (4)); `TestDeleteAnswersAreClassifiedByRelisting` (412 with another version current → `superseded`; 412 with a marker current → `marked`; 404 → `absent` or `marked`; 409 with the version still current → `pending`; a timeout → stop, `QuietUntil` kept); `TestOutstandingDeleteCannotHideANewerVersion` (Review Focus 1: the fake holds a delete and times it out; a row appears; the obligation supersedes, waits out `QuietUntil`, unmarks and releases; a wake acquires and writes a new version with other bytes; the held delete is delivered: 412, the new version current, no marker); `TestByteIdenticalRewriteWaitsOutTheQuiet` (an uncertain delete, then supersession: `releasing` stays with the mark and the reservation held until `QuietUntil`; a crash after `QuietUntil` is set and before the delete keeps it set and moves `RunKeys` into `RepairKeys`); `TestLateDeleteOnAByteIdenticalVersionIsRepaired` (the witness: the fake holds a delete past `QuietUntil`; the obligation ends; a re-promotion writes a byte-identical version of the key; the held delete lands and puts a marker over it; the next visit's repair removes that marker by its version id, the re-promoted version is current again and the event `restored` is in the report; a key whose newest non-marker version is the inventoried one is `done` and untouched); `TestRepairNeverTouchesAnotherShape` (a marker over a version with another ETag, a marker older than `RepairSince`, a current non-marker version, a marker whose owner wrote again between the listing and the delete: nothing becomes current that was not the later owner's newest; a listing per repair key per pass, none for an obligation whose runs were all answered); `TestAdmittedReconcileAndWakeCannotWriteUnderTheMark` (Review Focus 1, with `ExecBackend.Acquire` over the fake); `TestLockAndBuildStateNeverDeleted`. |
| -- [ ] **Step 2: Run to verify they fail**: `go test ./internal/lease -run 'DeleteIfMatch' -count=1`, `go test ./internal/host -run 'DeletesAreConditional|UnconditionalDelete|FourConditions|DeleteAnswers|OutstandingDelete|ByteIdentical|LateDeleteOnAByteIdentical|RepairNever|AdmittedReconcile|LockAndBuildState' -count=1`. |
| -- [ ] **Step 3: Implement. Step 4: Run to verify they pass.** **Dev witness** (recorded in the commit message, before any arming): on the dev hibernate bucket, a test object, a `DeleteObject` with a stale `If-Match` → 412 and no marker; with the current ETag → a marker. |
| -- [ ] **Step 5: Commit** `lease, moves: every cleanup deletion is conditional at the server on the inventoried ETag, and a late delete that hit a byte-identical later version is repaired by removing its marker`. |
| +- [ ] **Step 1: Write the failing tests**: `TestDeleteNamesTheVersion` (the fake saw a version id on every delete, never a bare key and never `If-Match`; an empty version id sends nothing); `TestDeletingAVersionThatIsNotCurrentChangesNothingCurrent` (two versions: deleting the older leaves the newer current, no marker written); `TestDeletingTheLastVersionLeavesNoObject` (the key lists nothing; no marker); `TestDeletingAMarkerByItsID` (an inventoried marker removed, the version beneath it, also inventoried and already deleted, stays gone); `TestAGoneVersionIsDone` (`NoSuchVersion` → nil); `TestCleanupDeletesOnlyThroughThePrimitive`. |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/host -run 'DeleteNames|DeletingAVersion|DeletingTheLast|DeletingAMarker|AGoneVersion|DeletesOnlyThrough' -count=1`. |
| +- [ ] **Step 3: Implement. Step 4: Run to verify they pass.** **Dev witness** (recorded in the commit message, before any arming): on the dev hibernate bucket, a test key under a test prefix with versions v1 and v2: `DeleteObject` with `VersionId` v1 → v2 still current and no marker; with v2 → the key lists no version and no marker; v2 again → `NoSuchVersion` (the primitive answers nil); a write v3, then a `DeleteObject` of v2 → v3 current. The test key is removed by its version ids afterwards. |
| +- [ ] **Step 5: Commit** `moves: the cleanup's one delete names an inventoried version id; dev witness of versioned deletion`. |
| -### Task 4c: Scheduling and renewal, outside the pass lock |
| +### Task 4c: Retiring a key, oldest version first, and answering every delete |
| **Files:** |
| -- Modify: `internal/host/movespass.go` (`movesWorker` gains `cleanupMu sync.Mutex`; `moveCfg` gains `PlainMirror func(ctx context.Context, region string) (lease.MirrorStore, error)` and `CleanupArmed bool`, set in `armOffline` from `CARLOS_OFFLINE_CLEANUP`; `movesPass` ends with `w.kickCleanup(ctx, s)`), `internal/host/movescleanup.go` (`kickCleanup`, `cleanupRun`, the reassert) |
| +- Modify: `internal/host/movescleanup_delete.go` (`retireKey`, `settleInFlight`) |
| +- Test: `internal/host/movescleanup_retire_test.go` |
| + |
| +**Interfaces:** |
| +- `func (w *movesWorker) retireKey(ctx context.Context, s release.Store, r *offlinecleanup.Retired, k offlinecleanup.InventoryKey, vs lease.VersionedStore, now time.Time) (outcome string, superseded string, err error)`, in order: `supersededBy` (conditions (1) and (3)); the mark (condition (2): Task 4d reasserts first; then `lease.LockState` at the location: owner not ours, or under `cleanupMarkMargin` left → `lock-taken`); condition (4): `vs.ListVersions(Bucket, key)` filtered to `Key == key`: the current entry an object version the inventory does not hold → `superseded` (nothing sent; the skipped page, Task 5a); otherwise, for each version `i` from `Keys[key].Done` in deletion order: not in the listing → done (`Done = i + 1`); in it → one `writeRetired` with `InFlight = {key, versionID, now}` and `Done = i`, then `deleteInventoriedVersion`: nil → `Done = i + 1`, `InFlight = nil` (written with the next `InFlight` or the key's last write); an error → event `refused/delete-unanswered`, `InFlight` stays set, stop this obligation for this run. Every version done → `deleted`. A read error → event (`versions-unreadable`, `lock-unreadable`, `registry-unreadable`, `record-unreadable`), stop for this run. |
| +- `func (w *movesWorker) settleInFlight(ctx context.Context, s release.Store, r *offlinecleanup.Retired, vs lease.VersionedStore, now time.Time) (state string, err error)`: runs only after rows 3 and 4 of this visit found both entries and the mark this cleanup's, and re-checks the mark's owner (`lease.LockState`) just before sending: ours → `deleteInventoriedVersion` for `InFlight` again (the same version id, so any earlier attempt and this one remove one version): nil → that key's `Done` moves past it, `InFlight = nil`, answer done; an error → stop. Not ours, or row 3 or 4 superseded this visit → nothing is sent: the key `unsettled`, `InFlight = nil`, a `PendingPage` `unsettled-<i>` ("cleanup of <host> lost its mark before a delete of <key> was answered") in the same write, answer `unsettled`. |
| +- **Why the order is safe** (the Global Constraint's argument, applied): a key's latest inventoried version is sent only after every older one is done, and a version that is not current can be removed without changing what is current; so at every point of a run, and after any crash, the current entry of every key is one of: the inventoried current version, a later owner's version, a marker someone else wrote, or nothing. |
| + |
| +- [ ] **Step 1: Write the failing tests**: `TestFourConditionsAtEveryKey` (each failing between key 1 and key 2: stop on a read error, supersede on (1), (2), (3); (4) with a later object version current → `superseded`, nothing sent; with a foreign marker current → our versions beneath it deleted, the marker kept); `TestDeletionNeverExposesAnOlderVersion` (a key with three versions; a crash and an error after each delete: the current entry is never v1 or v2 while v3 is present); `TestUnansweredDeleteIsAnsweredBeforeTheUnmark` (Review Focus 1: the fake holds a delete and times it out; a row appears; the obligation stays `cleaning` and sends the same version id again under the mark; only after the answer does it write `releasing`, unmark and release; the held first attempt is delivered after a wake wrote a new version: nothing changes); `TestRetriedAttemptsRemoveOnlyTheirVersion` (two attempts of one delete, the first delivered after the second was answered and after a later owner wrote v4: v4 current); `TestLostMarkLeavesTheDeleteUnsettled` (the mark replaced while a delete is in flight: nothing more is sent, the key `unsettled`, the page owed in `pages_pending`, the obligation `superseded`); `TestPromotionThatKeptAnInventoriedVersionKeepsIt` (a cleanup superseded with a delete in flight, answered under the mark; after the release a promotion through `lease.MirrorPrefix` finds an equal ETag and keeps the inventoried version in place: no delete of the cleanup reaches it, and it is current after the run ends); `TestAdmittedReconcileAndWakeCannotWriteUnderTheMark` (Review Focus 1, with `ExecBackend.Acquire` over the fake); `TestLockAndBuildStateNeverDeleted`. |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/host -run 'FourConditions|DeletionNeverExposes|UnansweredDelete|RetriedAttempts|LostMark|PromotionThatKept|AdmittedReconcile|LockAndBuildState' -count=1`. |
| +- [ ] **Step 3: Implement. Step 4: Run to verify they pass.** |
| +- [ ] **Step 5: Commit** `moves: a key is retired by its inventoried versions oldest first, and every delete is answered under the mark before the cleanup gives it up`. |
| + |
| +### Task 4d: Scheduling and renewal, outside the pass lock |
| + |
| +**Files:** |
| +- Modify: `internal/host/movespass.go` (`movesWorker` gains `cleanupMu sync.Mutex`; `moveCfg` gains `CleanupArmed bool`, set in `armOffline` from `CARLOS_OFFLINE_CLEANUP`; `movesPass` ends with `w.kickCleanup(ctx, s)`), `internal/host/movescleanup.go` (`kickCleanup`, `cleanupRun`, the reassert) |
| - Test: `internal/host/movescleanup_run_test.go` |
| **Interfaces:** |
| - `func (w *movesWorker) kickCleanup(ctx context.Context, s release.Store)`: nothing unless `cfg.OfflineCapable && cfg.Judge != nil`; `w.cleanupMu.TryLock()` false → return; else a goroutine runs `w.cleanupRun(runCtx, s)` with `runCtx, cancel := context.WithTimeout(ctx, cleanupRunDeadline)` and unlocks when it returns. **Locks:** the goroutine holds `cleanupMu` only, never `w.pass` and never `w.mu` across I/O; `movesPass` holds `w.pass` while it only try-locks `cleanupMu`, so neither waits on the other and `offlineReassert` (`movesoffline.go:795`, which takes `w.pass`) is never behind a cleanup. |
| -- `cleanupRun`: lists `<dataDir>/.offline-moves/*/*/retired.json` and the corrupt stubs, starts after the path recorded in `<dataDir>/.offline-moves/.cleanup-cursor`, calls `cleanupVisit` on each obligation in path order until the deadline (each read, list, delete, lock call and report write under `context.WithTimeout(runCtx, 10*time.Second)`), at most 50 keys per obligation, records the cursor after each obligation, then runs publication (Task 5a) and the reaper (Task 8). One `PlainMirror` and one `Versioned` store per run. |
| -- **Renewal:** in `cleaning` (after `MarkedAt`), `publishing`, `root-requested`, `cancel-pending` and `releasing` before the unmark, every visit reasserts the mark when under 30 min remain (`ReassertCleaning`; `ErrNotOurLock` → supersede, reason `lock-taken`, which in `cancel-pending` and `releasing` changes nothing), so the mark never lapses while root, a cancel acknowledgement or `QuietUntil` is waited for. |
| +- `cleanupRun`: lists `<dataDir>/.offline-moves/*/*/retired.json` and the corrupt stubs, starts after the path recorded in `<dataDir>/.offline-moves/.cleanup-cursor`, calls `cleanupVisit` on each obligation in path order until the deadline (each read, list, delete, lock call and report write under `context.WithTimeout(runCtx, 10*time.Second)`), at most 50 keys per obligation, records the cursor after each obligation, then runs publication (Task 5a) and the reaper (Task 8). One `cfg.Versioned` store per bucket per run. |
| +- **Renewal:** in `cleaning` (after `MarkedAt`), `publishing`, `root-requested`, `cancel-pending` and `releasing` before the unmark, every visit reasserts the mark when under 30 min remain (`ReassertCleaning`; `ErrNotOurLock` → supersede, reason `lock-taken`, which in `cancel-pending` and `releasing` changes nothing), so the mark never lapses while root, a cancel acknowledgement or an unanswered delete is waited for. |
| -- [ ] **Step 1: Write the failing tests**: `TestCleanupRunsTwentyFourHoursAfterTheObservedFinish` (at 23 h 59 m nothing; at 24 h reserved, marked, deleted); `TestCleanupUnarmedLeavesObligationsFinished`; `TestCleanupRunsOutsideThePassLock` (a versioned fake that blocks: `offlineReassert` returns while the run is blocked; the run stops at its deadline; the cursor moves on; a second kick while one runs starts nothing); `TestMarkIsReassertedWhileWaiting` (root unanswered for 3 h, and `releasing` under `QuietUntil`: the mark never under 30 min); `TestRunBudgetAndCursor` (120 keys: 50 per run; three obligations and a deadline: each visited in turn across runs). |
| +- [ ] **Step 1: Write the failing tests**: `TestCleanupRunsTwentyFourHoursAfterTheObservedFinish` (at 23 h 59 m nothing; at 24 h reserved, marked, deleted); `TestCleanupUnarmedLeavesObligationsFinished`; `TestCleanupRunsOutsideThePassLock` (a versioned fake that blocks: `offlineReassert` returns while the run is blocked; the run stops at its deadline; the cursor moves on; a second kick while one runs starts nothing); `TestMarkIsReassertedWhileWaiting` (root unanswered for 3 h, and a delete unanswered across runs: the mark never under 30 min); `TestRunBudgetAndCursor` (120 keys: 50 per run; three obligations and a deadline: each visited in turn across runs). |
| - [ ] **Step 2: Run to verify they fail**: `go test ./internal/host -run 'CleanupRuns|CleanupUnarmed|MarkIsReasserted|RunBudget' -count=1`. |
| - [ ] **Step 3: Implement. Step 4: Run to verify they pass.** |
| - [ ] **Step 5: Commit** `moves: the cleanup runs in its own goroutine under its own lock, with a run budget, a cursor and a mark that never lapses while it waits`. |
| @@ -192,31 +241,32 @@ Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and |
| - Test: `internal/host/movescleanup_report_test.go`, `internal/console/offline_cleanup_evidence_test.go` |
| **Interfaces:** |
| -- `type Event struct { Seq uint64; Kind, Key, Detail string; At, LastAt time.Time; Count int }`. Kinds and their stable details: `finished`; `busy` (the holder's kind); `reserved`; `marked`; `restored` (`Key`; the repair removed a late marker); `refused` (`record-unreadable`, `registry-unreadable`, `lock-unreadable`, `versions-unreadable`, `delete-uncertain`, `delete-unsupported`, `repair-unreadable`, `reservation-unreadable`, `mark-failed`, `unmark-failed`, `release-failed`, `report-unwritten`); `skipped` (`Key`); `superseded` (`record-moved`, `row-present`, `lock-taken`, `reservation-lost`); `quiet` (waiting for `QuietUntil`); `root-requested`; `cancel-requested`; `root-done`; `root-kept` (`Key` the path's `Which`, `Detail` root's reason); `root-cancelled`; `ended` (the outcome). An event equal in `Kind`, `Key` and `Detail` to the last one increments its `Count` and `LastAt` instead of appending, so every distinct refusal is kept and a repeated one is counted (the raw error goes to the box log only). |
| -- `type Report struct { V int; CleanupID, MoveID, Host, Box, State, Outcome string; Keys map[string]KeyOutcome; Events []Event; Left []KeptPath; Root *RootCleanupAck; Rev uint64; At time.Time }` at `moves.CleanupReportKey(Host, MoveID, Box)`, a plain replacing `Put`; `ValidateReport(r Report, key string) error` (the id recomputes, the key is the one its fields name). |
| -- After each obligation's transitions in a run: `PublishedRev < Rev` → write the report from the file → on success `markPublished` (`PublishedRev = Rev`); a failure → event `refused/report-unwritten` (deduped; the event itself bumps `Rev`, so the next run publishes it too). |
| -- **Pages** through `w.firePage(ctx, Host, ID, rule, alerting.LevelWarning, subject, body)` (one page per `(host, op, rule)` in the outbox, `movespaging.go:232`). Periodic rules, fired from persisted times, so a lost one is followed by the next tick's: `root-wait-<n>` with `n = floor((now - RootRequestedAt) / 10 min)` for `n >= 1`; `cancel-wait-<n>` from `CancelRequestedAt`; `corrupt-<h>` with `h = floor((now - the stub's FoundAt) / 1 h)`; `unreadable-<h>` while the record has been unreadable for over an hour (from that event's `At`). **One-shot rules (N8):** `skipped-<i>` for inventory index `i` is written as a `PendingPage{Rule, Subject, At}` into `PagesPending` in the same `writeRetired` that makes key `i` `superseded`, so no state has the key terminal without its page owed. After the transitions, each pending page is fired: it leaves `PagesPending` when `firePage` answers sent, paged before, unwired (no pager on this box, logged as every moves page is), or pending with its entry on disk (`readPageEntry(pageEntryPath(...))` succeeds); it stays on undelivered and on pending held only in memory (`firePage`'s index, `movespaging.go:280-292`, which a restart loses). Each run then calls `w.flushPageOutboxFor(ctx, Host, ID)`, whatever `cfg.Enabled` says, so the outbox's other entries keep plan 6's rule (with the switch off they wait on disk). |
| +- `type Event struct { Seq uint64; Kind, Key, Detail string; At, LastAt time.Time; Count int }`. Kinds and their stable details: `finished`; `busy` (the holder's kind); `reserved`; `released-unarmed`; `marked`; `refused` (`record-unreadable`, `registry-unreadable`, `lock-unreadable`, `versions-unreadable`, `delete-unanswered`, `reservation-unreadable`, `mark-failed`, `unmark-failed`, `release-failed`, `report-unwritten`); `skipped` (`Key`); `unsettled` (`Key`); `superseded` (`record-moved`, `row-present`, `lock-taken`, `reservation-lost`); `root-requested`; `root-refused` (root has not answered and its request directory's ancestry is not root-only); `cancel-requested`; `root-done`; `root-kept` (`Key` the path's `Which`, `Detail` root's reason); `root-cancelled`; `ended` (the outcome). An event equal in `Kind`, `Key` and `Detail` to the last one increments its `Count` and `LastAt` instead of appending, so every distinct refusal is kept and a repeated one is counted (the raw error goes to the box log only). |
| +- `type Report struct { V int; CleanupID, MoveID, Host, Account, App, Box, State, Outcome string; Keys map[string]KeyOutcome; Events []Event; Left []KeptPath; Root *RootCleanupAck; Rev uint64; At time.Time }` at `moves.CleanupReportKey(Host, MoveID, Box)`, a plain replacing `Put`; `ValidateReport(r Report, key string) error` (the id recomputes, the key is the one its fields name). `Account` and `App` let the console's door keep a report to the app it was asked about (Task 5b). |
| +- After each obligation's transitions in a run: `PublishedRev < Rev` → write the report from the file → on success `markPublished` (`PublishedRev = Rev`); a failure → event `refused/report-unwritten` (deduped; the event itself bumps `Rev`, so the next run publishes it too). The bucket report is never removed by this plan, so it outlives the op directory and the record. |
| +- **Pages** through `w.firePage(ctx, Host, ID, rule, alerting.LevelWarning, subject, body)` (one page per `(host, op, rule)` in the outbox, `movespaging.go:232`), the body naming `--cleanup <ID>`. Periodic rules, fired from persisted times, so a lost one is followed by the next tick's: `root-wait-<n>` with `n = floor((now - RootRequestedAt) / 10 min)` for `n >= 1`, until `root-refused` applies; `root-refused-<h>` with `h = floor((now - RootRequestedAt) / 1 h)` once `h >= 1` and the agent's own `RootOnlyChain` check of root's request directory fails (Task 6b); `cancel-wait-<n>` from `CancelRequestedAt`; `corrupt-<h>` with `h = floor((now - the stub's FoundAt) / 1 h)`; `unreadable-<h>` while the record has been unreadable for over an hour (from that event's `At`). **One-shot rules (N8):** `skipped-<i>` for inventory index `i` is written as a `PendingPage{Rule, Subject, At}` into `PagesPending` in the same `writeRetired` that makes key `i` `superseded`, and `unsettled-<i>` in the one that makes it `unsettled`, so no state has the key terminal without its page owed. After the transitions, each pending page is fired: it leaves `PagesPending` when `firePage` answers sent, paged before, unwired (no pager on this box, logged as every moves page is), or pending with its entry on disk (`readPageEntry(pageEntryPath(...))` succeeds); it stays on undelivered and on pending held only in memory (`firePage`'s index, `movespaging.go:280-292`, which a restart loses). Each run then calls `w.flushPageOutboxFor(ctx, Host, ID)`, whatever `cfg.Enabled` says, so the outbox's other entries keep plan 6's rule (with the switch off they wait on disk). |
| -- [ ] **Step 1: Write the failing tests**: `TestCleanupReportIsDurableAndBlocksTheReaper` (Review Focus 5, with Task 8's reaper once it exists; until then the report half: a failing `Put` for three runs, then success: one report with every event and `PublishedRev == Rev`); `TestRefusalsAreStableDedupedEvents` (a read error every run for an hour is one event with its count; a different detail appends); `TestPendingPagesSurviveARestart` (a skipped key; the outbox cannot create the entry and the pager refuses; restart: the page is fired again from `PagesPending` and leaves it only once on disk or sent); `TestPagesFireOncePerTickAcrossRestarts` (root unanswered for 35 min across a restart at 15 min: pages `root-wait-1`, `-2`, `-3`, each once); `TestPageFlushIsScopedToTheCleanup` (`cfg.Enabled` false: the cleanup's undelivered page is retried, another move's entry is not touched); `TestReadOfflineEvidenceSkipsCleanupReports`. |
| +- [ ] **Step 1: Write the failing tests**: `TestCleanupReportIsDurableAndBlocksTheReaper` (Review Focus 5, with Task 8's reaper once it exists; until then the report half: a failing `Put` for three runs, then success: one report with every event and `PublishedRev == Rev`); `TestRefusalsAreStableDedupedEvents` (a read error every run for an hour is one event with its count; a different detail appends); `TestPendingPagesSurviveARestart` (a skipped key and an unsettled key; the outbox cannot create the entry and the pager refuses; restart: each page is fired again from `PagesPending` and leaves it only once on disk or sent); `TestPagesFireOncePerTickAcrossRestarts` (root unanswered for 35 min across a restart at 15 min: pages `root-wait-1`, `-2`, `-3`, each once; with the ancestry refused, `root-refused-1` at 1 h and `-2` at 2 h, and no `root-wait` after 1 h); `TestPageFlushIsScopedToTheCleanup` (`cfg.Enabled` false: the cleanup's undelivered page is retried, another move's entry is not touched); `TestReadOfflineEvidenceSkipsCleanupReports`. |
| - [ ] **Step 2: Run to verify they fail**: `go test ./internal/host -run 'CleanupReport|RefusalsAre|PendingPages|PagesFire|PageFlush' -count=1`, `go test ./internal/console -run 'SkipsCleanup' -count=1`. |
| - [ ] **Step 3: Implement. Step 4: Run to verify they pass.** |
| - [ ] **Step 5: Commit** `moves: the cleanup's report under the move id, republished until it lands, with stable de-duplicated events, and pages owed in retired.json until the outbox has them`. |
| -### Task 5b: The operator's log: `carlos move offline status --log` |
| +### Task 5b: The operator's log: `carlos move offline status --log`, by move or by cleanup |
| **Files:** |
| -- Modify: `internal/console/apiinstancemove_offline.go` (`offlineView` gains `Cleanup []offlineCleanupView \`json:"cleanup,omitempty"\``; `handleAPIInstanceOfflineStatus`, line 1950, calls `s.offlineCleanupOf(r.Context(), &v, d)` when `log=1`, before and independent of `offlineSelfServeOf`, which lives in `selfserve_events.go:168` and returns early when there is no self-serve view, `:172-176`), `move.go` (the `log` flag for `offline status`; `offlineView` gains `Cleanup` and `Events`; `runStatus`), `move_test.go`, `approved_moves_copy_test.go` (the new ids), `docs/superpowers/plans/2026-10-06-moves-phase1-copy.md` (the rows, pending), `docs/site/cli.md` (the `--log` line and paragraph, after line 2469) |
| -- Test: `internal/console/offline_cleanup_log_test.go`, `move_test.go` |
| +- Create: `internal/console/offline_cleanups.go` (`handleAPIInstanceOfflineCleanups`, `offlineCleanupsOf`) |
| +- Modify: `internal/console/authcli.go` (the route beside line 721), `move.go` (the `log`, `move` and `cleanup` flags for `offline status`; the decoded cleanup view; `runStatus`), `move_test.go`, `approved_moves_copy_test.go` (the new ids), `docs/superpowers/plans/2026-10-06-moves-phase1-copy.md` (the rows, pending), `docs/site/cli.md` (the `--log` line and paragraph, after line 2469) |
| +- Test: `internal/console/offline_cleanups_test.go`, `move_test.go` |
| **Interfaces:** |
| -- **The door** (`GET .../instances/{host}/move/offline?log=1`, the existing status door): `offlineCleanupOf` lists `moves.OfflineReportPrefix(host, id) + "cleanup/"` for `id` = the record's `LastMove.ID` (the record the gate read, `d.rec`), reads each key, keeps a report that decodes and passes `offlinecleanup.ValidateReport` for its key, and appends `offlineCleanupView{Box, State, Outcome, Keys []{Key, State}, Events []{Kind, Key, Detail, At, LastAt, Count}, Left []{Which, Reason}, Root *{Outcome, Removed, Absent []string; Kept []{Which, Reason}}, At}` (Unix seconds, as the view's other times); a listing or read error is logged and leaves `Cleanup` out, never failing the answer. Operator moves and legacy moves get it as self-serve moves do. |
| -- **The CLI** (`parseMoveArgs`, `move.go:138`, its flag switch at `:166-176`): `case "offline"` with `a.offline == "status"` adds `fs.BoolVar(&a.log, "log", false, "show the move's log, with the source box's cleanup")`; `runStatus` (`move.go:1507`) polls `w.m.path + "?log=1"` when set, and, after its lines, in both its `state == "none"` branch and the in-flight one, says per cleanup report: "Cleanup of <host> on <box> is <state>." or, ended, "Cleanup of <host> on <box> ended <outcome>."; per key not `deleted`: "Object <key> is <state>."; per left path: "<box> left <which> in place: <reason>." (the existing MV form); root's answer: "Root answered <outcome>: <n> removed, <n> already gone, <n> kept."; per event: "At <time>: <kind> <key> <detail>, <count> times until <time>." (count 1: "At <time>: <kind> <key> <detail>."); per self-serve event the door returns: "At <time>: <phase> <reason> <detail>."; no cleanup report with `--log`: "No cleanup has reported for move <id> yet." Times as the CLI says them today (`2006-01-02 15:04 UTC`). The decoded view's field names are kept apart from the tree's, as `offlineView`'s are, for the copy gate (`move.go:1092-1100`); event kinds, keys, details and reasons are box diagnostics, not copy. The exit code is unchanged by the log. |
| -- **Docs:** `docs/site/cli.md` gains `carlos move offline status --host <host> --log` in the verb list and one paragraph: what the log adds (the move's events and, a day after the move finished, the source box's cleanup of the old region's copy, object by object, what it left and why). Copy rows, pending. |
| +- **The door** `GET /api/cli/apps/{account}/{app}/instances/{host}/move/offline/cleanups[?move=<id>|?cleanup=<id>]`, `svcrole.Set{}`, operator only (`s.moveOperator`, a 404 otherwise, as `offlineGate`): the instance gate for the account and app and the host shape as `moveGate` checks them, and **no record read**, so a host whose record was deleted, or whose `LastMove` has moved on, still answers. `offlineCleanupsOf(ctx, acct, app, host, move, cleanup)` lists `control/offline-moves/<host>/` once and keeps the keys of the shape `<host>/<op>/cleanup/<box>.json`; with `move` set, only that `op`; it reads each kept key, keeps a report that decodes, passes `offlinecleanup.ValidateReport` for its key and names this `Account` and `App`, and, with `cleanup` set, only a report whose `CleanupID` matches. The answer is `{cleanup: [offlineCleanupView]}`, newest `At` first, with `offlineCleanupView{CleanupID, MoveID, Box, State, Outcome, Keys []{Key, State}, Events []{Kind, Key, Detail, At, LastAt, Count}, Left []{Which, Reason}, Root *{Outcome, Removed, Absent []string; Kept []{Which, Reason}}, At}` (Unix seconds, as the status view's other times). A listing error is a 502; a report that fails a read is logged and left out. A malformed `move` (not a plain segment) or `cleanup` (not `cu_` and 16 hex) is a 400. Operator moves and legacy moves get it as self-serve moves do. The status door (`handleAPIInstanceOfflineStatus`, line 1950) is unchanged. |
| +- **The CLI** (`parseMoveArgs`, `move.go:138`, its flag switch at `:166-176`): `case "offline"` with `a.offline == "status"` adds `fs.BoolVar(&a.log, "log", false, "show the move's log, with the source box's cleanup")`, `fs.StringVar(&a.logMove, "move", "", "with --log: the cleanup of this move")` and `fs.StringVar(&a.logCleanup, "cleanup", "", "with --log: the cleanup with this id")`; `--move` and `--cleanup` only with `--log` and not together (a usage error otherwise). `runStatus` (`move.go:1507`) polls `w.m.path + "?log=1"` when `--log` is set, as today, and then asks the cleanups door once with the filter, in both its `state == "none"` branch and the in-flight one; a 404 from the status door (the record is gone) with `--log` still asks the cleanups door and says its lines. Per report: "Cleanup <id> of <host> on <box> for move <move> is <state>." or, ended, "Cleanup <id> of <host> on <box> for move <move> ended <outcome>."; per key not `deleted`: "Object <key> is <state>."; per left path: "<box> left <which> in place: <reason>." (the existing MV form); root's answer: "Root answered <outcome>: <n> removed, <n> already gone, <n> kept."; per event: "At <time>: <kind> <key> <detail>, <count> times until <time>." (count 1: "At <time>: <kind> <key> <detail>."); per self-serve event the status door returns: "At <time>: <phase> <reason> <detail>."; no report: "No cleanup has reported for <host> yet." (with a filter: "No cleanup of <host> matches <id>."). Without a filter every cleanup of the host is said, newest first. Times as the CLI says them today (`2006-01-02 15:04 UTC`). The decoded view's field names are kept apart from the tree's, as `offlineView`'s are, for the copy gate (`move.go:1092-1100`); event kinds, keys, details and reasons are box diagnostics, not copy. The exit code is the status door's, unchanged by the log. |
| +- **Docs:** `docs/site/cli.md` gains `carlos move offline status --host <host> --log [--move <id> | --cleanup <id>]` in the verb list and one paragraph: what the log adds (the move's events and, a day after a move finished, the source box's cleanup of the old region's copy, object by object, what it left and why), and that `--cleanup` finds a cleanup a page names, even after a later move or a deleted instance. Copy rows, pending. |
| -- [ ] **Step 1: Write the failing tests**: `TestOfflineStatusLogListsCleanupWithoutASelfServeRow` (an operator move, no self-serve row: `?log=1` carries the report; a report at the wrong key or with another id is skipped; a listing error leaves `cleanup` out and answers 200); `TestOfflineStatusWithoutLogHasNoCleanup`; `TestMoveOfflineStatusLogFlag` (`--log` accepted only on `offline status`; the request carries `log=1`); `TestMoveOfflineStatusLogRendering` (state `none` and in flight; each line above; no report); `TestMovesStringsAreRecorded` and `TestMovesCopyRecordIsComplete` green with the new rows pending. |
| -- [ ] **Step 2: Run to verify they fail**: `go test ./internal/console -run 'StatusLog|StatusWithoutLog' -count=1`, `go test . -run 'MoveOfflineStatusLog|MovesStringsAreRecorded|MovesCopyRecord' -count=1`. |
| +- [ ] **Step 1: Write the failing tests**: `TestOfflineCleanupsDoorNeedsNoRecord` (an operator move, no self-serve row, and then the record deleted: the door still answers the report); `TestOlderCleanupIsFoundByItsID` (Review Focus 5: move A's cleanup is open, move B finishes and has its own report; `?cleanup=<A's id>` answers A's report only; no filter answers both, newest first; `?move=<A>` answers A's); `TestOfflineCleanupsDoorFilters` (a report at the wrong key, with another id, or naming another app is left out; a malformed filter → 400; a listing error → 502; a member's token → 404); `TestMoveOfflineStatusLogFlags` (`--log` accepted only on `offline status`; `--move` and `--cleanup` only with `--log` and not together; the requests carry `log=1` and the filter); `TestMoveOfflineStatusLogRendering` (state `none`, in flight, and a 404 status door; each line above; no report; no match); `TestMovesStringsAreRecorded` and `TestMovesCopyRecordIsComplete` green with the new rows pending. |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/console -run 'OfflineCleanups|OlderCleanup' -count=1`, `go test . -run 'MoveOfflineStatusLog|MovesStringsAreRecorded|MovesCopyRecord' -count=1`. |
| - [ ] **Step 3: Implement. Step 4: Run to verify they pass.** |
| -- [ ] **Step 5: Commit** `move: carlos move offline status --log shows the source box's cleanup report and events, for every kind of move`. |
| +- [ ] **Step 5: Commit** `move: carlos move offline status --log shows a host's cleanup reports, by move or by cleanup id, for every kind of move and after the record is gone`. |
| ### Task 6a: The registry host freeze and root's authority |
| @@ -234,45 +284,47 @@ Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and |
| - [ ] **Step 3: Implement. Step 4: Run to verify they pass.** |
| - [ ] **Step 5: Commit** `registry, root: a per-host registry freeze, and root's cleanup authority from the reservation, the record and the mark with the activator's own settings`. |
| -### Task 6b: The root-owned request and acknowledgement |
| +### Task 6b: The root-owned request and acknowledgement, as a stage of the adopt pass |
| **Files:** |
| -- Create: `internal/release/rootcleanup.go` (discovery, the copy, the dispatch, the acknowledgement; portable), `internal/release/rootcleanup_other.go` (answers every request `cancelled` with `Reason: "unsupported platform"`), `internal/host/movescleanup_root.go` (the agent's rows, `rootCleanupPass`) |
| -- Modify: `internal/host/agent.go` (`runAdoptUnderRootRelease`: `rootCleanupPass(...)` after `heldStagePass` at line 4932, only when `passLockErr == nil` (line 4655), nonfatal) |
| -- Test: `internal/release/rootcleanup_test.go`, `internal/host/movescleanup_root_test.go` |
| +- Create: `internal/release/rootcleanup.go` (discovery, the copy, the dispatch, the acknowledgement, `LockRootPass`; portable), `internal/release/rootcleanup_other.go` (answers every request `cancelled` with `Reason: "unsupported platform"`; `LockRootPass` answers `ErrNotRootOnly`), `internal/host/movescleanup_root.go` (the agent's rows, `rootCleanupPass`) |
| +- Modify: `internal/host/agent.go` (`runAdoptUnderRootRelease`: the root pass lock right after `lockAdoptPass` at line 4655, in every mode; `rootCleanupPass(...)` after `heldStagePass` at line 4932), `internal/host/adoptlock.go` (the lock-order comment at lines 52-54 names the root pass lock between the adopt-pass lock and the restart marks), `internal/host/movesheldlock.go` (the order comment, lines 26-31), `internal/host/claimcensus.go` (`claimOptCensus` gains `<optDir>/root-cleanup` as box state, if its completeness test requires it) |
| +- Test: `internal/release/rootcleanup_test.go`, `internal/host/movescleanup_root_test.go`, `internal/host/adopt_rootcleanup_stage_test.go` |
| **Interfaces:** |
| -- **Files.** The agent's drop `<OfflineOpDir>/root-cleanup.json` (create-only) and cancel `<OfflineOpDir>/root-cleanup.cancel` (create-only). Root's directory `release.RootCleanupDir(optDir, id) = <optDir>/root-cleanup/<id>/` holding `request.json` (root's copy), `journal` (Task 6c) and exactly one of `done.json`, `cancelled.json`. `<optDir>/root-cleanup/` and each `<id>/` are created by root `0755`; root uses one only when `offlinecleanup.RootOnlyChain` passes on it (else root removes nothing for that request and writes its acknowledgement there with every path kept, "writable layout", which `ReadRootCleanupAck` then reports as untrusted). |
| +- **The root pass lock (R5, F5).** `func LockRootPass(optDir string) (unlock func(), err error)`: `RootCleanupBase(optDir) = <optDir>/root-cleanup/` is created by root `0755` when absent; `offlinecleanup.RootOnlyChain` on it false → `ErrNotRootOnly`, no lock (root's step then removes nothing on this box: a writable `/opt` is a writable layout); else `pass.lock` in it opened with `safefile.OpenFile(O_CREATE|O_RDWR, 0o600)` and `lockFileEx` (blocking `flock`), as `lockExclusive` does. In `runAdoptUnderRootRelease`, right after `lockAdoptPass` (line 4655) and whatever that answered, in every mode (hourly, `-stage-only`, `-config-only`, `-restart-only`), before `recoverRootRestarts` and every other publisher: `unlockRoot, rootLockErr := release.LockRootPass(release.OptDir())`, deferred, logged and never fatal, as the adopt-pass lock's failure is. So a pass whose adopt-pass lock failed (a symlink planted at `/data/adopt-pass.lock`, which `safefile` refuses) still waits on the root pass lock before it can publish. **Why the step does not depend on the adopt-pass lock's identity:** that lock lives in the agent-writable `/data` (`adoptlock.go:16`), where a shared-uid writer can replace or plant its name; the root pass lock lives on a root-only chain, which no other uid can change, so its acquisition is unchanged and needs no identity check. The adopt-pass lock is still required for the step (it is held, so the sweep keeps its own rule), but nothing relies on its identity. The one case left is a pass whose root pass lock fails for a root-level fault (an I/O error on a root-only directory, which no other uid can cause): that pass runs no root step, but it is not excluded from another pass's step; see the Self-review notes. |
| +- **The stage (S12).** `rootCleanupPass(...)` runs after `heldStagePass` (line 4932), so after `release.Pass`, inside which named recovery runs (`recovery.passOption`, `agent.go:4890`, `namedrecovery_pass.go:15`), and only when `passLockErr == nil && rootLockErr == nil`; nonfatal. `-config-only` and `-restart-only` return before it (`agent.go:4758-4767`, `:4794-4801`) but have taken the root pass lock, so they wait for a running step and a running step waits for them. Inside one pass the stages run one after another, so no root publisher of this pass runs while root's step runs, and every other adopt pass is blocked on the root pass lock. |
| +- **Files.** The agent's drop `<OfflineOpDir>/root-cleanup.json` (create-only) and cancel `<OfflineOpDir>/root-cleanup.cancel` (create-only). Root's directory `release.RootCleanupDir(optDir, id) = <optDir>/root-cleanup/<id>/` holding `request.json` (root's copy), `journal` (Task 6c) and exactly one of `done.json`, `cancelled.json`. Each `<id>/` is created by root `0755`; root uses `RootCleanupBase` and `<id>/` only when `offlinecleanup.RootOnlyChain` passes on them. **When it does not (S16), root writes nothing there**, removes nothing, and leaves the drop where it is: it logs `root-cleanup: <id> refused: <dir> is not root-only; repair its ownership, or remove <drop> to give up` to the adopt unit's journal (`log.Printf`, the systemd journal of `carlos-adopt.service`), which is the only record of the refusal. A repaired ancestry lets the next pass take the request. |
| - The wire types (declared in Task 1): `type RootCleanupRequest struct { V int; CleanupID, MoveID, Host, Account, App, Box string; Incarnation uint64; From, To string; Bucket, BucketRegion, Prefix string; Paths []RootCleanupPath; WrittenAt time.Time }`; `type RootCleanupPath struct { Which string; Identity FileIdentity }`; `type RootCleanupAck struct { V int; CleanupID, MoveID, Host, Outcome string (done or cancelled); Removed, Absent []string; Kept []KeptPath; Reason string; At time.Time }`. |
| -- `type RootCleanupOptions struct { DataDir, OptDir, Box, Region string; Authorize func(ctx context.Context, req offlinecleanup.RootCleanupRequest) error; Freeze func(host string, fn func(rowPresent bool) error) error; HostLock func(host string, fn func() error) error; Processes func() ([]string, error); Now func() time.Time }`; `var ErrRootCleanupRefused`; `func ServeRootCleanupRequests(ctx context.Context, o RootCleanupOptions) error`. |
| +- `type RootCleanupOptions struct { DataDir, OptDir, Box, Region string; Authorize func(ctx context.Context, req offlinecleanup.RootCleanupRequest) error; Freeze func(host string, fn func(rowPresent bool) error) error; HostLock func(host string, fn func() error) error; Processes func() ([]string, error); Now func() time.Time }`; `var ErrRootCleanupRefused`, `ErrNotRootOnly`; `func ServeRootCleanupRequests(ctx context.Context, o RootCleanupOptions) error`. |
| - **Discovery (F6):** each pass walks, with `safefile` (no symlinks), (i) the drops `<DataDir>/.offline-moves/<host>/<move>/root-cleanup.json` (host passing `ValidHostShape`, move a plain segment; `id = CleanupID(o.Box, host, move)`) and (ii) root's own `<OptDir>/root-cleanup/<id>/` directories that hold `request.json` and no acknowledgement, so a request root took and then crashed on is served from its own copy whether or not a drop remains. |
| -- **Root, per drop:** (1) read bounded (4096 bytes) and validate: `Host`, `MoveID` equal the path; `Box == o.Box`; `CleanupID == id`; `From == o.Region`; `Which` values known and unique. Invalid → `cancelled` under `id` with `Reason: "malformed request"`, the drop removed. (2) Copy: `request.json` create-only (temp, `fsync`, `os.Link`, directory `fsync`) with the drop's bytes; the journal's first line `taken <at>` (`fsync`); then the drop is removed. A `request.json` already present with other bytes: the drop is removed and logged, root's copy stands (a later drop never changes or cancels a taken request). From here root reads only its copy. |
| +- **Root, per drop:** (0) `RootOnlyChain` on `RootCleanupBase` and, when it exists, `RootCleanupDir(id)`: false → the S16 refusal above, next drop. (1) Read bounded (4096 bytes) and validate: `Host`, `MoveID` equal the path; `Box == o.Box`; `CleanupID == id`; `From == o.Region`; `Which` values known and unique. Invalid → `cancelled` under `id` with `Reason: "malformed request"`, the drop removed. (2) Copy: `request.json` create-only (temp, `fsync`, `os.Link`, directory `fsync`) with the drop's bytes; the journal's first line `taken <at>` (`fsync`); then the drop is removed. A `request.json` already present with other bytes: the drop is removed and logged, root's copy stands (a later drop never changes or cancels a taken request). From here root reads only its copy. |
| - **Root, per taken request** (from (2) or from discovery (ii)): Task 6c's journal recovery; `o.Authorize(ctx, req)`: `ErrRootCleanupRefused` → `cancelled` with its reason, nothing touched; another error → the next pass; each path not yet terminal through Task 6c's step; every path terminal → `cancelled.json` when the authority refused or any path was kept for "cancelled", else `done.json` (temp, `fsync`, rename, directory `fsync`). `<optDir>/root-cleanup/<id>/` older than 30 days past its acknowledgement is removed by root. |
| - `func ReadRootCleanupAck(optDir, id string) (ack *offlinecleanup.RootCleanupAck, trusted bool, err error)`: `trusted` only when `RootOnlyChain(RootCleanupDir(optDir, id))` passes and the file, opened with `safefile` (no symlinks), is owned by uid 0 with no group or other write bit; an acknowledgement the agent cannot trust is advisory (Task 4a's outcome table: `superseded`, every requested path recorded kept "writable layout"). |
| - `func rootCleanupPass(ctx context.Context, dataDir, optDir string, reg *registry.Registry, store release.Store, judge *provision.LifetimeJudge, box, region string)`: `ServeRootCleanupRequests` with `Authorize` = Task 6a's callback, `Freeze` = `reg.WithHostFrozen` (a non-nil row is `rowPresent`), `HostLock` = `withHeldStageLock(dataDir, ·, ·)`, `Processes` reading `/proc/*/exe` (`ENOENT`/`ESRCH` skipped as exited, any other error returned). |
| -- **The agent's rows** (Task 4a's row 6): `publishing`: `supersededBy` → supersede; the drop absent and `RootCleanupDir` absent → `writeRootCleanupRequest` (create-only; an existing drop with equal bytes is fine) → touch the stage-request marker (`holdStageRequester(dataDir)(Host)`, `moveshold.go:277`) → `RootRequestedAt = now`, `root-requested`, event; either present (a crash after the drop) → `root-requested` with `RootRequestedAt` restored, in this order: the drop's mtime when the drop exists; else the `at` of root's journal `taken` line; else `now`. `root-requested`: `supersededBy` → supersede (to `cancel-pending`); `ReadRootCleanupAck(optDir, ID)` → an acknowledgement whose `CleanupID`, `Host`, `MoveID` match → `Root = ack`, event, `releasing` with the outcome table's answer; a mismatch → page `corrupt-<h>`, stay; none → the `root-wait` pages. `cancel-pending`: the cancel file written create-only and `CancelRequestedAt` set (once); an acknowledgement → `releasing`, `Outcome = superseded`; else the drop is read **first** and then `RootCleanupDir`: both absent → `releasing`, `Outcome = superseded` (root takes a request into its directory before it removes the drop, so a drop already gone means root's directory exists, and no drop was written after `publishing`); else the `cancel-wait` pages. |
| +- **The agent's rows** (Task 4a's row 6): `publishing`: `supersededBy` → supersede; the drop absent and `RootCleanupDir` absent → `writeRootCleanupRequest` (create-only; an existing drop with equal bytes is fine) → touch the stage-request marker (`holdStageRequester(dataDir)(Host)`, `moveshold.go:277`) → `RootRequestedAt = now`, `root-requested`, event; either present (a crash after the drop) → `root-requested` with `RootRequestedAt` restored, in this order: the drop's mtime when the drop exists; else the `at` of root's journal `taken` line; else `now`. `root-requested`: `supersededBy` → supersede (to `cancel-pending`); `ReadRootCleanupAck(optDir, ID)` → an acknowledgement whose `CleanupID`, `Host`, `MoveID` match → `Root = ack`, event, `releasing` with the outcome table's answer; a mismatch → page `corrupt-<h>`, stay; none → the drop is read **first** and then `RootCleanupDir`: both absent (an operator removed the drop before root took it; root takes a request into its directory before it removes the drop) → `releasing`, `Outcome = superseded`; else, after `rootRefusedAfter` (1 h) since `RootRequestedAt` with the agent's own `RootOnlyChain` check of `RootCleanupBase` (or of `RootCleanupDir` when it exists) failing → event `root-refused` and the hourly `root-refused-<h>` page, stay (**never a release**; the mark is reasserted and the entries stay until an operator repairs the ancestry or removes the drop); else the `root-wait` pages. `cancel-pending`: the cancel file written create-only and `CancelRequestedAt` set (once); an acknowledgement → `releasing`, `Outcome = superseded`; else the drop is read **first** and then `RootCleanupDir`: both absent → `releasing`, `Outcome = superseded` (root takes a request into its directory before it removes the drop, so a drop already gone means root's directory exists, and no drop was written after `publishing`); else the `cancel-wait` pages. |
| -- [ ] **Step 1: Write the failing tests**: `TestRootFindsItsOwnRequestsOnRestart` (root takes a request, removes the drop and stops: the next pass serves it from `request.json`); `TestForgedAgentFilesAreHarmless` (a `done.json` planted in the agent's directory is never read; a drop with another id is cancelled under the expected id; a second drop with other bytes after the take is removed and changes nothing; a forged cancel stops root and ends the obligation `cancelled`, nothing else); `TestRootDirOwnershipIsVerified` (a root directory not passing the chain check is not used); `TestAgentReadsOnlyRootsAcknowledgement` (an untrusted acknowledgement → `superseded`, paths kept "writable layout"); `TestRootRequestedAtIsRestored` (a crash after the drop: from its mtime; after the take: from the `taken` line); `TestUnansweredRootRequestPagesEveryTenMinutes`; `TestCancelBeforeAndDuringRootsWork`; `TestCancelPendingResolvesWithoutARequest` (a crash after `publishing` before the drop; root never wrote a directory); `TestRootCleanupNeedsTheAdoptLock`; `TestUnsupportedPlatformCancels`. |
| -- [ ] **Step 2: Run to verify they fail**: `go test ./internal/release -run 'RootFinds|ForgedAgent|RootDir|CancelBefore|Unsupported' -count=1`, `go test ./internal/host -run 'AgentReadsOnly|RootRequestedAt|Unanswered|CancelPending|RootCleanupNeeds' -count=1`. |
| +- [ ] **Step 1: Write the failing tests**: `TestRootCleanupStepRunsInsideThePass` (a pass driven through `runAdoptUnderRootRelease`'s stages with fakes: the step runs after `release.Pass` and `heldStagePass`; with the adopt-pass lock failed, with the root pass lock failed, and on `-config-only` and `-restart-only`, no step; every mode takes the root pass lock before its first publisher); `TestRootPassLockIsRootOnly` (a base directory the test uid owns → `ErrNotRootOnly`, no lock; the lock's chain check runs as any uid against an injected chain check); `TestRootFindsItsOwnRequestsOnRestart` (root takes a request, removes the drop and stops: the next pass serves it from `request.json`); `TestForgedAgentFilesAreHarmless` (a `done.json` planted in the agent's directory is never read; a drop with another id is cancelled under the expected id; a second drop with other bytes after the take is removed and changes nothing; a forged cancel stops root and ends the obligation `cancelled`, nothing else); `TestRejectedAncestryGetsNoWrite` (a `RootCleanupBase` or `<id>/` that fails the chain check: nothing created, written or removed under it, the drop kept, one log line; repaired, the next pass takes it); `TestAgentPagesARefusedAncestryAndNeverReleases` (no answer for 1 h with the chain failing from the agent's view: hourly `root-refused` pages, the obligation stays `root-requested` with both entries and the mark for a simulated week; the drop removed by an operator → `releasing`, `superseded`); `TestAgentReadsOnlyRootsAcknowledgement` (an untrusted acknowledgement → `superseded`, paths kept "writable layout"); `TestRootRequestedAtIsRestored` (a crash after the drop: from its mtime; after the take: from the `taken` line); `TestUnansweredRootRequestPagesEveryTenMinutes`; `TestCancelBeforeAndDuringRootsWork`; `TestCancelPendingResolvesWithoutARequest` (a crash after `publishing` before the drop; root never wrote a directory); `TestUnsupportedPlatformCancels`. |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/release -run 'RootFinds|ForgedAgent|RejectedAncestry|RootPassLock|CancelBefore|Unsupported' -count=1`, `go test ./internal/host -run 'RootCleanupStep|AgentPages|AgentReadsOnly|RootRequestedAt|Unanswered|CancelPending' -count=1`. |
| - [ ] **Step 3: Implement** (until Task 6c lands, the per-path step answers kept with reason "not built", so this commit removes nothing). **Step 4: Run to verify they pass**, then `GOOS=darwin go build ./internal/release/ ./internal/host/` and `GOOS=windows go vet ./internal/release/ ./internal/host/`. |
| -- [ ] **Step 5: Commit** `root: takes a cleanup request into its own directory, finds it there again after a crash, and answers in a file the agent trusts only on a root-only chain`. |
| +- [ ] **Step 5: Commit** `root: a stage of the adopt pass under a root-only pass lock takes a cleanup request into its own directory, writes nothing where the ancestry is not root's, and answers in a file the agent trusts only on a root-only chain`. |
| ### Task 6c: Root's filesystem step, its journal and recovery |
| **Files:** |
| - Create: `internal/release/rootcleanup_linux.go` (`fstatat`, `readlinkat`, `unlinkat` through `golang.org/x/sys/unix` on a parent opened with `O_DIRECTORY|O_NOFOLLOW` through `safefile.OpenDir`), `internal/release/rootcleanup_journal.go`, `internal/host/livelink_writers_inventory_test.go` |
| -- Test: `internal/release/rootcleanup_linux_test.go`, `internal/release/rootcleanup_journal_test.go` |
| +- Test: `internal/release/rootcleanup_linux_test.go`, `internal/release/rootcleanup_journal_test.go`, `internal/host/rootcleanup_exclusion_test.go` |
| **Interfaces:** |
| - **The journal** `<RootCleanupDir>/journal`: append-only JSON lines, each `fsync`ed before the next syscall: `{at, step: "taken"}`; per path `{at, which, step: "intent", identity}` then exactly one of `{at, which, step: "removed"}`, `{at, which, step: "kept", reason}`, `{at, which, step: "absent"}`. A path's result is its last result line; the acknowledgement is built from results alone, so no outcome is unknown. |
| - **Recovery (F6),** per path whose last line is `intent`: `fstatat` the name in the opened parent again: absent → journal `absent` (removed by us before the crash or by anyone: recorded as absent, never as removed); present with the recorded identity → the step is retried from the start below; present with another identity → journal `kept`, "identity changed". Nothing was ever renamed, so nothing needs restoring after a cancel or a refusal. |
| -- **The step,** per path not yet terminal (`live-link` first; root derives the path: `LiveLink(OptDir, host)`, `ProgramsPath(DataDir, host)`): for the live link only, outside the freeze: the build directory it targets; another link under `<OptDir>/live/` resolving to the same build → `kept`, "the build is still referenced"; `o.Processes()` naming an executable under the build → `kept`, "a process runs the build"; `Processes` failing → the next pass (unknown is never unreferenced). The programs file has no reference check: its schema (`{schema, programs}`, Go main-package paths, `programhistory.go:28-34`) belongs to this host alone and no reader resolves another host through it. Then `o.Authorize` again (the same two answers). Then `RootOnlyChain(parent)` false → `kept`, "writable layout". Then `o.Freeze(host, func(rowPresent) { rowPresent → kept, "the instance has a row on this box"; o.HostLock(host, func() { root-cleanup.cancel present (Lstat) → kept, "cancelled"; open the parent (O_DIRECTORY|O_NOFOLLOW) and fstat it: not uid 0 or group/other writable → kept, "writable layout"; journal intent; fstatat(parent, name, AT_SYMLINK_NOFOLLOW): ENOENT → journal absent; identity (dev, inode, file type, readlinkat target) differs → journal kept, "identity changed"; equal → unlinkat(parent, name, 0): ENOENT → journal absent; success → fsync the parent, journal removed }) })`. Between `fstatat` and `unlinkat` only root can change an entry of a root-only directory, and every root writer of these names is excluded (below). |
| -- **The writers of these names** (`TestRootOnlyLiveLinkWritersAreExcluded`, `go/ast` with `go/types` over non-test files under `./internal/...`): every call of `release.PointLive`, `release.RecordProgram` and `(*activator.ExecBackend).ensureLiveLink` is listed with what keeps it off the name during root's step: `adopt pass` (it runs in the root adopt pass, which runs root's step after it in the same process under the adopt pass lock: `adopt.go`, `rootrollback_resume.go:125`, `roothistory_recovery.go:216`, and the named-unit recovery sites `namedrecovery.go:120`, `namedrecovery_active.go:191` once their caller is confirmed), `held-stage lock` (held publication), `needs a row` (the freeze), `agent process` (the agent cannot write a root-only directory: `ensureLiveLink`, the agent's own adopt pass at `agent.go:1319`) or `e2e only`; a new site, or a removed one, fails until the list is edited, and a site the implementer cannot place in a class is a question for the reviewer, not a guess. |
| +- **The step,** per path not yet terminal (`live-link` first; root derives the path: `LiveLink(OptDir, host)`, `ProgramsPath(DataDir, host)`): for the live link only, outside the freeze: the build directory it targets; another link under `<OptDir>/live/` resolving to the same build → `kept`, "the build is still referenced"; `o.Processes()` naming an executable under the build → `kept`, "a process runs the build"; `Processes` failing → the next pass (unknown is never unreferenced). The programs file has no reference check: its schema (`{schema, programs}`, Go main-package paths, `programhistory.go:28-34`) belongs to this host alone and no reader resolves another host through it. Then `o.Authorize` again (the same two answers). Then `RootOnlyChain(parent)` false → `kept`, "writable layout". Then `o.Freeze(host, func(rowPresent) { rowPresent → kept, "the instance has a row on this box"; o.HostLock(host, func() { root-cleanup.cancel present (Lstat) → kept, "cancelled"; open the parent (O_DIRECTORY|O_NOFOLLOW) and fstat it: not uid 0 or group/other writable → kept, "writable layout"; journal intent; fstatat(parent, name, AT_SYMLINK_NOFOLLOW): ENOENT → journal absent; identity (dev, inode, file type, readlinkat target) differs → journal kept, "identity changed"; equal → unlinkat(parent, name, 0): ENOENT → journal absent; success → fsync the parent, journal removed }) })`. Between `fstatat` and `unlinkat` only root can change an entry of a root-only directory, and every root writer of these names is excluded: the root adoption publishers by the pass (this step is a stage of it, and every other adopt pass waits on the root pass lock), held publication by the held-stage lock, and anything that needs a row by the freeze. |
| +- **The writers of these names** (`TestRootOnlyLiveLinkWritersAreExcluded`, `go/ast` with `go/types` over non-test files under `./internal/...`): every call of `release.PointLive`, `release.RecordProgram` and `(*activator.ExecBackend).ensureLiveLink` is listed with what keeps it off the name during root's step: `root pass lock` (reached only from `runAdoptUnderRootRelease`, which holds the root pass lock for its whole run and runs root's step as a later stage: `adopt.go:1751`, `:1870`, `rollback.go:124`, `rootrollback_resume.go:125`, `roothistory_recovery.go:216`, and named recovery's `namedrecovery.go:120` and `namedrecovery_active.go:191` through `recovery.passOption` inside `release.Pass`, `agent.go:4890`, `namedrecovery_pass.go:15`), `held-stage lock` (held publication: `materializeheld.go:460`, `:494`), `needs a row` (the freeze), `agent process` (the agent cannot write a root-only directory: `ensureLiveLink`, the agent's own adopt pass at `agent.go:1319`) or `e2e only`; the test also proves each `root pass lock` site's caller chain reaches only `runAdoptUnderRootRelease`. A new site, or a removed one, fails until the list is edited, and a site the implementer cannot place in a class is a question for the reviewer, not a guess. |
| -- [ ] **Step 1: Write the failing tests**: `TestRootRemovesOnlyTheRecordedIdentity` (Review Focus 3: a publication replaces the link between the reference check and the `fstatat` → kept, the replacement at its name; the recorded entry → removed); `TestWritableLayoutIsNeverRemoved` (a parent the test uid owns; an ancestor writable by group: kept "writable layout", nothing unlinked, no intent line); `TestRootRowCheckIsAtomicWithTheUnlink` (an `Upsert` issued during the freeze returns after it; a row present at the freeze → kept); `TestRootJournalResolvesEveryIntent` (a crash after each journal line and after the `unlinkat`: each restart ends in removed, kept or absent; a crash after the unlink and before `removed` → `absent`); `TestReferencedBuildIsKept` (another host's link to the build; a process; `Processes` failing → left); `TestProgramsFileHasNoReferenceCheck`; `TestRootOnlyLiveLinkWritersAreExcluded`. The Linux tests that need a root-owned parent run when the test runs as root (the root runner, `root-runner.py`) and are skipped otherwise; the identity and journal logic run as any uid against an injected chain check. |
| -- [ ] **Step 2: Run to verify they fail**: `go test ./internal/release -run 'RootRemoves|WritableLayout|RootRow|RootJournal|ReferencedBuild|ProgramsFile' -count=1`, `go test ./internal/host -run 'LiveLinkWriters' -count=1`. |
| +- [ ] **Step 1: Write the failing tests**: `TestRootRemovesOnlyTheRecordedIdentity` (Review Focus 3: a publication replaces the link between the reference check and the `fstatat` → kept, the replacement at its name; the recorded entry → removed); `TestNoPublisherRunsBetweenFstatatAndUnlinkat` (the S12 witness: a hook between `fstatat` and `unlinkat` starts a second adopt pass through the production lock functions, with `/data/adopt-pass.lock` replaced by a symlink so its adopt-pass lock fails, and a held publication of the same host: the second pass blocks in `LockRootPass` and the held publication in the held-stage lock until the step returns; the recorded entry is removed, and the second pass's publication lands after it); `TestWritableLayoutIsNeverRemoved` (a parent the test uid owns; an ancestor writable by group: kept "writable layout", nothing unlinked, no intent line); `TestRootRowCheckIsAtomicWithTheUnlink` (an `Upsert` issued during the freeze returns after it; a row present at the freeze → kept); `TestRootJournalResolvesEveryIntent` (a crash after each journal line and after the `unlinkat`: each restart ends in removed, kept or absent; a crash after the unlink and before `removed` → `absent`); `TestReferencedBuildIsKept` (another host's link to the build; a process; `Processes` failing → left); `TestProgramsFileHasNoReferenceCheck`; `TestRootOnlyLiveLinkWritersAreExcluded`. The Linux tests that need a root-owned parent run when the test runs as root (the root runner, `root-runner.py`) and are skipped otherwise; the identity, journal and lock-order logic run as any uid against an injected chain check. |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/release -run 'RootRemoves|WritableLayout|RootRow|RootJournal|ReferencedBuild|ProgramsFile' -count=1`, `go test ./internal/host -run 'LiveLinkWriters|NoPublisherRuns' -count=1`. |
| - [ ] **Step 3: Implement. Step 4: Run to verify they pass**, then the cross-builds of Task 6b. |
| -- [ ] **Step 5: Commit** `root: removes a recorded leftover by fstatat and unlinkat on its opened parent under the registry freeze and the held-stage lock, never on a writable layout, journalling intent and result`. |
| +- [ ] **Step 5: Commit** `root: removes a recorded leftover by fstatat and unlinkat on its opened parent, inside the adopt pass, under the registry freeze and the held-stage lock, never on a writable layout, journalling intent and result`. |
| ### Task 7a: The create holds the hostname key |
| @@ -290,22 +342,39 @@ Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and |
| - [ ] **Step 3: Implement. Step 4: Run to verify they pass**, plus `go test ./internal/console -run 'Lifetime|Allocation' -count=1`. |
| - [ ] **Step 5: Commit** `console: the lifetime create holds its hostname in the reservation table, with the allocation row as its intent and a release that settles it`. |
| -### Task 7b: Placement holds every instance's hostname key, and its sweep |
| +### Task 7b: Placement registers, then holds every instance's hostname key, under one deadline |
| **Files:** |
| - Create: `internal/console/placereserve.go` |
| -- Modify: `internal/console/fleets.go` (`placeAppTarget`, line 568), `internal/console/offlinereserve.go` (`place` reclaim → `s.settlePlacement`), `internal/console/lifetimes_recovery.go` (`residenceSweep` gains the placement settle) |
| -- Test: `internal/console/placereserve_test.go`, `internal/console/residence_audit_test.go` |
| +- Modify: `internal/console/fleets.go` (`placeAppTarget`, line 568), `internal/console/store.go` (`claimRawSetAppPlacement`, line 1692, takes the context: `s.db.ExecContext(ctx, …)`; its adapter `SetAppPlacementContext`, line 2725, passes `claimContext`), `internal/registry/registry.go` (`SetPlacementContext(ctx, host, target)`, line 2369's body with `r.sql.ExecContext(ctx, …)` and `r.remote.call(ctx, …)`, `remote.go:138`; `SetPlacement` calls it with `context.Background()`), `internal/console/offlinereserve.go` (`place` reclaim → `s.settlePlacement`), `internal/console/residence_sweep.go` (the placement settle) |
| +- Test: `internal/console/placereserve_test.go`, `internal/console/residence_audit_test.go`, `internal/registry/setplacement_context_test.go` |
| **Interfaces:** |
| -- **The placement** (`placeAppTarget`, claims-off branch only; the host-claim branch returns before this as today): after the seed check, the app's record keys are listed (the listing at line 678 moves up, still under `instanceStoreTimeout`), then `id, ref := s.reservePlacement(ctx, acct, app, hosts)`: one `moves.Reserve(ctx, s.cfg.Releases, {place, id}, want)` with `want[HostKey(h)] = {place, id, h, acct.Sqid, app, now}` for every listed host (N10: one valid entry per key, its `Host` that key's host), and `s.placing[id] = true` (a mutex-guarded set); `ErrReserved` by a `cleanup` → 409 "an instance of this app is still being cleaned up. try again later"; by another → 409 "an instance of this app is moving. try again later"; unreadable or uncertain → 502 "could not reserve the app's instances. try again" (an uncertain write is settled by the sweep's release); nothing else is written on any refusal. Then app placement, routes and records exactly as today. `defer s.settlePlacement(ctx, id, uncertain)` on every return after the reservation: each host whose `SetRecordRegion` returned an error other than `nil`, `os.ErrNotExist` or a seed conflict is fenced (`s.lockAppHost`, `s.admit(host, "")`, `s.fenceRecord(ctx, acct, app, host, fenceOwner{})`, as `offlineMoveReclaimable` fences); every fence answered → `moves.Release(ctx, s.cfg.Releases, {place, id})`; a fence failing → the entry stays for the sweep; `delete(s.placing, id)`. A record created after the listing came through the create, which holds its own key. |
| -- **The sweep** (in `residenceSweep`): a `place` operation whose entries are older than `offlineLockGrace` and whose id is not in `s.placing` → fence the record of every host key it holds (each entry's host, account and app), then `moves.Release`; any failure → kept, logged hourly. `offlineReclaimable` for `place` runs the same settle (`reclaimHeld` while `s.placing` has the id). |
| +- **The placement** (`placeAppTarget`, claims-off branch only; the host-claim branch returns before this as today), after the claims branch and before `SetAppPlacementContext`: `ctx, cancel := context.WithTimeout(ctx, instanceStoreTimeout)` (15 s), **the one operation deadline**, covering the listing, the reservation, the app placement, the routes and the records; the app's record keys are listed (the listing at line 678 moves up); then **`id := newPlaceID(); s.registerPlacement(id)`** (`s.placing`, a mutex-guarded `map[string]time.Time`) **before** `s.reservePlacement(ctx, acct, app, hosts, id)`: one `moves.Reserve(ctx, s.cfg.Releases, {place, id}, want)` with `want[HostKey(h)] = {place, id, h, acct.Sqid, app, now}` for every listed host (N10: one valid entry per key, its `Host` that key's host); `ErrReserved` by a `cleanup` → 409 "an instance of this app is still being cleaned up. try again later"; by another → 409 "an instance of this app is moving. try again later"; unreadable or uncertain → 502 "could not reserve the app's instances. try again" (an uncertain write is settled by the settle's release); nothing else is written on any refusal. Then app placement, routes and records exactly as today, with the two local writes under the deadline: **`s.store.SetAppPlacementContext(ctx, …)`** (which now reaches SQLite through `ExecContext`) and **`reg.SetPlacementContext(ctx, rt.Host, target)`** in the routes loop (today `reg.SetPlacement`, which takes no context). A caller paused past the deadline therefore cannot make either local write, nor `SetRecordRegion`, after it. |
| +- **Settlement.** `defer s.settlePlacement(id, uncertain)` from the registration on, on every return: under its own `context.WithTimeout(context.Background(), instanceStoreTimeout)`, each host whose `SetRecordRegion` returned an error other than `nil`, `os.ErrNotExist` or a seed conflict is fenced (`s.lockAppHost`, `s.admit(host, "")`, `s.fenceRecord(ctx, acct, app, host, fenceOwner{})`, as `offlineMoveReclaimable` fences); every fence answered → `moves.Release(ctx, s.cfg.Releases, {place, id})` (a write, so it also settles an uncertain reservation); a fence or the release failing → the entry stays for the sweep; then, and only then, `s.deregisterPlacement(id)`. The registration therefore covers the whole of acquisition, the writes, the fence and the release. A record created after the listing came through the create, which holds its own key. |
| +- **The sweep** (in `residenceSweep`) and `offlineReclaimable` for `place`: an operation is abandoned only when its id is **not registered** in `s.placing` **and** every entry of it is older than `offlineLockGrace`; then the record of every host key it holds is fenced (each entry's host, account and app), and `moves.Release` follows; any failure → kept, logged hourly. A registered id is never swept, however old: its own deadline ends its writes and its own settle releases it. |
| - **The audit** `TestEveryResidenceChangeHoldsTheHostnameKey` (`go/ast` over non-test files under `./internal/...`): every call of `provision.SetRecordRegion`, `provision.OfflineFlip`, `provision.OfflineAbort`, `provision.MoveComplete`, `provision.MoveFlip`, `provision.MoveRollBack` and `(*Server).createRecordWithLifetimeLocked` is listed with the reservation that covers it (placement, a move's selection, the create) or its reviewed exemption: `MoveComplete` (`internal/host/movespass.go:1617`, the legacy cut's last act on the source box: tolerated, Self-review notes), `MoveFlip` and `MoveRollBack` (Phase 1, host claims on, where no box judge and so no cleanup exists). |
| -- [ ] **Step 1: Write the failing tests**: `TestPlacementReservesEveryInstanceFirst` (busy → app placement, routes and records unchanged; the table written holds one valid entry per host key, and plan 3's parser fixture reads it as malformed, which the rollback rule covers); `TestPlacementFencesAnUncertainWriteBeforeRelease` (a store that commits the record PUT after the client's context is cancelled: the fence makes it fail its condition, the entry released after the fence); `TestPlacementCrashIsSettledByTheSweep`; `TestResidenceChangesAndCleanupExclude` (Review Focus 4, with the real create, the real placement door and an offline move selection against a box cleanup's entries); `TestEveryResidenceChangeHoldsTheHostnameKey`; `TestHolderChecksPreserved` (today's placement refusals byte for byte). |
| -- [ ] **Step 2: Run to verify they fail**: `go test ./internal/console -run 'Placement|ResidenceChanges|EveryResidence|HolderChecks' -count=1`. |
| +- [ ] **Step 1: Write the failing tests**: `TestPlacementReservesEveryInstanceFirst` (busy → app placement, routes and records unchanged; the table written holds one valid entry per host key, and plan 3's frozen parser reads it as malformed, which the rollback procedure covers); `TestPlacementRegistersBeforeItReserves` (the id is in `s.placing` when the fake store sees the reservation `Put`, and leaves it only after the settle's release); `TestPausedPlacementIsNeverSwept` (review 5's R8 interleaving: the caller pauses after `Reserve` past the grace; the sweep runs and leaves the entry; another placement of the app is refused busy; the caller resumes past its deadline: `SetAppPlacementContext` and `SetPlacementContext` fail with the context's error, nothing is written, and its settle releases); `TestPlacementFencesAnUncertainWriteBeforeRelease` (a store that commits the record PUT after the client's context is cancelled: the fence makes it fail its condition, the entry released after the fence); `TestPlacementCrashIsSettledByTheSweep` (a new process with an empty `s.placing`); `TestSetPlacementContextHonoursTheDeadline` (registry, local and remote); `TestResidenceChangesAndCleanupExclude` (Review Focus 4, with the real create, the real placement door and an offline move selection against a box cleanup's entries); `TestEveryResidenceChangeHoldsTheHostnameKey`; `TestHolderChecksPreserved` (today's placement refusals byte for byte). |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/console -run 'Placement|PausedPlacement|ResidenceChanges|EveryResidence|HolderChecks' -count=1`, `go test ./internal/registry -run 'SetPlacementContext' -count=1`. |
| - [ ] **Step 3: Implement. Step 4: Run to verify they pass**, plus `go test ./internal/console -run 'Place' -count=1`. |
| -- [ ] **Step 5: Commit** `console: placement holds every instance's hostname with one entry per key, fences an uncertain record write before it releases, and a sweep settles a crashed one`. |
| +- [ ] **Step 5: Commit** `console: placement registers before it reserves every instance's hostname, writes under one deadline, fences an uncertain record write before it releases, and a sweep settles only an unregistered one`. |
| + |
| +### Task 7c: The rollback drain: `carlos system reservations drain` |
| + |
| +**Files:** |
| +- Create: `internal/console/reservations_drain.go` (`handleAPIReservationsDrain`, `s.draining`), `system_reservations.go` (the CLI verb, in `system_lifetimes.go`'s form) |
| +- Modify: `internal/console/authcli.go` (`s.cliRoute("POST /api/cli/system/reservations/drain", svcrole.Set{}, s.handleAPIReservationsDrain)`, beside line 1024), `internal/console/lifetimes_create.go` (a draining console refuses a create before `BeginAllocation`, line 102), `internal/console/fleets.go` (a draining console refuses a placement before its registration), `internal/console/selfserve_door.go` (a draining console refuses a self-serve move request before `AdmitSelfServeMove`, line 347), `system.go` (`reservations` in `systemVerb`'s list, line 53, `runSystem`'s switch and `systemUsage`), `docs/site/cli.md` (the verb line), the copy record and `approved_moves_copy_test.go` |
| +- Test: `internal/console/reservations_drain_test.go`, `system_reservations_test.go` |
| + |
| +**Interfaces:** |
| +- **The door** (operator only: `s.moveOperator`, a 404 otherwise, and a 404 without `s.cfg.Releases`, as `handleAPILifetimesCutover`, `lifetimes_cutover.go:526-531`). Idempotent; each call is one round. The first call sets `s.draining` (an `atomic.Bool` for the rest of this process; a restart clears it, which is the way to abandon a rollback): from then a create, a placement and a self-serve move request answer 503 "this console is being rolled back. try again in a few minutes", before anything is written. The round: (1) `moves.ReadActive`; (2) every `place` operation: registered in `s.placing` → counted in flight, untouched; else settled now as the sweep settles it (fence every host key's record, then `moves.Release`), **never released before its fences answered**; (3) every `create` operation: its allocation row through `recoverAllocationLocked` under the app lock (a terminal or voided row releases its key; a pending one stays and is counted); (4) the counts: `cleanup` operations in the table, open self-serve moves (`s.store.OpenSelfServeMoves()`, `selfserve_store.go:562`), `create` and `place` operations left; (5) all four zero → the settling write, `moves.Release(ctx, s.cfg.Releases, {Kind: "drain", ID: "drain"})` (it holds nothing and writes rev + 1), then one more `moves.ReadActive`: still zero → `drained`; else `draining`. The answer: `{state: "draining" | "drained", cleanups, creates, placements, self_serve_moves int, rev uint64}`. |
| +- **The CLI** `carlos system reservations drain [--console <origin>] [--no-wait]`: `requireDeployAPISession`, then the door every 5 s (`waitLifetimesCutover`'s loop and retry rules: a failed poll or a 5xx is asked again, up to 5 in a row or 60 s; a refusal is final), printing each line that differs from the last: "draining: <n> cleanups, <n> creates, <n> placements and <n> self-serve moves still hold reservations." and, at the end, "drained: no new kind of reservation is held. stop this console now, then start the old one." Copy rows, pending. The command never stops the console itself. |
| + |
| +- [ ] **Step 1: Write the failing tests**: `TestDrainSettlesBeforeItReleases` (Review Focus 4: a `place` entry whose record PUT the fake store commits late: the fence runs and makes it fail before the release; the entry goes only after the fence answered; a fence failure keeps it and the answer stays `draining`); `TestDrainRefusesNewWork` (after the first call a create, a placement and a self-serve request answer 503 and write nothing; a restarted console takes them again); `TestDrainWaitsForInFlightWork` (a registered placement, a pending allocation, an open self-serve move and a `cleanup` entry each keep the answer `draining` with its count); `TestDrainEndsWithASettlingWrite` (`drained` only after the console's own write and a re-read; a create `Put` held by the fake store from before the drain is delivered after it and fails its condition); `TestDrainIsOperatorOnly`; `TestSystemReservationsDrainCLI` (the lines, the wait and the retries). |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/console -run 'Drain' -count=1`, `go test . -run 'SystemReservationsDrain|MovesStringsAreRecorded|MovesCopyRecord' -count=1`. |
| +- [ ] **Step 3: Implement. Step 4: Run to verify they pass.** |
| +- [ ] **Step 5: Commit** `console: carlos system reservations drain settles every placement and create reservation and refuses new ones before a rollback`. |
| ### Task 8: The reaper |
| @@ -315,20 +384,20 @@ Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and |
| - Test: `internal/host/movesreap_test.go` |
| **Interfaces:** |
| -- `func (w *movesWorker) reapRetired(ctx context.Context, now time.Time) error`: an op directory `<dataDir>/.offline-moves/<host>/<move>/` is removed (`os.RemoveAll` of that directory, then the host directory when empty, then the obligation's delivered page entries under `pageEntryDir(dataDir, host, id)`) only when its `retired.json` is `ended`, `now - EndedAt >= 7 d`, the final repair pass has run (with `RepairKeys` non-empty: `w.repairKeys` once more, `RepairPasses = 2`, `RepairKeys` cleared and the report republished before the removal; an error leaves the directory for the next run), `PublishedRev == Rev`, `PagesPending` is empty, the page outbox holds no undelivered entry for `(host, id)`, and the drop is absent or root's acknowledgement exists. A directory without `retired.json`, with one in any other state, or `corrupt`, is never removed: a target's park, an aborted move's park, an untracked lifetime's park and every directory an earlier release wrote stay as they are today. |
| +- `func (w *movesWorker) reapRetired(ctx context.Context, now time.Time) error`: an op directory `<dataDir>/.offline-moves/<host>/<move>/` is removed (`os.RemoveAll` of that directory, then the host directory when empty, then the obligation's delivered page entries under `pageEntryDir(dataDir, host, id)`) only when its `retired.json` is `ended`, `now - EndedAt >= 7 d`, `PublishedRev == Rev`, `PagesPending` is empty, the page outbox holds no undelivered entry for `(host, id)`, and the drop is absent or root's acknowledgement exists. A directory without `retired.json`, with one in any other state, or `corrupt`, is never removed: a target's park, an aborted move's park, an untracked lifetime's park and every directory an earlier release wrote stay as they are today. The bucket report is not touched, so the log still answers after the reaper (Task 5b). |
| -- [ ] **Step 1: Write the failing tests**: `TestOpDirReapedSevenDaysAfterEnded`; `TestFinalRepairPassRunsBeforeTheReaper` (a marker landing between the first and the final pass is repaired before the directory goes; a repair error keeps the directory); `TestUnpublishedReportBlocksTheReaper`; `TestPendingPageBlocksTheReaper`; `TestUndeliveredPageBlocksTheReaper`; `TestOutstandingRootRequestBlocksTheReaper`; `TestDirectoriesWithoutAnObligationAreNeverReaped`; and Task 5a's `TestCleanupReportIsDurableAndBlocksTheReaper` completed with the reaper. |
| -- [ ] **Step 2: Run to verify they fail; Step 3: implement; Step 4: run** `go test ./internal/host -run 'Reap|DurableAndBlocks|FinalRepairPass' -count=1`. |
| +- [ ] **Step 1: Write the failing tests**: `TestOpDirReapedSevenDaysAfterEnded`; `TestUnpublishedReportBlocksTheReaper`; `TestPendingPageBlocksTheReaper`; `TestUndeliveredPageBlocksTheReaper`; `TestOutstandingRootRequestBlocksTheReaper`; `TestDirectoriesWithoutAnObligationAreNeverReaped`; `TestReportOutlivesTheReaper`; and Task 5a's `TestCleanupReportIsDurableAndBlocksTheReaper` completed with the reaper. |
| +- [ ] **Step 2: Run to verify they fail; Step 3: implement; Step 4: run** `go test ./internal/host -run 'Reap|DurableAndBlocks|ReportOutlives' -count=1`. |
| - [ ] **Step 5: Commit** `agent: an ended cleanup's directory is removed seven days on, once its report and pages are delivered and root has answered`. |
| ### Task 9: End to end and the census |
| **Files:** |
| -- Test: `internal/e2e/selfserve/cleanup_test.go` (plan 2's bridge `host.E2E`, a clock the test advances, a root pass the test drives with the adopt lock held, a hibernate bucket fake with S3's conditional-delete semantics) |
| +- Test: `internal/e2e/selfserve/cleanup_test.go` (plan 2's bridge `host.E2E`, a clock the test advances, a root pass the test drives through the adopt pass's stages with both locks held, a hibernate bucket fake with S3's versioned semantics and a hook that holds a delete) |
| - Create: `internal/provision/testdata/plan14-root-reads-self-serve-moves-4.tsv`, `internal/provision/self_serve_moves4_census_test.go`, `docs/superpowers/notes/census-proposal-self-serve-moves-4.md` |
| - Modify: `internal/provision/plan14_root_reads_test.go`, `internal/provision/plan9_b8_census_test.go` (line 372) |
| -- [ ] **Step 1: Write the tests**: `TestCleanupTwentyFourHoursAfterTheObservedFinish` (a forward move; at 24 h every inventoried key carries a delete marker, each delete conditional on its ETag, `lock.json` and `build-state.json` remain, the lifted tombstone is back, root removed the live link when its chain is root-only, the programs file is reported left, the entries are gone, the table is still there and empty, the report is under the move id and `carlos move offline status --log` shows it); `TestCleanupSupersededByAPlacementBack` (the placement is refused while the cleanup holds; after it ends, a placement back to the source runs); `TestCreateUnderCleanupRefusedThenAllowed`; `TestRowMintedUnderTheMarkSupersedes`; `TestLateDeleteAfterSupersessionIsRefused` (a held delete delivered after the release and a new wake's write: 412, the instance serves its new data); `TestByteIdenticalRepromotionIsRepaired` (a legacy move back into the source region re-promotes byte-identical files; a held delete lands on one after `QuietUntil`; the next cleanup visit restores it and the instance's next wake restores the full replica); `TestOperatorReturnAfterSupersededCleanupStillFindsItsLeftovers` (the return's start door accepts the forward's leftover set after a superseded cleanup that had marked and unmarked); `TestLegacyMoveIntoTheSourceRegionIsTolerated` (switch off: a legacy move back to the source region while the cleanup holds: the cleanup supersedes at its next key, waits out any `QuietUntil`, the wake waits for the unmark, the promotion restores the staged data and the instance serves it); `TestUntrackedLifetimeMoveFinishes` (an incarnation-0 record with no slot moved by an operator: `gone` reports, the move finishes, no obligation exists); `TestCleanupReportInTheMoveLog`. |
| +- [ ] **Step 1: Write the tests**: `TestCleanupTwentyFourHoursAfterTheObservedFinish` (a forward move; at 24 h every inventoried version is gone, each removed by its version id, oldest first, no marker written by the cleanup, `lock.json` and `build-state.json` remain, the lifted tombstone is back, root removed the live link when its chain is root-only, the programs file is reported left, the entries are gone, the table is still there and empty, the report is under the move id and `carlos move offline status --log` shows it); `TestCleanupSupersededByAPlacementBack` (the placement is refused while the cleanup holds; after it ends, a placement back to the source runs); `TestCreateUnderCleanupRefusedThenAllowed`; `TestRowMintedUnderTheMarkSupersedes`; `TestLateAttemptAfterSupersessionChangesNothing` (a delete's first attempt held past the supersession; the second attempt answered under the mark; the release; a new wake writes; the held attempt is delivered: the instance serves its new data); `TestPromotionBackIntoTheRegionKeepsItsData` (a legacy move back into the source region while a superseded cleanup had a delete in flight: the delete is answered under the mark, the wake waits for the unmark, the promotion keeps equal-ETag versions in place and copies the rest, and the instance's next wake restores the full replica); `TestOperatorReturnAfterSupersededCleanupStillFindsItsLeftovers` (the return's start door accepts the forward's leftover set after a superseded cleanup that had marked and unmarked); `TestLegacyMoveIntoTheSourceRegionIsTolerated` (switch off: a legacy move back to the source region while the cleanup holds: the cleanup supersedes at its next key, answers any delete in flight, the wake waits for the unmark, the promotion restores the staged data and the instance serves it); `TestUntrackedLifetimeMoveFinishes` (an incarnation-0 record with no slot moved by an operator: `gone` reports, the move finishes, no obligation exists); `TestOlderCleanupInTheMoveLog` (two moves of one host, the first's cleanup open: `--log --cleanup <first id>` shows it after the second finished and after the record is deleted); `TestRollbackDrain` (the procedure's steps 1 and 2 against a live cleanup, a placement and a create: the drain answers `drained` only after the cleanup ended and both were settled; the old console's parser then reads the table). |
| - [ ] **Step 2: Run them; fix what they find within this plan's scope.** |
| - [ ] **Step 3: Run both census tests and the cross-builds**; write the census proposal with the loader block, the changed envelopes (`internal/host`, `internal/release`, `internal/registry`, `internal/lease`, `internal/moves`, `internal/console`, `internal/offlinecleanup`, the root package) and the units. |
| - [ ] **Step 4: Commit** `self-serve moves: cleanup end to end`; push. |
| @@ -337,10 +406,11 @@ Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and |
| ## Self-review notes |
| -- Spec coverage: §7.6 the source replica (T1, T3, T4a, T4b, T4c), root-owned leftovers with the link and process checks (T6a, T6b, T6c), parked local state (T8), the staged objects of an aborted move (unchanged); §7.1's `cleanup` reservation and the hostname keys (T2, T7a, T7b); §8's cleanup pages and every cleanup key in the move log (T5a, T5b); §9 rollout (Global Constraints); §11's cleanup witnesses (T4a, T4b, T9). |
| -- **Stated deviations.** (1) Legacy moves hold no reservation. A legacy move into the source region carries a move on the record from its door onward, so condition (1) supersedes the cleanup at its next key, and its completion's wake cannot take the lease while the mark holds; after the unmark the wake's promotion mirrors the staged replica over the prefix (`promoteStagedReplica`, an exact mirror, called during the restore after acquisition, `backend_exec.go:819`), so no deleted key matters; review 4's writer audit confirms this normal-operation argument (completion changes the record; promotion waits for acquisition). A late delete from before the supersession is refused by its ETag condition, and the byte-identical copies such a promotion makes wait out `QuietUntil`. Holding a key from the box at `MoveComplete` (`movespass.go:1617`) would add a box-side release and crash path and extend a legacy cut's downtime behind root's answer, for no exclusion the mark does not already give. (2) Placement keeps `SetRecordRegion`'s ETag-conditional write and settles an uncertain one by `fenceRecord`, after which no earlier PUT can land; the reservation entry is the persisted intent, so no journal table is added. (3) A cleanup entry is never reclaimed by another reserver (§7.1 allowed a status-based reclaim); the console pages instead. (4) Only the sealing box's local paths are cleaned; other boxes in the source region keep theirs, as today. (5) The create reserves after `BeginAllocation`, not before, so every reservation has an intent row behind it. (6) `build-state.json` is kept with `lock.json`. (7) §7.6's "plain `DeleteObject`" becomes a conditional one on the inventoried ETag: still a marker, never a version delete. (8) The table is never deleted (plan 2 deleted it when empty). (9) Root removes a leftover only on a root-only chain; the programs file, under the agent-writable `/data` on every built box, is recorded as left at `gone` and stays the runbook's, and so do both paths on a writable-opt box. (10) Root's rename-aside is gone; identity is compared on the opened parent with `fstatat`. (11) `QuietUntil` holds the mark and the reservation for 15 minutes after a run whose deletes were not all answered, for the byte-identical case the ETag condition cannot separate; it is a bound, and the repair (two passes, after the end and before the reaper) removes any marker a later delete still put over a later owner's byte-identical version, by its version id: the one version delete in the plan, on a marker only. |
| +- Spec coverage: §7.6 the source replica (T1, T3, T4a, T4b, T4c, T4d), root-owned leftovers with the link and process checks (T6a, T6b, T6c), parked local state (T8), the staged objects of an aborted move (unchanged); §7.1's `cleanup` reservation and the hostname keys (T2a, T2b, T2c, T7a, T7b); §8's cleanup pages and every cleanup key in the move log (T5a, T5b); §9 rollout and rollback (Global Constraints, T7c); §11's cleanup witnesses (T4a, T4c, T9). |
| +- **Stated deviations.** (1) Legacy moves hold no reservation. A legacy move into the source region carries a move on the record from its door onward, so condition (1) supersedes the cleanup at its next key, and its completion's wake cannot take the lease while the mark holds; after the unmark the wake's promotion mirrors the staged replica over the prefix (`promoteStagedReplica`, an exact mirror, called during the restore after acquisition, `backend_exec.go:819`), so no deleted key matters; review 4's writer audit confirms this normal-operation argument (completion changes the record; promotion waits for acquisition). A delete sent before the supersession is answered under the mark, before the unmark, so none is outstanding when the promotion keeps an equal-ETag version in place. Holding a key from the box at `MoveComplete` (`movespass.go:1617`) would add a box-side release and crash path and extend a legacy cut's downtime behind root's answer, for no exclusion the mark does not already give. (2) Placement keeps `SetRecordRegion`'s ETag-conditional write and settles an uncertain one by `fenceRecord`, after which no earlier PUT can land; the reservation entry is the persisted intent, so no journal table is added. (3) A cleanup entry is never reclaimed by another reserver (§7.1 allowed a status-based reclaim); the console pages instead. (4) Only the sealing box's local paths are cleaned; other boxes in the source region keep theirs, as today. (5) The create reserves after `BeginAllocation`, not before, so every reservation has an intent row behind it. (6) `build-state.json` is kept with `lock.json`. (7) **§7.6's "plain `DeleteObject`, `VersionId` deletes are never used" is amended (Paul's question 9):** every inventoried version is deleted by its version id, oldest first, the version current at `gone` last, so no older version is ever exposed and no late request can touch another owner's version. The retired replica is then not recoverable from the bucket. (8) The table is never deleted (plan 2 deleted it when empty). (9) Root removes a leftover only on a root-only chain; the programs file, under the agent-writable `/data` on every built box, is recorded as left at `gone` and stays the runbook's, and so do both paths on a writable-opt box. (10) Root's rename-aside is gone; identity is compared on the opened parent with `fstatat`. (11) §7.6's "a key already carrying a delete marker is done" becomes: a key whose current entry is a marker the inventory does not hold has its inventoried versions beneath that marker deleted (nothing current changes), and the marker is left. (12) Root's step is a stage of the adopt pass, under a new root-only pass lock that every adopt pass takes; held publication's freeze and lock are unchanged. |
| +- **The one residual in root's exclusion.** A pass whose root pass lock fails for a root-level fault (an I/O error on a root-only directory, which no other uid can cause) runs no root step, but it still publishes, as a pass whose adopt-pass lock fails does today, so it is not excluded from another pass's step. Making that failure fatal would remove the residual at the cost of a box that stops converging on that fault; the plan keeps today's never-fatal rule and names the case here for the reviewer. |
| - Plan 2 amendments (built code, changed here): move selections hold `host/<hostname>`; the switch-off rule applies between moves only, so a cleanup neither blocks nor is blocked by an operator move on another box; recovery mode counts `move` entries only; the cleanup reclaim interface is deleted; `releaseBoxes` writes an empty table instead of deleting it; every table write carries `epoch` and `rev`. |
| -- Built-code changes this plan makes, by task: T1 `moves/offline.go`, `host/movesoffline.go`, `release/programhistory.go` (export); T2 `moves/offline.go` (kinds, epoch, rev), `console/offlinereserve.go`, `console/selfserve_pass.go`, `console/selfserve_pages.go`, `console/lifetimes_recovery.go`, `console/lifetimes_store.go`; T3 `lease` (new file only); T4b `lease/mirror.go` (the conditional delete); T4c `host/movespass.go`; T5a `host/movespaging.go`, `console/apiinstancemove_offline.go`; T5b `console/apiinstancemove_offline.go`, `move.go`, `docs/site/cli.md`, the copy record; T6a `registry`; T6b `host/agent.go`; T7a `console/store.go`, `lifetimes_store.go`, `lifetimes_create.go`, `lifetimes_recovery.go`; T7b `fleets.go`, `offlinereserve.go`, `lifetimes_recovery.go`. |
| -- Names used across tasks: `offlinecleanup.Retired` (`State`, `Outcome`, `Keys`, `Events`, `PagesPending`, `LocalLeft`, `Root`, `Rev`, `PublishedRev`, `MarkedAt`, `QuietUntil`, `RunKeys`, `RepairKeys`, `RepairSince`, `RepairPasses`, `RootRequestedAt`, `CancelRequestedAt`), `CorruptStub`, `CleanupID`, `InventoryObject`, `KeyOutcome`, `Event`, `PendingPage`, `Report`, `FileIdentity`, `LocalPath`, `KeptPath`, `RootCleanupRequest`, `RootCleanupPath`, `RootCleanupAck`, `ValidateRetired`, `ValidateCorruptStub`, `ValidateReport`, `Identify`, `RootOnlyChain`; `moves.HostKey`, `IsHostKey`, `Active`, `ReadActive`, `NextActiveBody`, `Op`, `Conflicts`, `Reserve`, `Holds`, `Release`, `ErrReserved`, `ErrUnreadable`, `ErrUncertain`, `ErrReservationMismatch`, `RecordAway`, `Away*`, `OfflineOpDir`, `RetiredFile`, `CleanupReportKey`, `OfflineKindCreate`, `OfflineKindPlace`; `lease.CleaningOwner`, `MarkCleaning`, `ReassertCleaning`, `UnmarkCleaning`, `ErrNotOurLock`, `ConditionalMirrorDeleter`, `ErrDeleteConditionFailed`; `w.cleanupVisit`, `w.supersededBy`, `w.retireKey`, `w.repairKeys`, `w.kickCleanup`, `cleanupRun`, `cleanupMu`, `cfg.PlainMirror`, `cfg.CleanupArmed`, `recordRetired`, `createRetired`, `writeRetired`, `markPublished`, `flushPageOutboxFor`, `reapRetired`; `release.ServeRootCleanupRequests`, `RootCleanupOptions`, `RootCleanupDir`, `ReadRootCleanupAck`, `ErrRootCleanupRefused`, `ProgramsPath`; `rootCleanupAuthority`, `rootCleanupPass`; `registry.WithHostFrozen`; `s.offlineCleanupOf`, `s.reservePlacement`, `s.settlePlacement`, `s.placing`, `s.residenceSweep`, `s.releaseAllocationKey`, `errHostUnderCleanup`; the CLI's `moveArgs.log`. |
| -- Review Focus 1 to 5 are pinned by T4b, T4a, T6b+T6c, T2+T7a+T7b, T5a+T8. |
| -- **Open questions for Paul** (amadan discussion 363, unchanged): (1) legacy moves without a hostname reservation: accept the exemption, or reserve around `MoveComplete` on the box (a box-side release and crash path, and longer legacy-cut downtime while root answers)? (2) Cleanup runs 24 hours after **every** offline move, operator moves included, replacing the runbook's manual cleanup; intended (§10 Q4 talks about self-serve)? (3) Is the box switch `CARLOS_OFFLINE_CLEANUP=on` acceptable as the rollout gate? (4) A move of the same host waits while its cleanup holds `host/<hostname>`, which can last until root answers (paged every 10 min); acceptable? (5) The copy rows: four door sentences, five page subjects and one page body (now also the CLI's lines and the `--log` paragraph). |
| +- Built-code changes this plan makes, by task: T1 `moves/offline.go`, `host/movesoffline.go`, `release/programhistory.go` (export); T2a `moves/offline.go` (kinds, epoch, rev, `valid()`); T2b `console/offlinereserve.go`, `console/selfserve_pass.go`, `console/lifetimes_store.go`; T2c `console/lifetimes_recovery.go`, `console/selfserve_pages.go`; T3 `lease` (new file only); T4b, T4c `host` (new files only); T4d `host/movespass.go`; T5a `host/movespaging.go`, `console/apiinstancemove_offline.go`; T5b `console/authcli.go`, `move.go`, `docs/site/cli.md`, the copy record; T6a `registry`; T6b `host/agent.go`, `host/adoptlock.go` and `host/movesheldlock.go` (comments), `host/claimcensus.go`; T7a `console/store.go`, `lifetimes_store.go`, `lifetimes_create.go`, `lifetimes_recovery.go`; T7b `fleets.go`, `console/store.go` (`claimRawSetAppPlacement`), `registry/registry.go` (`SetPlacementContext`), `offlinereserve.go`, `residence_sweep.go`; T7c `console/authcli.go`, `lifetimes_create.go`, `fleets.go`, `selfserve_door.go`, `system.go`, `docs/site/cli.md`. The mirror (`internal/lease/mirror.go`) is not changed. |
| +- Names used across tasks: `offlinecleanup.Retired` (`State`, `Outcome`, `Keys`, `Events`, `PagesPending`, `LocalLeft`, `Root`, `Rev`, `PublishedRev`, `MarkedAt`, `ReserveUncertain`, `InFlight`, `RootRequestedAt`, `CancelRequestedAt`), `CorruptStub`, `CleanupID`, `InventoryKey`, `InventoryVersion`, `InFlightDelete`, `KeyOutcome` (`State`, `Done`), `Event`, `PendingPage`, `Report`, `FileIdentity`, `LocalPath`, `KeptPath`, `RootCleanupRequest`, `RootCleanupPath`, `RootCleanupAck`, `ValidateRetired`, `ValidateCorruptStub`, `ValidateReport`, `Identify`, `RootOnlyChain`; `moves.HostKey`, `IsHostKey`, `Active`, `ReadActive`, `NextActiveBody`, `Op`, `Conflicts`, `Reserve`, `Holds`, `Release`, `ErrReserved`, `ErrUnreadable`, `ErrUncertain`, `ErrReservationMismatch`, `RecordAway`, `Away*`, `OfflineOpDir`, `RetiredFile`, `CleanupReportKey`, `OfflineKindCreate`, `OfflineKindPlace`; `lease.CleaningOwner`, `MarkCleaning`, `ReassertCleaning`, `UnmarkCleaning`, `ErrNotOurLock`, `VersionedStore.DeleteVersion`; `deleteInventoriedVersion`, `w.cleanupVisit`, `w.supersededBy`, `w.retireKey`, `w.settleInFlight`, `w.kickCleanup`, `cleanupRun`, `cleanupMu`, `cfg.CleanupArmed`, `recordRetired`, `createRetired`, `writeRetired`, `markPublished`, `flushPageOutboxFor`, `reapRetired`; `release.ServeRootCleanupRequests`, `RootCleanupOptions`, `RootCleanupBase`, `RootCleanupDir`, `ReadRootCleanupAck`, `LockRootPass`, `ErrRootCleanupRefused`, `ErrNotRootOnly`, `ProgramsPath`; `rootCleanupAuthority`, `rootCleanupPass`, `rootRefusedAfter`; `registry.WithHostFrozen`, `SetPlacementContext`; `s.offlineCleanupsOf`, `s.reservePlacement`, `s.registerPlacement`, `s.deregisterPlacement`, `s.settlePlacement`, `s.placing`, `s.residenceSweep`, `s.releaseAllocationKey`, `s.draining`, `errHostUnderCleanup`; the CLI's `moveArgs.log`, `logMove`, `logCleanup`, and `carlos system reservations drain`. |
| +- Review Focus 1 to 5 are pinned by T4b+T4c, T4a, T6b+T6c, T2a-c+T7a-c, T5a+T5b+T8. |
| +- **Open questions for Paul** (amadan discussion 363): (1) legacy moves without a hostname reservation: accept the exemption, or reserve around `MoveComplete` on the box (a box-side release and crash path, and longer legacy-cut downtime while root answers)? (2) Cleanup runs 24 hours after **every** offline move, operator moves included, replacing the runbook's manual cleanup; intended (§10 Q4 talks about self-serve)? (3) Is the box switch `CARLOS_OFFLINE_CLEANUP=on` acceptable as the rollout gate? (4) A move of the same host waits while its cleanup holds `host/<hostname>`, which can last until root answers (paged every 10 min, hourly once root's ancestry is refused); acceptable? (5) The copy rows: five door sentences, seven page subjects and one page body (now also the CLI's lines, the drain's lines and the `--log` paragraph). (6) On every built box the programs file sits under the agent-writable `/data`, so root never removes it and it stays the runbook's; acceptable, or should the agent remove it itself? (7) Withdrawn by rev 6: the 15-minute `QuietUntil` and the marker-removing repair are gone; question 9 replaces it. (8) Rollback needs no self-serve move in flight; the drain now refuses to finish until none is. Acceptable? (9) **The §7.6 amendment:** the cleanup deletes every inventoried version of the retired replica by its version id (oldest first, the one current at `gone` last) instead of writing delete markers, so the source copy of a finished move is not recoverable from the bucket after the cleanup, in exchange for a boundary the server enforces (a late or retried request can remove only the version it names). Accept, or keep markers (then only `deleteInventoriedVersion` changes, and the settlement rules need a fresh review)? |