amadan
plan 4 rev 5: conditional deletes on the inventoried ETag with a quiet period and a repair pass; active.json never deleted and never repeats a body; no rename-aside, no root removal on the writable layout; lifecycle-wide supersession; untracked lifetimes get no cleanup; fifteen right-sized tasks incl. the CLI log
Answers Astra's review of rev 4 (F1-F13, N1-N11). The two destructive boundaries change shape rather than gaining fences: every bucket delete is If-Match on the inventoried version's ETag (a late delete cannot hide a newer version), with a 15-minute quiet hold and a marker-removal repair for byte-identical versions; root compares identity and unlinks under the held-publication lock and a registry host freeze, journalling in a root-owned directory, and does nothing on the writable layout.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1 file changed,
+229
−118
docs/superpowers/plans/2026-10-22-self-serve-moves-4-cleanup.md+229 −118
CI failed — run details
CI log
=== gate === sigwho: pid 2 ppid 1 pgid 0 sid 0; child pid 3 .amadan/test-gate.sh gate: 2026-10-24T08:46:58Z on aarch64, 32 cpus, 61341 MB available, go1.26.8 linux/arm64 gate: +0s start launchctx gate: +0s start vet gate: +0s start gofmt gate: +1s start rest gate: +11s start internal_console@0 gate: +11s start internal_console@1 gate: +11s start internal_console@2 gate: +11s start internal_console@3 gate: +11s start internal_console@4 gate: +11s start internal_console@5 gate: +11s start internal_console@6 gate: +11s start internal_console@7 gate: +11s waiting for internal_console's shards before the next split package gate: +173s start root@0 gate: +173s start root@1 gate: +173s start root@2 gate: +173s waiting for root's shards before the next split package amadan: step gate failed: signal: terminated amadan: job failed at step gate — its output is above, not at the tail of this log
diff --git a/docs/superpowers/plans/2026-10-22-self-serve-moves-4-cleanup.md b/docs/superpowers/plans/2026-10-22-self-serve-moves-4-cleanup.md| index 6338a12a3..b8b107232 100644 |
| --- a/docs/superpowers/plans/2026-10-22-self-serve-moves-4-cleanup.md |
| +++ b/docs/superpowers/plans/2026-10-22-self-serve-moves-4-cleanup.md |
| @@ -2,89 +2,95 @@ |
| > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| -Rev 4, 2026-10-24. Rev 3's review (`after236/selfserve-plan4-review3.md`: F4 closed; F1 to F3, F5 to F13 and N1 to N3 open) is answered, mostly by removing mechanisms. **F1:** the bucket is protected by a **mark**, not by an argument about which writers can run: before its first deletion the cleanup rewrites the source bucket's `lock.json`, by a conditional write from this move's lifted tombstone, into a live lease owned by `cleaning:<cleanup id>`; every writer that mints a replica version under the hibernate prefix either acquires that lease or writes a tombstone over it, and both refuse a live lease of another owner (the inventory is a Global Constraint, file by file, pinned by an AST test); so a reconcile that mints a late row cannot wake the host into the prefix, and the row supersedes the obligation; `build-state.json`, the one object written outside the lease (the cut's request carry), is no longer inventoried. **F2:** the create reserves `host/<hostname>` immediately **after** `BeginAllocation`, so the allocation row is the intent and every create reservation has a row behind it: a refusal or a crash before the row holds nothing, a busy answer voids the row, and a terminal row whose release is owed is revisited by `PendingAllocations`, whose predicate gains `host_key = 1`. **F3:** one state enum (`open`, `finished`, `cleaning`, `publishing`, `root-requested`, `cancel-pending`, `releasing`, `ended`, `corrupt`) drives one transition table; there is no second publication field. **F5, F6, F9:** root's copy of the request, its journal and its acknowledgement live in a root-owned directory under the opt dir; the agent's request is a drop that root copies, checks and removes; root authorises through a callback `internal/host` supplies (the cleanup's entries on the box's own key and on `host/<hostname>` with the request's account and app, the judged record away from this region, the mark), so `internal/release` imports nothing new; the unlink runs inside the registry freeze and the held-stage lock in their existing order and removes only an entry it has first **renamed aside and verified**, so a replacement by any writer, locked or not, is never removed; every journal intent resolves to removed, kept or absent. **F7:** a source row appearing supersedes the obligation (root cancelled and acknowledged, the mark removed, the reservation released). **F8:** the programs file stores Go main-package paths for its own host (`internal/release/programhistory.go:17-34`) and nothing else refers to it, so its checks are identity and row; the live link keeps §7.6's link and process checks, and evidence that cannot be read is retried, never taken as unreferenced. **F10:** the reaper removes only directories whose obligation has ended (§7.6's own scope), so no park without an obligation is ever stamped or removed. **F11:** the cleanup runs in its own goroutine, started by the moves pass through a try-lock, and never takes `w.pass`. **F12:** the switch-dependent `Exclusive` option is removed: the box's acquisition conflicts key for key in every mode, which is safe because the console's switch-off rule is a console-side restriction between moves only, and `control/offline-moves/console.json` is `{console, until}` (`internal/console/selfserve_lease.go:44-48`), with no switch to read; the remaining contract gaps are decided where each task names them. **F13:** the cleanup report is republished under the move id until the persisted `PublishedRev` equals `Rev`; refusals are stable, de-duplicated events inside it; pages go through the existing page outbox; the reaper waits for both. **N1:** the prefix is normalised once, with its trailing slash, before listing, filtering and persistence. **N2:** `releasing` is persisted before the mark is removed and the reservation released, and a restarted obligation deletes only after finding both entries and the mark still its own. **N3:** a placement reserves every instance's hostname key before it writes anything; the entry is its persisted intent; an uncertain record write is fenced (`fenceRecord`) before the release; a crashed placement's entries are fenced and released by the console's sweep. The deviations this costs (legacy moves, the placement write, the cleanup entry's reclaim, other boxes' local paths) are stated in the Self-review notes. Reviews: rev 1 (`after236/selfserve-plan4-review.md`, 18 findings), rev 2 (`selfserve-plan4-review2.md`, F1 to F13), rev 3 (`selfserve-plan4-review3.md`). |
| +Rev 5, 2026-10-24. Rev 4's review (`after236/selfserve-plan4-review4.md`: F3, F4, F8 to F11 and N1 to N3 closed; F1, F2, F5 to F7, F12, F13 open; N4 to N11 new) is answered by making each destructive step conditional at the server, or by removing it. **N4, F1:** every bucket deletion is `DeleteIfMatch(bucket, key, etag)` on the ETag of the version the inventory recorded: an S3 `If-Match` on the `DeleteObjectInput`, which the mirror does not send today (`internal/lease/mirror.go:787-800` deletes unconditionally; `release.S3Store.DeleteIfMatch`, `internal/release/conditional_delete.go:86-97`, does, for the deployment bucket only). A late or outstanding delete can then only put a marker over the inventoried version, never over a newer one, so the mark is lifted and the reservation released without settling outstanding requests. The one case the condition cannot tell apart, a later version with byte-identical content (equal ETag), is bounded by a 15-minute quiet after any run whose deletes were not all answered, and then repaired: after the quiet, and once more on the obligation's final pass, the box re-lists every key such a run deleted and removes, by its version id, a delete marker that a late cleanup delete put over a later owner's byte-identical version, so the hazard heals instead of merely being bounded. **F2:** `active.json` is never deleted. Every write carries a rising `rev` and the table's `epoch`, so no body repeats and no stale conditional PUT can match a later ETag. Every reservation write is `If-Match`, only the empty bootstrap is create-only, and a release always writes, so its success settles any earlier PUT of the same operation. **F5, F6:** the rename-aside protocol is removed. Root opens the parent, compares the recorded identity by `fstatat`, unlinks by `unlinkat` if it is equal and keeps the entry otherwise, under the registry host freeze and the held-stage lock, journalling intent and then result in a root-owned directory that root also walks on restart. A path whose parent chain a non-root uid can write is never removed by root ("writable layout"). The agent's unit writes `/data` (`internal/host/agent.go:1285`, `:4335`), so the programs file is recorded at `gone` as left in place, not requested. **F7:** supersession is checked at every lifecycle step, and root's `kept` and `cancelled` answers map to `superseded` and `cancelled` in one outcome table. **N5:** an untracked lifetime (incarnation 0 with no slot) gets no `retired.json`, and its gone report proceeds as before this plan. **N6, N9, N10, F3's timestamp, F12, N8, F13:** a corrupt stub has its own valid shape, and the first damaged file is kept as `retired.json.corrupt-1`; root reads the hibernate settings with the activator's own defaults; a placement writes one entry per host key; `RootRequestedAt` is restored from the drop's mtime or root's `taken` line; a one-shot page is persisted in `pages_pending` in the same write that makes its key terminal. **N7:** `carlos move offline status --host H --log` shows the cleanup reports and their events, independent of a self-serve row. **N11:** rollback stops the new console and removes any `create` or `place` entry left in the table by hand before the old console starts. Tasks 4, 6 and 7 are split, the CLI has its own task, and the end-to-end task is last. |
| -Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and allocation recovery (`internal/console/lifetimes_create.go`, `lifetimes_recovery.go`, `lifetimes_store.go`), `fenceRecord` (`lifetimes_create.go:551`), the judge (`provision.LifetimeJudge.Observe`), the reservation table (`moves.OfflineActive`, `ParseOfflineActive`, `OfflineActiveBody`, `OfflineSentinel`; the console's `reserveBoxes`/`releaseBoxes` and three-way reclaim in `internal/console/offlinereserve.go`), the lease's tombstone (`internal/lease/tombstone.go`), the offline source steps (`internal/host/movesoffline.go`), held publication's freeze and lock (`registry.WithHoldsFrozen`, `internal/host/movesheldlock.go`, `heldStageAuthority` in `internal/host/heldstage.go`), and the page outbox (`internal/host/movespaging.go`). |
| +Rev 4's closed answers stand: one state enum and one transition table (F3); a cleanup entry is never reclaimed by another reserver (F4); the programs file has no reference check and the live link keeps §7.6's link and process checks, with unreadable evidence retried (F8); root authorises through a host-level callback, so `internal/release` imports only the leaf `offlinecleanup` (F9); the reaper removes only directories whose obligation has ended (F10); the cleanup runs under its own `cleanupMu`, outside `w.pass` (F11); the prefix is normalised once (N1); `releasing` is persisted before the unmark and the release, and `cleaning` never re-acquires (N2); placement keeps `SetRecordRegion`'s conditional write, and `fenceRecord` settles an uncertain one (N3). Reviews: rev 1 (`after236/selfserve-plan4-review.md`), rev 2 (`selfserve-plan4-review2.md`), rev 3 (`selfserve-plan4-review3.md`), rev 4 (`selfserve-plan4-review4.md`). |
| -**Goal:** Twenty-four hours after a move the source box observed as finished, the source box retires the old region's replica objects from evidence it recorded at `gone`, under a `cleanup` reservation on its own box and on the hostname and a cleaning mark on the replica's lock, key by key with the four conditions re-read at each deletion, never deleting a version it did not inventory; root removes the old live link and programs file only through a request it copies, authorises from the bucket and answers in its own directory; the obligation's directory goes seven days after it ended; nothing times out into a release. |
| +Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and allocation recovery (`internal/console/lifetimes_create.go`, `lifetimes_recovery.go`, `lifetimes_store.go`), `fenceRecord` (`lifetimes_create.go:551`), the judge (`provision.LifetimeJudge.Observe`), the reservation table (`moves.OfflineActive`, `ParseOfflineActive`, `OfflineActiveBody`, `OfflineSentinel`; the console's `reserveBoxes`/`releaseBoxes` and three-way reclaim in `internal/console/offlinereserve.go`), the lease's tombstone (`internal/lease/tombstone.go`), the offline source steps (`internal/host/movesoffline.go`), held publication's freeze and lock (`registry.WithHoldsFrozen`, `internal/host/movesheldlock.go`, `heldStageAuthority` in `internal/host/heldstage.go`), the page outbox (`internal/host/movespaging.go`) and the offline CLI (`move.go`). |
| -**Architecture:** At `gone` the sealing source box writes `retired.json` under the move's op directory: the exact current versions under the normalised hibernate prefix (`lock.json` and `build-state.json` aside) and the identities of the two root-owned paths. A cleanup goroutine, started by the moves pass and never under its lock, walks every obligation through one transition table: observe the finish, wait 24 hours, reserve the box's key and `host/<hostname>`, mark `lock.json`, delete key by key with the four conditions re-read, then hand root a request (or skip it when nothing local remains), then persist `releasing`, restore the lifted tombstone, release and end. Supersession (the record moved on, a row appeared, the lock was taken, the reservation lost) ends the obligation through the same `releasing` step after any root request is cancelled and acknowledged. The lifetime create and placement hold `host/<hostname>` in the same table; a move's selection holds it too. |
| +**Goal:** Twenty-four hours after a move the source box observed as finished, the source box retires the old region's replica objects from evidence it recorded at `gone`, under a `cleanup` reservation on its own box and on the hostname and a cleaning mark on the replica's lock, key by key with the four conditions re-read at each deletion, and each deletion conditional at the server on the inventoried version's ETag. Root removes the old live link only through a request it copies, authorises from the bucket and answers in its own directory, and only where no other uid can write the link's parent. The obligation's directory goes seven days after it ended. Nothing times out into a release. |
| -**Tech Stack:** Go. `internal/offlinecleanup` (new leaf: the wire types), `internal/moves` (host keys, the conflict relation, `Reserve`/`Release`, `RecordAway`), `internal/lease` (the cleaning mark), `internal/host` (the dispatcher, the report, the reaper, root's authority and the agent's root half), `internal/release` (root's request handler), `internal/registry` (`WithHostFrozen`), `internal/console` (the create, placement, the reservation wrappers, recovery mode, the sweeps, the status door's log). |
| +**Architecture:** At `gone` the sealing source box writes `retired.json` under the move's op directory, for a tracked lifetime only: the exact current versions (key, version id, ETag) under the normalised hibernate prefix (`lock.json` and `build-state.json` aside), and the identities of the root-owned paths whose parent chain is root's alone (any other path is recorded as left). A cleanup goroutine, started by the moves pass and never under its lock, walks every obligation through one transition table: observe the finish, wait 24 hours, reserve the box's key and `host/<hostname>`, mark `lock.json`, delete key by key with the conditions re-read and the delete conditional on the ETag, hand root a request (or skip it when nothing local remains), then persist `releasing`, restore the lifted tombstone, release and end. Supersession is checked at every step and ends the obligation through the same `releasing`, after any root request is cancelled and acknowledged. The lifetime create and placement hold `host/<hostname>` in the same table, and so does a move's selection. The table is never deleted. |
| + |
| +**Tech Stack:** Go. `internal/offlinecleanup` (new leaf: the wire types, file identity and the layout check), `internal/moves` (host keys, the table's read and write, the conflict relation, `Reserve`/`Release`, `RecordAway`), `internal/lease` (the cleaning mark, the mirror's conditional delete), `internal/host` (the dispatcher, the report, the reaper, root's authority and the agent's root half), `internal/release` (root's request handler), `internal/registry` (`WithHostFrozen`), `internal/console` (the create, placement, the reservation wrappers, recovery mode, the sweeps, the status door's cleanup log), the root package (`move.go`, the CLI's `--log`). |
| **Spec:** `docs/superpowers/specs/2026-10-21-self-serve-offline-moves-design.md` §7.6, §7.1 (the `cleanup` reservation), §8 (the cleanup pages), §9 (rollout), §11. Stated deviations: placement and the lifetime create hold the hostname key (Paul, 2026-10-22, amadan #343); the rest are in the Self-review notes. |
| ## Global Constraints |
| -- **Arming and rollout.** The box acquires a new cleanup reservation only with `CARLOS_OFFLINE_CLEANUP=on` in its environment, on an `OfflineCapable` box whose judge is set (`cfg.Judge != nil`, i.e. host claims off, `internal/host/boxjudge.go:21-26`); `retired.json` is written at `gone` whatever the switch says, and an obligation past `finished` runs to its end if the switch is turned off. Order: boxes, then consoles (plan 4 code everywhere), then `CARLOS_OFFLINE_CLEANUP=on` on the boxes. Rollback: unset the switch, wait until `control/offline-moves/active.json` holds no `cleanup`, `create` or `place` entry (a console before this plan reads those kinds as a malformed table, `moves/offline.go:584-587`), then roll back. |
| -- Every data deletion is a plain `DeleteObject` through `cfg.PlainMirror` (`lease.NewS3Mirror(ctx, region, region)` for the source bucket's region); `VersionId` deletes are never used; `lock.json` and `build-state.json` are never inventoried and never deleted. |
| +- **Arming and rollout.** The box acquires a new cleanup reservation only with `CARLOS_OFFLINE_CLEANUP=on` in its environment, on an `OfflineCapable` box whose judge is set (`cfg.Judge != nil`, i.e. host claims off, `internal/host/boxjudge.go:21-26`); `retired.json` is written at `gone` whatever the switch says, and an obligation past `finished` runs to its end if the switch is turned off. Order: boxes, then consoles (plan 4 code everywhere), then `CARLOS_OFFLINE_CLEANUP=on` on the boxes. Before the switch goes on in any deployment, a check on dev confirms that a `DeleteObject` with a stale `If-Match` on the versioned hibernate bucket answers 412 and writes no delete marker (Task 4b's dev witness). **Rollback:** unset the box switch; stop the new console (only consoles write `create` and `place` entries, so none can be added after it stops; `cleanup` entries are accepted by the console before this plan, whose `valid()` takes `move` and `cleanup`, `moves/offline.go:584-587`, and `create` and `place` are the kinds it reads as a malformed table); read `control/offline-moves/active.json`; any `create` or `place` entry left is a stopped console's and is removed by hand before the old console starts: `aws s3api get-object --profile <profile> --bucket <deployment bucket> --key control/offline-moves/active.json active.json` (note the ETag), delete the entries from `reservations`, add 1 to `rev`, keep `epoch`, then `aws s3api put-object --profile <profile> --bucket <deployment bucket> --key control/offline-moves/active.json --body active.json --if-match <ETag>` (never `delete-object`); for a removed `create` entry the old console's recovery continues its allocation row as in plan 3; for a removed `place` entry the app's records are read and the placement is run again on the old console when they disagree. A self-serve move's `move` entries now include its host key, which the old console's own retry reads as a mismatch (`offlinereserve.go:177-183`), so roll back only when no self-serve move holds an entry. Only then start the old console. |
| +- **Tracked lifetimes only.** A cleanup obligation exists only for a tracked lifetime: the judged observation at `gone` is `LifetimeValid` with `Rec.Incarnation > 0` and a live slot at that incarnation. A record with incarnation 0 and no slot (an untracked lifetime, before the lazy migration; `provision.LifetimeOf`, `internal/provision/lifetime.go:127-130`, still answers it `LifetimeValid`), a migrating record, or any other verdict gets no `retired.json`, and its gone report proceeds exactly as before this plan; nothing is ever deleted for it, and its leftovers stay the runbook's. |
| +- **Every data deletion is conditional at the server.** A key is deleted only by `lease.ConditionalMirrorDeleter.DeleteIfMatch(ctx, bucket, key, etag)` on `cfg.PlainMirror` (`lease.NewS3Mirror(ctx, region, region)` for the source bucket's region), with the inventory's ETag for that key: S3 writes a delete marker only while the key's current version carries that ETag and answers 412 otherwise. `VersionId` deletes are never used on a data version (the one exception is the repair's removal of a delete marker, below), a plain `Delete` is never a fallback, and `lock.json` and `build-state.json` are never inventoried and never deleted. **The argument, once:** a delete issued under the conditions and still outstanding after them (a client timeout, a crash, a supersession, the unmark, the release) can land only on the version the inventory recorded; any later writer's version of that key carries another ETag unless its bytes are identical, so the late delete fails and the newer version stays current. That is why the mark may be lifted and the reservation released without settling outstanding requests. For the byte-identical case (a move back into the region, or a legacy promotion, copying the same LTX files back under the same keys), the bound is `QuietUntil`: set before the first delete of each run to the run's end plus `cleanupUncertainQuiet` (15 min), cleared at the end of a run in which every delete was answered, and `releasing` waits for it with the mark and the reservation held. It is a bound on how long the service can hold a request, not a proof, so a **repair** follows it: the keys a run with an unanswered delete issued deletes for are persisted (`RepairKeys`), and once the obligation has ended (so `QuietUntil` has passed and any later owner may have written) and again on its final pass before the reaper, each such key's versions are listed again. When the current version is a delete marker newer than `RepairSince` (the start of the first such run), the version directly beneath it is not the inventoried one, is newer than it and carries the inventoried ETag (the only version a late conditional delete of this cleanup could have landed on), the box removes **that marker** by its version id (`VersionedStore.DeleteVersion`, `internal/lease/versioned.go:261-275`) and records the key `restored`; when the newest version that is not a marker is the inventoried one, or any other shape is current, the key is `done` and nothing is touched. This marker removal is the one `VersionId` delete the plan allows, and it is safe: deleting a delete marker changes the current version only while that marker is current, and then the version directly beneath it becomes current; versions are only ever added on top, so that version is the later owner's newest, never one older than what was current before the late delete landed; a marker that is no longer current when the delete arrives (the owner wrote or deleted again after the listing) is removed without changing anything. The residue: an owner that itself deleted a byte-identical copy of an inventoried file after `RepairSince` gets that copy back once (litestream's retention removes it again on its next enforcement). The cost is one `ListVersions` (`ListObjectVersions` with the key as prefix) per repair key per repair pass, two passes, and only for obligations that had a run with an unanswered delete. |
| +- **The table is never deleted.** `control/offline-moves/active.json` is created once, by the first writer, as `{v: 1, epoch: <16 random bytes, hex>, rev: 1}` with `If-None-Match: *` (`release.Store.Put` with `ifMatch ""`, `internal/release/store.go:794-798`); emptiness is a table with no entries. Every later write is a `Put` with `If-Match` the ETag it read, of a body whose `rev` is the read body's plus one and whose `epoch` is the read body's (a fresh one when the read body has none, as an older console's writes do), so no two bodies the table ever holds are equal and an ETag never comes back. A conditional `Put` that read ETag E can land only while E is current, so once any later write succeeds, every earlier outstanding `Put` that read E can never land: an uncertain acquisition is settled by the next successful conditional write by anyone. A release always writes (rev + 1), even when its operation holds nothing, so its own success is the settlement its caller needs. |
| - **The prefix (N1).** `Prefix = activator.HibernatePrefix(cfg.OwnPrefix, host) + "/"` (`HibernatePrefix` is `path.Join`, no trailing slash, `internal/activator/activator.go:147`), computed once at `gone` and persisted; the listing is `ListVersions(bucket, Prefix)`; the excluded keys are exactly `Prefix + "lock.json"` and `Prefix + "build-state.json"`; the lease location is `strings.TrimSuffix(Prefix, "/")`. |
| - **The cleanup id** is `offlinecleanup.CleanupID(box, host, moveID) = "cu_" + hex(sha256(box + "\n" + host + "\n" + moveID))[:16]`: recomputable from the directory and the box, so a corrupt file, root and the agent all name the same obligation. |
| -- **The mark.** Owner `lease.CleaningOwner(id) = "cleaning:" + id`; TTL `cleanupMarkTTL = 1 h`; reasserted whenever less than 30 min remain; condition (2) requires at least `cleanupMarkMargin = 15 min` left (the margin exceeds the per-operation bound of 10 s by far, and covers clock skew as the lease's own `RenewMargin` does). Written only by a conditional PUT from the lifted tombstone of this move (`moves.TombstoneOwner(MoveID)`, expired); removed by a conditional PUT back to exactly that lifted tombstone (`{Generation: as read, ExpiresAt: 1970-01-01T00:00:00Z, Owner: moved:<move id>}`, what `LiftTombstone` writes), so the operator return's leftover set (spec §7.5) is unchanged after any cleanup. An expired mark whose bytes are unchanged still proves no lease was taken since, because every acquisition rewrites the object. |
| +- **The mark.** Owner `lease.CleaningOwner(id) = "cleaning:" + id`; TTL `cleanupMarkTTL = 1 h`; reasserted whenever less than 30 min remain, in every state from the mark to the unmark; condition (2) requires at least `cleanupMarkMargin = 15 min` left, a bound that keeps the mark from lapsing inside a run (the safety of a late delete is the condition on its ETag, above). Written only by a conditional PUT from the lifted tombstone of this move (`moves.TombstoneOwner(MoveID)`, expired); removed by a conditional PUT back to exactly that lifted tombstone (`{Generation: as read, ExpiresAt: 1970-01-01T00:00:00Z, Owner: moved:<move id>}`, what `LiftTombstone` writes), so the operator return's leftover set (spec §7.5) is unchanged after any cleanup. An expired mark whose bytes are unchanged still proves no lease was taken since, because every acquisition rewrites the object. |
| - **The writers the mark excludes** (each writer's check, by file): the wake (`internal/activator/backend_exec.go:701`, `lease.Acquire` in `ExecBackend.Acquire`, step 1 of `doWake` in `activator.go`, before the restore, the staged-replica promotion `promoteStagedReplica` at `backend_exec.go:3397`, `BeforeExec`'s build-state write and litestream replication, all of which run under that lease); the legacy final checkpoint (`internal/host/movespass.go:321-328`, `lease.ReplicateWithLease`, which acquires); the offline seal's checkpoint (`internal/host/movesoffline_seal.go:417`, the same `Checkpoint`) and its tombstone (`movesoffline_seal.go:431`, `lease.Tombstone`, which never replaces a live lease of another owner, so the seal's carry at `:469` is not reached). Not excluded by the mark and harmless: the legacy cut's request carry (`movespass.go:1594`) and the offline seal's carry write only `build-state.json`, which is not inventoried; the seed rollback and finish (`seedwiring.go:247`, `console/seedfinish.go:254`) only delete; the target landing (`movesoffline_land.go:611`, `:1373`) belongs to a move into this region, which holds `host/<hostname>`. `TestHibernatePrefixSitesAreInventoried` (Task 3) pins this list. |
| -- **Reservation keys and the one conflict relation.** Keys are a box id (today's bare label, unchanged), the sentinel `"*"`, or `moves.HostKey(host) = "host/" + host`. Kinds: `move`, `cleanup`, `create`, `place`. An existing entry of another `(Kind, ID)` conflicts with a requested set of keys when (a) it holds a requested key; or (b) one side holds `"*"` and the other holds a box key; or (c) the requester is the console with `cfg.SelfServeMoves != "offline"` and both are of kind `move` (plan 2's one-move-at-a-time rule, now between moves only). Host keys never meet (b) or (c). Every move selection (self-serve and operator) holds `host/<hostname>` beside its box keys or its sentinel; a cleanup holds `[cfg.BoxID, host/<hostname>]`; a create holds `[host/<hostname>]`; a placement holds `host/<h>` for every instance of the app. |
| +- **Reservation keys and the one conflict relation.** Keys are a box id (today's bare label, unchanged), the sentinel `"*"`, or `moves.HostKey(host) = "host/" + host`. Kinds: `move`, `cleanup`, `create`, `place`. An entry is valid with a known kind, a non-empty id and host, a non-zero `reserved_at`, and, for `create` and `place`, a non-empty account and app; an entry under a host key names that host (`Host == strings.TrimPrefix(key, "host/")`). An existing entry of another `(Kind, ID)` conflicts with a requested set of keys when (a) it holds a requested key; or (b) one side holds `"*"` and the other holds a box key; or (c) the requester is the console with `cfg.SelfServeMoves != "offline"` and both are of kind `move` (plan 2's one-move-at-a-time rule, now between moves only). Host keys never meet (b) or (c). Every move selection (self-serve and operator) holds `host/<hostname>` beside its box keys or its sentinel; a cleanup holds `[cfg.BoxID, host/<hostname>]` with one entry; a create holds `[host/<hostname>]`; a placement holds `host/<h>` for every instance of the app, one entry per key with `Host = h` and the placement's account and app. |
| - **A cleanup entry is never reclaimed by another reserver.** Its own box resumes it after any crash (its own retry is accepted); a box that never returns leaves an entry the console pages after 6 hours (operator repair). Plan 2's `Server.cleanupReports` and `cleanupReportEvidence` (`offlinereserve.go:95-104`) are deleted. |
| -- **The four conditions at every deletion**, re-read before each `Delete`: (1) `moves.RecordAway(obs, MoveID, Incarnation, From) == AwayYes`: the judged observation is `LifetimeValid` (the judge's verdict that the slot vouches for the record) at `Incarnation`, with no move, `LastMove.ID == MoveID && LastMove.Outcome == "done"`, and `Region != From`; (2) `lock.json`'s owner is `CleaningOwner(ID)` with at least `cleanupMarkMargin` left after any reassert; (3) the box's registry has no row for the host (`w.offlineRow(host)`, `movesoffline.go:996`); (4) `ListVersions(bucket, key)` filtered to `Key == key` shows the inventory's `VersionID` as `IsLatest` and not a delete marker. A read error in any: the obligation stops for this run. (1), (2) or (3) failing definitely: **supersession** (spec §7.6's "superseded as a whole"; never an indefinite refusal). (4): another current version → that key `superseded` and a page; a delete marker current → `marked`; no version and no marker → `absent`. |
| -- The cleanup writes nothing to the registry, the edges file or the offline view, so plan 3's holding entries (built from held rows and the view) are untouched; root's freeze holds the registry's write lock for a rename and an unlink, as held publication's does. |
| -- **Root's contract.** Root removes a path only inside `reg.WithHostFrozen(host, …)` (no row for the host) and then the host's held-stage lock (`withHeldStageLock`, `movesheldlock.go:68`), in the order adopt pass lock > the registry's write lock > the held-stage lock (`movesheldlock.go:26-31`), after the authority callback has answered yes for this request in this pass and `root-cleanup.cancel` is absent, and only the entry it renamed aside with `RENAME_NOREPLACE` and found carrying the recorded identity (device, inode, mode, link target). Nothing inside the held-stage lock reads the network or takes a lock. |
| -- Timings: 24 h after `FinishedAt`; per run 60 s (`cleanupRunDeadline`), 10 s per operation, at most 50 keys per obligation; pages every 10 min while root has not answered or has not acknowledged a cancel, hourly for a corrupt obligation; the console pages a `cleanup` entry older than 6 h and a `create` entry with no allocation row older than `offlineLockGrace` (2 min); a crashed placement is settled after `offlineLockGrace`; the op directory goes 7 d after `EndedAt`; root removes its own directory 30 d after its acknowledgement. |
| -- Census: `internal/provision/testdata/plan14-root-reads-self-serve-moves-4.tsv`, loaded by a new `loadRootReadInventory(root+"/internal/provision/testdata/plan14-root-reads-self-serve-moves-4.tsv", &inv, true)` block in `internal/provision/plan14_root_reads_test.go` after plan 3's block (`:1383`) and its seal check and before `compareRootReads`, `rev3=PROPOSED-self-serve-moves-4`, rows at the `authority` class with evidence `E2,J3,J4` for the destructive readers (the box's record, registry, lock and versions reads; root's drop, `active.json`, record, lock, registry-under-freeze and journal reads); `TestPlan9B8RecordWriteCensus` (`plan9_b8_census_test.go:372`) takes `plan14SelfServeMoves4WriteCensusInventory()` wrapping `plan14SelfServeMoves3WriteCensusInventory()` (`self_serve_moves3_census_test.go:22`) with every new or changed bucket writer as an added or re-pinned unit (`moves.Reserve`, `moves.Release`, `reserveBoxes`, `releaseBoxes`, the cleanup report writer; the placement settle and the place sweep reach `fenceRecord`'s existing record unit); both green. |
| -- Copy rows (pending Paul's review): "the previous instance at this hostname is still being cleaned up. try again later"; "an instance of this app is still being cleaned up. try again later"; "an instance of this app is moving. try again later"; "could not reserve the app's instances. try again"; page subjects "cleanup of <host> waits for root", "cleanup of <host> skipped <key>: another version is current", "cleanup of <host> waits for root to acknowledge its cancellation", "cleanup of <host> cannot read its record", "a cleanup has held <host> for over six hours", each with the body "run carlos move offline status --host <host> --log to see what it waits for". No tenant names in tests (`shop.example.com`, account `bab`, app `shop`). |
| +- **The four conditions at every deletion**, re-read before each `DeleteIfMatch`: (1) `moves.RecordAway(obs, MoveID, Incarnation, From) == AwayYes`: the judged observation is `LifetimeValid` (the judge's verdict that the slot vouches for the record) at `Incarnation`, with no move, `LastMove.ID == MoveID && LastMove.Outcome == "done"`, and `Region != From`; (2) `lock.json`'s owner is `CleaningOwner(ID)` with at least `cleanupMarkMargin` left after any reassert; (3) the box's registry has no row for the host (`w.offlineRow(host)`, `movesoffline.go:996`); (4) `ListVersions(bucket, key)` filtered to `Key == key` shows the inventory's `VersionID` as `IsLatest` and not a delete marker. A read error in any: the obligation stops for this run. (1), (2) or (3) failing definitely: **supersession** (spec §7.6's "superseded as a whole"; never an indefinite refusal). (4): another current version → that key `superseded` and a page; a delete marker current → `marked`; no version and no marker → `absent`. Conditions (1) and (3) are also checked at every other step of the lifecycle (Task 4a's `supersededBy`). |
| +- The cleanup writes nothing to the registry, the edges file or the offline view, so plan 3's holding entries (built from held rows and the view) are untouched; root's freeze holds the registry's write lock for an `fstatat` and an `unlinkat`, as held publication's does for its swap. |
| +- **Root's contract.** Root removes a path only (i) when the path's parent directory and every ancestor up to `/` is a directory owned by uid 0 with no group or other write bit (`offlinecleanup.RootOnlyChain`; otherwise the path is `kept`, reason "writable layout", because no exclusion against a writer that shares the agent's uid is possible there by construction); (ii) inside `reg.WithHostFrozen(host, …)` (no row for the host) and then the host's held-stage lock (`withHeldStageLock`, `movesheldlock.go:68`), in the order adopt pass lock > the registry's write lock > the held-stage lock (`movesheldlock.go:26-31`); (iii) after the authority callback has answered yes for this request in this pass and `root-cleanup.cancel` is absent; (iv) when `fstatat(parent, name, AT_SYMLINK_NOFOLLOW)` on the opened parent shows the recorded identity (device, inode, file type, link target); then `unlinkat(parent, name)`. An entry with another identity is `kept`, never removed. Every step is journalled as intent and then result in root's own directory, which is used only when it passes the same root-only check. Nothing inside the held-stage lock reads the network or takes a lock. |
| +- Timings: 24 h after `FinishedAt`; per run 60 s (`cleanupRunDeadline`), 10 s per operation, at most 50 keys per obligation; `cleanupUncertainQuiet` 15 min; pages every 10 min while root has not answered or has not acknowledged a cancel, hourly for a corrupt obligation and for a record unreadable for over an hour; the console pages a `cleanup` entry older than 6 h and a `create` entry with no allocation row older than `offlineLockGrace` (2 min); a crashed placement is settled after `offlineLockGrace`; the op directory goes 7 d after `EndedAt`; root removes its own directory 30 d after its acknowledgement. |
| +- Census: `internal/provision/testdata/plan14-root-reads-self-serve-moves-4.tsv`, loaded by a new `loadRootReadInventory(root+"/internal/provision/testdata/plan14-root-reads-self-serve-moves-4.tsv", &inv, true)` block in `internal/provision/plan14_root_reads_test.go` after plan 3's block (`:1383`) and its seal check and before `compareRootReads`, `rev3=PROPOSED-self-serve-moves-4`, rows at the `authority` class with evidence `E2,J3,J4` for the destructive readers (the box's record, registry, lock and versions reads; root's drop, its own request copy and journal, `active.json`, record, lock, registry-under-freeze, the layout `lstat` chain and the `fstatat` of each path); `TestPlan9B8RecordWriteCensus` (`plan9_b8_census_test.go:372`) takes `plan14SelfServeMoves4WriteCensusInventory()` wrapping `plan14SelfServeMoves3WriteCensusInventory()` (`self_serve_moves3_census_test.go:22`) with every new or changed bucket writer as an added or re-pinned unit (`moves.ReadActive`'s bootstrap, `moves.Reserve`, `moves.Release`, `reserveBoxes`, `releaseBoxes`, the cleanup report writer, the cleanup's `DeleteIfMatch`, the repair's `DeleteVersion` of a marker; the placement settle and the place sweep reach `fenceRecord`'s existing record unit); both green. |
| +- Copy rows (pending Paul's review; each task that emits one appends its row, at the next free `MV-O` id, status pending, to `docs/superpowers/plans/2026-10-06-moves-phase1-copy.md` and its id to `movesCopyIDs` in `approved_moves_copy_test.go`, in the same commit): "the previous instance at this hostname is still being cleaned up. try again later"; "an instance of this app is still being cleaned up. try again later"; "an instance of this app is moving. try again later"; "could not reserve the app's instances. try again"; page subjects "cleanup of <host> waits for root", "cleanup of <host> skipped <key>: another version is current", "cleanup of <host> waits for root to acknowledge its cancellation", "cleanup of <host> cannot read its record", "a cleanup has held <host> for over six hours", each with the body "run carlos move offline status --host <host> --log to see what it waits for"; the CLI's lines of Task 5b; the `--log` paragraph of `docs/site/cli.md`. No tenant names in tests (`shop.example.com`, account `bab`, app `shop`). |
| - Local tests: touched packages with `-run`; CI runs the full suite and `go test -tags e2e ./internal/e2e/...`. |
| ## Review Focus |
| -1. The F1 interleaving: a reconcile paused with an old observation mints the source row after the cleanup marked the lock, and a wake through `ExecBackend.Acquire` then tries the lease: it gets `HeldError` naming `cleaning:<id>`, no version is written under the prefix, the next key's condition (3) supersedes, root is cancelled and acknowledged, the lifted tombstone is restored, the reservation released, and the wake then succeeds (Task 4, `TestAdmittedReconcileAndWakeCannotWriteUnderTheMark`). |
| -2. A crash after every write and every side effect of the lifecycle (after `Reserve` before `cleaning`, after `cleaning` before the mark, after the mark before `MarkedAt`, after a `Delete` before its key outcome, after `releasing` before the unmark, after the unmark before the release, after the release before `ended`), restart, completion: no key is deleted unless both entries and the mark are found to be this cleanup's, an entry an operator removed or a mark replaced turns the obligation `superseded`, and nothing is re-acquired from `cleaning` (Task 4, `TestLifecycleResumesAtEveryBoundary`). |
| -3. Root under attack and crash: a shared-uid writer plants an acknowledgement, a request with another id and a cancel in the agent's directory; an agent publication replaces the live link between root's reference check and its rename, and again after the rename; a route is upserted during the freeze; root crashes after each journal line and each syscall: root removes only the entry it took and verified, the replacement survives, the upsert waits for the freeze, every intent resolves to removed, kept or absent, and the forged files change nothing but stopping (Task 6, `TestRootRemovesOnlyTheEntryItTook`, `TestRootJournalResolvesEveryIntent`, `TestForgedAgentFilesAreHarmless`). |
| -4. Residence changes and cleanup exclude each other through `host/<hostname>`: a create (busy → 409, row void, nothing written), a crash after `BeginAllocation` and after `Reserve`, a terminal row whose release failed; a placement that is busy (nothing written), whose record PUT commits after the client gave up (fenced before release) and whose console crashed (the sweep fences and releases); an offline move selection; with the switch off two creates of different hosts and a cleanup beside an operator move all proceed, and the sentinel never blocks a create (Tasks 2 and 7, `TestConflictRelation`, `TestResidenceChangesAndCleanupExclude`). |
| -5. Publication is durable: the report under the move id survives failed writes and is republished until `PublishedRev == Rev`; a refusal repeated every run is one event with a count; a page fires once per 10-minute tick across restarts; the reaper waits for the report, the pages and root's acknowledgement (Tasks 5 and 8, `TestCleanupReportIsDurableAndBlocksTheReaper`). |
| +1. The F1 and N4 interleavings: a reconcile paused with an old observation mints the source row after the cleanup marked the lock, and a wake through `ExecBackend.Acquire` then tries the lease: it gets `HeldError` naming `cleaning:<id>`, no version is written under the prefix, the next key's condition (3) supersedes, the lifted tombstone is restored, the reservation released, and the wake then succeeds. A delete issued before the supersession and held by the service lands after the wake wrote a new version of its key: the service refuses it (412), the new version stays current; with byte-identical bytes the release waits out `QuietUntil` first, and a delete that lands even later on a byte-identical re-promoted version is repaired on the next pass with that newer version current again (Task 4b, `TestAdmittedReconcileAndWakeCannotWriteUnderTheMark`, `TestOutstandingDeleteCannotHideANewerVersion`, `TestByteIdenticalRewriteWaitsOutTheQuiet`, `TestLateDeleteOnAByteIdenticalVersionIsRepaired`). |
| +2. A crash after every write and every side effect of the lifecycle (after `Reserve` before `cleaning`, after `cleaning` before the mark, after the mark before `MarkedAt`, after `QuietUntil` before a delete, after a delete before its key outcome, after the last key before `publishing`, after `releasing` before the unmark, after the unmark before the release, after the release before `ended`), restart, completion: no key is deleted unless both entries and the mark are found to be this cleanup's, an entry an operator removed or a mark replaced turns the obligation `superseded`, nothing is re-acquired from `cleaning`, and the outcome is the table's (Task 4a, `TestLifecycleResumesAtEveryBoundary`). |
| +3. Root under attack and crash: a shared-uid writer plants an acknowledgement, a request with another id, a changed copy of the request and a cancel in the agent's directory; a publication replaces the live link between root's reference check and its `fstatat`; a route is upserted during the freeze; the link's parent chain is writable; root crashes after each journal line and each syscall: root removes only an entry carrying the recorded identity, the replacement survives, the upsert waits for the freeze, the writable layout is never touched, every intent resolves to removed, kept or absent, root finds its own taken request on restart, and the forged files change nothing but stopping (Tasks 6b and 6c, `TestRootRemovesOnlyTheRecordedIdentity`, `TestWritableLayoutIsNeverRemoved`, `TestRootJournalResolvesEveryIntent`, `TestRootFindsItsOwnRequestsOnRestart`, `TestForgedAgentFilesAreHarmless`). |
| +4. Residence changes and cleanup exclude each other through `host/<hostname>`: a create (busy → 409, row void, no reservation written), a crash after `BeginAllocation` and after `Reserve`, a terminal row whose release failed, a create's reservation PUT that lands after its recovery voided and released it (refused by its condition); a placement that is busy (nothing written), whose record PUT commits after the client gave up (fenced before release) and whose console crashed (the sweep fences and releases); an offline move selection; with the switch off two creates of different hosts and a cleanup beside an operator move all proceed, and the sentinel never blocks a create (Tasks 2, 7a and 7b, `TestConflictRelation`, `TestUncertainReservationIsSettledByTheNextWrite`, `TestResidenceChangesAndCleanupExclude`). |
| +5. Publication is durable: the report under the move id survives failed writes and is republished until `PublishedRev == Rev`; a refusal repeated every run is one event with a count; a skipped key's page survives an outbox that cannot create it, a pager that refuses it and a restart; a periodic page fires once per tick across restarts; the reaper waits for the report, `pages_pending`, the outbox and root's acknowledgement (Tasks 5a and 8, `TestCleanupReportIsDurableAndBlocksTheReaper`, `TestPendingPagesSurviveARestart`). |
| --- |
| ### Task 1: The obligation and its evidence at `gone` |
| **Files:** |
| -- Create: `internal/offlinecleanup/wire.go` (standard library only; every wire type of this plan, `Retired`, `InventoryObject`, `KeyOutcome`, `Event`, `Report`, `FileIdentity`, `LocalPath`, `RootCleanupRequest`, `RootCleanupPath`, `RootCleanupAck`, `KeptPath`, with Tasks 5 and 6 giving the last six their semantics), `internal/host/movescleanup_retired.go`, `internal/host/fileid_linux.go`, `internal/host/fileid_other.go` |
| -- Modify: `internal/moves/offline.go` (`OfflineOpDir`, `RetiredFile`, `CleanupReportKey`), `internal/host/movesoffline.go` (`offlineSourceGone`, line 1493: with `rt == nil` and this box the move's sealer by `cfg.offlineRoles(m)`, `w.recordRetired(ctx, s, rec)` runs before the gone report; an error writes no gone report this pass), `internal/release/programhistory.go` (export `ProgramsPath(bakeDir, host)`, today's `programPath`) |
| -- Test: `internal/offlinecleanup/wire_test.go`, `internal/host/movescleanup_retired_test.go` |
| +- Create: `internal/offlinecleanup/wire.go` (standard library only; every wire type of this plan, `Retired`, `CorruptStub`, `InventoryObject`, `KeyOutcome`, `Event`, `PendingPage`, `Report`, `FileIdentity`, `LocalPath`, `KeptPath`, `RootCleanupRequest`, `RootCleanupPath`, `RootCleanupAck`, with Tasks 5a and 6b giving the last ones their semantics), `internal/offlinecleanup/layout_linux.go` and `layout_other.go` (`Identify`, `RootOnlyChain`), `internal/host/movescleanup_retired.go` |
| +- Modify: `internal/moves/offline.go` (`OfflineOpDir`, `RetiredFile`, `CleanupReportKey`), `internal/host/movesoffline.go` (`offlineSourceGone`, line 1493: with `rt == nil` and this box the move's sealer by `cfg.offlineRoles(m)`, `w.recordRetired(ctx, s, rec)` runs before the gone report; an error writes no gone report this pass; "untracked" is not an error), `internal/release/programhistory.go` (export `ProgramsPath(bakeDir, host)`, today's `programPath`) |
| +- Test: `internal/offlinecleanup/wire_test.go`, `internal/offlinecleanup/layout_linux_test.go`, `internal/host/movescleanup_retired_test.go` |
| **Interfaces:** |
| - `func CleanupID(box, host, moveID string) string` as the Global Constraint; `const MarkPrefix = "cleaning:"`. |
| -- `type Retired struct { V int; ID, MoveID, Host, Account, App, Box string; Incarnation uint64; From, To string; OpenAt time.Time; FinalTXID uint64; TombstoneOwner string; Bucket, BucketRegion, Prefix string; BucketInventory []InventoryObject; LocalInventory []LocalPath; WrittenAt time.Time; State, Outcome string; FinishedAt, ReservedAt, MarkedAt, RootRequestedAt, CancelRequestedAt, EndedAt time.Time; Keys map[string]KeyOutcome; Events []Event; Root *RootCleanupAck; Rev, PublishedRev uint64 }`, every field tagged snake_case, times `omitzero`, `V = 1`. |
| -- **States** (`State`, the only lifecycle field): `open`, `finished`, `cleaning`, `publishing`, `root-requested`, `cancel-pending`, `releasing`, `ended`, `corrupt`. `Outcome` is "" until `releasing`, then `cleaned` or `superseded`. |
| -- `type InventoryObject struct { Key, VersionID, ETag string; Size int64 }`; `type KeyOutcome struct { State string (pending, deleted, marked, superseded, absent); VersionID string; At time.Time }`; `type Event` (Task 5); `type FileIdentity struct { Dev, Inode uint64; Mode string (symlink or regular); LinkTarget string }`; `type LocalPath struct { Which string (live-link or programs-file); Identity FileIdentity }` (root derives the path itself; the agent's file names none). |
| -- `ValidateRetired(r Retired) error`: `V == 1`; `ID == CleanupID(Box, Host, MoveID)`; `MoveID` a plain segment; `Host` passes the host shape; `Account`, `App`, `Box` non-empty; `Incarnation > 0`; `From != To`; `State` in the set; `Outcome` "" before `releasing` and in the set after; `WrittenAt` non-zero; `Prefix` ends with `/`; inventory keys unique, under `Prefix`, neither excluded key; `Which` values unique and known; every `Keys` entry names an inventory key. |
| +- `type Retired struct { V int; ID, MoveID, Host, Account, App, Box string; Incarnation uint64; From, To string; OpenAt time.Time; FinalTXID uint64; TombstoneOwner string; Bucket, BucketRegion, Prefix string; BucketInventory []InventoryObject; LocalInventory []LocalPath; LocalLeft []KeptPath; WrittenAt time.Time; State, Outcome string; FinishedAt, ReservedAt, MarkedAt, QuietUntil, RepairSince, RootRequestedAt, CancelRequestedAt, EndedAt time.Time; RunKeys, RepairKeys []string; RepairPasses int; Keys map[string]KeyOutcome; Events []Event; PagesPending []PendingPage; Root *RootCleanupAck; Rev, PublishedRev uint64 }`, every field tagged snake_case, times `omitzero`, `V = 1`. |
| +- **States** (`State`, the only lifecycle field): `open`, `finished`, `cleaning`, `publishing`, `root-requested`, `cancel-pending`, `releasing`, `ended`, `corrupt`. `Outcome` is "" until `releasing`, then `cleaned`, `superseded` or `cancelled` (Task 4a's outcome table). |
| +- `type InventoryObject struct { Key, VersionID, ETag string; Size int64 }` (`ETag` unquoted, as `lease.ObjectVersion` carries it, `internal/lease/versioned.go:25-33`); `type KeyOutcome struct { State string (pending, deleted, marked, superseded, absent); VersionID string; At time.Time }`; `type Event`, `type PendingPage` (Task 5a); `type FileIdentity struct { Dev, Inode uint64; Mode string (symlink or regular); LinkTarget string }`; `type LocalPath struct { Which string (live-link or programs-file); Identity FileIdentity }` (root derives the path itself; the agent's file names none); `type KeptPath struct { Which, Reason string }`. |
| +- `func Identify(path string) (FileIdentity, bool, error)`: `Lstat` (never follows), `Readlink` for a symlink; absent → `false`; any other error fails the step. `func RootOnlyChain(dir string) (bool, error)`: `Lstat` of every component from `/` to `dir`: each a directory (a symlink component answers false), owned by uid 0, `mode & 0o022 == 0`. Off Linux both answer absent and false (the cleanup is Linux-only, as the box is). |
| +- `ValidateRetired(r Retired) error`: `V == 1`; `ID == CleanupID(Box, Host, MoveID)`; `MoveID` a plain segment; `Host` passes the host shape; `Account`, `App`, `Box` non-empty; `Incarnation > 0`; `From != To`; `State` in the set and not `corrupt`; `Outcome` "" before `releasing` and in the set after; `WrittenAt` non-zero; `Prefix` ends with `/`; inventory keys unique, under `Prefix`, neither excluded key; `ETag` and `VersionID` non-empty for each; `Which` values unique across `LocalInventory` and `LocalLeft`, and known; every `Keys` entry, `RunKeys` and `RepairKeys` entry names an inventory key; `RepairPasses` in 0 to 2. |
| +- `type CorruptStub struct { Corrupt bool; ID, Host string; FoundAt time.Time }` (`{corrupt: true, id, host, found_at}`); `ValidateCorruptStub(c, box, host, move string) error`: `Corrupt`, `Host == host`, `ID == CleanupID(box, host, move)`, `FoundAt` non-zero. |
| - `func OfflineOpDir(dataDir, host, op string) string` (`<dataDir>/.offline-moves/<host>/<op>`, "" on a bad name, the checks `OfflineParkDir` applies); `func RetiredFile(dataDir, host, op string) string`; `func CleanupReportKey(host, op, box string) string` (`OfflineReportPrefix(host, op) + "cleanup/" + box + ".json"`). |
| -- **The evidence at `gone`** (`recordRetired`): the record is in `offline-finishing` carrying this move, and `cfg.Region == m.From` (else nothing is written and the box logs once per day); `OpenAt = *Move.OpenedAt`; `FinalTXID` from the seal at `moves.OfflineSealKey(host, id)` validated by `moves.ValidateOfflineSeal`; `TombstoneOwner = moves.TombstoneOwner(id)`; `Bucket/BucketRegion = cfg.OwnBucket/OwnBucketRegion`; `Prefix` as the Global Constraint; `BucketInventory` = `cfg.Versioned(ctx, Bucket, BucketRegion).ListVersions(ctx, Bucket, Prefix)` keeping `IsLatest && !DeleteMarker && strings.HasPrefix(Key, Prefix)` without the two excluded keys; `LocalInventory` from `fileIdentity` on `release.LiveLink(cfg.OptDir, host)` and `release.ProgramsPath(cfg.DataDir, host)`; `State = open`, `Rev = 1`. |
| -- `fileIdentity(path) (FileIdentity, bool, error)`: `Lstat` (never follows), `Readlink` for a symlink; absent → `false`; any other error fails the step. Off Linux it answers absent for both (the cleanup is Linux-only, as the box is). |
| -- Writes: `createRetired` is create-only (temp file, `fsync`, `os.Link` into place, directory `fsync`; an existing file is never replaced); `writeRetired` (temp, `fsync`, rename, directory `fsync`) increments `Rev` on every write except `markPublished` (Task 5), which sets `PublishedRev` alone. |
| -- **Corrupt:** a file that does not decode or fails `ValidateRetired` is renamed `retired.json.corrupt` and replaced by the stub `{v, id (CleanupID from the box and the directory), move_id, host, box, state: corrupt, written_at}`, which keeps the obligation listed and pages hourly (Task 5) until an operator repairs it; a `corrupt` obligation is never reaped and never released, so any entry or mark it held stays until the repair. |
| +- **The evidence at `gone`** (`recordRetired`): the record is in `offline-finishing` carrying this move, `cfg.Judge != nil`, and `cfg.Region == m.From` (else nothing is written and the box logs once per day). **Tracked (N5):** `obs := cfg.Judge.Observe(ctx, s, rec.Account, rec.App, host)`; `LifetimeUnreadable` → an error (the step retries next pass, as any read in it does); `obs.Verdict == LifetimeValid && obs.Rec.Incarnation > 0 && obs.Slot != nil && obs.Slot.Incarnation == obs.Rec.Incarnation` → tracked; any other answer (incarnation 0 with no slot, a migrating record, stale, corrupt) → untracked: no `retired.json`, logged once per day, and `recordRetired` returns nil so the gone report is written as before this plan. Then: `OpenAt = *Move.OpenedAt`; `FinalTXID` from the seal at `moves.OfflineSealKey(host, id)` validated by `moves.ValidateOfflineSeal`; `TombstoneOwner = moves.TombstoneOwner(id)`; `Bucket/BucketRegion = cfg.OwnBucket/OwnBucketRegion`; `Prefix` as the Global Constraint; `BucketInventory` = `cfg.Versioned(ctx, Bucket, BucketRegion).ListVersions(ctx, Bucket, Prefix)` keeping `IsLatest && !DeleteMarker && strings.HasPrefix(Key, Prefix)` without the two excluded keys, with each version's id and ETag; for each of `release.LiveLink(cfg.OptDir, host)` and `release.ProgramsPath(cfg.DataDir, host)`: absent → nothing; present and `RootOnlyChain(filepath.Dir(path))` → `LocalInventory`; present otherwise → `LocalLeft` with reason "writable layout" (on every built box the programs file is here: the agent's unit has `ReadWritePaths=/data`); `State = open`, `Rev = 1`. |
| +- Writes: `createRetired` is create-only (temp file, `fsync`, `os.Link` into place, directory `fsync`; an existing file is never replaced); `writeRetired` (temp, `fsync`, rename, directory `fsync`) increments `Rev` on every write except `markPublished` (Task 5a), which sets `PublishedRev` alone. |
| +- **Corrupt (N6):** a file that is a valid `CorruptStub` stays as it is, never rewritten. A file that is neither a valid `Retired` nor a valid stub is moved aside create-only (`os.Link` then remove): to `retired.json.corrupt-1` when that name is free, else to the next free `retired.json.corrupt-<n>`, so the first damaged file is never overwritten; then the stub is written with `FoundAt` = `retired.json.corrupt-1`'s mtime. A stub keeps the obligation listed as `corrupt` and pages hourly from `FoundAt` (Task 5a) until an operator repairs it (a valid `retired.json` restored from `.corrupt-1`, or the directory removed after the entries and the mark are released by hand); it is never reaped and never released, so any entry or mark it held stays until the repair. |
| -- [ ] **Step 1: Write the failing tests**: `TestRetiredJSONWrittenOnceAtGoneBySealer` (fields; `FinalTXID` from the seal; a non-sealer source box writes nothing; a second gone pass does not rewrite; an identity error writes nothing and no gone report); `TestRetiredPrefixIsNormalisedOnce` (`cfg.OwnPrefix = "instances"` → `Prefix == "instances/shop.example.com/"`; the fake versioned store was listed with exactly that prefix; `instances/shop.example.com/lock.json` and `.../build-state.json` excluded; `instances/shop.example.co` keys not listed; `ValidateRetired` passes on what gone wrote); `TestCleanupIDIsDeterministic`; `TestValidateRetired` (each rule); `TestCorruptRetiredStaysAnObligation` (renamed, stub listed `corrupt` with the recomputed id); `TestFileIdentityUsesLstat` (a dangling link has an identity; a regular file; absent). |
| -- [ ] **Step 2: Run to verify they fail**: `go test ./internal/offlinecleanup -count=1`, `go test ./internal/host -run 'RetiredJSON|RetiredPrefix|CorruptRetired|FileIdentity' -count=1`. |
| +- [ ] **Step 1: Write the failing tests**: `TestRetiredJSONWrittenOnceAtGoneBySealer` (fields, each inventory object with its ETag; `FinalTXID` from the seal; a non-sealer source box writes nothing; a second gone pass does not rewrite; an identity error writes nothing and no gone report); `TestUntrackedLifetimeWritesNoRetiredAndReportsGone` (incarnation 0 with no slot, a migrating record, a stale verdict: no file, the gone report written; an unreadable lifetime: no file and no gone report this pass, both next pass); `TestRetiredPrefixIsNormalisedOnce` (`cfg.OwnPrefix = "instances"` → `Prefix == "instances/shop.example.com/"`; the fake versioned store was listed with exactly that prefix; `instances/shop.example.com/lock.json` and `.../build-state.json` excluded; `instances/shop.example.co` keys not listed; `ValidateRetired` passes on what gone wrote); `TestWritablePathsAreLeftNotRequested` (a path under a directory the test uid owns → `LocalLeft`, "writable layout"; a path under a root-only chain the test cannot build is covered by injecting `RootOnlyChain`); `TestCleanupIDIsDeterministic`; `TestValidateRetired` (each rule); `TestCorruptStubIsValidAndKeepsTheFirstDamage` (damaged → `.corrupt-1` and a stub that validates; a second scan changes nothing; a damaged stub → `.corrupt-2`, `FoundAt` still `.corrupt-1`'s mtime); `TestIdentifyUsesLstat` (a dangling link has an identity; a regular file; absent); `TestRootOnlyChain` (`/usr` true on a normal system; a temp directory false; a symlink component false). |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/offlinecleanup -count=1`, `go test ./internal/host -run 'RetiredJSON|Untracked|RetiredPrefix|WritablePaths|CorruptStub' -count=1`. |
| - [ ] **Step 3: Implement. Step 4: Run to verify they pass.** |
| -- [ ] **Step 5: Commit** `moves: the sealing source box records retired.json at gone, with a normalised prefix and a deterministic cleanup id`. |
| +- [ ] **Step 5: Commit** `moves: the sealing source box records retired.json at gone for a tracked lifetime, with ETags, a normalised prefix and a deterministic cleanup id`. |
| -### Task 2: Hostname keys and one conflict relation |
| +### Task 2: Hostname keys, a table that is never deleted, and one conflict relation |
| **Files:** |
| - Create: `internal/moves/reserve.go` |
| -- Modify: `internal/moves/offline.go` (`OfflineKindCreate = "create"`, `OfflineKindPlace = "place"`; `OfflineReservation.valid()` accepts the four kinds; `HostKey`, `IsHostKey`), `internal/console/lifetimes_store.go` (`AllocationState(opID) (state string, found bool, err error)`), `internal/console/offlinereserve.go` (`reserveBoxes`' steps (4) and (5) become one call to `moves.Conflicts`; the wanted keys add `moves.HostKey(r.Host)`; `releaseBoxes` wraps `moves.Release`; `offlineReclaimable`: `cleanup` → `reclaimHeld` always, `create` and `place` below; `cleanupReportEvidence` and `Server.cleanupReports` deleted), `internal/console/selfserve_pass.go` (`SelfServeEvidence`, lines 41-53, counts only entries of kind `move`), `internal/console/lifetimes_recovery.go` (each iteration also runs `s.residenceSweep(ctx, now)`), `internal/console/selfserve_pages.go` (two page conditions) |
| +- Modify: `internal/moves/offline.go` (`OfflineKindCreate = "create"`, `OfflineKindPlace = "place"`; `OfflineActive` gains `Epoch string \`json:"epoch,omitempty"\`` and `Rev uint64 \`json:"rev,omitempty"\``; `OfflineReservation.valid()` and `ParseOfflineActive` take the Global Constraint's entry rules; `HostKey`, `IsHostKey`), `internal/console/lifetimes_store.go` (`AllocationState(opID) (state string, found bool, err error)`), `internal/console/offlinereserve.go` (`reserveBoxes` reads through `moves.ReadActive` and writes `moves.NextActiveBody` with `If-Match`; its steps (4) and (5) become one call to `moves.Conflicts`; the wanted keys add `moves.HostKey(r.Host)`; `releaseBoxes` wraps `moves.Release`, so the `DeleteIfMatch` at line 478 goes; `offlineReclaimable`: `cleanup` → `reclaimHeld` always, `create` and `place` below; `cleanupReportEvidence` and `Server.cleanupReports` deleted), `internal/console/selfserve_pass.go` (`SelfServeEvidence`, lines 41-53, counts only entries of kind `move`), `internal/console/lifetimes_recovery.go` (each iteration also runs `s.residenceSweep(ctx, now)`), `internal/console/selfserve_pages.go` (two page conditions) |
| - Test: `internal/moves/reserve_test.go`, `internal/console/offlinereserve_hostkey_test.go` |
| **Interfaces:** |
| - `func HostKey(host string) string`; `func IsHostKey(key string) bool` (`strings.HasPrefix(key, "host/")`; a box label never contains `/`, `controlname.ValidLabel`). |
| +- `type Active struct { Body OfflineActive; Table map[string]OfflineReservation; ETag string }`; `func ReadActive(ctx context.Context, s release.Store) (Active, error)`: `Get`; absent → the bootstrap `Put` of `{v: 1, epoch: newEpoch(), rev: 1}` with `ifMatch ""` (create-only), then `Get` again whatever the `Put` answered (a lost create means another writer created it; an uncertain one can only ever create an empty table, so it is harmless); a read error or `ParseOfflineActive` failing → `ErrUnreadable`. `func NextActiveBody(prev OfflineActive, table map[string]OfflineReservation) OfflineActive`: `OfflineActiveBody(table)` with `Epoch = prev.Epoch` (a new one when empty) and `Rev = prev.Rev + 1`. Readers that only read (`SelfServeEvidence`, `selfserve_pages.go:143`) keep treating an absent object as an empty table. |
| - `type Op struct{ Kind, ID string }`; `func Conflicts(table map[string]OfflineReservation, keys []string, mine Op, movesExclusive bool) map[Op]map[string]OfflineReservation` = the Global Constraint's relation, `movesExclusive` applying rule (c) only when `mine.Kind == move`. |
| -- `var ErrUnreadable`, `ErrReservationMismatch`; `type ErrReserved struct { Key string; Holder OfflineReservation }`; `func Reserve(ctx context.Context, s release.Store, keys []string, r OfflineReservation) (OfflineReservation, error)` for the box, the create and the placement: read (absent = empty; a read error or `ParseOfflineActive` failing → `ErrUnreadable`); own entries on a key outside `keys` → `ErrReservationMismatch`; own entries covering `keys` → the held entry returned, nothing written; any conflict → `ErrReserved` naming the oldest conflicting entry (no reclaim on these paths); else one conditional `Put` of the table with the missing keys (at the held entry's `ReservedAt` when partial) and `OfflineActiveBody`'s projection; a CAS conflict re-reads once; a second loss → `ErrReserved` from the last read; a `Put` error that is not a CAS conflict is returned wrapped as uncertain (`errors.Is(err, ErrUncertain)`): the caller keeps its state as if it might hold the keys, and its own retry resolves it. |
| -- `func Holds(table map[string]OfflineReservation, keys []string, mine Op) bool`; `func Release(ctx context.Context, s release.Store, kind, id string) error` (`releaseBoxes`' algorithm, moved: absence is success; a malformed table is an error and is left as it is). |
| -- The console's `reserveBoxes`: `want = sel.reservationKeys()` plus `moves.HostKey(r.Host)`; the own retry is unchanged (an own reservation made before this plan, without the host key, is partial and completed in the same write; an operator operation holding the sentinel covers every box key and has its host key completed); conflicts = `moves.Conflicts(table, want, mine, s.cfg.SelfServeMoves != "offline")`; reclaim as today for `move`; `create`: `s.store.AllocationState(opID)` found and terminal → `reclaimYes`, pending → `reclaimHeld`, not found or an error → `reclaimUnreadable`; `place`: `reclaimHeld` until Task 7 replaces it with `s.settlePlacement`. |
| -- `s.residenceSweep(ctx, now)` (in each lifetime recovery iteration, `lifetimeRecoveryIteration`): a `cleanup` entry older than 6 h → `s.selfServePage(ctx, id, ssPageCleanupHeld, "a cleanup has held <host> for over six hours", body, now)`; a `create` entry older than `offlineLockGrace` with no allocation row → `ssPageCreateOrphan` (same body); Task 7 adds the placement settle. |
| +- `var ErrUnreadable`, `ErrReservationMismatch`, `ErrUncertain`; `type ErrReserved struct { Key string; Holder OfflineReservation }`; `func Reserve(ctx context.Context, s release.Store, mine Op, want map[string]OfflineReservation) (map[string]OfflineReservation, error)` for the box, the create and the placement (N10: the caller names the entry for each key; every entry carries `mine` and is valid, else an error and nothing written): `ReadActive`; own entries on a key outside `want` → `ErrReservationMismatch`; own entries covering `want` → the held entries returned, nothing written; any conflict → `ErrReserved` naming the oldest conflicting entry (no reclaim on these paths); else one `Put` with `If-Match` the read ETag of `NextActiveBody` with the missing keys added (at the held entries' `ReservedAt` when partial); a CAS conflict re-reads once; a second loss → `ErrReserved` from the last read; a `Put` error that is not a CAS conflict → `ErrUncertain` wrapped: the caller keeps its state as if it might hold the keys, and the next successful write by anyone, its own retry or its release included, settles it. |
| +- `func Holds(table map[string]OfflineReservation, keys []string, mine Op) bool`; `func Release(ctx context.Context, s release.Store, mine Op) error`: `ReadActive`; remove `mine`'s entries; **always** one `Put` with `If-Match` of `NextActiveBody`, even when `mine` held nothing (the settling write); a CAS conflict re-reads, three rounds as `releaseBoxes`; a malformed table is an error and is left as it is. A release that read no entry and wrote successfully proves that no earlier `Put` of `mine` can land (the Global Constraint), which a read alone never does. |
| +- The console's `reserveBoxes`: `want` = an entry `r` for each of `sel.reservationKeys()` and for `moves.HostKey(r.Host)`; the own retry is unchanged (an own reservation made before this plan, without the host key, is partial and completed in the same write; an operator operation holding the sentinel covers every box key and has its host key completed); conflicts = `moves.Conflicts(table, keys, mine, s.cfg.SelfServeMoves != "offline")`; reclaim as today for `move`; `create`: `s.store.AllocationState(opID)` found and terminal → `reclaimYes`, pending → `reclaimHeld`, not found or an error → `reclaimUnreadable`; `place`: `reclaimHeld` until Task 7b replaces it with `s.settlePlacement`. |
| +- `s.residenceSweep(ctx, now)` (in each lifetime recovery iteration, `lifetimeRecoveryIteration`): a `cleanup` entry older than 6 h → `s.selfServePage(ctx, id, ssPageCleanupHeld, "a cleanup has held <host> for over six hours", body, now)`; a `create` entry older than `offlineLockGrace` with no allocation row → `ssPageCreateOrphan` (same body); Task 7b adds the placement settle. |
| -- [ ] **Step 1: Write the failing tests**: `TestConflictRelation` (every pair of kinds over box keys, host keys and the sentinel, with and without `movesExclusive`: two creates of different hosts never conflict; a create and a sentinel move of another host do not; a create and a move of the same host do; a cleanup and an operator move on disjoint boxes do not, with the switch off as on; two moves on disjoint boxes conflict only with the switch off); `TestReserveReturnsTheHeldEntry`; `TestReserveCompletesAPartialOwnReservation`; `TestReserveUncertainPutIsNotAnAnswer` (a store that commits and then errors → `ErrUncertain`; the own retry returns the held entry); `TestMoveSelectionsHoldTheHostKey` (a reservation written by plan 2's code, without the host key, is completed by its own retry; a cleanup on the host key makes the move busy); `TestReleaseIsTheConsolesAlgorithm` (plan 2's `releaseBoxes` tests green through the wrapper); `TestCleanupEntriesAreNeverReclaimed`; `TestCreateEntryReclaimOnTheAllocationRow`; `TestRecoveryModeCountsMoveEntriesOnly` (cleanup, create and place entries alone leave `SelfServeEvidence` false); `TestResidenceSweepPagesOldEntries`. |
| -- [ ] **Step 2: Run to verify they fail**: `go test ./internal/moves -run 'Conflict|Reserve|Release' -count=1`, `go test ./internal/console -run 'HostKey|Reserve|Reclaim|Release|RecoveryMode|ResidenceSweep|OwnRetry|Sentinel' -count=1`. |
| +- [ ] **Step 1: Write the failing tests**: `TestConflictRelation` (every pair of kinds over box keys, host keys and the sentinel, with and without `movesExclusive`: two creates of different hosts never conflict; a create and a sentinel move of another host do not; a create and a move of the same host do; a cleanup and an operator move on disjoint boxes do not, with the switch off as on; two moves on disjoint boxes conflict only with the switch off); `TestEntryShapeRules` (a `place` or `create` entry without account or app, a host-key entry naming another host → malformed; a fixture parsed by plan 3's `ParseOfflineActive` accepts `move` and `cleanup` entries and ignores `epoch` and `rev`); `TestActiveTableIsNeverDeleted` (releasing the last entry writes an empty table; the console's `releaseBoxes` likewise; no `Delete` or `DeleteIfMatch` reaches the key); `TestBootstrapIsCreateOnlyAndEmpty` (two first writers: one bootstrap lands, both reserve through `If-Match`); `TestRevAndEpochNeverRepeatABody` (alternating reserves and releases of one entry: every ETag distinct; a body without `epoch`, as plan 3 writes, gets one); `TestUncertainReservationIsSettledByTheNextWrite` (review 4's interleaving: create C's `Put` is held by the fake store and answers a timeout; a cleanup acquires; C's recovery reads no entry of C, voids and calls `Release(C)`, which writes; the cleanup releases; C's held `Put` is then delivered and fails its condition; the table never holds C); `TestReleaseAlwaysWrites`; `TestReserveReturnsTheHeldEntry`; `TestReserveCompletesAPartialOwnReservation`; `TestReserveUncertainPutIsNotAnAnswer` (a store that commits and then errors → `ErrUncertain`; the own retry returns the held entries); `TestMoveSelectionsHoldTheHostKey` (a reservation written by plan 2's code, without the host key, is completed by its own retry; a cleanup on the host key makes the move busy); `TestReleaseIsTheConsolesAlgorithm` (plan 2's `releaseBoxes` tests green through the wrapper, with "deleted" read as "empty"); `TestCleanupEntriesAreNeverReclaimed`; `TestCreateEntryReclaimOnTheAllocationRow`; `TestRecoveryModeCountsMoveEntriesOnly` (cleanup, create and place entries alone, and an empty table, leave `SelfServeEvidence` false); `TestResidenceSweepPagesOldEntries`. |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/moves -run 'Conflict|EntryShape|ActiveTable|Bootstrap|RevAndEpoch|Uncertain|Reserve|Release' -count=1`, `go test ./internal/console -run 'HostKey|Reserve|Reclaim|Release|RecoveryMode|ResidenceSweep|OwnRetry|Sentinel|NeverDeleted' -count=1`. |
| - [ ] **Step 3: Implement. Step 4: Run to verify they pass.** |
| -- [ ] **Step 5: Commit** `moves: hostname keys in the reservation table and one conflict relation shared by the console, the box, creates and placements`. |
| +- [ ] **Step 5: Commit** `moves: hostname keys in a reservation table that is never deleted and never repeats a body, with one conflict relation shared by the console, the box, creates and placements`. |
| ### Task 3: The cleaning mark and the writer inventory |
| @@ -104,98 +110,202 @@ Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and |
| - [ ] **Step 3: Implement. Step 4: Run to verify they pass.** |
| - [ ] **Step 5: Commit** `lease: the cleaning mark, a live lease written only over this move's lifted tombstone and put back exactly; every hibernate-prefix site inventoried`. |
| -### Task 4: The cleanup lifecycle on the box |
| +### Task 4a: The persisted lifecycle and its recovery |
| **Files:** |
| -- Create: `internal/host/movescleanup.go`, `internal/moves/cleanup.go` |
| -- Modify: `internal/host/movespass.go` (`movesWorker` gains `cleanupMu sync.Mutex`; `moveCfg` gains `PlainMirror func(ctx context.Context, region string) (lease.MirrorStore, error)` and `CleanupArmed bool`, set in `armOffline` from `CARLOS_OFFLINE_CLEANUP`; `movesPass` ends with `w.kickCleanup(ctx, s)`) |
| +- Create: `internal/host/movescleanup.go` (`cleanupVisit`, the transition table, `supersededBy`, the outcome table), `internal/moves/cleanup.go` (`RecordAway`) |
| - Test: `internal/host/movescleanup_test.go`, `internal/moves/cleanup_test.go` |
| **Interfaces:** |
| -- `type Away int` (`AwayYes`, `AwayFinishing`, `AwayMoved`, `AwayUnreadable`); `func RecordAway(obs provision.Observation, moveID string, incarnation uint64, from string) Away`: `LifetimeUnreadable`/`LifetimeCorrupt` → `AwayUnreadable`; absent, not `LifetimeValid`, `Rec.Retired` or `Rec.Incarnation != incarnation` → `AwayMoved`; `Rec.Move != nil` → `AwayFinishing` when `Move.ID == moveID`, else `AwayMoved`; `LastMove` nil, another id or not `done` → `AwayMoved`; `Rec.Region == from` → `AwayMoved`; else `AwayYes`. Root's authority (Task 6) calls the same function. |
| -- `func (w *movesWorker) kickCleanup(ctx context.Context, s release.Store)`: nothing unless `cfg.OfflineCapable && cfg.Judge != nil`; `w.cleanupMu.TryLock()` false → return; else a goroutine runs `w.cleanupRun(runCtx, s)` with `runCtx, cancel := context.WithTimeout(ctx, cleanupRunDeadline)` and unlocks when it returns. **Locks:** the goroutine holds `cleanupMu` only, never `w.pass` and never `w.mu` across I/O; `movesPass` holds `w.pass` while it only try-locks `cleanupMu`, so neither waits on the other and `offlineReassert` (`movesoffline.go:795`, which takes `w.pass`) is never behind a cleanup. |
| -- `cleanupRun`: lists `<dataDir>/.offline-moves/*/*/retired.json` and the corrupt stubs, starts after the path recorded in `<dataDir>/.offline-moves/.cleanup-cursor`, visits each obligation in path order until the deadline (each read, list, delete, lock call and report write under `context.WithTimeout(runCtx, 10*time.Second)`), at most 50 keys per obligation, records the cursor after each obligation, then runs publication (Task 5) and the reaper (Task 8). One `PlainMirror` and one `Versioned` store per run. |
| -- **The transition table** (first matching row; "observe" is `moves.RecordAway(cfg.Judge.Observe(ctx, s, Account, App, Host), MoveID, Incarnation, From)`; `AwayUnreadable` anywhere → stay, event `refused/record-unreadable`, next obligation; each state change is one `writeRetired`): |
| - 1. `open`: `AwayFinishing` → stay (the original move's finishing has precedence; never supersession); `AwayYes` → `finished`, `FinishedAt = LastMove.At`; `AwayMoved` → supersede. |
| - 2. `finished`: `AwayMoved` → supersede; `now - FinishedAt < 24 h` or `!cfg.CleanupArmed` → stay; else `moves.Reserve([cfg.BoxID, HostKey(Host)], {cleanup, ID, Host, Account, App, now})`: `ErrReserved` → stay, event `busy`; `ErrUnreadable` or uncertain → stay, event `refused/reservation-unreadable`; held → `ReservedAt` from the returned entry, every inventory key `pending`, `State = cleaning`. |
| - 3. `cleaning`, `publishing`, `root-requested` (**ownership**, before anything else; rows 3 to 5 run in order within one visit): read `active.json`; `!moves.Holds(table, [BoxID, HostKey(Host)], {cleanup, ID})` → supersede (an operator removed it; never re-acquired); a read error → stay. With `MarkedAt` set, in these three states and in `cancel-pending`, the mark is reasserted whenever under 30 min remain (`ErrNotOurLock` → supersede, which in `cancel-pending` changes nothing), so it never lapses while root is waited for. |
| - 4. `cleaning` with `MarkedAt` zero: `MarkCleaning(..., TombstoneOwner, CleaningOwner(ID), cleanupMarkTTL, now)`: `ErrNotOurLock` → supersede; another error → stay; marked → `MarkedAt = now`. |
| - 5. `cleaning`, for each `pending` key in inventory order within the budget: observe → `AwayMoved` → supersede; `w.offlineRow(Host)` → a row → supersede; the mark (reassert when under 30 min remain; then `LockState`: owner not ours or under `cleanupMarkMargin` → `ErrNotOurLock` → supersede); `ListVersions(Bucket, key)` filtered to `Key == key`: latest is the inventory's version → `PlainMirror.Delete(Bucket, key)` → `Keys[key] = {deleted, VersionID, now}` (written before the next key; a crash between the delete and the write finds a marker current next run → `marked`, the same end); a marker current → `marked`; another version current → `superseded` and the skipped page (Task 5); none → `absent`. A read or delete error → event (`versions-unreadable`, `lock-unreadable`, `registry-unreadable`, `delete-failed`), stop this obligation for this run. Every key terminal: `LocalInventory` empty → `releasing`, `Outcome = cleaned`; else `publishing`. |
| - 6. `publishing`, `root-requested`, `cancel-pending`: Task 6's rows. |
| - 7. **Supersede**: from `publishing` or `root-requested` → `cancel-pending` (Task 6), so the reservation and the mark stay until root has acknowledged; from `cancel-pending` nothing changes; from any other state → `releasing`, `Outcome = superseded` (from `open` and `finished` too: `Release` and `UnmarkCleaning` are no-ops when nothing is held, and a `finished` obligation may hold an entry its crash left). |
| - 8. `releasing`: `UnmarkCleaning(..., CleaningOwner(ID), TombstoneOwner)` → error → stay; `moves.Release(cleanup, ID)` → error → stay; → `ended`, `EndedAt = now`. The mark goes before the reservation, so a residence change admitted by the release never meets a mark, and no key is deleted after `releasing` is written. |
| - 9. `ended`: Task 5's publication and Task 8's reaper only. `corrupt`: Task 5's hourly page only. |
| +- `type Away int` (`AwayYes`, `AwayFinishing`, `AwayMoved`, `AwayUnreadable`); `func RecordAway(obs provision.Observation, moveID string, incarnation uint64, from string) Away`: `LifetimeUnreadable`/`LifetimeCorrupt` → `AwayUnreadable`; absent, not `LifetimeValid`, `Rec.Retired` or `Rec.Incarnation != incarnation` → `AwayMoved`; `Rec.Move != nil` → `AwayFinishing` when `Move.ID == moveID`, else `AwayMoved`; `LastMove` nil, another id or not `done` → `AwayMoved`; `Rec.Region == from` → `AwayMoved`; else `AwayYes`. Root's authority (Task 6a) calls the same function. |
| +- `func (w *movesWorker) cleanupVisit(ctx context.Context, s release.Store, path string, now time.Time) error`: one obligation through the table below, each state change one `writeRetired`; Task 4c's runner calls it; the tests call it directly. The `cleaning` row's per-key step is `w.retireKey` (Task 4b); until 4b lands it is a field the tests set. |
| +- `func (w *movesWorker) supersededBy(ctx context.Context, s release.Store, r *offlinecleanup.Retired) (reason string, err error)` (F7: the one supersession check): observe (`moves.RecordAway(cfg.Judge.Observe(ctx, s, Account, App, Host), MoveID, Incarnation, From)`): `AwayUnreadable` → an error; `AwayMoved`, or `AwayFinishing` in any state but `open` → `record-moved`; then `w.offlineRow(Host)`: an error → an error; a row → `row-present`; else "". |
| +- **The transition table** (first matching row; an error from a check → stay, event `refused/<what>`, next obligation): |
| + 1. `open`: observe: `AwayFinishing` → stay (the original move's finishing has precedence; never supersession); `AwayMoved` → supersede; `AwayYes` → then the row check (a row → supersede) → `finished`, `FinishedAt = LastMove.At`. |
| + 2. `finished`: `supersededBy` → supersede; `now - FinishedAt < 24 h` or `!cfg.CleanupArmed` → stay; else `moves.Reserve(ctx, s, {cleanup, ID}, {cfg.BoxID: e, HostKey(Host): e})` with `e = {cleanup, ID, Host, Account, App, now}`: `ErrReserved` → stay, event `busy`; `ErrUnreadable` or uncertain → stay, event `refused/reservation-unreadable` (the next visit's own retry settles it); held → `ReservedAt` from the returned entry, every inventory key `pending`, `State = cleaning`. |
| + 3. `cleaning`, `publishing`, `root-requested` (**ownership**, before anything else; rows 3 to 5 run in order within one visit): read `active.json`; `!moves.Holds(table, [BoxID, HostKey(Host)], {cleanup, ID})` → supersede, reason `reservation-lost` (an operator removed it; never re-acquired); a read error → stay. |
| + 4. `cleaning` with `MarkedAt` zero: `MarkCleaning(..., TombstoneOwner, CleaningOwner(ID), cleanupMarkTTL, now)`: `ErrNotOurLock` → supersede, reason `lock-taken`; another error → stay; marked → `MarkedAt = now`. |
| + 5. `cleaning`, for each `pending` key in inventory order within the budget: `w.retireKey` (Task 4b), which answers the key's terminal outcome, a supersession reason, or stop. Every key terminal: `supersededBy` once more → supersede; else `LocalInventory` empty → `releasing`, `Outcome = cleaned`; else `publishing`. |
| + 6. `publishing`, `root-requested`, `cancel-pending`: Task 6b's rows; `publishing` and `root-requested` begin with `supersededBy` (a reason → supersede). |
| + 7. **Supersede**: from `publishing` or `root-requested` → `cancel-pending` (Task 6b), so the reservation and the mark stay until root has acknowledged; from `cancel-pending` nothing changes; from any other state → `releasing`, `Outcome = superseded` (from `open` and `finished` too: `Release` and `UnmarkCleaning` change nothing of ours when nothing is held, and a `finished` obligation may hold an entry its crash left). Event `superseded/<reason>`. |
| + 8. `releasing`: `now < QuietUntil` → stay (Task 4c keeps the mark); `UnmarkCleaning(..., CleaningOwner(ID), TombstoneOwner)` → error → stay; `moves.Release(ctx, s, {cleanup, ID})` → error → stay; → `ended`, `EndedAt = now`. The mark goes before the reservation, so a residence change admitted by the release never meets a mark, and no key is deleted after `releasing` is written. |
| + 9. `ended`: with `RepairKeys` non-empty and `RepairPasses == 0`, Task 4b's repair pass (then `RepairPasses = 1`); then Task 5a's publication and Task 8's reaper only. `corrupt`: Task 5a's hourly page only. |
| +- **The outcome table** (F7; the only place `Outcome` is chosen; never `cleaned` otherwise): |
| + |
| + | How the obligation reached `releasing` | `Outcome` | |
| + |---|---| |
| + | every key terminal, the final `supersededBy` "", `LocalInventory` empty | `cleaned` | |
| + | in `root-requested`, a trusted `done` acknowledgement with every path removed or absent | `cleaned` | |
| + | in `root-requested`, a trusted `done` acknowledgement with any path kept (identity changed, a row on this box, a referenced build, cancelled, writable layout) | `superseded` | |
| + | in `root-requested`, an acknowledgement the agent cannot trust (Task 6b) | `superseded` | |
| + | in `root-requested`, a trusted `cancelled` acknowledgement the agent did not ask for (authority refused, malformed request, unsupported platform, a forged cancel) | `cancelled` | |
| + | supersession from `open`, `finished` or `cleaning` (`record-moved`, `row-present`, `lock-taken`, `reservation-lost`) | `superseded` | |
| + | `cancel-pending` resolved (any acknowledgement, or no request was ever taken) | `superseded` | |
| + |
| + Root's per-path results go into the report whatever the outcome (`Root`). |
| - **Restart** resumes every state through the same table: `finished` re-runs `Reserve`, whose own retry returns the held entry; `cleaning` never reserves, and deletes only after rows 3 and 4 found both entries and the mark this cleanup's. |
| -- [ ] **Step 1: Write the failing tests** against a `DirStore` deployment bucket, a versioned fake recording every `Delete` (none with a version id), the lease's S3 fake for `lock.json`, a real registry and an injected clock: `TestRecordAway` (each branch); `TestCleanupRunsTwentyFourHoursAfterTheObservedFinish` (at 23 h 59 m nothing; at 24 h reserved, marked, deleted); `TestCleanupUnarmedLeavesObligationsFinished`; `TestFourConditionsAtEveryKey` (each failing between key 1 and key 2: stop on a read error, supersede on (1), (2), (3), the per-key outcomes of (4)); `TestCleanupSupersedesWhenTheRecordMovesOn` (a second move B to C, a recreation, a deletion, a placement back to `From`: each superseded, nothing more deleted, the lifted tombstone restored, the entries released); `TestCleanupSupersedesWhenARowAppears`; `TestOriginalFinishingMoveIsNotSupersession`; `TestAdmittedReconcileAndWakeCannotWriteUnderTheMark` (Review Focus 1, with `ExecBackend.Acquire` over the fake); `TestLifecycleResumesAtEveryBoundary` (Review Focus 2); `TestOwnershipLostIsSupersession` (an entry deleted by an operator; the mark replaced by a lease; each → `superseded`, no delete); `TestCleanupRunsOutsideThePassLock` (a versioned fake that blocks: `offlineReassert` returns while the run is blocked; the run stops at its deadline; the cursor moves on; a second kick while one runs starts nothing); `TestLockAndBuildStateNeverDeleted`; `TestNoVersionIdOnDeletes`. |
| -- [ ] **Step 2: Run to verify they fail**: `go test ./internal/moves -run 'RecordAway' -count=1`, `go test ./internal/host -run 'Cleanup|FourConditions|AdmittedReconcile|LifecycleResumes|OwnershipLost|OriginalFinishing|LockAndBuildState|NoVersionId' -count=1`. |
| +- [ ] **Step 1: Write the failing tests** against a `DirStore` deployment bucket, the lease's S3 fake for `lock.json`, a real registry, an injected clock and a key step the test controls: `TestRecordAway` (each branch); `TestSupersessionIsCheckedAtEveryStep` (a row appearing and, separately, the record moving on, at: `open`, `finished`, between two keys, after the last key and before `publishing`, in `publishing` before the drop, in `root-requested`: each ends `superseded` through `releasing` or `cancel-pending`, never `cleaned`); `TestOutcomeTable` (each row, with root's answers from a fake acknowledgement reader); `TestCleanupSupersedesWhenTheRecordMovesOn` (a second move B to C, a recreation, a deletion, a placement back to `From`, each before the reservation: superseded, the lifted tombstone untouched, nothing held); `TestOriginalFinishingMoveIsNotSupersession`; `TestLifecycleResumesAtEveryBoundary` (Review Focus 2); `TestOwnershipLostIsSupersession` (an entry deleted by an operator; the mark replaced by a lease; each → `superseded`, no delete). |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/moves -run 'RecordAway' -count=1`, `go test ./internal/host -run 'SupersessionIsChecked|OutcomeTable|CleanupSupersedes|OriginalFinishing|LifecycleResumes|OwnershipLost' -count=1`. |
| - [ ] **Step 3: Implement. Step 4: Run to verify they pass.** |
| -- [ ] **Step 5: Commit** `moves: the source box retires the old replica through one restartable lifecycle under a reservation and a cleaning mark, re-reading the four conditions at every key, outside the moves pass lock`. |
| +- [ ] **Step 5: Commit** `moves: the cleanup's persisted lifecycle, supersession checked at every step and one outcome table`. |
| -### Task 5: Publication: the report under the move, its events, the pages |
| +### Task 4b: The conditional deletion |
| + |
| +**Files:** |
| +- Create: `internal/host/movescleanup_delete.go` (`retireKey`) |
| +- Modify: `internal/lease/mirror.go` (`ConditionalMirrorDeleter`, `ErrDeleteConditionFailed`, `(*s3Mirror).DeleteIfMatch`), the versioned test fake the host tests use (it gains `DeleteIfMatch` with S3's semantics and a hook to hold a request and deliver it later) |
| +- Test: `internal/lease/mirror_deleteifmatch_test.go`, `internal/host/movescleanup_delete_test.go` |
| + |
| +**Interfaces:** |
| +- `type ConditionalMirrorDeleter interface { DeleteIfMatch(ctx context.Context, bucket, key, etag string) error }`; `var ErrDeleteConditionFailed`. `(*s3Mirror).DeleteIfMatch`: an empty or malformed `etag` → an error without a request; `m.clientFor(bucket, m.src).DeleteObject(ctx, &awss3.DeleteObjectInput{Bucket: &bucket, Key: &key, IfMatch: aws.String("\"" + normalizeETag(etag) + "\"")})` (the one-line change against `Delete` at `mirror.go:787-800`); `PreconditionFailed` or `ConditionalRequestConflict` (412, 409, by code and by HTTP status, as `release.s3CASConflict`, `store.go:742-756`) → `ErrDeleteConditionFailed`; `NoSuchKey`/404 → `ErrMirrorNotFound`; anything else returned as it is. The cleanup asserts the interface on `cfg.PlainMirror`'s store; a store without it → event `refused/delete-unsupported`, stop, nothing deleted (never a fallback to `Delete`). |
| +- `func (w *movesWorker) retireKey(ctx context.Context, s release.Store, r *offlinecleanup.Retired, obj offlinecleanup.InventoryObject, mirror lease.ConditionalMirrorDeleter, vs lease.VersionedStore, now time.Time) (outcome string, superseded string, err error)`, in order: `supersededBy` (conditions (1) and (3)); the mark (condition (2): Task 4c reasserts first; then `lease.LockState` at the location: owner not ours, or under `cleanupMarkMargin` left → `lock-taken`); condition (4): `vs.ListVersions(Bucket, key)` filtered to `Key == key`: a marker current → `marked`; another version current → `superseded` (the skipped page, Task 5a); none → `absent`; the inventory's version current → `mirror.DeleteIfMatch(Bucket, key, obj.ETag)`: nil → `deleted` (the condition held at the server); `ErrDeleteConditionFailed` or `ErrMirrorNotFound` → list the key again and classify as above (the inventory's version still current, after a 409 race → stays `pending` for the next run); any other error → event `refused/delete-uncertain`, `QuietUntil` stays set (below), stop this obligation for this run. `Keys[key]` is written before the next key; a crash between the delete and the write finds a marker current next run → `marked`, the same end. A read error → event (`versions-unreadable`, `lock-unreadable`, `registry-unreadable`, `record-unreadable`), stop for this run. |
| +- **`QuietUntil`** (the byte-identical bound): before the first `DeleteIfMatch` of a run, one `writeRetired` sets `QuietUntil = runStart + cleanupRunDeadline + cleanupUncertainQuiet` and `RunKeys` to the pending keys this run may delete (at most 50; a crash after it leaves both set); at the end of a run in which every delete issued was answered (nil, 412, 409 or 404), both are cleared in the run's last state write; a run with any other answer leaves `QuietUntil` at that delete's issue time plus `cleanupUncertainQuiet`, if later, moves `RunKeys` into `RepairKeys` and sets `RepairSince` (once) to the run's start. A visit that finds `RunKeys` set from a crashed run does the same move first. Task 4a's row 8 waits for `QuietUntil`. |
| +- `func (w *movesWorker) repairKeys(ctx context.Context, r *offlinecleanup.Retired, vs lease.VersionedStore, now time.Time) error` (the **repair**, the Global Constraint's rule): per key of `RepairKeys`, `vs.ListVersions(Bucket, key)` filtered to `Key == key` (the current entry first, the rest newest first, `versioned.go:45-50`): the current entry a delete marker with `LastModified >= RepairSince`, the next entry not a marker, its `VersionID` not the inventory's, its `LastModified` after the inventoried version's and its `ETag` the inventory's → `vs.DeleteVersion(Bucket, key, marker.VersionID)` → event `restored` for the key; anything else → `done`, nothing touched. A listing or delete error → event `refused/repair-unreadable`, the pass stops and runs again next visit (`RepairPasses` unchanged). Called from Task 4a's row 9 (the first pass after `ended`) and from Task 8's reaper (the final pass, which then clears `RepairKeys`). |
| + |
| +- [ ] **Step 1: Write the failing tests**: `TestMirrorDeleteIfMatchSendsIfMatch` (`s3Mirror` against an `httptest` S3 stub: the DELETE carries `If-Match: "<etag>"`; a 412 → `ErrDeleteConditionFailed`; a 409 → the same; a 404 → `ErrMirrorNotFound`; an empty ETag sends nothing); `TestDeletesAreConditionalOnTheInventoriedETag` (every delete the fake saw names the inventory's ETag; none is a plain `Delete` and none carries a version id); `TestUnconditionalDeleteIsNeverAFallback`; `TestFourConditionsAtEveryKey` (each failing between key 1 and key 2: stop on a read error, supersede on (1), (2), (3), the per-key outcomes of (4)); `TestDeleteAnswersAreClassifiedByRelisting` (412 with another version current → `superseded`; 412 with a marker current → `marked`; 404 → `absent` or `marked`; 409 with the version still current → `pending`; a timeout → stop, `QuietUntil` kept); `TestOutstandingDeleteCannotHideANewerVersion` (Review Focus 1: the fake holds a delete and times it out; a row appears; the obligation supersedes, waits out `QuietUntil`, unmarks and releases; a wake acquires and writes a new version with other bytes; the held delete is delivered: 412, the new version current, no marker); `TestByteIdenticalRewriteWaitsOutTheQuiet` (an uncertain delete, then supersession: `releasing` stays with the mark and the reservation held until `QuietUntil`; a crash after `QuietUntil` is set and before the delete keeps it set and moves `RunKeys` into `RepairKeys`); `TestLateDeleteOnAByteIdenticalVersionIsRepaired` (the witness: the fake holds a delete past `QuietUntil`; the obligation ends; a re-promotion writes a byte-identical version of the key; the held delete lands and puts a marker over it; the next visit's repair removes that marker by its version id, the re-promoted version is current again and the event `restored` is in the report; a key whose newest non-marker version is the inventoried one is `done` and untouched); `TestRepairNeverTouchesAnotherShape` (a marker over a version with another ETag, a marker older than `RepairSince`, a current non-marker version, a marker whose owner wrote again between the listing and the delete: nothing becomes current that was not the later owner's newest; a listing per repair key per pass, none for an obligation whose runs were all answered); `TestAdmittedReconcileAndWakeCannotWriteUnderTheMark` (Review Focus 1, with `ExecBackend.Acquire` over the fake); `TestLockAndBuildStateNeverDeleted`. |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/lease -run 'DeleteIfMatch' -count=1`, `go test ./internal/host -run 'DeletesAreConditional|UnconditionalDelete|FourConditions|DeleteAnswers|OutstandingDelete|ByteIdentical|LateDeleteOnAByteIdentical|RepairNever|AdmittedReconcile|LockAndBuildState' -count=1`. |
| +- [ ] **Step 3: Implement. Step 4: Run to verify they pass.** **Dev witness** (recorded in the commit message, before any arming): on the dev hibernate bucket, a test object, a `DeleteObject` with a stale `If-Match` → 412 and no marker; with the current ETag → a marker. |
| +- [ ] **Step 5: Commit** `lease, moves: every cleanup deletion is conditional at the server on the inventoried ETag, and a late delete that hit a byte-identical later version is repaired by removing its marker`. |
| + |
| +### Task 4c: Scheduling and renewal, outside the pass lock |
| + |
| +**Files:** |
| +- Modify: `internal/host/movespass.go` (`movesWorker` gains `cleanupMu sync.Mutex`; `moveCfg` gains `PlainMirror func(ctx context.Context, region string) (lease.MirrorStore, error)` and `CleanupArmed bool`, set in `armOffline` from `CARLOS_OFFLINE_CLEANUP`; `movesPass` ends with `w.kickCleanup(ctx, s)`), `internal/host/movescleanup.go` (`kickCleanup`, `cleanupRun`, the reassert) |
| +- Test: `internal/host/movescleanup_run_test.go` |
| + |
| +**Interfaces:** |
| +- `func (w *movesWorker) kickCleanup(ctx context.Context, s release.Store)`: nothing unless `cfg.OfflineCapable && cfg.Judge != nil`; `w.cleanupMu.TryLock()` false → return; else a goroutine runs `w.cleanupRun(runCtx, s)` with `runCtx, cancel := context.WithTimeout(ctx, cleanupRunDeadline)` and unlocks when it returns. **Locks:** the goroutine holds `cleanupMu` only, never `w.pass` and never `w.mu` across I/O; `movesPass` holds `w.pass` while it only try-locks `cleanupMu`, so neither waits on the other and `offlineReassert` (`movesoffline.go:795`, which takes `w.pass`) is never behind a cleanup. |
| +- `cleanupRun`: lists `<dataDir>/.offline-moves/*/*/retired.json` and the corrupt stubs, starts after the path recorded in `<dataDir>/.offline-moves/.cleanup-cursor`, calls `cleanupVisit` on each obligation in path order until the deadline (each read, list, delete, lock call and report write under `context.WithTimeout(runCtx, 10*time.Second)`), at most 50 keys per obligation, records the cursor after each obligation, then runs publication (Task 5a) and the reaper (Task 8). One `PlainMirror` and one `Versioned` store per run. |
| +- **Renewal:** in `cleaning` (after `MarkedAt`), `publishing`, `root-requested`, `cancel-pending` and `releasing` before the unmark, every visit reasserts the mark when under 30 min remain (`ReassertCleaning`; `ErrNotOurLock` → supersede, reason `lock-taken`, which in `cancel-pending` and `releasing` changes nothing), so the mark never lapses while root, a cancel acknowledgement or `QuietUntil` is waited for. |
| + |
| +- [ ] **Step 1: Write the failing tests**: `TestCleanupRunsTwentyFourHoursAfterTheObservedFinish` (at 23 h 59 m nothing; at 24 h reserved, marked, deleted); `TestCleanupUnarmedLeavesObligationsFinished`; `TestCleanupRunsOutsideThePassLock` (a versioned fake that blocks: `offlineReassert` returns while the run is blocked; the run stops at its deadline; the cursor moves on; a second kick while one runs starts nothing); `TestMarkIsReassertedWhileWaiting` (root unanswered for 3 h, and `releasing` under `QuietUntil`: the mark never under 30 min); `TestRunBudgetAndCursor` (120 keys: 50 per run; three obligations and a deadline: each visited in turn across runs). |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/host -run 'CleanupRuns|CleanupUnarmed|MarkIsReasserted|RunBudget' -count=1`. |
| +- [ ] **Step 3: Implement. Step 4: Run to verify they pass.** |
| +- [ ] **Step 5: Commit** `moves: the cleanup runs in its own goroutine under its own lock, with a run budget, a cursor and a mark that never lapses while it waits`. |
| + |
| +### Task 5a: Publication: the report under the move, its events, the pages |
| **Files:** |
| - Create: `internal/host/movescleanup_report.go` |
| -- Modify: `internal/host/movespaging.go` (`flushPageOutboxFor(ctx, host, op)`: `flushPageOutbox`'s delivery restricted to `pageEntryDir(dataDir, host, op)`), `internal/console/apiinstancemove_offline.go` (`readOfflineEvidence`, line 233, skips keys under `<prefix>cleanup/` as it skips `offlineReturnKey`; `offlineSelfServeOf` with `?log=1` adds `Cleanup []offlinecleanup.Report` for the record's `LastMove.ID`) |
| -- Test: `internal/host/movescleanup_report_test.go`, `internal/console/offline_cleanup_log_test.go` |
| +- Modify: `internal/host/movespaging.go` (`flushPageOutboxFor(ctx, host, op)`: `flushPageOutbox`'s delivery restricted to `pageEntryDir(dataDir, host, op)`), `internal/console/apiinstancemove_offline.go` (`readOfflineEvidence`, line 233, skips keys under `<prefix>cleanup/` as it skips `offlineReturnKey`) |
| +- Test: `internal/host/movescleanup_report_test.go`, `internal/console/offline_cleanup_evidence_test.go` |
| **Interfaces:** |
| -- `type Event struct { Seq uint64; Kind, Key, Detail string; At, LastAt time.Time; Count int }`. Kinds and their stable details: `finished`; `busy` (the holder's kind); `reserved`; `marked`; `refused` (`record-unreadable`, `registry-unreadable`, `lock-unreadable`, `versions-unreadable`, `delete-failed`, `reservation-unreadable`, `mark-failed`, `unmark-failed`, `release-failed`, `report-unwritten`); `skipped` (`Key`); `superseded` (`record-moved`, `row-present`, `lock-taken`, `reservation-lost`); `root-requested`; `cancel-requested`; `root-done`; `root-cancelled`; `ended` (the outcome). An event equal in `Kind`, `Key` and `Detail` to the last one increments its `Count` and `LastAt` instead of appending, so every distinct refusal is kept and a repeated one is counted (the raw error goes to the box log only). |
| -- `type Report struct { V int; CleanupID, MoveID, Host, Box, State, Outcome string; Keys map[string]KeyOutcome; Events []Event; Root *RootCleanupAck; Rev uint64; At time.Time }` at `moves.CleanupReportKey(Host, MoveID, Box)`, a plain replacing `Put`. |
| +- `type Event struct { Seq uint64; Kind, Key, Detail string; At, LastAt time.Time; Count int }`. Kinds and their stable details: `finished`; `busy` (the holder's kind); `reserved`; `marked`; `restored` (`Key`; the repair removed a late marker); `refused` (`record-unreadable`, `registry-unreadable`, `lock-unreadable`, `versions-unreadable`, `delete-uncertain`, `delete-unsupported`, `repair-unreadable`, `reservation-unreadable`, `mark-failed`, `unmark-failed`, `release-failed`, `report-unwritten`); `skipped` (`Key`); `superseded` (`record-moved`, `row-present`, `lock-taken`, `reservation-lost`); `quiet` (waiting for `QuietUntil`); `root-requested`; `cancel-requested`; `root-done`; `root-kept` (`Key` the path's `Which`, `Detail` root's reason); `root-cancelled`; `ended` (the outcome). An event equal in `Kind`, `Key` and `Detail` to the last one increments its `Count` and `LastAt` instead of appending, so every distinct refusal is kept and a repeated one is counted (the raw error goes to the box log only). |
| +- `type Report struct { V int; CleanupID, MoveID, Host, Box, State, Outcome string; Keys map[string]KeyOutcome; Events []Event; Left []KeptPath; Root *RootCleanupAck; Rev uint64; At time.Time }` at `moves.CleanupReportKey(Host, MoveID, Box)`, a plain replacing `Put`; `ValidateReport(r Report, key string) error` (the id recomputes, the key is the one its fields name). |
| - After each obligation's transitions in a run: `PublishedRev < Rev` → write the report from the file → on success `markPublished` (`PublishedRev = Rev`); a failure → event `refused/report-unwritten` (deduped; the event itself bumps `Rev`, so the next run publishes it too). |
| -- **Pages** through `w.firePage(ctx, Host, ID, rule, alerting.LevelWarning, subject, body)` (one page per `(host, op, rule)` in the outbox, durable across restarts): `root-wait-<n>` with `n = floor((now - RootRequestedAt) / 10 min)` for `n >= 1`; `cancel-wait-<n>` from `CancelRequestedAt`; `skipped-<i>` for inventory index `i`; `corrupt-<h>` with `h = floor((now - the stub's written_at) / 1 h)`. Each run then calls `w.flushPageOutboxFor(ctx, Host, ID)`, whatever `cfg.Enabled` says, so the outbox's other entries keep plan 6's rule (with the switch off they wait on disk). |
| +- **Pages** through `w.firePage(ctx, Host, ID, rule, alerting.LevelWarning, subject, body)` (one page per `(host, op, rule)` in the outbox, `movespaging.go:232`). Periodic rules, fired from persisted times, so a lost one is followed by the next tick's: `root-wait-<n>` with `n = floor((now - RootRequestedAt) / 10 min)` for `n >= 1`; `cancel-wait-<n>` from `CancelRequestedAt`; `corrupt-<h>` with `h = floor((now - the stub's FoundAt) / 1 h)`; `unreadable-<h>` while the record has been unreadable for over an hour (from that event's `At`). **One-shot rules (N8):** `skipped-<i>` for inventory index `i` is written as a `PendingPage{Rule, Subject, At}` into `PagesPending` in the same `writeRetired` that makes key `i` `superseded`, so no state has the key terminal without its page owed. After the transitions, each pending page is fired: it leaves `PagesPending` when `firePage` answers sent, paged before, unwired (no pager on this box, logged as every moves page is), or pending with its entry on disk (`readPageEntry(pageEntryPath(...))` succeeds); it stays on undelivered and on pending held only in memory (`firePage`'s index, `movespaging.go:280-292`, which a restart loses). Each run then calls `w.flushPageOutboxFor(ctx, Host, ID)`, whatever `cfg.Enabled` says, so the outbox's other entries keep plan 6's rule (with the switch off they wait on disk). |
| + |
| +- [ ] **Step 1: Write the failing tests**: `TestCleanupReportIsDurableAndBlocksTheReaper` (Review Focus 5, with Task 8's reaper once it exists; until then the report half: a failing `Put` for three runs, then success: one report with every event and `PublishedRev == Rev`); `TestRefusalsAreStableDedupedEvents` (a read error every run for an hour is one event with its count; a different detail appends); `TestPendingPagesSurviveARestart` (a skipped key; the outbox cannot create the entry and the pager refuses; restart: the page is fired again from `PagesPending` and leaves it only once on disk or sent); `TestPagesFireOncePerTickAcrossRestarts` (root unanswered for 35 min across a restart at 15 min: pages `root-wait-1`, `-2`, `-3`, each once); `TestPageFlushIsScopedToTheCleanup` (`cfg.Enabled` false: the cleanup's undelivered page is retried, another move's entry is not touched); `TestReadOfflineEvidenceSkipsCleanupReports`. |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/host -run 'CleanupReport|RefusalsAre|PendingPages|PagesFire|PageFlush' -count=1`, `go test ./internal/console -run 'SkipsCleanup' -count=1`. |
| +- [ ] **Step 3: Implement. Step 4: Run to verify they pass.** |
| +- [ ] **Step 5: Commit** `moves: the cleanup's report under the move id, republished until it lands, with stable de-duplicated events, and pages owed in retired.json until the outbox has them`. |
| + |
| +### Task 5b: The operator's log: `carlos move offline status --log` |
| + |
| +**Files:** |
| +- Modify: `internal/console/apiinstancemove_offline.go` (`offlineView` gains `Cleanup []offlineCleanupView \`json:"cleanup,omitempty"\``; `handleAPIInstanceOfflineStatus`, line 1950, calls `s.offlineCleanupOf(r.Context(), &v, d)` when `log=1`, before and independent of `offlineSelfServeOf`, which lives in `selfserve_events.go:168` and returns early when there is no self-serve view, `:172-176`), `move.go` (the `log` flag for `offline status`; `offlineView` gains `Cleanup` and `Events`; `runStatus`), `move_test.go`, `approved_moves_copy_test.go` (the new ids), `docs/superpowers/plans/2026-10-06-moves-phase1-copy.md` (the rows, pending), `docs/site/cli.md` (the `--log` line and paragraph, after line 2469) |
| +- Test: `internal/console/offline_cleanup_log_test.go`, `move_test.go` |
| -- [ ] **Step 1: Write the failing tests**: `TestCleanupReportIsDurableAndBlocksTheReaper` (Review Focus 5, with Task 8's reaper once it exists; until then the report half: a failing `Put` for three runs, then success: one report with every event and `PublishedRev == Rev`); `TestRefusalsAreStableDedupedEvents` (a read error every run for an hour is one event with its count; a different detail appends); `TestPagesFireOncePerTickAcrossRestarts` (root unanswered for 35 min across a restart at 15 min: pages `root-wait-1`, `-2`, `-3`, each once); `TestPageFlushIsScopedToTheCleanup` (`cfg.Enabled` false: the cleanup's undelivered page is retried, another move's entry is not touched); `TestReadOfflineEvidenceSkipsCleanupReports`; `TestOfflineStatusLogListsCleanup`. |
| -- [ ] **Step 2: Run to verify they fail**: `go test ./internal/host -run 'CleanupReport|RefusalsAre|PagesFire|PageFlush' -count=1`, `go test ./internal/console -run 'SkipsCleanup|StatusLogListsCleanup' -count=1`. |
| +**Interfaces:** |
| +- **The door** (`GET .../instances/{host}/move/offline?log=1`, the existing status door): `offlineCleanupOf` lists `moves.OfflineReportPrefix(host, id) + "cleanup/"` for `id` = the record's `LastMove.ID` (the record the gate read, `d.rec`), reads each key, keeps a report that decodes and passes `offlinecleanup.ValidateReport` for its key, and appends `offlineCleanupView{Box, State, Outcome, Keys []{Key, State}, Events []{Kind, Key, Detail, At, LastAt, Count}, Left []{Which, Reason}, Root *{Outcome, Removed, Absent []string; Kept []{Which, Reason}}, At}` (Unix seconds, as the view's other times); a listing or read error is logged and leaves `Cleanup` out, never failing the answer. Operator moves and legacy moves get it as self-serve moves do. |
| +- **The CLI** (`parseMoveArgs`, `move.go:138`, its flag switch at `:166-176`): `case "offline"` with `a.offline == "status"` adds `fs.BoolVar(&a.log, "log", false, "show the move's log, with the source box's cleanup")`; `runStatus` (`move.go:1507`) polls `w.m.path + "?log=1"` when set, and, after its lines, in both its `state == "none"` branch and the in-flight one, says per cleanup report: "Cleanup of <host> on <box> is <state>." or, ended, "Cleanup of <host> on <box> ended <outcome>."; per key not `deleted`: "Object <key> is <state>."; per left path: "<box> left <which> in place: <reason>." (the existing MV form); root's answer: "Root answered <outcome>: <n> removed, <n> already gone, <n> kept."; per event: "At <time>: <kind> <key> <detail>, <count> times until <time>." (count 1: "At <time>: <kind> <key> <detail>."); per self-serve event the door returns: "At <time>: <phase> <reason> <detail>."; no cleanup report with `--log`: "No cleanup has reported for move <id> yet." Times as the CLI says them today (`2006-01-02 15:04 UTC`). The decoded view's field names are kept apart from the tree's, as `offlineView`'s are, for the copy gate (`move.go:1092-1100`); event kinds, keys, details and reasons are box diagnostics, not copy. The exit code is unchanged by the log. |
| +- **Docs:** `docs/site/cli.md` gains `carlos move offline status --host <host> --log` in the verb list and one paragraph: what the log adds (the move's events and, a day after the move finished, the source box's cleanup of the old region's copy, object by object, what it left and why). Copy rows, pending. |
| + |
| +- [ ] **Step 1: Write the failing tests**: `TestOfflineStatusLogListsCleanupWithoutASelfServeRow` (an operator move, no self-serve row: `?log=1` carries the report; a report at the wrong key or with another id is skipped; a listing error leaves `cleanup` out and answers 200); `TestOfflineStatusWithoutLogHasNoCleanup`; `TestMoveOfflineStatusLogFlag` (`--log` accepted only on `offline status`; the request carries `log=1`); `TestMoveOfflineStatusLogRendering` (state `none` and in flight; each line above; no report); `TestMovesStringsAreRecorded` and `TestMovesCopyRecordIsComplete` green with the new rows pending. |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/console -run 'StatusLog|StatusWithoutLog' -count=1`, `go test . -run 'MoveOfflineStatusLog|MovesStringsAreRecorded|MovesCopyRecord' -count=1`. |
| - [ ] **Step 3: Implement. Step 4: Run to verify they pass.** |
| -- [ ] **Step 5: Commit** `moves: the cleanup's report under the move id, republished until it lands, with stable de-duplicated events and paged through the outbox`. |
| +- [ ] **Step 5: Commit** `move: carlos move offline status --log shows the source box's cleanup report and events, for every kind of move`. |
| -### Task 6: Root's leftovers |
| +### Task 6a: The registry host freeze and root's authority |
| **Files:** |
| -- Create: `internal/release/rootcleanup.go` (the protocol, portable), `internal/release/rootcleanup_linux.go` (`renameat2`, `fstatat`, `readlinkat`, `unlinkat` through `golang.org/x/sys/unix` on handles from `safefile.OpenDir`), `internal/release/rootcleanup_other.go` (answers every request `cancelled` with `Reason: "unsupported platform"`), `internal/host/movescleanup_root.go` (the agent's rows, the authority callback, `rootCleanupPass`) |
| -- Modify: `internal/registry/registry.go` (`WithHostFrozen`), `internal/registry/remote_dispatch.go` (the `WithHostFrozen` method, in `WithHoldsFrozen`'s form at line 228), `internal/host/agent.go` (`runAdoptUnderRootRelease`, line 4932: `rootCleanupPass(...)` after `heldStagePass`, only when `passLockErr == nil`, nonfatal) |
| -- Test: `internal/release/rootcleanup_test.go`, `internal/release/rootcleanup_linux_test.go`, `internal/host/movescleanup_root_test.go`, `internal/registry/hostfrozen_test.go` |
| +- Create: `internal/host/movescleanup_authority.go` (`rootCleanupAuthority`) |
| +- Modify: `internal/registry/registry.go` (`WithHostFrozen`), `internal/registry/remote_dispatch.go` (the `WithHostFrozen` method, in `WithHoldsFrozen`'s form at line 228) |
| +- Test: `internal/registry/hostfrozen_test.go`, `internal/host/movescleanup_authority_test.go` |
| **Interfaces:** |
| -- **Files.** The agent's drop `<OfflineOpDir>/root-cleanup.json` (create-only) and cancel `<OfflineOpDir>/root-cleanup.cancel` (create-only). Root's directory `release.RootCleanupDir(optDir, id) = <optDir>/root-cleanup/<id>/` holding `request.json` (root's copy), `journal`, and exactly one of `done.json`, `cancelled.json`; `<optDir>/root-cleanup/` and each `<id>/` are created by root `0755` and used only when `fstat` on the opened handle shows owner uid 0 and no group or other write bit (else nothing this pass, logged). The agent reads the acknowledgement there with `safefile` (no symlinks); on a hardened box its uid cannot write that directory, and on a writable box the agent is already trusted with the opt dir, the same trust the live link rests on. |
| -- The wire types (declared in Task 1): `type RootCleanupRequest struct { V int; CleanupID, MoveID, Host, Account, App, Box string; Incarnation uint64; From, To string; Bucket, BucketRegion, Prefix string; Paths []RootCleanupPath; WrittenAt time.Time }`; `type RootCleanupPath struct { Which string; Identity FileIdentity }`; `type RootCleanupAck struct { V int; CleanupID, MoveID, Host, Outcome string (done or cancelled); Removed, Absent []string; Kept []KeptPath; Reason string; At time.Time }`; `type KeptPath struct { Which, Reason string }`. |
| -- `type RootCleanupOptions struct { DataDir, OptDir, Box, Region string; Authorize func(ctx context.Context, req offlinecleanup.RootCleanupRequest) error; Freeze func(host string, fn func(rowPresent bool) error) error; HostLock func(host string, fn func() error) error; Processes func() ([]string, error); Now func() time.Time }`; `var ErrRootCleanupRefused` (the callback's definite no; any other error is a read failure and leaves the request for the next pass); `func ServeRootCleanupRequests(ctx context.Context, o RootCleanupOptions) error`. |
| -- **Root, per drop** (found by a `safefile` walk of `<DataDir>/.offline-moves/<host>/<move>/root-cleanup.json`, host passing `ValidHostShape`, move a plain segment; `id = CleanupID(o.Box, host, move)`): |
| - 1. Read bounded (4096 bytes) and validate: `Host`, `MoveID` equal the path; `Box == o.Box`; `CleanupID == id`; `From == o.Region`; `Which` values known and unique. Invalid → `cancelled` under `id` with `Reason: "malformed request"`. |
| - 2. Copy: `request.json` create-only with the drop's bytes; present with other bytes → `cancelled`, `Reason: "the request changed"`; then the drop is removed. From here root reads only its copy. |
| - 3. Recovery from the journal, per `Which`: last line `intent` and the aside name `.carlos-cleanup-<id>-<which>` present → as `taken`; `intent` and no aside → nothing was taken (the next attempt journals a new intent); `taken` and the aside present → resume at the identity check; `taken` and no aside → the unlink happened: `fsync` the parent, journal `removed`. Terminal lines are `removed`, `kept <reason>`, `absent`; the acknowledgement is built from them alone, so no outcome is unknown. |
| - 4. `o.Authorize(ctx, req)`: `ErrRootCleanupRefused` → `cancelled` with its reason, nothing touched; another error → the next pass. |
| - 5. Per `Which` not yet terminal (`live-link` first; root derives the path: `LiveLink(OptDir, host)`, `ProgramsPath(DataDir, host)`): for the live link only, outside the freeze: the build directory it targets; another link under `<OptDir>/live/` resolving to the same build → `kept the build is still referenced`; `o.Processes()` naming an executable under the build → `kept a process runs the build`; `Processes` failing → the next pass (unknown is never unreferenced). The programs file has no reference check: its schema (`{schema, programs}`, Go main-package paths, `programhistory.go:28-34`) belongs to this host alone and no reader resolves another host through it. Then `o.Authorize` again (the same two answers). Then `o.Freeze(host, func(rowPresent) { rowPresent → kept the instance has a row on this box; o.HostLock(host, func() { root-cleanup.cancel present (Lstat) → kept cancelled; journal intent and fsync; renameat2(parent, name, parent, aside, RENAME_NOREPLACE): ENOENT → journal absent; fsync the parent; journal taken; fstatat(aside, AT_SYMLINK_NOFOLLOW) and readlinkat; identity differs → renameat2(aside, name, RENAME_NOREPLACE), or on EEXIST (a newer entry is at the name) unlinkat(aside); fsync the parent; journal kept identity changed; identity matches → unlinkat(aside); fsync the parent; journal removed and fsync }) })`. |
| - 6. Every `Which` terminal → `cancelled.json` when the authority refused or any path was kept for `cancelled`, else `done.json` (temp, `fsync`, rename, directory `fsync`). |
| - 7. `<optDir>/root-cleanup/<id>/` older than 30 days past its acknowledgement is removed by root. |
| -- **The authority** (`internal/host`): `func rootCleanupAuthority(store release.Store, judge *provision.LifetimeJudge, box, region string, env func(string) string) func(context.Context, offlinecleanup.RootCleanupRequest) error`; `func rootCleanupPass(ctx context.Context, dataDir, optDir string, reg *registry.Registry, store release.Store, judge *provision.LifetimeJudge, box, region string)` with `judge` the root pass's `rootJudge` (`agent.go`, `newBoxJudge`), `box` `BoxLabel()` (the same label the agent uses as `cfg.BoxID`, `movespass.go:342-346`) and `region` root's `CARLOS_REGION`: (a) `active.json` parses and both `table[box]` and `table[moves.HostKey(Host)]` are `{cleanup, CleanupID, Host, Account, App}`; (b) `moves.RecordAway(judge.Observe(ctx, store, Account, App, Host), MoveID, Incarnation, From) == AwayYes`; (c) `req.Bucket`, `BucketRegion` and `Prefix` equal root's own `CARLOS_HIBERNATE_BUCKET`, `CARLOS_HIBERNATE_REGION` and the prefix built from `CARLOS_HIBERNATE_PREFIX` (unset → a read failure, logged), and `lease.LockState` at `strings.TrimSuffix(Prefix, "/")` names owner `lease.CleaningOwner(CleanupID)` (live or expired: the owner alone proves no lease was taken since the mark). (a), (b), (c) answering no → `ErrRootCleanupRefused` with the reason; a read error → itself. `Freeze` is `reg.WithHostFrozen`; `HostLock` is `withHeldStageLock(dataDir, ·, ·)`; `Processes` reads `/proc/*/exe` (`ENOENT`/`ESRCH` skipped as exited, any other error returned). |
| -- `func (r *Registry) WithHostFrozen(host string, fn func(row *Route) error) error`: `BEGIN IMMEDIATE`, `SELECT routeCols FROM routes WHERE host = ?` on the same connection, `fn`, `ROLLBACK`; remote: method `WithHostFrozen`, the callback carrying the row. |
| -- **The agent's rows** (Task 4's table): `publishing`: the drop absent and `RootCleanupDir` absent → `writeRootCleanupRequest` (create-only; an existing drop with equal bytes is fine) → touch the stage-request marker (`holdStageRequester(dataDir)(Host)`) → `RootRequestedAt = now`, `root-requested`, event; either present → `root-requested`. `root-requested`: `release.ReadRootCleanupAck(optDir, ID)` → an acknowledgement whose `CleanupID`, `Host`, `MoveID` match → `Root = ack`, event, `releasing`, `Outcome = cleaned`; a mismatch → page `corrupt-<h>`, stay; none → the `root-wait` pages. `cancel-pending`: the cancel file written create-only and `CancelRequestedAt` set (once); an acknowledgement → `releasing`, `Outcome = superseded`; else the drop is read **first** and then `RootCleanupDir`: both absent → `releasing`, `Outcome = superseded` (root copies before it removes the drop, so a drop already gone means root's directory exists, and none was ever written after `publishing`); else the `cancel-wait` pages. |
| - |
| -- [ ] **Step 1: Write the failing tests**: `TestRootRemovesOnlyTheEntryItTook` (Review Focus 3's replacements: before the rename → kept, the replacement back at its name; after the rename and before the identity check → the newer entry at the name survives, the aside removed); `TestRootRowCheckIsAtomicWithTheUnlink` (an `Upsert` issued during the freeze returns after it; a row present at the freeze → kept); `TestRootAuthorityComesFromTheBucket` (no entries; only the host key; another account or app; the record home again; the mark not ours; the request's bucket not root's → cancelled, nothing touched; a read error → left); `TestForgedAgentFilesAreHarmless` (a `done.json` planted in the agent's directory is never read; a drop with another id is cancelled under the expected id; a forged cancel stops root and nothing else); `TestRootJournalResolvesEveryIntent` (a crash after each journal line and after the rename and the unlink: each restart ends in removed, kept or absent); `TestCancelBeforeAndDuringRootsWork`; `TestReferencedBuildIsKept` (another host's link to the build; a process; `Processes` failing → left); `TestProgramsFileHasNoReferenceCheck`; `TestRootDirOwnershipIsVerified`; `TestAgentReadsOnlyRootsAcknowledgement`; `TestUnansweredRootRequestPagesEveryTenMinutes`; `TestCancelPendingResolvesWithoutARequest` (a crash after `publishing` before the drop; root never wrote a directory); `TestRootCleanupNeedsTheAdoptLock`; `TestUnsupportedPlatformCancels`; `TestWithHostFrozenLocalAndRemote`. |
| -- [ ] **Step 2: Run to verify they fail**: `go test ./internal/release -run 'RootCleanup|RootRemoves|RootRow|RootAuthority|ForgedAgent|RootJournal|CancelBefore|ReferencedBuild|ProgramsFile|RootDir|Unsupported' -count=1`, `go test ./internal/host -run 'RootCleanup|AgentReadsOnly|Unanswered|CancelPending' -count=1`, `go test ./internal/registry -run 'HostFrozen' -count=1`. |
| -- [ ] **Step 3: Implement. Step 4: Run to verify they pass**, then `GOOS=darwin go build ./internal/release/ ./internal/host/` and `GOOS=windows go vet ./internal/release/ ./internal/host/`. |
| -- [ ] **Step 5: Commit** `root: removes an old instance's live link and programs file only through a request it copies and authorises from the bucket, taking each entry aside and verifying it before the unlink`. |
| - |
| -### Task 7: Creates and placements hold the hostname key |
| +- `func (r *Registry) WithHostFrozen(host string, fn func(row *Route) error) error`: `BEGIN IMMEDIATE`, `SELECT routeCols FROM routes WHERE host = ?` on the same connection, `fn`, `ROLLBACK`, as `WithHoldsFrozen` (`registry.go:1626-1660`); remote: method `WithHostFrozen`, the callback carrying the row. |
| +- `func rootCleanupAuthority(store release.Store, judge *provision.LifetimeJudge, box, region string, hib func() (*activator.Config, error)) func(context.Context, offlinecleanup.RootCleanupRequest) error`, with `judge` the root pass's `rootJudge` (`agent.go`, `newBoxJudge`), `box` `BoxLabel()` (the same label the agent uses as `cfg.BoxID`, `movespass.go:342-346`), `region` root's `CARLOS_REGION` and `hib` `activator.ConfigFromEnv` (N9: the activator's own reading and defaults, `activator.go:114-133`): (a) `active.json` parses and both `table[box]` and `table[moves.HostKey(Host)]` are `{cleanup, CleanupID, Host, Account, App}`; (b) `moves.RecordAway(judge.Observe(ctx, store, Account, App, Host), MoveID, Incarnation, From) == AwayYes`; (c) `hib()` is not nil (nil: refused, "no hibernation bucket on this box"), `req.Bucket == cfg.Bucket`, `req.BucketRegion == cfg.Region` (both "" when the AWS chain resolves the region, `activator.go:99`), `req.Prefix == activator.HibernatePrefix(cfg.Prefix, Host) + "/"` (an empty `cfg.Prefix` is valid and gives `<host>/`), and `lease.LockState(ctx, cfg.Bucket, strings.TrimSuffix(req.Prefix, "/"), cfg.Region)` names owner `lease.CleaningOwner(CleanupID)` (live or expired: the owner alone proves no lease was taken since the mark). (a), (b) or (c) answering no → `ErrRootCleanupRefused` with the reason; a read error → itself (the request waits for the next pass). |
| + |
| +- [ ] **Step 1: Write the failing tests**: `TestWithHostFrozenLocalAndRemote` (an `Upsert` issued during `fn` returns after it; the row given is the row at the freeze); `TestRootAuthorityComesFromTheBucket` (no entries; only the host key; another account or app; the record home again; the mark not ours; the request's bucket, region or prefix not root's → refused; a read error → the error); `TestRootAuthorityUsesTheActivatorDefaults` (`CARLOS_HIBERNATE_REGION` unset with a request region "" → authorised; `CARLOS_HIBERNATE_PREFIX` unset → the prefix `<host>/` authorised; the bucket unset → refused). |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/registry -run 'HostFrozen' -count=1`, `go test ./internal/host -run 'RootAuthority' -count=1`. |
| +- [ ] **Step 3: Implement. Step 4: Run to verify they pass.** |
| +- [ ] **Step 5: Commit** `registry, root: a per-host registry freeze, and root's cleanup authority from the reservation, the record and the mark with the activator's own settings`. |
| + |
| +### Task 6b: The root-owned request and acknowledgement |
| + |
| +**Files:** |
| +- Create: `internal/release/rootcleanup.go` (discovery, the copy, the dispatch, the acknowledgement; portable), `internal/release/rootcleanup_other.go` (answers every request `cancelled` with `Reason: "unsupported platform"`), `internal/host/movescleanup_root.go` (the agent's rows, `rootCleanupPass`) |
| +- Modify: `internal/host/agent.go` (`runAdoptUnderRootRelease`: `rootCleanupPass(...)` after `heldStagePass` at line 4932, only when `passLockErr == nil` (line 4655), nonfatal) |
| +- Test: `internal/release/rootcleanup_test.go`, `internal/host/movescleanup_root_test.go` |
| + |
| +**Interfaces:** |
| +- **Files.** The agent's drop `<OfflineOpDir>/root-cleanup.json` (create-only) and cancel `<OfflineOpDir>/root-cleanup.cancel` (create-only). Root's directory `release.RootCleanupDir(optDir, id) = <optDir>/root-cleanup/<id>/` holding `request.json` (root's copy), `journal` (Task 6c) and exactly one of `done.json`, `cancelled.json`. `<optDir>/root-cleanup/` and each `<id>/` are created by root `0755`; root uses one only when `offlinecleanup.RootOnlyChain` passes on it (else root removes nothing for that request and writes its acknowledgement there with every path kept, "writable layout", which `ReadRootCleanupAck` then reports as untrusted). |
| +- The wire types (declared in Task 1): `type RootCleanupRequest struct { V int; CleanupID, MoveID, Host, Account, App, Box string; Incarnation uint64; From, To string; Bucket, BucketRegion, Prefix string; Paths []RootCleanupPath; WrittenAt time.Time }`; `type RootCleanupPath struct { Which string; Identity FileIdentity }`; `type RootCleanupAck struct { V int; CleanupID, MoveID, Host, Outcome string (done or cancelled); Removed, Absent []string; Kept []KeptPath; Reason string; At time.Time }`. |
| +- `type RootCleanupOptions struct { DataDir, OptDir, Box, Region string; Authorize func(ctx context.Context, req offlinecleanup.RootCleanupRequest) error; Freeze func(host string, fn func(rowPresent bool) error) error; HostLock func(host string, fn func() error) error; Processes func() ([]string, error); Now func() time.Time }`; `var ErrRootCleanupRefused`; `func ServeRootCleanupRequests(ctx context.Context, o RootCleanupOptions) error`. |
| +- **Discovery (F6):** each pass walks, with `safefile` (no symlinks), (i) the drops `<DataDir>/.offline-moves/<host>/<move>/root-cleanup.json` (host passing `ValidHostShape`, move a plain segment; `id = CleanupID(o.Box, host, move)`) and (ii) root's own `<OptDir>/root-cleanup/<id>/` directories that hold `request.json` and no acknowledgement, so a request root took and then crashed on is served from its own copy whether or not a drop remains. |
| +- **Root, per drop:** (1) read bounded (4096 bytes) and validate: `Host`, `MoveID` equal the path; `Box == o.Box`; `CleanupID == id`; `From == o.Region`; `Which` values known and unique. Invalid → `cancelled` under `id` with `Reason: "malformed request"`, the drop removed. (2) Copy: `request.json` create-only (temp, `fsync`, `os.Link`, directory `fsync`) with the drop's bytes; the journal's first line `taken <at>` (`fsync`); then the drop is removed. A `request.json` already present with other bytes: the drop is removed and logged, root's copy stands (a later drop never changes or cancels a taken request). From here root reads only its copy. |
| +- **Root, per taken request** (from (2) or from discovery (ii)): Task 6c's journal recovery; `o.Authorize(ctx, req)`: `ErrRootCleanupRefused` → `cancelled` with its reason, nothing touched; another error → the next pass; each path not yet terminal through Task 6c's step; every path terminal → `cancelled.json` when the authority refused or any path was kept for "cancelled", else `done.json` (temp, `fsync`, rename, directory `fsync`). `<optDir>/root-cleanup/<id>/` older than 30 days past its acknowledgement is removed by root. |
| +- `func ReadRootCleanupAck(optDir, id string) (ack *offlinecleanup.RootCleanupAck, trusted bool, err error)`: `trusted` only when `RootOnlyChain(RootCleanupDir(optDir, id))` passes and the file, opened with `safefile` (no symlinks), is owned by uid 0 with no group or other write bit; an acknowledgement the agent cannot trust is advisory (Task 4a's outcome table: `superseded`, every requested path recorded kept "writable layout"). |
| +- `func rootCleanupPass(ctx context.Context, dataDir, optDir string, reg *registry.Registry, store release.Store, judge *provision.LifetimeJudge, box, region string)`: `ServeRootCleanupRequests` with `Authorize` = Task 6a's callback, `Freeze` = `reg.WithHostFrozen` (a non-nil row is `rowPresent`), `HostLock` = `withHeldStageLock(dataDir, ·, ·)`, `Processes` reading `/proc/*/exe` (`ENOENT`/`ESRCH` skipped as exited, any other error returned). |
| +- **The agent's rows** (Task 4a's row 6): `publishing`: `supersededBy` → supersede; the drop absent and `RootCleanupDir` absent → `writeRootCleanupRequest` (create-only; an existing drop with equal bytes is fine) → touch the stage-request marker (`holdStageRequester(dataDir)(Host)`, `moveshold.go:277`) → `RootRequestedAt = now`, `root-requested`, event; either present (a crash after the drop) → `root-requested` with `RootRequestedAt` restored, in this order: the drop's mtime when the drop exists; else the `at` of root's journal `taken` line; else `now`. `root-requested`: `supersededBy` → supersede (to `cancel-pending`); `ReadRootCleanupAck(optDir, ID)` → an acknowledgement whose `CleanupID`, `Host`, `MoveID` match → `Root = ack`, event, `releasing` with the outcome table's answer; a mismatch → page `corrupt-<h>`, stay; none → the `root-wait` pages. `cancel-pending`: the cancel file written create-only and `CancelRequestedAt` set (once); an acknowledgement → `releasing`, `Outcome = superseded`; else the drop is read **first** and then `RootCleanupDir`: both absent → `releasing`, `Outcome = superseded` (root takes a request into its directory before it removes the drop, so a drop already gone means root's directory exists, and no drop was written after `publishing`); else the `cancel-wait` pages. |
| + |
| +- [ ] **Step 1: Write the failing tests**: `TestRootFindsItsOwnRequestsOnRestart` (root takes a request, removes the drop and stops: the next pass serves it from `request.json`); `TestForgedAgentFilesAreHarmless` (a `done.json` planted in the agent's directory is never read; a drop with another id is cancelled under the expected id; a second drop with other bytes after the take is removed and changes nothing; a forged cancel stops root and ends the obligation `cancelled`, nothing else); `TestRootDirOwnershipIsVerified` (a root directory not passing the chain check is not used); `TestAgentReadsOnlyRootsAcknowledgement` (an untrusted acknowledgement → `superseded`, paths kept "writable layout"); `TestRootRequestedAtIsRestored` (a crash after the drop: from its mtime; after the take: from the `taken` line); `TestUnansweredRootRequestPagesEveryTenMinutes`; `TestCancelBeforeAndDuringRootsWork`; `TestCancelPendingResolvesWithoutARequest` (a crash after `publishing` before the drop; root never wrote a directory); `TestRootCleanupNeedsTheAdoptLock`; `TestUnsupportedPlatformCancels`. |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/release -run 'RootFinds|ForgedAgent|RootDir|CancelBefore|Unsupported' -count=1`, `go test ./internal/host -run 'AgentReadsOnly|RootRequestedAt|Unanswered|CancelPending|RootCleanupNeeds' -count=1`. |
| +- [ ] **Step 3: Implement** (until Task 6c lands, the per-path step answers kept with reason "not built", so this commit removes nothing). **Step 4: Run to verify they pass**, then `GOOS=darwin go build ./internal/release/ ./internal/host/` and `GOOS=windows go vet ./internal/release/ ./internal/host/`. |
| +- [ ] **Step 5: Commit** `root: takes a cleanup request into its own directory, finds it there again after a crash, and answers in a file the agent trusts only on a root-only chain`. |
| + |
| +### Task 6c: Root's filesystem step, its journal and recovery |
| + |
| +**Files:** |
| +- Create: `internal/release/rootcleanup_linux.go` (`fstatat`, `readlinkat`, `unlinkat` through `golang.org/x/sys/unix` on a parent opened with `O_DIRECTORY|O_NOFOLLOW` through `safefile.OpenDir`), `internal/release/rootcleanup_journal.go`, `internal/host/livelink_writers_inventory_test.go` |
| +- Test: `internal/release/rootcleanup_linux_test.go`, `internal/release/rootcleanup_journal_test.go` |
| + |
| +**Interfaces:** |
| +- **The journal** `<RootCleanupDir>/journal`: append-only JSON lines, each `fsync`ed before the next syscall: `{at, step: "taken"}`; per path `{at, which, step: "intent", identity}` then exactly one of `{at, which, step: "removed"}`, `{at, which, step: "kept", reason}`, `{at, which, step: "absent"}`. A path's result is its last result line; the acknowledgement is built from results alone, so no outcome is unknown. |
| +- **Recovery (F6),** per path whose last line is `intent`: `fstatat` the name in the opened parent again: absent → journal `absent` (removed by us before the crash or by anyone: recorded as absent, never as removed); present with the recorded identity → the step is retried from the start below; present with another identity → journal `kept`, "identity changed". Nothing was ever renamed, so nothing needs restoring after a cancel or a refusal. |
| +- **The step,** per path not yet terminal (`live-link` first; root derives the path: `LiveLink(OptDir, host)`, `ProgramsPath(DataDir, host)`): for the live link only, outside the freeze: the build directory it targets; another link under `<OptDir>/live/` resolving to the same build → `kept`, "the build is still referenced"; `o.Processes()` naming an executable under the build → `kept`, "a process runs the build"; `Processes` failing → the next pass (unknown is never unreferenced). The programs file has no reference check: its schema (`{schema, programs}`, Go main-package paths, `programhistory.go:28-34`) belongs to this host alone and no reader resolves another host through it. Then `o.Authorize` again (the same two answers). Then `RootOnlyChain(parent)` false → `kept`, "writable layout". Then `o.Freeze(host, func(rowPresent) { rowPresent → kept, "the instance has a row on this box"; o.HostLock(host, func() { root-cleanup.cancel present (Lstat) → kept, "cancelled"; open the parent (O_DIRECTORY|O_NOFOLLOW) and fstat it: not uid 0 or group/other writable → kept, "writable layout"; journal intent; fstatat(parent, name, AT_SYMLINK_NOFOLLOW): ENOENT → journal absent; identity (dev, inode, file type, readlinkat target) differs → journal kept, "identity changed"; equal → unlinkat(parent, name, 0): ENOENT → journal absent; success → fsync the parent, journal removed }) })`. Between `fstatat` and `unlinkat` only root can change an entry of a root-only directory, and every root writer of these names is excluded (below). |
| +- **The writers of these names** (`TestRootOnlyLiveLinkWritersAreExcluded`, `go/ast` with `go/types` over non-test files under `./internal/...`): every call of `release.PointLive`, `release.RecordProgram` and `(*activator.ExecBackend).ensureLiveLink` is listed with what keeps it off the name during root's step: `adopt pass` (it runs in the root adopt pass, which runs root's step after it in the same process under the adopt pass lock: `adopt.go`, `rootrollback_resume.go:125`, `roothistory_recovery.go:216`, and the named-unit recovery sites `namedrecovery.go:120`, `namedrecovery_active.go:191` once their caller is confirmed), `held-stage lock` (held publication), `needs a row` (the freeze), `agent process` (the agent cannot write a root-only directory: `ensureLiveLink`, the agent's own adopt pass at `agent.go:1319`) or `e2e only`; a new site, or a removed one, fails until the list is edited, and a site the implementer cannot place in a class is a question for the reviewer, not a guess. |
| + |
| +- [ ] **Step 1: Write the failing tests**: `TestRootRemovesOnlyTheRecordedIdentity` (Review Focus 3: a publication replaces the link between the reference check and the `fstatat` → kept, the replacement at its name; the recorded entry → removed); `TestWritableLayoutIsNeverRemoved` (a parent the test uid owns; an ancestor writable by group: kept "writable layout", nothing unlinked, no intent line); `TestRootRowCheckIsAtomicWithTheUnlink` (an `Upsert` issued during the freeze returns after it; a row present at the freeze → kept); `TestRootJournalResolvesEveryIntent` (a crash after each journal line and after the `unlinkat`: each restart ends in removed, kept or absent; a crash after the unlink and before `removed` → `absent`); `TestReferencedBuildIsKept` (another host's link to the build; a process; `Processes` failing → left); `TestProgramsFileHasNoReferenceCheck`; `TestRootOnlyLiveLinkWritersAreExcluded`. The Linux tests that need a root-owned parent run when the test runs as root (the root runner, `root-runner.py`) and are skipped otherwise; the identity and journal logic run as any uid against an injected chain check. |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/release -run 'RootRemoves|WritableLayout|RootRow|RootJournal|ReferencedBuild|ProgramsFile' -count=1`, `go test ./internal/host -run 'LiveLinkWriters' -count=1`. |
| +- [ ] **Step 3: Implement. Step 4: Run to verify they pass**, then the cross-builds of Task 6b. |
| +- [ ] **Step 5: Commit** `root: removes a recorded leftover by fstatat and unlinkat on its opened parent under the registry freeze and the held-stage lock, never on a writable layout, journalling intent and result`. |
| + |
| +### Task 7a: The create holds the hostname key |
| + |
| +**Files:** |
| +- Modify: `internal/console/store.go` (`ALTER TABLE host_allocations ADD COLUMN host_key INTEGER NOT NULL DEFAULT 0`), `internal/console/lifetimes_store.go` (`BeginAllocation` inserts `host_key = 1`; `ClearAllocationHostKey(opID)`; `PendingAllocations` selects `state = 'pending' OR host_key = 1`; `Allocation.HostKey bool`), `internal/console/lifetimes_create.go` (the reservation between `BeginAllocation` and `allocSlotLocked`, line 103), `internal/console/lifetimes_recovery.go` (`recoverAllocationLocked`, line 152) |
| +- Test: `internal/console/lifetimes_hostkey_test.go` |
| + |
| +**Interfaces:** |
| +- **The create**, after `BeginAllocation` (the row is the intent; `admit`'s `PendingForHost` is unchanged): `moves.Reserve(ctx, s.cfg.Releases, {create, a.OpID}, {HostKey(host): {create, a.OpID, host, acct, app, now}})`: held → step 4 as today; `ErrReserved` by a `cleanup` → `settleAllocation(a, lifetimeVoid, errHostUnderCleanup)` (409 "the previous instance at this hostname is still being cleaned up. try again later"); by another holder → void, `errLifetimeSettling`; `ErrUnreadable`, `ErrReservationMismatch` or uncertain → `errLifetimePending` (recovery owns the row). After every terminal state the create path or recovery writes, `s.releaseAllocationKey(a)` = `moves.Release(ctx, s.cfg.Releases, {create, OpID})` (a write, so its success settles an uncertain reservation `Put`) then `ClearAllocationHostKey(OpID)`; a failure leaves `host_key = 1`. |
| +- **"Busy writes no reservation", qualified (F2):** step 2 of the create (`planCreateLocked`, `lifetimes_create.go:74`, `:162`) can retire an orphan slot (`retireSlotOrphanLocked`, `:215`) or clean a stale body (`retireStaleLocked`, `:205`) before `BeginAllocation` and before any reservation. That is the existing create protocol's own work, outside the reservation, and safe beside a cleanup: it acts only on a lifetime its slot no longer vouches for (an orphan slot has no record; a stale body is not the slot's), which a cleanup's condition (1) already judges `AwayMoved`, so that cleanup supersedes; and neither write mints a replica version or a registry row (plan 1's rules). A busy create therefore leaves nothing written by the reservation or by step 4 onward. |
| +- **Recovery** (`recoverAllocationLocked`): first branch, `a.State != pending && a.HostKey` → `releaseAllocationKey`, return; every branch that writes the slot, the stale body or the record first calls the same `moves.Reserve` (its own retry returns the held entry, or writes the entry again under a fresh ETag): `ErrReserved` with no slot of its own → void and release; `ErrReserved` with its own slot → stays pending, logged hourly; unreadable or uncertain → pending. A read that finds no entry of the create is never taken as settlement; only the release's successful write is. |
| + |
| +- [ ] **Step 1: Write the failing tests**: `TestCreateReservesTheHostnameKeyAfterBeginAllocation` (busy by cleanup → 409 cleanup copy, the row void, no slot, no record, no entry; busy by a move → settling; a crash after `BeginAllocation` and before `Reserve` → recovery reserves and continues, or voids when busy; a crash after `Reserve` → recovery continues under its own retry; a crash after the terminal state and before the release → the sweep releases and clears; `LastIncarnation` ignores the void row); `TestCreateUncertainReservationIsSettledByItsRelease` (review 4's F2 interleaving through the real create and recovery: the create's held `Put` delivered after the release fails its condition; `host_key` cleared only after the release's write); `TestPendingAllocationHoldsTheKeyUntilRecoveryEnds`; `TestCreateStepTwoIsOutsideTheReservation` (an orphan slot retired before a busy answer; a cleanup of that host, if any, supersedes); `TestHolderChecksPreserved` (today's create refusals byte for byte). |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/console -run 'HostnameKeyAfter|CreateUncertain|PendingAllocation|CreateStepTwo|HolderChecks' -count=1`. |
| +- [ ] **Step 3: Implement. Step 4: Run to verify they pass**, plus `go test ./internal/console -run 'Lifetime|Allocation' -count=1`. |
| +- [ ] **Step 5: Commit** `console: the lifetime create holds its hostname in the reservation table, with the allocation row as its intent and a release that settles it`. |
| + |
| +### Task 7b: Placement holds every instance's hostname key, and its sweep |
| **Files:** |
| -- Modify: `internal/console/store.go` (`ALTER TABLE host_allocations ADD COLUMN host_key INTEGER NOT NULL DEFAULT 0`), `internal/console/lifetimes_store.go` (`BeginAllocation` inserts `host_key = 1`; `ClearAllocationHostKey(opID)`; `PendingAllocations` selects `state = 'pending' OR host_key = 1`; `Allocation.HostKey bool`), `internal/console/lifetimes_create.go` (the reservation between `BeginAllocation` and `allocSlotLocked`, line 103), `internal/console/lifetimes_recovery.go` (`recoverAllocationLocked`, line 152), `internal/console/fleets.go` (`placeAppTarget`, line 568), `internal/console/offlinereserve.go` (`place` reclaim → `s.settlePlacement`), `internal/console/lifetimes_recovery.go` (`residenceSweep` gains the placement settle) |
| - Create: `internal/console/placereserve.go` |
| -- Test: `internal/console/lifetimes_hostkey_test.go`, `internal/console/placereserve_test.go`, `internal/console/residence_audit_test.go` |
| +- Modify: `internal/console/fleets.go` (`placeAppTarget`, line 568), `internal/console/offlinereserve.go` (`place` reclaim → `s.settlePlacement`), `internal/console/lifetimes_recovery.go` (`residenceSweep` gains the placement settle) |
| +- Test: `internal/console/placereserve_test.go`, `internal/console/residence_audit_test.go` |
| **Interfaces:** |
| -- **The create**, after `BeginAllocation` (the row is the intent; `admit`'s `PendingForHost` is unchanged): `moves.Reserve(ctx, s.cfg.Releases, []string{moves.HostKey(host)}, {create, a.OpID, host, acct, app, now})`: held → step 4 as today; `ErrReserved` by a `cleanup` → `settleAllocation(a, lifetimeVoid, errHostUnderCleanup)` (409 "the previous instance at this hostname is still being cleaned up. try again later"); by another holder → void, `errLifetimeSettling`; `ErrUnreadable`, `ErrReservationMismatch` or uncertain → `errLifetimePending` (recovery owns the row). After every terminal state the create path or recovery writes, `s.releaseAllocationKey(a)` = `moves.Release(create, OpID)` then `ClearAllocationHostKey(OpID)`; a failure leaves `host_key = 1`. |
| -- **Recovery** (`recoverAllocationLocked`): first branch, `a.State != pending && a.HostKey` → `releaseAllocationKey`, return; every branch that writes the slot, the stale body or the record first calls the same `moves.Reserve` (its own retry returns the held entry): `ErrReserved` with no slot of its own → void and release; `ErrReserved` with its own slot → stays pending, logged hourly; unreadable → pending. |
| -- **The placement** (`placeAppTarget`, claims-off branch only; the host-claim branch returns before this as today): after the seed check, the app's record keys are listed (the listing at line 678 moves up, still under `instanceStoreTimeout`), then `id, ref := s.reservePlacement(ctx, acct, app, hosts)`: one `moves.Reserve` with `HostKey(h)` for every listed host, `{place, id}`, and `s.placing[id] = true` (a mutex-guarded set); `ErrReserved` by a `cleanup` → 409 "an instance of this app is still being cleaned up. try again later"; by another → 409 "an instance of this app is moving. try again later"; unreadable or uncertain → 502 "could not reserve the app's instances. try again" (an uncertain write is settled by the sweep); nothing else is written on any refusal. Then app placement, routes and records exactly as today. `defer s.settlePlacement(ctx, id, uncertain)` on every return after the reservation: each host whose `SetRecordRegion` returned an error other than `nil`, `os.ErrNotExist` or a seed conflict is fenced (`s.lockAppHost`, `s.admit(host, "")`, `s.fenceRecord(ctx, acct, app, host, fenceOwner{})`, as `offlineMoveReclaimable` fences); every fence answered → `moves.Release(place, id)`; a fence failing → the entry stays for the sweep; `delete(s.placing, id)`. A record created after the listing came through the create, which holds its own key. |
| -- **The sweep** (in `residenceSweep`): a `place` entry older than `offlineLockGrace` whose id is not in `s.placing` → fence the record of every host key it holds (account and app from the entry), then `moves.Release`; any failure → kept, logged hourly. `offlineReclaimable` for `place` runs the same settle (`reclaimHeld` while `s.placing` has the id). |
| +- **The placement** (`placeAppTarget`, claims-off branch only; the host-claim branch returns before this as today): after the seed check, the app's record keys are listed (the listing at line 678 moves up, still under `instanceStoreTimeout`), then `id, ref := s.reservePlacement(ctx, acct, app, hosts)`: one `moves.Reserve(ctx, s.cfg.Releases, {place, id}, want)` with `want[HostKey(h)] = {place, id, h, acct.Sqid, app, now}` for every listed host (N10: one valid entry per key, its `Host` that key's host), and `s.placing[id] = true` (a mutex-guarded set); `ErrReserved` by a `cleanup` → 409 "an instance of this app is still being cleaned up. try again later"; by another → 409 "an instance of this app is moving. try again later"; unreadable or uncertain → 502 "could not reserve the app's instances. try again" (an uncertain write is settled by the sweep's release); nothing else is written on any refusal. Then app placement, routes and records exactly as today. `defer s.settlePlacement(ctx, id, uncertain)` on every return after the reservation: each host whose `SetRecordRegion` returned an error other than `nil`, `os.ErrNotExist` or a seed conflict is fenced (`s.lockAppHost`, `s.admit(host, "")`, `s.fenceRecord(ctx, acct, app, host, fenceOwner{})`, as `offlineMoveReclaimable` fences); every fence answered → `moves.Release(ctx, s.cfg.Releases, {place, id})`; a fence failing → the entry stays for the sweep; `delete(s.placing, id)`. A record created after the listing came through the create, which holds its own key. |
| +- **The sweep** (in `residenceSweep`): a `place` operation whose entries are older than `offlineLockGrace` and whose id is not in `s.placing` → fence the record of every host key it holds (each entry's host, account and app), then `moves.Release`; any failure → kept, logged hourly. `offlineReclaimable` for `place` runs the same settle (`reclaimHeld` while `s.placing` has the id). |
| - **The audit** `TestEveryResidenceChangeHoldsTheHostnameKey` (`go/ast` over non-test files under `./internal/...`): every call of `provision.SetRecordRegion`, `provision.OfflineFlip`, `provision.OfflineAbort`, `provision.MoveComplete`, `provision.MoveFlip`, `provision.MoveRollBack` and `(*Server).createRecordWithLifetimeLocked` is listed with the reservation that covers it (placement, a move's selection, the create) or its reviewed exemption: `MoveComplete` (`internal/host/movespass.go:1617`, the legacy cut's last act on the source box: tolerated, Self-review notes), `MoveFlip` and `MoveRollBack` (Phase 1, host claims on, where no box judge and so no cleanup exists). |
| -- [ ] **Step 1: Write the failing tests**: `TestCreateReservesTheHostnameKeyAfterBeginAllocation` (busy by cleanup → 409 cleanup copy, the row void, no slot, no record, no entry; busy by a move → settling; a crash after `BeginAllocation` and before `Reserve` → recovery reserves and continues, or voids when busy; a crash after `Reserve` → recovery continues under its own retry; a crash after the terminal state and before the release → the sweep releases and clears; `LastIncarnation` ignores the void row); `TestPendingAllocationHoldsTheKeyUntilRecoveryEnds`; `TestResidenceChangesAndCleanupExclude` (Review Focus 4, with the real create, the real placement door and an offline move selection against a box cleanup's entries); `TestPlacementReservesEveryInstanceFirst` (busy → app placement, routes and records unchanged); `TestPlacementFencesAnUncertainWriteBeforeRelease` (a store that commits the record PUT after the client's context is cancelled: the fence makes it fail its condition, the entry released after the fence); `TestPlacementCrashIsSettledByTheSweep`; `TestEveryResidenceChangeHoldsTheHostnameKey`; `TestHolderChecksPreserved` (today's create and placement refusals byte for byte). |
| -- [ ] **Step 2: Run to verify they fail**: `go test ./internal/console -run 'HostnameKey|PendingAllocation|ResidenceChanges|Placement|EveryResidence|HolderChecks' -count=1`. |
| -- [ ] **Step 3: Implement. Step 4: Run to verify they pass**, plus `go test ./internal/console -run 'Lifetime|Allocation|Place' -count=1`. |
| -- [ ] **Step 5: Commit** `console: the lifetime create and placement hold every hostname they change in the reservation table, with the allocation row and the entry as their intents`. |
| +- [ ] **Step 1: Write the failing tests**: `TestPlacementReservesEveryInstanceFirst` (busy → app placement, routes and records unchanged; the table written holds one valid entry per host key, and plan 3's parser fixture reads it as malformed, which the rollback rule covers); `TestPlacementFencesAnUncertainWriteBeforeRelease` (a store that commits the record PUT after the client's context is cancelled: the fence makes it fail its condition, the entry released after the fence); `TestPlacementCrashIsSettledByTheSweep`; `TestResidenceChangesAndCleanupExclude` (Review Focus 4, with the real create, the real placement door and an offline move selection against a box cleanup's entries); `TestEveryResidenceChangeHoldsTheHostnameKey`; `TestHolderChecksPreserved` (today's placement refusals byte for byte). |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/console -run 'Placement|ResidenceChanges|EveryResidence|HolderChecks' -count=1`. |
| +- [ ] **Step 3: Implement. Step 4: Run to verify they pass**, plus `go test ./internal/console -run 'Place' -count=1`. |
| +- [ ] **Step 5: Commit** `console: placement holds every instance's hostname with one entry per key, fences an uncertain record write before it releases, and a sweep settles a crashed one`. |
| ### Task 8: The reaper |
| @@ -205,31 +315,32 @@ Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and |
| - Test: `internal/host/movesreap_test.go` |
| **Interfaces:** |
| -- `func (w *movesWorker) reapRetired(ctx context.Context, now time.Time) error`: an op directory `<dataDir>/.offline-moves/<host>/<move>/` is removed (`os.RemoveAll` of that directory, then the host directory when empty, then the obligation's delivered page entries under `pageEntryDir(dataDir, host, id)`) only when its `retired.json` is `ended`, `now - EndedAt >= 7 d`, `PublishedRev == Rev`, the page outbox holds no undelivered entry for `(host, id)`, and the drop is absent or root's acknowledgement exists. A directory without `retired.json`, with one in any other state, or `corrupt`, is never removed: a target's park, an aborted move's park and every directory an earlier release wrote stay as they are today. |
| +- `func (w *movesWorker) reapRetired(ctx context.Context, now time.Time) error`: an op directory `<dataDir>/.offline-moves/<host>/<move>/` is removed (`os.RemoveAll` of that directory, then the host directory when empty, then the obligation's delivered page entries under `pageEntryDir(dataDir, host, id)`) only when its `retired.json` is `ended`, `now - EndedAt >= 7 d`, the final repair pass has run (with `RepairKeys` non-empty: `w.repairKeys` once more, `RepairPasses = 2`, `RepairKeys` cleared and the report republished before the removal; an error leaves the directory for the next run), `PublishedRev == Rev`, `PagesPending` is empty, the page outbox holds no undelivered entry for `(host, id)`, and the drop is absent or root's acknowledgement exists. A directory without `retired.json`, with one in any other state, or `corrupt`, is never removed: a target's park, an aborted move's park, an untracked lifetime's park and every directory an earlier release wrote stay as they are today. |
| -- [ ] **Step 1: Write the failing tests**: `TestOpDirReapedSevenDaysAfterEnded`; `TestUnpublishedReportBlocksTheReaper`; `TestUndeliveredPageBlocksTheReaper`; `TestOutstandingRootRequestBlocksTheReaper`; `TestDirectoriesWithoutAnObligationAreNeverReaped`; and Task 5's `TestCleanupReportIsDurableAndBlocksTheReaper` completed with the reaper. |
| -- [ ] **Step 2: Run to verify they fail; Step 3: implement; Step 4: run** `go test ./internal/host -run 'Reap|DurableAndBlocks' -count=1`. |
| +- [ ] **Step 1: Write the failing tests**: `TestOpDirReapedSevenDaysAfterEnded`; `TestFinalRepairPassRunsBeforeTheReaper` (a marker landing between the first and the final pass is repaired before the directory goes; a repair error keeps the directory); `TestUnpublishedReportBlocksTheReaper`; `TestPendingPageBlocksTheReaper`; `TestUndeliveredPageBlocksTheReaper`; `TestOutstandingRootRequestBlocksTheReaper`; `TestDirectoriesWithoutAnObligationAreNeverReaped`; and Task 5a's `TestCleanupReportIsDurableAndBlocksTheReaper` completed with the reaper. |
| +- [ ] **Step 2: Run to verify they fail; Step 3: implement; Step 4: run** `go test ./internal/host -run 'Reap|DurableAndBlocks|FinalRepairPass' -count=1`. |
| - [ ] **Step 5: Commit** `agent: an ended cleanup's directory is removed seven days on, once its report and pages are delivered and root has answered`. |
| ### Task 9: End to end and the census |
| **Files:** |
| -- Test: `internal/e2e/selfserve/cleanup_test.go` (plan 2's bridge `host.E2E`, a clock the test advances, a root pass the test drives with the adopt lock held) |
| +- Test: `internal/e2e/selfserve/cleanup_test.go` (plan 2's bridge `host.E2E`, a clock the test advances, a root pass the test drives with the adopt lock held, a hibernate bucket fake with S3's conditional-delete semantics) |
| - Create: `internal/provision/testdata/plan14-root-reads-self-serve-moves-4.tsv`, `internal/provision/self_serve_moves4_census_test.go`, `docs/superpowers/notes/census-proposal-self-serve-moves-4.md` |
| - Modify: `internal/provision/plan14_root_reads_test.go`, `internal/provision/plan9_b8_census_test.go` (line 372) |
| -- [ ] **Step 1: Write the tests**: `TestCleanupTwentyFourHoursAfterTheObservedFinish` (a forward move; at 24 h every inventoried key carries a delete marker, `lock.json` and `build-state.json` remain, the lifted tombstone is back, root removed the live link and the programs file, the entries are gone, the report is under the move id); `TestCleanupSupersededByAPlacementBack` (the placement is refused while the cleanup holds; after it ends, a placement back to the source runs); `TestCreateUnderCleanupRefusedThenAllowed`; `TestRowMintedUnderTheMarkSupersedes`; `TestOperatorReturnAfterSupersededCleanupStillFindsItsLeftovers` (the return's start door accepts the forward's leftover set after a superseded cleanup that had marked and unmarked); `TestLegacyMoveIntoTheSourceRegionIsTolerated` (switch off: a legacy move back to the source region while the cleanup holds: the cleanup supersedes at its next key, the wake waits for the unmark, the promotion restores the staged data and the instance serves it); `TestCleanupReportInTheMoveLog`. |
| +- [ ] **Step 1: Write the tests**: `TestCleanupTwentyFourHoursAfterTheObservedFinish` (a forward move; at 24 h every inventoried key carries a delete marker, each delete conditional on its ETag, `lock.json` and `build-state.json` remain, the lifted tombstone is back, root removed the live link when its chain is root-only, the programs file is reported left, the entries are gone, the table is still there and empty, the report is under the move id and `carlos move offline status --log` shows it); `TestCleanupSupersededByAPlacementBack` (the placement is refused while the cleanup holds; after it ends, a placement back to the source runs); `TestCreateUnderCleanupRefusedThenAllowed`; `TestRowMintedUnderTheMarkSupersedes`; `TestLateDeleteAfterSupersessionIsRefused` (a held delete delivered after the release and a new wake's write: 412, the instance serves its new data); `TestByteIdenticalRepromotionIsRepaired` (a legacy move back into the source region re-promotes byte-identical files; a held delete lands on one after `QuietUntil`; the next cleanup visit restores it and the instance's next wake restores the full replica); `TestOperatorReturnAfterSupersededCleanupStillFindsItsLeftovers` (the return's start door accepts the forward's leftover set after a superseded cleanup that had marked and unmarked); `TestLegacyMoveIntoTheSourceRegionIsTolerated` (switch off: a legacy move back to the source region while the cleanup holds: the cleanup supersedes at its next key, waits out any `QuietUntil`, the wake waits for the unmark, the promotion restores the staged data and the instance serves it); `TestUntrackedLifetimeMoveFinishes` (an incarnation-0 record with no slot moved by an operator: `gone` reports, the move finishes, no obligation exists); `TestCleanupReportInTheMoveLog`. |
| - [ ] **Step 2: Run them; fix what they find within this plan's scope.** |
| -- [ ] **Step 3: Run both census tests and the cross-builds**; write the census proposal with the loader block, the changed envelopes (`internal/host`, `internal/release`, `internal/registry`, `internal/lease`, `internal/moves`, `internal/console`, `internal/offlinecleanup`) and the units. |
| +- [ ] **Step 3: Run both census tests and the cross-builds**; write the census proposal with the loader block, the changed envelopes (`internal/host`, `internal/release`, `internal/registry`, `internal/lease`, `internal/moves`, `internal/console`, `internal/offlinecleanup`, the root package) and the units. |
| - [ ] **Step 4: Commit** `self-serve moves: cleanup end to end`; push. |
| --- |
| ## Self-review notes |
| -- Spec coverage: §7.6 the source replica (T1, T3, T4), root-owned leftovers with the link and process checks (T6), parked local state (T8), the staged objects of an aborted move (unchanged); §7.1's `cleanup` reservation and the hostname keys (T2, T7); §8's cleanup pages and every cleanup key in the move log (T5); §9 rollout (Global Constraints); §11's cleanup witnesses (T4, T9). |
| -- **Stated deviations.** (1) Legacy moves hold no reservation: a legacy move into the source region carries a move on the record from its door onward, so condition (1) supersedes the cleanup at its next key, and its completion's wake cannot take the lease while the mark holds; after the unmark the wake's promotion mirrors the staged replica over the prefix (`promoteStagedReplica`, an exact mirror), so no deleted key matters. Holding a key from the box at `MoveComplete` (`movespass.go:1617`) would add a box-side release and crash path and extend a legacy cut's downtime behind root's answer, for no exclusion the mark does not already give. (2) Placement keeps `SetRecordRegion`'s ETag-conditional write and settles an uncertain one by `fenceRecord`, after which no earlier PUT can land; the reservation entry is the persisted intent, so no journal table is added. A bound write would add a refusal on unrelated concurrent writes and change the door's answers. (3) A cleanup entry is never reclaimed by another reserver (§7.1 allowed a status-based reclaim): its own box resumes it after any restart, and the only case a status report could settle, a box that died before its first state write, is one whose report is stale and so could never settle it; the console pages instead. (4) Only the sealing box's local paths are cleaned; other boxes in the source region keep theirs, as today; the region-has-one-box check is removed because the mark excludes every box's wake. (5) The create reserves after `BeginAllocation`, not before, so every reservation has an intent row behind it. (6) `build-state.json` is kept with `lock.json`. |
| -- Plan 2 amendments (built code, changed here): move selections hold `host/<hostname>`; the switch-off rule applies between moves only, so a cleanup neither blocks nor is blocked by an operator move on another box; recovery mode counts `move` entries only, so a create, a placement or a cleanup never turns the legacy move doors into the self-serve family's 404; the cleanup reclaim interface is deleted. |
| -- Built-code changes this plan makes, by task: T2 `moves/offline.go` (kinds), `console/offlinereserve.go`, `console/selfserve_pass.go`, `console/selfserve_pages.go`; T3 `lease` (new file only); T4 `host/movespass.go`; T5 `host/movespaging.go`, `console/apiinstancemove_offline.go`; T6 `registry`, `host/agent.go`, `release/programhistory.go` (export); T7 `console/store.go`, `lifetimes_store.go`, `lifetimes_create.go`, `lifetimes_recovery.go`, `fleets.go`. |
| -- Names used across tasks: `offlinecleanup.Retired` (`State`, `Outcome`, `Keys`, `Events`, `Root`, `Rev`, `PublishedRev`, `MarkedAt`, `RootRequestedAt`, `CancelRequestedAt`), `CleanupID`, `InventoryObject`, `KeyOutcome`, `Event`, `Report`, `FileIdentity`, `LocalPath`, `RootCleanupRequest`, `RootCleanupPath`, `RootCleanupAck`, `KeptPath`, `ValidateRetired`; `moves.HostKey`, `IsHostKey`, `Op`, `Conflicts`, `Reserve`, `Holds`, `Release`, `ErrReserved`, `ErrUnreadable`, `ErrUncertain`, `ErrReservationMismatch`, `RecordAway`, `Away*`, `OfflineOpDir`, `RetiredFile`, `CleanupReportKey`, `OfflineKindCreate`, `OfflineKindPlace`; `lease.CleaningOwner`, `MarkCleaning`, `ReassertCleaning`, `UnmarkCleaning`, `ErrNotOurLock`; `w.kickCleanup`, `cleanupRun`, `cleanupMu`, `cfg.PlainMirror`, `cfg.CleanupArmed`, `recordRetired`, `createRetired`, `writeRetired`, `markPublished`, `flushPageOutboxFor`, `reapRetired`; `release.ServeRootCleanupRequests`, `RootCleanupOptions`, `RootCleanupDir`, `ReadRootCleanupAck`, `ErrRootCleanupRefused`, `ProgramsPath`; `rootCleanupAuthority`, `rootCleanupPass`; `registry.WithHostFrozen`; `s.reservePlacement`, `s.settlePlacement`, `s.placing`, `s.residenceSweep`, `s.releaseAllocationKey`, `errHostUnderCleanup`. |
| -- Review Focus 1 to 5 are pinned by T4, T4, T6, T2+T7, T5+T8. |
| +- Spec coverage: §7.6 the source replica (T1, T3, T4a, T4b, T4c), root-owned leftovers with the link and process checks (T6a, T6b, T6c), parked local state (T8), the staged objects of an aborted move (unchanged); §7.1's `cleanup` reservation and the hostname keys (T2, T7a, T7b); §8's cleanup pages and every cleanup key in the move log (T5a, T5b); §9 rollout (Global Constraints); §11's cleanup witnesses (T4a, T4b, T9). |
| +- **Stated deviations.** (1) Legacy moves hold no reservation. A legacy move into the source region carries a move on the record from its door onward, so condition (1) supersedes the cleanup at its next key, and its completion's wake cannot take the lease while the mark holds; after the unmark the wake's promotion mirrors the staged replica over the prefix (`promoteStagedReplica`, an exact mirror, called during the restore after acquisition, `backend_exec.go:819`), so no deleted key matters; review 4's writer audit confirms this normal-operation argument (completion changes the record; promotion waits for acquisition). A late delete from before the supersession is refused by its ETag condition, and the byte-identical copies such a promotion makes wait out `QuietUntil`. Holding a key from the box at `MoveComplete` (`movespass.go:1617`) would add a box-side release and crash path and extend a legacy cut's downtime behind root's answer, for no exclusion the mark does not already give. (2) Placement keeps `SetRecordRegion`'s ETag-conditional write and settles an uncertain one by `fenceRecord`, after which no earlier PUT can land; the reservation entry is the persisted intent, so no journal table is added. (3) A cleanup entry is never reclaimed by another reserver (§7.1 allowed a status-based reclaim); the console pages instead. (4) Only the sealing box's local paths are cleaned; other boxes in the source region keep theirs, as today. (5) The create reserves after `BeginAllocation`, not before, so every reservation has an intent row behind it. (6) `build-state.json` is kept with `lock.json`. (7) §7.6's "plain `DeleteObject`" becomes a conditional one on the inventoried ETag: still a marker, never a version delete. (8) The table is never deleted (plan 2 deleted it when empty). (9) Root removes a leftover only on a root-only chain; the programs file, under the agent-writable `/data` on every built box, is recorded as left at `gone` and stays the runbook's, and so do both paths on a writable-opt box. (10) Root's rename-aside is gone; identity is compared on the opened parent with `fstatat`. (11) `QuietUntil` holds the mark and the reservation for 15 minutes after a run whose deletes were not all answered, for the byte-identical case the ETag condition cannot separate; it is a bound, and the repair (two passes, after the end and before the reaper) removes any marker a later delete still put over a later owner's byte-identical version, by its version id: the one version delete in the plan, on a marker only. |
| +- Plan 2 amendments (built code, changed here): move selections hold `host/<hostname>`; the switch-off rule applies between moves only, so a cleanup neither blocks nor is blocked by an operator move on another box; recovery mode counts `move` entries only; the cleanup reclaim interface is deleted; `releaseBoxes` writes an empty table instead of deleting it; every table write carries `epoch` and `rev`. |
| +- Built-code changes this plan makes, by task: T1 `moves/offline.go`, `host/movesoffline.go`, `release/programhistory.go` (export); T2 `moves/offline.go` (kinds, epoch, rev), `console/offlinereserve.go`, `console/selfserve_pass.go`, `console/selfserve_pages.go`, `console/lifetimes_recovery.go`, `console/lifetimes_store.go`; T3 `lease` (new file only); T4b `lease/mirror.go` (the conditional delete); T4c `host/movespass.go`; T5a `host/movespaging.go`, `console/apiinstancemove_offline.go`; T5b `console/apiinstancemove_offline.go`, `move.go`, `docs/site/cli.md`, the copy record; T6a `registry`; T6b `host/agent.go`; T7a `console/store.go`, `lifetimes_store.go`, `lifetimes_create.go`, `lifetimes_recovery.go`; T7b `fleets.go`, `offlinereserve.go`, `lifetimes_recovery.go`. |
| +- Names used across tasks: `offlinecleanup.Retired` (`State`, `Outcome`, `Keys`, `Events`, `PagesPending`, `LocalLeft`, `Root`, `Rev`, `PublishedRev`, `MarkedAt`, `QuietUntil`, `RunKeys`, `RepairKeys`, `RepairSince`, `RepairPasses`, `RootRequestedAt`, `CancelRequestedAt`), `CorruptStub`, `CleanupID`, `InventoryObject`, `KeyOutcome`, `Event`, `PendingPage`, `Report`, `FileIdentity`, `LocalPath`, `KeptPath`, `RootCleanupRequest`, `RootCleanupPath`, `RootCleanupAck`, `ValidateRetired`, `ValidateCorruptStub`, `ValidateReport`, `Identify`, `RootOnlyChain`; `moves.HostKey`, `IsHostKey`, `Active`, `ReadActive`, `NextActiveBody`, `Op`, `Conflicts`, `Reserve`, `Holds`, `Release`, `ErrReserved`, `ErrUnreadable`, `ErrUncertain`, `ErrReservationMismatch`, `RecordAway`, `Away*`, `OfflineOpDir`, `RetiredFile`, `CleanupReportKey`, `OfflineKindCreate`, `OfflineKindPlace`; `lease.CleaningOwner`, `MarkCleaning`, `ReassertCleaning`, `UnmarkCleaning`, `ErrNotOurLock`, `ConditionalMirrorDeleter`, `ErrDeleteConditionFailed`; `w.cleanupVisit`, `w.supersededBy`, `w.retireKey`, `w.repairKeys`, `w.kickCleanup`, `cleanupRun`, `cleanupMu`, `cfg.PlainMirror`, `cfg.CleanupArmed`, `recordRetired`, `createRetired`, `writeRetired`, `markPublished`, `flushPageOutboxFor`, `reapRetired`; `release.ServeRootCleanupRequests`, `RootCleanupOptions`, `RootCleanupDir`, `ReadRootCleanupAck`, `ErrRootCleanupRefused`, `ProgramsPath`; `rootCleanupAuthority`, `rootCleanupPass`; `registry.WithHostFrozen`; `s.offlineCleanupOf`, `s.reservePlacement`, `s.settlePlacement`, `s.placing`, `s.residenceSweep`, `s.releaseAllocationKey`, `errHostUnderCleanup`; the CLI's `moveArgs.log`. |
| +- Review Focus 1 to 5 are pinned by T4b, T4a, T6b+T6c, T2+T7a+T7b, T5a+T8. |
| +- **Open questions for Paul** (amadan discussion 363, unchanged): (1) legacy moves without a hostname reservation: accept the exemption, or reserve around `MoveComplete` on the box (a box-side release and crash path, and longer legacy-cut downtime while root answers)? (2) Cleanup runs 24 hours after **every** offline move, operator moves included, replacing the runbook's manual cleanup; intended (§10 Q4 talks about self-serve)? (3) Is the box switch `CARLOS_OFFLINE_CLEANUP=on` acceptable as the rollout gate? (4) A move of the same host waits while its cleanup holds `host/<hostname>`, which can last until root answers (paged every 10 min); acceptable? (5) The copy rows: four door sentences, five page subjects and one page body (now also the CLI's lines and the `--log` paragraph). |