amadan
plan 4 rev 4: cleanup excludes admitted work by marking the lifted tombstone; root authority through a host-level callback and root-owned acknowledgements; hostname keys, placement intent, durable release and publication obligations
Answers Astra's review of rev 3 (F1-F13, N1-N3). Departures from the controller's rulings are argued in the rev paragraph and listed for Paul: the legacy move door takes no reservation (the mark tolerates it), placement keeps SetRecordRegion with the reservation as its intent, Exclusive is removed, the reaper's scope is narrowed instead of park provenance files, cleanup entries are never reclaimed, the one-box fleet check is gone, and a box switch CARLOS_OFFLINE_CLEANUP=on gates the rollout.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1 file changed,
+183
−100
docs/superpowers/plans/2026-10-22-self-serve-moves-4-cleanup.md+183 −100
CI failed — run details
CI log
=== gate === sigwho: pid 2 ppid 1 pgid 0 sid 0; child pid 3 .amadan/test-gate.sh gate: 2026-10-24T08:07:21Z on aarch64, 32 cpus, 61401 MB available, go1.26.8 linux/arm64 gate: +0s start launchctx gate: +0s start vet gate: +0s start gofmt gate: +1s start rest gate: +11s start internal_console@0 gate: +11s start internal_console@1 gate: +11s start internal_console@2 gate: +11s start internal_console@3 gate: +11s start internal_console@4 gate: +11s start internal_console@5 gate: +11s start internal_console@6 gate: +11s start internal_console@7 gate: +11s waiting for internal_console's shards before the next split package gate: +172s start root@0 gate: +172s start root@1 gate: +172s start root@2 gate: +172s waiting for root's shards before the next split package amadan: step gate failed: signal: terminated amadan: job failed at step gate — its output is above, not at the tail of this log
diff --git a/docs/superpowers/plans/2026-10-22-self-serve-moves-4-cleanup.md b/docs/superpowers/plans/2026-10-22-self-serve-moves-4-cleanup.md| index 46c41f140..6338a12a3 100644 |
| --- a/docs/superpowers/plans/2026-10-22-self-serve-moves-4-cleanup.md |
| +++ b/docs/superpowers/plans/2026-10-22-self-serve-moves-4-cleanup.md |
| @@ -2,151 +2,234 @@ |
| > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| -Rev 3, 2026-10-22. Rev 1 was reviewed by Astra (`after236/selfserve-plan4-review.md`, NOT READY, 18 findings); rev 2 (f1681c63) answered them and was reviewed (`selfserve-plan4-review2.md`, NOT READY, F1 to F13). Rev 3 answers F1 to F13 on the strength of **Paul's ruling of 2026-10-22** (amadan #343): every change of a hostname's residence goes through the per-box reservation table of plan 2. Concretely: the table gains **hostname keys** beside box keys, and the console's lifetime create (plan 1 Task 3), `carlos apps place` and the legacy move door, the self-serve and operator moves (plan 2) and this plan's cleanup all hold an entry on `host/<hostname>` for the duration of their durable work, so the exclusion is **durable and cross-process**, not a console mutex (F1, F2); condition (1) regains the region check (F1); a root request is preceded by a persisted `publishing` state and reconciled on restart (F3); a corrupt `retired.json` stays a **blocking obligation** that the status report names and never an absence, and the status contract is reconciled with plan 2's reclaim rule (F4); root's unlink has an intent/result journal, re-checks identity and cancellation immediately before `unlinkat` under its own exclusion, and derives its authority from the bucket (the `cleanup` reservation on `host/<hostname>` and the judged record), not from the request body (F5, F6); one ordered transition table with the original finishing move waiting and the fleet check before acquisition (F7); §7.6's process and reference checks are kept with the candidate link excluded (F8); the shared wire types live in a new leaf package `internal/offlinecleanup` and the root filesystem code is platform-tagged (F9); every park class persists an end observation and the reaper measures seven days from it, legacy parks kept (F10); a deadline-bearing cleanup context with per-operation bounds and a lock argument from the adopt entry point (F11); the remaining contract gaps are decided (F12); the report is published under the move id as well as the cleanup id and pending pages are durable per key (F13). |
| +Rev 4, 2026-10-24. Rev 3's review (`after236/selfserve-plan4-review3.md`: F4 closed; F1 to F3, F5 to F13 and N1 to N3 open) is answered, mostly by removing mechanisms. **F1:** the bucket is protected by a **mark**, not by an argument about which writers can run: before its first deletion the cleanup rewrites the source bucket's `lock.json`, by a conditional write from this move's lifted tombstone, into a live lease owned by `cleaning:<cleanup id>`; every writer that mints a replica version under the hibernate prefix either acquires that lease or writes a tombstone over it, and both refuse a live lease of another owner (the inventory is a Global Constraint, file by file, pinned by an AST test); so a reconcile that mints a late row cannot wake the host into the prefix, and the row supersedes the obligation; `build-state.json`, the one object written outside the lease (the cut's request carry), is no longer inventoried. **F2:** the create reserves `host/<hostname>` immediately **after** `BeginAllocation`, so the allocation row is the intent and every create reservation has a row behind it: a refusal or a crash before the row holds nothing, a busy answer voids the row, and a terminal row whose release is owed is revisited by `PendingAllocations`, whose predicate gains `host_key = 1`. **F3:** one state enum (`open`, `finished`, `cleaning`, `publishing`, `root-requested`, `cancel-pending`, `releasing`, `ended`, `corrupt`) drives one transition table; there is no second publication field. **F5, F6, F9:** root's copy of the request, its journal and its acknowledgement live in a root-owned directory under the opt dir; the agent's request is a drop that root copies, checks and removes; root authorises through a callback `internal/host` supplies (the cleanup's entries on the box's own key and on `host/<hostname>` with the request's account and app, the judged record away from this region, the mark), so `internal/release` imports nothing new; the unlink runs inside the registry freeze and the held-stage lock in their existing order and removes only an entry it has first **renamed aside and verified**, so a replacement by any writer, locked or not, is never removed; every journal intent resolves to removed, kept or absent. **F7:** a source row appearing supersedes the obligation (root cancelled and acknowledged, the mark removed, the reservation released). **F8:** the programs file stores Go main-package paths for its own host (`internal/release/programhistory.go:17-34`) and nothing else refers to it, so its checks are identity and row; the live link keeps §7.6's link and process checks, and evidence that cannot be read is retried, never taken as unreferenced. **F10:** the reaper removes only directories whose obligation has ended (§7.6's own scope), so no park without an obligation is ever stamped or removed. **F11:** the cleanup runs in its own goroutine, started by the moves pass through a try-lock, and never takes `w.pass`. **F12:** the switch-dependent `Exclusive` option is removed: the box's acquisition conflicts key for key in every mode, which is safe because the console's switch-off rule is a console-side restriction between moves only, and `control/offline-moves/console.json` is `{console, until}` (`internal/console/selfserve_lease.go:44-48`), with no switch to read; the remaining contract gaps are decided where each task names them. **F13:** the cleanup report is republished under the move id until the persisted `PublishedRev` equals `Rev`; refusals are stable, de-duplicated events inside it; pages go through the existing page outbox; the reaper waits for both. **N1:** the prefix is normalised once, with its trailing slash, before listing, filtering and persistence. **N2:** `releasing` is persisted before the mark is removed and the reservation released, and a restarted obligation deletes only after finding both entries and the mark still its own. **N3:** a placement reserves every instance's hostname key before it writes anything; the entry is its persisted intent; an uncertain record write is fenced (`fenceRecord`) before the release; a crashed placement's entries are fenced and released by the console's sweep. The deviations this costs (legacy moves, the placement write, the cleanup entry's reclaim, other boxes' local paths) are stated in the Self-review notes. Reviews: rev 1 (`after236/selfserve-plan4-review.md`, 18 findings), rev 2 (`selfserve-plan4-review2.md`, F1 to F13), rev 3 (`selfserve-plan4-review3.md`). |
| -Builds on plans 1 to 3: `withHostname`/`admit`, the create protocol and allocation recovery (plan 1 Task 3), judged reads (plan 1 Task 6), `moves.Reserve`/`Release` with `(Kind, ID)` identity and the legacy projection (plan 2 Task 3, extended here with hostname keys), the self-serve finaliser (plan 2 Task 8), the box's `cfg.Versioned` arming (plan 2 Task 6), the box's offline steps. |
| +Builds on plans 1 to 3 as built on `self-serve-moves-3`: the create protocol and allocation recovery (`internal/console/lifetimes_create.go`, `lifetimes_recovery.go`, `lifetimes_store.go`), `fenceRecord` (`lifetimes_create.go:551`), the judge (`provision.LifetimeJudge.Observe`), the reservation table (`moves.OfflineActive`, `ParseOfflineActive`, `OfflineActiveBody`, `OfflineSentinel`; the console's `reserveBoxes`/`releaseBoxes` and three-way reclaim in `internal/console/offlinereserve.go`), the lease's tombstone (`internal/lease/tombstone.go`), the offline source steps (`internal/host/movesoffline.go`), held publication's freeze and lock (`registry.WithHoldsFrozen`, `internal/host/movesheldlock.go`, `heldStageAuthority` in `internal/host/heldstage.go`), and the page outbox (`internal/host/movespaging.go`). |
| -**Goal:** Twenty-four hours after a move the box observed as finished, the source box retires the old region's replica objects from evidence it recorded at finish, under a `cleanup` reservation on its own box and on the hostname, key by key with the four conditions re-read at each deletion, never deleting a version it did not inventory; root removes its own leftovers only through a request it answers with `.done` or `.cancelled` after checking the bucket's authority; parked local state goes seven days after its end observation; nothing times out into a release. |
| +**Goal:** Twenty-four hours after a move the source box observed as finished, the source box retires the old region's replica objects from evidence it recorded at `gone`, under a `cleanup` reservation on its own box and on the hostname and a cleaning mark on the replica's lock, key by key with the four conditions re-read at each deletion, never deleting a version it did not inventory; root removes the old live link and programs file only through a request it copies, authorises from the bucket and answers in its own directory; the obligation's directory goes seven days after it ended; nothing times out into a release. |
| -**Architecture:** At `gone` the source box writes `retired.json` under the move's op directory with the exact bucket inventory (keys and current version ids under the host's hibernate prefix, `lock.json` excluded) and the root-owned local paths with their identities. The box's moves pass runs a dispatcher over every `retired.json` on the box: it observes the finish, waits 24 hours, takes a `cleanup` reservation on `box/<own box>` and `host/<hostname>` under the cleanup's own id, and deletes key by key, re-reading the record (with its region), the slot, the lifted tombstone, the registry and the object's current version before each plain `Delete`; it records a terminal outcome per key; then it records `publishing`, writes `root-cleanup.json`, root verifies the reservation and the record in the bucket and answers `.done` or `.cancelled`, and the reservation is released. Supersession (a later move, a recreation, a deletion, a residence change) ends the obligation and cancels any outstanding root request. Because every residence change of the hostname holds `host/<hostname>` in the same table, none can run while cleanup holds it, and cleanup cannot begin while one does. |
| +**Architecture:** At `gone` the sealing source box writes `retired.json` under the move's op directory: the exact current versions under the normalised hibernate prefix (`lock.json` and `build-state.json` aside) and the identities of the two root-owned paths. A cleanup goroutine, started by the moves pass and never under its lock, walks every obligation through one transition table: observe the finish, wait 24 hours, reserve the box's key and `host/<hostname>`, mark `lock.json`, delete key by key with the four conditions re-read, then hand root a request (or skip it when nothing local remains), then persist `releasing`, restore the lifted tombstone, release and end. Supersession (the record moved on, a row appeared, the lock was taken, the reservation lost) ends the obligation through the same `releasing` step after any root request is cancelled and acknowledged. The lifetime create and placement hold `host/<hostname>` in the same table; a move's selection holds it too. |
| -**Tech Stack:** Go. `internal/offlinecleanup` (new leaf package: the wire types), `internal/moves` (`Retired`, the reservation helper with hostname keys), `internal/host` (the moves pass, the status report, the reaper, root's adopt pass), `internal/release` (root's request handler), `internal/lease` (`VersionedStore`, `MirrorStore`, `LockState`, `IsLiftedTombstone`), `internal/status`, `internal/console` (admission and the residence-change doors), `internal/provision` (`SetRecordRegion` callers), `internal/alerting`. |
| +**Tech Stack:** Go. `internal/offlinecleanup` (new leaf: the wire types), `internal/moves` (host keys, the conflict relation, `Reserve`/`Release`, `RecordAway`), `internal/lease` (the cleaning mark), `internal/host` (the dispatcher, the report, the reaper, root's authority and the agent's root half), `internal/release` (root's request handler), `internal/registry` (`WithHostFrozen`), `internal/console` (the create, placement, the reservation wrappers, recovery mode, the sweeps, the status door's log). |
| -**Spec:** `docs/superpowers/specs/2026-10-21-self-serve-offline-moves-design.md` §7.6, §7.1 (the `cleanup` reservation), §8 (the cleanup pages), §11. Deviation stated and ruled (Paul, 2026-10-22): placement and the legacy move door take a reservation, so §7.1's table serialises every residence change, not only moves. |
| +**Spec:** `docs/superpowers/specs/2026-10-21-self-serve-offline-moves-design.md` §7.6, §7.1 (the `cleanup` reservation), §8 (the cleanup pages), §9 (rollout), §11. Stated deviations: placement and the lifetime create hold the hostname key (Paul, 2026-10-22, amadan #343); the rest are in the Self-review notes. |
| ## Global Constraints |
| -- Every data deletion is a plain `DeleteObject` (a delete marker); `VersionId` deletes are never used for cleanup; `lock.json` is never deleted. |
| -- **Hostname keys in the reservation table (the ruling).** `active.json`'s `Reservations` map holds two key forms: `box/<box id>` (plan 2's per-box entries, today's keys renamed by the normaliser) and `host/<hostname>`. An acquisition names a set of keys; it conflicts when any key is held by another `(Kind, ID)`. Kinds: `move` (plan 2), `cleanup` (this plan), `create` (plan 1's lifetime create: `{create, OpID}` on `host/<hostname>`, acquired in step 1 of the create protocol **before** `BeginAllocation` and released by the allocation's terminal state, recovery holding it meanwhile), `place` (`carlos apps place`, the browser placement, and the legacy move door's `MoveComplete`: `{place, opID}` on `host/<hostname>` and on the boxes of the regions involved for the duration of the record write, released after it). The sentinel `"*"` of plan 2 conflicts with every key. **No code path changes a record's `Region`, creates a record, or deletes a record's replica without holding `host/<hostname>`**; `TestEveryResidenceChangeHoldsTheHostnameKey` (an AST audit over `SetRecordRegion`, `OfflineFlip`, `MoveComplete`, `createRecordWithLifetime`, `cleanupDispatch` callers) pins it. |
| -- **The four conditions at every deletion**, re-read immediately before each `Delete`: (1) the judged observation of the retirement's `(Account, App, Host)` is present and valid, at `Incarnation == Retired.Incarnation`, with `Region == Retired.To` **or a region the record has since moved on to that is not `Retired.From`** (the spec's "names the destination or a later region"), carries no move, and `LastMove.ID == Retired.MoveID` with `Outcome == done`; **and** the hostname's slot is `{Retired.Incarnation, live, owned by the record}`; (2) the source bucket's `<hibernate prefix>/lock.json` is the lifted tombstone of `Retired.MoveID` (`lease.LockState` owner `== moves.TombstoneOwner(MoveID)` and `lease.IsLiftedTombstone`); (3) the box's registry has no row for the host; (4) `ListVersions(bucket, key)` filtered to `Key == key` shows the inventory's `VersionID` as `IsLatest` and not a delete marker. A read error in any → stop this pass (the obligation stays `cleaning`). (1) failing because the record moved on, was recreated or deleted → `superseded`. (1) failing because `Region == Retired.From` (a placement back) → `superseded` (the data is live again). (4) failing because another version is current → that key `superseded`; (4) with no latest object and no marker → that key `absent` (terminal). |
| -- **The exclusion argument** (why the window between (4) and `Delete` is safe), now a **lock argument**: every writer to the hibernate prefix needs the host to reside in this region, and every residence change of the hostname holds `host/<hostname>` in the same table cleanup holds it in: the lifetime create (`create`), placement and the legacy move (`place`), the moves (`move`). The activator's wake needs a registry row, which reconcile mints only from a record naming this region, which no path can produce while cleanup holds the key; litestream runs only in a woken container; the stager writes `moves/`, not the hibernate prefix, and cannot start (its move holds the box). Already-admitted work: a residence change that acquired `host/<hostname>` before cleanup finishes first (cleanup's acquisition is refused `busy` until the key is free), and a create admitted before cleanup acquired is refused at its durable step because its `create` acquisition is attempted **before** `BeginAllocation`, inside `withHostname`, so there is no admitted-but-unreserved create. A second box in the same region cannot wake the host without a row either; cleanup still refuses to run while the fleet shows more than one box in the source region (belt and braces; re-checked before acquisition and under it). |
| -- The `cleanup` reservation is held from before the first deletion through root's `.done` or `.cancelled`; a request root never answers keeps it held and pages every 10 minutes; nothing releases on a timer. |
| -- Root removes a recorded path only when, at the unlink boundary and under its adopt exclusion, the bucket shows `{cleanup, CleanupID}` holding `host/<hostname>` and the judged record away from this region, the entry's identity matches (device, inode, and for a link its target), the registry has no row for the host, and `.cancel` is absent; the identity is re-read immediately before `unlinkat` on the same directory handle. |
| -- The reservation helper lives in `internal/moves` (`Reserve`, `Release`) with typed options and results; the console's `reserveBoxes`/`releaseBoxes` (plan 2) are thin wrappers. |
| -- Every deletion, skip and refusal is durable in `retired.json` (`Keys` and `Refusals`) and published in the cleanup report under **both** `control/offline-moves/<host>/<move id>/cleanup/<box>.json` (so the move's log reader finds it) and `.../<cleanup id>/cleanup/<box>.json`. |
| -- Census: the box's cleanup units and root's handler take rows in `plan14-root-reads-self-serve-moves-4.tsv`, loaded by a new `loadRootReadInventory(..., true)` block in `internal/provision/plan14_root_reads_test.go` after every existing overlay and before `compareRootReads`, `rev3=PROPOSED-self-serve-moves-4`, with rows for the actual readers (the record, the slot, the lock state, the registry, the versions listing, the request file, root's reservation and record reads) at the `authority` class with the evidence basis each needs (`E2,J3,J4` where the destructive reader's obligations apply), not the wrapper's label; both census tests green. |
| -- Copy rows (pending Paul's review): "the previous instance at this hostname is still being cleaned up. try again later"; "cleanup of <host> waits for root"; "cleanup of <host> skipped <key>: another version is current". No tenant names in tests. |
| -- Local tests: touched packages with `-run`; CI runs the full suite. |
| +- **Arming and rollout.** The box acquires a new cleanup reservation only with `CARLOS_OFFLINE_CLEANUP=on` in its environment, on an `OfflineCapable` box whose judge is set (`cfg.Judge != nil`, i.e. host claims off, `internal/host/boxjudge.go:21-26`); `retired.json` is written at `gone` whatever the switch says, and an obligation past `finished` runs to its end if the switch is turned off. Order: boxes, then consoles (plan 4 code everywhere), then `CARLOS_OFFLINE_CLEANUP=on` on the boxes. Rollback: unset the switch, wait until `control/offline-moves/active.json` holds no `cleanup`, `create` or `place` entry (a console before this plan reads those kinds as a malformed table, `moves/offline.go:584-587`), then roll back. |
| +- Every data deletion is a plain `DeleteObject` through `cfg.PlainMirror` (`lease.NewS3Mirror(ctx, region, region)` for the source bucket's region); `VersionId` deletes are never used; `lock.json` and `build-state.json` are never inventoried and never deleted. |
| +- **The prefix (N1).** `Prefix = activator.HibernatePrefix(cfg.OwnPrefix, host) + "/"` (`HibernatePrefix` is `path.Join`, no trailing slash, `internal/activator/activator.go:147`), computed once at `gone` and persisted; the listing is `ListVersions(bucket, Prefix)`; the excluded keys are exactly `Prefix + "lock.json"` and `Prefix + "build-state.json"`; the lease location is `strings.TrimSuffix(Prefix, "/")`. |
| +- **The cleanup id** is `offlinecleanup.CleanupID(box, host, moveID) = "cu_" + hex(sha256(box + "\n" + host + "\n" + moveID))[:16]`: recomputable from the directory and the box, so a corrupt file, root and the agent all name the same obligation. |
| +- **The mark.** Owner `lease.CleaningOwner(id) = "cleaning:" + id`; TTL `cleanupMarkTTL = 1 h`; reasserted whenever less than 30 min remain; condition (2) requires at least `cleanupMarkMargin = 15 min` left (the margin exceeds the per-operation bound of 10 s by far, and covers clock skew as the lease's own `RenewMargin` does). Written only by a conditional PUT from the lifted tombstone of this move (`moves.TombstoneOwner(MoveID)`, expired); removed by a conditional PUT back to exactly that lifted tombstone (`{Generation: as read, ExpiresAt: 1970-01-01T00:00:00Z, Owner: moved:<move id>}`, what `LiftTombstone` writes), so the operator return's leftover set (spec §7.5) is unchanged after any cleanup. An expired mark whose bytes are unchanged still proves no lease was taken since, because every acquisition rewrites the object. |
| +- **The writers the mark excludes** (each writer's check, by file): the wake (`internal/activator/backend_exec.go:701`, `lease.Acquire` in `ExecBackend.Acquire`, step 1 of `doWake` in `activator.go`, before the restore, the staged-replica promotion `promoteStagedReplica` at `backend_exec.go:3397`, `BeforeExec`'s build-state write and litestream replication, all of which run under that lease); the legacy final checkpoint (`internal/host/movespass.go:321-328`, `lease.ReplicateWithLease`, which acquires); the offline seal's checkpoint (`internal/host/movesoffline_seal.go:417`, the same `Checkpoint`) and its tombstone (`movesoffline_seal.go:431`, `lease.Tombstone`, which never replaces a live lease of another owner, so the seal's carry at `:469` is not reached). Not excluded by the mark and harmless: the legacy cut's request carry (`movespass.go:1594`) and the offline seal's carry write only `build-state.json`, which is not inventoried; the seed rollback and finish (`seedwiring.go:247`, `console/seedfinish.go:254`) only delete; the target landing (`movesoffline_land.go:611`, `:1373`) belongs to a move into this region, which holds `host/<hostname>`. `TestHibernatePrefixSitesAreInventoried` (Task 3) pins this list. |
| +- **Reservation keys and the one conflict relation.** Keys are a box id (today's bare label, unchanged), the sentinel `"*"`, or `moves.HostKey(host) = "host/" + host`. Kinds: `move`, `cleanup`, `create`, `place`. An existing entry of another `(Kind, ID)` conflicts with a requested set of keys when (a) it holds a requested key; or (b) one side holds `"*"` and the other holds a box key; or (c) the requester is the console with `cfg.SelfServeMoves != "offline"` and both are of kind `move` (plan 2's one-move-at-a-time rule, now between moves only). Host keys never meet (b) or (c). Every move selection (self-serve and operator) holds `host/<hostname>` beside its box keys or its sentinel; a cleanup holds `[cfg.BoxID, host/<hostname>]`; a create holds `[host/<hostname>]`; a placement holds `host/<h>` for every instance of the app. |
| +- **A cleanup entry is never reclaimed by another reserver.** Its own box resumes it after any crash (its own retry is accepted); a box that never returns leaves an entry the console pages after 6 hours (operator repair). Plan 2's `Server.cleanupReports` and `cleanupReportEvidence` (`offlinereserve.go:95-104`) are deleted. |
| +- **The four conditions at every deletion**, re-read before each `Delete`: (1) `moves.RecordAway(obs, MoveID, Incarnation, From) == AwayYes`: the judged observation is `LifetimeValid` (the judge's verdict that the slot vouches for the record) at `Incarnation`, with no move, `LastMove.ID == MoveID && LastMove.Outcome == "done"`, and `Region != From`; (2) `lock.json`'s owner is `CleaningOwner(ID)` with at least `cleanupMarkMargin` left after any reassert; (3) the box's registry has no row for the host (`w.offlineRow(host)`, `movesoffline.go:996`); (4) `ListVersions(bucket, key)` filtered to `Key == key` shows the inventory's `VersionID` as `IsLatest` and not a delete marker. A read error in any: the obligation stops for this run. (1), (2) or (3) failing definitely: **supersession** (spec §7.6's "superseded as a whole"; never an indefinite refusal). (4): another current version → that key `superseded` and a page; a delete marker current → `marked`; no version and no marker → `absent`. |
| +- The cleanup writes nothing to the registry, the edges file or the offline view, so plan 3's holding entries (built from held rows and the view) are untouched; root's freeze holds the registry's write lock for a rename and an unlink, as held publication's does. |
| +- **Root's contract.** Root removes a path only inside `reg.WithHostFrozen(host, …)` (no row for the host) and then the host's held-stage lock (`withHeldStageLock`, `movesheldlock.go:68`), in the order adopt pass lock > the registry's write lock > the held-stage lock (`movesheldlock.go:26-31`), after the authority callback has answered yes for this request in this pass and `root-cleanup.cancel` is absent, and only the entry it renamed aside with `RENAME_NOREPLACE` and found carrying the recorded identity (device, inode, mode, link target). Nothing inside the held-stage lock reads the network or takes a lock. |
| +- Timings: 24 h after `FinishedAt`; per run 60 s (`cleanupRunDeadline`), 10 s per operation, at most 50 keys per obligation; pages every 10 min while root has not answered or has not acknowledged a cancel, hourly for a corrupt obligation; the console pages a `cleanup` entry older than 6 h and a `create` entry with no allocation row older than `offlineLockGrace` (2 min); a crashed placement is settled after `offlineLockGrace`; the op directory goes 7 d after `EndedAt`; root removes its own directory 30 d after its acknowledgement. |
| +- Census: `internal/provision/testdata/plan14-root-reads-self-serve-moves-4.tsv`, loaded by a new `loadRootReadInventory(root+"/internal/provision/testdata/plan14-root-reads-self-serve-moves-4.tsv", &inv, true)` block in `internal/provision/plan14_root_reads_test.go` after plan 3's block (`:1383`) and its seal check and before `compareRootReads`, `rev3=PROPOSED-self-serve-moves-4`, rows at the `authority` class with evidence `E2,J3,J4` for the destructive readers (the box's record, registry, lock and versions reads; root's drop, `active.json`, record, lock, registry-under-freeze and journal reads); `TestPlan9B8RecordWriteCensus` (`plan9_b8_census_test.go:372`) takes `plan14SelfServeMoves4WriteCensusInventory()` wrapping `plan14SelfServeMoves3WriteCensusInventory()` (`self_serve_moves3_census_test.go:22`) with every new or changed bucket writer as an added or re-pinned unit (`moves.Reserve`, `moves.Release`, `reserveBoxes`, `releaseBoxes`, the cleanup report writer; the placement settle and the place sweep reach `fenceRecord`'s existing record unit); both green. |
| +- Copy rows (pending Paul's review): "the previous instance at this hostname is still being cleaned up. try again later"; "an instance of this app is still being cleaned up. try again later"; "an instance of this app is moving. try again later"; "could not reserve the app's instances. try again"; page subjects "cleanup of <host> waits for root", "cleanup of <host> skipped <key>: another version is current", "cleanup of <host> waits for root to acknowledge its cancellation", "cleanup of <host> cannot read its record", "a cleanup has held <host> for over six hours", each with the body "run carlos move offline status --host <host> --log to see what it waits for". No tenant names in tests (`shop.example.com`, account `bab`, app `shop`). |
| +- Local tests: touched packages with `-run`; CI runs the full suite and `go test -tags e2e ./internal/e2e/...`. |
| ## Review Focus |
| -1. The record changes **after** one deletion (a second move B→C, a recreation in the same region, a deletion, a placement back to the source): the next key's condition (1) fails, nothing more is deleted, the obligation is `superseded`, and an outstanding root request is cancelled (Task 2, `TestCleanupRefusesWhenTheRecordMovedAgainMidPass`). |
| -2. Writer exclusion is durable: a lifetime create, a placement and a legacy move each attempted while cleanup holds `host/<hostname>` are refused at their durable step (409 / busy) and leave nothing behind; cleanup attempted while any of them holds the key is `busy`; a create paused **after** its pre-lock checks and before its `create` acquisition, with cleanup acquiring meanwhile, is refused; the fake-writer witness proves the writer is excluded, not that its version is hidden (Tasks 2 and 5, `TestResidenceChangesAndCleanupExclude`). |
| -3. A delayed finaliser release of the move's reservation cannot touch the cleanup's (distinct `(Kind, ID)`); a `create` entry held by a pending allocation blocks cleanup until recovery ends the allocation (Tasks 2 and 5, `TestCleanupExclusionIsMutualAndDurable`). |
| -4. Root never answers: the reservation stays, a page every 10 minutes with persisted cadence across restarts, no release; root answers after an hour → `.done`, release; a failed release retries; a failed page delivery is logged and retried; a request published but not recorded (crash) is reconciled on restart and cancelled under supersession (Tasks 2 and 4, `TestUnansweredRootRequestHoldsAndPages`, `TestPublishedRequestSurvivesTheCrash`). |
| -5. A cancel landing during root's checks, a root crash after one unlink, and `.done` winning the cancellation race: root deletes nothing after seeing `.cancel` at the boundary, `.cancelled` or `.done` names exactly what was **removed** (from the journal's `removed` lines, never from intents), an intent with no result is reported `unknown`, and the agent releases on either (Task 4, `TestCancelIsAcknowledgedBeforeRelease`, `TestRootCrashAfterOneUnlinkIsReportedFromProgress`). |
| +1. The F1 interleaving: a reconcile paused with an old observation mints the source row after the cleanup marked the lock, and a wake through `ExecBackend.Acquire` then tries the lease: it gets `HeldError` naming `cleaning:<id>`, no version is written under the prefix, the next key's condition (3) supersedes, root is cancelled and acknowledged, the lifted tombstone is restored, the reservation released, and the wake then succeeds (Task 4, `TestAdmittedReconcileAndWakeCannotWriteUnderTheMark`). |
| +2. A crash after every write and every side effect of the lifecycle (after `Reserve` before `cleaning`, after `cleaning` before the mark, after the mark before `MarkedAt`, after a `Delete` before its key outcome, after `releasing` before the unmark, after the unmark before the release, after the release before `ended`), restart, completion: no key is deleted unless both entries and the mark are found to be this cleanup's, an entry an operator removed or a mark replaced turns the obligation `superseded`, and nothing is re-acquired from `cleaning` (Task 4, `TestLifecycleResumesAtEveryBoundary`). |
| +3. Root under attack and crash: a shared-uid writer plants an acknowledgement, a request with another id and a cancel in the agent's directory; an agent publication replaces the live link between root's reference check and its rename, and again after the rename; a route is upserted during the freeze; root crashes after each journal line and each syscall: root removes only the entry it took and verified, the replacement survives, the upsert waits for the freeze, every intent resolves to removed, kept or absent, and the forged files change nothing but stopping (Task 6, `TestRootRemovesOnlyTheEntryItTook`, `TestRootJournalResolvesEveryIntent`, `TestForgedAgentFilesAreHarmless`). |
| +4. Residence changes and cleanup exclude each other through `host/<hostname>`: a create (busy → 409, row void, nothing written), a crash after `BeginAllocation` and after `Reserve`, a terminal row whose release failed; a placement that is busy (nothing written), whose record PUT commits after the client gave up (fenced before release) and whose console crashed (the sweep fences and releases); an offline move selection; with the switch off two creates of different hosts and a cleanup beside an operator move all proceed, and the sentinel never blocks a create (Tasks 2 and 7, `TestConflictRelation`, `TestResidenceChangesAndCleanupExclude`). |
| +5. Publication is durable: the report under the move id survives failed writes and is republished until `PublishedRev == Rev`; a refusal repeated every run is one event with a count; a page fires once per 10-minute tick across restarts; the reaper waits for the report, the pages and root's acknowledgement (Tasks 5 and 8, `TestCleanupReportIsDurableAndBlocksTheReaper`). |
| --- |
| -### Task 1: `retired.json` at finish, the finish observation, and the status report's summary |
| +### Task 1: The obligation and its evidence at `gone` |
| **Files:** |
| -- Create: `internal/offlinecleanup/wire.go` (the leaf package: `Retired`, `InventoryObject`, `KeyOutcome`, `Refusal`, `FileIdentity`, `LocalPath`, `RootCleanupRequest`, `RootCleanupPath`, `RootCleanupAck`, `RetiredSummary`; imports only the standard library), `internal/host/fileid_linux.go`, `internal/host/fileid_other.go` |
| -- Modify: `internal/moves/offline.go` (`OfflineOpDir`, `RetiredFile`), `internal/host/movesoffline.go` (`offlineSourceGone`: when the row is gone, before the `gone` report, write `retired.json` once, state `open`; a present file is never rewritten), `internal/host/movescleanup.go` (new: the dispatcher's first step, the **finish observation**), `internal/host/agent.go` (the status tick fills `p.Retirements`, `p.RetirementsComplete`, `p.RetirementsErr`), `internal/status/snapshot.go` (`Probes.Retirements []offlinecleanup.RetiredSummary`, `Probes.RetirementsComplete bool`, `Probes.RetirementsErr string`; `CollectBox`'s attachment condition includes them; the copies into `BoxMetrics`, private) |
| -- Test: `internal/host/movesoffline_retired_test.go`, `internal/status/snapshot_test.go`, `internal/offlinecleanup/wire_test.go` |
| +- Create: `internal/offlinecleanup/wire.go` (standard library only; every wire type of this plan, `Retired`, `InventoryObject`, `KeyOutcome`, `Event`, `Report`, `FileIdentity`, `LocalPath`, `RootCleanupRequest`, `RootCleanupPath`, `RootCleanupAck`, `KeptPath`, with Tasks 5 and 6 giving the last six their semantics), `internal/host/movescleanup_retired.go`, `internal/host/fileid_linux.go`, `internal/host/fileid_other.go` |
| +- Modify: `internal/moves/offline.go` (`OfflineOpDir`, `RetiredFile`, `CleanupReportKey`), `internal/host/movesoffline.go` (`offlineSourceGone`, line 1493: with `rt == nil` and this box the move's sealer by `cfg.offlineRoles(m)`, `w.recordRetired(ctx, s, rec)` runs before the gone report; an error writes no gone report this pass), `internal/release/programhistory.go` (export `ProgramsPath(bakeDir, host)`, today's `programPath`) |
| +- Test: `internal/offlinecleanup/wire_test.go`, `internal/host/movescleanup_retired_test.go` |
| **Interfaces:** |
| -- `type offlinecleanup.Retired struct { ID string \`json:"id"\` ("cu_" + 16 random hex); MoveID, Host, Account, App string; Incarnation uint64; From, To string; OpenAt time.Time; FinalTXID uint64 (from the immutable seal at `moves.OfflineSealKey(host, moveID)`, validated as the landing validates it; never from the mutable sealer report); TombstoneOwner string; Bucket, BucketRegion string; Prefix string (`activator.HibernatePrefix(cfg.OwnPrefix, host)`, persisted: the actual location, never recomputed); BucketInventory []InventoryObject; LocalInventory []LocalPath; WrittenAt, FinishedAt time.Time; State string; CleaningSince, ReservedAt time.Time; Keys map[string]KeyOutcome; Refusals []Refusal; RootRequest string; EndedAt, LastPagedAt time.Time; Reason string; PendingPages []PendingPage }` (every field tagged snake_case; times `omitzero`). |
| -- `State` ∈ `open`, `finished`, `cleaning`, `bucket-done`, `publishing`, `root-requested`, `cancel-pending`, `ended`, `corrupt` (a file that does not decode or fails validation: renamed `retired.json.corrupt` **and** replaced by a stub `retired.json` `{id (from the directory when the original cannot be read), state: corrupt, reason}`, so the obligation stays listed, blocks the hostname key through its reservation if one is held (the stub keeps `ReservedAt` when recoverable) and pages once per hour until an operator repairs it; never an absence). `RootRequest` ∈ "", `pending`, `done`, `cancelled`; it is **cleared to ""** when `State` becomes `ended`. |
| -- **Validation** (`offlinecleanup.ValidateRetired`): `ID` matches `^cu_[0-9a-f]{16}$`; `MoveID` non-empty; `Host` passes the host shape; `Account`, `App` non-empty; `Incarnation > 0`; `From != To`; `State` in the set; `WrittenAt` non-zero; `Prefix` non-empty and ends with `/`; inventory keys non-empty, unique, under `Prefix`, none `Prefix + "lock.json"`; `LocalInventory` paths among the two canonical ones; a failing validation is `corrupt`. |
| -- `type InventoryObject struct { Key, VersionID, ETag string; Size int64 }`; `type KeyOutcome struct { State string (pending, deleted, marked, superseded, absent); VersionID string; At time.Time; Detail string }`; `type Refusal struct { At time.Time; Condition int (1 to 4); Detail string }` (one entry per stop-the-pass refusal; capped at the last 50); `type FileIdentity struct { Dev, Inode uint64; Mode string (symlink or regular); LinkTarget string }`; `type LocalPath struct { Which string (live-link, programs-file); Path string; Identity FileIdentity }`; `type PendingPage struct { Condition string; Key string; Since time.Time; Attempts int }`. |
| -- `func OfflineOpDir(dataDir, host, moveID string) string`, `func RetiredFile(dataDir, host, moveID string) string`; `fileIdentity(path) (FileIdentity, error)` uses `Lstat` (never follows), records `Mode`, and `Readlink` for a symlink; a path that is absent is omitted; any other error fails the `gone` step this pass (nothing written). |
| -- Evidence at `gone`: the record is still in `offline-finishing` with `Move` present: `OpenAt = *Move.OpenedAt`, `Incarnation = rec.Incarnation`, `From/To` from the move, `FinalTXID` from the seal, `TombstoneOwner = moves.TombstoneOwner(MoveID)`, `Bucket/BucketRegion = cfg.OwnBucket/OwnBucketRegion`, `Prefix` as above; `BucketInventory` from `cfg.Versioned(ctx, bucket, region).ListVersions(ctx, bucket, Prefix)` keeping `IsLatest && !DeleteMarker` and `Key != Prefix + "lock.json"`; `LocalInventory` from `fileIdentity` on `release.LiveLink(opt, host)` and `filepath.Join(dataDir, "programs", host + ".json")`. The write is temp-file, `fsync`, rename, directory `fsync` (every state write below the same; `writeRetired` is the one writer). |
| -- **The finish observation** (`open` → `finished`): the judged record cleared with `LastMove.ID == MoveID && Outcome == done` → `FinishedAt = LastMove.At`; the record still carrying **this** move (`Move != nil && Move.ID == MoveID`, in `offline-finishing`) → wait (never supersession: the original move's finishing has precedence over the supersession rule); any other record state → `superseded` (Task 2's rule). |
| -- `type RetiredSummary struct { ID, MoveID, Host, State, RootRequest string; ReleasePending bool (a reservation is held: `ReservedAt` non-zero and `State != ended`); UpdatedAt int64 (the `retired.json`'s mtime at collection, unix seconds) }`; `BoxMetrics.Retirements`, `RetirementsComplete` (every `*/*/retired.json` and `*.corrupt` under the op root was read), `RetirementsErr`. **The reclaim contract with plan 2** (one statement, both plans): a `cleanup` reservation is reclaimable only when the owning box's fresh report (`Metrics.BoxID` equal to the entry's box key, `GeneratedAt` later than the reservation's `ReservedAt` plus one status interval, so the observation post-dates the reservation) shows `RetirementsComplete` and an entry with that id in `State == ended` with `ReleasePending == false`; an absent id is **never** evidence (a corrupt file is listed as `corrupt`); plan 2's Task 3 reclaim text is amended to this wording. |
| - |
| -- [ ] **Step 1: Write the failing tests**: `TestRetiredJSONWrittenOnceAtGone` (fields, `Prefix`, `FinalTXID` from the seal; `lock.json` excluded; a second gone pass does not rewrite; an identity error writes nothing; `Lstat` semantics for a dangling link); `TestFinishObservedFromTheClearedRecord` (`FinishedAt` from `LastMove.At`; the original move still finishing → unchanged; another `LastMove.ID` → superseded); `TestCorruptRetiredStaysAnObligation` (renamed, the stub listed as `corrupt`, paged hourly, never absent from the summary); `TestValidateRetired` (each rule); `TestStatusRetirementsThroughTheRealPath` (`CollectBox → WriteBox → LoadBoxesComplete → BuildFleet → PublishFleet`: private only; `RetirementsComplete` false with `RetirementsErr` when a directory is unreadable; `ReleasePending` and `UpdatedAt` as defined). |
| -- [ ] **Step 2: Run to verify they fail**: `go test ./internal/host -run 'RetiredJSON|FinishObserved|CorruptRetired|StatusRetirements' -count=1`, `go test ./internal/offlinecleanup -count=1`, `go test ./internal/status -run 'Retirements' -count=1`. |
| -- [ ] **Step 3: Implement.** **Step 4: Run to verify they pass.** |
| -- [ ] **Step 5: Commit** `moves: the source box records retired.json at gone and observes the finish; the status report summarises retirements with completeness, identity and release state`. |
| - |
| -### Task 2: Hostname keys in the reservation table, the shared helper, and the cleanup lifecycle |
| +- `func CleanupID(box, host, moveID string) string` as the Global Constraint; `const MarkPrefix = "cleaning:"`. |
| +- `type Retired struct { V int; ID, MoveID, Host, Account, App, Box string; Incarnation uint64; From, To string; OpenAt time.Time; FinalTXID uint64; TombstoneOwner string; Bucket, BucketRegion, Prefix string; BucketInventory []InventoryObject; LocalInventory []LocalPath; WrittenAt time.Time; State, Outcome string; FinishedAt, ReservedAt, MarkedAt, RootRequestedAt, CancelRequestedAt, EndedAt time.Time; Keys map[string]KeyOutcome; Events []Event; Root *RootCleanupAck; Rev, PublishedRev uint64 }`, every field tagged snake_case, times `omitzero`, `V = 1`. |
| +- **States** (`State`, the only lifecycle field): `open`, `finished`, `cleaning`, `publishing`, `root-requested`, `cancel-pending`, `releasing`, `ended`, `corrupt`. `Outcome` is "" until `releasing`, then `cleaned` or `superseded`. |
| +- `type InventoryObject struct { Key, VersionID, ETag string; Size int64 }`; `type KeyOutcome struct { State string (pending, deleted, marked, superseded, absent); VersionID string; At time.Time }`; `type Event` (Task 5); `type FileIdentity struct { Dev, Inode uint64; Mode string (symlink or regular); LinkTarget string }`; `type LocalPath struct { Which string (live-link or programs-file); Identity FileIdentity }` (root derives the path itself; the agent's file names none). |
| +- `ValidateRetired(r Retired) error`: `V == 1`; `ID == CleanupID(Box, Host, MoveID)`; `MoveID` a plain segment; `Host` passes the host shape; `Account`, `App`, `Box` non-empty; `Incarnation > 0`; `From != To`; `State` in the set; `Outcome` "" before `releasing` and in the set after; `WrittenAt` non-zero; `Prefix` ends with `/`; inventory keys unique, under `Prefix`, neither excluded key; `Which` values unique and known; every `Keys` entry names an inventory key. |
| +- `func OfflineOpDir(dataDir, host, op string) string` (`<dataDir>/.offline-moves/<host>/<op>`, "" on a bad name, the checks `OfflineParkDir` applies); `func RetiredFile(dataDir, host, op string) string`; `func CleanupReportKey(host, op, box string) string` (`OfflineReportPrefix(host, op) + "cleanup/" + box + ".json"`). |
| +- **The evidence at `gone`** (`recordRetired`): the record is in `offline-finishing` carrying this move, and `cfg.Region == m.From` (else nothing is written and the box logs once per day); `OpenAt = *Move.OpenedAt`; `FinalTXID` from the seal at `moves.OfflineSealKey(host, id)` validated by `moves.ValidateOfflineSeal`; `TombstoneOwner = moves.TombstoneOwner(id)`; `Bucket/BucketRegion = cfg.OwnBucket/OwnBucketRegion`; `Prefix` as the Global Constraint; `BucketInventory` = `cfg.Versioned(ctx, Bucket, BucketRegion).ListVersions(ctx, Bucket, Prefix)` keeping `IsLatest && !DeleteMarker && strings.HasPrefix(Key, Prefix)` without the two excluded keys; `LocalInventory` from `fileIdentity` on `release.LiveLink(cfg.OptDir, host)` and `release.ProgramsPath(cfg.DataDir, host)`; `State = open`, `Rev = 1`. |
| +- `fileIdentity(path) (FileIdentity, bool, error)`: `Lstat` (never follows), `Readlink` for a symlink; absent → `false`; any other error fails the step. Off Linux it answers absent for both (the cleanup is Linux-only, as the box is). |
| +- Writes: `createRetired` is create-only (temp file, `fsync`, `os.Link` into place, directory `fsync`; an existing file is never replaced); `writeRetired` (temp, `fsync`, rename, directory `fsync`) increments `Rev` on every write except `markPublished` (Task 5), which sets `PublishedRev` alone. |
| +- **Corrupt:** a file that does not decode or fails `ValidateRetired` is renamed `retired.json.corrupt` and replaced by the stub `{v, id (CleanupID from the box and the directory), move_id, host, box, state: corrupt, written_at}`, which keeps the obligation listed and pages hourly (Task 5) until an operator repairs it; a `corrupt` obligation is never reaped and never released, so any entry or mark it held stays until the repair. |
| + |
| +- [ ] **Step 1: Write the failing tests**: `TestRetiredJSONWrittenOnceAtGoneBySealer` (fields; `FinalTXID` from the seal; a non-sealer source box writes nothing; a second gone pass does not rewrite; an identity error writes nothing and no gone report); `TestRetiredPrefixIsNormalisedOnce` (`cfg.OwnPrefix = "instances"` → `Prefix == "instances/shop.example.com/"`; the fake versioned store was listed with exactly that prefix; `instances/shop.example.com/lock.json` and `.../build-state.json` excluded; `instances/shop.example.co` keys not listed; `ValidateRetired` passes on what gone wrote); `TestCleanupIDIsDeterministic`; `TestValidateRetired` (each rule); `TestCorruptRetiredStaysAnObligation` (renamed, stub listed `corrupt` with the recomputed id); `TestFileIdentityUsesLstat` (a dangling link has an identity; a regular file; absent). |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/offlinecleanup -count=1`, `go test ./internal/host -run 'RetiredJSON|RetiredPrefix|CorruptRetired|FileIdentity' -count=1`. |
| +- [ ] **Step 3: Implement. Step 4: Run to verify they pass.** |
| +- [ ] **Step 5: Commit** `moves: the sealing source box records retired.json at gone, with a normalised prefix and a deterministic cleanup id`. |
| + |
| +### Task 2: Hostname keys and one conflict relation |
| **Files:** |
| -- Create: `internal/moves/reserve.go`, `internal/host/movescleanup.go` (the dispatcher and the bucket step) |
| -- Modify: `internal/moves/offline.go` (`OfflineReservation` unchanged; the key forms `box/<id>`, `host/<hostname>`, `"*"`; the normaliser renames legacy bare box ids to `box/<id>` on the first write), `internal/console/offlinereserve.go` (wrappers over `moves.Reserve`/`Release`; plan 2's selections add `host/<hostname>`), `internal/host/movespass.go` (one `w.cleanupDispatch(ctx, s)` per moves pass, after the record loop and only when this box has no offline record in flight, under a context with a 60 s deadline and per-operation 10 s bounds (`context.WithTimeout` around each read, list, delete and report write); fairness: obligations are visited round-robin from a persisted cursor (`cleanup-cursor` file), each getting at most 50 keys per pass; the offline report refresh is **not** blocked: the dispatcher releases `w.mu` between keys), `armOffline` arms `cfg.Versioned` (plan 2) and `cfg.PlainMirror func(ctx context.Context, bucket, region string) (lease.MirrorStore, error)` (built as `offlineMirror` builds it, through `lease.NewS3Mirror(ctx, srcRegion, dstRegion)` with both regions the source bucket's) |
| -- Test: `internal/moves/reserve_test.go`, `internal/host/movescleanup_test.go` |
| +- Create: `internal/moves/reserve.go` |
| +- Modify: `internal/moves/offline.go` (`OfflineKindCreate = "create"`, `OfflineKindPlace = "place"`; `OfflineReservation.valid()` accepts the four kinds; `HostKey`, `IsHostKey`), `internal/console/lifetimes_store.go` (`AllocationState(opID) (state string, found bool, err error)`), `internal/console/offlinereserve.go` (`reserveBoxes`' steps (4) and (5) become one call to `moves.Conflicts`; the wanted keys add `moves.HostKey(r.Host)`; `releaseBoxes` wraps `moves.Release`; `offlineReclaimable`: `cleanup` → `reclaimHeld` always, `create` and `place` below; `cleanupReportEvidence` and `Server.cleanupReports` deleted), `internal/console/selfserve_pass.go` (`SelfServeEvidence`, lines 41-53, counts only entries of kind `move`), `internal/console/lifetimes_recovery.go` (each iteration also runs `s.residenceSweep(ctx, now)`), `internal/console/selfserve_pages.go` (two page conditions) |
| +- Test: `internal/moves/reserve_test.go`, `internal/console/offlinereserve_hostkey_test.go` |
| **Interfaces:** |
| -- `type moves.ReserveOptions struct { Exclusive bool; Now func() time.Time; Reclaim func(ctx context.Context, key string, held OfflineReservation) (bool, error) }`; `type ErrReserved struct { Key string; Holder OfflineReservation }`; `var ErrUnreadable`; `func Reserve(ctx context.Context, s release.Store, keys []string, r OfflineReservation, opt ReserveOptions) (OfflineReservation, error)` (returns the entry as written or as already held by this `(Kind, ID)`, so a caller that crashed after acquiring recovers `ReservedAt` by calling again; an own retry must cover exactly the expected keys, as plan 2 says; a legacy body is normalised; a malformed entry → `ErrUnreadable`; the projection; one conditional write; one CAS retry); `func Release(ctx context.Context, s release.Store, kind, id string) error`. |
| -- The box's reclaim callback refuses every `move`, `create` and `place` entry (it has no console state) and reclaims a `cleanup` entry only when `key == "box/" + cfg.BoxID` (or the paired `host/` key of the same id) and its own `retired.json` with that id is `ended`; absent or `corrupt` → not reclaimable. `Exclusive` on the box comes from `cfg.SelfServeMoves != "offline"` read from the deployment manifest's console switch published in `control/console.json` (plan 2 writes it) and defaults to true when unreadable. |
| -- `func (w *movesWorker) cleanupDispatch(ctx context.Context, s release.Store) error`, the **ordered transition table** (the first matching row of each state applies; "the record" is the judged observation; `unreadable` anywhere → wait and log once per hour): |
| - - any non-terminal state, the record carries **this** move (`Move.ID == MoveID`) → wait (the original move is finishing; never supersession). |
| - - any non-terminal state, the record moved on (`Move != nil` with another id, `LastMove.ID != MoveID`, `Incarnation` differs, absent, stale, or `Region == From`) → **supersession**: `RootRequest ∈ {publishing, pending}` → write `.cancel`, `State = cancel-pending`; else release (if `ReservedAt` set) → `ended`, `Reason: superseded`, `RootRequest = ""`. |
| - - `open` → the finish observation (Task 1) → `finished`. |
| - - `finished`, `now - FinishedAt >= 24 h` → **the fleet check first** (exactly one box publishes the source region: else stay, page hourly) → `Reserve(["box/" + cfg.BoxID, "host/" + host], {cleanup, r.ID, host, acct, app, now}, opts)` (busy → stay, log once per hour) → `ReservedAt` from the returned entry, `CleaningSince = now`, `State = cleaning`, every inventory key `pending` (one write) → the fleet check **again under the reservation** (a change → release, stay `finished`, page). |
| - - `cleaning`: for each `pending` key in order within the budget: the four conditions (a read error → `Refusals` entry, stop the pass); (1) failing by supersession → the supersession row; (3) failing (a row appeared) → `Refusals`, stop, page (an intended indefinite refusal until an operator or supersession resolves it; the reservation stays); (2) failing (a live lease, another owner, no tombstone) → `Refusals`, stop, page (likewise); (4): the inventory's version is latest → `PlainMirror.Delete(bucket, key)` → `Keys[key] = {deleted, VersionID, now}` (persisted **before** the next key; a crash between the delete and the write → the next pass finds a marker current → `marked`, which is the same end state); a marker current → `marked`; another version current → `superseded` + a `PendingPage{skipped, key}`; no latest and no marker → `absent`; every key terminal → `bucket-done`. |
| - - `bucket-done`: `LocalInventory` empty → release → `ended` (`cleaned`); else `RootRequest = publishing` (persisted **first**) → write `root-cleanup.json` create-only (an existing file for this id → fine) → touch the stage marker → `RootRequest = pending`, `State = root-requested`. |
| - - `publishing` (found on restart): the request file exists → `pending`, `root-requested`; absent → write it, continue; supersession from `publishing` writes `.cancel` regardless (the request may exist). |
| - - `root-requested`: `.done` → validate the ack (`CleanupID`, `Host`, `MoveID` match; else `corrupt` ack → page hourly, stay) → release → `ended` (`cleaned`), `RootRequest = ""`; `.cancelled` → release → `ended`; neither and `now - LastPagedAt >= 10 min` → page ("cleanup of <host> waits for root"), `LastPagedAt = now` (persisted before the page is sent). |
| - - `cancel-pending`: `.done` or `.cancelled` → release → `ended` (`superseded`); neither → page every 10 minutes. |
| - - `ended`: Task 3's reaper. `corrupt`: page hourly, nothing else. |
| - - **Pending pages** (`PendingPages`): a page is persisted before it is sent and removed after a successful send; a failed send increments `Attempts` and is retried next pass; pages survive terminal transitions and are drained before the reaper may remove the directory. |
| -- The cleanup report: role `cleanup` added to `offlineRoles`, phases `cleaning`, `bucket-done`, `root-requested`, `ended`, `superseded`, `corrupt`; `OfflineReport.Cleanup *offlinecleanup.CleanupReport{CleanupID string; MoveID string; Keys map[string]KeyOutcome; Refusals []Refusal; RootRequest string}` (tagged), validated by plan 2's `ValidateOfflineReportFor` matrix (role `cleanup` ↔ those phases with `Cleanup != nil`), rewritten each pass at both paths of the Global Constraint; the move's log reader (`carlos move offline status --log`) lists it under the move. |
| - |
| -- [ ] **Step 1: Write the failing tests** against a `DirStore` (deployment), a versioned fake (data bucket) recording every `Delete` with no version id, a real registry, an injected clock: `TestReserveHostnameKeys` (two key forms; legacy bare ids normalised to `box/`; `host/` conflicts across kinds; `Reserve` returns the held entry on a retry); `TestReserveSharedByConsoleAndBox`; `TestCleanupRunsTwentyFourHoursAfterTheObservedFinish`; `TestFleetCheckBeforeAndUnderTheReservation`; `TestFourConditionsAtEveryKey` (each condition failing between key 1 and key 2: stop, supersede or the per-key outcome as specified; the region check both ways); `TestCleanupRefusesWhenTheRecordMovedAgainMidPass` (B→C; a recreation; a deletion; a placement back to the source → each superseded, root cancelled when requested or publishing); `TestResidenceChangesAndCleanupExclude` (Review Focus 2 with the real `createRecordWithLifetime`, the real placement door and the legacy move door); `TestCleanupSkipsAChangedVersionAndFinishes` (the key superseded, a pending page persisted, sent, removed; a failed send retried); `TestLockJsonNeverDeletedAcrossRetries`; `TestNoVersionIdOnDeletes`; `TestLifecycleResumesAtEveryState` (a crash after each state write and **after each external side effect before its state write**: after the request file, after the marker, after a delete, after a page; restart; completion; `ReservedAt`/`CleaningSince` preserved; `publishing` reconciled); `TestOriginalFinishingMoveIsNotSupersession`; `TestBoxReclaimRefusesOtherKinds`; `TestCleanupReportUnderBothIds`; `TestBudgetAndFairness` (two obligations, slow I/O: the pass ends at its deadline, the cursor advances, the offline report refresh is not blocked). |
| -- [ ] **Step 2: Run to verify they fail**: `go test ./internal/moves -run 'Reserve' -count=1`, `go test ./internal/host -run 'Cleanup|FourConditions|Residence|Lifecycle|LockJson|NoVersionId|BoxReclaim|FleetCheck|Budget' -count=1`. |
| -- [ ] **Step 3: Implement.** **Step 4: Run to verify they pass**, plus `go test ./internal/console -run 'Reserve|Reclaim' -count=1`. |
| -- [ ] **Step 5: Commit** `moves: hostname keys in the reservation table; the source box retires the old replica through one restartable lifecycle, re-reading the four conditions at every key`. |
| - |
| -### Task 3: The per-box reaper with end observations |
| +- `func HostKey(host string) string`; `func IsHostKey(key string) bool` (`strings.HasPrefix(key, "host/")`; a box label never contains `/`, `controlname.ValidLabel`). |
| +- `type Op struct{ Kind, ID string }`; `func Conflicts(table map[string]OfflineReservation, keys []string, mine Op, movesExclusive bool) map[Op]map[string]OfflineReservation` = the Global Constraint's relation, `movesExclusive` applying rule (c) only when `mine.Kind == move`. |
| +- `var ErrUnreadable`, `ErrReservationMismatch`; `type ErrReserved struct { Key string; Holder OfflineReservation }`; `func Reserve(ctx context.Context, s release.Store, keys []string, r OfflineReservation) (OfflineReservation, error)` for the box, the create and the placement: read (absent = empty; a read error or `ParseOfflineActive` failing → `ErrUnreadable`); own entries on a key outside `keys` → `ErrReservationMismatch`; own entries covering `keys` → the held entry returned, nothing written; any conflict → `ErrReserved` naming the oldest conflicting entry (no reclaim on these paths); else one conditional `Put` of the table with the missing keys (at the held entry's `ReservedAt` when partial) and `OfflineActiveBody`'s projection; a CAS conflict re-reads once; a second loss → `ErrReserved` from the last read; a `Put` error that is not a CAS conflict is returned wrapped as uncertain (`errors.Is(err, ErrUncertain)`): the caller keeps its state as if it might hold the keys, and its own retry resolves it. |
| +- `func Holds(table map[string]OfflineReservation, keys []string, mine Op) bool`; `func Release(ctx context.Context, s release.Store, kind, id string) error` (`releaseBoxes`' algorithm, moved: absence is success; a malformed table is an error and is left as it is). |
| +- The console's `reserveBoxes`: `want = sel.reservationKeys()` plus `moves.HostKey(r.Host)`; the own retry is unchanged (an own reservation made before this plan, without the host key, is partial and completed in the same write; an operator operation holding the sentinel covers every box key and has its host key completed); conflicts = `moves.Conflicts(table, want, mine, s.cfg.SelfServeMoves != "offline")`; reclaim as today for `move`; `create`: `s.store.AllocationState(opID)` found and terminal → `reclaimYes`, pending → `reclaimHeld`, not found or an error → `reclaimUnreadable`; `place`: `reclaimHeld` until Task 7 replaces it with `s.settlePlacement`. |
| +- `s.residenceSweep(ctx, now)` (in each lifetime recovery iteration, `lifetimeRecoveryIteration`): a `cleanup` entry older than 6 h → `s.selfServePage(ctx, id, ssPageCleanupHeld, "a cleanup has held <host> for over six hours", body, now)`; a `create` entry older than `offlineLockGrace` with no allocation row → `ssPageCreateOrphan` (same body); Task 7 adds the placement settle. |
| + |
| +- [ ] **Step 1: Write the failing tests**: `TestConflictRelation` (every pair of kinds over box keys, host keys and the sentinel, with and without `movesExclusive`: two creates of different hosts never conflict; a create and a sentinel move of another host do not; a create and a move of the same host do; a cleanup and an operator move on disjoint boxes do not, with the switch off as on; two moves on disjoint boxes conflict only with the switch off); `TestReserveReturnsTheHeldEntry`; `TestReserveCompletesAPartialOwnReservation`; `TestReserveUncertainPutIsNotAnAnswer` (a store that commits and then errors → `ErrUncertain`; the own retry returns the held entry); `TestMoveSelectionsHoldTheHostKey` (a reservation written by plan 2's code, without the host key, is completed by its own retry; a cleanup on the host key makes the move busy); `TestReleaseIsTheConsolesAlgorithm` (plan 2's `releaseBoxes` tests green through the wrapper); `TestCleanupEntriesAreNeverReclaimed`; `TestCreateEntryReclaimOnTheAllocationRow`; `TestRecoveryModeCountsMoveEntriesOnly` (cleanup, create and place entries alone leave `SelfServeEvidence` false); `TestResidenceSweepPagesOldEntries`. |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/moves -run 'Conflict|Reserve|Release' -count=1`, `go test ./internal/console -run 'HostKey|Reserve|Reclaim|Release|RecoveryMode|ResidenceSweep|OwnRetry|Sentinel' -count=1`. |
| +- [ ] **Step 3: Implement. Step 4: Run to verify they pass.** |
| +- [ ] **Step 5: Commit** `moves: hostname keys in the reservation table and one conflict relation shared by the console, the box, creates and placements`. |
| + |
| +### Task 3: The cleaning mark and the writer inventory |
| **Files:** |
| -- Create: `internal/host/movesreap.go` |
| -- Modify: `internal/host/movespass.go` (`w.observeParkEnds(ctx)` and `w.reapParked(ctx, now)` once per moves pass after the dispatcher), `internal/host/movesoffline.go` (every role writes nothing new; the observer below writes `park-ended.json`) |
| -- Test: `internal/host/movesreap_test.go` |
| +- Create: `internal/lease/cleaning.go`, `internal/host/hibernateprefix_inventory_test.go` |
| +- Test: `internal/lease/cleaning_test.go` (over the S3 fake the tombstone tests use) |
| + |
| +**Interfaces:** |
| +- `const CleaningPrefix = "cleaning:"`; `func CleaningOwner(id string) string`; `var ErrNotOurLock` (the lock is absent, another owner's, or this move's tombstone still live: nothing written). |
| +- `func MarkCleaning(ctx context.Context, bucket, location, region, moveOwner, cleaningOwner string, ttl time.Duration, now time.Time) (expires time.Time, err error)`: `readLock` with its ETag (`tombstoneLeaser`, `tombstone.go:99-148`); owner `cleaningOwner` → `ReassertCleaning`; owner `moveOwner` and `IsLiftedTombstone` → `PutObject` `IfMatch` `{Generation: cur.Generation, ExpiresAt: now+ttl, Owner: cleaningOwner}`; else `ErrNotOurLock`; a lost condition → `ErrTombstoneChanged`; `ttl` in `(0, MaxTombstoneTTL]`. |
| +- `func ReassertCleaning(ctx, bucket, location, region, cleaningOwner string, ttl time.Duration, now time.Time) (time.Time, error)`: owner `cleaningOwner`, live or expired → conditional rewrite of the expiry; else `ErrNotOurLock`. |
| +- `func UnmarkCleaning(ctx, bucket, location, region, cleaningOwner, moveOwner string) error`: owner `cleaningOwner` → conditional PUT of `{Generation, ExpiresAt: liftedExpiry, Owner: moveOwner}`; absent or another owner → nil (nothing of ours; the caller logs it); a lost condition → `ErrTombstoneChanged`. |
| +- **The inventory test** (`go/ast` with `go/types` over every non-test file under `./internal/...`): every call of `activator.HibernatePrefix` and `(activator.Config).hibernatePrefix` is in the test's list with its class: `reader`, `lease-gated writer`, `tombstone-gated writer`, `build-state only`, `delete only`, `a move into the region (hostname key)` or `excluded: <reason>`, as the Global Constraint lists them; a new site, or a removed one, fails until the list is edited. |
| + |
| +- [ ] **Step 1: Write the failing tests**: `TestMarkOnlyFromThisMovesLiftedTombstone` (lifted tombstone of this move → marked; of another move, live tombstone, an ordinary lease, absent → `ErrNotOurLock`, nothing written); `TestMarkRacesAnAcquire` (an `Acquire` landing between the read and the PUT → the mark's condition fails); `TestAcquireReplicateAndTombstoneRefuseTheMark` (`Acquire` → `*HeldError` with owner `cleaning:<id>`; `ReplicateWithLease` → the same; `Tombstone` → error, nothing written; `LiftTombstone` with the move's `mine` → `HeldError`); `TestReassertKeepsAnUnchangedExpiredMark` (expired and unchanged → extended; taken and released, so absent → `ErrNotOurLock`); `TestUnmarkRestoresTheLiftedTombstone` (`IsLiftedTombstone(owner, expires)` true afterwards, the generation kept); `TestHibernatePrefixSitesAreInventoried`. |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/lease -run 'Cleaning|Mark|Unmark' -count=1`, `go test ./internal/host -run 'HibernatePrefixSites' -count=1`. |
| +- [ ] **Step 3: Implement. Step 4: Run to verify they pass.** |
| +- [ ] **Step 5: Commit** `lease: the cleaning mark, a live lease written only over this move's lifted tombstone and put back exactly; every hibernate-prefix site inventoried`. |
| + |
| +### Task 4: The cleanup lifecycle on the box |
| + |
| +**Files:** |
| +- Create: `internal/host/movescleanup.go`, `internal/moves/cleanup.go` |
| +- Modify: `internal/host/movespass.go` (`movesWorker` gains `cleanupMu sync.Mutex`; `moveCfg` gains `PlainMirror func(ctx context.Context, region string) (lease.MirrorStore, error)` and `CleanupArmed bool`, set in `armOffline` from `CARLOS_OFFLINE_CLEANUP`; `movesPass` ends with `w.kickCleanup(ctx, s)`) |
| +- Test: `internal/host/movescleanup_test.go`, `internal/moves/cleanup_test.go` |
| + |
| +**Interfaces:** |
| +- `type Away int` (`AwayYes`, `AwayFinishing`, `AwayMoved`, `AwayUnreadable`); `func RecordAway(obs provision.Observation, moveID string, incarnation uint64, from string) Away`: `LifetimeUnreadable`/`LifetimeCorrupt` → `AwayUnreadable`; absent, not `LifetimeValid`, `Rec.Retired` or `Rec.Incarnation != incarnation` → `AwayMoved`; `Rec.Move != nil` → `AwayFinishing` when `Move.ID == moveID`, else `AwayMoved`; `LastMove` nil, another id or not `done` → `AwayMoved`; `Rec.Region == from` → `AwayMoved`; else `AwayYes`. Root's authority (Task 6) calls the same function. |
| +- `func (w *movesWorker) kickCleanup(ctx context.Context, s release.Store)`: nothing unless `cfg.OfflineCapable && cfg.Judge != nil`; `w.cleanupMu.TryLock()` false → return; else a goroutine runs `w.cleanupRun(runCtx, s)` with `runCtx, cancel := context.WithTimeout(ctx, cleanupRunDeadline)` and unlocks when it returns. **Locks:** the goroutine holds `cleanupMu` only, never `w.pass` and never `w.mu` across I/O; `movesPass` holds `w.pass` while it only try-locks `cleanupMu`, so neither waits on the other and `offlineReassert` (`movesoffline.go:795`, which takes `w.pass`) is never behind a cleanup. |
| +- `cleanupRun`: lists `<dataDir>/.offline-moves/*/*/retired.json` and the corrupt stubs, starts after the path recorded in `<dataDir>/.offline-moves/.cleanup-cursor`, visits each obligation in path order until the deadline (each read, list, delete, lock call and report write under `context.WithTimeout(runCtx, 10*time.Second)`), at most 50 keys per obligation, records the cursor after each obligation, then runs publication (Task 5) and the reaper (Task 8). One `PlainMirror` and one `Versioned` store per run. |
| +- **The transition table** (first matching row; "observe" is `moves.RecordAway(cfg.Judge.Observe(ctx, s, Account, App, Host), MoveID, Incarnation, From)`; `AwayUnreadable` anywhere → stay, event `refused/record-unreadable`, next obligation; each state change is one `writeRetired`): |
| + 1. `open`: `AwayFinishing` → stay (the original move's finishing has precedence; never supersession); `AwayYes` → `finished`, `FinishedAt = LastMove.At`; `AwayMoved` → supersede. |
| + 2. `finished`: `AwayMoved` → supersede; `now - FinishedAt < 24 h` or `!cfg.CleanupArmed` → stay; else `moves.Reserve([cfg.BoxID, HostKey(Host)], {cleanup, ID, Host, Account, App, now})`: `ErrReserved` → stay, event `busy`; `ErrUnreadable` or uncertain → stay, event `refused/reservation-unreadable`; held → `ReservedAt` from the returned entry, every inventory key `pending`, `State = cleaning`. |
| + 3. `cleaning`, `publishing`, `root-requested` (**ownership**, before anything else; rows 3 to 5 run in order within one visit): read `active.json`; `!moves.Holds(table, [BoxID, HostKey(Host)], {cleanup, ID})` → supersede (an operator removed it; never re-acquired); a read error → stay. With `MarkedAt` set, in these three states and in `cancel-pending`, the mark is reasserted whenever under 30 min remain (`ErrNotOurLock` → supersede, which in `cancel-pending` changes nothing), so it never lapses while root is waited for. |
| + 4. `cleaning` with `MarkedAt` zero: `MarkCleaning(..., TombstoneOwner, CleaningOwner(ID), cleanupMarkTTL, now)`: `ErrNotOurLock` → supersede; another error → stay; marked → `MarkedAt = now`. |
| + 5. `cleaning`, for each `pending` key in inventory order within the budget: observe → `AwayMoved` → supersede; `w.offlineRow(Host)` → a row → supersede; the mark (reassert when under 30 min remain; then `LockState`: owner not ours or under `cleanupMarkMargin` → `ErrNotOurLock` → supersede); `ListVersions(Bucket, key)` filtered to `Key == key`: latest is the inventory's version → `PlainMirror.Delete(Bucket, key)` → `Keys[key] = {deleted, VersionID, now}` (written before the next key; a crash between the delete and the write finds a marker current next run → `marked`, the same end); a marker current → `marked`; another version current → `superseded` and the skipped page (Task 5); none → `absent`. A read or delete error → event (`versions-unreadable`, `lock-unreadable`, `registry-unreadable`, `delete-failed`), stop this obligation for this run. Every key terminal: `LocalInventory` empty → `releasing`, `Outcome = cleaned`; else `publishing`. |
| + 6. `publishing`, `root-requested`, `cancel-pending`: Task 6's rows. |
| + 7. **Supersede**: from `publishing` or `root-requested` → `cancel-pending` (Task 6), so the reservation and the mark stay until root has acknowledged; from `cancel-pending` nothing changes; from any other state → `releasing`, `Outcome = superseded` (from `open` and `finished` too: `Release` and `UnmarkCleaning` are no-ops when nothing is held, and a `finished` obligation may hold an entry its crash left). |
| + 8. `releasing`: `UnmarkCleaning(..., CleaningOwner(ID), TombstoneOwner)` → error → stay; `moves.Release(cleanup, ID)` → error → stay; → `ended`, `EndedAt = now`. The mark goes before the reservation, so a residence change admitted by the release never meets a mark, and no key is deleted after `releasing` is written. |
| + 9. `ended`: Task 5's publication and Task 8's reaper only. `corrupt`: Task 5's hourly page only. |
| +- **Restart** resumes every state through the same table: `finished` re-runs `Reserve`, whose own retry returns the held entry; `cleaning` never reserves, and deletes only after rows 3 and 4 found both entries and the mark this cleanup's. |
| + |
| +- [ ] **Step 1: Write the failing tests** against a `DirStore` deployment bucket, a versioned fake recording every `Delete` (none with a version id), the lease's S3 fake for `lock.json`, a real registry and an injected clock: `TestRecordAway` (each branch); `TestCleanupRunsTwentyFourHoursAfterTheObservedFinish` (at 23 h 59 m nothing; at 24 h reserved, marked, deleted); `TestCleanupUnarmedLeavesObligationsFinished`; `TestFourConditionsAtEveryKey` (each failing between key 1 and key 2: stop on a read error, supersede on (1), (2), (3), the per-key outcomes of (4)); `TestCleanupSupersedesWhenTheRecordMovesOn` (a second move B to C, a recreation, a deletion, a placement back to `From`: each superseded, nothing more deleted, the lifted tombstone restored, the entries released); `TestCleanupSupersedesWhenARowAppears`; `TestOriginalFinishingMoveIsNotSupersession`; `TestAdmittedReconcileAndWakeCannotWriteUnderTheMark` (Review Focus 1, with `ExecBackend.Acquire` over the fake); `TestLifecycleResumesAtEveryBoundary` (Review Focus 2); `TestOwnershipLostIsSupersession` (an entry deleted by an operator; the mark replaced by a lease; each → `superseded`, no delete); `TestCleanupRunsOutsideThePassLock` (a versioned fake that blocks: `offlineReassert` returns while the run is blocked; the run stops at its deadline; the cursor moves on; a second kick while one runs starts nothing); `TestLockAndBuildStateNeverDeleted`; `TestNoVersionIdOnDeletes`. |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/moves -run 'RecordAway' -count=1`, `go test ./internal/host -run 'Cleanup|FourConditions|AdmittedReconcile|LifecycleResumes|OwnershipLost|OriginalFinishing|LockAndBuildState|NoVersionId' -count=1`. |
| +- [ ] **Step 3: Implement. Step 4: Run to verify they pass.** |
| +- [ ] **Step 5: Commit** `moves: the source box retires the old replica through one restartable lifecycle under a reservation and a cleaning mark, re-reading the four conditions at every key, outside the moves pass lock`. |
| + |
| +### Task 5: Publication: the report under the move, its events, the pages |
| + |
| +**Files:** |
| +- Create: `internal/host/movescleanup_report.go` |
| +- Modify: `internal/host/movespaging.go` (`flushPageOutboxFor(ctx, host, op)`: `flushPageOutbox`'s delivery restricted to `pageEntryDir(dataDir, host, op)`), `internal/console/apiinstancemove_offline.go` (`readOfflineEvidence`, line 233, skips keys under `<prefix>cleanup/` as it skips `offlineReturnKey`; `offlineSelfServeOf` with `?log=1` adds `Cleanup []offlinecleanup.Report` for the record's `LastMove.ID`) |
| +- Test: `internal/host/movescleanup_report_test.go`, `internal/console/offline_cleanup_log_test.go` |
| **Interfaces:** |
| -- `park-ended.json` (`{move_id, host, incarnation, at, reason}`) in an op directory **without** a `retired.json`: written by `observeParkEnds` when the judged record shows `LastMove.ID == id` (`at = LastMove.At`, `reason = LastMove.Outcome`) or `Move.ID == id` has gone from the record and `LastMove.ID` is a later move (`at = now`, `reason = superseded`), or the record is absent and the directory's op id is not referenced by any `retired.json` on this box (`at = now`, `reason = record gone`); written once; so a later move overwriting `LastMove` never loses the evidence, because the observation was taken while it was available (each pass observes every unended park). |
| -- `func (w *movesWorker) reapParked(ctx, now) error`: an op directory is reaped when (a) it has `retired.json` in `ended` with `now - EndedAt >= 7 d` and no `PendingPages`, or (b) it has `park-ended.json` with `now - at >= 7 d`; a directory with neither, or with an outstanding `root-cleanup.json` lacking `.done`/`.cancelled`, or a `retired.json` in any other state, is never reaped; legacy directories (no `retired.json`, no `park-ended.json`, and no record evidence this pass) are left and logged once per day. `os.RemoveAll` of the op directory only; the host directory when empty. |
| +- `type Event struct { Seq uint64; Kind, Key, Detail string; At, LastAt time.Time; Count int }`. Kinds and their stable details: `finished`; `busy` (the holder's kind); `reserved`; `marked`; `refused` (`record-unreadable`, `registry-unreadable`, `lock-unreadable`, `versions-unreadable`, `delete-failed`, `reservation-unreadable`, `mark-failed`, `unmark-failed`, `release-failed`, `report-unwritten`); `skipped` (`Key`); `superseded` (`record-moved`, `row-present`, `lock-taken`, `reservation-lost`); `root-requested`; `cancel-requested`; `root-done`; `root-cancelled`; `ended` (the outcome). An event equal in `Kind`, `Key` and `Detail` to the last one increments its `Count` and `LastAt` instead of appending, so every distinct refusal is kept and a repeated one is counted (the raw error goes to the box log only). |
| +- `type Report struct { V int; CleanupID, MoveID, Host, Box, State, Outcome string; Keys map[string]KeyOutcome; Events []Event; Root *RootCleanupAck; Rev uint64; At time.Time }` at `moves.CleanupReportKey(Host, MoveID, Box)`, a plain replacing `Put`. |
| +- After each obligation's transitions in a run: `PublishedRev < Rev` → write the report from the file → on success `markPublished` (`PublishedRev = Rev`); a failure → event `refused/report-unwritten` (deduped; the event itself bumps `Rev`, so the next run publishes it too). |
| +- **Pages** through `w.firePage(ctx, Host, ID, rule, alerting.LevelWarning, subject, body)` (one page per `(host, op, rule)` in the outbox, durable across restarts): `root-wait-<n>` with `n = floor((now - RootRequestedAt) / 10 min)` for `n >= 1`; `cancel-wait-<n>` from `CancelRequestedAt`; `skipped-<i>` for inventory index `i`; `corrupt-<h>` with `h = floor((now - the stub's written_at) / 1 h)`. Each run then calls `w.flushPageOutboxFor(ctx, Host, ID)`, whatever `cfg.Enabled` says, so the outbox's other entries keep plan 6's rule (with the switch off they wait on disk). |
| -- [ ] **Step 1: Write the failing tests**: `TestParkedStateReapedSevenDaysAfterTheObligationEnds`; `TestParkEndObservedOnceAndSurvivesALaterMove` (a second move inside seven days overwrites `LastMove`; the park is still reaped seven days after its own end); `TestOldParkWithoutEvidenceIsLeft` (an old directory, record absent today → `park-ended.json` today, reaped seven days later, not today); `TestOutstandingRootRequestIsNeverReaped`; `TestPendingPagesBlockTheReaper`; `TestLegacyDirectoriesAreLeft`. |
| -- [ ] **Step 2: Run to verify they fail; Step 3: implement; Step 4: run** `go test ./internal/host -run 'Reap|ParkEnd' -count=1`. |
| -- [ ] **Step 5: Commit** `agent: parked offline-move state records its end observation and is reaped seven days after it`. |
| +- [ ] **Step 1: Write the failing tests**: `TestCleanupReportIsDurableAndBlocksTheReaper` (Review Focus 5, with Task 8's reaper once it exists; until then the report half: a failing `Put` for three runs, then success: one report with every event and `PublishedRev == Rev`); `TestRefusalsAreStableDedupedEvents` (a read error every run for an hour is one event with its count; a different detail appends); `TestPagesFireOncePerTickAcrossRestarts` (root unanswered for 35 min across a restart at 15 min: pages `root-wait-1`, `-2`, `-3`, each once); `TestPageFlushIsScopedToTheCleanup` (`cfg.Enabled` false: the cleanup's undelivered page is retried, another move's entry is not touched); `TestReadOfflineEvidenceSkipsCleanupReports`; `TestOfflineStatusLogListsCleanup`. |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/host -run 'CleanupReport|RefusalsAre|PagesFire|PageFlush' -count=1`, `go test ./internal/console -run 'SkipsCleanup|StatusLogListsCleanup' -count=1`. |
| +- [ ] **Step 3: Implement. Step 4: Run to verify they pass.** |
| +- [ ] **Step 5: Commit** `moves: the cleanup's report under the move id, republished until it lands, with stable de-duplicated events and paged through the outbox`. |
| -### Task 4: Root's leftovers through a request it verifies in the bucket |
| +### Task 6: Root's leftovers |
| **Files:** |
| -- Create: `internal/release/rootcleanup.go` (the protocol, portable), `internal/release/rootcleanup_linux.go` (`fstatat`, `readlinkat`, `unlinkat`), `internal/release/rootcleanup_other.go` (an implementation that answers every request `.cancelled` with `Reason: "unsupported platform"`), `internal/host/movescleanup_root.go` |
| -- Modify: `internal/host/agent.go` (`runAdoptUnderRootRelease`: `rootCleanupPass(ctx, dataDir, release.OptDir(), store, reg, locked bool, now func() time.Time)` beside `heldStagePass`, in the stage-requested and hourly modes, nonfatal to adoption, `locked` from the pass's own lock result (`passLockErr == nil`)) |
| -- Test: `internal/release/rootcleanup_test.go`, `internal/host/movescleanup_root_test.go` |
| +- Create: `internal/release/rootcleanup.go` (the protocol, portable), `internal/release/rootcleanup_linux.go` (`renameat2`, `fstatat`, `readlinkat`, `unlinkat` through `golang.org/x/sys/unix` on handles from `safefile.OpenDir`), `internal/release/rootcleanup_other.go` (answers every request `cancelled` with `Reason: "unsupported platform"`), `internal/host/movescleanup_root.go` (the agent's rows, the authority callback, `rootCleanupPass`) |
| +- Modify: `internal/registry/registry.go` (`WithHostFrozen`), `internal/registry/remote_dispatch.go` (the `WithHostFrozen` method, in `WithHoldsFrozen`'s form at line 228), `internal/host/agent.go` (`runAdoptUnderRootRelease`, line 4932: `rootCleanupPass(...)` after `heldStagePass`, only when `passLockErr == nil`, nonfatal) |
| +- Test: `internal/release/rootcleanup_test.go`, `internal/release/rootcleanup_linux_test.go`, `internal/host/movescleanup_root_test.go`, `internal/registry/hostfrozen_test.go` |
| **Interfaces:** |
| -- Files under `OfflineOpDir`: `root-cleanup.json` (`offlinecleanup.RootCleanupRequest{Host, MoveID, CleanupID string; Paths []RootCleanupPath{Which string; Identity FileIdentity}; WrittenAt time.Time}`), `root-cleanup.journal` (lines `intent <which>` before each unlink and `removed <which>` after), `root-cleanup.cancel` (the agent's), `root-cleanup.done` and `root-cleanup.cancelled` (`RootCleanupAck{CleanupID, Host, MoveID string; Removed, Kept, Unknown []string; Reason string; At time.Time}`; `Unknown` = intents with no `removed` line). |
| -- **Authority** (`release.ServeRootCleanupRequests(ctx, dataDir, optDir string, s release.Store, reg rowAbsent func(host string) (bool, error), locked bool, now func() time.Time) error`): `locked == false` → return; for each request without an ack (listed through `safefile`, the host from the directory name passing `ValidHostShape`, the move id its parent): read bounded (4096 bytes), validate (`Host`, `MoveID` agree with the directory; `CleanupID` matches `^cu_`; each `Which` known and unique; a malformed request → `.cancelled` `Reason: "malformed request"`); **then the bucket**: `active.json` must hold `{cleanup, CleanupID}` on `host/<Host>` (else `.cancelled` `Reason: "no cleanup reservation"`), and the judged record of the hostname (found through the request's account and app, which the request carries: add `Account, App` to the request) must be valid with `Region != this box's region` and `LastMove.ID == MoveID` (else `.cancelled` `Reason: "the instance is not away"`); a bucket read error → leave the request for the next pass. Root derives the two paths itself (`release.LiveLink(optDir, host)`, `filepath.Join(dataDir, "programs", host + ".json")`). **The spec's process and reference checks** (§7.6), with the candidate link excluded: for the live link, no other host's live link under `optDir` targets the same build **and** no running process (procfs scan of `/proc/*/exe` and `/proc/*/cwd`) lives under the build the link targets → else `Kept` ("the build is still referenced"); for the programs file, no other programs file references the host → else `Kept`. |
| -- **The unlink boundary**, per `Which`, inside `release.WithRootCleanupLock(host)` (a flock on `<OfflineOpDir>/root-cleanup.lock`, root-only, serialising root's cleanup against root's own adopt writes to the same paths, which are the only other writers of the live link and programs file: the agent never writes them): open the parent (`safefile.OpenDir`); `fstatat` by name (no follow) and `readlinkat` for a link; compare with the recorded identity (`Dev`, `Inode`, `Mode`, `LinkTarget`) → mismatch → `Kept` ("identity changed"); `rowAbsent(host)` → a row → `Kept`; append `intent <which>` to the journal and `fsync`; **re-check `.cancel` and re-`fstatat` the entry** (identity unchanged, else `Kept`); `unlinkat`; append `removed <which>`, `fsync`. `.cancel` present at the re-check → stop, write `.cancelled` from the journal. After all paths → `.done` from the journal (`Removed` = `removed` lines, `Unknown` = intents without one, `Kept` with reasons). A request older than 30 days without an answer → `.cancelled` `Reason: "stale request"`. Read, append and unlink failures: the pass stops for this request and retries next time (the journal keeps the truth); an ack write failure → retried next pass. |
| -- The agent: `writeRootCleanupRequest(dataDir string, r offlinecleanup.Retired) error` (create-only); the dispatcher (Task 2) handles `.done`/`.cancelled`/paging/cancel; an ack whose identity does not match the request → the obligation `corrupt` (paged). |
| +- **Files.** The agent's drop `<OfflineOpDir>/root-cleanup.json` (create-only) and cancel `<OfflineOpDir>/root-cleanup.cancel` (create-only). Root's directory `release.RootCleanupDir(optDir, id) = <optDir>/root-cleanup/<id>/` holding `request.json` (root's copy), `journal`, and exactly one of `done.json`, `cancelled.json`; `<optDir>/root-cleanup/` and each `<id>/` are created by root `0755` and used only when `fstat` on the opened handle shows owner uid 0 and no group or other write bit (else nothing this pass, logged). The agent reads the acknowledgement there with `safefile` (no symlinks); on a hardened box its uid cannot write that directory, and on a writable box the agent is already trusted with the opt dir, the same trust the live link rests on. |
| +- The wire types (declared in Task 1): `type RootCleanupRequest struct { V int; CleanupID, MoveID, Host, Account, App, Box string; Incarnation uint64; From, To string; Bucket, BucketRegion, Prefix string; Paths []RootCleanupPath; WrittenAt time.Time }`; `type RootCleanupPath struct { Which string; Identity FileIdentity }`; `type RootCleanupAck struct { V int; CleanupID, MoveID, Host, Outcome string (done or cancelled); Removed, Absent []string; Kept []KeptPath; Reason string; At time.Time }`; `type KeptPath struct { Which, Reason string }`. |
| +- `type RootCleanupOptions struct { DataDir, OptDir, Box, Region string; Authorize func(ctx context.Context, req offlinecleanup.RootCleanupRequest) error; Freeze func(host string, fn func(rowPresent bool) error) error; HostLock func(host string, fn func() error) error; Processes func() ([]string, error); Now func() time.Time }`; `var ErrRootCleanupRefused` (the callback's definite no; any other error is a read failure and leaves the request for the next pass); `func ServeRootCleanupRequests(ctx context.Context, o RootCleanupOptions) error`. |
| +- **Root, per drop** (found by a `safefile` walk of `<DataDir>/.offline-moves/<host>/<move>/root-cleanup.json`, host passing `ValidHostShape`, move a plain segment; `id = CleanupID(o.Box, host, move)`): |
| + 1. Read bounded (4096 bytes) and validate: `Host`, `MoveID` equal the path; `Box == o.Box`; `CleanupID == id`; `From == o.Region`; `Which` values known and unique. Invalid → `cancelled` under `id` with `Reason: "malformed request"`. |
| + 2. Copy: `request.json` create-only with the drop's bytes; present with other bytes → `cancelled`, `Reason: "the request changed"`; then the drop is removed. From here root reads only its copy. |
| + 3. Recovery from the journal, per `Which`: last line `intent` and the aside name `.carlos-cleanup-<id>-<which>` present → as `taken`; `intent` and no aside → nothing was taken (the next attempt journals a new intent); `taken` and the aside present → resume at the identity check; `taken` and no aside → the unlink happened: `fsync` the parent, journal `removed`. Terminal lines are `removed`, `kept <reason>`, `absent`; the acknowledgement is built from them alone, so no outcome is unknown. |
| + 4. `o.Authorize(ctx, req)`: `ErrRootCleanupRefused` → `cancelled` with its reason, nothing touched; another error → the next pass. |
| + 5. Per `Which` not yet terminal (`live-link` first; root derives the path: `LiveLink(OptDir, host)`, `ProgramsPath(DataDir, host)`): for the live link only, outside the freeze: the build directory it targets; another link under `<OptDir>/live/` resolving to the same build → `kept the build is still referenced`; `o.Processes()` naming an executable under the build → `kept a process runs the build`; `Processes` failing → the next pass (unknown is never unreferenced). The programs file has no reference check: its schema (`{schema, programs}`, Go main-package paths, `programhistory.go:28-34`) belongs to this host alone and no reader resolves another host through it. Then `o.Authorize` again (the same two answers). Then `o.Freeze(host, func(rowPresent) { rowPresent → kept the instance has a row on this box; o.HostLock(host, func() { root-cleanup.cancel present (Lstat) → kept cancelled; journal intent and fsync; renameat2(parent, name, parent, aside, RENAME_NOREPLACE): ENOENT → journal absent; fsync the parent; journal taken; fstatat(aside, AT_SYMLINK_NOFOLLOW) and readlinkat; identity differs → renameat2(aside, name, RENAME_NOREPLACE), or on EEXIST (a newer entry is at the name) unlinkat(aside); fsync the parent; journal kept identity changed; identity matches → unlinkat(aside); fsync the parent; journal removed and fsync }) })`. |
| + 6. Every `Which` terminal → `cancelled.json` when the authority refused or any path was kept for `cancelled`, else `done.json` (temp, `fsync`, rename, directory `fsync`). |
| + 7. `<optDir>/root-cleanup/<id>/` older than 30 days past its acknowledgement is removed by root. |
| +- **The authority** (`internal/host`): `func rootCleanupAuthority(store release.Store, judge *provision.LifetimeJudge, box, region string, env func(string) string) func(context.Context, offlinecleanup.RootCleanupRequest) error`; `func rootCleanupPass(ctx context.Context, dataDir, optDir string, reg *registry.Registry, store release.Store, judge *provision.LifetimeJudge, box, region string)` with `judge` the root pass's `rootJudge` (`agent.go`, `newBoxJudge`), `box` `BoxLabel()` (the same label the agent uses as `cfg.BoxID`, `movespass.go:342-346`) and `region` root's `CARLOS_REGION`: (a) `active.json` parses and both `table[box]` and `table[moves.HostKey(Host)]` are `{cleanup, CleanupID, Host, Account, App}`; (b) `moves.RecordAway(judge.Observe(ctx, store, Account, App, Host), MoveID, Incarnation, From) == AwayYes`; (c) `req.Bucket`, `BucketRegion` and `Prefix` equal root's own `CARLOS_HIBERNATE_BUCKET`, `CARLOS_HIBERNATE_REGION` and the prefix built from `CARLOS_HIBERNATE_PREFIX` (unset → a read failure, logged), and `lease.LockState` at `strings.TrimSuffix(Prefix, "/")` names owner `lease.CleaningOwner(CleanupID)` (live or expired: the owner alone proves no lease was taken since the mark). (a), (b), (c) answering no → `ErrRootCleanupRefused` with the reason; a read error → itself. `Freeze` is `reg.WithHostFrozen`; `HostLock` is `withHeldStageLock(dataDir, ·, ·)`; `Processes` reads `/proc/*/exe` (`ENOENT`/`ESRCH` skipped as exited, any other error returned). |
| +- `func (r *Registry) WithHostFrozen(host string, fn func(row *Route) error) error`: `BEGIN IMMEDIATE`, `SELECT routeCols FROM routes WHERE host = ?` on the same connection, `fn`, `ROLLBACK`; remote: method `WithHostFrozen`, the callback carrying the row. |
| +- **The agent's rows** (Task 4's table): `publishing`: the drop absent and `RootCleanupDir` absent → `writeRootCleanupRequest` (create-only; an existing drop with equal bytes is fine) → touch the stage-request marker (`holdStageRequester(dataDir)(Host)`) → `RootRequestedAt = now`, `root-requested`, event; either present → `root-requested`. `root-requested`: `release.ReadRootCleanupAck(optDir, ID)` → an acknowledgement whose `CleanupID`, `Host`, `MoveID` match → `Root = ack`, event, `releasing`, `Outcome = cleaned`; a mismatch → page `corrupt-<h>`, stay; none → the `root-wait` pages. `cancel-pending`: the cancel file written create-only and `CancelRequestedAt` set (once); an acknowledgement → `releasing`, `Outcome = superseded`; else the drop is read **first** and then `RootCleanupDir`: both absent → `releasing`, `Outcome = superseded` (root copies before it removes the drop, so a drop already gone means root's directory exists, and none was ever written after `publishing`); else the `cancel-wait` pages. |
| + |
| +- [ ] **Step 1: Write the failing tests**: `TestRootRemovesOnlyTheEntryItTook` (Review Focus 3's replacements: before the rename → kept, the replacement back at its name; after the rename and before the identity check → the newer entry at the name survives, the aside removed); `TestRootRowCheckIsAtomicWithTheUnlink` (an `Upsert` issued during the freeze returns after it; a row present at the freeze → kept); `TestRootAuthorityComesFromTheBucket` (no entries; only the host key; another account or app; the record home again; the mark not ours; the request's bucket not root's → cancelled, nothing touched; a read error → left); `TestForgedAgentFilesAreHarmless` (a `done.json` planted in the agent's directory is never read; a drop with another id is cancelled under the expected id; a forged cancel stops root and nothing else); `TestRootJournalResolvesEveryIntent` (a crash after each journal line and after the rename and the unlink: each restart ends in removed, kept or absent); `TestCancelBeforeAndDuringRootsWork`; `TestReferencedBuildIsKept` (another host's link to the build; a process; `Processes` failing → left); `TestProgramsFileHasNoReferenceCheck`; `TestRootDirOwnershipIsVerified`; `TestAgentReadsOnlyRootsAcknowledgement`; `TestUnansweredRootRequestPagesEveryTenMinutes`; `TestCancelPendingResolvesWithoutARequest` (a crash after `publishing` before the drop; root never wrote a directory); `TestRootCleanupNeedsTheAdoptLock`; `TestUnsupportedPlatformCancels`; `TestWithHostFrozenLocalAndRemote`. |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/release -run 'RootCleanup|RootRemoves|RootRow|RootAuthority|ForgedAgent|RootJournal|CancelBefore|ReferencedBuild|ProgramsFile|RootDir|Unsupported' -count=1`, `go test ./internal/host -run 'RootCleanup|AgentReadsOnly|Unanswered|CancelPending' -count=1`, `go test ./internal/registry -run 'HostFrozen' -count=1`. |
| +- [ ] **Step 3: Implement. Step 4: Run to verify they pass**, then `GOOS=darwin go build ./internal/release/ ./internal/host/` and `GOOS=windows go vet ./internal/release/ ./internal/host/`. |
| +- [ ] **Step 5: Commit** `root: removes an old instance's live link and programs file only through a request it copies and authorises from the bucket, taking each entry aside and verifying it before the unlink`. |
| + |
| +### Task 7: Creates and placements hold the hostname key |
| -- [ ] **Step 1: Write the failing tests**: `TestRootRemovesOnlyRecordedIdentities`; `TestRootAuthorityComesFromTheBucket` (no reservation → cancelled; the record home again → cancelled; a forged request for a host that is live → cancelled, nothing unlinked); `TestProcessAndReferenceChecksKeep` (another host's link to the same build; a process under the build; another programs file naming the host); `TestCancelIsAcknowledgedBeforeRelease` (`.cancel` landing between the intent line and the re-check → nothing unlinked, `.cancelled` with `Unknown` empty; after one unlink → `.cancelled` names exactly that path from `removed`); `TestRootCrashAfterOneUnlinkIsReportedFromProgress` (a crash after `intent` and before `unlinkat` → `Unknown`, not `Removed`; after `unlinkat` and before `removed` → the restart re-checks the entry: absent → `removed`); `TestEntryReplacedBeforeUnlinkIsKept` (the entry replaced between the first `fstatat` and the re-check → `Kept`); `TestUnansweredRootRequestHoldsAndPages`; `TestPublishedRequestSurvivesTheCrash`; `TestNoLocalLeftoversEndsAtOnce`; `TestRootCleanupNeedsTheAdoptLock`; `TestRootCleanupIsServedInBothModes`; `TestUnsupportedPlatformCancels`. |
| -- [ ] **Step 2: Run to verify they fail**: `go test ./internal/release -run 'RootCleanup|RootRemoves|RootAuthority|ProcessAndReference|CancelIs|RootCrash|EntryReplaced' -count=1`, `go test ./internal/host -run 'RootCleanup|Unanswered|PublishedRequest|NoLocalLeftovers' -count=1`. |
| -- [ ] **Step 3: Implement.** **Step 4: Run to verify they pass**, then the root-read census and `GOOS=darwin go build ./internal/release/ ./internal/host/`, `GOOS=windows go vet ./internal/release/ ./internal/host/`. |
| -- [ ] **Step 5: Commit** `root: removes an old instance's live link and programs file only through a cleanup request it verifies in the bucket, journalling intent and result at the unlink`. |
| +**Files:** |
| +- Modify: `internal/console/store.go` (`ALTER TABLE host_allocations ADD COLUMN host_key INTEGER NOT NULL DEFAULT 0`), `internal/console/lifetimes_store.go` (`BeginAllocation` inserts `host_key = 1`; `ClearAllocationHostKey(opID)`; `PendingAllocations` selects `state = 'pending' OR host_key = 1`; `Allocation.HostKey bool`), `internal/console/lifetimes_create.go` (the reservation between `BeginAllocation` and `allocSlotLocked`, line 103), `internal/console/lifetimes_recovery.go` (`recoverAllocationLocked`, line 152), `internal/console/fleets.go` (`placeAppTarget`, line 568), `internal/console/offlinereserve.go` (`place` reclaim → `s.settlePlacement`), `internal/console/lifetimes_recovery.go` (`residenceSweep` gains the placement settle) |
| +- Create: `internal/console/placereserve.go` |
| +- Test: `internal/console/lifetimes_hostkey_test.go`, `internal/console/placereserve_test.go`, `internal/console/residence_audit_test.go` |
| + |
| +**Interfaces:** |
| +- **The create**, after `BeginAllocation` (the row is the intent; `admit`'s `PendingForHost` is unchanged): `moves.Reserve(ctx, s.cfg.Releases, []string{moves.HostKey(host)}, {create, a.OpID, host, acct, app, now})`: held → step 4 as today; `ErrReserved` by a `cleanup` → `settleAllocation(a, lifetimeVoid, errHostUnderCleanup)` (409 "the previous instance at this hostname is still being cleaned up. try again later"); by another holder → void, `errLifetimeSettling`; `ErrUnreadable`, `ErrReservationMismatch` or uncertain → `errLifetimePending` (recovery owns the row). After every terminal state the create path or recovery writes, `s.releaseAllocationKey(a)` = `moves.Release(create, OpID)` then `ClearAllocationHostKey(OpID)`; a failure leaves `host_key = 1`. |
| +- **Recovery** (`recoverAllocationLocked`): first branch, `a.State != pending && a.HostKey` → `releaseAllocationKey`, return; every branch that writes the slot, the stale body or the record first calls the same `moves.Reserve` (its own retry returns the held entry): `ErrReserved` with no slot of its own → void and release; `ErrReserved` with its own slot → stays pending, logged hourly; unreadable → pending. |
| +- **The placement** (`placeAppTarget`, claims-off branch only; the host-claim branch returns before this as today): after the seed check, the app's record keys are listed (the listing at line 678 moves up, still under `instanceStoreTimeout`), then `id, ref := s.reservePlacement(ctx, acct, app, hosts)`: one `moves.Reserve` with `HostKey(h)` for every listed host, `{place, id}`, and `s.placing[id] = true` (a mutex-guarded set); `ErrReserved` by a `cleanup` → 409 "an instance of this app is still being cleaned up. try again later"; by another → 409 "an instance of this app is moving. try again later"; unreadable or uncertain → 502 "could not reserve the app's instances. try again" (an uncertain write is settled by the sweep); nothing else is written on any refusal. Then app placement, routes and records exactly as today. `defer s.settlePlacement(ctx, id, uncertain)` on every return after the reservation: each host whose `SetRecordRegion` returned an error other than `nil`, `os.ErrNotExist` or a seed conflict is fenced (`s.lockAppHost`, `s.admit(host, "")`, `s.fenceRecord(ctx, acct, app, host, fenceOwner{})`, as `offlineMoveReclaimable` fences); every fence answered → `moves.Release(place, id)`; a fence failing → the entry stays for the sweep; `delete(s.placing, id)`. A record created after the listing came through the create, which holds its own key. |
| +- **The sweep** (in `residenceSweep`): a `place` entry older than `offlineLockGrace` whose id is not in `s.placing` → fence the record of every host key it holds (account and app from the entry), then `moves.Release`; any failure → kept, logged hourly. `offlineReclaimable` for `place` runs the same settle (`reclaimHeld` while `s.placing` has the id). |
| +- **The audit** `TestEveryResidenceChangeHoldsTheHostnameKey` (`go/ast` over non-test files under `./internal/...`): every call of `provision.SetRecordRegion`, `provision.OfflineFlip`, `provision.OfflineAbort`, `provision.MoveComplete`, `provision.MoveFlip`, `provision.MoveRollBack` and `(*Server).createRecordWithLifetimeLocked` is listed with the reservation that covers it (placement, a move's selection, the create) or its reviewed exemption: `MoveComplete` (`internal/host/movespass.go:1617`, the legacy cut's last act on the source box: tolerated, Self-review notes), `MoveFlip` and `MoveRollBack` (Phase 1, host claims on, where no box judge and so no cleanup exists). |
| -### Task 5: Residence changes hold the hostname key |
| +- [ ] **Step 1: Write the failing tests**: `TestCreateReservesTheHostnameKeyAfterBeginAllocation` (busy by cleanup → 409 cleanup copy, the row void, no slot, no record, no entry; busy by a move → settling; a crash after `BeginAllocation` and before `Reserve` → recovery reserves and continues, or voids when busy; a crash after `Reserve` → recovery continues under its own retry; a crash after the terminal state and before the release → the sweep releases and clears; `LastIncarnation` ignores the void row); `TestPendingAllocationHoldsTheKeyUntilRecoveryEnds`; `TestResidenceChangesAndCleanupExclude` (Review Focus 4, with the real create, the real placement door and an offline move selection against a box cleanup's entries); `TestPlacementReservesEveryInstanceFirst` (busy → app placement, routes and records unchanged); `TestPlacementFencesAnUncertainWriteBeforeRelease` (a store that commits the record PUT after the client's context is cancelled: the fence makes it fail its condition, the entry released after the fence); `TestPlacementCrashIsSettledByTheSweep`; `TestEveryResidenceChangeHoldsTheHostnameKey`; `TestHolderChecksPreserved` (today's create and placement refusals byte for byte). |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/console -run 'HostnameKey|PendingAllocation|ResidenceChanges|Placement|EveryResidence|HolderChecks' -count=1`. |
| +- [ ] **Step 3: Implement. Step 4: Run to verify they pass**, plus `go test ./internal/console -run 'Lifetime|Allocation|Place' -count=1`. |
| +- [ ] **Step 5: Commit** `console: the lifetime create and placement hold every hostname they change in the reservation table, with the allocation row and the entry as their intents`. |
| + |
| +### Task 8: The reaper |
| **Files:** |
| -- Modify: `internal/console/lifetimes_create.go` (plan 1's create protocol step 1: after `admit(host, "")` and before `BeginAllocation`, `moves.Reserve(["host/" + host], {create, OpID, host, acct, app, now}, consoleOpts)`; `ErrReserved` by a `cleanup` holder → `errHostUnderCleanup` (409 "the previous instance at this hostname is still being cleaned up. try again later"); by any other holder → `errLifetimeSettling` (409 settling); `ErrUnreadable` → 502; the entry is released by the allocation's terminal state (`committed`, `superseded`, `void`, `refused`) in the same code path that sets it, and by recovery when it terminalises a row; a pending allocation keeps the key, so cleanup is `busy` until recovery ends it), `internal/console/apiinstancemove.go` (the legacy move door's `MoveComplete`: `{place, opID}` on `host/<host>` and the two regions' boxes around the record write), `internal/console/appsettings.go` and `fleets.go` (`placeAppTarget`'s callers, `carlos apps place`: `{place, opID}` on `host/<host>` per instance around each `SetRecordRegion`), `internal/console/offlinereserve.go` (plan 2's selections include `host/<hostname>`) |
| -- Test: `internal/console/lifetimes_cleanup_test.go`, `internal/console/residence_reservation_test.go` |
| +- Create: `internal/host/movesreap.go` |
| +- Modify: `internal/host/movescleanup.go` (`cleanupRun` ends with `w.reapRetired(ctx, now)`) |
| +- Test: `internal/host/movesreap_test.go` |
| + |
| +**Interfaces:** |
| +- `func (w *movesWorker) reapRetired(ctx context.Context, now time.Time) error`: an op directory `<dataDir>/.offline-moves/<host>/<move>/` is removed (`os.RemoveAll` of that directory, then the host directory when empty, then the obligation's delivered page entries under `pageEntryDir(dataDir, host, id)`) only when its `retired.json` is `ended`, `now - EndedAt >= 7 d`, `PublishedRev == Rev`, the page outbox holds no undelivered entry for `(host, id)`, and the drop is absent or root's acknowledgement exists. A directory without `retired.json`, with one in any other state, or `corrupt`, is never removed: a target's park, an aborted move's park and every directory an earlier release wrote stay as they are today. |
| -- [ ] **Step 1: Write the failing tests**: `TestCreateReservesTheHostnameKeyBeforeAllocating` (paused after `admit` and before `Reserve`, cleanup acquires → the create is refused 409 cleanup, no slot, no row; paused after `Reserve` → cleanup is `busy`; the key released at every terminal state); `TestPendingAllocationHoldsTheKeyUntilRecoveryEnds`; `TestPlacementAndLegacyMoveReserveTheHostname` (each refused `busy` under a cleanup holder; each blocks cleanup while held; released after the write); `TestEveryResidenceChangeHoldsTheHostnameKey` (the AST audit); `TestHolderChecksPreserved`. |
| -- [ ] **Step 2: Run to verify they fail; Step 3: implement; Step 4: run** `go test ./internal/console -run 'Cleanup|Reserve|Residence|HolderChecks' -count=1`. |
| -- [ ] **Step 5: Commit** `console: every residence change of a hostname holds its key in the reservation table; a hostname under cleanup is refused at durable admission`. |
| +- [ ] **Step 1: Write the failing tests**: `TestOpDirReapedSevenDaysAfterEnded`; `TestUnpublishedReportBlocksTheReaper`; `TestUndeliveredPageBlocksTheReaper`; `TestOutstandingRootRequestBlocksTheReaper`; `TestDirectoriesWithoutAnObligationAreNeverReaped`; and Task 5's `TestCleanupReportIsDurableAndBlocksTheReaper` completed with the reaper. |
| +- [ ] **Step 2: Run to verify they fail; Step 3: implement; Step 4: run** `go test ./internal/host -run 'Reap|DurableAndBlocks' -count=1`. |
| +- [ ] **Step 5: Commit** `agent: an ended cleanup's directory is removed seven days on, once its report and pages are delivered and root has answered`. |
| -### Task 6: End to end |
| +### Task 9: End to end and the census |
| **Files:** |
| -- Test: `internal/e2e/selfserve/cleanup_test.go` (plan 2's bridge, a clock the test advances, a root pass driven by the test with `locked = true`) |
| +- Test: `internal/e2e/selfserve/cleanup_test.go` (plan 2's bridge `host.E2E`, a clock the test advances, a root pass the test drives with the adopt lock held) |
| +- Create: `internal/provision/testdata/plan14-root-reads-self-serve-moves-4.tsv`, `internal/provision/self_serve_moves4_census_test.go`, `docs/superpowers/notes/census-proposal-self-serve-moves-4.md` |
| +- Modify: `internal/provision/plan14_root_reads_test.go`, `internal/provision/plan9_b8_census_test.go` (line 372) |
| -- [ ] **Step 1: Write the tests**: `TestCleanupTwentyFourHoursAfterTheObservedFinish`; `TestCleanupRefusalsMidPass` (the record moved again after one key; a placement back; a current version differs; a move holds the box; two boxes in the region); `TestCleanupSupersededCancelsRoot`; `TestCreateUnderCleanupRefusedThenAllowed`; `TestCleanupStatusSupportsReclaim` (plan 2's reclaim rule against the real status report through its lifecycle, including a corrupt file and a report older than the reservation: not reclaimable until `ended` in a report that post-dates the reservation); `TestCleanupReportInTheMoveLog`. |
| +- [ ] **Step 1: Write the tests**: `TestCleanupTwentyFourHoursAfterTheObservedFinish` (a forward move; at 24 h every inventoried key carries a delete marker, `lock.json` and `build-state.json` remain, the lifted tombstone is back, root removed the live link and the programs file, the entries are gone, the report is under the move id); `TestCleanupSupersededByAPlacementBack` (the placement is refused while the cleanup holds; after it ends, a placement back to the source runs); `TestCreateUnderCleanupRefusedThenAllowed`; `TestRowMintedUnderTheMarkSupersedes`; `TestOperatorReturnAfterSupersededCleanupStillFindsItsLeftovers` (the return's start door accepts the forward's leftover set after a superseded cleanup that had marked and unmarked); `TestLegacyMoveIntoTheSourceRegionIsTolerated` (switch off: a legacy move back to the source region while the cleanup holds: the cleanup supersedes at its next key, the wake waits for the unmark, the promotion restores the staged data and the instance serves it); `TestCleanupReportInTheMoveLog`. |
| - [ ] **Step 2: Run them; fix what they find within this plan's scope.** |
| -- [ ] **Step 3: Run both census tests and the cross-builds**; write `docs/superpowers/notes/census-proposal-self-serve-moves-4.md`. |
| +- [ ] **Step 3: Run both census tests and the cross-builds**; write the census proposal with the loader block, the changed envelopes (`internal/host`, `internal/release`, `internal/registry`, `internal/lease`, `internal/moves`, `internal/console`, `internal/offlinecleanup`) and the units. |
| - [ ] **Step 4: Commit** `self-serve moves: cleanup end to end`; push. |
| --- |
| ## Self-review notes |
| -- Spec coverage: §7.6 the source replica (T1, T2), parked local state (T3), root-owned leftovers with the process and reference checks (T4), the staged objects of an aborted move (unchanged); §7.1's `cleanup` reservation and the hostname keys (T2, T5); §8's cleanup pages (T2, T4); §11's cleanup witnesses (T6). Stated strengthenings: cleanup refuses while the source region has more than one box; every residence change holds the hostname key (Paul's ruling 2026-10-22). |
| -- Plan 2 amendments this plan requires (small, listed for the plan 2 build): the reservation keys become `box/<id>` and `host/<hostname>`; selections include `host/<hostname>`; the reclaim rule's cleanup clause reads as Task 1 states it. |
| -- Names used across tasks: `offlinecleanup.Retired` (`State`, `Keys`, `Refusals`, `RootRequest`, `FinishedAt`, `LastPagedAt`, `PendingPages`, `Prefix`), `InventoryObject`, `KeyOutcome`, `Refusal`, `PendingPage`, `FileIdentity`, `LocalPath`, `RetiredSummary` (`ReleasePending`, `UpdatedAt`), `ValidateRetired`, `OfflineOpDir`, `RetiredFile`, `fileIdentity`, `writeRetired`, `moves.Reserve` (returns the entry), `moves.Release`, `ReserveOptions`, `ErrReserved`, `ErrUnreadable`, `cleanupDispatch`, `observeParkEnds`, `reapParked`, `park-ended.json`, `RootCleanupRequest` (`Account`, `App`), `RootCleanupPath`, `RootCleanupAck` (`Unknown`), `ServeRootCleanupRequests`, `WithRootCleanupLock`, `writeRootCleanupRequest`, `errHostUnderCleanup`, `cfg.PlainMirror`, the key forms `box/`, `host/`, `"*"`. |
| -- Review Focus 1 to 5 are pinned by T2, T2+T5, T2+T5, T2+T4, T4. |
| +- Spec coverage: §7.6 the source replica (T1, T3, T4), root-owned leftovers with the link and process checks (T6), parked local state (T8), the staged objects of an aborted move (unchanged); §7.1's `cleanup` reservation and the hostname keys (T2, T7); §8's cleanup pages and every cleanup key in the move log (T5); §9 rollout (Global Constraints); §11's cleanup witnesses (T4, T9). |
| +- **Stated deviations.** (1) Legacy moves hold no reservation: a legacy move into the source region carries a move on the record from its door onward, so condition (1) supersedes the cleanup at its next key, and its completion's wake cannot take the lease while the mark holds; after the unmark the wake's promotion mirrors the staged replica over the prefix (`promoteStagedReplica`, an exact mirror), so no deleted key matters. Holding a key from the box at `MoveComplete` (`movespass.go:1617`) would add a box-side release and crash path and extend a legacy cut's downtime behind root's answer, for no exclusion the mark does not already give. (2) Placement keeps `SetRecordRegion`'s ETag-conditional write and settles an uncertain one by `fenceRecord`, after which no earlier PUT can land; the reservation entry is the persisted intent, so no journal table is added. A bound write would add a refusal on unrelated concurrent writes and change the door's answers. (3) A cleanup entry is never reclaimed by another reserver (§7.1 allowed a status-based reclaim): its own box resumes it after any restart, and the only case a status report could settle, a box that died before its first state write, is one whose report is stale and so could never settle it; the console pages instead. (4) Only the sealing box's local paths are cleaned; other boxes in the source region keep theirs, as today; the region-has-one-box check is removed because the mark excludes every box's wake. (5) The create reserves after `BeginAllocation`, not before, so every reservation has an intent row behind it. (6) `build-state.json` is kept with `lock.json`. |
| +- Plan 2 amendments (built code, changed here): move selections hold `host/<hostname>`; the switch-off rule applies between moves only, so a cleanup neither blocks nor is blocked by an operator move on another box; recovery mode counts `move` entries only, so a create, a placement or a cleanup never turns the legacy move doors into the self-serve family's 404; the cleanup reclaim interface is deleted. |
| +- Built-code changes this plan makes, by task: T2 `moves/offline.go` (kinds), `console/offlinereserve.go`, `console/selfserve_pass.go`, `console/selfserve_pages.go`; T3 `lease` (new file only); T4 `host/movespass.go`; T5 `host/movespaging.go`, `console/apiinstancemove_offline.go`; T6 `registry`, `host/agent.go`, `release/programhistory.go` (export); T7 `console/store.go`, `lifetimes_store.go`, `lifetimes_create.go`, `lifetimes_recovery.go`, `fleets.go`. |
| +- Names used across tasks: `offlinecleanup.Retired` (`State`, `Outcome`, `Keys`, `Events`, `Root`, `Rev`, `PublishedRev`, `MarkedAt`, `RootRequestedAt`, `CancelRequestedAt`), `CleanupID`, `InventoryObject`, `KeyOutcome`, `Event`, `Report`, `FileIdentity`, `LocalPath`, `RootCleanupRequest`, `RootCleanupPath`, `RootCleanupAck`, `KeptPath`, `ValidateRetired`; `moves.HostKey`, `IsHostKey`, `Op`, `Conflicts`, `Reserve`, `Holds`, `Release`, `ErrReserved`, `ErrUnreadable`, `ErrUncertain`, `ErrReservationMismatch`, `RecordAway`, `Away*`, `OfflineOpDir`, `RetiredFile`, `CleanupReportKey`, `OfflineKindCreate`, `OfflineKindPlace`; `lease.CleaningOwner`, `MarkCleaning`, `ReassertCleaning`, `UnmarkCleaning`, `ErrNotOurLock`; `w.kickCleanup`, `cleanupRun`, `cleanupMu`, `cfg.PlainMirror`, `cfg.CleanupArmed`, `recordRetired`, `createRetired`, `writeRetired`, `markPublished`, `flushPageOutboxFor`, `reapRetired`; `release.ServeRootCleanupRequests`, `RootCleanupOptions`, `RootCleanupDir`, `ReadRootCleanupAck`, `ErrRootCleanupRefused`, `ProgramsPath`; `rootCleanupAuthority`, `rootCleanupPass`; `registry.WithHostFrozen`; `s.reservePlacement`, `s.settlePlacement`, `s.placing`, `s.residenceSweep`, `s.releaseAllocationKey`, `errHostUnderCleanup`. |
| +- Review Focus 1 to 5 are pinned by T4, T4, T6, T2+T7, T5+T8. |