amadan
plan: self-serve moves 4, cleanup (rev 1)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1 file changed,
+130
−0
docs/superpowers/plans/2026-10-22-self-serve-moves-4-cleanup.md+130 −0
CI passed — run details
CI log
=== gate ===
.amadan/test-gate.sh
=== launchctx (0 s)
ok amadan.net/carlos/platform.git/launchctx 0.016s
? amadan.net/carlos/platform.git/launchctx/controlname [no test files]
? amadan.net/carlos/platform.git/launchctx/wireid [no test files]
=== vet (1 s)
=== gofmt (0 s)
gofmt -l .: clean
=== rest (143 s)
? github.com/carlosframework/platform/cmd/changelog [no test files]
ok github.com/carlosframework/platform/cmd/console 5.773s
ok github.com/carlosframework/platform/cmd/moveprobe 12.890s
ok github.com/carlosframework/platform/cmd/zddprobe 0.717s
ok github.com/carlosframework/platform/internal/activator 61.886s
ok github.com/carlosframework/platform/internal/agentsock 0.033s
ok github.com/carlosframework/platform/internal/alerting 0.006s
ok github.com/carlosframework/platform/internal/analytics 0.029s
ok github.com/carlosframework/platform/internal/appconfig 3.902s
ok github.com/carlosframework/platform/internal/apperr 0.177s
ok github.com/carlosframework/platform/internal/applog 0.428s
ok github.com/carlosframework/platform/internal/awsinv 0.016s
ok github.com/carlosframework/platform/internal/boxchoice 0.011s
ok github.com/carlosframework/platform/internal/boxid 1.090s
? github.com/carlosframework/platform/internal/boxid/boxidtest [no test files]
? github.com/carlosframework/platform/internal/boxid/internal/signtest [no test files]
ok github.com/carlosframework/platform/internal/boxvolumes 0.016s
ok github.com/carlosframework/platform/internal/caps 0.020s
ok github.com/carlosframework/platform/internal/certwarm 0.003s
ok github.com/carlosframework/platform/internal/cgroup 0.893s
ok github.com/carlosframework/platform/internal/changelog 0.710s
ok github.com/carlosframework/platform/internal/claimactivation 1.734s
ok github.com/carlosframework/platform/internal/clamd 0.081s
? github.com/carlosframework/platform/internal/cliflag [no test files]
? github.com/carlosframework/platform/internal/controlname [no test files]
? github.com/carlosframework/platform/internal/controlsock [no test files]
ok github.com/carlosframework/platform/internal/dbdoor 6.172s
ok github.com/carlosframework/platform/internal/dbdoor/dbdoortest 0.013s
ok github.com/carlosframework/platform/internal/dbverify 0.020s
ok github.com/carlosframework/platform/internal/dnsrec 0.003s
ok github.com/carlosframework/platform/internal/docsite 0.028s
ok github.com/carlosframework/platform/internal/domains 4.570s
ok github.com/carlosframework/platform/internal/doorkey 0.016s
ok github.com/carlosframework/platform/internal/economics 0.007s
ok github.com/carlosframework/platform/internal/edge 46.507s
? github.com/carlosframework/platform/internal/edgeproof [no test files]
ok github.com/carlosframework/platform/internal/edgesfile 0.008s
ok github.com/carlosframework/platform/internal/fleetapi 0.201s
ok github.com/carlosframework/platform/internal/gate 2.149s
ok github.com/carlosframework/platform/internal/genselect 0.030s
ok github.com/carlosframework/platform/internal/geo 0.018s
ok github.com/carlosframework/platform/internal/geoip 0.046s
ok github.com/carlosframework/platform/internal/historyreadback 0.258s
ok github.com/carlosframework/platform/internal/host 139.450s
ok github.com/carlosframework/platform/internal/hostactors 7.632s
ok github.com/carlosframework/platform/internal/hostactors/awsmembership 0.031s
ok github.com/carlosframework/platform/internal/hostbox 4.523s
ok github.com/carlosframework/platform/internal/hostclaim 1.394s
ok github.com/carlosframework/platform/internal/hostenv 0.034s
ok github.com/carlosframework/platform/internal/hostevidence 0.032s
ok github.com/carlosframework/platform/internal/hostexec 7.451s
ok github.com/carlosframework/platform/internal/hostfence 0.062s
ok github.com/carlosframework/platform/internal/hostfiles 11.467s
ok github.com/carlosframework/platform/internal/hostfloor 0.033s
ok github.com/carlosframework/platform/internal/hostguard 1.431s
ok github.com/carlosframework/platform/internal/hostops 3.602s
? github.com/carlosframework/platform/internal/hostops/fencerelease [no test files]
ok github.com/carlosframework/platform/internal/hostops/internal/fencerelease 0.032s
ok github.com/carlosframework/platform/internal/hostquarantine 0.032s
ok github.com/carlosframework/platform/internal/hostrelocate 1.292s
ok github.com/carlosframework/platform/internal/hostresolve 0.284s
ok github.com/carlosframework/platform/internal/hostwrite 0.032s
ok github.com/carlosframework/platform/internal/infraview 0.029s
ok github.com/carlosframework/platform/internal/instancectl 5.309s
ok github.com/carlosframework/platform/internal/instanceexec 2.900s
ok github.com/carlosframework/platform/internal/instancefs 0.033s
ok github.com/carlosframework/platform/internal/instancekey 0.035s
ok github.com/carlosframework/platform/internal/instanceupdates 0.008s
ok github.com/carlosframework/platform/internal/layout 3.194s
ok github.com/carlosframework/platform/internal/lbhealth 0.003s
ok github.com/carlosframework/platform/internal/lease 10.271s
ok github.com/carlosframework/platform/internal/lease/leasetest 0.752s
ok github.com/carlosframework/platform/internal/ledger 0.019s
ok github.com/carlosframework/platform/internal/mail 0.380s
ok github.com/carlosframework/platform/internal/maintenance 0.113s
ok github.com/carlosframework/platform/internal/maintenanceexec 5.747s
ok github.com/carlosframework/platform/internal/maintenanceguard 0.115s
ok github.com/carlosframework/platform/internal/metrics 0.009s
ok github.com/carlosframework/platform/internal/moves 23.924s
ok github.com/carlosframework/platform/internal/moves/archive 1.704s
ok github.com/carlosframework/platform/internal/objstore 0.576s
ok github.com/carlosframework/platform/internal/ops 0.626s
ok github.com/carlosframework/platform/internal/park 0.388s
ok github.com/carlosframework/platform/internal/plans 0.007s
ok github.com/carlosframework/platform/internal/privatefile 0.004s
ok github.com/carlosframework/platform/internal/provision 18.532s
ok github.com/carlosframework/platform/internal/publicrelease 0.008s
? github.com/carlosframework/platform/internal/regionname [no test files]
ok github.com/carlosframework/platform/internal/registry 6.721s
ok github.com/carlosframework/platform/internal/relay 7.281s
ok github.com/carlosframework/platform/internal/release 28.830s
ok github.com/carlosframework/platform/internal/rollout 4.782s
ok github.com/carlosframework/platform/internal/rolloutwords 0.003s
ok github.com/carlosframework/platform/internal/rootaws 0.009s
ok github.com/carlosframework/platform/internal/rootparticipation 0.064s
ok github.com/carlosframework/platform/internal/rootstate 0.031s
ok github.com/carlosframework/platform/internal/runs 0.520s
ok github.com/carlosframework/platform/internal/s3compat 0.016s
ok github.com/carlosframework/platform/internal/safefile 0.005s
ok github.com/carlosframework/platform/internal/schedule 3.684s
ok github.com/carlosframework/platform/internal/seeds 0.049s
ok github.com/carlosframework/platform/internal/sim/hostlife 17.675s
ok github.com/carlosframework/platform/internal/sim/litestreamfacts 3.986s
ok github.com/carlosframework/platform/internal/status 0.143s
ok github.com/carlosframework/platform/internal/steering 0.009s
ok github.com/carlosframework/platform/internal/stopevidence 0.229s
ok github.com/carlosframework/platform/internal/svcrole 0.002s
ok github.com/carlosframework/platform/internal/telemetry 0.005s
ok github.com/carlosframework/platform/internal/termfence 0.006s
ok github.com/carlosframework/platform/internal/tunnelclient 0.005s
ok github.com/carlosframework/platform/internal/unixdial 0.004s
ok github.com/carlosframework/platform/internal/updatewords 0.004s
ok github.com/carlosframework/platform/internal/usage 0.006s
ok github.com/carlosframework/platform/internal/version 0.004s
ok github.com/carlosframework/platform/internal/vet 2.128s
=== internal_console@0 (88 s)
ok github.com/carlosframework/platform/internal/console 84.146s
=== internal_console@1 (41 s)
ok github.com/carlosframework/platform/internal/console 38.007s
=== internal_console@2 (45 s)
ok github.com/carlosframework/platform/internal/console 41.915s
=== internal_console@3 (40 s)
ok github.com/carlosframework/platform/internal/console 37.087s
=== internal_console@4 (41 s)
ok github.com/carlosframework/platform/internal/console 37.740s
=== internal_console@5 (38 s)
ok github.com/carlosframework/platform/internal/console 34.798s
=== internal_console@6 (40 s)
ok github.com/carlosframework/platform/internal/console 36.652s
=== internal_console@7 (48 s)
ok github.com/carlosframework/platform/internal/console 44.174s
=== root@0 (37 s)
ok github.com/carlosframework/platform 34.525s
=== root@1 (47 s)
ok github.com/carlosframework/platform 44.134s
=== root@2 (44 s)
ok github.com/carlosframework/platform 41.340s
=== cross ===
--- shard 0 of 5 ---
cross-compiling darwin/arm64
--- shard 1 of 5 ---
cross-compiling darwin/amd64
--- shard 2 of 5 ---
cross-compiling linux/arm64
linux/arm64 carlos binary: 50069666 bytes (ceiling 52000000)
--- shard 3 of 5 ---
cross-compiling windows/amd64
--- shard 4 of 5 ---
cross-compiling windows/arm64
=== minio-contract ===
minio contract: arm64, minio RELEASE.2025-09-07T16-13-09Z, mc RELEASE.2025-08-13T08-35-41Z
minio: cached, digest verified
mc: cached, digest verified
minio healthy on 127.0.0.1:20999
Added `local` successfully.
Bucket created successfully `local/contract-bucket`.
Bucket created successfully `local/denied-bucket`.
Created policy `contract-only` successfully.
Added user `scopeduser` successfully.
Attached Policies: [contract-only]
To User: scopeduser
buckets and scoped user ready
=== RUN TestDeliveredContractMinio
=== RUN TestDeliveredContractMinio/put
=== RUN TestDeliveredContractMinio/get_round-trips_the_body
=== RUN TestDeliveredContractMinio/list_shows_the_key
=== RUN TestDeliveredContractMinio/head_and_list_agree_on_the_version
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed
=== RUN TestDeliveredContractMinio/delete_removes_it
=== RUN TestDeliveredContractMinio/cross-bucket_denial
--- PASS: TestDeliveredContractMinio (0.02s)
--- PASS: TestDeliveredContractMinio/put (0.00s)
--- PASS: TestDeliveredContractMinio/get_round-trips_the_body (0.00s)
--- PASS: TestDeliveredContractMinio/list_shows_the_key (0.00s)
--- PASS: TestDeliveredContractMinio/head_and_list_agree_on_the_version (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route (0.01s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/delete_removes_it (0.00s)
--- PASS: TestDeliveredContractMinio/cross-bucket_denial (0.00s)
PASS
ok github.com/carlosframework/platform/internal/objstore 0.023s
=== RUN TestT4StorageProbeMinio
2026/10/22 07:16:34 moves: T4: STORE: list: operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: 18E0C8C3BEA8C2D1, HostID: 7c420053e0fbcd7f41653fdd92ffa2802c903d05303b1aa15adc2e8bb65d183f, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
--- PASS: TestT4StorageProbeMinio (0.00s)
PASS
ok github.com/carlosframework/platform/internal/host 0.020s
minio contract: PASS
diff --git a/docs/superpowers/plans/2026-10-22-self-serve-moves-4-cleanup.md b/docs/superpowers/plans/2026-10-22-self-serve-moves-4-cleanup.md| new file mode 100644 |
| index 000000000..0b236430b |
| --- /dev/null |
| +++ b/docs/superpowers/plans/2026-10-22-self-serve-moves-4-cleanup.md |
| @@ -0,0 +1,130 @@ |
| +# Self-serve moves, plan 4: cleanup |
| + |
| +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| + |
| +Rev 1, 2026-10-22. Builds on plans 1 to 3: per-box reservations in `active.json` (plan 2 Task 3, which already reads the `cleanup` kind), the self-serve job's `finish` (plan 2 Task 8), the box's offline steps (`offlineSourceGone`, `internal/host/movesoffline.go`). |
| + |
| +**Goal:** Twenty-four hours after a finished move, the source box retires the old region's replica objects from evidence it recorded at finish, under a `cleanup` reservation, key by key, never deleting a version it did not inventory; root removes its own leftovers only through a request file it answers with `.done` or `.cancelled`; parked local state goes seven days after the obligation ends; nothing times out into a release. |
| + |
| +**Architecture:** At `finish` the source box writes `retired.json` under its park directory with the exact bucket inventory (keys and current version ids under `instances/<host>/`, `lock.json` excluded) and the root-owned local paths with their identities. The box's moves pass runs the cleanup as one operation with the `retired.json`'s id: a `cleanup` reservation for its own box, `cleaning_since`, then per key the four conditions at the moment of deletion, a plain `Delete` (a delete marker on these versioned buckets), `cleaned` only when every key is done or superseded. The agent then writes `root-cleanup.json` and touches the stage-requested marker; root's adopt pass removes exactly the recorded identities and writes `root-cleanup.done`; the reservation is released only on `.done` or an acknowledged `.cancelled`. Supersession (the host moved again or recreated) cancels and waits for the same acknowledgement. |
| + |
| +**Tech Stack:** Go. `internal/moves` (the `retired.json` shape, the shared reservation helper), `internal/host` (the moves pass, the status report, the agent's retention loop, root's adopt pass), `internal/lease` (`VersionedStore.ListVersions`, `MirrorStore.Delete`, `LockState`, `IsLiftedTombstone`), `internal/console` (the create door's reservation check, the reclaim rule's `cleanup` evidence), `internal/alerting`. |
| + |
| +**Spec:** `docs/superpowers/specs/2026-10-21-self-serve-offline-moves-design.md` §7.6, §7.1 (the `cleanup` reservation), §8 (the cleanup pages), §11. |
| + |
| +## Global Constraints |
| + |
| +- `VersionId` deletes are never used for cleanup: every deletion is a plain `DeleteObject`, which writes a delete marker and keeps every version recoverable. `lock.json` is never deleted. |
| +- A key is deleted only when, at that moment: the record names `to` or a later region and carries no move for the host; the source bucket's `lock.json` is the lifted tombstone of this move id (`lease.LockState` owner `moves.TombstoneOwner(moveID)` and `lease.IsLiftedTombstone`); the box's registry has no row for the host; and the key is in the inventory with the inventory's current version id (`ListVersions` shows that version `IsLatest`). A key whose current version differs is `superseded`, never deleted. Any read that fails stops the pass for this obligation (it retries next pass). |
| +- The `cleanup` reservation is held from before the first deletion through root's `.done` or `.cancelled`. A request root never answers keeps it held and pages every 10 minutes. Nothing releases on a timer. |
| +- Root removes a recorded path only when its identity still matches (device, inode, and for a link its target), the registry has no row for the host, and no live link or running process names the build directory. Root checks for `.cancel` immediately before each deletion. |
| +- The reservation helper moves to `internal/moves` so the console (plan 2) and the box share one implementation (`moves.Reserve`, `moves.Release`); the console's `reserveBoxes`/`releaseBoxes` become thin wrappers. |
| +- Every deletion and every refusal is a move log line (the box's `notice`/`logTrouble` and the report under `control/offline-moves/<host>/<id>/cleanup/<box>.json`). |
| +- Census: the box's cleanup units and root's request handler take rows in `plan14-root-reads-self-serve-moves-4.tsv`, loaded last, `rev3=PROPOSED-self-serve-moves-4`; root's handler is keyed on the request file at the version sweep's class (`authority`, basis E2, like `sweepStagedVersions`); both census tests green. |
| +- Copy rows pending Paul's review: "the previous instance at this hostname is still being cleaned up. try again later". No tenant names in tests. |
| +- Local tests: touched packages with `-run`; CI runs the full suite. |
| + |
| +## Review Focus |
| + |
| +1. The record moves again (a second move, or a recreation) between the inventory and a deletion: the pass refuses that key and marks the obligation `superseded` as a whole; nothing more is deleted (Task 2, `TestCleanupRefusesWhenTheRecordMovedAgain`). |
| +2. An object under `instances/<host>/` whose current version differs from the inventory (a wake wrote it): skipped as `superseded`, never deleted; the obligation stays open with the key named (Task 2, `TestCleanupSkipsAChangedVersion`). |
| +3. The reservation is held by a move (an operator's return in progress): cleanup does not start; a cleanup reservation in place refuses a create of the hostname and a move of it (Task 2 and Task 5, `TestCleanupWaitsForAMoveAndBlocksOne`). |
| +4. Root never answers the request: the reservation stays, a page every 10 minutes, no release; root answers after an hour: `.done`, release (Task 4, `TestUnansweredRootRequestHoldsAndPages`). |
| +5. A cancel racing root's deletions: root deletes nothing after seeing `.cancel`, writes `.cancelled` naming what it had removed, and the agent releases only then (Task 4, `TestCancelIsAcknowledgedBeforeRelease`). |
| + |
| +--- |
| + |
| +### Task 1: `retired.json` at finish, and the status report's summary |
| + |
| +**Files:** |
| +- Modify: `internal/moves/offline.go` (the `Retired` shape and `RetiredKey`), `internal/host/movesoffline.go` (`offlineSourceGone`: when the row is gone, before the `gone` report, write `retired.json` once; a present file is not rewritten), `internal/host/agent.go` (the status tick fills `p.Retirements`), `internal/status/snapshot.go` (`BoxMetrics.Retirements []RetiredSummary \`json:"retirements,omitempty"\``, private) |
| +- Test: `internal/host/movesoffline_retired_test.go`, `internal/status/snapshot_test.go` |
| + |
| +**Interfaces:** |
| +- `type moves.Retired struct { ID string \`json:"id"\`; MoveID string \`json:"move_id"\`; Host, Account, App string; Incarnation uint64 \`json:"incarnation"\`; From, To string; OpenAt time.Time \`json:"open_at"\`; FinalTXID uint64 \`json:"final_txid"\`; TombstoneOwner string \`json:"tombstone_owner"\`; Bucket, BucketRegion string; BucketInventory []InventoryObject \`json:"bucket_inventory"\`; LocalInventory []LocalPath \`json:"local_inventory"\`; WrittenAt time.Time \`json:"written_at"\`; CleaningSince *time.Time \`json:"cleaning_since,omitempty"\`; State string \`json:"state"\` (`open`, `cleaning`, `cleaned`, `superseded`); Skipped []string \`json:"skipped,omitempty"\`; EndedAt *time.Time \`json:"ended_at,omitempty"\` }`; `type InventoryObject struct { Key, VersionID, ETag string }`; `type LocalPath struct { Path string; Dev, Inode uint64; LinkTarget string }`; `func RetiredPath(dataDir, host, id string) string` (`<park dir>/retired.json`, the id is the move id). |
| +- The inventory: `VersionedStore.ListVersions(ctx, bucket, "instances/<host>/")`, keeping `IsLatest && !DeleteMarker` versions, excluding `lock.json`; the local inventory: `/opt/carlos/live/<host>` (a symlink: its target) and `/data/programs/<host>.json`, with `Lstat`'s device and inode; a missing path is omitted. |
| +- `type status.RetiredSummary struct { ID string \`json:"id"\`; Host string \`json:"host"\`; State string \`json:"state"\` }` from every `retired.json` under `/data/.offline-moves/*/*/`. |
| + |
| +- [ ] **Step 1: Write the failing tests**: `TestRetiredJSONWrittenOnceAtGone` (the file's inventory lists the two replica objects with their version ids and not `lock.json`; the live link's target and the programs file identities; a second gone pass does not rewrite it); `TestStatusReportsRetirements` (private block; absent from `fleet.json`). |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/host -run 'RetiredJSON|StatusReportsRetirements' -count=1`. |
| +- [ ] **Step 3: Implement.** |
| +- [ ] **Step 4: Run to verify they pass.** |
| +- [ ] **Step 5: Commit** `moves: the source box records retired.json at finish; the status report summarises retirements`. |
| + |
| +### Task 2: The cleanup pass |
| + |
| +**Files:** |
| +- Create: `internal/moves/reserve.go` (`Reserve`, `Release`, the reclaim predicate as a callback), `internal/host/movescleanup.go` |
| +- Modify: `internal/console/offlinereserve.go` (wrappers over `moves.Reserve`/`Release`), `internal/host/movespass.go` (the moves pass calls `w.cleanupStep(ctx, s)` after the offline round, under the pass lock) |
| +- Test: `internal/moves/reserve_test.go`, `internal/host/movescleanup_test.go` |
| + |
| +**Interfaces:** |
| +- `func moves.Reserve(ctx context.Context, s release.Store, boxes []string, r OfflineReservation, reclaimable func(ctx context.Context, held OfflineReservation) (bool, error)) error` (plan 2 Task 3's protocol: one conditional write; `ErrReserved` naming the holder's host when busy; the legacy body reserves every box); `func moves.Release(ctx context.Context, s release.Store, id string) error`. |
| +- `func (w *movesWorker) cleanupStep(ctx context.Context, s release.Store) error`: for each `retired.json` on this box with `State == open` and `WrittenAt + 24 h <= now`, or `State == cleaning`: (1) `Reserve([]string{w.cfg.BoxID}, {cleanup, r.ID, host, acct, app, now}, reclaim)` (busy → log once per hour, next pass); (2) set `CleaningSince`, `State = cleaning` (written through a temp file and rename); (3) the gate: `ReadRecordJudged` (plan 1) for `(acct, app, host)` → the record names `To` or a later region and `Move == nil`, else `superseded`; `lease.LockState(bucket, "instances/<host>/", region)` → owner `== TombstoneOwner(MoveID)` and `IsLiftedTombstone`, else refuse this pass (log); `w.cfg.Route(host) == nil`, else `superseded`; (4) per inventory key in order: `ListVersions(bucket, key)` → the inventory's version `IsLatest` → `MirrorStore.Delete(bucket, key)`, log "cleanup <id>: deleted <key> (version <v>)"; a delete marker already current → done; the current version differs → `Skipped` += key, log `superseded <key>`; the gate re-read every 50 keys; (5) every key done or skipped-and-recorded → `State = cleaned` when `Skipped` is empty; else stay `cleaning` and retry the skipped keys next pass (they stay `superseded` while their version differs; the obligation ends `cleaned` only when the skipped set empties, which it never does for a live successor; an operator reads it); a host whose record moved again or whose box has a row again → `State = superseded`, `EndedAt`. Then Task 4's root step; the reservation is released after `.done`/`.cancelled` (Task 4), or at once when the state is `superseded` and no root request was written. |
| +- A `cleanup` reservation reclaim rule (the console's and the box's `reclaimable`): the owning box's status report names no `RetiredSummary` with that id, or names it `cleaned`/`superseded` with no root request outstanding; a report older than two status ticks → not reclaimable. |
| + |
| +- [ ] **Step 1: Write the failing tests** against a `DirStore` for the deployment bucket, a versioned fake for the data bucket, a real registry and a clock: `TestReserveSharedByConsoleAndBox` (the console's wrapper and the box reserve the same file; a cleanup reservation refuses a move's reservation of that box and the reverse); `TestCleanupRunsAfter24Hours` (23 h → nothing; 24 h → reserved, cleaning, every key gets a delete marker, `lock.json` untouched, `cleaned`); `TestCleanupRefusesWhenTheRecordMovedAgain` (a second move in flight → nothing deleted; a record naming the source region again → `superseded`); `TestCleanupSkipsAChangedVersion`; `TestCleanupRefusesWithoutTheLiftedTombstone` (a live lease on `lock.json` → refused this pass); `TestCleanupRefusesWhileARowExists`; `TestCleanupWaitsForAMoveAndBlocksOne` (a move's reservation on the box → cleanup waits; a cleanup reservation → the operator stage door and plan 2's job wait with "waiting for another move on this box"); `TestCleanupLogsEveryKey`; `TestReclaimOnlyOnStatusEvidence`. |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/moves -run 'Reserve' -count=1`, `go test ./internal/host -run 'Cleanup' -count=1`. |
| +- [ ] **Step 3: Implement.** |
| +- [ ] **Step 4: Run to verify they pass**, plus `go test ./internal/console -run 'Reserve|Reclaim' -count=1`. |
| +- [ ] **Step 5: Commit** `moves: the source box retires the old replica from its inventory under a cleanup reservation, key by key`. |
| + |
| +### Task 3: Parked local state goes seven days after the obligation ends |
| + |
| +**Files:** |
| +- Modify: `internal/host/agent.go` (`retentionLoop` gains `reapParkedMoves(dataDir, now)`), `internal/host/movescleanup.go` |
| +- Test: `internal/host/movescleanup_test.go` |
| + |
| +- [ ] **Step 1: Write the failing tests**: `TestParkedStateReapedSevenDaysAfterTheObligationEnds` (`cleaned` at T → the park directory is present at T + 6 d 23 h and gone at T + 7 d; `superseded` likewise; `open` and `cleaning` never reaped; a directory with an outstanding `root-cleanup.json` and no `.done`/`.cancelled` never reaped). |
| +- [ ] **Step 2: Run to verify it fails.** |
| +- [ ] **Step 3: Implement** (`os.RemoveAll` of `/data/.offline-moves/<host>/<id>/` only; the host directory when empty). |
| +- [ ] **Step 4: Run** `go test ./internal/host -run 'ParkedStateReaped' -count=1`. |
| +- [ ] **Step 5: Commit** `agent: parked offline-move state is reaped seven days after its cleanup obligation ends`. |
| + |
| +### Task 4: Root's leftovers through a request file |
| + |
| +**Files:** |
| +- Create: `internal/release/rootcleanup.go` (the request, done, cancel and cancelled files and root's handler), `internal/host/movescleanup_root.go` (the agent's half) |
| +- Modify: `internal/host/agent.go` (`runAdoptUnderRootRelease`: `rootCleanupPass(ctx, dataDir, reg)` beside `heldStagePass`, served on the stage-requested pass and the hourly one), `internal/host/movescleanup.go` (after `cleaned`: write the request, touch `stageRequestMarker`; poll for `.done`/`.cancelled` each pass; page every 10 minutes while unanswered) |
| +- Test: `internal/release/rootcleanup_test.go`, `internal/host/movescleanup_root_test.go` |
| + |
| +**Interfaces:** |
| +- `type release.RootCleanupRequest struct { Host string; MoveID string; Paths []RootCleanupPath; WrittenAt time.Time }`, `type RootCleanupPath struct { Path string; Dev, Inode uint64; LinkTarget string; BuildDir string }`; files under the park directory: `root-cleanup.json`, `root-cleanup.done` (`{removed: [...], at}`), `root-cleanup.cancel` (the agent's), `root-cleanup.cancelled` (`{removed: [...], at}`). |
| +- `func release.ServeRootCleanupRequests(ctx context.Context, dataDir string, rowAbsent func(host string) (bool, error), liveLinkNames func(buildDir string) (bool, error), running func(buildDir string) (bool, error)) error`: lists `/data/.offline-moves/*/*/root-cleanup.json` (`safefile`, as the held-stage reader does); for each request without `.done`/`.cancelled`: for each path in order: if `.cancel` exists → write `.cancelled` with what was removed so far and stop; `Lstat` the path → identity equal (`Dev`, `Inode`, and `Readlink` for a link) else skip (recorded as `kept`); `rowAbsent(host)`; `!liveLinkNames(buildDir) && !running(buildDir)` → remove (`safefile.Remove`); then `.done`. A request older than 30 days with no answer is left as it is (root never guesses). |
| +- The agent: `writeRootCleanupRequest` after `cleaned` (only when `LocalInventory` is non-empty; otherwise the reservation is released at once); `touch(filepath.Join(dataDir, "moves", stageRequestMarker))`; on each moves pass: `.done` or `.cancelled` → release the reservation, record `EndedAt`; neither and `WrittenAt + 10 min` passed → `alerting.Page(LevelWarning, "cleanup waits for root", ...)`, repeated every 10 minutes; supersession while a request is outstanding → write `.cancel` and wait for `.cancelled`. |
| + |
| +- [ ] **Step 1: Write the failing tests**: `TestRootRemovesOnlyRecordedIdentities` (a matching link and programs file → removed and in `.done`; a link whose target changed → kept; an inode that changed → kept; a row present → kept; a live link naming the build dir → kept); `TestCancelIsAcknowledgedBeforeRelease` (`.cancel` written after the first path was removed → `.cancelled` names that one path, the second untouched; the agent releases only on `.cancelled`); `TestUnansweredRootRequestHoldsAndPages` (no root pass for an hour: pages at 10, 20, ... minutes, the reservation held; then root answers → `.done`, release); `TestNoLocalLeftoversReleasesAtOnce`; `TestRootCleanupIsServedOnTheStageRequestedPass`. |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/release -run 'RootCleanup' -count=1`, `go test ./internal/host -run 'RootCleanup|Cancel|Unanswered' -count=1`. |
| +- [ ] **Step 3: Implement.** |
| +- [ ] **Step 4: Run to verify they pass**, then the root-read census (root's handler row at the version sweep's class). |
| +- [ ] **Step 5: Commit** `root: removes an old instance's live link and programs file only through a cleanup request it answers with done or cancelled`. |
| + |
| +### Task 5: The create door and the move doors respect a cleanup reservation |
| + |
| +**Files:** |
| +- Modify: `internal/console/apiinstances.go` (the create door, after `hostHolder`: `active.json` holds a `cleanup` reservation whose `Host` is this host → 409 "the previous instance at this hostname is still being cleaned up. try again later"; a read failure → the existing "could not check the host" 500), `internal/console/offlinereserve.go` (plan 2's `reserveBoxes` already refuses a held box; the sentence for a `cleanup` holder is "waiting for another move on this box" in the job and the existing busy sentence at the operator doors) |
| +- Test: `internal/console/offlinereserve_test.go` |
| + |
| +- [ ] **Step 1: Write the failing tests**: `TestCreateRefusesUnderACleanupReservation` (409 with the sentence; no record, no slot); `TestCreateProceedsAfterRelease`; `TestMoveWaitsUnderACleanupReservation`. |
| +- [ ] **Step 2: Run to verify they fail.** |
| +- [ ] **Step 3: Implement.** |
| +- [ ] **Step 4: Run** `go test ./internal/console -run 'CleanupReservation|Reserve' -count=1`. |
| +- [ ] **Step 5: Commit** `console: a hostname under cleanup is not recreated or moved until the cleanup ends`. |
| + |
| +### Task 6: End to end |
| + |
| +**Files:** |
| +- Test: `internal/host/selfserve_cleanup_e2e_test.go` (plan 2's two-box harness, a clock the test advances) |
| + |
| +- [ ] **Step 1: Write the tests**: `TestCleanupTwentyFourHoursAfterDone` (forward move done; at +24 h the source bucket's replica keys carry delete markers, `lock.json` stands, `retired.json` is `cleaned`, root's `.done` names the live link and programs file, the reservation is released; at +7 d the park directory is gone); `TestCleanupRefusals` (the record moved again; a current version differs; a move holds the reservation); `TestCleanupSupersededCancelsRoot` (a recreation during the root step: `.cancel`, `.cancelled`, release, `superseded`). |
| +- [ ] **Step 2: Run them; fix what they find within this plan's scope.** |
| +- [ ] **Step 3: Run both census tests and the cross-builds**; write `docs/superpowers/notes/census-proposal-self-serve-moves-4.md`. |
| +- [ ] **Step 4: Commit** `self-serve moves: cleanup end to end`; push. |
| + |
| +--- |
| + |
| +## Self-review notes |
| + |
| +- Spec coverage: §7.6 the source replica (T1, T2), parked local state (T3), root-owned leftovers (T4), the staged objects of an aborted move (unchanged: the stager removes them on abort today); §7.1's `cleanup` reservation and the create door's check (T2, T5); §8's cleanup pages (T4, T2's skipped keys page through the same sink); §11's cleanup witnesses (T6). |
| +- Names used across tasks: `moves.Retired`, `InventoryObject`, `LocalPath`, `RetiredPath`, `status.RetiredSummary`, `moves.Reserve`, `moves.Release`, `cleanupStep`, `reapParkedMoves`, `release.RootCleanupRequest`, `ServeRootCleanupRequests`, `writeRootCleanupRequest`. |
| +- Review Focus 1 to 5 are pinned by T2, T2, T2+T5, T4, T4. |