carlos / platform Public
Moves Phase 1, Plan 5: Cleanup and Parks Implementation Plan
> For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.
Goal: Leave no copy of the instance in a live namespace of the region that lost: abort parks staging from the target's side, every staging lane carries its seal bound as a persisted park scope, the target parks its staging after the commit, the move releases the claim to serving only when every obligation is gone, the at-rest sweep covers moves/<host>, the db door refuses move kinds, and the console finishes the claim halves and runs the deadline sweeps.
Architecture: Parks are claim obligations (park-replica scopes and park-generation chores, Plan 2 Task 9, index D6) executed by internal/park.Engine with a persisted ParkScope whose evidence_at is the genuine bound (the cut's stop evidence, the release's stop or fenced_at, the lane's seal bound, the sweep's listing). The box runs the immediate half (copies and version deletes) inside the move's phases; acknowledgement after the horizon is the regional chore runner's (claim Plan 6's hostbox.Regional.Run, widened to the two kinds). The release to serving is hostclaim.ReleaseMove (Plan 4 Task 2). The console's movesRecoveryPass writes claim halves the records decided and runs the deadline sweeps; it is a plain function claim Plan 5's Recovery.Pass can call.
Tech Stack: Go, internal/park, internal/hostclaim, internal/console.
Spec: rev 7.18 §6.4 (destinations, exact versions, verify after the horizon, the park scope grammar, the straggler sweep, moves/<host>/), §7.3 (retire staging after its writers' horizon), §7.5 (recovery owner for the claim half), §7.8 obligations, §7.9 steps 4 and 5, §7.11, §7.12, §13 paging, §19.1 (the db door), §19.2 K7, K8. Index: 2026-10-06-moves-phase1-0-index.md. Plans 1 to 4 merged (Plan 4 Tasks 9 and 10 may be pending).
Global Constraints
- Gate, cross-compile, sandboxed
GOCACHE; tests serial and niced. No em dashes. evidence_atis always the original evidence or fencing bound, never the cleanup's start time; a scope is written once and every retry uses it (claim rev 6.23 §2; witnesseswParkScopeSurvivesRestart,wStagingScopeDeadline).- One deadline for every scope: acknowledge only after a verification listing later than
evidence_at + H,H= 15 minutes +request_limit(120 s in production). - A park executes strictly within its scope; an out-of-scope key is never archived or deleted (INV-23; witness
wSweepScopeMembership). - Abort is legal in
preparing,readyandcutting/requested; the target region's box parks staging; the source parks its own snapshots (joint N13). - The release to
servingnamesexec(target afterdone, source afterrolled_backoraborted) and happens only when every obligation this op created is gone (K8). - The db door refuses on every move kind (§19.1).
Review Focus
- A staging lane sealed at time T with lease TTL L must be acknowledged no earlier than
T + L + Hand no later than the first pass after it, even across an agent restart (witnesswStagingScopeDeadline). Test in Task 3. - An aborted move must not leave the source trying to delete the target's staging (an IAM violation on every abort in the model) and must not close until the target's park acknowledged (§7.11). Test in Task 1 (
TestAbortParksFromTheTargetAndWaits). - A sweep of
moves/<host>while an operation is staging there must not run; one on a free host must run as a holderlessparkingop and free the claim again in the same CAS as its last acknowledgement (witnesswFreeSweepVsCreate). Test in Task 5. - A
carlos db pullduring any move kind must get the family's 409 withRetry-After, including duringlanding/committedbefore the close (§19.1). Test in Task 6. - The console pass must never write a claim half the record did not decide: a record at
landingwith the claim atmoving/sealedis a flip half; a record atcommittedwith the claim atlanding/probedis a commit half; a record atrolling_backwith the claim atlandingis a rollback half; anything else is left alone and logged once (§7.5). Test in Task 7 (TestRecoveryPassWritesOnlyDecidedHalves).
---
File structure
| path | responsibility | |---|---| | internal/provision/provision.go | MoveAbortRequested (the auto-abort bound), MoveAbortComplete guard | | internal/hostclaim/move.go | Abort, QueueStagingPark use, BindScope | | internal/host/movesabort.go | abort on the source and the target | | internal/host/movesparks.go | the target's staging park after the commit; the finalization close | | internal/host/movessweep.go | the moves/<host> sweep scope on the box | | internal/console/apidbclaim.go | move kinds refused | | internal/console/movesrecovery.go | movesRecoveryPass | | internal/console/server.go | the pass's ticker |
---
Task 0: The paging primitive
Files: modify internal/host/alertsink.go; create internal/host/alertsink_page_test.go.
Interfaces: (a *alertSinks) PageOperator(ctx, level, subject, body string) error delivering through the same webhook and SNS topic as deliver (alertsink.go:99), and the package function host.PageOperator(ctx, level, subject, body string) error on the armed sinks (a no-op with a log line when none are configured, as today's alert delivery does). Moved here from Plan 6 (index D12) because Task 7's console pass and Plan 6's paging rules both need it, and Plan 6 depends on this plan.
- [ ] Test
TestPageOperatorUsesTheArmedSinks(a fake webhook poster records the subject and body; with no sinks the call returns nil and logs once); implement; commit:git commit -m "host: PageOperator, the paging primitive for moves (§13)" ...
---
Task 1: Abort with the target-owned staging park
Files: modify internal/hostclaim/move.go (Abort), internal/host/movespass.go (the MoveAborting arm), create internal/host/movesabort.go, tests.
Interfaces:
hostclaim.Abort(opID, targetRegion string, scope ParkScope, newID func() string) func(*Claim) error:KindMovingwith phase""(before the seal; a cut pastrequestedcannot abort) → phase"aborting", appends the lane'spark-replica(QueueStagingPark, D6) fortargetRegionwithscope(ownerregion:<target>); idempotent. Plan 1 Task 14's interim abort (release with no park) is replaced here.- The source, on
aborting:hostclaim.Update(Abort(id, to, scope{Location: ScopeLocation(host, id), Incarnation, EvidenceAt: <the last round's lease SafeUntil the receipts recorded, i.e. the staging writers' horizon start>, RequestLimit: 120 s}, newID)); delete its ownmove-snapshots/<host>/<id>/objects (own bucket; by version throughDeleteVersion) andmove-receipts/<host>/<id>*.json; then wait:MoveAbortCompleteonly when no obligation under the op remains and the claim is released (ReleaseMove(id, cfg.Box)first; the record clears second, §7.5 "the close is the other way round"). - The target region's box, for a record in
abortingorrevertingwhoseTo == cfg.Region:cfg.Park.Runon the op's lanepark-replicaobligation (copies and deletes at once; acknowledgement after the horizon through the regional runner). The reverting case's obligation is the onehostclaim.Revertqueued (Plan 2 Task 2); Plan 4 Task 6'sopenedwaits on it. - The staging horizon start for the abort scope: the receipts ledger records
LeaseUntilof the last round (Plan 1 Task 14'smoveLedger);evidence_at = LeaseUntil(the time after which no staging writer that held the lease can still write; the same bound the seal records for a cut). - Auto-abort:
preparingwith no promotable round for 60 minutes pages (§13) and, at 6 hours (moves.AbortAfter), the console sweep (Task 7) CASesMoveAbort. - The record's
MoveAbortstays Plan 1 Task 2's;MoveAbortCompletegains the guard check throughrec.Move.Legacy()(a legacy record completes as before, with the Plan 1DeletePrefixpath kept only for legacy records; the per-op path never deletes from the source).
Tests: TestAbortParksFromTheTargetAndWaits (Review Focus 2: the source's pass issues no delete against the target bucket; the target's pass archives moves/<host>/<id>/... under parked/<inc>/<host>/<ret>/moves/...; the record clears only after the obligation is acknowledged and the claim says serving with exec = source); TestAbortScopeIsTheWritersHorizon (witness wStagingQuiet: a staging PUT signed before the lease ended and landing after the park's first pass is found by the verification listing and parked as a further retirement event; the park acknowledges only after LeaseUntil + H).
- [ ] Steps 1 to 5. Commit:
git commit -m "host: abort parks staging from the target region and waits for the obligation (§7.11, joint N13)" ...
---
Task 2: BindScope and the scope rules
Files: modify internal/hostclaim/move.go, internal/hostclaim/move_test.go.
Interfaces:
hostclaim.BindScope(opID, obligationID string, scope ParkScope) func(*Claim) error: binds a scope to an unscoped obligation once (Park == nil), validates withscope.Validate(c.Host), refuses to rebind (ErrWrongState), so a retry or a restart never changesevidence_at(INV-22, witnesswParkScopeSurvivesRestart).moves.ScopeFor(kind string, host string, inc string, evidenceAt time.Time) hostclaim.ParkScopehelpers:SourceLocationScope(location, inc, stopEvidenceAt),TargetLocationScope(location, inc, releaseEvidenceAt),StagingScope(host, opID, inc, sealBound),ReturnLaneScope(host, rbID, inc, returnSealBound), eachRequestLimit: 120 s.
- [ ] Test
TestBindScopeOnce(witnesswParkScopeSurvivesRestart: a secondBindScopewith a laterEvidenceAtis refused and the first scope stands) andTestScopeHelpersUseTheGenuineBound(every helper'sEvidenceAtis the argument, nevertime.Now()); implement; commit:git commit -m "hostclaim: BindScope; moves: scope helpers with genuine evidence bounds (claim rev 6.23 §2, INV-22)" ...
---
Task 3: The target's staging park after the commit, and the close
Files: create internal/host/movesparks.go, internal/host/movesparks_test.go; modify internal/host/movespass.go.
Interfaces:
- For a
committedrecord whoseTo == cfg.Region: run the lane'spark-replicaobligation (cfg.Park.Run;ErrPendinguntil the horizon; C4's runner acknowledges). Its scope is the oneCommitqueued (evidence_at = seal.Bound(), Plan 3 Task 7). - The close (§7.9 step 5): any participant that sees
OpenedAt,SourceParkedAtset and no obligation under the op: the terminal log eventdone(Plan 6'smoves.LogEvent; until then alog.Printf),hostclaim.Update(ReleaseMove(id, targetBox)),provision.SetRecordMove(MoveClose(g, now))(which writesLastMove{Outcome: "done"}, index D11). The console pass also tries (Task 7). park.Engine'sNowis the real clock; the test drivesNowthrough the harness and a fakeLeases.
Tests: TestStagingParkAcknowledgesAtSealBoundPlusH (Review Focus 1, witness wStagingScopeDeadline: seal bound T; Run at T + 16 min is ErrPending; a new worker (restart) at T + 17 min + 1 s acknowledges; the scope's EvidenceAt is still T); TestCloseWaitsForBothObligations (with the source's location park-replica still pending → no release; after both acknowledged → claim serving with exec = uk-box, Move == nil, region uk); TestCloseIsRecordLast (the claim is released before the record clears; a crash between leaves a finished move that admits nothing, and the next pass clears it).
- [ ] Steps 1 to 5. Commit:
git commit -m "host: the target's staging park, the horizon, and the close (§7.9 steps 4 and 5, K8)" ...
---
Task 4: Local parks of move retirements
Files: internal/park/local_linux_test.go only.
Interfaces: none new (D6). park.Local.RetireArtifacts and RetireInstallation keep the engine's layout, .parked/<incarnation>/<host>/<retirement>/artifacts/ (local_linux.go:196-201); the local journal already records Op, so F3 selects a move's local retirements by op the way it selects remote ones by manifest. The spec's .parked/superseded-<op id>/... wording is part of the rev 7.20 amendment D6 names.
- [ ] Test
TestLocalRetirementUnderAMoveOpIsFindableByOp(apark-localwith a move id asOplands under the engine's path and its journal names the op); commit:git commit -m "park: local retirements under a move op keep the engine's layout; the journal names the op (§6.4, D6)" ...
---
Task 5: The moves/<host> sweep scope
Files: create internal/host/movessweep.go, internal/host/movessweep_test.go; modify internal/park/sweep.go only if Decide does not already group moves/<host>/<op> lanes by location (claim rev 6.24 §2 says it does; verify with a test first).
Interfaces:
sweepMovesNamespace(ctx, s, w, host string): once a day and after each operation's release on this box (aw.releasedset the close writes into), when no operation holds the claim (Op.Kind == KindServingor free) and no move stages undermoves/<host>/:cfg.Park.Sweep(ctx, host, legacyIncarnation, evidenceAt = the sweep's complete listing time, 120 s); the decision's candidates undermoves/<host>/…are parked under one chore per lane scopedmoves/<host>/<op>, ormoves/<host>for a free host; on a free host the holderless sweep (hostclaimon main: theparkingop withintent park_sweep,FreeAfterPark); on a serving host the chore's op issession:<holder.session>.- The regional runner (claim Plan 6) runs and acknowledges; this task only decides and queues.
Tests: TestSweepSkipsWhileAMoveStages (Review Focus 3); TestFreeHostSweepIsHolderlessAndFrees (witness wFreeSweepVsCreate); TestSweepScopeMembership (witness wSweepScopeMembership: a late object under instances/<host>/<old inc>/g-old/ is never referenced by the moves/<host> chore; it is the instances/<host>/<old inc> chore's).
- [ ] Steps 1 to 5. Commit:
git commit -m "host: the at-rest sweep covers moves/<host>, holderless on a free host (§6.4, CC20)" ...
---
Task 6: The db door refuses move kinds
Files: modify internal/console/apidbclaim.go:139-142, internal/console/apidbclaim_test.go.
- Add
hostclaim.KindMoving, hostclaim.KindLanding, hostclaim.KindReverting, hostclaim.KindRestoringto theswitchreturningerrDBClaimClosed(409 withRetry-After: 1), for every phase includinglanding/committed(the door reopens at the close, when the claim isservingagain). - Test
TestDBDoorRefusesDuringMoves(Review Focus 4): a table over the four kinds andlanding/committed.
- [ ] Commit:
git commit -m "console: the db door refuses on every move kind (§19.1)" ...
---
Task 7: The console's move pass
Files: create internal/console/movesrecovery.go, internal/console/movesrecovery_test.go; modify internal/console/server.go (a ticker every 60 s, after startup; s.cfg.MovesPassDisabled for tests).
Interfaces:
movesRecoveryPass(ctx, s release.Store, now time.Time, alert func(level, subject, body string)) error(D8). Two inputs, joined by host:provision.ListAllRecords(records withMove != nil) and claim discovery,s.List(ctx, "control/hosts/")(hostclaim.Key's prefix) read withhostclaim.Readfor every claim whoseOp.Kindis a move kind; a claim with a move kind whose record has no move, or no record at all, is an orphan (the request door failed after its claim CAS, or a record was deleted under a finished move). For each record with a move,hostclaim.Read: - the claim half the record decided (§7.5): recordlanding+ claimmoving/sealedwith exec none →Flip(needssource_stampandsrc_gen: the flip half is written by whoever sees it; the console computessourceStampfromholder.stampas observed, the pre-flip stamp, andsrcGenfromholder.generation); recordcommitted+ claimlandingnotcommitted→Commitwith the scopes from the seal (seal.Bound()) and the evidence (stopped-source.json'sCheckedAt); recordrolling_back+ claimlanding→RollBack(id, rollback.id); anything else untouched (Review Focus 5). - an orphanmoving/""claim (from the discovery above) older than 10 minutes byOp.StartedAt→hostclaim.Update(ReleaseOrphanMove(id)), a guarded transition added tohostclaim/move.goin this task: legal only forKindMovingwith phase"",Exec == participants[0], no obligation under the op, and the caller's assertion that the record carries no move for this id (the pass re-reads the record inside the retry loop;ReleaseMoveitself rejects phase""on purpose, so the orphan case has its own door);KindLanding/KindRestoring/KindRevertingorphans are never released automatically: they page (an operation past the flip with no record is an incident). - deadline sweeps:landingpastlanding_deadline→MoveRollBack(g, NewRollbackID, "landing deadline", now)(the box writes the claim half or this pass does on its next tick);rolling_backpastrelease_deadlinewith no target evidence →alert("incident", ...)once per hour (M-C10's sentence for the operator view);preparingwithUpdatedAtolder than 60 minutes →alert("warning", ...); older thanmoves.AbortAfter(6 h) →MoveAbort(). - the finalization halves:committedwith both stamps and no obligations →ReleaseMovethenMoveClose;rolling_backwith the three stamps and no obligations →ReleaseMove(source)thenMoveRolledBackClose;abortingwith no obligations →ReleaseMove(source)thenMoveAbortComplete; arolling_backrecord whose target is onclaim.Lostand whose target phase isreleased(the loss committed) getsTargetParkedAtstamped by this pass once no obligation under the op is owned by that box or by its region (Plan 4 Task 10 step 5). - the landing phase catch-up (Plan 2 Task 10syncClaimPhase) for anylandingrecord whose claim phase is behind itslanding_phase. - a source terminated before the flip (spec rev 7.20 §7.10, index D16; BLOCKED-ON-CLAIM C7): amovingclaim whose source box carries a verified termination fence and whose record is notlanding→ the claim's one-CAS conversion torelocating/pinning(C7's helper) and the record'sMoveRelocating(g, reason)(aprovisiontransition this task adds:preparing|ready|cutting → relocating, keepingMove.ID; the record clears when the relocation releases, through theservingfinalization half above, which already clears a finishedmove); when the source region has no live enrolled box, one incident page with M-C22's sentence (host.PageOperator, rate-limited by the op journal), andGET …/moveshowsstate: relocatingwithdetail.waiting_for_box = <region>(Plan 6 Task 2). Amovingclaim whose record is alreadylandingis not converted: the flip is the record's decision and §7.10's lost-target path applies.Server.startMovesPass(ctx): the ticker, logging each pass's errors;alertishost.PageOperator(Task 0), which the console binary links already (internal/hostis in the CLI).- The closes write
MoveClose(g, now),MoveRolledBackClose(g, now)andMoveAbortCompletewithLastMove(index D11).
Tests: TestRecoveryPassWritesOnlyDecidedHalves (Review Focus 5); TestRecoveryPassRollsBackAtTheLandingDeadline; TestRecoveryPassReleasesAnOrphanMovingClaim (the whole pass after a request-door crash: a moving claim at control/hosts/<host>.json, a record with no move; after 10 minutes the claim is serving with exec = source; a landing orphan is paged and left alone); TestRecoveryPassFinishesTheClose.
- [ ] Steps 1 to 5. Commit:
git commit -m "console: the move pass writes decided claim halves, sweeps deadlines and finishes closes (§7.5, §13, D8)" ...
---
Task 8: Gate and copy
- [ ] Full gate, cross-compile.
- [ ] Copy list for Plan 6: M-C13 (abort refused after the cut started), M-C19 (delete during a move, the delete door's 409: add the refusal to the delete doors' claim CAS failure path:
hostclaim.BeginDelete'sErrWrongStateon a move kind maps to M-C19), the paging subjects: "move rolled back", "move stuck", "move waiting for <to> to release", "move preparing for over an hour".
Self-review notes
Coverage: §7.11 (Task 1), §6.4 scopes, horizon, local destinations, the sweep and moves/<host>/ (Tasks 2 to 5), §7.9 steps 4 and 5 and K8 (Task 3), §7.12 delete refused (Task 8's M-C19 mapping; the claim's StartGuard already refuses), §19.1 db door (Task 6), §7.5 recovery owner and §13 paging (Task 7). The regional acknowledgement after the horizon is claim Plan 6's runner, listed in the index.