carlos / platform Public

Clone
git clone https://amadan.net/carlos/platform

Plain git — no account needed to clone.

Download

Download this file

Moves Phase 1, Plan 5: Cleanup and Parks Implementation Plan

> For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Leave no copy of the instance in a live namespace of the region that lost: abort parks staging from the target's side, every staging lane carries its seal bound as a persisted park scope, the target parks its staging after the commit, the move releases the claim to serving only when every obligation is gone, the at-rest sweep covers moves/<host>, the db door refuses move kinds, and the console finishes the claim halves and runs the deadline sweeps.

Architecture: Parks are claim obligations (park-replica scopes and park-generation chores, Plan 2 Task 9, index D6) executed by internal/park.Engine with a persisted ParkScope whose evidence_at is the genuine bound (the cut's stop evidence, the release's stop or fenced_at, the lane's seal bound, the sweep's listing). The box runs the immediate half (copies and version deletes) inside the move's phases; acknowledgement after the horizon is the regional chore runner's (claim Plan 6's hostbox.Regional.Run, widened to the two kinds). The release to serving is hostclaim.ReleaseMove (Plan 4 Task 2). The console's movesRecoveryPass writes claim halves the records decided and runs the deadline sweeps; it is a plain function claim Plan 5's Recovery.Pass can call.

Tech Stack: Go, internal/park, internal/hostclaim, internal/console.

Spec: rev 7.18 §6.4 (destinations, exact versions, verify after the horizon, the park scope grammar, the straggler sweep, moves/<host>/), §7.3 (retire staging after its writers' horizon), §7.5 (recovery owner for the claim half), §7.8 obligations, §7.9 steps 4 and 5, §7.11, §7.12, §13 paging, §19.1 (the db door), §19.2 K7, K8. Index: 2026-10-06-moves-phase1-0-index.md. Plans 1 to 4 merged (Plan 4 Tasks 9 and 10 may be pending).

Global Constraints

  • Gate, cross-compile, sandboxed GOCACHE; tests serial and niced. No em dashes.
  • evidence_at is always the original evidence or fencing bound, never the cleanup's start time; a scope is written once and every retry uses it (claim rev 6.23 §2; witnesses wParkScopeSurvivesRestart, wStagingScopeDeadline).
  • One deadline for every scope: acknowledge only after a verification listing later than evidence_at + H, H = 15 minutes + request_limit (120 s in production).
  • A park executes strictly within its scope; an out-of-scope key is never archived or deleted (INV-23; witness wSweepScopeMembership).
  • Abort is legal in preparing, ready and cutting/requested; the target region's box parks staging; the source parks its own snapshots (joint N13).
  • The release to serving names exec (target after done, source after rolled_back or aborted) and happens only when every obligation this op created is gone (K8).
  • The db door refuses on every move kind (§19.1).

Review Focus

  1. A staging lane sealed at time T with lease TTL L must be acknowledged no earlier than T + L + H and no later than the first pass after it, even across an agent restart (witness wStagingScopeDeadline). Test in Task 3.
  2. An aborted move must not leave the source trying to delete the target's staging (an IAM violation on every abort in the model) and must not close until the target's park acknowledged (§7.11). Test in Task 1 (TestAbortParksFromTheTargetAndWaits).
  3. A sweep of moves/<host> while an operation is staging there must not run; one on a free host must run as a holderless parking op and free the claim again in the same CAS as its last acknowledgement (witness wFreeSweepVsCreate). Test in Task 5.
  4. A carlos db pull during any move kind must get the family's 409 with Retry-After, including during landing/committed before the close (§19.1). Test in Task 6.
  5. The console pass must never write a claim half the record did not decide: a record at landing with the claim at moving/sealed is a flip half; a record at committed with the claim at landing/probed is a commit half; a record at rolling_back with the claim at landing is a rollback half; anything else is left alone and logged once (§7.5). Test in Task 7 (TestRecoveryPassWritesOnlyDecidedHalves).

---

File structure

| path | responsibility | |---|---| | internal/provision/provision.go | MoveAbortRequested (the auto-abort bound), MoveAbortComplete guard | | internal/hostclaim/move.go | Abort, QueueStagingPark use, BindScope | | internal/host/movesabort.go | abort on the source and the target | | internal/host/movesparks.go | the target's staging park after the commit; the finalization close | | internal/host/movessweep.go | the moves/<host> sweep scope on the box | | internal/console/apidbclaim.go | move kinds refused | | internal/console/movesrecovery.go | movesRecoveryPass | | internal/console/server.go | the pass's ticker |

---

Task 0: The paging primitive

Files: modify internal/host/alertsink.go; create internal/host/alertsink_page_test.go.

Interfaces: (a *alertSinks) PageOperator(ctx, level, subject, body string) error delivering through the same webhook and SNS topic as deliver (alertsink.go:99), and the package function host.PageOperator(ctx, level, subject, body string) error on the armed sinks (a no-op with a log line when none are configured, as today's alert delivery does). Moved here from Plan 6 (index D12) because Task 7's console pass and Plan 6's paging rules both need it, and Plan 6 depends on this plan.

  • [ ] Test TestPageOperatorUsesTheArmedSinks (a fake webhook poster records the subject and body; with no sinks the call returns nil and logs once); implement; commit: git commit -m "host: PageOperator, the paging primitive for moves (§13)" ...

---

Task 1: Abort with the target-owned staging park

Files: modify internal/hostclaim/move.go (Abort), internal/host/movespass.go (the MoveAborting arm), create internal/host/movesabort.go, tests.

Interfaces:

  • hostclaim.Abort(opID, targetRegion string, scope ParkScope, newID func() string) func(*Claim) error: KindMoving with phase "" (before the seal; a cut past requested cannot abort) → phase "aborting", appends the lane's park-replica (QueueStagingPark, D6) for targetRegion with scope (owner region:<target>); idempotent. Plan 1 Task 14's interim abort (release with no park) is replaced here.
  • The source, on aborting: hostclaim.Update(Abort(id, to, scope{Location: ScopeLocation(host, id), Incarnation, EvidenceAt: <the last round's lease SafeUntil the receipts recorded, i.e. the staging writers' horizon start>, RequestLimit: 120 s}, newID)); delete its own move-snapshots/<host>/<id>/ objects (own bucket; by version through DeleteVersion) and move-receipts/<host>/<id>*.json; then wait: MoveAbortComplete only when no obligation under the op remains and the claim is released (ReleaseMove(id, cfg.Box) first; the record clears second, §7.5 "the close is the other way round").
  • The target region's box, for a record in aborting or reverting whose To == cfg.Region: cfg.Park.Run on the op's lane park-replica obligation (copies and deletes at once; acknowledgement after the horizon through the regional runner). The reverting case's obligation is the one hostclaim.Revert queued (Plan 2 Task 2); Plan 4 Task 6's opened waits on it.
  • The staging horizon start for the abort scope: the receipts ledger records LeaseUntil of the last round (Plan 1 Task 14's moveLedger); evidence_at = LeaseUntil (the time after which no staging writer that held the lease can still write; the same bound the seal records for a cut).
  • Auto-abort: preparing with no promotable round for 60 minutes pages (§13) and, at 6 hours (moves.AbortAfter), the console sweep (Task 7) CASes MoveAbort.
  • The record's MoveAbort stays Plan 1 Task 2's; MoveAbortComplete gains the guard check through rec.Move.Legacy() (a legacy record completes as before, with the Plan 1 DeletePrefix path kept only for legacy records; the per-op path never deletes from the source).

Tests: TestAbortParksFromTheTargetAndWaits (Review Focus 2: the source's pass issues no delete against the target bucket; the target's pass archives moves/<host>/<id>/... under parked/<inc>/<host>/<ret>/moves/...; the record clears only after the obligation is acknowledged and the claim says serving with exec = source); TestAbortScopeIsTheWritersHorizon (witness wStagingQuiet: a staging PUT signed before the lease ended and landing after the park's first pass is found by the verification listing and parked as a further retirement event; the park acknowledges only after LeaseUntil + H).

  • [ ] Steps 1 to 5. Commit: git commit -m "host: abort parks staging from the target region and waits for the obligation (§7.11, joint N13)" ...

---

Task 2: BindScope and the scope rules

Files: modify internal/hostclaim/move.go, internal/hostclaim/move_test.go.

Interfaces:

  • hostclaim.BindScope(opID, obligationID string, scope ParkScope) func(*Claim) error: binds a scope to an unscoped obligation once (Park == nil), validates with scope.Validate(c.Host), refuses to rebind (ErrWrongState), so a retry or a restart never changes evidence_at (INV-22, witness wParkScopeSurvivesRestart).
  • moves.ScopeFor(kind string, host string, inc string, evidenceAt time.Time) hostclaim.ParkScope helpers: SourceLocationScope(location, inc, stopEvidenceAt), TargetLocationScope(location, inc, releaseEvidenceAt), StagingScope(host, opID, inc, sealBound), ReturnLaneScope(host, rbID, inc, returnSealBound), each RequestLimit: 120 s.
  • [ ] Test TestBindScopeOnce (witness wParkScopeSurvivesRestart: a second BindScope with a later EvidenceAt is refused and the first scope stands) and TestScopeHelpersUseTheGenuineBound (every helper's EvidenceAt is the argument, never time.Now()); implement; commit: git commit -m "hostclaim: BindScope; moves: scope helpers with genuine evidence bounds (claim rev 6.23 §2, INV-22)" ...

---

Task 3: The target's staging park after the commit, and the close

Files: create internal/host/movesparks.go, internal/host/movesparks_test.go; modify internal/host/movespass.go.

Interfaces:

  • For a committed record whose To == cfg.Region: run the lane's park-replica obligation (cfg.Park.Run; ErrPending until the horizon; C4's runner acknowledges). Its scope is the one Commit queued (evidence_at = seal.Bound(), Plan 3 Task 7).
  • The close (§7.9 step 5): any participant that sees OpenedAt, SourceParkedAt set and no obligation under the op: the terminal log event done (Plan 6's moves.LogEvent; until then a log.Printf), hostclaim.Update(ReleaseMove(id, targetBox)), provision.SetRecordMove(MoveClose(g, now)) (which writes LastMove{Outcome: "done"}, index D11). The console pass also tries (Task 7).
  • park.Engine's Now is the real clock; the test drives Now through the harness and a fake Leases.

Tests: TestStagingParkAcknowledgesAtSealBoundPlusH (Review Focus 1, witness wStagingScopeDeadline: seal bound T; Run at T + 16 min is ErrPending; a new worker (restart) at T + 17 min + 1 s acknowledges; the scope's EvidenceAt is still T); TestCloseWaitsForBothObligations (with the source's location park-replica still pending → no release; after both acknowledged → claim serving with exec = uk-box, Move == nil, region uk); TestCloseIsRecordLast (the claim is released before the record clears; a crash between leaves a finished move that admits nothing, and the next pass clears it).

  • [ ] Steps 1 to 5. Commit: git commit -m "host: the target's staging park, the horizon, and the close (§7.9 steps 4 and 5, K8)" ...

---

Task 4: Local parks of move retirements

Files: internal/park/local_linux_test.go only.

Interfaces: none new (D6). park.Local.RetireArtifacts and RetireInstallation keep the engine's layout, .parked/<incarnation>/<host>/<retirement>/artifacts/ (local_linux.go:196-201); the local journal already records Op, so F3 selects a move's local retirements by op the way it selects remote ones by manifest. The spec's .parked/superseded-<op id>/... wording is part of the rev 7.20 amendment D6 names.

  • [ ] Test TestLocalRetirementUnderAMoveOpIsFindableByOp (a park-local with a move id as Op lands under the engine's path and its journal names the op); commit: git commit -m "park: local retirements under a move op keep the engine's layout; the journal names the op (§6.4, D6)" ...

---

Task 5: The moves/<host> sweep scope

Files: create internal/host/movessweep.go, internal/host/movessweep_test.go; modify internal/park/sweep.go only if Decide does not already group moves/<host>/<op> lanes by location (claim rev 6.24 §2 says it does; verify with a test first).

Interfaces:

  • sweepMovesNamespace(ctx, s, w, host string): once a day and after each operation's release on this box (a w.released set the close writes into), when no operation holds the claim (Op.Kind == KindServing or free) and no move stages under moves/<host>/: cfg.Park.Sweep(ctx, host, legacyIncarnation, evidenceAt = the sweep's complete listing time, 120 s); the decision's candidates under moves/<host>/… are parked under one chore per lane scoped moves/<host>/<op>, or moves/<host> for a free host; on a free host the holderless sweep (hostclaim on main: the parking op with intent park_sweep, FreeAfterPark); on a serving host the chore's op is session:<holder.session>.
  • The regional runner (claim Plan 6) runs and acknowledges; this task only decides and queues.

Tests: TestSweepSkipsWhileAMoveStages (Review Focus 3); TestFreeHostSweepIsHolderlessAndFrees (witness wFreeSweepVsCreate); TestSweepScopeMembership (witness wSweepScopeMembership: a late object under instances/<host>/<old inc>/g-old/ is never referenced by the moves/<host> chore; it is the instances/<host>/<old inc> chore's).

  • [ ] Steps 1 to 5. Commit: git commit -m "host: the at-rest sweep covers moves/<host>, holderless on a free host (§6.4, CC20)" ...

---

Task 6: The db door refuses move kinds

Files: modify internal/console/apidbclaim.go:139-142, internal/console/apidbclaim_test.go.

  • Add hostclaim.KindMoving, hostclaim.KindLanding, hostclaim.KindReverting, hostclaim.KindRestoring to the switch returning errDBClaimClosed (409 with Retry-After: 1), for every phase including landing/committed (the door reopens at the close, when the claim is serving again).
  • Test TestDBDoorRefusesDuringMoves (Review Focus 4): a table over the four kinds and landing/committed.
  • [ ] Commit: git commit -m "console: the db door refuses on every move kind (§19.1)" ...

---

Task 7: The console's move pass

Files: create internal/console/movesrecovery.go, internal/console/movesrecovery_test.go; modify internal/console/server.go (a ticker every 60 s, after startup; s.cfg.MovesPassDisabled for tests).

Interfaces:

  • movesRecoveryPass(ctx, s release.Store, now time.Time, alert func(level, subject, body string)) error (D8). Two inputs, joined by host: provision.ListAllRecords (records with Move != nil) and claim discovery, s.List(ctx, "control/hosts/") (hostclaim.Key's prefix) read with hostclaim.Read for every claim whose Op.Kind is a move kind; a claim with a move kind whose record has no move, or no record at all, is an orphan (the request door failed after its claim CAS, or a record was deleted under a finished move). For each record with a move, hostclaim.Read: - the claim half the record decided (§7.5): record landing + claim moving/sealed with exec none → Flip (needs source_stamp and src_gen: the flip half is written by whoever sees it; the console computes sourceStamp from holder.stamp as observed, the pre-flip stamp, and srcGen from holder.generation); record committed + claim landing not committed → Commit with the scopes from the seal (seal.Bound()) and the evidence (stopped-source.json's CheckedAt); record rolling_back + claim landing → RollBack(id, rollback.id); anything else untouched (Review Focus 5). - an orphan moving/"" claim (from the discovery above) older than 10 minutes by Op.StartedAt → hostclaim.Update(ReleaseOrphanMove(id)), a guarded transition added to hostclaim/move.go in this task: legal only for KindMoving with phase "", Exec == participants[0], no obligation under the op, and the caller's assertion that the record carries no move for this id (the pass re-reads the record inside the retry loop; ReleaseMove itself rejects phase "" on purpose, so the orphan case has its own door); KindLanding/KindRestoring/KindReverting orphans are never released automatically: they page (an operation past the flip with no record is an incident). - deadline sweeps: landing past landing_deadline → MoveRollBack(g, NewRollbackID, "landing deadline", now) (the box writes the claim half or this pass does on its next tick); rolling_back past release_deadline with no target evidence → alert("incident", ...) once per hour (M-C10's sentence for the operator view); preparing with UpdatedAt older than 60 minutes → alert("warning", ...); older than moves.AbortAfter (6 h) → MoveAbort(). - the finalization halves: committed with both stamps and no obligations → ReleaseMove then MoveClose; rolling_back with the three stamps and no obligations → ReleaseMove(source) then MoveRolledBackClose; aborting with no obligations → ReleaseMove(source) then MoveAbortComplete; a rolling_back record whose target is on claim.Lost and whose target phase is released (the loss committed) gets TargetParkedAt stamped by this pass once no obligation under the op is owned by that box or by its region (Plan 4 Task 10 step 5). - the landing phase catch-up (Plan 2 Task 10 syncClaimPhase) for any landing record whose claim phase is behind its landing_phase. - a source terminated before the flip (spec rev 7.20 §7.10, index D16; BLOCKED-ON-CLAIM C7): a moving claim whose source box carries a verified termination fence and whose record is not landing → the claim's one-CAS conversion to relocating/pinning (C7's helper) and the record's MoveRelocating(g, reason) (a provision transition this task adds: preparing|ready|cutting → relocating, keeping Move.ID; the record clears when the relocation releases, through the serving finalization half above, which already clears a finished move); when the source region has no live enrolled box, one incident page with M-C22's sentence (host.PageOperator, rate-limited by the op journal), and GET …/move shows state: relocating with detail.waiting_for_box = <region> (Plan 6 Task 2). A moving claim whose record is already landing is not converted: the flip is the record's decision and §7.10's lost-target path applies.
  • Server.startMovesPass(ctx): the ticker, logging each pass's errors; alert is host.PageOperator (Task 0), which the console binary links already (internal/host is in the CLI).
  • The closes write MoveClose(g, now), MoveRolledBackClose(g, now) and MoveAbortComplete with LastMove (index D11).

Tests: TestRecoveryPassWritesOnlyDecidedHalves (Review Focus 5); TestRecoveryPassRollsBackAtTheLandingDeadline; TestRecoveryPassReleasesAnOrphanMovingClaim (the whole pass after a request-door crash: a moving claim at control/hosts/<host>.json, a record with no move; after 10 minutes the claim is serving with exec = source; a landing orphan is paged and left alone); TestRecoveryPassFinishesTheClose.

  • [ ] Steps 1 to 5. Commit: git commit -m "console: the move pass writes decided claim halves, sweeps deadlines and finishes closes (§7.5, §13, D8)" ...

---

Task 8: Gate and copy

  • [ ] Full gate, cross-compile.
  • [ ] Copy list for Plan 6: M-C13 (abort refused after the cut started), M-C19 (delete during a move, the delete door's 409: add the refusal to the delete doors' claim CAS failure path: hostclaim.BeginDelete's ErrWrongState on a move kind maps to M-C19), the paging subjects: "move rolled back", "move stuck", "move waiting for <to> to release", "move preparing for over an hour".

Self-review notes

Coverage: §7.11 (Task 1), §6.4 scopes, horizon, local destinations, the sweep and moves/<host>/ (Tasks 2 to 5), §7.9 steps 4 and 5 and K8 (Task 3), §7.12 delete refused (Task 8's M-C19 mapping; the claim's StartGuard already refuses), §19.1 db door (Task 6), §7.5 recovery owner and §13 paging (Task 7). The regional acknowledgement after the horizon is claim Plan 6's runner, listed in the index.