amadan
spec rev 7.19 (P7 cold-only, region HA L14 to L17 folded, L18 stated as a limit); plans rev 3 (review round 2, 25 findings and the round-1 partials)
Spec: §0.5 records claim rev 6.27 (989ee349) §1.1's #236 rulings and the region HA requests; §7.1 the target box in the start CAS by choose(); §7.10 the termination fence as M16's evidence; §7.6 one install routine, two sources, and the files manifest; §8.3, §9.4; §14.2.6 the one-box-per-region limit; §19.1 cold-only. Plans: index D6 drops park-superseded and park-staging for park-replica scopes and park-generation residue chores under parked/ (no new claim kind); D13 the operator-off switch CARLOS_MOVES_PHASE1; D14 cold-only; the BLOCKED-ON-CLAIM contracts C1 to C8 with platform-1e's work order; round-2 dispositions. Local fixes across plans 1 to 7 as the index table lists them. Round-2 findings and the region HA conflicts record committed beside the plans.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
11 files changed,
+679
−125
docs/superpowers/plans/2026-10-06-moves-phase1-0-index.md+86 −15docs/superpowers/plans/2026-10-06-moves-phase1-1-copy-and-staging.md+97 −18docs/superpowers/plans/2026-10-06-moves-phase1-2-cut-flip-landing.md+33 −26docs/superpowers/plans/2026-10-06-moves-phase1-3-probe-commit-open.md+23 −22docs/superpowers/plans/2026-10-06-moves-phase1-4-rollback-return-declare-lost.md+26 −25docs/superpowers/plans/2026-10-06-moves-phase1-5-cleanup-and-parks.md+11 −10docs/superpowers/plans/2026-10-06-moves-phase1-6-cli-status-observability.md+3 −3docs/superpowers/plans/2026-10-06-moves-phase1-7-rehearsal-runbook.md+4 −3docs/superpowers/plans/2026-10-06-moves-region-ha-conflicts.md+99 −0docs/superpowers/plans/CODEX-PLAN-MOVES-PHASE1-R2.md+148 −0docs/superpowers/specs/2026-10-02-move-any-app-design.md+149 −3
CI passed — run details
diff --git a/docs/superpowers/plans/2026-10-06-moves-phase1-0-index.md b/docs/superpowers/plans/2026-10-06-moves-phase1-0-index.md| index fe358ef85..0df5cc745 100644 |
| --- a/docs/superpowers/plans/2026-10-06-moves-phase1-0-index.md |
| +++ b/docs/superpowers/plans/2026-10-06-moves-phase1-0-index.md |
| @@ -11,24 +11,25 @@ on `main` at 67547f5 or later, as an ordered series of plans that each |
| produce working, tested, mergeable software. |
| **Spec:** `docs/superpowers/specs/2026-10-02-move-any-app-design.md` (rev |
| -7.18). The executable model is `internal/sim/hostlife/` at 9c417aaa on the |
| -same branch; its witness names are cited in the tests the plans port. |
| +7.19: P7 cold-only and region HA L14 to L17 folded in; L18 stated as a |
| +limit). The executable model is `internal/sim/hostlife/` at 9c417aaa on |
| +the same branch; its witness names are cited in the tests the plans port. |
| **Claim library on `main` (built on, never re-implemented):** |
| -`internal/hostclaim` (E, rev 6.26 semantics), `internal/genselect` (F), |
| -`internal/hostguard` and `internal/stopevidence` (claim Plan 3), |
| -`internal/park` (claim Plan 4), `internal/console/apidbclaim.go` (claim |
| -Plan 7). |
| +`internal/hostclaim` (E, rev 6.27 semantics; 989ee349 §1.1 records the |
| +#236 rulings), `internal/genselect` (F), `internal/hostguard` and |
| +`internal/stopevidence` (claim Plan 3), `internal/park` (claim Plan 4), |
| +`internal/console/apidbclaim.go` (claim Plan 7). |
| ## The series |
| | # | plan | produces | needs from claim plans 5, 6, 8 | |
| |---|---|---|---| |
| | 1 | [`2026-10-06-moves-phase1-1-copy-and-staging.md`](2026-10-06-moves-phase1-1-copy-and-staging.md) | op ids, the record's move journal, the `moving` claim CAS at the request door (operator-only), the per-op staging layout with `move.json`, seal-ready copy rounds, the files archive, the timers snapshot, the launch and readiness reports (T1 to T12, S1, S2), the T11 budget, the protocol floor and `moves_protocol`, the `hold_op` column, the copy gate | nothing | |
| -| 2 | [`2026-10-06-moves-phase1-2-cut-flip-landing.md`](2026-10-06-moves-phase1-2-cut-flip-landing.md) | the journaled cut (requested to flipped), `ReconstructStopEvidence`, the seal by version, the archive publication, the flip CAS, hold rows and reconcile's participant rules, timers frozen and drained, `RenderHeld`, `MaterializeHeld`, the landing's `checking` to `restored` phases including the generation build (snapshot, anchor, build intent, install manifest), `park-superseded` and `park-staging` destinations | nothing | |
| -| 3 | [`2026-10-06-moves-phase1-3-probe-commit-open.md`](2026-10-06-moves-phase1-3-probe-commit-open.md) | the probe policy (attempt windows, deadline checks, budgeted attempts), `ProbeWake` as a cold admission into the prepared generation, the commit record CAS and its claim half (obligations, `CoverRegionPredecessors`), the conditional open, the source's finalization park | Tasks 5 to 7 need claim Plan 6 (`hostbox.Writer`, the claim-aware start sequence); Tasks 1 to 4 and 8 do not | |
| -| 4 | [`2026-10-06-moves-phase1-4-rollback-return-declare-lost.md`](2026-10-06-moves-phase1-4-rollback-return-declare-lost.md) | `reverting` (pre-flip recovery), the rollback decision, the target's release with the clean/dirty rule (`src_covered`), the dirty return lane, the source's `claimed` CAS (`holder.stamp = source_stamp`, clean `holder.generation = src_gen`), the dirty install on the source, the target's `cleaning`, `stuck`, `--retry`, declare-lost | the recovery probes need claim Plan 6 (as Plan 3); declare-lost (Task 10) needs claim Plan 8 (`hostclaim.DeclareLost`, `LossOperationOwner`, `AdmitWriterWithLoss`, `LossRecord`) | |
| -| 5 | [`2026-10-06-moves-phase1-5-cleanup-and-parks.md`](2026-10-06-moves-phase1-5-cleanup-and-parks.md) | abort with the target-owned staging park, the staging horizon as a persisted `ParkScope`, the finalization parks and the release to `serving` (K8), the `moves/<host>` sweep scope, the db door's refusal on move kinds, the console's claim-half recovery pass and deadline sweeps | the box that runs `park-superseded` and `park-staging` chores is claim Plan 6's `hostbox.Regional.Run` (its kind allow-list must admit the two kinds, listed below); the console pass can later be called from claim Plan 5's `Recovery.Pass` | |
| +| 2 | [`2026-10-06-moves-phase1-2-cut-flip-landing.md`](2026-10-06-moves-phase1-2-cut-flip-landing.md) | the journaled cut (requested to flipped), `stopevidence.Reconstruct`, the seal by version, the archive publication, the flip CAS, hold rows and reconcile's participant rules, timers frozen and drained, `RenderHeld`, `MaterializeHeld`, the landing's `checking` to `restored` phases including the generation build (snapshot, anchor, build intent, install manifest), the park queue helpers and `Engine.Parked` | C3 and C8 for two sub-steps of `local`; everything else nothing | |
| +| 3 | [`2026-10-06-moves-phase1-3-probe-commit-open.md`](2026-10-06-moves-phase1-3-probe-commit-open.md) | the probe policy (attempt windows, deadline checks, budgeted attempts), `ProbeWake` as a cold admission into the prepared generation, the commit record CAS and its claim half (obligations, `CoverRegionPredecessors`), the conditional open, the source's finalization park | Tasks 5 to 7 BLOCKED-ON-CLAIM (C1, C2); Tasks 1 to 4 and 8 are not | |
| +| 4 | [`2026-10-06-moves-phase1-4-rollback-return-declare-lost.md`](2026-10-06-moves-phase1-4-rollback-return-declare-lost.md) | `reverting` (pre-flip recovery), the rollback decision, the target's release with the clean/dirty rule (`src_covered`), the dirty return lane, the source's `claimed` CAS (`holder.stamp = source_stamp`, clean `holder.generation = src_gen`), the dirty install on the source, the target's `cleaning`, `stuck`, `--retry`, the record side of declare-lost | Task 9 BLOCKED-ON-CLAIM (C1, C2); Task 10 BLOCKED-ON-CLAIM (C6, C7); the lost-target branch of Task 5 with it | |
| +| 5 | [`2026-10-06-moves-phase1-5-cleanup-and-parks.md`](2026-10-06-moves-phase1-5-cleanup-and-parks.md) | the paging primitive, abort with the target-owned staging park, the staging horizon as a persisted `ParkScope`, the finalization parks and the release to `serving` (K8), the `moves/<host>` sweep scope, the db door's refusal on move kinds, the console's claim-half recovery pass and deadline sweeps | the acknowledgement after the horizon of every chore is C4's runner; the immediate half is built here; the console pass can later be called from claim Plan 5's `Recovery.Pass` | |
| | 6 | [`2026-10-06-moves-phase1-6-cli-status-observability.md`](2026-10-06-moves-phase1-6-cli-status-observability.md) | `carlos move` and its four sub-verbs, `POST …/move/resolve`, `GET …/move?detail=1`, exit codes 0/3/4/5, the move log, the member stream events, paging rules, the copy gate, the `docs/site/cli.md` heading | nothing (the `--declare-lost` flag calls Plan 4 Task 10's door and is built with it) | |
| | 7 | [`2026-10-06-moves-phase1-7-rehearsal-runbook.md`](2026-10-06-moves-phase1-7-rehearsal-runbook.md) | the `moveprobe` test app, the cross-version tests, the rollout steps and the rehearsal runbook in M12's order | every plan above, merged and rolled with Paul's go | |
| @@ -72,9 +73,10 @@ changes with it. |
| Task 6 requires that sequence to be callable for a held row with a |
| probe policy: moves asks Plan 6 for one exported entry, |
| `(a *Service) startAdmitted(ctx, e *actEntry, admitted hostclaim.Observed, policy startPolicy) error`, or equivalent, where `startPolicy{MaxStarts int; Window time.Duration; Deadline time.Time; Probe bool}` is the only new input. Until that exists, Plan 3 Tasks 5 to 7 cannot be built. |
| -- `hostbox.Regional.Run` must admit obligation kinds `park-superseded` |
| - and `park-staging` beside `park-replica` and `park-generation` (Plan 5 |
| - Task 2 defines the kinds; Plan 6's allow-list is the one line to widen). |
| +- `hostbox.Regional.Run` runs `park-replica` chores whose scope is a |
| + `moves/<host>/<op id>` lane or a whole location, and `park-generation` |
| + chores, under a move or rollback op id (platform-1e work order item 1: |
| + "Regional.Run for park-replica moves scopes"); no new kind (D6). |
| - `spawnClaimed(ctx, enabled bool, manager *cgroup.Manager, host string, limit cgroup.Limit, selected hostwrite.Work, identity hostwrite.Identity, build func() *exec.Cmd) (*exec.Cmd, *cgroup.Prepared, error)` (claim Plan 6 line 878): every spawn under a move is strict (§5.3); `ProbeWake` spawns through it with `CARLOS_MOVE_PROBE=1`, keeps the returned `*cgroup.Prepared` for the process's whole life, and on every exit path (a failed start included) runs `Prepared.KillAndEmpty` then `Prepared.Close` before the attempt is counted. |
| - The wake admission of Plan 6 Task 9 must honour moves' K6: `doWake` reads the claim fresh and, when `op.exec` does not name this box, kills the host's process tree (`ReconstructStopEvidence`'s step 1, Plan 2 Task 3) before anything else; it admits a wake under `landing/committed` for the target (§7.14 row "landing, committed, not yet finalized") and under `reverting`/`restoring` phases `probed` and `opened` for the source. Its restore writes `<dbPath>.gen` and retires (`park.Local.RetireArtifacts`) a warm database whose `.gen` names another generation before restoring cold (§5.1 start check rule 3, §9.3). |
| - `schedule.SelectedTimers` (the timer-control adapter): Plan 2 Task 9's |
| @@ -102,6 +104,33 @@ additions, `internal/hostexec`, `internal/hostquarantine`, |
| - `Recovery.Pass(ctx) error` and `(s *Server) StartClaimRecoveryPass(ctx)`: Plan 5 Task 7's `movesRecoveryPass` is written to be called from `Recovery.Pass` once it exists; until then the console runs it on its own ticker. |
| - `hostops.RunOwnedRemote(ctx, e *park.Engine, w park.Work, scope ParkScopeProvider) error` and `ParkScopeProvider func(context.Context, hostclaim.Observed, hostclaim.Obligation) (hostclaim.ParkScope, error)`: the regional runner's scope provider for moves' chores returns the persisted scope the chore carries (Plan 5 Task 3 binds it at the queuing CAS), never a computed one. |
| +## BLOCKED-ON-CLAIM contracts (rev 3) |
| + |
| +Platform-1e's work order (2026-10-06): (1) claim Plan 6 Tasks 6 to 11 |
| +built to the signatures below: `startAdmitted`, K6 in the wake |
| +admission, `SelectedTimers` for held rows, `Regional.Run` for |
| +`park-replica` moves scopes; unknown kinds stay refused. (2) Plan 8b: |
| +roster, census, release identity and guard, the termination fence, the |
| +real declare-lost and readmit. (3) Plan 5 Tasks 5 to 9: the move, flip, |
| +commit, rollback and delete transitions, plus the delete's two parks. |
| +Claim rev 6.27 (989ee349) §1.1 records Paul's #236 rulings; P7 |
| +(cold-only) binds every plan here. |
| + |
| +Every task marked **BLOCKED-ON-CLAIM** names one of these contracts and |
| +is not built until the contract is merged on `main`; the moves plans do |
| +not guess the API beyond what is written here. |
| + |
| +| id | contract moves needs | from | blocks | |
| +|---|---|---|---| |
| +| C1 | `(a *Service) startAdmitted(ctx, e *actEntry, admitted hostclaim.Observed, policy startPolicy) error` with `startPolicy{MaxStarts int; Window time.Duration; Deadline time.Time; Probe bool; Prepared *moves.InstallManifest; Inputs *provision.MoveInputs}`: the admitted writer's start of claim rev 6.10 §6.4a (purge, restore of the admitted baseline or the prepared generation's pinned snapshot, start, readiness, first upload, `generation.json` by If-Match or If-None-Match, `PublishGeneration`, open), callable for a **held** row, returning `ErrAttemptOver` when the policy's window or deadline passes before readiness and never marking the host failed when `Probe` is true. Cold only (P7): the sequence never consults a certificate while P7 stands. | claim Plan 6 Tasks 7, 9 (work order 1) | Plan 3 Tasks 5, 6, 7; Plan 4 Task 9 | |
| +| C2 | K6 in `doWake`: a fresh claim read; when `op.exec` does not name this box the host's process tree is killed (`stopevidence.Reconstruct` step 1) before anything else; a wake is admitted under `landing/committed` for the target and under `reverting`/`restoring` phases `probed` and `opened` for the source; the restore writes `<dbPath>.gen` and retires a warm database whose `.gen` names another generation. | claim Plan 6 Task 9 | Plan 3 Task 7 (serving after the open); Plan 4 Tasks 5, 6 (serving after `opened`) | |
| +| C3 | `schedule.SelectedTimers` accepts a held row's write when the selection names the hold's op, and gains `ReplaceTimers(host string, rows []registry.Timer) error` delegating to `registry.ReplaceTimers` under that selection. | claim Plan 6 Task 7 | Plan 2 Task 10 `local` (until merged, `local` calls `registry.ReplaceTimers` directly behind the switch D13) | |
| +| C4 | `hostbox.Regional.Run` runs `park-replica` chores scoped to `moves/<host>/<op id>` and to `instances/<host>/<inc>`, and `park-generation` chores, under move and rollback op ids; its `Leases.Acquire(bucket, location)` **shares the activator's held lease for a location this box itself serves** (an aggregate lease adapter, claim Plan 5 D3's wording), else a landing's residue chores and the at-rest sweep on a serving host can never acknowledge (`park/remote.go:404` acquires the location's lease). | claim Plan 6 Task 10, Plan 5 Task 3 | Plan 5 Tasks 1, 3, 5 acknowledgement after the horizon (the immediate half runs without it) | |
| +| C5 | Production callers for `hostops.Controller.Adopt` (`adoptHostRecord`) on the reconcile pass and for `(s *Server) StartClaimRecoveryPass`: a legacy host gets its claim on the next reconcile pass. | claim Plan 5 integration (work order 3) | the rehearsal (Plan 7 P-5); the request door's "no host claim yet" refusal is correct meanwhile | |
| +| C6 | The loss sequence, end to end: `CreateLossRecord` of a `prepared` record with `WriteSources` composed as **the latest target session as reason `target` and every earlier target session since the cut and, when `src_covered` is false, the source's cut session as further sources** (`lost.go:149-191` requires the frozen set to equal the admission's primary `WritesLost` plus `AdditionalWritesLost`, unique by `(box, generation, session)`, each with a validated baseline); the transition `prepared → committed` with `ResultClaimGen` after `DeclareLost` (`lost.go:119` admits only a `committed` record); `DestroyedStorage` attestations for `DischargeLocal`; `Enrollment` and `QuarantineRevision` as the roster supplies them (Plan 8b), and a fence form of the declaration with neither (region HA L2 to L4). Moves supplies `moves.LossOwner` (`ValidateDeclareLost`, `ValidateLossRecovery`) and the record-side `lost` phase; the claim owns the record's lifecycle and the exact `LossRecord` composition. | claim Plan 8 + 8b (work order 2) | Plan 4 Task 10; the lost-target branch of Plan 4 Task 5; Plan 6 Task 1 `--declare-lost`; Plan 7 Run 4 | |
| +| C7 | The journal key under which a termination fence is cited on the claim (`loss.<id>` with a fence intent, or a `fence` key) so `ReturnToSource` can accept it as evidence (region HA L14). | claim 8b / region HA L2 to L4 | Plan 4 Task 2's fence form | |
| +| C8 | Region HA's files publisher API, `files.Publish(ctx, store lease.VersionedStore, location, generation, dir, envPath string) (manifestVersion string, err error)` or its real name, for `local`'s files manifest (spec rev 7.19 §7.6, L17). | region HA plan 1a | Plan 2 Task 10 `local`'s manifest step (`.files-gen` alone until then) | |
| + |
| ## Decisions locked across the series |
| - **D1 Journals.** The record's `move` field carries the move's phase |
| @@ -116,17 +145,20 @@ additions, `internal/hostexec`, `internal/hostquarantine`, |
| - **D3 Staging-work lease.** `lease.AcquireTTL(ctx, deploymentBucket, "control/staging-leases/"+targetRegion+"/"+host, deploymentRegion, moves.StagingLeaseTTL)` is the claim's shared staging-work lease (§7.3), with `StagingLeaseTTL = 45 * time.Minute` (the ordinary 15-minute `LeaseTTL` cannot leave the signature's 15-minute life plus skew in front of `SafeUntil`). Every staging request goes through `moves.Guarded.Before`, which checks `SafeUntil()` and then calls the SDK in the same goroutine with no await between; the staging S3 client (`lease.NewStagingMirror`) has SDK retries disabled and a 120 s request timeout, so the SDK signs exactly once, immediately after the check, and never re-signs; the lane's own retry loop makes a new check for each new signature. The residual gap, a pause between the SDK's SigV4 computation and the socket write, is the pause assumption the spec names (§7.3, model 14), bounded by the skew allowance. |
| - **D4 Seal versions from the copy responses.** `lease.MirrorPrefix` records each copy's destination `versionId` in the receipts through an optional `lease.VersionedCopier` interface the S3 mirror implements, so the source never needs to read the target's `moves/*` (M17). The seal is the receipts' versions for the current source keys, and a receipt is trusted only when it matches the current source ETag and size **and** the current destination ETag; anything else is recopied, which yields a fresh version id. |
| - **D5 The sealed restore is litestream's own planner.** `moves.SealedClient` implements `litestream.ReplicaClient` over the target bucket's versioned store and serves exactly the sealed `(key, versionId)` pairs; `litestream.NewReplicaWithClient(nil, client).Restore` builds the database (the pattern `internal/lease/lease.go:404` uses). Nothing reads the current listing. |
| -- **D6 Park destinations.** Two new obligation kinds, `park-superseded` (a whole location) and `park-staging` (a `moves/<host>/<op>` lane), both archived under `superseded/<op id>/<host>/<retirement id>/{instances,moves}/`. `internal/park` gains the destination; the engine's manifests, version deletes, horizon and completion are unchanged. |
| +- **D6 Park kinds and destinations (rev 3, after platform-1e's interface check).** No new obligation kind and no new destination. A staging lane is a `park-replica` scoped to `moves/<host>/<op id>` (the grammar claim rev 6.23 §2 allows). The source's finalization park and a rolled-back target's cleaning park are `park-replica` scoped to the whole location `instances/<host>/<inc>`: in that region the location is no longer authoritative, so `park.Engine`'s live-set rule (`remote.go:346`) selects every version. A landing's residue from an earlier stay in the target region is parked as one `park-generation` chore per residue generation (`@root` for the origin), the same decision `park.Decide` makes for the at-rest sweep, because a whole-location park there could never acknowledge once the landing's generation is live. The destination is the engine's `parked/<incarnation>/<host>/<retirement id>/`, and F3 selects a move's retirements by the manifest's `Op`, not by path. Spec §6.4's `superseded/<op id>/...` wording is therefore a spec amendment for rev 7.20 ("the destination is the claim engine's, the op is in the manifest"); the invariants it served, one manifest per retirement event (D5), nothing live overwritten, INV-10, hold. Reason no new kind is needed: nothing in a move reads superseded data back (a dirty return seals the target's live installation before `cleaning` parks; a clean return restores from the source's live replica; F3 deletes after M7's hold, by manifest), so the engine's existing kinds and paths carry every property the spec asked of `superseded/`. |
| - **D7 Reconcile learns holds from the host package.** `provision.ReconcileConfig` gains `Hold func(Record) (status, op string, ok bool)` and `KeepParticipant func(host string) bool`; the host package supplies both from its claim reads. Reconcile itself never reads a claim. |
| - **D8 The console's move pass.** `movesRecoveryPass` (Plan 5) writes the claim half the record decided (§7.5) and runs the deadline sweeps. It is a plain function so claim Plan 5's `Recovery.Pass` can call it. |
| - **D9 `provision.MoveRequest` is replaced by `provision.MoveStart`** in Plan 1, since a move without an id, an incarnation and a cut mode has no journal to run. `MoveComplete` is replaced by `MoveFlip` and `MoveClose`. The old staging-wins landing (`promoteStagedReplica`) is deleted in Plan 2. |
| - **D10 Flat landing only.** Phase 1 lands flat (§6.2, §21); a nested source is refused at the door and at every box step. |
| - **D11 The terminal outcome survives the close.** Every close CAS (`done`, `rolled_back`, `aborted`) writes `Record.LastMove = &MoveOutcome{ID, Outcome, At, OfflineS}` as it clears `Move`, so a CLI that missed the intermediate states reads the outcome of the move id it requested from `GET …/move` (`{"state":"none","last":{…}}`) and exits with the right code. |
| -- **D12 Order within the series.** The copy gate (the record file and the ship check) is Plan 1 Task 15, so every later plan's strings are checked from the day they are written; the paging primitive `host.PageOperator` is Plan 5 Task 0, before the console pass that needs it; `carlos move`'s dispatch and its `docs/site/cli.md` heading land in one task (Plan 6 Task 1), so no commit fails the docsite gate. |
| +- **D12 Order within the series.** The copy gate (the record file and the ship check) is Plan 1 Task 15, so every later plan's strings are checked from the day they are written; the paging primitive `host.PageOperator` is Plan 5 Task 0, before the console pass that needs it; `carlos move`'s dispatch and its `docs/site/cli.md` heading land in one task (Plan 6 Task 1), so no commit fails the docsite gate. `registry.DeleteIfHold` is Plan 1 Task 13's, beside `ReleaseHold`; `hostclaim.ReleaseMove` is Plan 1 Task 3's (Plan 4 widens its legal states); Plan 1 Task 11's host-side report pass is built inside Task 14 (it needs that task's harness). |
| +- **D13 The operator-off switch.** Every move entry point (the request door and its siblings, the box worker's handling of any record with `move`, the console's move pass, `carlos move`) is behind `CARLOS_MOVES_PHASE1=on`, read by the console and the agent, **off by default**. Off: the door answers 409 "moves are not enabled on this deployment", the worker skips every record with a move (one log line a minute), the pass does nothing. The switch is spec §15 step 4's dormancy made concrete: the series merges to `main` task by task without exposing a half-move, and the default stays off until every plan is merged and Paul's rehearsal order begins (Plan 7 P-1 turns it on per deployment). The copy gate ties to it: `TestPendingCopyIsNotWired` becomes `TestPendingCopyKeepsTheSwitchOff`: while any row of the copy record is `pending`, `moves.DefaultEnabled` must be `false`, and no task may flip it. Pending strings may therefore be wired as the tasks are built; nothing ships to an operator until Paul approves the record and the switch default changes in one commit. |
| +- **D14 Cold only (P7, claim rev 6.27 §1.1).** No task passes `Mode: "warm"` or `PredecessorCertified: true` to `AdmitWriter`; no task writes `closed/<session>.json`; `KeepStamp` follows the library's rule unchanged. The budget already prices every attempt as a cold admission (Plan 1 Task 10); the per-wake `park-generation` chore is outside the cut window. |
| ## Global constraints (every plan repeats these) |
| - Gate before every commit: `go build ./...`, `go vet ./...`, `go test ./... -count=1`, `gofmt -l .` clean, `make test`; cross-compile `GOOS=darwin`, `GOOS=windows`, `GOOS=linux GOARCH=arm64` (AGENTS.md "Build & test"). Sandboxed Go: `export GOCACHE=$TMPDIR/gocache GOFLAGS=-buildvcs=false`. |
| +- Cold only (P7, D14) and the operator-off switch (D13) in every plan. |
| - CPU policy: tests serial (`-p 1` when the whole tree runs), niced, no deep simulations. No new sweep of `internal/sim/hostlife` runs in these plans. |
| - Copy: every user-facing string goes through the copy-review gate before it ships; no em dashes anywhere (a full stop and a new sentence instead). Plan 6 Task 7 is the gate. |
| - Nothing rolls to production without Paul's go. Rehearsal order is M12's: the probe app, a disposable instance, kpmg (Plan 7). |
| @@ -175,6 +207,45 @@ fixed: |
| | 31 | the copy check needed an independent inventory | Plan 1 Task 15: `movesCopyIDs` pinned, negative fixtures | |
| | 32 | the clean-rollback rehearsal wrote before refusing | Plan 7 Tasks 1 and 3: the before-write mode suppresses every mutation | |
| +## Review round 2 (Astra, on da2832aa): dispositions (plan rev 3) |
| + |
| +Findings file: `~/.cache/nearest-region/CODEX-PLAN-MOVES-PHASE1-R2.md` |
| +(the coordinator's record; not re-reviewed, per the coordinator). The |
| +locked design stands; the work split into local fixes (rev 3) and |
| +contracts (the BLOCKED-ON-CLAIM table above). |
| + |
| +| # | finding | rev 3 | |
| +|---|---|---| |
| +| 1 | `complete()` needs `BoxesComplete`, which `readEdgesSnapshot` never sets | Plan 1 Task 4: the door checks `snap.Complete` (every edges file read), the evidence the floor needs; liveness is not the floor's question | |
| +| 2 | `MoveStuck` constant and function collide; a two-argument `Revert` call remained | Plan 4 Task 1: the transition is `MoveMarkStuck`; Plan 2 Task 5's call takes the five arguments | |
| +| 3 | `Claim.Validate` rejects the new kinds | D6: no new kinds; `park-replica` scopes and `park-generation` chores; `Engine.Parked` stays | |
| +| 4 | the dependency graph understated the integration | the BLOCKED-ON-CLAIM table and platform-1e's work order; `DeleteIfHold` moved to Plan 1 Task 13 | |
| +| 5 | intermediate merges exposed half-moves | D13: the operator-off switch `CARLOS_MOVES_PHASE1`; the interim abort releases the claim (`ReleaseMove` in Plan 1 Task 3) | |
| +| 6 | return-lane writes outside the lease and signing contract | Plan 4 Task 4: the lane is written under the source region's staging lease through the guarded staging mirror; the bound is the return seal's | |
| +| 7 | `pending` before the PUT defeats the foreign refusal | Plan 1 Task 14: the target validates `move.json` (its own bucket) in T9 and at `checking`; the source's ledger decides only its own replay; a foreign object is caught by the target, never by the source guessing | |
| +| 8 | inventory registration against the real path/stamp contract | Plan 2 Task 10: relative paths; the old stay's entries retired by their own tag before the new installation registers under F; `.gen` included | |
| +| 9 | the landing held the lease the engine acquires | Plan 2 Task 10: the location lease is taken after `parking`; the acknowledgement half is C4 | |
| +| 10 | a whole-location park could not finish once the new generation was live | D6: residue is parked per generation (`park-generation`) | |
| +| 11 | record/claim phase mirroring wedged on a half write; `ExecToNone` replay | Plan 2 Task 10 `syncClaimPhase` catch-up; Plan 2 Task 2 `ExecToNone` idempotent on a matching journal | |
| +| 12 | the finalization stamp could be lost | Plan 3 Task 8: `row-deleted` in the op journal before the delete; the entry condition reads it | |
| +| 13 | dirty-return probes rejected their prepared generation | Plan 3 Task 5: `Op`, `HoldOp` and `PreparedGeneration` distinguished; `PreparedCheck` for every prepared install | |
| +| 14 | the launch-input pin was never enforced at spawn | Plan 3 Task 5 step 2: render, then compare with `move.inputs` (landing) or record `recovery_inputs` (recovery) | |
| +| 15 | declare-lost rewrote the state its legality check read | Plan 4 Tasks 1, 10: a distinct `lost` target phase the ordinary recovery cannot consume | |
| +| 16 | the loss flow cannot pass the real Plan 8 API | C6; Plan 4 Task 10 BLOCKED-ON-CLAIM | |
| +| 17 | case (c) performed regional retirement | Plan 4 Task 10: a separate consequence path | |
| +| 18 | `TargetParkedAt` had no surviving producer after a loss | Plan 4 Task 10 and Plan 5 Task 7: the console pass stamps it when no chore owned by the lost box or its region remains under the op | |
| +| 19 | the readiness window started before the restore | Plan 3 Tasks 1, 6: `Opened` is the first app start, written by the probe; attempts are accounted separately | |
| +| 20 | randomized sealing broke the replay identity | Plan 2 Task 5: the archive identity (manifest, sealed key, temp file) is journaled before any PUT | |
| +| 21 | the converger could write after the barrier | Plan 2 Task 6: a shared `schedule.HostGate` the runner, the converger and the cut all take | |
| +| 22 | T11 used example rates | Plan 1 Task 11: measured rates with a one-hour freshness; missing measurements fail T11 | |
| +| 23 | log keys gave no sequencing guarantee | Plan 6 Task 3: transition-identity keys, create-only | |
| +| 24 | copy approval and prerequisites after their consumers | D13 (the switch ties approval to enablement); `hostguard.Capture` returns `(Captured, error)`; the lease fixture is `leasetest.NewVersionedBucket` + `NewS3Server`; `ProtocolVersion` stub stays in Task 4 | |
| +| 25 | the loss rehearsal lacked its prerequisites | Plan 7 P-6: Plan 8b, C5, the switch | |
| +| R1-6 | the SDK signs at send | Plan 1 Task 6: the staging mirror presigns at the check (`PresignPutObject`, `PresignCopyObject`, the check's clock as `X-Amz-Date`) and sends the presigned request once with no retry | |
| +| R1-8 | `Extract` lacked duplicate-destination rejection | Plan 1 Task 7 | |
| +| R1-11 | unknown-floor participation missed preparing participants | Plan 1 Task 12: a participant is a held row **or** a record whose move names this box | |
| +| R1-27 | Task 7's retry contradicted Task 1 | Plan 4 Task 7 resumes at `StuckAt` | |
| + |
| ## What the model proved and the plans port |
| Witnesses that become real tests, by plan: Plan 2 ports `wSealByVersion`, |
diff --git a/docs/superpowers/plans/2026-10-06-moves-phase1-1-copy-and-staging.md b/docs/superpowers/plans/2026-10-06-moves-phase1-1-copy-and-staging.md| index f5b1258d3..4fad3d74f 100644 |
| --- a/docs/superpowers/plans/2026-10-06-moves-phase1-1-copy-and-staging.md |
| +++ b/docs/superpowers/plans/2026-10-06-moves-phase1-1-copy-and-staging.md |
| @@ -20,6 +20,7 @@ |
| - The readiness bar is 16 MiB (`defaultMoveReadyDelta`), the report freshness bound is 10 minutes, the default cut window is 120 s, `open_budget` is 5 s (§7.3). |
| - The archive cap is 256 MiB enforced while streaming; regular files and directories only (§8.1). |
| - Phase 1 moves one exec-backed hibernating `kind:"instance"` record between non-home placements; everything §7.1 lists is refused at the door and at every box step. |
| +- Cold only (P7, claim rev 6.27 §1.1; index D14): no task here or later passes a warm admission or writes a certificate. The operator-off switch `CARLOS_MOVES_PHASE1` (index D13) gates every entry point from this plan on; its default is off. |
| ## Review Focus |
| @@ -609,7 +610,7 @@ Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| **Interfaces:** |
| - Consumes: `hostclaim.StartGuard`, `RequireOp`, `KindServing`, `KindMoving`, `Op`, `Touched`, `ErrWrongState`, `ErrMalformed`. |
| -- Produces: `hostclaim.BeginMove(opID, by, sourceBox, targetBox, targetRegion string, now time.Time) func(*Claim) error`; `hostclaim.Touch(opID string, regions, boxes []string) func(*Claim) error`; `hostclaim.RequireMoveOp(opID, kind, phase string) func(*Claim) error`; phase constants `PhaseMoving = ""`, `PhaseSealed = "sealed"`. |
| +- Produces: `hostclaim.BeginMove(opID, by, sourceBox, targetBox, targetRegion string, now time.Time) func(*Claim) error`; `hostclaim.Touch(opID string, regions, boxes []string) func(*Claim) error`; `hostclaim.RequireMoveOp(opID, kind, phase string) func(*Claim) error`; phase constants `PhaseMoving = ""`, `PhaseSealed = "sealed"`; `hostclaim.ReleaseMove(opID, execBox string) func(*Claim) error` (K8, in this plan so the interim abort can release: legal from `moving` with phase `""` or `"aborting"` when no obligation under the op remains, sets `Op = Op{Kind: KindServing, Exec: execBox}`; Plan 4 Task 2 widens it to `landing/committed`, `restoring/opened`, `reverting/opened`). Test `TestReleaseMoveFromMovingNeedsNoChores` beside the two below. |
| - [ ] **Step 1: Write the failing test** |
| @@ -936,6 +937,10 @@ func (s *Server) handleAPIInstanceMoveRequest(w http.ResponseWriter, r *http.Req |
| if !ok { |
| return |
| } |
| + if !s.cfg.MovesEnabled { // D13: CARLOS_MOVES_PHASE1=on, off by default |
| + jsonError(w, http.StatusConflict, "moves are not enabled on this deployment") |
| + return |
| + } |
| ctx := r.Context() |
| if rec.Role != "" { |
| jsonError(w, http.StatusBadRequest, "a replica does not move. delete it and create one in the new region") |
| @@ -1052,10 +1057,23 @@ func (s *Server) handleAPIInstanceMoveRequest(w http.ResponseWriter, r *http.Req |
| // be read may be below the floor, so the door refuses (complete == false). |
| func moveTargetBox(ctx context.Context, s release.Store, to, from string) (box string, floorOK, complete bool) { |
| snap, err := readEdgesSnapshot(ctx, s) |
| - if err != nil || !snap.complete() { |
| + // snap.Complete: every edges file listed was read and decoded. That is |
| + // the evidence the floor needs. snap.complete() also wants |
| + // BoxesComplete, which only readLiveEdgesSnapshot computes (liveness, |
| + // instancesplacement.go:103-130), and a floor decision has no liveness |
| + // question, so it is not used here. |
| + if err != nil || !snap.Complete { |
| return "", false, false |
| } |
| floorOK, complete = true, true |
| + // The target box in a multi-box region is region HA's choose() (spec |
| + // rev 7.19 §7.1, L15): prefer a box running no instance of the app, |
| + // then the most headroom (fewest claimed hosts), then the SHA-256 |
| + // tie-break. Until region HA's eligibleBoxes is exported, one box per |
| + // region is the deployment's shape and the rule degenerates to "the |
| + // only box"; the lexically-first fallback below is kept for a region |
| + // that already has two, and TestMoveTargetBoxPrefersChoose pins the |
| + // choose() order once the helper exists. |
| for _, f := range snap.Files { |
| if f.Placement != to && f.Placement != from { |
| continue |
| @@ -1313,7 +1331,7 @@ Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| - Create: `internal/moves/lease.go`, `internal/moves/lease_test.go` |
| **Interfaces:** |
| -- Consumes: `lease.acquireTTL(ctx, bucket, prefix, region, ttl)` (`lease.go:129`, exported in this task as `lease.AcquireTTL`), `(*lease.Held).SafeUntil() (time.Time, error)`, `Release`; `lease.NewS3Mirror` (`mirror.go:388`), extended in this task with `lease.NewStagingMirror(ctx, srcRegion, dstRegion string) (MirrorStore, error)`: the same client pair built with `o.RetryMaxAttempts = 1` (no SDK retries, so a request is signed once) and an `http.Client{Timeout: 120 * time.Second}` (the request limit), so a signature is computed immediately after the caller's check and never recomputed. |
| +- Consumes: `lease.acquireTTL(ctx, bucket, prefix, region, ttl)` (`lease.go:129`, exported in this task as `lease.AcquireTTL`), `(*lease.Held).SafeUntil() (time.Time, error)`, `Release`; `lease.NewS3Mirror` (`mirror.go:388`), extended in this task with `lease.NewStagingMirror(ctx, srcRegion, dstRegion string) (StagingMirror, error)`: a `MirrorStore` plus `VersionedCopier`/`VersionedPutter` whose every mutation is **presigned at the call** and sent once. `CopyVersioned` builds the request with `s3.NewPresignClient(client).PresignCopyObject(ctx, in, s3.WithPresignClientFromClientOptions(...), func(o *s3.PresignOptions){ o.Expires = 15 * time.Minute })` using `o.ClientOptions` with `aws.Config.Retryer = aws.NopRetryer{}`, so `X-Amz-Date` is the call's clock and the signature exists before the send; the presigned URL is sent with one `http.Client{Timeout: 120 * time.Second}` `Do`, no retry; the copy's `x-amz-version-id` response header is the version id. `PutVersioned` presigns `PutObject` the same way with the body's SHA-256 in the signed headers. `Delete` presigns `DeleteObject`. A caller that wants a retry calls again, through a fresh `Guarded.Before`, which signs again at that check. This is §7.3's "the signature is computed at that check, with the check's time as its `X-Amz-Date`, and never recomputed" made literal; the only pause the rule cannot cover is between `Before`'s clock read and the presigner's clock read, two statements apart in one goroutine, which the skew allowance covers. |
| - Produces: `moves.StagingLeaseTTL = 45 * time.Minute` (D3: the ordinary 15-minute `LeaseTTL` cannot leave the signature's life in front of `SafeUntil`); `moves.StagingLeasePrefix(targetRegion, host string) = "control/staging-leases/"+targetRegion+"/"+host`; `moves.SafeUntilGuard interface{ SafeUntil() (time.Time, error) }`; `moves.Guarded{Lease SafeUntilGuard; Now func() time.Time; Skew time.Duration}` with `(g Guarded) Before(fn func() error) error`: `lease.Held.SafeUntil` already subtracts the renewal margin; the §7.3 rule subtracts the signature's whole life, so `Before` refuses with `moves.ErrLeaseUnsafe` when `Now().Add(SignatureLife + Skew).After(safe)` (`SignatureLife = 15 * time.Minute`, `Skew` default 1 min) and otherwise calls `fn` at once in the same goroutine. The lane's retry loop (Task 14) calls `Before` again for each retry; nothing retries inside `fn`. |
| - [ ] **Step 1: Write the failing test** |
| @@ -1355,13 +1373,21 @@ func TestGuardedRefusesWithinTheSignatureLife(t *testing.T) { |
| } |
| } |
| -// With the lease package's local test leaser (internal/lease/leasetest), a |
| -// lease acquired with StagingLeaseTTL passes the guard right after |
| -// acquisition and a lease acquired with lease.LeaseTTL does not. |
| +// Against a local S3 (internal/lease/leasetest: NewVersionedBucket and |
| +// NewS3Server, the package's own fixture), a lease acquired with |
| +// StagingLeaseTTL passes the guard right after acquisition and a lease |
| +// acquired with lease.LeaseTTL does not. The client is pointed at the |
| +// server the way lease's own tests do (read lease_test.go for the |
| +// endpoint override the package uses; use that seam, not a new one). |
| func TestGuardAgainstRealLeaseSettings(t *testing.T) { |
| ctx := context.Background() |
| - store, bucket, region := leasetest.LocalStore(t) // the package's existing local S3-compatible fixture; name per leasetest |
| - held, err := lease.AcquireTTL(ctx, bucket, "control/staging-leases/uk/h", region, StagingLeaseTTL) |
| + bucket := leasetest.NewVersionedBucket() |
| + handler, port := leasetest.NewS3Server(bucket, "data-ie") |
| + srv := httptest.NewServer(handler) |
| + defer srv.Close() |
| + _ = port |
| + region := lease.TestEndpoint(t, srv.URL) // the package's endpoint seam for tests; name per lease_test.go |
| + held, err := lease.AcquireTTL(ctx, "data-ie", "control/staging-leases/uk/h", region, StagingLeaseTTL) |
| if err != nil { |
| t.Fatal(err) |
| } |
| @@ -1369,7 +1395,7 @@ func TestGuardAgainstRealLeaseSettings(t *testing.T) { |
| if err := (Guarded{Lease: held, Now: time.Now}).Before(func() error { return nil }); err != nil { |
| t.Fatalf("staging lease refused at acquisition: %v", err) |
| } |
| - short, err := lease.AcquireTTL(ctx, bucket, "control/staging-leases/uk/h2", region, lease.LeaseTTL) |
| + short, err := lease.AcquireTTL(ctx, "data-ie", "control/staging-leases/uk/h2", region, lease.LeaseTTL) |
| if err != nil { |
| t.Fatal(err) |
| } |
| @@ -1377,7 +1403,34 @@ func TestGuardAgainstRealLeaseSettings(t *testing.T) { |
| if err := (Guarded{Lease: short, Now: time.Now}).Before(func() error { return nil }); !errors.Is(err, ErrLeaseUnsafe) { |
| t.Fatalf("a 15 minute lease passed the guard: %v", err) |
| } |
| - _ = store |
| +} |
| + |
| +// The staging mirror signs at the check and never again: a presigned |
| +// request carries the X-Amz-Date of the moment it was built, and a second |
| +// attempt is a second Before. |
| +func TestStagingMirrorPresignsAtTheCheck(t *testing.T) { |
| + bucket := leasetest.NewVersionedBucket() |
| + handler, _ := leasetest.NewS3Server(bucket, "data-uk") |
| + var dates []string |
| + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { |
| + dates = append(dates, r.URL.Query().Get("X-Amz-Date")) |
| + handler.ServeHTTP(w, r) |
| + })) |
| + defer srv.Close() |
| + m := lease.NewStagingMirrorAt(t, srv.URL) // the test constructor the task adds beside NewStagingMirror |
| + t0 := time.Date(2026, 10, 6, 9, 0, 0, 0, time.UTC) |
| + clock := t0 |
| + m.Clock = func() time.Time { return clock } |
| + if _, err := m.PutVersioned(context.Background(), "data-uk", "moves/h/mv/timers/t.json", []byte("{}")); err != nil { |
| + t.Fatal(err) |
| + } |
| + clock = t0.Add(3 * time.Minute) |
| + if _, err := m.PutVersioned(context.Background(), "data-uk", "moves/h/mv/timers/t.json", []byte("{}")); err != nil { |
| + t.Fatal(err) |
| + } |
| + if len(dates) != 2 || dates[0] != "20261006T090000Z" || dates[1] != "20261006T090300Z" { |
| + t.Fatalf("X-Amz-Date per request: %v", dates) |
| + } |
| } |
| ``` |
| @@ -1552,6 +1605,24 @@ func TestArchiveRefusesSpecialFiles(t *testing.T) { |
| } |
| } |
| +// A crafted archive naming one destination twice (two "env" entries, or |
| +// "files/a" and "files/./a") is refused before any byte is written. |
| +func TestExtractRefusesDuplicateDestinations(t *testing.T) { |
| + var buf bytes.Buffer |
| + zw, _ := zstd.NewWriter(&buf) |
| + tw := tar.NewWriter(zw) |
| + for _, name := range []string{"env", "env"} { |
| + tw.WriteHeader(&tar.Header{Name: name, Typeflag: tar.TypeReg, Size: 1, Mode: 0o600}) |
| + tw.Write([]byte("x")) |
| + } |
| + tw.Close() |
| + zw.Close() |
| + out := t.TempDir() |
| + if err := Extract(bytes.NewReader(buf.Bytes()), out, filepath.Join(out, "h.env")); !errors.Is(err, ErrBadPath) { |
| + t.Fatalf("duplicate env: %v", err) |
| + } |
| +} |
| + |
| func TestArchiveCapIsEnforcedWhileStreaming(t *testing.T) { |
| dir, env := filesFixture(t) |
| f, _ := os.Create(filepath.Join(dir, "big")) |
| @@ -1840,7 +1911,8 @@ func PlainDigest(filesDir, envPath string) (plain string, bytes int64, entries i |
| } |
| // Extract writes an opened archive into filesDir and envPath, refusing |
| -// anything a tar could smuggle: absolute paths, "..", links and devices. |
| +// anything a tar could smuggle: absolute paths, "..", links and devices, |
| +// and two entries naming one destination (a replayed or crafted archive). |
| func Extract(r io.Reader, filesDir, envPath string) error { |
| zr, err := zstd.NewReader(r) |
| if err != nil { |
| @@ -1848,6 +1920,7 @@ func Extract(r io.Reader, filesDir, envPath string) error { |
| } |
| defer zr.Close() |
| tr := tar.NewReader(zr) |
| + seen := map[string]bool{} |
| for { |
| hdr, err := tr.Next() |
| if err == io.EOF { |
| @@ -1872,6 +1945,10 @@ func Extract(r io.Reader, filesDir, envPath string) error { |
| default: |
| return fmt.Errorf("%w: %q is outside files/ and env", ErrBadPath, hdr.Name) |
| } |
| + if seen[dst] { |
| + return fmt.Errorf("%w: %q named twice", ErrBadPath, hdr.Name) |
| + } |
| + seen[dst] = true |
| switch hdr.Typeflag { |
| case tar.TypeDir: |
| if err := os.MkdirAll(dst, fs.FileMode(hdr.Mode).Perm()|0o700); err != nil { |
| @@ -2556,7 +2633,7 @@ Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| **Interfaces:** |
| - Produces (pure): `moves.PutReplace(ctx, s release.Store, key string, body []byte) error`: `release.Store.Put` with an empty `ifMatch` is create-only (`store.go:38`), so a recurring object (a report, the console record) is written by `Get` for the current etag (absent → `""`), then `Put(key, body, etag)`, retrying once on `release.ErrCASConflict`; every mutable object the series writes to the deployment bucket goes through it (reports here, the console record in Task 12, the hold view's receipts ledger when it lives in a `release.Store`); `moves.Check{OK bool; Reason string}`; `moves.TargetReport{V int; Box, Region string; MoveID string; At time.Time; Checks map[string]Check; Inputs provision.MoveInputs; AttemptsAdmitted int; CutBudgetS int}`; check ids `T1 … T12` as constants `CheckProtocol = "T1"` etc.; `moves.SourceReport{V int; Box string; MoveID string; At time.Time; Env EnvReport; S1, S2 Check; FilesPlain string}`; `moves.ReportFreshness = 10 * time.Minute`; `moves.Ready(roundDelta int64, bar int64, target *TargetReport, source *SourceReport, now time.Time) (ok bool, reason string)`; `(r TargetReport) Passing() (bool, string)`. |
| -- Produces (host): `(w *movesWorker) writeSourceReport(ctx, s, rec, env map[string]string, filesPlain string) error`; `(w *movesWorker) targetReportPass(ctx, s release.Store, recs []provision.Record) error` run by the target region's box for every record whose `Move.To` is its placement: computes T1 (own `moves_protocol` vs floor), T2 (own `secrets_key_fp`, `appconfig.KeyFingerprint` of `CARLOS_SECRETS_KEY`'s public half, equals the console's `SecretsKeyFP` in `moves.ConsoleKey` = `control/moves-console.json`, which Task 12 has the console write at startup; no such object yet on main), T5 (the newest archive manifest under `moves/<host>/<id>/files/` opens with this box's key: list, read manifest, `archive.Open` to `io.Discard`, compare `plain`), T6 (`diskBytes(dataDir())`: free ≥ 2×replica + archive + 2 GiB and resulting use < 85%), T8 (one hosting box publishes the target placement), T9 (`hostclaim.Read`: `KindMoving`, id, holder incarnation = `move.json`'s; this region and box in `touched`), T10 (`PutIfAbsent`/`DeleteVersion` of `moves/_lane/<id>` in the source bucket through `lease.VersionedStore`), T11 (`moves.Estimate` with measured rates: `Rates` read from `CARLOS_MOVE_RATES="cross,s3,disk"` env, default `ExampleRates`, recorded in the report), T12 (`claim.QuietAfter[region]` passed), T3 (compose the flat held row's environment: `appconfig.LoadMergedHost` for the record's environment rendered to a temp config file, the carried `<host>.env` from the newest archive, the agent-owned variables, this process's inherited variables; `moves.CompareEnv`), T4 (for each store binding in the composed environment per `objstore.EnvNames`: list one key, put/get/delete `.carlos-move-probe/<move id>` through the S3 client of `internal/objstore`'s existing probe helper if one exists, else a minimal AWS SDK client), T7 (`release.MaterializeHeld`, Plan 2 Task 7: until it lands, T7 checks the pinned release's manifest exists in the store at `release.ManifestKey(acct, app, version)`); writes `TargetReportKey` with `Inputs` and `AttemptsAdmitted`. |
| +- Produces (host): `(w *movesWorker) writeSourceReport(ctx, s, rec, env map[string]string, filesPlain string) error`; `(w *movesWorker) targetReportPass(ctx, s release.Store, recs []provision.Record) error` run by the target region's box for every record whose `Move.To` is its placement: computes T1 (own `moves_protocol` vs floor), T2 (own `secrets_key_fp`, `appconfig.KeyFingerprint` of `CARLOS_SECRETS_KEY`'s public half, equals the console's `SecretsKeyFP` in `moves.ConsoleKey` = `control/moves-console.json`, which Task 12 has the console write at startup; no such object yet on main), T5 (the newest archive manifest under `moves/<host>/<id>/files/` opens with this box's key: list, read manifest, `archive.Open` to `io.Discard`, compare `plain`), T6 (`diskBytes(dataDir())`: free ≥ 2×replica + archive + 2 GiB and resulting use < 85%), T8 (one hosting box publishes the target placement), T9 (`hostclaim.Read`: `KindMoving`, id, holder incarnation = `move.json`'s; this region and box in `touched`), T10 (`PutIfAbsent`/`DeleteVersion` of `moves/_lane/<id>` in the source bucket through `lease.VersionedStore`), T11 (`moves.Estimate` with **measured** rates, never the worked example: `moves.Rates` come from `moves.Measurements{Cross, S3, Disk MeasuredRate}` with `MeasuredRate{MiBPerS float64; At time.Time; Bytes int64}`; the target measures regional S3 by putting and getting a 16 MiB object under `moves/<host>/<id>/probe/` in its own bucket once per report and the disk by writing and reading a 64 MiB file under `<DataDir>/.moves/` with `O_DIRECT` where available, fsync otherwise; the cross-region rate comes from the source's receipts ledger, which Task 14 extends with `LastRoundBytes` and `LastRoundSeconds` of its most recent copying round (copied through the target bucket's region, so it is the path the cut uses), published in the source report as `source.cross_rate`; a measurement older than one hour, or missing, fails T11 with "no fresh rate measurement for <which>"; `CARLOS_MOVE_RATES` exists only as a test override and is refused outside tests (`testing.Testing()`)), T12 (`claim.QuietAfter[region]` passed), T3 (compose the flat held row's environment: `appconfig.LoadMergedHost` for the record's environment rendered to a temp config file, the carried `<host>.env` from the newest archive, the agent-owned variables, this process's inherited variables; `moves.CompareEnv`), T4 (for each store binding in the composed environment per `objstore.EnvNames`: list one key, put/get/delete `.carlos-move-probe/<move id>` through the S3 client of `internal/objstore`'s existing probe helper if one exists, else a minimal AWS SDK client), T7 (`release.MaterializeHeld`, Plan 2 Task 7: until it lands, T7 checks the pinned release's manifest exists in the store at `release.ManifestKey(acct, app, version)`); writes `TargetReportKey` with `Inputs`, `AttemptsAdmitted` and the measurements. The host-side pass and its test are **written in Task 14** (they need its harness; D12); this task's deliverable is the pure package and `moves.PutReplace`. |
| Test (pure) `internal/moves/readiness_test.go`: |
| @@ -2745,7 +2822,7 @@ Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| **Interfaces:** |
| - Produces: `moves.ProtocolVersion = 1`; `moves.FloorKey = "control/moves-protocol.json"`; `moves.Floor{Min int; MinRelease string}`; `moves.ReadFloor(ctx, s release.Store) (*Floor, error)` (absent → nil, nil); `moves.ReleaseAtOrAbove(version, minRelease string) (bool, error)` using `release.CompareVersions` if it exists (grep `func Compare` in `internal/release`), else `golang.org/x/mod/semver` on `v`-prefixed strings; `moves.ConsoleKey = "control/moves-console.json"`, `moves.ConsoleRecord{V int; Protocol int; SecretsKeyFP string; At time.Time}`. |
| - `edgesFile.MovesProtocol int \`json:"moves_protocol"\`` (not omitempty: absent means a box below any floor). |
| -- `systemApplier` refuses to activate a build below `Floor.MinRelease` in `install`, `rollback` and `recoverIfNeeded`'s restore (`restoreFiles`). The floor is **armed locally**: every successful read of `control/moves-protocol.json` (the agent's edges-file loop once an hour, and `carlos system apply` before it stages) writes `<optDir>/moves-floor.json` durably (temp file, fsync, rename); activation reads the local copy first and the store second. Boot recovery runs before networking (`systemapply.go:109`), so the local copy is what it has. With a floor in hand, below it refuses. With **no** floor anywhere: when the box's registry (`/data/registry.db`, the agent's path) holds any row with `hold_op != ""`, the box is a participant and activation refuses ("the moves floor is unknown and this box holds an operation"); otherwise it proceeds with a log line (a box holding no operation cannot harm a move). The floor is never deleted locally once armed. |
| +- `systemApplier` refuses to activate a build below `Floor.MinRelease` in `install`, `rollback` and `recoverIfNeeded`'s restore (`restoreFiles`). The floor is **armed locally**: every successful read of `control/moves-protocol.json` (the agent's edges-file loop once an hour, and `carlos system apply` before it stages) writes `<optDir>/moves-floor.json` durably (temp file, fsync, rename); activation reads the local copy first and the store second. Boot recovery runs before networking (`systemapply.go:109`), so the local copy is what it has. With a floor in hand, below it refuses. With **no** floor anywhere: the box is a participant when its registry (`/data/registry.db`, the agent's path) holds any row with `hold_op != ""` **or** the hold view's last written snapshot (`<DataDir>/.moves/participants.json`, written by every moves pass with the hosts whose move names this box as source or target, `preparing` included) names any host; then activation refuses ("the moves floor is unknown and this box holds an operation"); otherwise it proceeds with a log line (a box holding no operation cannot harm a move). The floor is never deleted locally once armed. |
| - [ ] **Step 1: Write the failing tests** |
| @@ -2839,7 +2916,7 @@ Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| - Test: `internal/registry/hold_test.go` |
| **Interfaces:** |
| -- Produces: `Route.HoldOp string`; `(r *Registry) SetHold(host, status, op string) error` (status one of `moving`, `landing`, `restoring`, `releasing`; refuses others); `(r *Registry) ReleaseHold(host, op string) (bool, error)` (`UPDATE routes SET status='ready', hold_op='' WHERE host=? AND hold_op=?`, true when a row changed); `registry.HoldStatuses = map[string]bool{...}`; `registry.IsHold(status string) bool`. |
| +- Produces: `Route.HoldOp string`; `(r *Registry) SetHold(host, status, op string) error` (status one of `moving`, `landing`, `restoring`, `releasing`; refuses others); `(r *Registry) ReleaseHold(host, op string) (bool, error)` (`UPDATE routes SET status='ready', hold_op='' WHERE host=? AND hold_op=?`, true when a row changed); `(r *Registry) DeleteIfHold(host, op string) (bool, error)` (`DELETE FROM routes WHERE host=? AND hold_op=?`; the source's row deletion at finalization, §7.9 step 3; here rather than Plan 3 so Plan 3 Task 8 does not depend on Task 7, D12); `registry.HoldStatuses = map[string]bool{...}`; `registry.IsHold(status string) bool`. |
| - [ ] **Step 1: Write the failing test** |
| @@ -2943,10 +3020,12 @@ Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| **Interfaces:** |
| - `moveCfg` gains: `Box string` (the box label `metricsBoxLabel` gives), `DeployBucket, DeployBucketRegion string` (from `CARLOS_DEPLOYMENT_BUCKET` and `CARLOS_AWS_REGION`/the IMDS region the edges loop resolves; empty in dir-store deployments, where the lease falls back to a `lease.Acquire` on the own bucket under the same prefix), `Versioned func(ctx, bucket, region string) (lease.VersionedStore, error)`, `StagingLease func(ctx, targetRegion, host string) (moves.SafeUntilGuard, func(context.Context) error, error)`, `SecretsKey string` (`CARLOS_SECRETS_KEY`), `SecretsPub *ecdh.PublicKey`, `LaunchEnv func(host string) (map[string]string, error)`, `Timers func(host string) ([]registry.Timer, error)`, `FilesDir func(host string) (dir, envPath string)` (flat: `<DataDir>/<host>/`, `<DataDir>/<host>.env`), `ReadyProbe func(ctx, host string) moves.Check` (S1: `GET <ready path>` on the instance's socket through the activator's own request path; the activator's `ServeHTTP` is the edge's, so the probe dials the socket directly with `Host: <host>` and a 30 s timeout), `StrictNext func(host string) moves.Check` (S2: the activator reports whether the next spawn is strict; until Plan 2's strict-spawn task lands this returns OK when `cgroup` reports delegation). |
| -- The round, per in-flight record this box owns, in order (§7.3, K5, K12): refuse a legacy record (log once a minute, `return nil`); refuse a nested row (`rt.DBPath != provision.DefaultDBPath(dataDir, host)`: "nested instances cannot move yet"); refuse a source with `<DataDir>/<host>-sidecar.env` present or any refused registry column set on its row (`Caps`, `Unit`, `Backing`, `Platform`, `Tunnel`, `FleetBox`, `Pool`, `Canary`, `Addr`, `Addrs`, `Health`, `Replica`, `Replicas`; §6.2, §7.1), writing the reason into the source report's `S1` so the operator sees it in `carlos move status`; read the claim; `hostclaim.RequireMoveOp(id, KindMoving, "")` and holder incarnation = `rec.Move.Incarnation`; if the target region or the target box (`Op.Intent["target"]`) is not in `Touched`, `hostclaim.Update(... Touch(id, []string{to}, []string{targetBox}))` (K12 and T9 both hold from the first staging write; the target's `checking` re-adds its box idempotently); take the staging lease (`lease.AcquireTTL(..., moves.StagingLeaseTTL)`); `move.json`: the ledger (own bucket) first records `move_json: "pending"`, then `PutIfAbsent(move.json)` through `Versioned(target bucket)`; success records the returned version; `lease.ErrExists` with a `pending` entry is this op's own earlier PUT whose ledger write was lost (the lane has one writer, this op's source under the staging lease, and the id is 128 random bits), recorded as `move_json: "present"`; `ErrExists` with **no** ledger entry refuses the round ("a staging lane for this move id already exists"; Review Focus 2). The source never reads the target object (M17); the target validates its incarnation itself, in T9 and at `checking`; `MirrorPrefix(own replica prefix → ReplicaPrefix)` through the staging mirror (`lease.NewStagingMirror`) wrapped in a `guardedMirror` whose `CopyVersioned`, `PutVersioned` and `Delete` each call `Guarded.Before` around exactly one SDK call, and whose caller retries a failed call only by calling again (a new check, a new signature); compute `archive.PlainDigest`; when it differs from the receipts' recorded plain, `archive.Build` + `archive.Seal` to a temp file, `PutVersioned` to `FilesPrefix+"files-<sealed>.sealed"` and `.json` (manifest), record in receipts; write the source report with `LaunchEnv` → `moves.ReportEnv`, S1 and S2; read the target report; `moves.Ready(round.DeltaBytes(), cfg.ReadyDelta, target, source, now)`; on ready from preparing, `MovePromoteReady` and `nudgeSerial`; in `ready`, keep mirroring. For `cut == "auto"` and `ready`, the source itself CASes `MoveMarkCutting` (§7.4 "cut door or the source (auto-cut)"); Plan 2 picks it up. |
| +- The switch (D13): `moveCfg.Enabled bool` from `CARLOS_MOVES_PHASE1=on`; when off, every record with `Move != nil` is skipped with one log line a minute per host and nothing below runs; `armMovesPass` reads it once. |
| +- The interim abort (until Plan 5 Task 1 replaces it): for `aborting`, the source deletes its own `move-snapshots/<host>/<id>/` and receipts, then `hostclaim.Update(ReleaseMove(id, cfg.Box))` (Task 3), then `MoveAbortComplete`; the staging lane in the target bucket is left for Plan 5's target-owned park (the source deletes nothing there: M17 allows it, INV-8 does not want it). |
| +- The round, per in-flight record this box owns, in order (§7.3, K5, K12): refuse a legacy record (log once a minute, `return nil`); refuse a nested row (`rt.DBPath != provision.DefaultDBPath(dataDir, host)`: "nested instances cannot move yet"); refuse a source with `<DataDir>/<host>-sidecar.env` present or any refused registry column set on its row (`Caps`, `Unit`, `Backing`, `Platform`, `Tunnel`, `FleetBox`, `Pool`, `Canary`, `Addr`, `Addrs`, `Health`, `Replica`, `Replicas`; §6.2, §7.1), writing the reason into the source report's `S1` so the operator sees it in `carlos move status`; read the claim; `hostclaim.RequireMoveOp(id, KindMoving, "")` and holder incarnation = `rec.Move.Incarnation`; if the target region or the target box (`Op.Intent["target"]`) is not in `Touched`, `hostclaim.Update(... Touch(id, []string{to}, []string{targetBox}))` (K12 and T9 both hold from the first staging write; the target's `checking` re-adds its box idempotently); take the staging lease (`lease.AcquireTTL(..., moves.StagingLeaseTTL)`); `move.json`: `PutIfAbsent(move.json)` through `Versioned(target bucket)`; success records the returned version in the ledger; `lease.ErrExists` records `move_json: "present"` and the round goes on. The source cannot tell its own lost-response PUT from a foreign object (M17 forbids the read, and a ledger intent written first cannot tell them apart either: round-2 finding 7), so **the target owns that validation**: T9 reads `moves/<host>/<id>/move.json` in its own bucket and requires `move_id == rec.Move.ID`, `incarnation == holder.incarnation`, `host`, `from`, `to` all equal to the record's; a mismatch fails T9 with "the staging lane holds another move's move.json" and the move never becomes ready (the operator aborts; the target's park retires the lane); `checking` repeats the same check before any local write. The id is 128 random bits, so the case is a replayed request or a bug, never a collision in practice; `MirrorPrefix(own replica prefix → ReplicaPrefix)` through the staging mirror (`lease.NewStagingMirror`) wrapped in a `guardedMirror` whose `CopyVersioned`, `PutVersioned` and `Delete` each call `Guarded.Before` around exactly one SDK call, and whose caller retries a failed call only by calling again (a new check, a new signature); compute `archive.PlainDigest`; when it differs from the receipts' recorded plain, `archive.Build` + `archive.Seal` to a temp file, `PutVersioned` to `FilesPrefix+"files-<sealed>.sealed"` and `.json` (manifest), record in receipts; write the source report with `LaunchEnv` → `moves.ReportEnv`, S1 and S2; read the target report; `moves.Ready(round.DeltaBytes(), cfg.ReadyDelta, target, source, now)`; on ready from preparing, `MovePromoteReady` and `nudgeSerial`; in `ready`, keep mirroring. For `cut == "auto"` and `ready`, the source itself CASes `MoveMarkCutting` (§7.4 "cut door or the source (auto-cut)"); Plan 2 picks it up. |
| - Receipts: `lease.Receipts{Bucket: cfg.OwnBucket, Key: moves.ReceiptsKey(host, id)}`; a per-op `moveLedger` JSON beside it at `move-receipts/<host>/<id>-moves.json` records `move_json_version`, `archive_plain`, `archive_sealed`, `archive_versions`. |
| -Tests: rewrite `TestMovesPassPrepareAndPromote` for the new layout (keys under `moves/<host>/<id>/replica/`; `move.json` present; the round is not promoted until a passing fresh `target.json` exists in the harness store); add `TestStagingMoveJSONIsCreateOnly` (Review Focus 2: a `move.json` already there and no ledger entry → the round returns an error naming it and copies nothing), `TestStagingMoveJSONReplayAfterLostLedger` (Review Focus 2: the ledger says `pending`, the fake versioned store answers `ErrExists` and refuses every `GetVersion`/`OpenVersion` of the target lane with an access-denied error; the round proceeds and records `present`), `TestPreparingRefusesLegacyNestedAndRefusedColumns`, `TestPreparingTouchesTheTargetRegionAndBoxFirst` (the claim's `touched.regions` gains `uk` and `touched.boxes` gains `uk-box` before the fake mirror sees a copy, from a claim that never touched either), `TestStagingWriterRetriesWithANewCheck` (the fake mirror fails one copy; the lane calls `Before` again before the retry; a `Now` past the bound makes the retry `ErrLeaseUnsafe` and the round ends without a second SDK call), `TestSourceReportIsWrittenEachRound` (two rounds, the second's report replaces the first: `PutReplace`), `TestArchiveIsRebuiltOnlyWhenPlainChanges`, `TestAutoCutMarksCuttingFromReady`. |
| +Tests: rewrite `TestMovesPassPrepareAndPromote` for the new layout (keys under `moves/<host>/<id>/replica/`; `move.json` present; the round is not promoted until a passing fresh `target.json` exists in the harness store); add `TestStagingMoveJSONIsCreateOnly` (Review Focus 2: a `move.json` already there and no ledger entry → the round returns an error naming it and copies nothing), `TestStagingMoveJSONReplayAfterLostLedger` (Review Focus 2: the fake versioned store answers `ErrExists` and refuses every `GetVersion`/`OpenVersion` of the target lane with an access-denied error; the round proceeds and records `present`; the target-side `TestTargetReportPassFlagsEachCheck` has the matching case: a `move.json` naming another move id fails T9 with the lane sentence), `TestMovesPassIsInertWhenTheSwitchIsOff` (D13), `TestInterimAbortReleasesTheClaim`, `TestPreparingRefusesLegacyNestedAndRefusedColumns`, `TestPreparingTouchesTheTargetRegionAndBoxFirst` (the claim's `touched.regions` gains `uk` and `touched.boxes` gains `uk-box` before the fake mirror sees a copy, from a claim that never touched either), `TestStagingWriterRetriesWithANewCheck` (the fake mirror fails one copy; the lane calls `Before` again before the retry; a `Now` past the bound makes the retry `ErrLeaseUnsafe` and the round ends without a second SDK call), `TestSourceReportIsWrittenEachRound` (two rounds, the second's report replaces the first: `PutReplace`), `TestArchiveIsRebuiltOnlyWhenPlainChanges`, `TestAutoCutMarksCuttingFromReady`. |
| - [ ] **Step 1: Write the failing tests**, **Step 2: run**, **Step 3: implement**, **Step 4: run** `nice -n 10 go test ./internal/host -run 'TestMovesPass|TestStaging|TestPreparing|TestSourceReport|TestArchiveIsRebuilt|TestAutoCut' -count=1`. |
| @@ -2969,8 +3048,8 @@ The gate exists from this plan on, so every later plan's strings are checked the |
| - `movesCopyIDs` is a literal in the test file: `M-C1` to `M-C20` and every `MV-N<n>` id the plans introduce (each plan's "Gate and copy" task appends to this literal as it appends rows to the record). `TestMovesCopyRecordIsComplete` fails when the record lacks an id the literal names, or names an id the literal does not. |
| - `TestMovesApprovedCopyIsApplied`: every `approved` row's text appears byte for byte in the file the row names (as the existing test checks, through string literals for short texts). |
| -- `TestPendingCopyIsNotWired`: for every `pending` row, its text (and the text with its `<placeholder>` segments replaced by `%s`/`%d`/`%v` verbs, so a `fmt.Sprintf` form is caught too) does not appear as a string literal in any non-test Go file in a call to `jsonError`, `fmt.Fprint*(os.Stdout|os.Stderr, …)`, `fmt.Print*`, `emitEvent`, `PageOperator`, `log.Fatal*`, or as the argument of a helper whose name ends in `Sentence`, `Line` or `Message`; the check walks the file's AST with `go/ast`, finds call expressions whose callee matches, and inspects every string literal argument and every identifier bound to a string literal in the same file. |
| -- Negative fixtures under `testdata/moves-copy/`: a record missing an id, a row whose text changed after approval, a pending text reached through a `Sprintf` helper; each is loaded by a table test that expects the corresponding failure. |
| +- `TestPendingCopyKeepsTheSwitchOff` (D13): while any row of the record is `pending`, `moves.DefaultEnabled` (the compiled default of `CARLOS_MOVES_PHASE1`) must be `false`, so no pending sentence can reach an operator through a default deployment; the day Paul approves the record, one commit flips the default and this test's condition inverts (every row `approved`). Pending strings may be wired as the tasks are built; the gate is on enablement, not on each call site. |
| +- Negative fixtures under `testdata/moves-copy/`: a record missing an id, a row whose text changed after approval, a record with a pending row beside a `DefaultEnabled = true`; each is loaded by a table test that expects the corresponding failure. |
| - [ ] Write the tests, run them red (the record has only pending rows; the three negative fixtures fail as expected), implement, run green, commit: `git commit -m "copy gate for the moves series: the pinned inventory, the ship check (§20)" ...` |
diff --git a/docs/superpowers/plans/2026-10-06-moves-phase1-2-cut-flip-landing.md b/docs/superpowers/plans/2026-10-06-moves-phase1-2-cut-flip-landing.md| index 82b36f572..c9ee039b0 100644 |
| --- a/docs/superpowers/plans/2026-10-06-moves-phase1-2-cut-flip-landing.md |
| +++ b/docs/superpowers/plans/2026-10-06-moves-phase1-2-cut-flip-landing.md |
| @@ -4,7 +4,7 @@ |
| **Goal:** Cut a prepared move over: stop the source with durable evidence, seal the staged copy by version, publish the archive, flip the record and the claim, and install the generation on the target through `restored`, with every phase journaled and replayable and no process yet started on the target. |
| -**Architecture:** The source's cut phases live in `internal/host/movescut.go` beside `movespass.go`, the target's landing phases in `internal/host/movesland.go`; both drive `provision.MoveState` CASes guarded by move id and incarnation and then the claim CASes `internal/hostclaim/move.go` defines (`Flip`, `ExecToNone`, `SetMovePhase`, `SetJournal`). Stop evidence is `activator.ReconstructStopEvidence`, built on `cgroup.Cleanup` and `stopevidence.Collect`. The generation build is `internal/moves/build.go`: litestream's own planner over `moves.SealedClient` (exactly the sealed versions), then the snapshot and anchor encoded with `ltx.NewEncoder` as `litestreamfacts` does. Holds are registry rows with `hold_op`; reconcile mints and keeps participant rows through two hooks the host package supplies. `appconfig.RenderHeld` and `release.MaterializeHeld` render and stage for a held row. `internal/park` gains the `superseded/` destination. |
| +**Architecture:** The source's cut phases live in `internal/host/movescut.go` beside `movespass.go`, the target's landing phases in `internal/host/movesland.go`; both drive `provision.MoveState` CASes guarded by move id and incarnation and then the claim CASes `internal/hostclaim/move.go` defines (`Flip`, `ExecToNone`, `SetMovePhase`, `SetJournal`). Stop evidence is `activator.ReconstructStopEvidence`, built on `cgroup.Cleanup` and `stopevidence.Collect`. The generation build is `internal/moves/build.go`: litestream's own planner over `moves.SealedClient` (exactly the sealed versions), then the snapshot and anchor encoded with `ltx.NewEncoder` as `litestreamfacts` does. Holds are registry rows with `hold_op`; reconcile mints and keeps participant rows through two hooks the host package supplies. `appconfig.RenderHeld` and `release.MaterializeHeld` render and stage for a held row. `internal/park` gains `Engine.Parked`; the existing kinds and the `parked/` destination carry move retirements (index D6). |
| **Tech Stack:** Go, litestream v0.5.14 (`NewReplicaWithClient`, `RestoreOptions.TXID`), superfly/ltx v0.5.1 (`NewEncoder`, `NewDecoder`, `Verify`), modernc sqlite via `lease.CheckDB`, cgroup v2 (`internal/cgroup`), `internal/stopevidence`, `internal/hostguard`, `internal/park`, `internal/genselect`. |
| @@ -21,6 +21,8 @@ |
| - One absolute deadline: `cut_deadline = cut_at + cut_budget − open_budget`; every phase before the commit checks it (§7.4). |
| - `promoteStagedReplica` and the staging-wins restore are deleted (§5.1, M15). |
| - No em dashes in any string. |
| +- Cold only (P7, index D14); the switch (D13) gates the cut and the landing like every other entry point. |
| +- The install routine (Task 8) is shared with region HA's relocation (spec rev 7.19 §7.6, L16): its input is a pinned version set and `N`, never "a staged replica" by name. |
| ## Review Focus |
| @@ -46,7 +48,7 @@ |
| | `internal/moves/ltxbuild.go` | full-file LTX encoding, `BuildIntent`, `InstallManifest` | |
| | `internal/moves/sealedclient.go` | `SealedClient`, a `litestream.ReplicaClient` over the sealed versions | |
| | `internal/moves/build.go` | `BuildGeneration` | |
| -| `internal/park/remote.go` | `park-superseded`, `park-staging`, `superseded/` destination, `Engine.Parked` | |
| +| `internal/park/remote.go` | `Engine.Parked` (the existing kinds and `parked/` destination carry move retirements, index D6) | |
| | `internal/appconfig/renderheld.go` | `RenderHeld` and the move-bound ack | |
| | `internal/release/materializeheld.go` | `MaterializeHeld` | |
| | `internal/provision/reconcile.go` | `Hold`, `KeepParticipant` | |
| @@ -312,7 +314,7 @@ func MoveRevert(g MoveGuard, reason string) func(*Record) error { |
| - Modify: `internal/hostclaim/move.go`, `internal/hostclaim/move_test.go` |
| **Interfaces:** |
| -- Produces: journal keys `JournalSeal = "seal"`, `JournalSrcGen = "src_gen"`, `JournalSourceStamp = "source_stamp"`, `JournalCutSession = "cut_session"`, `JournalRevPhase = "rev_phase"`, `JournalEvidenceSource = "evidence_source"`, `JournalEvidenceTarget = "evidence_target"`; `SetMovePhase(opID, kind, from, to string) func(*Claim) error`; `SetJournal(opID string, kv map[string]string) func(*Claim) error` (op id only; any kind); `ExecToNone(opID, box, evidenceKey, journalKey string) func(*Claim) error` (K3: `c.Op.Exec == box` → `""`, records the evidence key under `journalKey`); `Flip(opID, targetBox string, sourceStamp uint64, srcGen string) func(*Claim) error` (K4: requires `KindMoving`, phase `sealed`, `Exec == ""`, `evidence_source` journalled; sets `Kind = KindLanding`, `Phase = "checking"`, `Exec = targetBox`, `holder.stamp = c.Gen + 1`, the gen this CAS produces: `hostclaim.Update` sets `gen = observed + 1` after `tr` runs (`store.go`, and `create.go:94` writes the serving stamp the same way); journals `source_stamp` and `src_gen`); `Revert(opID, sourceBox, targetRegion string, stagingScope *ParkScope, newID func() string) func(*Claim) error` (from `KindMoving`, `Exec == ""`, `evidence_source` journalled → `KindReverting`, `Exec = sourceBox`, `rev_phase = "evidence"` unless set, plus the `park-staging` obligation for the target region's lane with the writers' bound as its scope, which §7.11's "opened waits for the staging obligation" refers to; Plan 4 Task 6 waits on it and Plan 5 Task 1 dispatches it). |
| +- Produces: journal keys `JournalSeal = "seal"`, `JournalSrcGen = "src_gen"`, `JournalSourceStamp = "source_stamp"`, `JournalCutSession = "cut_session"`, `JournalRevPhase = "rev_phase"`, `JournalEvidenceSource = "evidence_source"`, `JournalEvidenceTarget = "evidence_target"`; `SetMovePhase(opID, kind, from, to string) func(*Claim) error`; `SetJournal(opID string, kv map[string]string) func(*Claim) error` (op id only; any kind); `ExecToNone(opID, box, evidenceKey, journalKey string) func(*Claim) error` (K3: `c.Op.Exec == box` → `""`, records the evidence key under `journalKey`); `Flip(opID, targetBox string, sourceStamp uint64, srcGen string) func(*Claim) error` (K4: requires `KindMoving`, phase `sealed`, `Exec == ""`, `evidence_source` journalled; sets `Kind = KindLanding`, `Phase = "checking"`, `Exec = targetBox`, `holder.stamp = c.Gen + 1`, the gen this CAS produces: `hostclaim.Update` sets `gen = observed + 1` after `tr` runs (`store.go`, and `create.go:94` writes the serving stamp the same way); journals `source_stamp` and `src_gen`); `Revert(opID, sourceBox, targetRegion string, stagingScope *ParkScope, newID func() string) func(*Claim) error` (from `KindMoving`, `Exec == ""`, `evidence_source` journalled → `KindReverting`, `Exec = sourceBox`, `rev_phase = "evidence"` unless set, plus the lane `park-replica` obligation for the target region's lane with the writers' bound as its scope, which §7.11's "opened waits for the staging obligation" refers to; Plan 4 Task 6 waits on it and Plan 5 Task 1 dispatches it). |
| - [ ] **Step 1: Write the failing test** |
| @@ -375,7 +377,7 @@ func TestRevertNeedsEvidenceAndReturnsExecToSource(t *testing.T) { |
| if err := Revert(op, "ie-box", "uk", &scope, newID)(c); err != nil || c.Op.Kind != KindReverting || c.Op.Exec != "ie-box" || c.Op.Journal[JournalRevPhase] != "evidence" { |
| t.Fatalf("revert: %v %+v", err, c.Op) |
| } |
| - if len(c.Obligations) != 1 || c.Obligations[0].Kind != "park-staging" || c.Obligations[0].Region != "uk" || c.Obligations[0].Park == nil { |
| + if len(c.Obligations) != 1 || c.Obligations[0].Kind != "park-replica" || c.Obligations[0].Region != "uk" || c.Obligations[0].Park == nil { |
| t.Fatalf("revert queued no target staging park: %+v", c.Obligations) |
| } |
| } |
| @@ -427,7 +429,9 @@ func SetMovePhase(opID, kind, from, to string) func(*Claim) error { |
| } |
| // ExecToNone is K3: a participant that has written stop evidence for this |
| -// op gives execution up. Nobody moves exec from one box to another. |
| +// op gives execution up. Nobody moves exec from one box to another. A |
| +// replay after a crash finds exec already empty and the same evidence |
| +// journalled: that is idempotent, not a conflict. |
| func ExecToNone(opID, box, evidenceKey, journalKey string) func(*Claim) error { |
| return func(c *Claim) error { |
| if err := RequireOp(opID)(c); err != nil { |
| @@ -436,6 +440,9 @@ func ExecToNone(opID, box, evidenceKey, journalKey string) func(*Claim) error { |
| if evidenceKey == "" || (journalKey != JournalEvidenceSource && journalKey != JournalEvidenceTarget) { |
| return fmt.Errorf("%w: execution is given up only with evidence", ErrMalformed) |
| } |
| + if c.Op.Exec == "" && c.Op.Journal[journalKey] == evidenceKey { |
| + return nil |
| + } |
| if c.Op.Exec != box { |
| return fmt.Errorf("%w: exec is %q, not %q", ErrWrongState, c.Op.Exec, box) |
| } |
| @@ -495,12 +502,12 @@ func Revert(opID, sourceBox, targetRegion string, stagingScope *ParkScope, newID |
| setJournal(c, JournalRevPhase, "evidence") |
| } |
| for _, o := range c.Obligations { |
| - if o.Op == opID && o.Kind == "park-staging" && o.Region == targetRegion { |
| + if o.Op == opID && o.Kind == "park-replica" && o.Region == targetRegion { |
| return nil |
| } |
| } |
| s := *stagingScope |
| - c.Obligations = append(c.Obligations, Obligation{ID: newID(), Op: opID, Kind: "park-staging", Region: targetRegion, Retirement: newID(), Owner: "region:" + targetRegion, Park: &s}) |
| + c.Obligations = append(c.Obligations, Obligation{ID: newID(), Op: opID, Kind: "park-replica", Region: targetRegion, Retirement: newID(), Owner: "region:" + targetRegion, Park: &s}) |
| return nil |
| } |
| } |
| @@ -713,12 +720,12 @@ func dbPathsWithSidecars(paths []string) []string { |
| - `stopping`: `SetHold(host, "moving", id)`; fence (`cfg.Fence`); wait `TimersDrained`; `cfg.Evidence` (steps 1 to 4); `!TreeStopped()` → retry next pass until `cut_deadline`, then `MoveRevert(g, "stop evidence budget")` and `hostclaim.Revert`; `TreeStopped()` → the four steps are written into the op journal (`Journal.Steps["evidence"]`, local; nothing is published yet, because the report's `StepReplica`, `FinalTXID` and digests do not exist until the checkpoint) → `MoveCutPhase(stopping → stopped)`. |
| - `stopped`: `Checkpoint` (final `SyncAndWait`); `N = NewestL0`; `StepReplica` OK with the database's sha256; `Replication`, `NextDue` from the row; timers snapshot → `move-snapshots/<host>/<id>/timers.json` (own bucket, create-only) and `TimersPrefix` (target, `PutVersioned`); `files.plain` via `archive.PlainDigest`; **now** the complete report: `stopped-source.json` create-only (K13: `release.Store.Put(ctx, key, body, "")` is create-only on both the S3 and the dir store, `store.go:38`, and loses with `release.ErrCASConflict`; on a conflict read the existing object, and equal `Host`, `OpID`, `Role` and `FinalTXID` is a replay and fine); `MoveRecordSource`; then `hostclaim.Update(ExecToNone(id, box, stoppedKey, JournalEvidenceSource))` and `SetJournal(cut_session = holder.session)` in the same transition (a crash between any two of these replays the phase: the checkpoint is idempotent on a stopped database, the report is create-only, the CAS is conditional); `MoveCutPhase(stopped → staged)` is written only after the final round below succeeds, so the phase `stopped` repeats the mirror until it does: |
| - the final round: `MirrorPrefix` into `replica/` under the lease; `Deferred > 0` → count in the journal (`moveLedger.DeferredRounds`), three → `MoveRevert`; else seal = `round.Versions` for every current key plus the files and timers versions, `SrcCovered = (N == LocalPosition)`, `FromDB = true` when the stopped database file exists; `SetJournal(seal)` and `SetMovePhase(moving, "" → sealed)` in one claim CAS; `MoveCutPhase(stopped → staged)`. |
| - - `staged`: the final archive from the stopped directory, every step replayable and all of them **before** the phase advances: `archive.Build` + `Seal` → `move-snapshots/<host>/<id>/files-<sealed>.sealed` create-only (`PutIfAbsent`; on `ErrExists` the recorded object is reused, codex 19), the manifest beside it; copy both into `FilesPrefix` with `PutVersioned`; `SetJournal(seal)` with `Files` filled (same `N`); only then `MoveCutPhase(staged → archived, stamp MoveSource.Files digests)`. A crash anywhere inside replays `staged` from the top: the create-only put returns the same object, the copies make new versions the next `SetJournal(seal)` names, and the phase never says `archived` without the seal naming the archive. |
| + - `staged`: the final archive from the stopped directory, every step replayable and all of them **before** the phase advances. Sealing is randomized (§8.2), so the archive's identity is fixed **before** any PUT: `archive.Build` + `Seal` into `<JournalDir>/files.sealed` and `files.json` (the manifest with the sealed key and both digests), then the identity (`sealed` digest, `plain` digest, sealed key, byte count) into the op journal (`Journal.Steps["archive"]`); a replay that finds the journal entry and the temp file re-puts those bytes under the same key `files-<sealed>.sealed` (create-only; `ErrExists` is its own earlier PUT); a replay that finds the entry but not the temp file checks the object by key (`ListVersions` of the exact key in the own bucket): present → reuse; absent → rebuild, reseal and **overwrite the journal entry** (nothing was published under the old identity). Then: `PutIfAbsent` to `move-snapshots/<host>/<id>/files-<sealed>.sealed` and the manifest beside it; copy both into `FilesPrefix` with `PutVersioned`; `SetJournal(seal)` with `Files` filled (same `N`); only then `MoveCutPhase(staged → archived, stamp MoveSource.Files digests)`. The phase never says `archived` without the seal naming the archive, and the record's digests are the journaled identity's. |
| - `archived`: precondition, re-read from the claim: `seal.Files` non-empty and `seal.N == Source.FinalTXID` (else the phase is a replay of `staged`'s tail: redo the copy and the journal, never flip); `MoveFlip(g, now, now + 15 min)`; then `hostclaim.Update(Flip(id, targetBox, holder.stamp, holder.generation))`; `nudgeSerial`. The claim half may be written by anyone who sees the record ahead (§7.5); the source tries once here and the console pass (Plan 5) finishes it. |
| - - every phase: `now > cut_deadline` → `MoveRevert` (phases `stopping..archived`) and `hostclaim.Revert` once exec is none. |
| + - every phase: `now > cut_deadline` → `MoveRevert` (phases `stopping..archived`) and `hostclaim.Revert(id, cfg.Box, rec.Move.To, &stagingScope, newID)` once exec is none (Task 2's five arguments; `stagingScope` as Plan 4 Task 6 builds it from the receipts ledger's `LeaseUntil`). |
| - Each phase's local progress is in the op journal F12 (`moves.Journal`, Task 7's shape, written here first as `internal/moves/journal.go`). |
| -Tests (`movescut_test.go`, harness from Plan 1 Task 14 extended with a `fakeEvidence` returning a configurable report, a fake versioned store with a `failAfter(n)` hook, and a claim written into the harness store): `TestCutAdmitsOnlyFreshPassingReports`, `TestCutGoesBackToReadyWhenInputsMove`, `TestCutResumesStoppingFromEvidenceAlone` (Review Focus 1: a fresh worker on the same store with the record at `stopping` calls `Evidence` and advances, with no in-memory state), `TestStoppedReportIsCompleteBeforeExecToNone` (the report object exists with `StepReplica` and `FinalTXID` before the claim's exec is `""`; a crash injected after the checkpoint and before the report leaves exec with the source; the replay publishes one report), `TestCutRevertsAfterThreeDeferredRounds` (Review Focus 2), `TestSealNamesEveryCurrentVersionAndN` (witness `wSealByVersion`: a stale copy landing after the seal creates a version the seal does not name), `TestCutArchiveIsCreateOnlyAndReused` (codex 19), `TestArchivedNeverPrecedesTheSealedArchive` (crashes injected after the create-only put, after the first `PutVersioned`, and after the second: in every case the record is still `staged`; the replay completes and the seal names both `files/` versions before `archived`; a record hand-edited to `archived` with an empty `seal.Files` does not flip), `TestFlipWritesRecordThenClaim` (record `landing` with region `uk`; claim `landing/checking`, exec `uk-box`, `source_stamp` journalled; `holder.generation` unchanged), `TestCutDeadlinePassedReverts`. |
| +Tests (`movescut_test.go`, harness from Plan 1 Task 14 extended with a `fakeEvidence` returning a configurable report, a fake versioned store with a `failAfter(n)` hook, and a claim written into the harness store): `TestCutAdmitsOnlyFreshPassingReports`, `TestCutGoesBackToReadyWhenInputsMove`, `TestCutResumesStoppingFromEvidenceAlone` (Review Focus 1: a fresh worker on the same store with the record at `stopping` calls `Evidence` and advances, with no in-memory state), `TestStoppedReportIsCompleteBeforeExecToNone` (the report object exists with `StepReplica` and `FinalTXID` before the claim's exec is `""`; a crash injected after the checkpoint and before the report leaves exec with the source; the replay publishes one report), `TestCutRevertsAfterThreeDeferredRounds` (Review Focus 2), `TestSealNamesEveryCurrentVersionAndN` (witness `wSealByVersion`: a stale copy landing after the seal creates a version the seal does not name), `TestCutArchiveIsCreateOnlyAndReused` (codex 19, with the real randomized sealer: crashes after the journal entry, after the create-only PUT, after the first copy; every replay publishes the **same** `files-<sealed>` key; a replay with the temp file deleted and no object yet reseals under a new key and the record names the new one; a replay with the temp file deleted and the object present reuses it), `TestArchivedNeverPrecedesTheSealedArchive` (crashes injected after the create-only put, after the first `PutVersioned`, and after the second: in every case the record is still `staged`; the replay completes and the seal names both `files/` versions before `archived`; a record hand-edited to `archived` with an empty `seal.Files` does not flip), `TestFlipWritesRecordThenClaim` (record `landing` with region `uk`; claim `landing/checking`, exec `uk-box`, `source_stamp` journalled; `holder.generation` unchanged), `TestCutDeadlinePassedReverts`. |
| - [ ] **Step 1 to 4** as above. Run: `nice -n 10 go test ./internal/host ./internal/moves -run 'TestCut|TestSeal|TestFlip' -count=1 -p 1` → PASS. |
| - [ ] **Step 5: Commit**: `git commit -m "host: the journaled cut, requested to flipped, with stop evidence and the seal by version (§7.4, §7.5)" ...` |
| @@ -735,10 +742,10 @@ Tests (`movescut_test.go`, harness from Plan 1 Task 14 extended with a `fakeEvid |
| **Interfaces:** |
| - `provision.ReconcileConfig.Hold func(rec Record) (status, op string, ok bool)`: when ok, a row minted for `rec` is `Upsert`ed with `Status = status, HoldOp = op` and never `ready`; an existing row under that op is left as is. `KeepParticipant func(host string) bool`: when true, the removal phase skips the row ("participant of a running operation"), and the orphan guard does not act on it. |
| -- `schedule.Runner.Busy(host string) bool` (the per-host claim, `runner.go:276`); the runner's `Tick` takes `claim(host)` **first** and then re-reads the route (`r.Reg.Get`) inside the claim: a held row (`registry.IsHold(rt.Status)`) releases the claim and continues with no bookkeeping. The order matters (the cut's drain): `SetHold` then "wait until `!Busy(host)`" is correct only if every tick that observes the row **after** the hold does so while holding the claim, so a tick paused between its route read and its claim cannot slip past the drain and advance `attempts` (`runner.go:350`). The route read before `claim` at `runner.go:143` stays as a cheap pre-filter; the decision is the second read. `schedule.Converger.Frozen func(host string) bool`: `convergeHost` returns `(false, nil)` for a frozen host, and the prune's `alive` set includes frozen hosts so `PruneTimers` keeps their rows. |
| +- `schedule.Runner.Busy(host string) bool` (the per-host claim, `runner.go:276`); the runner's `Tick` takes `claim(host)` **first** and then re-reads the route (`r.Reg.Get`) inside the claim: a held row (`registry.IsHold(rt.Status)`) releases the claim and continues with no bookkeeping. The order matters (the cut's drain): `SetHold` then "wait until `!Busy(host)`" is correct only if every tick that observes the row **after** the hold does so while holding the claim, so a tick paused between its route read and its claim cannot slip past the drain and advance `attempts` (`runner.go:350`). The route read before `claim` at `runner.go:143` stays as a cheap pre-filter; the decision is the second read. `schedule.Converger.Frozen func(host string) bool`: `convergeHost` returns `(false, nil)` for a frozen host, and the prune's `alive` set includes frozen hosts so `PruneTimers` keeps their rows. **One gate for all three writers** (round-2 finding 21): `schedule.HostGate` (`internal/schedule/hostgate.go`, a per-host `sync.RWMutex` map) is shared by the `Runner` (its `claim`/`release` take the host's read lock for the delivery), the `Converger` (`convergeHost` holds the read lock across its check of `Frozen` and its writes) and the cut's drain (`moveCfg.TimersDrained` takes the **write** lock after `SetHold`, which waits for in-flight deliveries and convergences, then releases it; from then on the hold check inside the lock refuses both). The agent's overlapping converge loops (`agent.go:1991`, `:2555`) share the one `HostGate` the schedule wiring creates. |
| - `movesHoldView` in the host package: built from each moves pass's record list plus `hostclaim.Read` per host with a move in flight; exposes `hold(rec) (status, op, ok)` (target after the flip: `landing`; source in `reverting`/`restoring`: `restoring`; target in a rollback's release: `releasing`; source during `moving` at or after `stopping`: `moving`), `keep(host) bool` (this box in `op.participants` of a move-kind op), `frozen(host) bool` (a move-kind op holds the claim), `strict(host) bool` (same). |
| -Test `TestReconcileHoldsParticipantRows` (Review Focus 5): a record with region `uk` and a `Hold` returning `("landing", id, true)` mints the row held on the uk box and never `ready`; the ie box with `KeepParticipant(host) == true` keeps its `moving` row though the record's region is `uk`; with `KeepParticipant == false` the row is removed as today. Test `TestHeldRowsFreezeTimers` (witness `wTimerFreeze`): a held row's due once-timer is not delivered and its `attempts` does not advance; a frozen host's declared schedule is neither upserted nor pruned by `Converger.Pass`; after the hold lifts both resume. Test `TestDrainRaceCannotAdvanceBookkeeping`: a tick reads the route (ready), pauses on a test hook before `claim`; the cut sets the hold and observes `Busy == false`; the tick resumes, claims, re-reads, sees the hold and releases; `attempts` is unchanged and nothing was delivered. |
| +Test `TestReconcileHoldsParticipantRows` (Review Focus 5): a record with region `uk` and a `Hold` returning `("landing", id, true)` mints the row held on the uk box and never `ready`; the ie box with `KeepParticipant(host) == true` keeps its `moving` row though the record's region is `uk`; with `KeepParticipant == false` the row is removed as today. Test `TestHeldRowsFreezeTimers` (witness `wTimerFreeze`): a held row's due once-timer is not delivered and its `attempts` does not advance; a frozen host's declared schedule is neither upserted nor pruned by `Converger.Pass`; after the hold lifts both resume. Test `TestDrainRaceCannotAdvanceBookkeeping`: a tick reads the route (ready), pauses on a test hook before `claim`; the cut sets the hold and observes `Busy == false`; the tick resumes, claims, re-reads, sees the hold and releases; `attempts` is unchanged and nothing was delivered. Test `TestConvergerCannotWriteAfterTheBarrier`: a `convergeHost` paused after its `Frozen` check (inside the gate's read lock) blocks the cut's drain until it finishes; a converger that starts after the drain sees `Frozen` and writes nothing; the timers snapshot taken after the drain equals the table. |
| - [ ] **Steps 1 to 4**. Run: `nice -n 10 go test ./internal/provision ./internal/schedule ./internal/host -count=1 -p 1` → PASS. |
| - [ ] **Step 5: Commit**: `git commit -m "reconcile and schedule honour move holds: participant rows kept, held rows never ready, timers frozen and drained (§5.2, §9.4, §9.6, K9)" ...` |
| @@ -772,7 +779,7 @@ Tests: `RenderHeld` writes the file for a `landing` row that `SyncExec` would sk |
| - `moves.EncodeFull(db io.ReaderAt, pageSize, pageCount uint32, min, max ltx.TXID, timestamp int64, walOffset int64) (data []byte, crc64 uint64, err error)`: `ltx.NewEncoder` with `Header{Version: ltx.Version, Flags: ltx.HeaderFlagNoChecksum, PageSize, Commit: pageCount, MinTXID: min, MaxTXID: max, Timestamp: timestamp, WALOffset: walOffset}`, every page in order **except SQLite's lock page**, `ltx.LockPgno(pageSize)` (`ltx.go:494`; the encoder refuses it, `encoder.go:219`, and litestream skips it), while `Commit` keeps the full page count; `Close`; the CRC-64 is the trailer's `FileChecksum` (read it back with `ltx.NewDecoder(bytes.NewReader(data)).Verify()` then `Trailer().FileChecksum`). |
| - `moves.BuildIntent{V int; Op, Host string; N uint64; PageSize, Commit uint32; Timestamp int64}`; `moves.InstallManifest{V int; Op, Host string; N uint64; SnapshotKey, SnapshotVersion string; SnapshotCRC64 uint64; AnchorKey, AnchorVersion string; AnchorCRC64 uint64; IntegrityCheck string; DBSHA256 string; BuiltAt time.Time}`; `moves.GenerationPrefix(location, op string) string` (`<location>g-<op>/`, location with its trailing slash); `moves.SnapshotKey(prefix string, n uint64)` = `prefix + "0009/" + ltx.FormatFilename(1, n)`; `moves.AnchorKey(prefix, n)` = `prefix + "0000/" + ltx.FormatFilename(n, n)`; `BuildIntentKey(prefix) = prefix+"build-intent.json"`, `InstallManifestKey(prefix) = prefix+"install.json"`. |
| - `moves.NewSealedClient(store lease.VersionedStore, bucket, replicaPrefix string, seal Seal) *SealedClient` implementing `litestream.ReplicaClient`: `Type() "sealed"`, `Init` no-op, `LTXFiles(ctx, level, seek, _)` returns `ltx.NewFileInfoSliceIterator` over the sealed objects at that level parsed from their keys (`ltx.ParseFilename`), `OpenLTXFile` opens exactly the sealed version by `OpenVersion(ctx, bucket, key, versionID)` (ranged through `VersionRangeOpener` when offset/size are set), `WriteLTXFile`, `DeleteLTXFiles`, `DeleteAll` return `ErrSealedReadOnly`, `SetLogger` stores it. |
| -- `moves.BuildGeneration(ctx, in BuildInputs) (InstallManifest, error)` with `BuildInputs{Store lease.VersionedStore; Bucket, ReplicaPrefix string; Seal Seal; GenerationPrefix string; Op, Host string; WorkDir string; Now func() time.Time; Verify func(dbPath string) error}`: (1) read or write the intent (`PutIfAbsent`; on `lease.ErrExists` read it and use its timestamp, page size, commit); (2) restore into a fresh scratch file: remove `<WorkDir>/<host>.db`, `-wal`, `-shm` and `-txid` first (litestream's `Restore` refuses an existing output path, `replica.go:601`; the file is reproducible from the sealed versions, so a replay after any crash starts clean), then `litestream.NewReplicaWithClient(nil, sealedClient).Restore(ctx, opt)` with `opt.OutputPath = <WorkDir>/<host>.db`, `opt.TXID = ltx.TXID(seal.N)`; refuse unless the restored position is `N` (litestream writes `<db>-txid`? Only in follow mode; instead compare the plan's final `MaxTXID` through `litestream.CalcRestorePlan(ctx, client, ltx.TXID(N), time.Time{}, logger)` whose last file's `MaxTXID` must be `N`, and verify every planned file with `ltx.NewDecoder(...).Verify()` before the restore); `Verify(dbPath)` = `lease.CheckDB` (integrity_check); (3) `EncodeFull` snapshot `[1, N]` and `PutIfAbsent(SnapshotKey)`; on `ErrExists`, `OpenVersion` the current one and require equal bytes (identical by construction from the intent; differing bytes refuse with `ErrBuildDiverged`); (4) the anchor `[N, N]` with `WALOffset = litestream.WALHeaderSize`, same rule; (5) sha256 of the database file; (6) `InstallManifest` via `PutIfAbsent` (on exists, read and compare `N`, keys, versions). |
| +- `moves.BuildGeneration(ctx, in BuildInputs) (InstallManifest, error)` with `BuildInputs{Store lease.VersionedStore; Bucket string; Source []SealedObject (the pinned version set: for a move the seal's replica/ objects, for region HA's relocation the versions a `genselect` plan or a current listing pins, L16); N uint64; GenerationPrefix string; Op, Host string; WorkDir string; Now func() time.Time; Verify func(dbPath string) error}` (`SealedClient` is built over `Source`, so nothing in the routine names a staging lane): (1) read or write the intent (`PutIfAbsent`; on `lease.ErrExists` read it and use its timestamp, page size, commit); (2) restore into a fresh scratch file: remove `<WorkDir>/<host>.db`, `-wal`, `-shm` and `-txid` first (litestream's `Restore` refuses an existing output path, `replica.go:601`; the file is reproducible from the sealed versions, so a replay after any crash starts clean), then `litestream.NewReplicaWithClient(nil, sealedClient).Restore(ctx, opt)` with `opt.OutputPath = <WorkDir>/<host>.db`, `opt.TXID = ltx.TXID(seal.N)`; refuse unless the restored position is `N` (litestream writes `<db>-txid`? Only in follow mode; instead compare the plan's final `MaxTXID` through `litestream.CalcRestorePlan(ctx, client, ltx.TXID(N), time.Time{}, logger)` whose last file's `MaxTXID` must be `N`, and verify every planned file with `ltx.NewDecoder(...).Verify()` before the restore); `Verify(dbPath)` = `lease.CheckDB` (integrity_check); (3) `EncodeFull` snapshot `[1, N]` and `PutIfAbsent(SnapshotKey)`; on `ErrExists`, `OpenVersion` the current one and require equal bytes (identical by construction from the intent; differing bytes refuse with `ErrBuildDiverged`); (4) the anchor `[N, N]` with `WALOffset = litestream.WALHeaderSize`, same rule; (5) sha256 of the database file; (6) `InstallManifest` via `PutIfAbsent` (on exists, read and compare `N`, keys, versions). |
| Tests (`build_test.go`, using `internal/lease/leasetest`'s local versioned store if it offers one, else a small in-memory `lease.VersionedStore` fake in `moves` tests): `TestSealedClientServesOnlySealedVersions` (a newer version of a sealed key exists; `OpenLTXFile` returns the sealed bytes; an unsealed key is absent); `TestBuildGenerationReachesNAndWritesSnapshotAndAnchor` (drive litestream over a real sqlite file through the file replica client as `litestreamfacts` does to produce a replica, seal its current versions, build, then assert: snapshot header `MinTXID 1, MaxTXID N, Flags NoChecksum`, anchor `[N, N]` with `WALOffset == WALHeaderSize`, both `Verify()`, manifest's `DBSHA256` equals the file's, `IntegrityCheck == "ok"`); `TestBuildReplayReusesIdenticalSnapshot` (Review Focus 3, witness `wBuildCrashes`: run the build with a `Store` that fails after the snapshot PUT, **in the same `WorkDir`**, with the half-restored database file left on disk; run again on the same store and directory; the snapshot key has one version, the manifest exists, bytes identical; a second variant fails during the restore itself, leaving a partial output file, and the replay still completes); `TestBuildRefusesAGapOrAShortRestore` (drop a sealed middle file → `ErrBuildIncomplete`, no snapshot PUT); `TestEncodeFullSkipsTheLockPage` (a sparse database file extended past 1 GiB with `Truncate`, page size 4096: the encoder succeeds, the file has `pageCount - 1` pages, `Verify()` passes, and litestream's own restore of it produces a database whose `PRAGMA page_count` equals the original). |
| @@ -781,21 +788,21 @@ Tests (`build_test.go`, using `internal/lease/leasetest`'s local versioned store |
| --- |
| -### Task 9: Park destinations for move retirements |
| +### Task 9: Park queue helpers for move retirements (no new kinds) |
| **Files:** |
| -- Modify: `internal/park/remote.go` (`archiveBase`, `destination`, kind checks), `internal/park/remote_test.go` |
| -- Modify: `internal/hostclaim/claim.go` (`Obligation.Kind` doc lists the two kinds), `internal/hostclaim/move.go` (`QueueSupersededPark`, `QueueStagingPark`) |
| +- Modify: `internal/park/remote.go` (`Engine.Parked` only), `internal/park/remote_test.go` |
| +- Modify: `internal/hostclaim/move.go` (`QueueLocationPark`, `QueueStagingPark`, `QueueLocalPark`, `QueueResidueParks`) |
| -**Interfaces:** |
| -- Kinds `"park-superseded"` (a whole location: `Park.Location` is `instances/<host>` or `instances/<host>/<inc>`) and `"park-staging"` (`Park.Location` is `moves/<host>/<op>`); `archiveBase` for both: `"superseded/" + o.Op + "/" + host + "/" + o.Retirement + "/"`; `destination` appends `instances/...` or `moves/...` under it as today's `kind + "/"` rule does. |
| +**Interfaces (index D6: the engine's existing kinds and destination):** |
| +- A staging lane is a `park-replica` scoped to `moves/<host>/<op>`; a whole location to retire in a region where it is no longer authoritative (the source after the commit, a rolled-back target at `cleaning`) is a `park-replica` scoped to `instances/<host>/<inc>`; a landing's earlier-stay residue in the target region is one `park-generation` per residue generation (`hostclaim.RootGeneration` for the origin). `Claim.Validate` (`claim.go:266-288`) accepts all three unchanged; the archive lands under `parked/<incarnation>/<host>/<retirement>/`; F3 reads the manifest's `Op`. |
| - `(e Engine) Parked(ctx, w Work) (bool, error)`: the manifest exists and every version in its deletion set is gone (what the landing's `parking` phase waits for, §7.6; acknowledgement still waits for the horizon in `Run`). |
| -- `hostclaim.QueueSupersededPark(opID, region, box string, scope ParkScope, newID func() string) func(*Claim) error` (appends `{Kind: "park-superseded", Region, Owner: "region:"+region, Park: &scope, Retirement: newID(), ID: newID()}` and, when `box != ""`, a `park-local` for the box; scope validated with `Validate(host)`; idempotent per `(opID, region)` and `(opID, box)`); `hostclaim.QueueLocalPark(opID, box string, newID func() string) func(*Claim) error` (a `park-local` alone, for the landing's `local` phase: `park.Local.RetireArtifacts` runs only against a persisted `park-local` chore of the claim, `local.go:62`, and treats an unknown id as already completed, so the chore must exist in the claim before the call); `hostclaim.QueueStagingPark(opID, region string, scope ParkScope, newID func() string) func(*Claim) error`. `Plan 5 Task 4`'s destination patch covers `local_linux.go:196-201` (the `localWalk` segments). |
| +- `hostclaim.QueueLocationPark(opID, region, box string, scope ParkScope, newID func() string) func(*Claim) error` (appends `{Kind: "park-replica", Region, Owner: "region:"+region, Park: &scope, Retirement: newID(), ID: newID()}` and, when `box != ""`, a `park-local` for the box; scope validated with `Validate(host)`; idempotent per `(opID, region)` and `(opID, box)`); `hostclaim.QueueStagingPark(opID, region string, scope ParkScope, newID func() string) func(*Claim) error` (a `park-replica` whose scope location is `moves/<host>/<op>`); `hostclaim.QueueLocalPark(opID, box string, newID func() string) func(*Claim) error` (a `park-local` alone, for the landing's `local` phase: `park.Local.RetireArtifacts` runs only against a persisted `park-local` chore of the claim, `local.go:62`, and treats an unknown id as already completed, so the chore must exist in the claim before the call); `hostclaim.QueueResidueParks(opID, region string, location ParkScope, generations []string, newID func() string) func(*Claim) error` (one `park-generation` per listed generation, each `{Kind: "park-generation", Region, Generation: g, Owner: "region:"+region, Park: &scope}`, deduplicated by `(opID, region, g)`; the landing's `parking` phase feeds it `park.Decide`'s candidates grouped by generation). |
| -Tests: `TestSupersededDestination` (a `park-superseded` obligation archives `instances/h/<inc>/0000/x.ltx@v1` to `superseded/<op>/h/<ret>/instances/h/<inc>/0000/x.ltx@v1`); `TestParkedReportsManifestAndDeletions`; `TestQueueSupersededParkValidatesScope`. |
| +Tests: `TestLocationParkIsAParkReplicaTheStoreAccepts` (a real `hostclaim.Update` round trip through `release.NewDirStore`: `Validate` passes, the obligation reads back); `TestStagingParkScopeIsTheLane`; `TestResidueParksArePerGeneration` (two residue generations and the root → three chores, `@root` named); `TestParkedReportsManifestAndDeletions`; `TestEngineArchivesAMoveRetirementUnderParked` (the destination is `parked/<inc>/h/<ret>/instances/h/<inc>/0000/x.ltx@v1` and the manifest's `Op` is the move id). |
| - [ ] **Steps 1 to 4**. Run: `nice -n 10 go test ./internal/park ./internal/hostclaim -count=1 -p 1` → PASS. |
| -- [ ] **Step 5: Commit**: `git commit -m "park: superseded/ destinations for move retirements; Parked; hostclaim queues for them (§6.4, D6)" ...` |
| +- [ ] **Step 5: Commit**: `git commit -m "park: Parked; hostclaim: park queue helpers for move retirements on the existing kinds (§6.4, D6)" ...` |
| --- |
| @@ -808,18 +815,18 @@ Tests: `TestSupersededDestination` (a `park-superseded` obligation archives `ins |
| **Interfaces:** |
| - `moveCfg` gains: `Guard hostguard.Guard`, `Park park.Engine` (own bucket and region; `View`, `Authorize` from Plan 5's providers until then a view that lists the location and authorizes only the landing's own op), `Local park.Local`, `RenderHeld func(ctx, rt registry.Route, op string) (appconfig.HeldRender, error)`, `MaterializeHeld func(ctx, rt registry.Route, version string) error`, `ReplaceTimers func(host string, rows []registry.Timer) error`, `SetReplication func(host, class string) error`, `SetNextDue func(host string, due int64) error`, `RemoveVerifyEntry func(ctx, host string) error` (loads `dbverify.Load`, deletes `Hosts[host]`, `Publish`), `OwnLease func(ctx, prefix string) (*lease.Held, error)`, `SecretsPriv *ecdh.PrivateKey`. |
| - `landingPass(ctx, s, w, rec) error` per phase (§7.6), each phase first `MoveLandingPhase(g, from, to)` then its side effects, journaled in `moves.Journal`: |
| - - every landing phase: the record CAS (`MoveLandingPhase`) first, then the claim mirrors it (`hostclaim.Update(SetMovePhase(id, KindLanding, from, to))`), so the claim reaches `probing` and `probed` and the probe's admission observes the phase it binds to (`AdmitWriter` checks the phase and never advances it, `generation.go:35`); a replay that finds the claim already ahead of the record phase it is writing skips the claim half. |
| - - `checking`: read `stopped-source.json` (must be `Complete()`); claim: `RequireMoveOp(id, KindLanding, "checking")`, `Exec == cfg.Box`; `hostclaim.Update(Touch(id, nil, []string{cfg.Box}))` if absent (Plan 1 Task 14 added it at the first staging write; this is the idempotent K12 re-add); `move.json` (own bucket, readable here) incarnation == `rec.Move.Incarnation` == `holder.incarnation`; `cfg.Guard.Activate(ctx, host, cfg.Box, nil)` (sets `active_stamp = holder.stamp`, the flip's); `hostclaim.Update(QueueLocalPark(id, cfg.Box, newID))` and `cfg.Local.RecordInventory(ctx, host, captured, "flat", []string{dbPath, filesDir, envPath}, check)` so the `local` phase has a persisted chore to retire against; acquire `OwnLease` on `holder.location` and keep it in the worker's per-host map through `published`. |
| - - `parking`: list `holder.location` in the own bucket (`Store.ListVersions`); nothing there → done; else queue `QueueSupersededPark(id, cfg.Region, "", ParkScope{Location: strings.TrimSuffix(location, "/"), Incarnation: holder.incarnation, EvidenceAt: rec.Move.FlippedAt, RequestLimit: 120 s}, newID)` once (journal step), `cfg.Park.Run(ctx, Work{Host, ID, Op})` (ErrPending expected), then `cfg.Park.Parked` must be true before advancing. |
| + - every landing phase begins with `syncClaimPhase(ctx, rec, obs)`: the record's `landing_phase` is the decision and the claim's `op.phase` follows it (§7.5); when the claim is **behind** (a crash after the record CAS and before the claim CAS), the function walks `landingOrder` from the claim's phase to the record's, one `SetMovePhase(id, KindLanding, from, to)` per step, before the phase's work; when the claim is **ahead** (a crash after the claim CAS of a later step, impossible by construction since the claim never leads, but checked) it is left alone and logged; equal is the common case. Then the phase does its work, writes the record CAS (`MoveLandingPhase`) and mirrors it (`SetMovePhase`). The console pass (Plan 5 Task 7) runs the same catch-up for a landing it finds behind. `AdmitWriter` checks the phase and never advances it (`generation.go:35`), so the probe's admission observes `probing` only through this mirroring. |
| + - `checking`: read `stopped-source.json` (must be `Complete()`); claim: `RequireMoveOp(id, KindLanding, "checking")`, `Exec == cfg.Box`; `hostclaim.Update(Touch(id, nil, []string{cfg.Box}))` if absent (Plan 1 Task 14 added it at the first staging write; this is the idempotent K12 re-add); `move.json` (own bucket, readable here) incarnation == `rec.Move.Incarnation` == `holder.incarnation`; before activating anything, read the guard record (`hostguard.Capture(dataDir, host)` returns `(Captured, error)`; `ErrNoRecord` means no earlier stay) and the claim's `Inventory[cfg.Box]`: an earlier stay's paths and their stamp `S'` are what `local` will retire, so they are noted in the op journal (`Journal.Steps["previous"] = S'`) **before** `cfg.Guard.Activate(ctx, host, cfg.Box, nil)` sets `active_stamp = holder.stamp` (the flip's `F`); `hostclaim.Update(QueueLocalPark(id, cfg.Box, newID))` so `local` has a persisted chore. **No lease is taken here** (round-2 finding 9): `park.Engine.Run` acquires the location's lease itself (`remote.go:404`), so the landing acquires `OwnLease` on `holder.location` only after `parking`, at the start of `building`, and keeps it in the worker's per-host map through `published`. |
| + - `parking`: list `holder.location` in the own bucket (`Store.ListVersions`); nothing there → done; else group the residue by generation as `park.Decide` does (every generation other than the one the landing will build; the origin as `@root`) and `hostclaim.Update(QueueResidueParks(id, cfg.Region, scope{Location: TrimSuffix(location, "/"), Incarnation, EvidenceAt: rec.Move.FlippedAt, RequestLimit: 120 s}, generations, newID))` once (journal step); `cfg.Park.Run(ctx, Work{Host, ID, Op})` per chore (ErrPending expected), then `cfg.Park.Parked` must be true for each before advancing. Per-generation chores are what let the acknowledgement finish after the probe makes the new generation live in this location (D6; a whole-location park would meet `remote.go:346`'s live-set refusal forever); the acknowledgement half is C4's runner. |
| - `building`: `moves.BuildGeneration` with the seal from the claim journal (`DecodeSeal(obs.Claim.Op.Journal[JournalSeal])`), `ReplicaPrefix` in the own bucket, `GenerationPrefix(holder.location, id)`, `WorkDir = JournalDir`. |
| - `published`: journal only (no `generation.json`, R10-3). |
| - - `local`: `cfg.Local.RetireArtifacts(ctx, host, hostguard.Capture(dataDir, host), <the park-local chore id queued at checking>, id, previousStamp)` for a warm database, files directory and `<host>.env` of an earlier stay (previous stamp = the guard record's retired stamp; when none, the chore is acknowledged empty with `AcknowledgeEmpty`); the test asserts on the filesystem that the old files are under `.parked/` and gone from the live paths, not on the call returning nil; `archive.Open` + `Extract` the newest archive from `FilesPrefix` (by the seal's files version) into `<DataDir>/<host>/` and `<DataDir>/<host>.env`; `ReplaceTimers(host, snapshot.Rows)` from the sealed timers version; `SetReplication(host, Source.Replication)`, `SetNextDue(host, Source.NextDue)`; `RemoveVerifyEntry`; fsync the directory; write `.files-gen` `{"op": id, "incarnation": inc}` last. |
| + - `local`, in this order (round-2 finding 8; `park/local.go:202` requires paths **relative to `DataDir`**, tags them with the captured stamp and refuses a path already tagged with another stamp): (1) `captured, err := hostguard.Capture(dataDir, host)` (now `F`); (2) when the journal's `previous` names `S'` and `Inventory[cfg.Box]` has paths tagged `S'`: `cfg.Local.RetireArtifacts(ctx, host, captured, <the park-local chore id>, id, S')` parks every artifact tagged `S'` (the old `<host>.db`, `<host>.db.gen`, `<host>/`, `<host>.env`) into `.parked/<inc>/<host>/<retirement>/artifacts/`, which clears their inventory entries; when there is no `S'`, `AcknowledgeEmpty`; the test asserts on the filesystem that the old files are gone from the live paths and present under `.parked/`, not on the call returning nil; (3) `archive.Open` + `Extract` the newest archive from `FilesPrefix` (by the seal's files version) into `<DataDir>/<host>/` and `<DataDir>/<host>.env`; (4) `cfg.Local.RecordInventory(ctx, host, captured, "flat", []string{host + ".db", host + ".db.gen", host, host + ".env"}, check)` registers the new installation under `F`; (5) `ReplaceTimers(host, snapshot.Rows)` from the sealed timers version (through `SelectedTimers.ReplaceTimers` once C3 lands; `registry.ReplaceTimers` directly behind the switch until then); `SetReplication(host, Source.Replication)`, `SetNextDue(host, Source.NextDue)`; `RemoveVerifyEntry`; (6) the files manifest of spec rev 7.19 §7.6 (L17) through C8's publisher when it exists (a journal step, skipped with a log line until then); (7) fsync the directory; write `.files-gen` `{"op": id, "incarnation": inc}` last. |
| - `configured`: `RenderHeld`; `MaterializeHeld(rt, rec.Move.Inputs.Release)`; the ack's `BundleETags` must equal `rec.Move.Inputs.BundleETags` and its overlay etag `Inputs.OverlayETag`, else `MoveRollBack` (Plan 4 Task 1's transition; until it exists, write `Reason` and stop: `markForRollback(rec, "configuration changed during the move")` is the one call site Plan 4 replaces). |
| - `restored`: copy the built database to the row's `DBPath` (flat), sha256 == manifest's, `lease.CheckDB`, write `<dbPath>.gen` `{"op": id, "incarnation": inc, "location": holder.location}`, `MoveLandingPhase(configured → restored, stamp Landing.InstalledDBSHA256)`. |
| - `landing_deadline` passed before `restored` → the rollback marker as above. |
| - `armMovesPass` wiring in `agent.go` for the new hooks; the hold view (Task 6) is updated from both passes. |
| -Tests (`movesland_test.go`, the harness with `cfg.Region = "uk"`, a flipped record and a `landing/checking` claim with `exec = uk-box`, the sealed replica staged in the harness's versioned store from a real small sqlite replica built as in Task 8's test): `TestLandingCheckingRequiresClaimEvidenceAndTouch`, `TestLandingParkingWaitsForTheManifest` (an earlier stay's object under `holder.location` is archived to `superseded/<id>/...` before `building`), `TestLandingBuildsInstallsAndRestores` (end to end through `restored`: `.files-gen` last, timers replaced with registrations preserved, `<dbPath>.gen` written, `installed_db_sha256` recorded), `TestLandingMarkersAreWrittenLast` (witness `wMarkersLast`: a crash injected before the final sub-step of `local` leaves no `.files-gen`; the replay finishes it), `TestLandingConfiguredPinsInputs` (Review Focus 4), `TestLandingDeadlineMarksRollback`, `TestLandingReplaysArePhaseIdempotent` (every phase run twice yields one object per key). |
| +Tests (`movesland_test.go`, the harness with `cfg.Region = "uk"`, a flipped record and a `landing/checking` claim with `exec = uk-box`, the sealed replica staged in the harness's versioned store from a real small sqlite replica built as in Task 8's test, and a fake `park.Leases` that **refuses a second acquisition of a held location**, so a lease the landing still held would fail the test): `TestLandingCheckingRequiresClaimEvidenceAndTouch`, `TestLandingParkingWaitsForTheManifest` (an earlier stay's objects under `holder.location` in two generations are archived to `parked/<inc>/<host>/<ret>/...` before `building`, by two `park-generation` chores; after the probe publishes the new generation, `Engine.Run` on those chores past the horizon acknowledges rather than refusing), `TestLandingPhaseCatchUp` (round-2 finding 11: a crash after the record's `parking` and before the claim's; the restarted worker advances the claim to `parking` and continues; a claim left at `checking` with the record at `restored` is walked forward step by step), `TestLandingInventoryOrder` (an earlier stay's inventory tagged `S'`: `local` parks it, then registers the new paths under `F`; registering first is refused by the real `RecordInventory`), `TestLandingBuildsInstallsAndRestores` (end to end through `restored`: `.files-gen` last, timers replaced with registrations preserved, `<dbPath>.gen` written, `installed_db_sha256` recorded), `TestLandingMarkersAreWrittenLast` (witness `wMarkersLast`: a crash injected before the final sub-step of `local` leaves no `.files-gen`; the replay finishes it), `TestLandingConfiguredPinsInputs` (Review Focus 4), `TestLandingDeadlineMarksRollback`, `TestLandingReplaysArePhaseIdempotent` (every phase run twice yields one object per key). |
| - [ ] **Steps 1 to 4**. Run: `nice -n 10 go test ./internal/host -run 'TestLanding' -count=1` → PASS. |
| - [ ] **Step 5: Commit**: `git commit -m "host: the landing from checking to restored: park, build, local install, held render and materialization (§7.6)" ...` |
diff --git a/docs/superpowers/plans/2026-10-06-moves-phase1-3-probe-commit-open.md b/docs/superpowers/plans/2026-10-06-moves-phase1-3-probe-commit-open.md| index 99ccd71a9..25fa2e0a1 100644 |
| --- a/docs/superpowers/plans/2026-10-06-moves-phase1-3-probe-commit-open.md |
| +++ b/docs/superpowers/plans/2026-10-06-moves-phase1-3-probe-commit-open.md |
| @@ -4,7 +4,7 @@ |
| **Goal:** Prove the target can run the instance (the probe as a cold admission into the prepared generation, within the budgeted attempts and the cut deadline), commit by one record CAS, let the claim follow with its obligations, open the target row, and park the source. |
| -**Architecture:** The probe policy is pure (`internal/moves/probe.go`): attempt windows, the retry backoff, deadline checks before the start and before readiness. `activator.ProbeWake` runs claim Plan 6's admitted-writer start sequence for a held row with that policy, spawning strictly with `CARLOS_MOVE_PROBE=1` and requiring the readiness answer from the spawned tree. The commit is `provision.MoveCommit` conditional on `now ≤ cut_deadline`, then `hostclaim.Commit` queues `park-superseded` for the source and `park-staging` for the target and covers the source region's predecessors. The open is `registry.ReleaseHold`. The source's finalization retires its installation under the host guard and records `source_parked_at`. |
| +**Architecture:** The probe policy is pure (`internal/moves/probe.go`): attempt windows, the retry backoff, deadline checks before the start and before readiness. `activator.ProbeWake` runs claim Plan 6's admitted-writer start sequence for a held row with that policy, spawning strictly with `CARLOS_MOVE_PROBE=1` and requiring the readiness answer from the spawned tree. The commit is `provision.MoveCommit` conditional on `now ≤ cut_deadline`, then `hostclaim.Commit` queues a location `park-replica` for the source and a lane `park-replica` for the target and covers the source region's predecessors. The open is `registry.ReleaseHold`. The source's finalization retires its installation under the host guard and records `source_parked_at`. |
| **Tech Stack:** Go, `internal/hostbox` (claim Plan 6), `internal/hostclaim`, `internal/genselect`, `internal/park`, `internal/hostguard`, unix sockets with `SO_PEERCRED` (`golang.org/x/sys/unix`), cgroup v2. |
| @@ -17,6 +17,7 @@ |
| - A landing has at most three attempts and never more than T11 admitted; a probe never marks the host failed (§7.7). |
| - The commit is a record CAS conditional on `now ≤ cut_deadline`; the claim follows unconditionally; `holder.generation` is never written by the commit (R10-3). |
| - The open is the conditional `ReleaseHold(host, moveID)`; the target serves on the session its probe was admitted with; nothing rolls back after a commit. |
| +- Cold only (P7, index D14): every probe and every wake after the open is a cold admission; nothing here reads a certificate. The switch (D13) gates the probe and the open. |
| - No em dashes. |
| ## Review Focus |
| @@ -177,7 +178,7 @@ with `MoveOpened`, `MoveSourceParked`, `MoveClose`, `MoveProbeAttempt` in the sa |
| **Files:** modify `internal/hostclaim/move.go`, test `internal/hostclaim/move_test.go`. |
| **Interfaces:** |
| -- `hostclaim.Commit(opID, sourceRegion, sourceBox, targetRegion string, sourceScope, stagingScope ParkScope, newID func() string) func(*Claim) error`: requires `KindLanding` with phase `probing` or `probed` and `Exec == participants[1]`; sets phase `"committed"`; appends `park-superseded` for `sourceRegion` with `sourceScope` (owner `region:<source>`), `park-local` for `sourceBox`, `park-staging` for `targetRegion` with `stagingScope`; runs `CoverRegionPredecessors(opID, "committed", sourceRegion)`; idempotent: a claim already at `committed` under this op returns `nil` with no second set of obligations (witness `wCommitReplayed`). `holder.generation` untouched. |
| +- `hostclaim.Commit(opID, sourceRegion, sourceBox, targetRegion string, sourceScope, stagingScope ParkScope, newID func() string) func(*Claim) error`: requires `KindLanding` with phase `probing` or `probed` and `Exec == participants[1]`; sets phase `"committed"`; appends location `park-replica` for `sourceRegion` with `sourceScope` (owner `region:<source>`), `park-local` for `sourceBox`, lane `park-replica` for `targetRegion` with `stagingScope`; runs `CoverRegionPredecessors(opID, "committed", sourceRegion)`; idempotent: a claim already at `committed` under this op returns `nil` with no second set of obligations (witness `wCommitReplayed`). `holder.generation` untouched. |
| - `PhaseCommitted = "committed"`. |
| - [ ] **Step 1: Write the failing test** (witnesses `wCommitCoverage`, `wCommitInterrupted`, `wCommitReplayed`): |
| @@ -216,7 +217,7 @@ func TestCommitQueuesObligationsAndCoversTheSourceRegion(t *testing.T) { |
| for _, o := range c.Obligations { |
| kinds[o.Kind+"/"+o.Region+o.Box]++ |
| } |
| - if kinds["park-superseded/ie"] != 1 || kinds["park-local/ie-box"] != 1 || kinds["park-staging/uk"] != 1 || len(c.Obligations) != 3 { |
| + if kinds["park-replica/ie"] != 1 || kinds["park-local/ie-box"] != 1 || kinds["park-replica/uk"] != 1 || len(c.Obligations) != 3 { |
| t.Fatalf("obligations: %+v", c.Obligations) |
| } |
| if !c.Holder.Admissions[0].Covered { |
| @@ -242,8 +243,8 @@ func TestCommitQueuesObligationsAndCoversTheSourceRegion(t *testing.T) { |
| const PhaseCommitted = "committed" |
| // Commit is the claim half of the record's commit (§7.5, §7.9 step 1): |
| -// phase committed plus, in the same write, park-superseded for the |
| -// source's location in its region and box, park-staging for the target's |
| +// phase committed plus, in the same write, a location park-replica for the |
| +// source's location in its region and box, a lane park-replica for the target's |
| // staging, and CoverRegionPredecessors for the source region. It writes |
| // nothing into holder.generation (R10-3). Idempotent under the op. |
| func Commit(opID, sourceRegion, sourceBox, targetRegion string, sourceScope, stagingScope ParkScope, newID func() string) func(*Claim) error { |
| @@ -272,9 +273,9 @@ func Commit(opID, sourceRegion, sourceBox, targetRegion string, sourceScope, sta |
| c.Op.Phase = PhaseCommitted |
| s, g := sourceScope, stagingScope |
| c.Obligations = append(c.Obligations, |
| - Obligation{ID: newID(), Op: opID, Kind: "park-superseded", Region: sourceRegion, Retirement: newID(), Owner: "region:" + sourceRegion, Park: &s}, |
| + Obligation{ID: newID(), Op: opID, Kind: "park-replica", Region: sourceRegion, Retirement: newID(), Owner: "region:" + sourceRegion, Park: &s}, |
| Obligation{ID: newID(), Op: opID, Kind: "park-local", Box: sourceBox, Retirement: newID(), Owner: "box:" + sourceBox}, |
| - Obligation{ID: newID(), Op: opID, Kind: "park-staging", Region: targetRegion, Retirement: newID(), Owner: "region:" + targetRegion, Park: &g}) |
| + Obligation{ID: newID(), Op: opID, Kind: "park-replica", Region: targetRegion, Retirement: newID(), Owner: "region:" + targetRegion, Park: &g}) |
| return CoverRegionPredecessors(opID, PhaseCommitted, sourceRegion)(c) |
| } |
| } |
| @@ -297,19 +298,19 @@ func Commit(opID, sourceRegion, sourceBox, targetRegion string, sourceScope, sta |
| --- |
| -### Task 5: `ProbeWake` (needs claim Plan 6) |
| +### Task 5: `ProbeWake` (BLOCKED-ON-CLAIM: C1, C2) |
| **Files:** create `internal/activator/probewake.go`, `internal/activator/peercred_linux.go`, `internal/activator/peercred_other.go`, `internal/activator/probewake_test.go`. |
| **Interfaces:** |
| - Consumes (claim Plan 6): `hostbox.Writer` (`Baseline`, `Admit`, `Published`), `hostbox.GenerationPrefix`, `spawnClaimed(ctx, enabled, manager, host, limit, selected, identity, build) (*exec.Cmd, *cgroup.Prepared, error)` (the `*cgroup.Prepared` is kept for the process's life and, on every exit path including a failed start, `KillAndEmpty` then `Close` run before the attempt is counted), and Plan 6 Task 9's admitted-start sequence exposed as `(a *Service) startAdmitted(ctx, e *actEntry, admitted hostclaim.Observed, policy startPolicy) error` (index: "exact interfaces needed"). Consumes `genselect.SelectPrepared(ctx, s, req, genselect.PreparedPin{Generation, SnapshotKey, SnapshotVersion, SnapshotTXID, SnapshotCRC64})` to plan the prepared generation's restore from the install manifest. |
| -- Produces: `activator.ProbeConfig{Host, Op string; Kind string ("landing" | "recovery"); Prepared *moves.InstallManifest (nil for a recovery probe into the source's own generation); Seal *moves.Seal; CutDeadline time.Time; Attempt moves.ProbeAttempt; RenderHeld func(ctx) error; ReadyPath string; LossID string (empty except for a lost-target return, Plan 4 Task 10, where the admission goes through hostclaim.AdmitWriterWithLoss)}`; `activator.ProbeResult{Session uint64; Generation string; AnsweredAt time.Time}`; `(a *Service) ProbeWake(ctx, cfg ProbeConfig) (ProbeResult, error)`: |
| - 1. the row must be held with `hold_op == cfg.Op` and the claim must name this box `op.exec` (K1, K6), else `ErrProbeRefused`; |
| - 2. `cfg.RenderHeld(ctx)` (the caller composes `RenderHeld` + `MaterializeHeld`; every probe kind, model 4); |
| - 3. **prepared-input validation**, before any admission: for a landing probe, `.files-gen` and `<dbPath>.gen` must name the prepared generation (`cfg.Op`) and the holder's incarnation and location, and `cfg.Prepared` (the install manifest) must exist with `N == cfg.Seal.N`; this is not the §5.1 start check, which compares against `holder.generation`: before the first probe's admission `holder.generation` is still the source's (the flip preserves it, §7.5; only `AdmitWriter` moves it, `generation.go:27`), so a start check here would refuse the landing's own files. For a recovery probe, `.files-gen` and `<dbPath>.gen`, when present, must name `holder.generation` as observed (the source's own, or the return generation after a dirty install); |
| - 4. `cfg.Attempt.MayStart(now, cfg.CutDeadline)`; |
| +- Produces: `activator.ProbeConfig{Host string; Op string (the claim's op id: the move id, in every kind); HoldOp string (the row's hold_op: the move id for a landing or revert, the rollback id for a rollback's source); PreparedGeneration string (the generation a prepared install produced and the admission opens: the move id for a landing, the rollback id for a dirty return, "" for a clean recovery into the source's own generation); Kind string ("landing" | "recovery"); Prepared *moves.InstallManifest (non-nil exactly when PreparedGeneration != ""); Seal *moves.Seal; Inputs *provision.MoveInputs (the pin a landing probe enforces at spawn; nil for a recovery probe, which records what it rendered); CutDeadline time.Time; Attempt *moves.ProbeAttempt (Index set by the caller; Opened written by the probe at the first app start); RenderHeld func(ctx) (appconfig.HeldRender, error); ReadyPath string; LossID string (empty except for a lost-target return, Plan 4 Task 10, where the admission goes through hostclaim.AdmitWriterWithLoss)}`; `activator.ProbeResult{Session uint64; Generation string; AnsweredAt time.Time}`; `(a *Service) ProbeWake(ctx, cfg ProbeConfig) (ProbeResult, error)`: |
| + 1. the row must be held with `hold_op == cfg.HoldOp` and the claim must name this box `op.exec` under op `cfg.Op` (K1, K6), else `ErrProbeRefused`; |
| + 2. `render, err := cfg.RenderHeld(ctx)` (the caller composes `RenderHeld` + `MaterializeHeld`; every probe kind, model 4); **the launch-input pin, at spawn** (§7.7, round-2 finding 14): for a landing probe (`cfg.Inputs != nil`) the rendered `BundleETags` and `OverlayETag` must equal `cfg.Inputs`', the materialized release must be `cfg.Inputs.Release`, and `moves.ReportEnv(key, b.LaunchEnvironment(rt)).Digest` must equal `cfg.Inputs.EnvReport`; a mismatch is `ErrProbeFailed("configuration changed during the move")` before any admission, and the landing rolls back; a recovery probe records what it rendered into `Rollback.RecoveryInputs` (through the caller's `RecordInputs` hook) and is bound to nothing (J2-R2-5). Plan 2's `configured` check stays as the early refusal; this one is the authoritative one, since the environment can change between the two; |
| + 3. **prepared-input validation**, before any admission, when `cfg.PreparedGeneration != ""` (a landing, or a dirty return): `moves.PreparedCheck(cfg.PreparedGeneration, holder, filesGen, dbGen)`: `.files-gen` and `<dbPath>.gen` name that generation and the holder's incarnation and location, and `cfg.Prepared` exists with `N == cfg.Seal.N`. This is not the §5.1 start check against `holder.generation`: before the first probe's admission `holder.generation` is still the predecessor's (the source's after a flip, the target's after a dirty `ReturnToSource`, which deliberately keeps it, Plan 4 Task 2; only `AdmitWriter` moves it, `generation.go:27`), so a start check here would refuse the install's own files (round-2 finding 13). When `cfg.PreparedGeneration == ""` (a clean recovery or a revert), the markers, when present, must name `holder.generation` as observed; |
| + 4. `cfg.Attempt.MayStart(now, cfg.CutDeadline)` is checked with `Opened` **unset** (the window has not begun); the cut deadline alone applies here; |
| 5. the admission: `decide` per §7.14: cold; `NewGeneration = cfg.Op` when `holder.generation` is neither `cfg.Op` nor a descendant admitted from it, else a fresh id (`newID()`); `KeepStamp = holder.stamp > holder.session`; `Coverage = "sealed"` when `cfg.Seal != nil && cfg.Seal.FromDB` and the generation is the prepared one, else `""`; `Region`, `PredecessorRegion` per the last admission record; `RemoteMaxTXID` from the baseline plan (`SelectPrepared` for the prepared generation; `Writer.Baseline` for a fresh one); then `Writer.Admit`; |
| - 6. `startAdmitted` with `startPolicy{MaxStarts: moves.ProbeMaxStarts, Window: moves.ProbeWindow, Deadline: min(attempt deadline, cut deadline), Probe: true}`; **the §5.1 start check runs here, inside the admitted sequence**, with `moves.StartCheck(claimAfterCAS, marker, filesGen, dbGen)` against the generation the admission CAS just named (`holder.generation == cfg.Op` for a first probe; a fresh id for a restarted one, whose `.files-gen` still names `cfg.Op` and is accepted through the admission chain: Task 4's `StartCheck` takes the claim and walks `holder.admissions`); the spawn goes through `spawnClaimed` with `CARLOS_MOVE_PROBE=1` in the child's environment and is strict; |
| + 6. `startAdmitted` with `startPolicy{MaxStarts: moves.ProbeMaxStarts, Window: moves.ProbeWindow, Deadline: cfg.CutDeadline, Probe: true, Prepared: cfg.Prepared}` (C1): the sequence purges and restores **before** the window opens, then sets `cfg.Attempt.Opened = now` at the first app start and journals it (round-2 finding 19: the 30 s window is the app's, the restore is priced separately in T11); **the §5.1 start check runs here, inside the admitted sequence**, with `moves.StartCheck(claimAfterCAS, marker, filesGen, dbGen)` against the generation the admission CAS just named (`holder.generation == cfg.Op` for a first probe; a fresh id for a restarted one, whose `.files-gen` still names `cfg.Op` and is accepted through the admission chain: Task 4's `StartCheck` takes the claim and walks `holder.admissions`); the spawn goes through `spawnClaimed` with `CARLOS_MOVE_PROBE=1` in the child's environment and is strict; |
| 7. readiness: `GET cfg.ReadyPath` over the row's socket with `Host: <host>`, 200 to 399, the peer's `SO_PEERCRED` pid a member of `carlos-exec@<host>` (`cgroup.Cleanup.Members`), `cfg.Attempt.AcceptReadiness(answeredAt, cfg.CutDeadline)`; a 20 s settle, the process still running, a second request; |
| 8. the marker and publication are `startAdmitted`'s (first upload, `generation.json` by If-Match or If-None-Match, `Writer.Published`); the entry stays held: `RetireExcept` keeps entries whose row `IsHold`. |
| A failure after the admission stops the tree (`FenceAndStop` with a fresh generation) and returns `ErrProbeFailed` with the reason; the attempt is over; the operation decides what next. |
| @@ -322,27 +323,27 @@ Tests: `TestProbeRefusesUnheldRowOrForeignExec`; `TestFirstProbeFromTheSourceGen |
| --- |
| -### Task 6: The landing's `probing` and `probed` (needs claim Plan 6) |
| +### Task 6: The landing's `probing` and `probed` (BLOCKED-ON-CLAIM: C1) |
| **Files:** modify `internal/host/movesland.go`, `internal/host/movesland_test.go`. |
| **Interfaces:** |
| - `moveCfg.Probe func(ctx, activator.ProbeConfig) (activator.ProbeResult, error)`; `moveCfg.RenderAndMaterialize func(ctx, rt registry.Route, op, version string) (appconfig.HeldRender, error)`. |
| -- `landingPass` phases: `restored → probing` (`MoveLandingPhase`, then the claim's `SetMovePhase(id, KindLanding, "restored", "probing")`, Plan 2 Task 10's mirroring rule, so the admission's `ObservedPhase` is `probing`); in `probing`, attempts from the journal (`Journal.Attempt`): `n = AttemptsFor("landing", targetReport.AttemptsAdmitted)`; for each attempt not yet recorded: `MoveProbeAttempt(g, i)`, journal `Attempt = i` and `Opened = now` before the call (a crash restarts only the probe, §7.13; the journalled attempt whose window passed counts as failed), `cfg.Probe(...)`; success → `MoveLandingPhase(probing → probed)` then `SetMovePhase(id, KindLanding, "probing", "probed")`; `ErrAttemptOver`/`ErrProbeFailed` → next attempt, if any; attempts exhausted or `cut_deadline` passed → the rollback marker (Plan 4 Task 1's `MoveRollBack`; until it lands, `markForRollback`). |
| +- `landingPass` phases: `restored → probing` (`MoveLandingPhase`, then the claim's `SetMovePhase(id, KindLanding, "restored", "probing")`, Plan 2 Task 10's mirroring rule, so the admission's `ObservedPhase` is `probing`); in `probing`, attempts from the journal (`Journal.Attempt`): `n = AttemptsFor("landing", targetReport.AttemptsAdmitted)`; for each attempt not yet recorded: `MoveProbeAttempt(g, i)` and journal `Attempt = i` before the call (the attempt is accounted here; its `Opened` is written by the probe at the first app start, Task 5 step 6, so the restore does not eat the window; a crash restarts only the probe, §7.13, and a journalled attempt with an `Opened` whose window passed counts as failed), `cfg.Probe(...)`; success → `MoveLandingPhase(probing → probed)` then `SetMovePhase(id, KindLanding, "probing", "probed")`; `ErrAttemptOver`/`ErrProbeFailed` → next attempt, if any; attempts exhausted or `cut_deadline` passed → the rollback marker (Plan 4 Task 1's `MoveRollBack`; until it lands, `markForRollback`). |
| - Tests: `TestLandingProbesWithinTheBudget` (two admitted attempts: the first fails, the second passes → `probed`; a third is never attempted); `TestLandingProbeCrashRestartsOnlyTheProbe` (a fresh worker on a journal at `probing` with `Attempt = 1` does not rerun `local` or `restored`: their journal steps are untouched and no second `.files-gen` write happens); `TestLandingExhaustedAttemptsMarkRollback`. |
| - [ ] **Steps 1 to 5**. Commit: `git commit -m "host: the landing's probing and probed phases (§7.6, §7.7)" ...` |
| --- |
| -### Task 7: Commit and open on the target (Tasks 5 and 6 first) |
| +### Task 7: Commit and open on the target (BLOCKED-ON-CLAIM: C1, C2; Tasks 5 and 6 first) |
| **Files:** modify `internal/host/movesland.go`, `internal/host/movesland_test.go`; modify `internal/registry/registry.go` (`DeleteIfHold`). |
| **Interfaces:** |
| - In `landingPass`, at `probed`: `provision.SetRecordMove(MoveCommit(g, cfg.Now()))`; `ErrCutDeadline` → the rollback marker; success or an already-`committed` record → the claim half: `hostclaim.Update(Commit(id, rec.Move.From, sourceBox, cfg.Region, sourceScope, stagingScope, newID))` where `sourceScope = {Location: TrimSuffix(holder.location, "/"), Incarnation, EvidenceAt: the cut's stop evidence time (`stopped-source.json`'s `CheckedAt`), RequestLimit: 120 s}` and `stagingScope = {Location: moves.ScopeLocation(host, id), Incarnation, EvidenceAt: seal.Bound(), RequestLimit: 120 s}` (the staging lane's own seal bound, claim rev 6.23; Plan 5 Task 3 proves the deadline); the claim half is written by whoever sees the record ahead (§7.5): this pass, or the console's pass (Plan 5). |
| - The open: `cfg.ReleaseHold(host, id)` (`registry.ReleaseHold`); `changed == false` with the row already `ready` and `hold_op == ""` is a replay and fine; `MoveOpened(g, now)`; `nudgeSerial`; the activator's entry from the probe is the serving process: no restart. |
| -- `registry.DeleteIfHold(host, op string) (bool, error)`: `DELETE FROM routes WHERE host = ? AND hold_op = ?` (the source's row deletion in Task 8, conditional on `hold_op = move id`, §7.9 step 3). |
| +- `registry.DeleteIfHold` is Plan 1 Task 13's (D12); Task 8 uses it without depending on this task. |
| Tests: `TestCommitThenOpenInOrder` (witness `wCommitBeforeOpen`: driven from the flip through every landing phase with the real transitions, no fixture phase edits; the row is `ready` only after the record says `committed` and the claim says `committed`); `TestOpenAfterCrashDoesNotReprobe` (Review Focus 3: a worker restarted on a record at `committed` with the claim still `landing/probed` writes the claim half, opens, and never calls `Probe`); `TestLateCommitRollsBack` (Review Focus 2). |
| @@ -355,14 +356,14 @@ Tests: `TestCommitThenOpenInOrder` (witness `wCommitBeforeOpen`: driven from the |
| **Files:** create `internal/host/movesfinal.go`, `internal/host/movesfinal_test.go`; modify `internal/host/movespass.go` (records in `committed` whose `From == cfg.Region` run `finalizeSource`). |
| **Interfaces:** |
| -- `finalizeSource(ctx, s, w, rec) error` (§7.9 step 3), on the source box, only when the record is `committed` and this box's row has `hold_op == rec.Move.ID` (Review Focus 5): |
| - 1. the remote half: `cfg.Park.Run(ctx, park.Work{Host, ID: <the park-superseded obligation for cfg.Region under this op>, Op: id})` while holding the lease on its own `instances/<host>/` (the engine takes `Leases.Acquire` itself); `ErrPending` is fine: the copies and version deletes are done when `cfg.Park.Parked` is true; acknowledgement after the horizon is the regional runner's (claim Plan 6's `hostbox.Regional.Run`, kinds widened per the index); |
| +- `finalizeSource(ctx, s, w, rec) error` (§7.9 step 3), on the source box, only when the record is `committed` and **either** this box's row has `hold_op == rec.Move.ID` **or** the op journal (`moves.Journal` for this op) has `Steps["row-deleted"]` (round-2 finding 12: a crash between the row's deletion and `MoveSourceParked` must not lock the close out; a replacement row with another `hold_op` is refused by the first condition and does not match the second); Review Focus 5: |
| + 1. the remote half: `cfg.Park.Run(ctx, park.Work{Host, ID: <the park-replica obligation for cfg.Region under this op, D6>, Op: id})` (the engine takes `Leases.Acquire` itself; the source holds no lease of its own here); `ErrPending` is fine: the copies and version deletes are done when `cfg.Park.Parked` is true; acknowledgement after the horizon is the regional runner's (C4); |
| 2. the local half under the host guard: `cfg.Local.RetireInstallation(ctx, host, hostguard.Capture(dataDir, host), <park-local obligation id>, id)` parks the database with `<dbPath>.gen`, the files directory, `<host>.env`, `<db>.corrupt` and the op journals into `.parked/superseded-<id>/<host>/<retirement>/` (the destination `park.Local` derives from the obligation's op; if `park.Local` hardcodes `.parked/<inc>/...`, add the `superseded-<op>` form beside Task 9's remote destination); `CompleteObligation(opID, parkLocalID)` through `hostclaim.Update`; |
| - 3. `cfg.ReplaceTimers(host, nil)` (clears the timers rows), `cfg.DeleteIfHold(host, id)` (the row); the source box's config ack (`appconfig.ConfigAckKey(cfg.Box, host)`, §6.3) is copied to `moves.ReportPrefix(...) + "config-ack-source.json"` and deleted; |
| + 3. `cfg.ReplaceTimers(host, nil)` (clears the timers rows); journal `Steps["row-deleted"] = "pending"`, then `cfg.DeleteIfHold(host, id)` (the row), then `Steps["row-deleted"] = "done"` (a replay that finds `pending` and no row with this `hold_op` treats the delete as done); the source box's config ack (`appconfig.ConfigAckKey(cfg.Box, host)`, §6.3) is copied to `moves.ReportPrefix(...) + "config-ack-source.json"` and deleted; |
| 4. `MoveSourceParked(g, now)`; `nudgeSerial`. |
| Every step journaled in `moves.Journal`; replayable. |
| -Tests: `TestSourceParkWaitsForCommit` (Review Focus 5: a `landing` record, or a `committed` one whose row's `hold_op` is another op → nothing parked, nothing deleted); `TestSourceParkRetiresInstallationAndRow` (the database, `.gen`, files dir and env are gone from live paths and present under `.parked/superseded-<id>/`; the row is gone; timers rows gone; `source_parked_at` set; the `park-local` obligation acknowledged); `TestSourceParkReplayIsIdempotent`. |
| +Tests: `TestSourceParkWaitsForCommit` (Review Focus 5: a `landing` record, or a `committed` one whose row's `hold_op` is another op → nothing parked, nothing deleted); `TestSourceParkSurvivesACrashAfterTheRowDelete` (round-2 finding 12: the journal says `row-deleted: pending`, the row is gone; the replay stamps `source_parked_at`); `TestSourceParkRetiresInstallationAndRow` (the database, `.gen`, files dir and env are gone from live paths and present under `.parked/superseded-<id>/`; the row is gone; timers rows gone; `source_parked_at` set; the `park-local` obligation acknowledged); `TestSourceParkReplayIsIdempotent`. |
| - [ ] **Steps 1 to 5**. Commit: `git commit -m "host: the source's finalization park after the commit (§7.9 step 3)" ...` |
diff --git a/docs/superpowers/plans/2026-10-06-moves-phase1-4-rollback-return-declare-lost.md b/docs/superpowers/plans/2026-10-06-moves-phase1-4-rollback-return-declare-lost.md| index c1794046a..b8d97e89e 100644 |
| --- a/docs/superpowers/plans/2026-10-06-moves-phase1-4-rollback-return-declare-lost.md |
| +++ b/docs/superpowers/plans/2026-10-06-moves-phase1-4-rollback-return-declare-lost.md |
| @@ -19,7 +19,8 @@ |
| - A recovery probe renders current inputs and records them as `recovery_inputs`; it is never gated on `move.inputs` (J2-R2-5). |
| - The `claimed` CAS writes `holder.stamp = op.journal.source_stamp`; a dirty install issues a new stamp at its probe's cold admission and never keeps `S` (claim rev 6.5 §6.4). |
| - `verify` is never re-run once `probing` is journaled (INV-2, joint N11). |
| -- Declare-lost takes compute fencing only; an expired lease is never enough (model 5). |
| +- Declare-lost takes compute fencing only; an expired lease is never enough (model 5). A termination fence (spec rev 7.19 §7.10, region HA L14) is compute fencing with no operator, on the same path. |
| +- Cold only (P7, index D14): every recovery probe is a cold admission; `clean-return` coverage still applies to a clean return (it is about declarations, not warmth). The switch (D13) gates every entry point here. |
| ## Review Focus |
| @@ -54,12 +55,12 @@ |
| **Interfaces:** |
| - `MoveRollBack(g MoveGuard, rbID, reason string, now time.Time) func(*Record) error`: from `landing` (any landing phase) → `rolling_back`; `Region = From`; `Rollback = &MoveRollback{ID, RequestedAt, Reason, TargetPhase: RbTargetReleasing, SourcePhase: RbSourceWaiting, ReleaseDeadline: now + 15 min}`; refused from `committed` (`ErrMoveConflict`): the commit won. |
| - `MoveRollbackTargetPhase(g, from, to string, stamp func(*MoveRollback))` over the target order `releasing, released, returning, returned, cleaning, cleaned` (a clean release skips `returning`/`returned`: `released → cleaning` is legal when `!Dirty`); `MoveRollbackSourcePhase(g, from, to, stamp)` over `waiting, claimed, restoring, verified | parking, building, published, local, configured, restored, probing, probed, recovered, opened` (clean: `restoring → verified → probing`; dirty: `restoring → parking → … → restored → probing`). |
| -- `MoveStuck(g, reason)`: from `rolling_back` with the source at `probing` **or at `restoring`/`verified` (a verification mismatch, INV-4)**, or from `reverting` → `stuck`, recording `Rollback.StuckAt = SourcePhase`; `MoveRetry(g)`: `stuck → rolling_back` (the source phase returns to what `StuckAt` names: a mismatch retries verification, never bypasses it) or `→ reverting` (by `Rollback == nil`), clears `Reason`. |
| -- `MoveRollbackTargetLost(g, lossKey string, now)`: the declared-loss transition (§7.10): legal from target phase `releasing`, `returning`, or `released` with `Dirty` (the "released dirty, not returned" window); sets `TargetPhase = released`, `Dirty = false`, `Evidence = lossKey`, `LostAt = now`; the forward-only `MoveRollbackTargetPhase` never accepts `returning → released`, so the loss has its own closure. |
| +- `MoveMarkStuck(g, reason)` (the constant `MoveStuck` is the state, Plan 1 Task 2; round-2 finding 2): from `rolling_back` with the source at `probing` **or at `restoring`/`verified` (a verification mismatch, INV-4)**, or from `reverting` → `stuck`, recording `Rollback.StuckAt = SourcePhase`; `MoveRetry(g)`: `stuck → rolling_back` (the source phase returns to what `StuckAt` names: a mismatch retries verification, never bypasses it) or `→ reverting` (by `Rollback == nil`), clears `Reason`. |
| +- `MoveRollbackTargetLost(g, lossRef string, now)`: the declared-loss decision (§7.10; round-2 finding 15): legal from target phase `releasing`, `returning`, or `released` with `Dirty` (the "released dirty, not returned" window); sets `TargetPhase = RbTargetLost` (`"lost"`, a **distinct** phase), `Rollback.LossRef = lossRef` (the loss record key, or the termination fence record key, spec rev 7.19 §7.10), `LostAt = now`, and leaves `Dirty` as it was. The ordinary source recovery never consumes `lost`: its `waiting` phase requires `released` (clean) or `returned` (dirty), and the lost branch requires `lost` **and** the claim's `loss.<id>` (or C7's fence) journal entry. `MoveRollbackLossCommitted(g)`: `lost → released` with `Dirty = false`, `Evidence = LossRef`, legal only after the claim carries the declaration (the caller checks). The forward-only `MoveRollbackTargetPhase` never accepts `returning → released`, so the loss has its own closures. |
| - `MoveRolledBackClose(g, now)`: `rolling_back` with `RecoveredAt`, `OpenedAt`, `TargetParkedAt` set → `Move = nil` (region is `From` already) and `LastMove = &MoveOutcome{ID, Outcome: "rolled_back", At: now, OfflineS: Rollback.OpenedAt - CutAt}` (index D11). `MoveRevertClose(g, now)`: `reverting` with `rev_phase` opened (the caller checks the claim) → `Move = nil`, `LastMove{Outcome: "rolled_back"}` with `OfflineS` from `CutAt` to the reopen. |
| - `MoveLanding` with `landing_deadline` passed is the console sweep's trigger (Plan 5) through `MoveRollBack`. |
| -- [ ] **Step 1: Write the failing test**: the commit/rollback race (`MoveCommit` then `MoveRollBack` → `ErrMoveConflict`; `MoveRollBack` then `MoveCommit` → `ErrMoveConflict`); `MoveRollBack` sets region `ie`, the deadline and both phases; target phases in order, the clean skip, a dirty attempt to skip `returning` refused; `MoveRollbackTargetLost` legal from `releasing`, `returning` and `released`+`Dirty`, refused from `cleaning` and from `released` clean; source phases clean and dirty orders; `MoveStuck` from `verified` (mismatch) and from `probing`, and `MoveRetry` returning the source to the phase it was stuck in; `MoveRolledBackClose` refuses until all three stamps exist and writes `LastMove`. |
| +- [ ] **Step 1: Write the failing test**: the commit/rollback race (`MoveCommit` then `MoveRollBack` → `ErrMoveConflict`; `MoveRollBack` then `MoveCommit` → `ErrMoveConflict`); `MoveRollBack` sets region `ie`, the deadline and both phases; target phases in order, the clean skip, a dirty attempt to skip `returning` refused; `MoveRollbackTargetLost` legal from `releasing`, `returning` and `released`+`Dirty`, refused from `cleaning` and from `released` clean, lands in `lost` with `Dirty` untouched; `MoveRollbackLossCommitted` from `lost` only; source phases clean and dirty orders; `MoveMarkStuck` from `verified` (mismatch) and from `probing`, and `MoveRetry` returning the source to the phase it was stuck in; `MoveRolledBackClose` refuses until all three stamps exist and writes `LastMove`. |
| - [ ] **Steps 2 to 4**. **Step 5: Commit**: `git commit -m "provision: rollback, revert, stuck and retry transitions (§7.8, §7.10)" ...` |
| --- |
| @@ -70,8 +71,8 @@ |
| **Interfaces:** |
| - `RollBack(opID, rbID string) func(*Claim) error`: `KindLanding` (any phase but `committed`) → `KindRestoring`, phase `"releasing"`, `Intent["rollback"] = rbID`; exec unchanged (the target still owes evidence). |
| -- `TargetReleased(opID, targetBox, targetRegion, evidenceKey string, scope ParkScope, newID func() string) func(*Claim) error`: `KindRestoring`, `Exec == targetBox` → `ExecToNone(... JournalEvidenceTarget)`, phase `"released"`, append `park-superseded` for `targetRegion` with `scope` (owner `region:<target>`) and `park-local` for `targetBox`, then `CoverRegionPredecessors(opID, "released", targetRegion)` (claim rev 6.14 item 6, R12-3). |
| -- `ReturnToSource(opID, sourceBox string, clean bool) func(*Claim) error` (§7.8 `claimed`, K4): `KindRestoring`; `Exec == ""` **or `"none"`** (claim Plan 8's `declareLostTransition` writes the literal `"none"`, line 1511); either `evidence_target` journalled **or** a `loss.<id>` journal entry present (a declared-lost target gives no evidence; the declaration is the cited fence, §7.10); `sourceBox == participants[0]` (the lost target has been removed from `participants` by the declaration, so the source may be the only participant left) → `Exec = sourceBox`, phase `"claimed"`, `holder.stamp = ParseUint(journal[source_stamp])`; when `clean`, `holder.generation = journal[src_gen]`; `holder.admissions` untouched (R11-4). `TargetReleased` is not called for a declared-lost target: the console's resolve door queues the target region's `park-superseded` and runs `CoverRegionPredecessors` itself (Task 10), since no box can give evidence for the lost one. |
| +- `TargetReleased(opID, targetBox, targetRegion, evidenceKey string, scope ParkScope, newID func() string) func(*Claim) error`: `KindRestoring`, `Exec == targetBox` → `ExecToNone(... JournalEvidenceTarget)`, phase `"released"`, `QueueLocationPark(opID, targetRegion, targetBox, scope, newID)` (a `park-replica` scoped to the whole location plus the box's `park-local`, D6), then `CoverRegionPredecessors(opID, "released", targetRegion)` (claim rev 6.14 item 6, R12-3). |
| +- `ReturnToSource(opID, sourceBox string, clean bool) func(*Claim) error` (§7.8 `claimed`, K4): `KindRestoring`; `Exec == ""` **or `"none"`** (claim Plan 8's `declareLostTransition` writes the literal `"none"`, line 1511); either `evidence_target` journalled **or** a `loss.<id>` journal entry present (a declared-lost target gives no evidence; the declaration is the cited fence, §7.10); `sourceBox == participants[0]` (the lost target has been removed from `participants` by the declaration, so the source may be the only participant left) → `Exec = sourceBox`, phase `"claimed"`, `holder.stamp = ParseUint(journal[source_stamp])`; when `clean`, `holder.generation = journal[src_gen]`; `holder.admissions` untouched (R11-4). `TargetReleased` is not called for a declared-lost target: the console's resolve door queues the target region's location `park-replica` and runs `CoverRegionPredecessors` itself (Task 10), since no box can give evidence for the lost one. |
| - `ReleaseMove(opID, execBox string) func(*Claim) error` (K8): kind in `landing/committed`, `restoring/opened`, `reverting/opened` or `moving/aborting`; no obligation with `Op == opID` remains; sets `Op = Op{Kind: KindServing, Exec: execBox}` (participants cleared; journal dropped; the admissions stay in the holder). |
| - `SetRevPhase(opID, from, to string)` (`rev_phase` in the journal under `KindReverting`, order `evidence, verify, probing, probed, opened`). |
| - `QueueReturnLanePark(opID, sourceRegion string, scope ParkScope, newID)` (dirty: the lane `moves/<host>/<rb id>` in the source's region, queued at `probed`, §7.8). |
| @@ -86,7 +87,7 @@ |
| **Files:** create `internal/host/movesinstall.go`, `internal/host/movesinstall_test.go`; modify `internal/host/movesland.go` (the landing's `parking` to `restored` call `installGeneration`). |
| **Interfaces:** |
| -- `installSource{OpID, Host string; Lane string (the staging or return-lane prefix in this box's own bucket); Seal moves.Seal; GenerationOp string (the generation name: the move id for a landing, the rollback id for a dirty return); Destination string (the superseded op id for what the install parks); PinInputs *provision.MoveInputs (nil for a recovery install); Phase func(from, to string, stamp func(*provision.MoveState)) func(*provision.Record) error (MoveLandingPhase or MoveRollbackSourcePhase bound to the guard); Phases [6]string (the six phase names of this install in order)}`. |
| +- `installSource{OpID, Host string; Lane string (the staging or return-lane prefix in this box's own bucket); Seal moves.Seal; GenerationOp string (the generation name: the move id for a landing, the rollback id for a dirty return); PinInputs *provision.MoveInputs (nil for a recovery install); Phase func(from, to string, stamp func(*provision.MoveState)) func(*provision.Record) error (MoveLandingPhase or MoveRollbackSourcePhase bound to the guard); Phases [6]string (the six phase names of this install in order)}`. |
| - `installGeneration(ctx, s, w, rec, src installSource) (done bool, rollbackReason string, err error)` runs `parking → building → published → local → configured → restored` exactly as Plan 2 Task 10 wrote them, with two differences for a recovery install: `configured` renders current inputs and records them in `Rollback.RecoveryInputs` (never compared with `move.inputs`); the stamps: a recovery install runs under the source's pre-move stamp `S` (already active after `ReturnToSource` + `Guard.Activate`), `local` runs `RetireArtifacts(S)` on the pre-move database and any leftover tagged with a retired stamp. |
| - The landing test suite of Plan 2 must pass unchanged after the refactor. |
| @@ -103,11 +104,11 @@ |
| - `rollbackTarget(ctx, s, w, rec)` phases (§7.8 target table): |
| - `releasing`: `SetHold(host, "releasing", rbID)`; fence; `Evidence` (ReconstructStopEvidence); incomplete → retry (the release deadline is the console's to page on). |
| - `released`: digests after a final checkpoint (`Checkpoint`): database sha256, files plain (`archive.PlainDigest`), timers (`moves.TimersDigest`); `clean := (landing_phase never reached probing || all three equal Landing.InstalledDBSHA256, Source.Files.Plain, Source.TimersSHA256) && seal.SrcCovered`; write `stopped-target.json` (create-only); `MoveRollbackTargetPhase(releasing → released, Dirty = !clean, Evidence = key)`; then `hostclaim.Update(TargetReleased(id, cfg.Box, cfg.Region, key, scope{Location: TrimSuffix(holder.location,"/"), Incarnation, EvidenceAt: report.CheckedAt, RequestLimit: 120 s}, newID))`. |
| - - `returning` (dirty): if a database was restored: final checkpoint and `SyncAndWait`, `N'` = `LocalPosition`, `moves.EncodeFull` of the stopped database `[1, N']` → `PutVersioned` to `ReturnLane(host, rbID) + "replica/0009/" + ltx.FormatFilename(1, N')` in the **source** bucket (M17: the target can put into the source's `moves/*`), the files archive and the timers snapshot under `files/` and `timers/`, `move.json` create-only; else (dirty only for want of `src_covered`): copy the staged seal's versions as given (`CopyVersion` by version id from own `moves/<host>/<mv>/replica/...` into the lane, same keys) and the staged files and timers; seal → `SetJournal(JournalReturnSeal = "return_seal")`; `MoveRollbackTargetPhase(returning → returned)`. |
| - - `cleaning`: only when `rec.Move.Rollback.RecoveredAt != nil` (Review Focus 4): run the `park-superseded` obligation for own `instances/<host>/…` and a `park-staging` for own `moves/<host>/<mv>/` (queued here if absent, scope `evidence_at = seal.Bound()`), `cfg.Park.Run` each until `Parked`; `Local.RetireInstallation` (database, `.gen`, files, env); `DeleteIfHold(host, rbID)`; `ReplaceTimers(host, nil)`; `MoveRollbackTargetPhase(cleaning → cleaned, TargetParkedAt = now)`; `CompleteObligation(park-local)`. |
| + - `returning` (dirty): **the return lane is a staging lane** (§7.3's rules apply to it; round-2 finding 6): the target takes the staging-work lease for `(source region, host)` (`lease.AcquireTTL(..., moves.StagingLeasePrefix(from, host), ..., moves.StagingLeaseTTL)`), writes every lane object through the guarded staging mirror (`lease.NewStagingMirror(ctx, targetRegion, sourceRegion)` under `moves.Guarded.Before`, one presigned request per call), and records the lease's `SafeUntil` at the seal as the lane's bound. If a database was restored: final checkpoint and `SyncAndWait`, `N'` = `LocalPosition`, `moves.EncodeFull` of the stopped database `[1, N']` → `PutVersioned` to `ReturnLane(host, rbID) + "replica/0009/" + ltx.FormatFilename(1, N')` in the **source** bucket (M17: the target can put into the source's `moves/*`), the files archive and the timers snapshot under `files/` and `timers/`, `move.json` create-only; else (dirty only for want of `src_covered`): copy the staged seal's versions as given (`CopyVersion` by version id from own `moves/<host>/<mv>/replica/...` into the lane, same keys) and the staged files and timers; the return seal (`moves.ReturnSeal`, with `LeaseUntil` = the lease's bound) → `SetJournal(JournalReturnSeal = "return_seal")`; `MoveRollbackTargetPhase(returning → returned)`; the lease is released. `TestReturnLaneWriterIsGuarded`: a paused return writer past its bound sends nothing, and the lane's later park (Plan 5) acknowledges at `LeaseUntil + H`, not before. |
| + - `cleaning`: only when `rec.Move.Rollback.RecoveredAt != nil` (Review Focus 4): run the `park-replica` obligation scoped to own `instances/<host>/<inc>` (queued by `TargetReleased`) and the `park-replica` scoped to own `moves/<host>/<mv>` (queued here if absent through `QueueStagingPark`, scope `evidence_at = seal.Bound()`), `cfg.Park.Run` each until `Parked` (D6: the location is no longer authoritative here, so the engine selects everything); `Local.RetireInstallation` (database, `.gen`, files, env); `DeleteIfHold(host, rbID)`; `ReplaceTimers(host, nil)`; `MoveRollbackTargetPhase(cleaning → cleaned, TargetParkedAt = now)`; `CompleteObligation(park-local)`. |
| - The rollback decision on the target: a landing failure (`markForRollback` sites) or `landing_deadline` → `provision.SetRecordMove(MoveRollBack(g, moves.NewRollbackID, reason, now))`; `ErrMoveConflict` from a won commit → proceed to open instead; then `hostclaim.Update(RollBack(id, rbID))`. |
| -Tests (harness with a target-side cfg, a `landing/probing` record and claim): `TestRollbackCASLosesToCommit`; `TestReleaseIsCleanWhenNothingChanged` (digests equal, `src_covered` true → `Dirty == false`, exec none, the target region's predecessors covered); `TestReleaseIsDirtyWithoutSrcCovered` (Review Focus 1, witness `wSealReturn`: digests equal, `src_covered` false → `Dirty == true`, `returning` copies the staged seal's exact versions into the lane); `TestDirtyReturnSnapshotsTheStoppedDatabase` (witness `wDirtyReturn`: `[1, N']` in the lane, `Verify()` passes, the seal names it); `TestTargetCleaningWaitsForRecoveredAt` (Review Focus 4); `TestTargetCleaningLeavesNoLiveCopy` (INV-10: nothing under own `instances/<host>/<inc>/` or `moves/<host>/<mv>/` after `cleaned`; archives under `superseded/<mv>/<host>/`). |
| +Tests (harness with a target-side cfg, a `landing/probing` record and claim): `TestRollbackCASLosesToCommit`; `TestReleaseIsCleanWhenNothingChanged` (digests equal, `src_covered` true → `Dirty == false`, exec none, the target region's predecessors covered); `TestReleaseIsDirtyWithoutSrcCovered` (Review Focus 1, witness `wSealReturn`: digests equal, `src_covered` false → `Dirty == true`, `returning` copies the staged seal's exact versions into the lane); `TestDirtyReturnSnapshotsTheStoppedDatabase` (witness `wDirtyReturn`: `[1, N']` in the lane, `Verify()` passes, the seal names it); `TestTargetCleaningWaitsForRecoveredAt` (Review Focus 4); `TestTargetCleaningLeavesNoLiveCopy` (INV-10: nothing under own `instances/<host>/<inc>/` or `moves/<host>/<mv>/` after `cleaned`; archives under `parked/<inc>/<host>/<ret>/`). |
| - [ ] **Steps 1 to 5**. Commit: `git commit -m "host: the rollback decision and the target's release, return lane and cleaning (§7.8)" ...` |
| @@ -123,8 +124,8 @@ Tests (harness with a target-side cfg, a `landing/probing` record and claim): `T |
| - `claimed`: `hostclaim.Update(ReturnToSource(id, cfg.Box, clean))`; `cfg.Guard.Activate(ctx, host, cfg.Box, nil)` (the fresh read finds `holder.stamp == S`, matching `active_stamp`, or re-initialises a lost record to `S`); `MoveRollbackSourcePhase(waiting → claimed)`. |
| - `restoring`: `SetHold(host, "restoring", rbID)`. |
| - clean `verified`: files plain and timers digest equal `Source.Files.Plain` and `Source.TimersSHA256`, the warm database's `<dbPath>.gen` names the pre-move generation (`src_gen`) and location; a mismatch → `MoveStuck(g, "recovery baseline mismatch")` from `restoring` (legal there, Task 1; INV-4), and `--retry` reruns the verification, never skips it. |
| - - dirty: `installGeneration` with `installSource{Lane: ReturnLane(host, rbID) in own bucket, Seal: return_seal, GenerationOp: rbID, Destination: rbID, PinInputs: nil, Phase: MoveRollbackSourcePhase}`. |
| - - `probing` (Task 9; needs claim Plan 6): `ProbeWake` kind `recovery`, nine attempts, each a cold admission (dirty: into the return generation, `Coverage: "sealed"`; clean: warm or cold into the source's own generation, `Coverage: "clean-return"`), `recovery_inputs` recorded; a restarted probe restarts only the probe; nine failures → `MoveStuck(g, reason)`. |
| + - dirty: `installGeneration` with `installSource{Lane: ReturnLane(host, rbID) in own bucket, Seal: return_seal, GenerationOp: rbID, PinInputs: nil, Phase: MoveRollbackSourcePhase}` (its `parking` step parks the pre-move generation's residue per generation as the landing does, D6; its `local` step retires the pre-move artifacts tagged `S` by tag, Plan 2 Task 10's order). |
| + - `probing` (Task 9, BLOCKED-ON-CLAIM C1): `ProbeWake` kind `recovery`, nine attempts, each a cold admission (dirty: into the return generation, `Coverage: "sealed"`; clean: warm or cold into the source's own generation, `Coverage: "clean-return"`), `recovery_inputs` recorded; a restarted probe restarts only the probe; nine failures → `MoveStuck(g, reason)`. |
| - `probed`: `MoveRollbackSourcePhase(probing → probed)`; dirty: `QueueReturnLanePark(id, cfg.Region, scope{Location: ScopeLocation(host, rbID), Incarnation, EvidenceAt: returnSeal.Bound(), RequestLimit: 120 s}, newID)`. |
| - `recovered`: `MoveRollbackSourcePhase(probed → recovered, RecoveredAt = now)`; `nudgeSerial` (the target's `cleaning` waits on it). |
| - `opened`: `ReleaseHold(host, rbID)`; `OpenedAt`; `hostclaim.Update(SetMovePhase(id, KindRestoring, "claimed"|…, "opened"))`. |
| @@ -141,9 +142,9 @@ Tests: `TestSourceWaitsForTargetEvidence`; `TestCleanReturnVerifiesAndOpens` (no |
| **Files:** create `internal/host/movesrevert.go`, `internal/host/movesrevert_test.go`; modify `internal/host/movespass.go` (`reverting` records whose `From == cfg.Region`). |
| **Interfaces:** |
| -- `revertSource(ctx, s, w, rec)` (§7.4 reverting): claim `hostclaim.Revert(id, cfg.Box, rec.Move.To, &stagingScope, newID)` once exec is none, where `stagingScope = {Location: ScopeLocation(host, id), Incarnation, EvidenceAt: <the lane's last LeaseUntil from the receipts ledger>, RequestLimit: 120 s}` (from `stopping` the evidence was never written: `rev_phase = evidence` computes it, writes `stopped-source.json` once it is complete, `ExecToNone`, then `Revert`); `SetHold(host, "restoring", id)`; `rev_phase` from the claim journal: `evidence` (compute, start nothing until it holds), `verify` (from `stopped`, `staged`, `archived`: files and timers digests equal `Source`'s; a mismatch is `stuck`, and `--retry` reruns the verification), `probing` (Task 9: `ProbeWake` kind `recovery`, warm or cold into the source's own generation; `SetRevPhase(verify → probing)` written before the first attempt; never re-runs `verify`), `probed` (`ReleaseHold(host, id)` conditional), `opened` (wait for the `park-staging` obligation `Revert` queued, which the target region's box runs and acknowledges through its regional runner and the target-side pass dispatches exactly as Plan 5 Task 1 dispatches an abort's: a record in `reverting` whose `To == cfg.Region` runs `cfg.Park.Run` on it; then `ReleaseMove(id, cfg.Box)` and `MoveRevertClose(g, now)`). Fifteen minutes of failed probes: `MoveStuck`. |
| +- `revertSource(ctx, s, w, rec)` (§7.4 reverting): claim `hostclaim.Revert(id, cfg.Box, rec.Move.To, &stagingScope, newID)` once exec is none, where `stagingScope = {Location: ScopeLocation(host, id), Incarnation, EvidenceAt: <the lane's last LeaseUntil from the receipts ledger>, RequestLimit: 120 s}` (from `stopping` the evidence was never written: `rev_phase = evidence` computes it, writes `stopped-source.json` once it is complete, `ExecToNone`, then `Revert`); `SetHold(host, "restoring", id)`; `rev_phase` from the claim journal: `evidence` (compute, start nothing until it holds), `verify` (from `stopped`, `staged`, `archived`: files and timers digests equal `Source`'s; a mismatch is `stuck`, and `--retry` reruns the verification), `probing` (Task 9: `ProbeWake` kind `recovery`, warm or cold into the source's own generation; `SetRevPhase(verify → probing)` written before the first attempt; never re-runs `verify`), `probed` (`ReleaseHold(host, id)` conditional), `opened` (wait for the lane `park-replica` obligation `Revert` queued, which the target region's box runs and acknowledges through its regional runner and the target-side pass dispatches exactly as Plan 5 Task 1 dispatches an abort's: a record in `reverting` whose `To == cfg.Region` runs `cfg.Park.Run` on it; then `ReleaseMove(id, cfg.Box)` and `MoveRevertClose(g, now)`). Fifteen minutes of failed probes: `MoveStuck`. |
| -Tests: `TestRevertFromStoppingComputesEvidenceFirst`; `TestRevertVerifyThenProbeNeverVerifiesAgain` (witness `wRollbackJournal` for the revert); `TestRevertOpensAndReleases` (the target-side pass archives the lane under `superseded/<mv>/<host>/…/moves/`, the obligation acknowledges after the horizon, and only then the claim is `serving` with exec = source and `LastMove.Outcome == "rolled_back"`). |
| +Tests: `TestRevertFromStoppingComputesEvidenceFirst`; `TestRevertVerifyThenProbeNeverVerifiesAgain` (witness `wRollbackJournal` for the revert); `TestRevertOpensAndReleases` (the target-side pass archives the lane under `parked/<inc>/<host>/<ret>/moves/`, the obligation acknowledges after the horizon, and only then the claim is `serving` with exec = source and `LastMove.Outcome == "rolled_back"`). |
| - [ ] **Steps 1 to 5**. Commit: `git commit -m "host: reverting, the pre-flip recovery, journaled in rev_phase (§7.4)" ...` |
| @@ -153,7 +154,7 @@ Tests: `TestRevertFromStoppingComputesEvidenceFirst`; `TestRevertVerifyThenProbe |
| **Files:** modify `internal/console/apiinstancemove.go` (`handleAPIInstanceMoveResolve`: body `{"retry": true}`), `internal/console/apiinstancemove_test.go`. |
| -**Interfaces:** `moveOperatorGate`; `{"retry": true}` → `provision.SetRecordMove(MoveRetry(g))` with the guard from the record; 409 unless `stuck`; the box reruns the recovery's `probing` phase on its next pass (Tasks 5 and 6 read `stuck → rolling_back|reverting` as "probe again", resetting the journal's attempt counter to 0). Copy: M-C11 is the CLI's; the door answers the flat view. |
| +**Interfaces:** `moveOperatorGate`; `{"retry": true}` → `provision.SetRecordMove(MoveRetry(g))` with the guard from the record; 409 unless `stuck`; the box resumes at the phase `Rollback.StuckAt` names (Task 1): a probe that failed nine times probes again with the journal's attempt counter reset to 0; a verification mismatch verifies again and is stuck again unless the operator fixed the baseline (never bypassed, INV-4); a revert resumes its `rev_phase`. Copy: M-C11 is the CLI's; the door answers the flat view. |
| - [ ] Test `TestResolveRetryFromStuckOnly`; implement; commit: `git commit -m "console: move resolve --retry reruns the recovery probe (§7.10)" ...` |
| @@ -171,32 +172,32 @@ Tests: `TestRevertFromStoppingComputesEvidenceFirst`; `TestRevertVerifyThenProbe |
| --- |
| -### Task 9: Recovery probes (needs claim Plan 6) |
| +### Task 9: Recovery probes (BLOCKED-ON-CLAIM: C1, C2) |
| **Files:** modify `internal/host/movesrecover.go`, `internal/host/movesrevert.go`, their tests. |
| -- Wire `cfg.Probe` with `activator.ProbeConfig{Kind: "recovery", Prepared: <the return generation's install manifest for a dirty return, nil otherwise>, Seal: <return seal or nil>, CutDeadline: far future (the recovery's own deadlines apply, codex R2-10), Attempt: ...}`, nine attempts (`moves.AttemptsFor("recovery", 0)`), the admission's `Coverage` `"sealed"` (dirty) or `"clean-return"` (clean with `src_covered`), `KeepStamp == false` for a dirty return (a writer was admitted since `S`). |
| +- Wire `cfg.Probe` with `activator.ProbeConfig{Op: moveID, HoldOp: rbID (a rollback) or moveID (a revert), PreparedGeneration: rbID for a dirty return, "" otherwise, Kind: "recovery", Prepared: <the return generation's install manifest for a dirty return, nil otherwise>, Seal: <return seal or nil>, Inputs: nil, CutDeadline: far future (the recovery's own deadlines apply, codex R2-10), Attempt: ...}`, nine attempts (`moves.AttemptsFor("recovery", 0)`), the admission's `Coverage` `"sealed"` (dirty) or `"clean-return"` (clean with `src_covered`), **cold in every case** (P7; a clean return is a cold admission into the source's own generation restored from this region's remote, never warm), `KeepStamp == false` for a dirty return (a writer was admitted since `S`). |
| - Tests: `TestDirtyReturnProbeIssuesANewStamp` (the probe's admission moves `holder.stamp` from `S` to a new value and `holder.generation` to the return generation; the guard record retires `S`); `TestCleanReturnProbeIsCoveredCleanReturn` (the admission record's `Coverage == "clean-return"`, no `writes_lost`). |
| - [ ] Commit: `git commit -m "host: recovery probes as cold admissions, nine attempts (§7.8, §7.14)" ...` |
| --- |
| -### Task 10: Declare-lost (needs claim Plan 8) |
| +### Task 10: Declare-lost (BLOCKED-ON-CLAIM: C6, C7; the record side below is local and built first) |
| **Files:** create `internal/moves/lossowner.go`, `internal/moves/lossowner_test.go`; modify `internal/console/apiinstancemove.go` (`resolve` with `{"declare_lost": "<box>", "confirm": "<host>", "cloud": {...}}`), `internal/console/apiinstancemove_test.go`; modify `internal/host/movesrecover.go` (the lost-target return), `internal/host/movesrollback.go` (a returning box quarantines). |
| **Interfaces:** |
| -- `moves.LossOwner{Record provision.Record}` implementing `hostclaim.LossOperationOwner.ValidateDeclareLost(c hostclaim.Claim, in hostclaim.LostDeclaration) error` with §7.10's table: legal only when `in.Box` is exactly one of `c.Op.Participants` and is **not** `participants[0]` (the source); the op is a move kind; the record's `State` is not `committed` (the record's commit decides, whatever the claim says); and one of: (a) the target owes stop evidence (`KindLanding`, or `KindRestoring` with `rollback.target_phase == releasing`); (b) the target released dirty and has not returned (`KindRestoring`, `target_phase == returning`); (c) the box owes only `park-local` obligations (legal after commit too: `State == committed` is allowed for this case alone). Everything else `ErrLossRefused` with the sentence. |
| +- `moves.LossOwner{Record provision.Record}` implementing `hostclaim.LossOperationOwner.ValidateDeclareLost(c hostclaim.Claim, in hostclaim.LostDeclaration) error` with §7.10's table, read against **the record as the door re-read it after `MoveRollbackTargetLost`** (round-2 finding 15: the legality is in the `lost` phase's provenance): legal only when `in.Box` is exactly one of `c.Op.Participants` and is **not** `participants[0]` (the source); the op is a move kind; the record's `State` is not `committed` (the record's commit decides, whatever the claim says); and one of: (a) and (b) the record's target phase is `lost` with `LossRef == in.LossID`'s record key (the door wrote it from `releasing`, `returning` or `released`+`Dirty`, the only phases `MoveRollbackTargetLost` accepts); (c) the box owes only `park-local` obligations and the record's target phase is `cleaned` or the move is `committed` (legal after commit for this case alone; no record phase changes). Everything else `ErrLossRefused` with the sentence. |
| - `moves.LossOwner.ValidateLossRecovery(c, loss hostclaim.LossRecord, in hostclaim.Admission) error` (`hostclaim.LossRecoveryOwner`): the admission is the source's, `in.Box == participants[0]`, `c.Op.Kind == KindRestoring`, phase `claimed`, and `in.AdditionalWritesLost` names **every** session the target was admitted into since the cut (every `holder.admissions` entry with `Box == lost box` and `Session > cut_session`) plus, when the seal's `src_covered` is false, the source's cut session `{Box: source, Generation: src_gen, Session: cut_session, RemoteMaxTXID: seal.N}`; unique by `(box, session)` (witnesses `wTwoSessionDeclaration`, `wNoDuplicateDeclarationEarly`, `wNoDuplicateDeclarationLate`; the legacy identity `{session: 0, generation: "" read as @root, box: holder.writer_box}` is valid, R16-3). |
| - The console door, in this order (the record decides, the claim follows, §7.5; a commit that wins between the two leaves the claim untouched): |
| 1. `moveOperatorGate`; body `{"declare_lost": box, "confirm": host, "cloud": {"instance_id", "state", "observed_at", "region", "account", "request_id"}}` validated (`confirm` must equal the host, M-C18; `state` is `stopped` or `terminated`; every cloud field present). |
| - 2. **The record first**: if `landing`, `provision.SetRecordMove(MoveRollBack(g, NewRollbackID, "declared lost: "+box, now))`; `ErrMoveConflict` because the record is `committed` → 409 "this move has committed. the target is authoritative and cannot be declared lost" and nothing else happens. Then `MoveRollbackTargetLost(g, lossKey, now)` (Task 1) when the target phase is `releasing`, `returning` or `released`+`Dirty`; a target already `released` clean or past `cleaning` → 409 (nothing to declare). Case (c), a box owing only `park-local` chores, skips both record writes. |
| - 3. **The claim**: `hostclaim.Read` fresh; if `Op.Kind == KindLanding` the claim half of the rollback is still owed → `hostclaim.Update(RollBack(id, rbID))` first. Build Plan 8's `LossRecord` exactly: `{V: 1, ID: lossID, State: "prepared", Actor: addr, AcceptedAt: now, Host, HolderIncarnation, HolderStamp, HolderSession, ExpectedClaimGen: obs.Claim.Gen, ExpectedClaimETag: obs.ETag, OpID, OpKind, OpPhase: as observed, Box, Enrollment: <the box's enrollment id from the actor directory, Plan 8b; until then the box label, which Plan 8's validation accepts as a non-empty string>, QuarantineRevision: <previous LostRevisions[box] + 1>, CloudObservation: LossCloudObservation{ObservedAt, Account, Region, InstanceID, State, RequestID} from the body, OperatorAttestation: LossOperatorAttestation{MaintainComputeFence: true, ColdQuarantineStartOnly: true}, FencedAt: cloud.ObservedAt, ConfirmedConsequences: LossConsequences{RevokeExecution: obs.Claim.Op.Exec == box, RemoveParticipation: true, DischargeLocal: <every park-local of the box>, ReassignRegional: <every park-replica/park-generation owned by box:<box>>, WritesPotentiallyLost: true, TimersPotentiallyLost: true, RecoveryIntent: json({"kind":"move-rollback","rollback":rbID,"clean":true})}, Timers: TimerLoss{Knowledge: TimersUnknown}}` (`TimersPartial` with the staged timers snapshot as `Source` when the cut's `timers/` object exists: the registered timers the target held are declared, §7.10). `hostclaim.CreateLossRecord(ctx, s.cfg.Releases, signer, record)` with the console's root `release.Signer`, then `hostclaim.DeclareLost(ctx, s.cfg.Releases, pub, obs, lossID, moves.LossOwner{Record: *rec})`. |
| - 4. In one more claim CAS: `QueueSupersededPark(id, targetRegion, "", scope{Location: TrimSuffix(holder.location, "/"), Incarnation, EvidenceAt: FencedAt, RequestLimit: 120 s}, newID)` and `CoverRegionPredecessors(id, obs.Claim.Op.Phase, targetRegion)` (what `TargetReleased` does for a live target), and for case (b) `QueueReturnLanePark(id, sourceRegion, scope{Location: ScopeLocation(host, rbID), Incarnation, EvidenceAt: FencedAt, RequestLimit: 120 s}, newID)` (model 27). |
| - 5. The declared timers are alerted through `host.PageOperator` (Plan 5 Task 0); the box is on `claim.Lost`. |
| - Until claim Plan 8b lands (index), the door answers 409 "declare-lost is not enabled in this deployment yet" unless `CARLOS_MOVES_DECLARE_LOST=1` is set on the console, which only a test sets. |
| -- The source's recovery with a lost target: `waiting` ends when the claim carries the `loss.<id>` entry and `Exec == "none"` (no `released`/`returned` target phase will come; the door wrote `MoveRollbackTargetLost`); `claimed` is `ReturnToSource(id, cfg.Box, clean = true)` (Task 2's loss form); the probe admits through `hostclaim.AdmitWriterWithLoss(ctx, s, pub, obs, admission, lossID, validated, moves.LossOwner{...})` instead of `Writer.Admit` (the moves pass passes `lossID` through `ProbeConfig.LossID`), with `validated` built from the admissions table as above (`ValidatedWriteSource{Reason: "target" | "source-cut", Box, Generation, Session, Region, Baseline, RemoteMaxTXID}`); `TestLostTargetRecoveryRunsAgainstARealDeclaration` drives the source pass from the claim Plan 8's `DeclareLost` actually produced in the previous test, not a hand-written claim. |
| + 2. **The record first** (local, built now): if `landing`, `provision.SetRecordMove(MoveRollBack(g, NewRollbackID, "declared lost: "+box, now))`; `ErrMoveConflict` because the record is `committed` → 409 "this move has committed. the target is authoritative and cannot be declared lost" and nothing else happens. Then `MoveRollbackTargetLost(g, lossKey, now)` (Task 1) when the target phase is `releasing`, `returning` or `released`+`Dirty`, which lands in `lost`; a target already `released` clean or past `cleaning` → 409 (nothing to declare). **Case (c)**, a box owing only `park-local` chores, skips both record writes and every claim step below except the declaration itself, and asserts afterwards that `holder`, the regional obligations and the admissions' coverage are unchanged (round-2 finding 17). |
| + 3. **The claim** (BLOCKED-ON-CLAIM C6: the exact lifecycle is the claim's; moves supplies the owner and the inputs listed here): `hostclaim.Read` fresh; if `Op.Kind == KindLanding` the claim half of the rollback is still owed → `hostclaim.Update(RollBack(id, rbID))` first. Then the loss sequence as `lost.go` on `feat/hostclaim-leases` requires it: `CreateLossRecord` of a `prepared` record whose `WriteSources` are composed per C6 (the latest target session as `target`, every earlier target session since `cut_session`, and the source's cut session when `src_covered` is false; moves computes the list from `holder.admissions` and the seal), with `CloudObservation{ObservedAt, Account, Region, InstanceID, State, RequestID}` from the body (every field required), `OperatorAttestation{MaintainComputeFence: true, ColdQuarantineStartOnly: true, DestroyedStorage: <the body's attested volume ids, required when any park-local is to be discharged>}`, `FencedAt: cloud.ObservedAt`, `ConfirmedConsequences` as the claim computes them from the observed claim, `Timers: TimerLoss{Knowledge: TimersPartial, Source: <the cut's timers/ object>}` when it exists else `TimersUnknown`, `Enrollment` and `QuarantineRevision` from Plan 8b's roster; `DeclareLost(ctx, s.cfg.Releases, pub, obs, lossID, moves.LossOwner{Record: *rec})`; then the record's `prepared → committed` with `ResultClaimGen` through `ReplaceLossRecord` (the claim's function; `AdmitWriterWithLoss` admits only a `committed` record, `lost.go:119`); then `MoveRollbackLossCommitted(g)` (Task 1) moves the record from `lost` to `released` clean with `Evidence = LossRef`. A crash between any two of these replays from the loss record's `State`. |
| + 4. In one more claim CAS, cases (a) and (b) only: `QueueLocationPark(id, targetRegion, "", scope{Location: TrimSuffix(holder.location, "/"), Incarnation, EvidenceAt: FencedAt, RequestLimit: 120 s}, newID)` and `CoverRegionPredecessors(id, obs.Claim.Op.Phase, targetRegion)` (what `TargetReleased` does for a live target), and for case (b) `QueueReturnLanePark(id, sourceRegion, scope{Location: ScopeLocation(host, rbID), Incarnation, EvidenceAt: FencedAt, RequestLimit: 120 s}, newID)` (model 27). The declaration itself discharged the box's `park-local` chores (or left them for the operator path) and reassigned its regional chores to `region:<target>`. |
| + 5. The declared timers are alerted through `host.PageOperator` (Plan 5 Task 0); the box is on `claim.Lost`. **`TargetParkedAt` for a lost target** (round-2 finding 18): the lost box's `cleaning` never runs; the console pass (Plan 5 Task 7) stamps `MoveRollbackTargetPhase(… → cleaned, TargetParkedAt)` when no obligation under the op is owned by `box:<lost box>` or by `region:<target region>` remains unacknowledged, which the surviving regional runner (C4) brings about; a region with no surviving box leaves the chores to the at-rest sweep and the close waits, paged once an hour (§7.8 "Waiting" applies to the park as it does to the evidence). |
| + Until claim Plan 8b lands (index), the door answers 409 "declare-lost is not enabled in this deployment yet" unless `CARLOS_MOVES_DECLARE_LOST=1` is set on the console, which only a test sets. **The termination fence (spec rev 7.19 §7.10, L14)** is the same path with the fence record as `LossRef`, driven by the console pass instead of an operator, once C7 names the journal key; BLOCKED-ON-CLAIM with the rest of this task. |
| +- The source's recovery with a lost target: `waiting` ends when the record's target phase is `released` with `Evidence == LossRef` (the door's `MoveRollbackLossCommitted`, after the loss record is `committed`) and the claim carries the `loss.<id>` entry with `Exec == "none"`; a record still at `lost` is **not** consumed (round-2 finding 15); `claimed` is `ReturnToSource(id, cfg.Box, clean = true)` (Task 2's loss form); the probe admits through `hostclaim.AdmitWriterWithLoss(ctx, s, pub, obs, admission, lossID, validated, moves.LossOwner{...})` instead of `Writer.Admit` (the moves pass passes `lossID` through `ProbeConfig.LossID`), with `validated` built from the admissions table as above (`ValidatedWriteSource{Reason: "target" | "source-cut", Box, Generation, Session, Region, Baseline, RemoteMaxTXID}`); `TestLostTargetRecoveryRunsAgainstARealDeclaration` drives the source pass from the claim Plan 8's `DeclareLost` actually produced in the previous test, not a hand-written claim. |
| - A returning box (`Lost` contains it) refuses every move step and runs `Local.RetireInstallation` for the host under the guard before anything else (claim §7.2 quarantine; Plan 8's `hostquarantine` when it lands, else the `park.Local` call directly). |
| Tests: `TestDeclareLostLegalityTable` (Review Focus 5: a table over (kind, phase, record state, box) with the expected error; the refused cases leave the claim unchanged); `TestDeclareLostLosesToAWinningCommit` (the target commits the record between the door's gate and its record write: the door answers 409, writes no loss record and no claim change, and the target opens); `TestDeclareLostCreatesARealSignedLossRecord` (through Plan 8's `CreateLossRecord` and `ReadLossRecord` with a test root signer, every field above present, `DeclareLost` accepts it, the claim's `Lost` names the box, `Exec == "none"`, the target is out of `participants`, the `loss.<id>` journal entry exists); `TestDeclareLostRefusesWithoutACloudObservation` (witness `wDeclareLostCompute`: a body whose `cloud.state` is not `stopped` or `terminated`, or absent, is refused at the door; an expired lease is never an argument); `TestReturningLostBoxQuarantinesFirst` (witness `wLostQuarantine`: a box in `claim.Lost` that comes back retires the host's local artifacts before any move step runs and refuses the step); `TestLostTargetReturnDeclaresEverySessionOnce` (witnesses `wTwoSessionDeclaration`, `wNoDuplicateDeclarationEarly/Late`: a target with a probe session that closed cleanly and a warm continuation that died → two `additional_writes_lost`; a target destroyed before its first admission with `src_covered` false → the cut session once, as the predecessor itself, never twice); `TestDeclareLostQueuesTheReturnLanePark` (model 27, witness `wLostReturnPark`: the lane's scope has `evidence_at = fenced_at`); `TestResolveDeclareLostRefusesWithoutConfirm`. |
diff --git a/docs/superpowers/plans/2026-10-06-moves-phase1-5-cleanup-and-parks.md b/docs/superpowers/plans/2026-10-06-moves-phase1-5-cleanup-and-parks.md| index 5f53af638..a32565108 100644 |
| --- a/docs/superpowers/plans/2026-10-06-moves-phase1-5-cleanup-and-parks.md |
| +++ b/docs/superpowers/plans/2026-10-06-moves-phase1-5-cleanup-and-parks.md |
| @@ -4,7 +4,7 @@ |
| **Goal:** Leave no copy of the instance in a live namespace of the region that lost: abort parks staging from the target's side, every staging lane carries its seal bound as a persisted park scope, the target parks its staging after the commit, the move releases the claim to `serving` only when every obligation is gone, the at-rest sweep covers `moves/<host>`, the db door refuses move kinds, and the console finishes the claim halves and runs the deadline sweeps. |
| -**Architecture:** Parks are claim obligations (`park-superseded`, `park-staging`, Plan 2 Task 9) executed by `internal/park.Engine` with a persisted `ParkScope` whose `evidence_at` is the genuine bound (the cut's stop evidence, the release's stop or `fenced_at`, the lane's seal bound, the sweep's listing). The box runs the immediate half (copies and version deletes) inside the move's phases; acknowledgement after the horizon is the regional chore runner's (claim Plan 6's `hostbox.Regional.Run`, widened to the two kinds). The release to `serving` is `hostclaim.ReleaseMove` (Plan 4 Task 2). The console's `movesRecoveryPass` writes claim halves the records decided and runs the deadline sweeps; it is a plain function claim Plan 5's `Recovery.Pass` can call. |
| +**Architecture:** Parks are claim obligations (`park-replica` scopes and `park-generation` chores, Plan 2 Task 9, index D6) executed by `internal/park.Engine` with a persisted `ParkScope` whose `evidence_at` is the genuine bound (the cut's stop evidence, the release's stop or `fenced_at`, the lane's seal bound, the sweep's listing). The box runs the immediate half (copies and version deletes) inside the move's phases; acknowledgement after the horizon is the regional chore runner's (claim Plan 6's `hostbox.Regional.Run`, widened to the two kinds). The release to `serving` is `hostclaim.ReleaseMove` (Plan 4 Task 2). The console's `movesRecoveryPass` writes claim halves the records decided and runs the deadline sweeps; it is a plain function claim Plan 5's `Recovery.Pass` can call. |
| **Tech Stack:** Go, `internal/park`, `internal/hostclaim`, `internal/console`. |
| @@ -60,14 +60,14 @@ |
| **Files:** modify `internal/hostclaim/move.go` (`Abort`), `internal/host/movespass.go` (the `MoveAborting` arm), create `internal/host/movesabort.go`, tests. |
| **Interfaces:** |
| -- `hostclaim.Abort(opID, targetRegion string, scope ParkScope, newID func() string) func(*Claim) error`: `KindMoving` with phase `""` (before the seal; a cut past `requested` cannot abort) → phase `"aborting"`, appends `park-staging` for `targetRegion` with `scope` (owner `region:<target>`); idempotent. |
| +- `hostclaim.Abort(opID, targetRegion string, scope ParkScope, newID func() string) func(*Claim) error`: `KindMoving` with phase `""` (before the seal; a cut past `requested` cannot abort) → phase `"aborting"`, appends the lane's `park-replica` (`QueueStagingPark`, D6) for `targetRegion` with `scope` (owner `region:<target>`); idempotent. Plan 1 Task 14's interim abort (release with no park) is replaced here. |
| - The source, on `aborting`: `hostclaim.Update(Abort(id, to, scope{Location: ScopeLocation(host, id), Incarnation, EvidenceAt: <the last round's lease SafeUntil the receipts recorded, i.e. the staging writers' horizon start>, RequestLimit: 120 s}, newID))`; delete its own `move-snapshots/<host>/<id>/` objects (own bucket; by version through `DeleteVersion`) and `move-receipts/<host>/<id>*.json`; then wait: `MoveAbortComplete` only when no obligation under the op remains and the claim is released (`ReleaseMove(id, cfg.Box)` first; the record clears second, §7.5 "the close is the other way round"). |
| -- The target region's box, for a record in `aborting` **or `reverting`** whose `To == cfg.Region`: `cfg.Park.Run` on the op's `park-staging` obligation (copies and deletes at once; acknowledgement after the horizon through the regional runner). The reverting case's obligation is the one `hostclaim.Revert` queued (Plan 2 Task 2); Plan 4 Task 6's `opened` waits on it. |
| +- The target region's box, for a record in `aborting` **or `reverting`** whose `To == cfg.Region`: `cfg.Park.Run` on the op's lane `park-replica` obligation (copies and deletes at once; acknowledgement after the horizon through the regional runner). The reverting case's obligation is the one `hostclaim.Revert` queued (Plan 2 Task 2); Plan 4 Task 6's `opened` waits on it. |
| - The staging horizon start for the abort scope: the receipts ledger records `LeaseUntil` of the last round (Plan 1 Task 14's `moveLedger`); `evidence_at = LeaseUntil` (the time after which no staging writer that held the lease can still write; the same bound the seal records for a cut). |
| - Auto-abort: `preparing` with no promotable round for 60 minutes pages (§13) and, at 6 hours (`moves.AbortAfter`), the console sweep (Task 7) CASes `MoveAbort`. |
| - The record's `MoveAbort` stays Plan 1 Task 2's; `MoveAbortComplete` gains the guard check through `rec.Move.Legacy()` (a legacy record completes as before, with the Plan 1 `DeletePrefix` path kept **only** for legacy records; the per-op path never deletes from the source). |
| -Tests: `TestAbortParksFromTheTargetAndWaits` (Review Focus 2: the source's pass issues no delete against the target bucket; the target's pass archives `moves/<host>/<id>/...` under `superseded/<id>/<host>/<ret>/moves/...`; the record clears only after the obligation is acknowledged and the claim says `serving` with `exec = source`); `TestAbortScopeIsTheWritersHorizon` (witness `wStagingQuiet`: a staging PUT signed before the lease ended and landing after the park's first pass is found by the verification listing and parked as a further retirement event; the park acknowledges only after `LeaseUntil + H`). |
| +Tests: `TestAbortParksFromTheTargetAndWaits` (Review Focus 2: the source's pass issues no delete against the target bucket; the target's pass archives `moves/<host>/<id>/...` under `parked/<inc>/<host>/<ret>/moves/...`; the record clears only after the obligation is acknowledged and the claim says `serving` with `exec = source`); `TestAbortScopeIsTheWritersHorizon` (witness `wStagingQuiet`: a staging PUT signed before the lease ended and landing after the park's first pass is found by the verification listing and parked as a further retirement event; the park acknowledges only after `LeaseUntil + H`). |
| - [ ] **Steps 1 to 5**. Commit: `git commit -m "host: abort parks staging from the target region and waits for the obligation (§7.11, joint N13)" ...` |
| @@ -90,11 +90,11 @@ Tests: `TestAbortParksFromTheTargetAndWaits` (Review Focus 2: the source's pass |
| **Files:** create `internal/host/movesparks.go`, `internal/host/movesparks_test.go`; modify `internal/host/movespass.go`. |
| **Interfaces:** |
| -- For a `committed` record whose `To == cfg.Region`: run the `park-staging` obligation (`cfg.Park.Run`; `ErrPending` until the horizon; the regional runner acknowledges). Its scope is the one `Commit` queued (`evidence_at = seal.Bound()`, Plan 3 Task 7). |
| +- For a `committed` record whose `To == cfg.Region`: run the lane's `park-replica` obligation (`cfg.Park.Run`; `ErrPending` until the horizon; C4's runner acknowledges). Its scope is the one `Commit` queued (`evidence_at = seal.Bound()`, Plan 3 Task 7). |
| - The close (§7.9 step 5): any participant that sees `OpenedAt`, `SourceParkedAt` set and no obligation under the op: the terminal log event `done` (Plan 6's `moves.LogEvent`; until then a `log.Printf`), `hostclaim.Update(ReleaseMove(id, targetBox))`, `provision.SetRecordMove(MoveClose(g, now))` (which writes `LastMove{Outcome: "done"}`, index D11). The console pass also tries (Task 7). |
| - `park.Engine`'s `Now` is the real clock; the test drives `Now` through the harness and a fake `Leases`. |
| -Tests: `TestStagingParkAcknowledgesAtSealBoundPlusH` (Review Focus 1, witness `wStagingScopeDeadline`: seal bound `T`; `Run` at `T + 16 min` is `ErrPending`; a new worker (restart) at `T + 17 min + 1 s` acknowledges; the scope's `EvidenceAt` is still `T`); `TestCloseWaitsForBothObligations` (with the source's `park-superseded` still pending → no release; after both acknowledged → claim `serving` with `exec = uk-box`, `Move == nil`, region `uk`); `TestCloseIsRecordLast` (the claim is released before the record clears; a crash between leaves a finished `move` that admits nothing, and the next pass clears it). |
| +Tests: `TestStagingParkAcknowledgesAtSealBoundPlusH` (Review Focus 1, witness `wStagingScopeDeadline`: seal bound `T`; `Run` at `T + 16 min` is `ErrPending`; a new worker (restart) at `T + 17 min + 1 s` acknowledges; the scope's `EvidenceAt` is still `T`); `TestCloseWaitsForBothObligations` (with the source's location `park-replica` still pending → no release; after both acknowledged → claim `serving` with `exec = uk-box`, `Move == nil`, region `uk`); `TestCloseIsRecordLast` (the claim is released before the record clears; a crash between leaves a finished `move` that admits nothing, and the next pass clears it). |
| - [ ] **Steps 1 to 5**. Commit: `git commit -m "host: the target's staging park, the horizon, and the close (§7.9 steps 4 and 5, K8)" ...` |
| @@ -102,11 +102,11 @@ Tests: `TestStagingParkAcknowledgesAtSealBoundPlusH` (Review Focus 1, witness `w |
| ### Task 4: Local parks of move retirements |
| -**Files:** modify `internal/park/local.go` (destination `.parked/superseded-<op>/<host>/<retirement>/` for obligations whose op is a move or rollback id), `internal/park/local_linux_test.go`. |
| +**Files:** `internal/park/local_linux_test.go` only. |
| -**Interfaces:** `park.Local.RetireArtifacts` and `RetireInstallation` choose the destination by the obligation's op: today `parkLocal` walks `.parked/<incarnation>/<host>/<retirement>/artifacts/` (`local_linux.go:196-201`, `localWalk(parked, []string{want.Incarnation, want.Host, want.Retirement}, true)`); for a `park-local` whose `Op` has `moves.OpIDKind(op) != ""` the first segment becomes `"superseded-" + op` so the event lands under `.parked/superseded-<op>/<host>/<retirement>/artifacts/`; everything else keeps today's path. (`internal/park` importing `internal/moves` would invert the dependency: put `OpIDKind`'s regexp in `park` as `supersededOp(op string) bool` with the same shape, and a test in `moves` asserting the two regexps are byte-equal.) The journal `localJournal` gains no field: the op is already in `want.Op`. |
| +**Interfaces:** none new (D6). `park.Local.RetireArtifacts` and `RetireInstallation` keep the engine's layout, `.parked/<incarnation>/<host>/<retirement>/artifacts/` (`local_linux.go:196-201`); the local journal already records `Op`, so F3 selects a move's local retirements by op the way it selects remote ones by manifest. The spec's `.parked/superseded-<op id>/...` wording is part of the rev 7.20 amendment D6 names. |
| -- [ ] Test, implement, commit: `git commit -m "park: local retirements of move operations land under .parked/superseded-<op>/ (§6.4)" ...` |
| +- [ ] Test `TestLocalRetirementUnderAMoveOpIsFindableByOp` (a `park-local` with a move id as `Op` lands under the engine's path and its journal names the op); commit: `git commit -m "park: local retirements under a move op keep the engine's layout; the journal names the op (§6.4, D6)" ...` |
| --- |
| @@ -144,7 +144,8 @@ Tests: `TestSweepSkipsWhileAMoveStages` (Review Focus 3); `TestFreeHostSweepIsHo |
| - the claim half the record decided (§7.5): record `landing` + claim `moving/sealed` with exec none → `Flip` (needs `source_stamp` and `src_gen`: the flip half is written by whoever sees it; the console computes `sourceStamp` from `holder.stamp` as observed, the pre-flip stamp, and `srcGen` from `holder.generation`); record `committed` + claim `landing` not `committed` → `Commit` with the scopes from the seal (`seal.Bound()`) and the evidence (`stopped-source.json`'s `CheckedAt`); record `rolling_back` + claim `landing` → `RollBack(id, rollback.id)`; anything else untouched (Review Focus 5). |
| - an orphan `moving/""` claim (from the discovery above) older than 10 minutes by `Op.StartedAt` → `hostclaim.Update(ReleaseOrphanMove(id))`, a guarded transition added to `hostclaim/move.go` in this task: legal only for `KindMoving` with phase `""`, `Exec == participants[0]`, no obligation under the op, and the caller's assertion that the record carries no move for this id (the pass re-reads the record inside the retry loop; `ReleaseMove` itself rejects phase `""` on purpose, so the orphan case has its own door); `KindLanding`/`KindRestoring`/`KindReverting` orphans are never released automatically: they page (an operation past the flip with no record is an incident). |
| - deadline sweeps: `landing` past `landing_deadline` → `MoveRollBack(g, NewRollbackID, "landing deadline", now)` (the box writes the claim half or this pass does on its next tick); `rolling_back` past `release_deadline` with no target evidence → `alert("incident", ...)` once per hour (M-C10's sentence for the operator view); `preparing` with `UpdatedAt` older than 60 minutes → `alert("warning", ...)`; older than `moves.AbortAfter` (6 h) → `MoveAbort()`. |
| - - the finalization halves: `committed` with both stamps and no obligations → `ReleaseMove` then `MoveClose`; `rolling_back` with the three stamps and no obligations → `ReleaseMove(source)` then `MoveRolledBackClose`; `aborting` with no obligations → `ReleaseMove(source)` then `MoveAbortComplete`. |
| + - the finalization halves: `committed` with both stamps and no obligations → `ReleaseMove` then `MoveClose`; `rolling_back` with the three stamps and no obligations → `ReleaseMove(source)` then `MoveRolledBackClose`; `aborting` with no obligations → `ReleaseMove(source)` then `MoveAbortComplete`; a `rolling_back` record whose target is on `claim.Lost` and whose target phase is `released` (the loss committed) gets `TargetParkedAt` stamped by this pass once no obligation under the op is owned by that box or by its region (Plan 4 Task 10 step 5). |
| + - the landing phase catch-up (Plan 2 Task 10 `syncClaimPhase`) for any `landing` record whose claim phase is behind its `landing_phase`. |
| - `Server.startMovesPass(ctx)`: the ticker, logging each pass's errors; `alert` is `host.PageOperator` (Task 0), which the console binary links already (`internal/host` is in the CLI). |
| - The closes write `MoveClose(g, now)`, `MoveRolledBackClose(g, now)` and `MoveAbortComplete` with `LastMove` (index D11). |
diff --git a/docs/superpowers/plans/2026-10-06-moves-phase1-6-cli-status-observability.md b/docs/superpowers/plans/2026-10-06-moves-phase1-6-cli-status-observability.md| index 8b16a6ca1..58794b246 100644 |
| --- a/docs/superpowers/plans/2026-10-06-moves-phase1-6-cli-status-observability.md |
| +++ b/docs/superpowers/plans/2026-10-06-moves-phase1-6-cli-status-observability.md |
| @@ -49,7 +49,7 @@ |
| **Interfaces:** |
| - `runMove(args []string)`: `carlos move --host <h> --to <region> [--manual-cut] [--no-wait] [--app <a>] [--console <url>]`; sub-verbs `status`, `cut`, `abort`, `resolve --retry | --declare-lost <box> [--instance-id <id>] [--region <aws region>]`; `--host` resolves the account and app through `resolvedApp` when `--app` is absent (the host's record is found through `GET /api/cli/apps/{acct}/{app}/instances/{host}` under the chosen app; without `--app`, the CLI lists the account's apps and finds the one whose instance domain the host ends in, as `instances_cli.go`'s host resolution does if it exists; else `--app` is required with a sentence saying so). |
| - `moveWait(ctx, sess *apiSession, path, moveID string, out io.Writer, poll time.Duration) int` polls `GET <path>?detail=1` every `poll` (2 s), prints a sentence on each state change (M-C1 at start, M-C2 while preparing, M-C3 when a target report appears, M-C4 per failing check, M-C5 at `cutting/admitted` with the budget, M-C6 at `landing/probing`, M-C7 at the open with `opened_at - cut_at`, M-C8 at `rolling_back`, M-C9 at `rolled_back` clean, M-C10 while the target owes evidence past the release deadline, M-C11 at `stuck`, M-C12 at `ready` with `cut == manual`, M-C13 for an abort refused after the cut). The terminal result is **never inferred from the last state seen**: when the answer is `state: none`, the CLI reads `last` (`Record.LastMove`, index D11) and requires `last.id == moveID`; `done` → 0 with M-C7 (`offline_s` from the outcome), `rolled_back` → 3 with M-C9 (or M-C8 and M-C9 when it never saw `rolling_back`), `aborted` → 3; `stuck` and `rolling_back` past the release deadline → 4; a `none` whose `last.id` is another move, or no `last`, is "this move's outcome is not recorded" and exits 4 (Review Focus 1). A disconnected CLI resumed with `carlos move status --host <h> --wait` reads the same outcome. |
| -- `--declare-lost`: before calling the door, the CLI verifies the box through the operator's own cloud credentials: `ec2.DescribeInstances` for the instance id the operator passes (or the box's instance id from `GET /api/cli/status/instances` if that door lists it); the state must be `stopped` or `terminated` (§7.10, model 5); it prints M-C18 and reads the host name from stdin as the confirmation; then `POST …/move/resolve {"declare_lost": box, "confirm": host, "cloud": {"instance_id", "state", "observed_at", "region"}}`. Without AWS credentials the verb refuses: "declare-lost needs your AWS credentials to confirm the box is stopped or terminated". |
| +- `--declare-lost`: before calling the door, the CLI verifies the box through the operator's own cloud credentials: `ec2.DescribeInstances` for the instance id the operator passes (or the box's instance id from `GET /api/cli/status/instances` if that door lists it); the state must be `stopped` or `terminated` (§7.10, model 5); it prints M-C18 and reads the host name from stdin as the confirmation; then `POST …/move/resolve {"declare_lost": box, "confirm": host, "cloud": {"instance_id", "state", "observed_at", "region", "account", "request_id"}, "destroyed_storage": [<volume ids the operator attests destroyed, optional>]}` (every `cloud` field is what Plan 8's `LossCloudObservation` requires: `account` is the `DescribeInstances` reservation's `OwnerId`, `request_id` the response's request id). Without AWS credentials the verb refuses: "declare-lost needs your AWS credentials to confirm the box is stopped or terminated". The verb is BLOCKED-ON-CLAIM with Plan 4 Task 10 (C6) and refused in production until Plan 8b (D13's switch does not open it). |
| Tests: `TestMoveWaitExitCodes` (a fake console serving a scripted sequence of detail answers: preparing → ready → cutting → landing → rolling_back → none with `last{id, rolled_back}`; the CLI exits 3 and printed M-C8 and M-C9; the done sequence exits 0 with M-C7; stuck exits 4; a 409 at the request exits 5); `TestMoveWaitSurvivesMissedStates` (Review Focus 1: the sequence preparing → none with `last{id, rolled_back}`, every intermediate state skipped: exit 3 with M-C8 and M-C9 printed from the outcome; none with `last.id` of another move → exit 4 with the "not recorded" sentence); `TestMoveManualCutPrintsM12AndStops`; `TestMoveDeclareLostNeedsStoppedOrTerminated` (witness `wDeclareLostCompute`: a fake EC2 answering `running` → refused before any console call). |
| @@ -76,11 +76,11 @@ Tests: `TestMoveDetailLeaksNothing` (Review Focus 2: the response body contains |
| **Files:** create `internal/moves/log.go`, `internal/moves/log_test.go`, `internal/host/movesevents.go`, `internal/host/movesevents_test.go`; modify the phase writers in `movescut.go`, `movesland.go`, `movesrollback.go`, `movesrecover.go`, `movesrevert.go`, `movesparks.go`, `internal/console/apiinstancemove.go` (request, cut, abort, resolve) to call `w.logEvent(ctx, rec, event, fields)`. |
| **Interfaces:** |
| -- `moves.LogEvent{V int; Op, Host, Event, Actor string; At time.Time; Fields map[string]string}`; `moves.WriteLog(ctx, s release.Store, prefix string, seq int, ev LogEvent) error`: `s.Put(key, body, "")` **is** create-only on `release.Store` (`store.go:38`), which is what a log wants: the key carries `seq` and the event name, so two actors racing to write the same sequence number collide with `release.ErrCASConflict` and the loser takes the next number (`NextLogSeq` again, bounded to three tries); a replay of a phase that already logged its event finds its own event under the previous sequence (the writer checks the last 20 keys for `-<event>.json` under this op before writing, so a replayed phase logs once). `moves.NextLogSeq(ctx, s, prefix) (int, error)` (lists `log/`, max + 1). |
| +- `moves.LogEvent{V int; Op, Host, Event, Actor string; At time.Time; Fields map[string]string}`; `moves.WriteLog(ctx, s release.Store, prefix string, ev LogEvent, identity string) error`: the key is the **transition's identity**, not a counter (round-2 finding 23): `log/<identity>.json` where `identity` is `<record state>-<phase>` for a phase transition (`cutting-stopped`, `landing-probing`, `rolling_back-source-claimed`), `probe_attempt-<n>` for the n-th attempt (`Landing.Attempts` or the recovery journal's counter), `stuck-<n>` and `retry-<n>` for the n-th entry into `stuck` (`Rollback.StuckCount`, a record field this task adds), and the terminal `done`, `rolled_back`, `aborted`, `declared_lost`; `s.Put(key, body, "")` is create-only (`store.go:38`), so two actors writing the same transition collide harmlessly on identical content and a replayed phase writes nothing new; the log is ordered by `At` and by the record's own phase order, never by a sequence number. `moves.ReadLog(ctx, s, prefix) ([]LogEvent, error)` lists `log/` and sorts by `At`. |
| - Events: `requested, ready, cut_requested, admitted, stopping, stopped, staged, archived, flipped, checking, parking, building, published, local, configured, restored, probing, probe_attempt, probed, committed, opened, source_parked, done, reverting, rev_evidence, rev_verify, rev_probing, rev_opened, rolling_back, released, returning, returned, claimed, verified, recovered, cleaning, cleaned, rolled_back, stuck, retry, aborting, aborted, declared_lost`; `Fields` carry timings (`offline_s` at `opened`/`recovered`: `cut_at` to the open) and digests. |
| - The member stream: `emitEvent(sink, rt, level, msg)` per transition with the operator sentence (M-C1, M-C5, M-C6, M-C7, M-C8, M-C9, M-C11 as member-safe lines); `movesWorker` gets `Events activator.EventSink` from `agent.go:464`. |
| -Tests: `TestLogEventsAreSequential`; `TestEveryTransitionEmitsOneEvent` (Review Focus 3: the harness counts events across a full done flow against the transition list; a status poll emits none). |
| +Tests: `TestLogKeysAreTransitionIdentities` (two writers of `landing-probing` produce one object; three probe attempts produce three `probe_attempt-<n>` objects; a replayed `cutting-stopped` writes nothing new); `TestEveryTransitionEmitsOneEvent` (Review Focus 3: the harness counts events across a full done flow against the transition list; a status poll emits none). |
| - [ ] Commit: `git commit -m "moves: the move log and member-stream events (§13)" ...` |
diff --git a/docs/superpowers/plans/2026-10-06-moves-phase1-7-rehearsal-runbook.md b/docs/superpowers/plans/2026-10-06-moves-phase1-7-rehearsal-runbook.md| index e1c432115..1eeed2588 100644 |
| --- a/docs/superpowers/plans/2026-10-06-moves-phase1-7-rehearsal-runbook.md |
| +++ b/docs/superpowers/plans/2026-10-06-moves-phase1-7-rehearsal-runbook.md |
| @@ -81,7 +81,8 @@ The runbook has one section per run in §14.4's order; each section is a checkli |
| - [ ] P-2 §15 step 5 applied: `sibling_staging_arns` and the versioned-object grants (Paul, scoped `tofu apply -target`); verify with T10's probe through `carlos move status --host <h>` after the first request (T10 passes). |
| - [ ] P-3 `control/moves-protocol.json` written create-only: `{"min":1,"min_release":"<the rolled version>"}` (an `aws s3api put-object --if-none-match '*'` from an operator shell with the deployment profile); `carlos system apply` of an older build on dev now refuses. |
| - [ ] P-4 `moveprobe` shipped as app `moveprobe` under the ops account with a store binding `MOVEPROBE`, a sealed secret `MOVEPROBE_SECRET`, a declared schedule, and `moves.ready_path` unset (`/` serves `/ready`'s answer). |
| -- [ ] P-5 Every rehearsal host is adopted (claim Plan 5's adoption has run: `control/hosts/<host>.json` exists with `op.kind: serving`). |
| +- [ ] P-5 Every rehearsal host is adopted (claim Plan 5's adoption has run: `control/hosts/<host>.json` exists with `op.kind: serving`). This needs C5 merged (production callers for adoption and the claim recovery pass): merging the three claim branches alone does not adopt anything. |
| +- [ ] P-6 The switch is on for this deployment: `CARLOS_MOVES_PHASE1=on` on the console and every box (index D13), with the copy record approved and the default flipped in one commit before any run on the flagship. Run 4 (declare-lost) needs claim Plan 8b merged and rolled (the roster, census, release identity and guard, the termination fence, the real declare-lost and readmit) and `CARLOS_MOVES_DECLARE_LOST` is never set outside tests; until 8b is rolled, Run 4 is skipped and the series continues with Run 5, and the loss run is repeated on dev once 8b lands. |
| **Run 1, dev IE → uk, done:** |
| - [ ] S-1 `carlos instances create --app moveprobe --host mp1-<date>.<domain> --region ie` (a fresh name; Review Focus 3); wake it (`curl https://<host>/`), `POST /arm-once?in=600`, note `/report`. |
| @@ -91,7 +92,7 @@ The runbook has one section per run in §14.4's order; each section is a checkli |
| - [ ] R-2 the `bg` and `worker` row counts are at least S-1's and every `seq` is contiguous (no missing row). |
| - [ ] R-3 the `every-minute` rows continue on uk with region `uk` from the first tick after the open. |
| - [ ] R-4 the once-timer fired exactly once, on uk, with S-1's `due_at` (Review Focus 2). |
| -- [ ] R-5 `aws s3api list-object-versions --bucket <ie data bucket> --prefix instances/<host>/` is empty after the source park acknowledged (`carlos move status` detail shows no obligation; the log has `source_parked` and `done`); `superseded/<mv>/<host>/` holds the archive. |
| +- [ ] R-5 `aws s3api list-object-versions --bucket <ie data bucket> --prefix instances/<host>/` is empty after the source park acknowledged (`carlos move status` detail shows no obligation; the log has `source_parked` and `done`); `parked/<inc>/<host>/<ret>/` holds the archive. |
| - [ ] R-6 the move log has no `stray` page; the ie box's journal shows `ReconstructStopEvidence` killed the worker (the `tree` step OK) (Review Focus 1). |
| - [ ] R-7 (not for this run). |
| @@ -99,7 +100,7 @@ The runbook has one section per run in §14.4's order; each section is a checkli |
| **Run 3, dev forced dirty rollback:** `MOVEPROBE_FAIL_READY_IN=ie,after-write`; expect exit 3; R-7: the source's `/report` holds a `bg` row with region `ie` written on the target during its probe (Review Focus 4); the log's `released` event has `dirty: true`, `returned` names the lane snapshot; `moves/<host>/<rb>/` in the uk bucket is parked after the horizon. |
| -**Run 4, dev declare-lost:** request a move to a dev box Paul designates as disposable; after the flip, stop that box through the AWS console (dev only); `carlos move resolve --host <host> --declare-lost <box> --instance-id <id>`; type the host name at M-C18; expect the clean rollback to complete on the source, exit 3; the loss record names the box, `fenced_at`, the declared timers; restart the box and watch it quarantine the host's local state before anything else (S-9, Review Focus 5). |
| +**Run 4, dev declare-lost (needs P-6's Plan 8b clause):** request a move to a dev box Paul designates as disposable; after the flip, stop that box through the AWS console (dev only); `carlos move resolve --host <host> --declare-lost <box> --instance-id <id>`; type the host name at M-C18; expect the clean rollback to complete on the source, exit 3; the loss record names the box, `fenced_at`, the declared timers; restart the box and watch it quarantine the host's local state before anything else (S-9, Review Focus 5). |
| **Run 5, flagship, scoped approval:** runs 1 to 3 again on the flagship with a `moveprobe` instance, each with Paul's explicit go in the discussion thread. |
diff --git a/docs/superpowers/plans/2026-10-06-moves-region-ha-conflicts.md b/docs/superpowers/plans/2026-10-06-moves-region-ha-conflicts.md| new file mode 100644 |
| index 000000000..7583f1292 |
| --- /dev/null |
| +++ b/docs/superpowers/plans/2026-10-06-moves-region-ha-conflicts.md |
| @@ -0,0 +1,99 @@ |
| +# Region HA L14 to L18 against the moves spec and the claim (platform-d8, 2026-10-06) |
| + |
| +Written while folding region HA spec 1 rev 5.4 §9 (branch |
| +`design-region-ha-rev5`) into moves rev 7.19, with claim rev 6.27 |
| +(989ee349) §1.1's rulings in hand. "Claim change" means platform-1e has |
| +to do something for the moves text to hold. |
| + |
| +## Folded into moves rev 7.19 with no conflict |
| + |
| +- **L15, the target box in the start CAS.** Moves' `hostclaim.BeginMove` |
| + (plan 1 Task 3) already takes `targetBox` and records |
| + `participants = {source, target}` and `intent.target`. Rev 7.19 §7.1 |
| + names `choose()` as the chooser for a multi-box region. No claim |
| + change: the participants list and the single-exec rule are claim §2 |
| + as they stand. Moves' door reads the roster and the edges snapshot; |
| + it needs the roster's eligibility read (region HA §5.2 |
| + `eligibleBoxes`) exported from wherever region HA builds it, or it |
| + falls back to "the lexically-first box publishing the placement" for |
| + a one-box region, which is what it does today. |
| +- **L16, one install routine.** `moves.BuildGeneration` (plan 2 Task 8) |
| + takes a pinned version set and `N`; relocation can call it with a |
| + `genselect.SelectPrepared` plan or a current listing. No claim change. |
| +- **L17, the files manifest per installation.** Moves' `local` phase |
| + publishes the installed directory through region HA's files publisher |
| + before `.files-gen`. **Claim change (already region HA's L13):** the |
| + sweep's classifier must treat `<location>files/<generation>/` as that |
| + generation's and protect the objects the files live set names; without |
| + L13, a `park-generation` chore would archive a live manifest's objects. |
| + Moves also needs the publisher's API (region HA plan 1a's) exported: |
| + `files.Publish(ctx, store, location, generation, dir, envPath) (manifestVersion string, err error)` or equivalent; the moves plan names |
| + it as a contract, not a fact. |
| + |
| +## Folded with a claim change required |
| + |
| +- **L14, the termination fence as M16's evidence.** Moves §7.10 now |
| + says a fenced target is taken off the move automatically before the |
| + record's commit, through the declared-loss path, with `fenced_at` as |
| + the scope's evidence time. On the claim side this needs L2 to L4 as |
| + region HA requests them: `declareLostTransition` today requires |
| + `QuarantineRevision != 0` and a non-empty `Enrollment` from an |
| + operator-prepared `LossRecord` (leases branch `lost.go:221`); a fence |
| + with no quarantine revision and no operator must be a legal |
| + declaration input (L4 says a terminated box needs no readmission). |
| + Moves needs the journal to carry **one readable reference** to the |
| + fence (the `loss.<id>` entry's intent, or a new `fence` key) so |
| + `ReturnToSource` (plan 4 Task 2) can accept it as the cited evidence; |
| + the key name is the claim's to choose. Until that lands, the moves |
| + plan marks the fence path BLOCKED-ON-CLAIM beside `--declare-lost`. |
| +- **A terminated source before the flip.** Moves §7.10 says the move |
| + aborts and relocation takes the host. The claim's `moving` op must be |
| + releasable by the fence's lost-box path while the move's staging park |
| + is still owed (abort's `park-staging`, owned by the target region): |
| + the release of a `moving` claim with outstanding obligations is today |
| + `ErrPendingChore` (`ReleaseMove`); region HA's L1 "source-loss |
| + conversion to failover from `starting` or `releasing`" needs the same |
| + for `moving`. Claim change: a fence may convert `moving` to |
| + `relocating` with the staging obligation carried over, or the |
| + obligation's owner must be allowed to acknowledge under the new op. |
| + |
| +## Not done, stated as a limit |
| + |
| +- **L18, two boxes per region in the model.** `internal/sim/hostlife` |
| + keeps one box per region plus replacement boxes. The requested |
| + extension models relocation, an operation outside the moves spec, |
| + over a world whose `box.Region` bookkeeping assumes one box per |
| + region; it is not tractable in this revision within the CPU policy |
| + (the gate plus one niced 3000 sweep). Rev 7.19 §14.2.6 states the |
| + limit. It belongs to a joint round with the region HA session on its |
| + own branch once relocation's protocol is fixed. |
| + |
| +## Rulings that touch both |
| + |
| +- **P7, cold-only.** Moves rev 7.19 §0.5 and §19.1: every admission |
| + cold, no certificate issued. The claim library keeps `Mode: "warm"` |
| + and `PredecessorCertified`; moves never passes them. No claim change; |
| + the claim's own §1.1 already records P7. |
| +- **Park kinds (plan review round 2, finding 3; platform-1e's interface |
| + check).** Moves drops `park-superseded` and `park-staging`: staging |
| + lanes are `park-replica` scoped to `moves/<host>/<op id>`; a source's |
| + finalization and a rolled-back target's cleaning are `park-replica` |
| + scoped to the whole location (non-authoritative there, so the engine |
| + selects everything); a landing's earlier-stay residue is |
| + `park-generation` per residue generation (the engine's live-set rule |
| + lets those acknowledge after the new generation is live). Destination |
| + stays the engine's `parked/<incarnation>/<host>/<retirement>/`; F3 |
| + selects by manifest `Op`. **No new claim kind is needed in 6.27.** |
| + Reason: nothing in the move reads superseded data back (a dirty return |
| + seals the target's live installation before `cleaning` parks; a clean |
| + return restores from the source's live replica), M7's seven-day hold |
| + is F3's rule over immutable manifests, not a path, and D5's one |
| + manifest per retirement event is the engine's own. |
| +- **One claim contract moves still needs for parks on a serving box:** |
| + `Engine.Run` acquires the location's replica lease (`park/remote.go:404`). |
| + A `park-generation` chore for a location this box itself serves (the |
| + landing's earlier-stay residue acknowledging after the probe; the |
| + at-rest sweep on a serving host) can only acknowledge if the regional |
| + runner's `Leases.Acquire` shares the activator's held lease for a |
| + location this box serves (an aggregate lease adapter, claim Plan 5 |
| + D3's wording). Moves lists it as a BLOCKED-ON-CLAIM contract. |
diff --git a/docs/superpowers/plans/CODEX-PLAN-MOVES-PHASE1-R2.md b/docs/superpowers/plans/CODEX-PLAN-MOVES-PHASE1-R2.md| new file mode 100644 |
| index 000000000..ed79435aa |
| --- /dev/null |
| +++ b/docs/superpowers/plans/CODEX-PLAN-MOVES-PHASE1-R2.md |
| @@ -0,0 +1,148 @@ |
| +**PLAN: NOT READY** |
| + |
| +Reviewed `da2832aa` against the approved moves spec, claim spec at `98b45332`, and these local remote-tracking tips: |
| + |
| +| Reference | Tip | |
| +|---|---| |
| +| `origin/main` | `01f890ce` | |
| +| `origin/feat/hostclaim-console` | `5422d8fc` | |
| +| `origin/feat/hostclaim-box` | `2bd3ea1f` | |
| +| `origin/feat/hostclaim-leases` | `ca5a79d8` | |
| + |
| +No files changed, hosts contacted, or tests run. “Resolved” below means the revised plan specifies a satisfactory fix, not that an implementation has passed verification. |
| + |
| +Round-1 dispositions: |
| + |
| +| R1 | Finding | Disposition | Assessment | |
| +|---:|---|---|---| |
| +| 1 | `MoveLanding` collision | Resolved | Journal renamed `MoveLandingJournal`. A separate collision remains below. | |
| +| 2 | Stop-evidence import cycle | Resolved | Moving reconstruction to `stopevidence` and passing `eb.Cgroups` removes the identified cycle. | |
| +| 3 | Strict-spawn return signature | Resolved | Three results and prepared-handle lifetime are now specified. | |
| +| 4 | Stale receipt version | Resolved | Receipt must match current source and destination; otherwise recopy. | |
| +| 5 | Unusable staging TTL | Resolved | The 45-minute staging TTL fixes the arithmetic. The proposed real-lease fixture still needs correction. | |
| +| 6 | Sign-at-check | **Partly** | Disabling retries helps, but calling the SDK in the same goroutine does not bind its signature timestamp to the check. An arbitrary pause before SDK signing remains possible. | |
| +| 7 | Forbidden `move.json` recovery read | **Partly** | GET removed; the replacement pending-intent protocol cannot distinguish an existing foreign object. R2-7 below. | |
| +| 8 | Application `env` collision | **Partly** | Namespaces fixed. `Extract` still has no duplicate-destination rejection or corresponding negative test. | |
| +| 9 | Target absent from `touched` | Resolved | First staging write now follows target-region and target-box registration. | |
| +| 10 | Empty ETag report writes | Resolved | Mutable reports use read/ETag/CAS replacement. | |
| +| 11 | Floor fails open | **Partly** | Local floor added, but the new door rejects every request. Unknown-floor handling also equates participation with a held registry row, missing preparing participants. | |
| +| 12 | Incomplete immutable stop report | Resolved | Complete report follows checkpoint and precedes execution release. | |
| +| 13 | Archive publication after `archived` | Resolved | Publication and seal update now precede advancement. Archive identity replay remains a separate issue. | |
| +| 14 | Runner slips past drain | Resolved | Runner claims before its decisive fresh route read. Converger remains outside this handshake. | |
| +| 15 | LTX lock page | Resolved | Skip explicitly specified while retaining `Commit`. | |
| +| 16 | Restore over existing scratch | Resolved | Replay removes operation-owned scratch before restoring. | |
| +| 17 | Retirement without a chore | **Partly** | Chore added, but inventory registration does not match the real API’s path/stamp contract. | |
| +| 18 | First probe checks source generation | **Partly** | First landing validation separated. Dirty recovery and subsequent marker handling still do not compose. | |
| +| 19 | Claim never reaches commit phase | **Partly** | Happy-path mirroring added; interrupted record/claim phase writes have no complete catch-up procedure. | |
| +| 20 | Declare-lost overrides winning commit | Resolved | Rollback now competes on the record first. Other loss-ordering failures remain below. | |
| +| 21 | Loss payload mismatch | **Partly** | Plan 4 corrects several fields; Plan 6 still omits required `account` and `request_id`. Current branch requirements also remain unmet. | |
| +| 22 | Recovery cannot consume declaration | **Partly** | `none` and loss journal accepted, but actual loss admission and terminal completion still fail. | |
| +| 23 | Orphan claims undiscoverable | Resolved | Claim discovery and a distinct orphan-release transition added. | |
| +| 24 | Polling misses terminal outcome | Resolved | Operation-bound `LastMove` is persisted and returned. | |
| +| 25 | Probe app launch contract | Resolved | Uses `--socket` and `--db`. | |
| +| 26 | Unlisted Plan 8b activation dependency | Resolved | Index explicitly names 8b and refuses production activation until it lands. Runbook needs to carry that gate too. | |
| +| 27 | Forbidden recovery transitions | **Partly** | New closures help, but Task 7 still says retry always resumes probing, contradicting Task 1’s verification retry. Loss ordering also invalidates its own legality test. | |
| +| 28 | Reverting lacks staging cleanup | Resolved | Producer and target-side dispatcher now specified. | |
| +| 29 | MinIO test excluded from CI | Resolved | Explicit host-package invocation and PASS assertion added. | |
| +| 30 | Task dependency/order failures | **Partly** | Paging and CLI heading corrected. Other forward dependencies and premature copy wiring remain. | |
| +| 31 | Copy inventory derived from approval rows | Resolved | Independent ID inventory and negative fixtures specified. | |
| +| 32 | Clean rehearsal writes before failure | Resolved | Before-write mode explicitly suppresses mutations. | |
| + |
| +New findings, including failures introduced by the revisions: |
| + |
| +1. **P1 — Every move request fails the new completeness check.** |
| + [Plan 1:1054](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-1-copy-and-staging.md:1054) calls `readEdgesSnapshot` and then `snap.complete()`. On main, `instancesplacement.go:236` defines that as `Complete && BoxesComplete`; `readEdgesSnapshot:242` never sets `BoxesComplete`. Every invocation therefore returns incomplete. Use the appropriate complete reader, or explicitly define the narrower evidence check. Test a successful request through the real reader. |
| + |
| +2. **P1 — Another duplicate Go identifier prevents compilation.** |
| + [Plan 1:449](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-1-copy-and-staging.md:449) declares the constant `MoveStuck`; [Plan 4:57](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-4-rollback-return-declare-lost.md:57) declares the transition function `MoveStuck` in the same package. Rename one. The revised snippets also retain a two-argument `Revert` call at Plan 2:365 despite its new five-argument signature. |
| + |
| +3. **P1 — The real claim store rejects both new park kinds.** |
| + [Plan 2:788](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-2-cut-flip-landing.md:788) changes `Obligation.Kind`’s documentation, but does not extend `Claim.Validate`. Main’s `claim.go:266` rejects scopes on either new kind, and `:274–288` rejects the kinds themselves. `hostclaim.Update` always validates before persistence. Consequently revert, commit, abort and retirement CASes fail. Direct closure tests such as Plan 3’s commit test bypass this failure. Extend all relevant validators and exercise real `Update`/`Read` round trips. |
| + |
| +4. **P1 — The dependency graph promises integration the branches do not provide.** |
| + [Index:69](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-0-index.md:69) correctly admits the missing `startAdmitted` entry point, but understates the missing work: `hostclaim-box` adds adapters without modifying the live activator wake/start path. `SelectedTimers` also has no `ReplaceTimers` method. On `hostclaim-console`, `adoptHostRecord` and `StartClaimRecoveryPass` have no production callers; merging it does not cause the promised next-reconcile adoption. Assign explicit integration tasks and prerequisites. Also, Plan 3 Task 8 consumes `DeleteIfHold`, introduced only by the dependency-blocked Task 7. |
| + |
| +5. **P1 — Intermediate merges expose unfinished move behavior.** |
| + [Plan 1:584](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-1-copy-and-staging.md:584) deletes working cut tests and turns the cut into a no-op, while Task 14 still automatically enters `cutting`. Its interim abort clears the record without releasing the new move claim. Plan 2 then enables stopping/flipping before recovery and probing exist. There is no deployment feature gate covering these partially implemented mutations. “Dormant release” and withholding production rollout do not make each advertised merge independently shippable. Keep incomplete entry points explicitly disabled until their completion and recovery paths exist. |
| + |
| +6. **P1 — Return-lane writes omit the staging lease and signing contract.** |
| + [Plan 4:106](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-4-rollback-return-declare-lost.md:106) writes snapshots, archives, timers and `move.json` through `PutVersioned`/`CopyVersion`, without acquiring the source-region staging-work lease or applying the guarded signing path. The moves spec explicitly applies the forward lane’s exclusion and horizon rules to the return lane. Specify that adapter and persist its actual bound before publishing the return seal; test a paused return writer against cleanup. This is additional to unresolved R1-6. |
| + |
| +7. **P1 — Writing `pending` before PUT defeats the foreign-lane refusal.** |
| + [Plan 1:2946](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-1-copy-and-staging.md:2946) always records `pending` before `PutIfAbsent`. Thus an already-existing object with no prior ledger becomes indistinguishable from a successful earlier PUT whose response was lost. The “ErrExists with no ledger” branch is unreachable in that sequence. Even retaining the pre-call state cannot resolve a crash immediately after intent persistence. Use target-owned identity validation before accepting the lane, and test that exact crash with a foreign object already present. |
| + |
| +8. **P1 — The revised local inventory step cannot retire an earlier installation correctly.** |
| + [Plan 2:812](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-2-cut-flip-landing.md:812) activates stamp F, then calls `RecordInventory` with database/files/env paths before retiring the previous stamp. Main’s `park/local.go` requires paths relative to `DataDir`, tags them with the captured stamp, and rejects an existing path tagged with another stamp. Absolute paths fail immediately; registering old files under F either conflicts or makes `RetireArtifacts(previousStamp)` select nothing. Preserve the old inventory/stamp for retirement, then register installed artifacts under F. Include `.gen` and other required artifacts explicitly. |
| + |
| +9. **P1 — Landing holds the lease that its park engine must acquire.** |
| + [Plan 2:812](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-2-cut-flip-landing.md:812) retains `OwnLease(holder.location)` through `published`; the next phase calls `Park.Run` for that location. Main’s `park/remote.go:404` independently acquires that same lease. A landing with earlier remote residue cannot pass parking. Define lease ownership transfer/reuse, or acquire the build lease after retirement. The fake lease harness must reject a second acquisition. |
| + |
| +10. **P1 — A landing’s whole-location park cannot finish after the new generation becomes live.** |
| + [Plan 2:813](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-2-cut-flip-landing.md:813) advances after `Parked`, leaving acknowledgement pending until the horizon, then builds and serves in the same location. On later verification, main’s park engine lists the new generation and marker; `selected` at `remote.go:346` refuses protected live objects for a whole-location park. The old obligation therefore blocks finalization. Define how this retirement distinguishes earlier residue from the newly authoritative generation without weakening live-data protection, and test acknowledgement after the probe publishes. |
| + |
| +11. **P1 — Record/claim phase mirroring still wedges at interrupted writes.** |
| + [Plan 2:811](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-2-cut-flip-landing.md:811) handles a claim already ahead, but not one behind: crash after record `parking`, before claim `checking → parking`; the next record step expects the claim at `parking`, and fails against `checking`. The console pass handles flip/commit/rollback halves, not these intermediate phases. A similar cut window exists at [Plan 2:714](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-2-cut-flip-landing.md:714): replay after successful `ExecToNone` calls a closure that rejects already-empty execution. Specify catch-up/idempotency before further work and test each half-write boundary. |
| + |
| +12. **P1 — Source finalization can permanently lose its completion stamp.** |
| + [Plan 3:358](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-3-probe-commit-open.md:358) requires the source row still to carry this hold. It deletes that row at line 361 and writes `SourceParkedAt` afterward. A crash between them makes the next pass fail its entry condition forever, leaving the close blocked. Replay must recognize its own completed deletion using durable operation state while still refusing a replacement row. |
| + |
| +13. **P1 — Dirty-return probes reject their prepared generation before admission.** |
| + [Plan 3:309](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-3-probe-commit-open.md:309) requires every recovery probe’s local markers to name observed `holder.generation`. A dirty return has installed `rb_…` markers, while `ReturnToSource(clean=false)` deliberately retains the target’s generation until admission. The probe refuses its own valid input. Separate prepared recovery from recovery into the source’s existing generation, and distinguish claim operation ID, hold ID and prepared generation ID. |
| + |
| +14. **P1 — The probe never enforces the actual launch-input pin.** |
| + [Plan 3:307](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-3-probe-commit-open.md:307) renders current configuration, then validates markers and admits. No step compares the newly rendered input versions and actual spawn environment with `move.inputs`. The comparison in Plan 2’s `configured` phase is earlier and can become stale. A bundle change or agent restart between configured and probing can therefore run different inputs and still commit. Enforce §7.7’s check at spawn; recovery must record its current inputs separately. |
| + |
| +15. **P1 — Declare-lost rewrites the state its own legality check requires.** |
| + [Plan 4:194](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-4-rollback-return-declare-lost.md:194) first sets the target to `released`, `Dirty=false`; line 190’s owner accepts restoring targets in `releasing` or `returning`, so validation against the resulting record refuses. Retaining an old record would conceal a different problem: a previously released dirty target now looks like an ordinary clean return before the loss claim is committed. Persist a distinct loss decision that ordinary recovery cannot consume until the corresponding durable declaration is complete. |
| + |
| +16. **P1 — The loss flow still cannot pass the real Plan 8 admission API.** |
| + [Plan 4:195](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-4-rollback-return-declare-lost.md:195) creates a prepared record without `WriteSources` and never advances it to committed with `ResultClaimGen`. On `hostclaim-leases`, `lost.go:119` requires committed state; `:149–191` requires exact frozen sources and validated baselines. The plan also puts **every** target session in `AdditionalWritesLost`, duplicating the latest predecessor already represented by primary `WritesLost`. Local chore discharge additionally needs exact `DestroyedStorage` attestations, which the proposed payload omits. Port the branch’s real declaration/handover/admission sequence, including crashes, instead of stopping the test at `DeclareLost`. |
| + |
| +17. **P1 — The destroyed-local-only loss case performs unrelated regional retirement.** |
| + [Plan 4:196](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-4-rollback-return-declare-lost.md:196) unconditionally queues a target-region whole-location park and covers predecessors after declaration. Case (c) is explicitly legal after commit because it should discharge only local chores and change no data authority. Give it a separate consequence path; assert that holder authority, regional chores and coverage remain unchanged. |
| + |
| +18. **P1 — Declared-lost rollback still waits for a stamp only the lost box writes.** |
| + [Plan 4:131](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-4-rollback-return-declare-lost.md:131) requires `TargetParkedAt` to close. Its only producer is the target’s `cleaning` phase at line 107. Task 10 removes that box from participation and requires it to refuse move work; no surviving actor completes the corresponding record state. Define terminal accounting for discharged local work and surviving regional cleanup, then test through `LastMove`, not merely source re-admission. |
| + |
| +19. **P2 — The readiness window starts before restore, contrary to the specified budget.** |
| + [Plan 3:331](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-3-probe-commit-open.md:331) records `Opened=now` before `ProbeWake`, which then renders, admits, purges and restores. The spec starts the 30-second window at the first app start and prices restore separately. Large restores consume or exhaust the readiness allowance before the app runs. Persist admission/attempt accounting separately from the first-start timestamp; include restore delay in the test. |
| + |
| +20. **P2 — Randomized archive sealing defeats the promised replay identity.** |
| + [Plan 2:716](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-2-cut-flip-landing.md:716) replays `Build + Seal` from the top and claims create-only PUT returns the same object. `Seal` generates a fresh key and ciphertext, so `files-<sealed>` is a different key every time. Persist the chosen archive identity, manifest and sealed key before publication, then reuse those bytes. Test a crash after each PUT using the real randomized sealer. |
| + |
| +21. **P2 — Timer convergence can still write after the snapshot barrier.** |
| + [Plan 2:738](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-2-cut-flip-landing.md:738) drains runner claims but only checks `Frozen` on entry to `convergeHost`. Main explicitly runs convergence from overlapping agent loops. A converger paused after its check can resume and upsert/delete/fire timers after the cut snapshots them; `Runner.Busy` does not cover it. Include convergence in the freeze/drain protocol and test that interleaving. |
| + |
| +22. **P2 — T11 admits moves using example rates rather than measured rates.** |
| + [Plan 1:2559](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-1-copy-and-staging.md:2559) defaults missing `CARLOS_MOVE_RATES` to `ExampleRates`. The approved T11 requirement is explicitly based on measured cross-region, regional-storage and disk rates. No measurement producer or freshness requirement is planned. Require usable measurements before passing T11; missing measurements must not silently become the worked example. |
| + |
| +23. **P2 — Log sequencing and deduplication do not provide the claimed guarantees.** |
| + [Plan 6:79](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-6-cli-status-observability.md:79) assumes writers choosing the same sequence collide, but keys contain both sequence and event name: two different events at sequence N both succeed. Deduplicating by event name also suppresses legitimate repeated `probe_attempt`, `stuck` and `retry` occurrences. Use durable transition/attempt identities and test concurrent writers plus repeated occurrences. |
| + |
| +24. **P2 — Copy approval and several concrete APIs still arrive after their consumers.** |
| + [Plan 1:2975](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-1-copy-and-staging.md:2975) asks the new gate to become green although Task 4 already writes strings whose record remains pending. Approval must precede those implementation tasks. Separately, Task 4 uses `moves.ProtocolVersion` before Task 12 defines it; Task 11 uses Task 14’s harness. `leasetest.LocalStore` at line 1363 does not exist on main, and inline `hostguard.Capture(...)` calls ignore its `(Captured,error)` signature. Replace placeholders with actual setup and move prerequisites before the per-task commit gates. |
| + |
| +25. **P2 — The rehearsal’s declared prerequisites cannot execute its loss run.** |
| + [Plan 7:79](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-7-rehearsal-runbook.md:79) lists rollout, grants, floor and adoption, but not claim Plan 8b’s activation prerequisites. Run 4 then invokes a door that Plan 4 explicitly refuses outside tests until 8b. Add that prerequisite and the real adoption integration; merging the three named branches alone does not satisfy either. |
| + |
| +The approved design can remain intact, but **the remaining work is not entirely local**. Several fixes are small edits; admission, loss recovery, retirement and activation require coordinated contracts across the moves plans and claim integration work. The series needs another revision before implementation can safely proceed. |
| + |
| +--- |
| + |
| +**FIRST PROMPT:** |
| +Read-only round-2 review of the moves Phase 1 plans at `da2832aa`. |
| + |
| +**WHAT:** |
| +Check round-1 fixes, new defects and build readiness. |
| + |
| +**STATUS:** |
| +Review complete. PLAN: NOT READY. Repository unchanged. |
| + |
| +**NEEDED:** |
| +Nothing for this review. |
| + |
| +**RECOMMENDED:** |
| +Revise the blocked interfaces and recovery sequences, then review the updated series. |
| + |
| +**SESSION:** |
| +Safe to close. Nothing running; no handoff file written. |
| \ No newline at end of file |
diff --git a/docs/superpowers/specs/2026-10-02-move-any-app-design.md b/docs/superpowers/specs/2026-10-02-move-any-app-design.md| index e2588c24e..0f8487eef 100644 |
| --- a/docs/superpowers/specs/2026-10-02-move-any-app-design.md |
| +++ b/docs/superpowers/specs/2026-10-02-move-any-app-design.md |
| @@ -1,8 +1,30 @@ |
| # Move any app or instance to another region: Phase 1, a safe single-instance move |
| **Date:** 2026-10-06 |
| -**Revision:** 7.18. |
| - |
| +**Revision:** 7.19. |
| + |
| +- **Rev 7.19** records Paul's rulings of 2026-10-06 on amadan #236 as |
| + claim rev 6.27 (989ee349) §1.1 states them, and folds in region HA's |
| + requests L14 to L17 (spec 1 rev 5.4, branch `design-region-ha-rev5`, |
| + `2026-10-04-region-ha-box-placement-design.md` §9). New §0.5. **P7, |
| + cold-only admissions:** until a reviewed Litestream fix (#234) makes a |
| + clean-close certificate safe, every admission is cold, no certificate |
| + is issued, and nothing in Phase 1 depends on warm; §5.1 step 9's warm |
| + mode stays as specified future behaviour. **L14** (§7.10): a target |
| + terminated mid-move is taken off the move by the termination fence, |
| + as M16's evidence, automatically, under the before-commit rules. |
| + **L15** (§7.1, §7.3, §19.1): a move names its target box in the start |
| + CAS, chosen by region HA's `choose()`; readiness, evidence, install and |
| + recovery bind to it. **L16** (§5.1, §7.6): the install is one shared |
| + routine with relocation, fed by a sealed staged replica or a pinned set |
| + of remote versions. **L17** (§6.1, §8, §9.4): every installation writes |
| + a files manifest in its new generation; the archive at the cut stays; |
| + timer freezing and draining apply to `relocating` holds. **L18** |
| + (§14.2.6): the model stays at one box per region plus replacements; |
| + the two-box extension is stated as a limit, not done here. Nothing in |
| + the protocol of rev 7.18 changes otherwise. Conflicts and claim |
| + changes these raise are listed in `region-ha-conflicts.md` (the |
| + coordinator's record) for platform-1e. |
| - **Rev 7.18** folds in **joint round 17**'s two P3s |
| (`CODEX-FINDINGS-JOINT-R17.md`, cited "R17-n"); the round was **clean**, |
| A, B and C ready for moves Phase 1 implementation planning, after |
| @@ -872,6 +894,42 @@ and the moves session's three questions; rev 7.5 encodes it (§5.1, §6.4, |
| Until platform-1e adopts them, this spec states each as the behaviour it |
| relies on. |
| +### 0.5 Paul's rulings of 2026-10-06 (#236; claim rev 6.27 §1.1) and region HA's requests |
| + |
| +Claim rev 6.27 (989ee349) §1.1 records every ruling of amadan #236. The |
| +ones this spec acts on: |
| + |
| +- **P1, the floor is the first fully integrated release.** §11's |
| + `control/moves-protocol.json` names a real release that carries the |
| + claim, the moves and the region HA code together; no placeholder tag. |
| +- **P5, record-backed placement only.** A move's target is a record's |
| + region and a box the roster knows; nothing places by hand. |
| +- **P6, rename and transfer stay refused.** Out of scope here as before. |
| +- **P7, cold-only admissions.** Until a reviewed Litestream fix (#234) |
| + makes a clean-close certificate safe, **every admission is cold**: no |
| + `closed/<session>.json` is written, no `PredecessorCertified` admission |
| + is made, and no plan task depends on warm. §5.1 step 9's warm mode, |
| + the certificate ledger and `KeepStamp`'s interaction with warm stay in |
| + this spec as the behaviour a later revision turns on; the model keeps |
| + both modes so the rule stays checked. The cost in Phase 1: every wake |
| + restores from the remote (one regional S3 read and one disk write of |
| + the database), writes a fresh generation's first upload (one disk read |
| + and one regional S3 write) and its `generation.json`, and |
| + `PublishGeneration` queues one `park-generation` chore for the dead |
| + predecessor per wake. T11 already prices every probe attempt as a |
| + cold admission (rev 7.11, R10-8), so the readiness budget is |
| + unchanged by P7; the per-wake chore is a claim CAS and a regional |
| + park after the horizon, outside the cut window. |
| + |
| +Region HA (spec 1 rev 5.4, §9) asks four things of this spec, folded in |
| +as follows; the fifth, L18, is stated as a limit: |
| + |
| +- **L14** (§7.10): the termination fence is M16's evidence, automatically. |
| +- **L15** (§7.1, §7.3, §19.1): the target box is chosen in the start CAS. |
| +- **L16** (§5.1 step 2, §7.6): one install routine, two sources. |
| +- **L17** (§6.1, §8.3, §9.4): a files manifest per installation. |
| +- **L18** (§14.2.6): two boxes per region in the model is not done here. |
| + |
| ## 1. What happened in production |
| ### 1.1 Finding 1: busy instances never reached `ready` (fixed on main) |
| @@ -2249,7 +2307,17 @@ versions created after `H`, and INV-10 everywhere at the end of the run. |
| `POST /api/cli/apps/{account}/{app}/instances/{host}/move` |
| `{"region":"uk","cut":"auto"|"manual"}`, operator-only (§12). The door |
| CASes the claim from `serving` (holder = this incarnation, execution |
| -holder = the source box) to `moving` with op id = the move id (§19 K2). It |
| +holder = the source box) to `moving` with op id = the move id (§19 K2). |
| +**The target box is named in that CAS** (region HA L15, ruled H12): for |
| +a region with several boxes the door chooses it with region HA's |
| +`choose(eligibleBoxes(region, pool), app)` (spec 1 §5.2: prefer a box |
| +running no instance of the app, then the most headroom, then the SHA-256 |
| +tie-break), over the roster and the edges snapshot; for a one-box region |
| +it is that box. The choice is made once and recorded as `op.participants[1]` |
| +and `op.intent.target`; readiness (T8, T9), evidence, install and |
| +recovery bind to that box, and nothing substitutes another box: a target |
| +that cannot complete is replaced only by an explicit fenced transition |
| +(§7.10, L14), never by recovery picking a new one (P5-D2, A1-10). It |
| is refused (400 or 409, with the sentence), and every box step re-checks: |
| - not exec-backed `kind:"instance"`, addr-backed, pooled, fleet-placed, |
| @@ -2543,6 +2611,24 @@ journaled in `<DataDir>/.moves/<host>/<move id>/` (F12). |
| Any failure from `checking` to `probing`, or `landing_deadline`, triggers a |
| rollback (§7.8). |
| +**One install routine, two sources (region HA L16; rev 7.19).** |
| +`building` and `published` are one routine shared with region HA's |
| +relocation (spec 1 §5.8): its input is a **pinned set of remote |
| +versions** (key, version id, level, TXID range) plus the required final |
| +TXID, and its output is the prepared generation's snapshot, anchor, build |
| +intent and completed install manifest exactly as above. A move feeds it |
| +the seal's `replica/` versions and `N`; a relocation feeds it the |
| +versions `genselect.SelectPrepared` or the generation's current listing |
| +pins and that listing's maximum TXID. The anchor rule, the two durable |
| +states, the replay rule and the marker rule are the routine's, so both |
| +callers get them once. **Every installation writes a files manifest in |
| +its new generation** (L17, §8.3): after `local` has extracted the |
| +archive, the installed files directory is published as region HA §5.6's |
| +objects and manifest under `<location>files/<generation>/`, before |
| +`.files-gen` is written, so a failure after this landing can restore the |
| +files without an archive; the archive at the cut stays as the sealed |
| +transfer. |
| + |
| ### 7.7 The probe (codex 14) |
| `ProbeWake(host, op)`, legal only on a row held with `hold_op = op` and |
| @@ -2716,6 +2802,29 @@ runs anywhere. It converges when the target returns (INV-3). |
| ### 7.10 Stuck and declare-lost (M16) |
| +**The termination fence is M16's evidence, automatically (region HA |
| +L14, ruled H10; rev 7.19).** Region HA §5.9 defines the termination |
| +fence: a regional worker observes state exactly `terminated` for the |
| +box's instance id, account and region as the roster records them, and |
| +writes the create-only fence record `control/box-fences/<enrollment>.json`; |
| +the claim puts the box on `lost` citing it (claim L2 to L4). For a move |
| +whose **target** is fenced that way, the move takes the fence as the |
| +evidence `--declare-lost` would have supplied, with no operator: the |
| +same legality table below applies (the target owes stop evidence, or |
| +released dirty and has not returned), **under the same before-commit |
| +rule**: before the record's commit CAS the move rolls back clean through |
| +the declared-loss path, with `fenced_at` as every scope's evidence time |
| +and the fence record as `rollback.evidence`; after the record's commit |
| +the target is authoritative and the move is not what recovers the |
| +instance: the move finishes its own close (its remaining steps are parks |
| +run by regional workers), and the instance's recovery is region HA's |
| +relocation, outside this spec. A terminated **source** is likewise not |
| +the move's to recover before the flip: the move aborts (its `moving` |
| +claim releases through the fence's lost-box path) and relocation takes |
| +the host. The fence supplies the evidence; this spec decides what the |
| +move does with it; the operator's `--declare-lost` stays for the |
| +stopped, unclear and vanished boxes the fence does not cover (claim D2). |
| + |
| **What a declared-lost box leaves behind (rev 7.11).** Its acknowledged |
| writes beyond its uploads are declared by the next admission's |
| `writes_lost` (the record's `remote_max_txid` names where an operator |
| @@ -3146,6 +3255,17 @@ deterministic. |
| reuses the recorded object (codex 19). |
| - **Retention:** the source's snapshots stay until F3; staging is parked at |
| finalization (§7.9, §7.8). |
| +- **The files manifest (region HA L17; rev 7.19).** The archive is the |
| + transfer; durability between moves is region HA §5.6's files publisher: |
| + immutable objects `<location>files/<generation>/objects/<seq>-<sha256>` |
| + and one manifest per generation, replaced with `If-Match`. Every |
| + installation, a landing's and a dirty return's included, publishes its |
| + installed files directory and `<host>.env` as that generation's first |
| + manifest in `local`, before `.files-gen`; the serving writer then keeps |
| + it current per the app's `files_durability`. The classifier treats |
| + `files/<generation>/` as that generation's (claim L13), so a |
| + `park-generation` chore retires it with the generation, except objects |
| + the files live set names. |
| ## 9. Changes to existing code (codex 13, 15) |
| @@ -3203,6 +3323,12 @@ version)` stages exactly the pinned release (`move.inputs.release`). |
| one for INV-2, and finds timers lost when the files marker is written |
| before the timers are imported (R2-6). |
| - Declared schedules converge again once the claim returns to `serving`. |
| +- **`relocating` holds (region HA L17; rev 7.19).** The same freezing |
| + and draining apply to a row held by region HA's `relocating` op: the |
| + runner skips the held row with no bookkeeping, the converger neither |
| + upserts nor prunes the host's timers while the claim op is |
| + `relocating`, and the relocation's release flush carries the timers |
| + snapshot as §7.4 `stopped` does. |
| ### 9.5 Edge |
| @@ -5783,6 +5909,22 @@ baselines, this sweep and the witnesses are the evidence for rev 7.10. |
| #### 14.2.6 What the model does not prove |
| +- **One box per region (region HA L18; rev 7.19).** The model keeps one |
| + box per region plus replacement boxes (a destroyed box's successor). |
| + Region HA asks for two boxes per region with create spread, |
| + relocation by drain, termination mid-serve and in every relocation |
| + phase, nested failover, a stale edges file, an unenrolled reference, a |
| + volume without `DeleteOnTermination`, a crash mid-install, late file |
| + uploads and late PUTs after termination. That is a model of |
| + relocation, an operation this spec does not define, over a world |
| + whose `box.Region` and per-region bookkeeping assume one box; it is |
| + not a change this revision can make within the CPU policy (the gate |
| + plus one niced 3000 sweep per revision) or its scope. **Stated as a |
| + limit:** nothing in §14.2.3's results speaks for two boxes in one |
| + region, for `choose()`'s spread, or for a termination fence taking a |
| + participant off an operation. The extension belongs to a joint round |
| + with the region HA session, on its own branch, with its own sweeps, |
| + after relocation's protocol is fixed in claim rev 6.27's successors. |
| - **It is a model.** It checks the protocol, not the code that will |
| implement it. Its S3 is a versioned map with the documented semantics, |
| every one checked against the real service (§15 step 8). Its |
| @@ -5991,6 +6133,10 @@ to settle, not a product choice. |
| One op id (the move id) runs through every kind below. The kind changes by |
| CAS; the record's state is the move's own detailed journal. |
| +`op.participants` is `{source box, target box}` from the start CAS, the |
| +target chosen by `choose()` (§7.1, L15). **Every admission in every row |
| +below is cold while P7 stands** (§0.5): "warm or cold as for `serving`" |
| +reads "cold" until #234 lands. |
| | claim `op.kind` | record states | source may spawn | target may spawn | |
| |---|---|---|---| |