amadan
plans: moves Phase 1 review round 1 (Astra on 616f2499): all 32 findings fixed, findings committed unedited
MoveLandingJournal; stopevidence.Reconstruct (no activator cycle); the spawnClaimed handle; receipts vouch only on a complete match; the 45 min staging lease and the single-signature staging mirror; move.json replay without a target read; files/ and env archive namespaces; the target box touched first; PutReplace for mutable objects; the floor armed locally; stop evidence complete before ExecToNone; the archive in the seal before archived; the drain race; the LTX lock page; scratch restores; persisted local chores; PreparedCheck before admission; claim phases mirrored; declare-lost record-first with Plan 8's real schema; orphan claim discovery; Record.LastMove for the CLI's outcome; the probe app's flags and before-write mode; Plan 8b named; task order (copy gate first, PageOperator in Plan 5, dispatch with its heading).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
10 files changed,
+457
−146
docs/superpowers/plans/2026-10-06-moves-phase1-0-index.md+50 −4docs/superpowers/plans/2026-10-06-moves-phase1-1-copy-and-staging.md+151 −38docs/superpowers/plans/2026-10-06-moves-phase1-2-cut-flip-landing.md+70 −47docs/superpowers/plans/2026-10-06-moves-phase1-3-probe-commit-open.md+18 −11docs/superpowers/plans/2026-10-06-moves-phase1-4-rollback-return-declare-lost.md+18 −11docs/superpowers/plans/2026-10-06-moves-phase1-5-cleanup-and-parks.md+17 −6docs/superpowers/plans/2026-10-06-moves-phase1-6-cli-status-observability.md+19 −25docs/superpowers/plans/2026-10-06-moves-phase1-7-rehearsal-runbook.md+5 −4docs/superpowers/plans/2026-10-06-moves-phase1-copy.md+4 −0docs/superpowers/plans/CODEX-PLAN-MOVES-PHASE1-R1.md+105 −0
CI passed — run details
diff --git a/docs/superpowers/plans/2026-10-06-moves-phase1-0-index.md b/docs/superpowers/plans/2026-10-06-moves-phase1-0-index.md| index 0037b473f..fe358ef85 100644 |
| --- a/docs/superpowers/plans/2026-10-06-moves-phase1-0-index.md |
| +++ b/docs/superpowers/plans/2026-10-06-moves-phase1-0-index.md |
| @@ -24,7 +24,7 @@ Plan 7). |
| | # | plan | produces | needs from claim plans 5, 6, 8 | |
| |---|---|---|---| |
| -| 1 | [`2026-10-06-moves-phase1-1-copy-and-staging.md`](2026-10-06-moves-phase1-1-copy-and-staging.md) | op ids, the record's move journal, the `moving` claim CAS at the request door (operator-only), the per-op staging layout with `move.json`, seal-ready copy rounds, the files archive, the timers snapshot, the launch and readiness reports (T1 to T12, S1, S2), the T11 budget, the protocol floor and `moves_protocol`, the `hold_op` column | nothing | |
| +| 1 | [`2026-10-06-moves-phase1-1-copy-and-staging.md`](2026-10-06-moves-phase1-1-copy-and-staging.md) | op ids, the record's move journal, the `moving` claim CAS at the request door (operator-only), the per-op staging layout with `move.json`, seal-ready copy rounds, the files archive, the timers snapshot, the launch and readiness reports (T1 to T12, S1, S2), the T11 budget, the protocol floor and `moves_protocol`, the `hold_op` column, the copy gate | nothing | |
| | 2 | [`2026-10-06-moves-phase1-2-cut-flip-landing.md`](2026-10-06-moves-phase1-2-cut-flip-landing.md) | the journaled cut (requested to flipped), `ReconstructStopEvidence`, the seal by version, the archive publication, the flip CAS, hold rows and reconcile's participant rules, timers frozen and drained, `RenderHeld`, `MaterializeHeld`, the landing's `checking` to `restored` phases including the generation build (snapshot, anchor, build intent, install manifest), `park-superseded` and `park-staging` destinations | nothing | |
| | 3 | [`2026-10-06-moves-phase1-3-probe-commit-open.md`](2026-10-06-moves-phase1-3-probe-commit-open.md) | the probe policy (attempt windows, deadline checks, budgeted attempts), `ProbeWake` as a cold admission into the prepared generation, the commit record CAS and its claim half (obligations, `CoverRegionPredecessors`), the conditional open, the source's finalization park | Tasks 5 to 7 need claim Plan 6 (`hostbox.Writer`, the claim-aware start sequence); Tasks 1 to 4 and 8 do not | |
| | 4 | [`2026-10-06-moves-phase1-4-rollback-return-declare-lost.md`](2026-10-06-moves-phase1-4-rollback-return-declare-lost.md) | `reverting` (pre-flip recovery), the rollback decision, the target's release with the clean/dirty rule (`src_covered`), the dirty return lane, the source's `claimed` CAS (`holder.stamp = source_stamp`, clean `holder.generation = src_gen`), the dirty install on the source, the target's `cleaning`, `stuck`, `--retry`, declare-lost | the recovery probes need claim Plan 6 (as Plan 3); declare-lost (Task 10) needs claim Plan 8 (`hostclaim.DeclareLost`, `LossOperationOwner`, `AdmitWriterWithLoss`, `LossRecord`) | |
| @@ -75,7 +75,7 @@ changes with it. |
| - `hostbox.Regional.Run` must admit obligation kinds `park-superseded` |
| and `park-staging` beside `park-replica` and `park-generation` (Plan 5 |
| Task 2 defines the kinds; Plan 6's allow-list is the one line to widen). |
| -- `spawnClaimed(ctx, enabled bool, manager *cgroup.Manager, host string, limit cgroup.Limit, selected hostwrite.Work, identity hostwrite.Identity, build func() *exec.Cmd) (*exec.Cmd, error)`: every spawn under a move is strict (§5.3); `ProbeWake` spawns through it with `CARLOS_MOVE_PROBE=1`. |
| +- `spawnClaimed(ctx, enabled bool, manager *cgroup.Manager, host string, limit cgroup.Limit, selected hostwrite.Work, identity hostwrite.Identity, build func() *exec.Cmd) (*exec.Cmd, *cgroup.Prepared, error)` (claim Plan 6 line 878): every spawn under a move is strict (§5.3); `ProbeWake` spawns through it with `CARLOS_MOVE_PROBE=1`, keeps the returned `*cgroup.Prepared` for the process's whole life, and on every exit path (a failed start included) runs `Prepared.KillAndEmpty` then `Prepared.Close` before the attempt is counted. |
| - The wake admission of Plan 6 Task 9 must honour moves' K6: `doWake` reads the claim fresh and, when `op.exec` does not name this box, kills the host's process tree (`ReconstructStopEvidence`'s step 1, Plan 2 Task 3) before anything else; it admits a wake under `landing/committed` for the target (§7.14 row "landing, committed, not yet finalized") and under `reverting`/`restoring` phases `probed` and `opened` for the source. Its restore writes `<dbPath>.gen` and retires (`park.Local.RetireArtifacts`) a warm database whose `.gen` names another generation before restoring cold (§5.1 start check rule 3, §9.3). |
| - `schedule.SelectedTimers` (the timer-control adapter): Plan 2 Task 9's |
| `ReplaceTimers` call on the target runs under the landing's own |
| @@ -93,6 +93,8 @@ additions, `internal/hostexec`, `internal/hostquarantine`, |
| - `hostclaim.LossRecoveryOwner interface{ ValidateLossRecovery(Claim, LossRecord, Admission) error }` and `AdmitWriterWithLoss(ctx, s, pub, observed, in Admission, lossID string, validated []ValidatedWriteSource, owner LossRecoveryOwner) (Observed, error)`: the source's re-admission after a declared-lost target (the `writes_lost` for every target session since the cut and the source's cut session when `src_covered` is false; witnesses `wTwoSessionDeclaration`, `wNoDuplicateDeclarationEarly`, `wNoDuplicateDeclarationLate`). |
| - `hostclaim.Readmit(box string, quarantineRevision uint64) func(*Claim) error` and the `hostquarantine` retirement a returning box runs before `--readmit` (§7.10). |
| - `hostexec.Open(ctx, cfg hostexec.Config) (*Controller, error)` with `Ready/Lost/Close`: the lease controller every admitted writer runs under; `ProbeWake` runs its spawn under it once Plan 8 lands (Plan 3 Task 6 notes the seam). |
| +- `declareLostTransition`'s effects (claim Plan 8 line 1444): it journals `loss.<id>` (the canonical recovery intent), sets `op.exec = "none"` (the literal string, not `""`) when the lost box was the executor, removes the box from `op.participants`, reassigns its regional chores to `region:<r>`, discharges its local chores, and keeps the phase. Moves' `ReturnToSource` (Plan 4 Task 2) accepts that state: exec `""` or `"none"`, and either `evidence_target` or a `loss.<id>` journal entry. |
| +- **Activation prerequisites are claim Plan 8b's, not Plan 8's** (Plan 8 "Rollout runbook", line 5678): the authenticated release identity behind the floor, the root-signed actor directory and census, and the live quarantine, declare-lost and readmission wiring. Plan 4 Task 10 can be built against Plan 8's library and tests, but `carlos move resolve --declare-lost` stays refused in production (the door answers "declare-lost is not enabled in this deployment yet") until Plan 8b lands and Paul rules on #236's P1 to P3. A direct `park.Local.RetireInstallation` is the build-time stand-in for the returning box's quarantine in tests only; it is not an activation path. |
| **From claim Plan 5 (`feat/hostclaim-console`, package `internal/hostops`):** |
| @@ -111,14 +113,16 @@ additions, `internal/hostexec`, `internal/hostquarantine`, |
| - **D2 zstd.** The archive is tar then zstd (§8.1). The repository pins no |
| zstd library; Plan 1 adds `github.com/klauspost/compress` (pure Go) as |
| the one new module dependency of the series. |
| -- **D3 Staging-work lease.** `lease.Acquire(ctx, deploymentBucket, "control/staging-leases/"+targetRegion+"/"+host, deploymentRegion)` is the claim's shared staging-work lease (§7.3). Every staging writer takes it, checks `SafeUntil()` immediately before each request, and signs nothing after it. |
| -- **D4 Seal versions from the copy responses.** `lease.MirrorPrefix` records each copy's destination `versionId` in the receipts through an optional `lease.VersionedCopier` interface the S3 mirror implements, so the source never needs to read the target's `moves/*` (M17). The seal is the receipts' versions for the current source keys. |
| +- **D3 Staging-work lease.** `lease.AcquireTTL(ctx, deploymentBucket, "control/staging-leases/"+targetRegion+"/"+host, deploymentRegion, moves.StagingLeaseTTL)` is the claim's shared staging-work lease (§7.3), with `StagingLeaseTTL = 45 * time.Minute` (the ordinary 15-minute `LeaseTTL` cannot leave the signature's 15-minute life plus skew in front of `SafeUntil`). Every staging request goes through `moves.Guarded.Before`, which checks `SafeUntil()` and then calls the SDK in the same goroutine with no await between; the staging S3 client (`lease.NewStagingMirror`) has SDK retries disabled and a 120 s request timeout, so the SDK signs exactly once, immediately after the check, and never re-signs; the lane's own retry loop makes a new check for each new signature. The residual gap, a pause between the SDK's SigV4 computation and the socket write, is the pause assumption the spec names (§7.3, model 14), bounded by the skew allowance. |
| +- **D4 Seal versions from the copy responses.** `lease.MirrorPrefix` records each copy's destination `versionId` in the receipts through an optional `lease.VersionedCopier` interface the S3 mirror implements, so the source never needs to read the target's `moves/*` (M17). The seal is the receipts' versions for the current source keys, and a receipt is trusted only when it matches the current source ETag and size **and** the current destination ETag; anything else is recopied, which yields a fresh version id. |
| - **D5 The sealed restore is litestream's own planner.** `moves.SealedClient` implements `litestream.ReplicaClient` over the target bucket's versioned store and serves exactly the sealed `(key, versionId)` pairs; `litestream.NewReplicaWithClient(nil, client).Restore` builds the database (the pattern `internal/lease/lease.go:404` uses). Nothing reads the current listing. |
| - **D6 Park destinations.** Two new obligation kinds, `park-superseded` (a whole location) and `park-staging` (a `moves/<host>/<op>` lane), both archived under `superseded/<op id>/<host>/<retirement id>/{instances,moves}/`. `internal/park` gains the destination; the engine's manifests, version deletes, horizon and completion are unchanged. |
| - **D7 Reconcile learns holds from the host package.** `provision.ReconcileConfig` gains `Hold func(Record) (status, op string, ok bool)` and `KeepParticipant func(host string) bool`; the host package supplies both from its claim reads. Reconcile itself never reads a claim. |
| - **D8 The console's move pass.** `movesRecoveryPass` (Plan 5) writes the claim half the record decided (§7.5) and runs the deadline sweeps. It is a plain function so claim Plan 5's `Recovery.Pass` can call it. |
| - **D9 `provision.MoveRequest` is replaced by `provision.MoveStart`** in Plan 1, since a move without an id, an incarnation and a cut mode has no journal to run. `MoveComplete` is replaced by `MoveFlip` and `MoveClose`. The old staging-wins landing (`promoteStagedReplica`) is deleted in Plan 2. |
| - **D10 Flat landing only.** Phase 1 lands flat (§6.2, §21); a nested source is refused at the door and at every box step. |
| +- **D11 The terminal outcome survives the close.** Every close CAS (`done`, `rolled_back`, `aborted`) writes `Record.LastMove = &MoveOutcome{ID, Outcome, At, OfflineS}` as it clears `Move`, so a CLI that missed the intermediate states reads the outcome of the move id it requested from `GET …/move` (`{"state":"none","last":{…}}`) and exits with the right code. |
| +- **D12 Order within the series.** The copy gate (the record file and the ship check) is Plan 1 Task 15, so every later plan's strings are checked from the day they are written; the paging primitive `host.PageOperator` is Plan 5 Task 0, before the console pass that needs it; `carlos move`'s dispatch and its `docs/site/cli.md` heading land in one task (Plan 6 Task 1), so no commit fails the docsite gate. |
| ## Global constraints (every plan repeats these) |
| @@ -129,6 +133,48 @@ additions, `internal/hostexec`, `internal/hostquarantine`, |
| - A new `carlos` command is not done until `docs/site/cli.md` names it (the `internal/docsite` gate). |
| - Wire shapes are additive: every new record field is `omitempty`, every new registry column has a default, every new claim journal key is a string. |
| +## Review round 1 (Astra, codex, on 616f2499): dispositions |
| + |
| +The findings are committed unedited as `CODEX-PLAN-MOVES-PHASE1-R1.md` |
| +beside this index. Verdict then: PLAN: NOT READY. Every finding was |
| +verified against the pinned main checkout and accepted; where each is |
| +fixed: |
| + |
| +| # | finding | fixed in | |
| +|---|---|---| |
| +| 1 | `MoveLanding` type and constant collide | Plan 1 Task 2: the journal type is `MoveLandingJournal` | |
| +| 2 | `activator` importing `stopevidence` is a cycle; `a.cgroups` is no field | Plan 2 Task 3: `stopevidence.Reconstruct(ctx, ReconstructInputs{Cgroups: eb.Cgroups, …})` | |
| +| 3 | `spawnClaimed` returns `(*exec.Cmd, *cgroup.Prepared, error)` | Index interfaces, Plan 3 Task 5: the handle is kept and cleaned on every exit path | |
| +| 4 | a receipt's version trusted on ETag equality alone | Plan 1 Task 5 and D4: a receipt vouches only when it matches source and destination completely; else recopy | |
| +| 5 | a 15-minute lease never passes the guard | Plan 1 Task 6 and D3: `moves.StagingLeaseTTL` 45 min, `lease.AcquireTTL`, a test on real settings | |
| +| 6 | the SDK signs at send and re-signs retries | Plan 1 Task 6 and D3: `lease.NewStagingMirror` with no SDK retries and a 120 s timeout; one request per `Before`; lane retries re-check | |
| +| 7 | `move.json` replay needed a forbidden read | Plan 1 Task 14, Review Focus 2: ledger intent first; `ErrExists` with a pending intent is our own PUT; without, refuse; the target validates | |
| +| 8 | an application file named `env` collides with the overlay | Plan 1 Task 7: `files/` and `env` namespaces, duplicate destinations refused, round-trip test | |
| +| 9 | T9 needs the target box in `touched` before the flip | Plan 1 Task 14: the first `Touch` adds the target region and box | |
| +| 10 | `Put(..., "")` is create-only; reports could not refresh | Plan 1 Task 11 `moves.PutReplace`, Task 12, Plan 6 Task 3 corrected | |
| +| 11 | the floor failed open on a read error; the door ignored completeness | Plan 1 Task 12: the floor is armed locally; unknown refuses a participant; the door needs `readEdgesSnapshot(...).complete()` | |
| +| 12 | stop evidence published before its final proof | Plan 2 Task 5: `stopping` keeps steps 1 to 4 local; the complete report is written at `stopped` before `ExecToNone` | |
| +| 13 | `archived` persisted before the archive was in the seal | Plan 2 Task 5: publication and the seal complete before the phase CAS; the flip checks `seal.Files` | |
| +| 14 | a paused tick could slip the drain | Plan 2 Task 6: the hold check runs after `claim`; `TestDrainRaceCannotAdvanceBookkeeping` | |
| +| 15 | the LTX encoder refuses the lock page | Plan 2 Task 8: `ltx.LockPgno` skipped, `Commit` kept, a >1 GiB sparse test | |
| +| 16 | a replayed build restored over an existing file | Plan 2 Task 8: the scratch database is removed before every restore | |
| +| 17 | `RetireArtifacts` without a persisted chore does nothing | Plan 2 Tasks 9 and 10: `QueueLocalPark` and `RecordInventory` at `checking`; the test asserts the filesystem | |
| +| 18 | the first probe's start check compared against the source generation | Plan 3 Tasks 4 and 5: `PreparedCheck` before admission; `StartCheck` after the CAS inside the admitted sequence | |
| +| 19 | the claim never reached `probing`/`probed` | Plan 2 Task 10 and Plan 3 Task 6: every landing phase mirrors into the claim with `SetMovePhase`; the commit test drives it | |
| +| 20 | declare-lost could override a winning commit | Plan 4 Task 10: the record decides first (`MoveRollBack`, `MoveRollbackTargetLost`), the claim follows | |
| +| 21 | the loss payload did not match claim Plan 8 | Plan 4 Task 10: the exact `LossRecord`, `LossCloudObservation`, `LossOperatorAttestation`, `LossConsequences` fields | |
| +| 22 | recovery could not consume the declaration result | Plan 4 Task 2: `ReturnToSource` accepts exec `"none"` and a `loss.<id>` journal; Task 10 runs the source pass against a real declaration | |
| +| 23 | orphan claims were never visited | Plan 5 Task 7: claim discovery from `control/hosts/`, `ReleaseOrphanMove` | |
| +| 24 | the CLI inferred the outcome from the last state seen | D11 and Plans 1, 3, 4, 5, 6: `Record.LastMove`, `GET …/move` returns `last`, the CLI reads it | |
| +| 25 | the probe app's launch contract differed from the backend | Plan 7 Task 1: `--socket`, `--db`, `TestMoveprobeLaunchContract` | |
| +| 26 | activation depends on claim Plan 8b | Index interfaces: 8b's prerequisites named; declare-lost stays refused until 8b lands | |
| +| 27 | forbidden transitions requested | Plan 4 Task 1: `MoveStuck` from `restoring`/`verified`, `MoveRollbackTargetLost` | |
| +| 28 | reverting had no staging cleanup producer | Plan 2 Task 2 `Revert` queues the park; Plan 4 Task 6 waits; Plan 5 Task 1 dispatches | |
| +| 29 | the MinIO test would never run in CI | Plan 1 Task 11: added to `30-minio-contract`; failing cases for the stubbed hooks | |
| +| 30 | task-order cycle and a failing docsite commit | D12: `PageOperator` is Plan 5 Task 0; dispatch and heading in Plan 6 Task 1; the copy gate is Plan 1 Task 15 | |
| +| 31 | the copy check needed an independent inventory | Plan 1 Task 15: `movesCopyIDs` pinned, negative fixtures | |
| +| 32 | the clean-rollback rehearsal wrote before refusing | Plan 7 Tasks 1 and 3: the before-write mode suppresses every mutation | |
| + |
| ## What the model proved and the plans port |
| Witnesses that become real tests, by plan: Plan 2 ports `wSealByVersion`, |
diff --git a/docs/superpowers/plans/2026-10-06-moves-phase1-1-copy-and-staging.md b/docs/superpowers/plans/2026-10-06-moves-phase1-1-copy-and-staging.md| index 88047b38c..f5b1258d3 100644 |
| --- a/docs/superpowers/plans/2026-10-06-moves-phase1-1-copy-and-staging.md |
| +++ b/docs/superpowers/plans/2026-10-06-moves-phase1-1-copy-and-staging.md |
| @@ -24,7 +24,7 @@ |
| ## Review Focus |
| 1. A record whose `move` was written by a v0.22 console (`MoveState` without `id`) must be refused by the new worker on every pass, logged once a minute, and never mirrored or cut. Test in Task 2 (`TestMoveStateLegacyRecordIsRefused`). |
| -2. A `move.json` that already exists in the target bucket under a fresh move id (an id collision or a replayed request) must stop the first round with a refusal, not overwrite it. Test in Task 5 (`TestStagingMoveJSONIsCreateOnly`). |
| +2. A `move.json` that already exists in the target bucket when the source has no record of writing it must stop the round with a refusal, not overwrite it; a `move.json` the source itself wrote but whose ledger entry was lost in a crash must be accepted without reading it back (M17 forbids the read). Test in Task 14 (`TestStagingMoveJSONIsCreateOnly`, `TestStagingMoveJSONReplayAfterLostLedger`). |
| 3. A files directory containing a symlink must refuse the archive with the path, and the round must keep mirroring the replica (the refusal is reported in `target.json`'s T5 input, not fatal to the move until the cut). Test in Task 7 (`TestArchiveRefusesSpecialFiles`). |
| 4. An operator who names the instance's own current region, or a region outside the manifest's `regions`, gets the refusal sentence and no claim write. Test in Task 4 (`TestMoveDoorRefusesWithoutClaimWrite`). |
| 5. A target report older than 10 minutes, even one that passed, must not promote `ready`; a stale report after `ready` must demote nothing (the cut re-checks freshness at `admitted`, Plan 2). Test in Task 11 (`TestReadyNeedsFreshReports`). |
| @@ -188,7 +188,7 @@ Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| - Test: `internal/provision/move_test.go` |
| **Interfaces:** |
| -- Produces: the extended `provision.MoveState` below; state constants `MovePreparing, MoveReady, MoveCutting, MoveReverting, MoveAborting, MoveLanding, MoveCommitted, MoveRollingBack, MoveStuck`; `provision.MoveStart{ID, Incarnation, From, To, Cut, By string; Now time.Time}` and `provision.MoveStartTransition(p MoveStart) func(*Record) error`; `(m *MoveState) Legacy() bool`; `Manifest.Moves *MovesManifest{ReadyPath string}`; `(m Manifest) ReadyPathOrDefault() string`. |
| +- Produces: the extended `provision.MoveState` below; `Record.LastMove *MoveOutcome \`json:"last_move,omitempty"\`` beside `Record.Move` (`provision.go:58`); state constants `MovePreparing, MoveReady, MoveCutting, MoveReverting, MoveAborting, MoveLanding, MoveCommitted, MoveRollingBack, MoveStuck`; `provision.MoveStart{ID, Incarnation, From, To, Cut, By string; Now time.Time}` and `provision.MoveStartTransition(p MoveStart) func(*Record) error`; `(m *MoveState) Legacy() bool`; `Manifest.Moves *MovesManifest{ReadyPath string}`; `(m Manifest) ReadyPathOrDefault() string`. |
| - Later plans add the phase transitions (Plan 2 Task 1 the cut phases, Plan 2 Task 8 the landing phases, Plan 4 Task 1 the rollback phases). |
| - [ ] **Step 1: Write the failing test** |
| @@ -358,8 +358,8 @@ type MoveState struct { |
| FlippedAt *time.Time `json:"flipped_at,omitempty"` |
| LandingDeadline *time.Time `json:"landing_deadline,omitempty"` |
| - LandingPhase string `json:"landing_phase,omitempty"` |
| - Landing *MoveLanding `json:"landing,omitempty"` |
| + LandingPhase string `json:"landing_phase,omitempty"` |
| + Landing *MoveLandingJournal `json:"landing,omitempty"` |
| CommittedAt *time.Time `json:"committed_at,omitempty"` |
| OpenedAt *time.Time `json:"opened_at,omitempty"` |
| @@ -397,8 +397,21 @@ type MoveFilesDigest struct { |
| Entries int `json:"entries,omitempty"` |
| } |
| -type MoveLanding struct { |
| +// MoveLandingJournal is the landing's own record (§7.2 "landing"). Named |
| +// so because MoveLanding is the state constant below. |
| +type MoveLandingJournal struct { |
| InstalledDBSHA256 string `json:"installed_db_sha256,omitempty"` |
| + Attempts int `json:"attempts,omitempty"` |
| +} |
| + |
| +// MoveOutcome is the terminal result the close CAS leaves on the record |
| +// as it clears Move (index D11), so a CLI that missed the intermediate |
| +// states still reads the outcome of the move it requested. |
| +type MoveOutcome struct { |
| + ID string `json:"id"` |
| + Outcome string `json:"outcome"` // done | rolled_back | aborted |
| + At time.Time `json:"at"` |
| + OfflineS float64 `json:"offline_s,omitempty"` |
| } |
| // MoveRollback is §7.8's journal. |
| @@ -978,7 +991,11 @@ func (s *Server) handleAPIInstanceMoveRequest(w http.ResponseWriter, r *http.Req |
| jsonError(w, http.StatusBadRequest, "region "+to+" cannot take a move in this deployment. it publishes no staging bucket. Regions that can: "+stagingCodes(caps)) |
| return |
| } |
| - targetBox, floorOK := moveTargetBox(ctx, s.cfg.Releases, to, rec.Region) |
| + targetBox, floorOK, complete := moveTargetBox(ctx, s.cfg.Releases, to, rec.Region) |
| + if !complete { |
| + jsonError(w, http.StatusConflict, "the fleet's edges files could not all be read. try again") |
| + return |
| + } |
| if targetBox == "" { |
| jsonError(w, http.StatusBadRequest, "no box in "+to+" publishes an edges file yet") |
| return |
| @@ -1030,10 +1047,16 @@ func (s *Server) handleAPIInstanceMoveRequest(w http.ResponseWriter, r *http.Req |
| // moveTargetBox is the lexically-first box publishing placement to (the |
| // same rule as moveTargets in internal/host), and whether every box in |
| -// either region is at or above the protocol floor (§11, K10). |
| -func moveTargetBox(ctx context.Context, s release.Store, to, from string) (box string, floorOK bool) { |
| - floorOK = true |
| - for _, f := range readEdgesFiles(ctx, s) { |
| +// either region is at or above the protocol floor (§11, K10). An |
| +// incomplete snapshot answers neither: a box whose edges file could not |
| +// be read may be below the floor, so the door refuses (complete == false). |
| +func moveTargetBox(ctx context.Context, s release.Store, to, from string) (box string, floorOK, complete bool) { |
| + snap, err := readEdgesSnapshot(ctx, s) |
| + if err != nil || !snap.complete() { |
| + return "", false, false |
| + } |
| + floorOK, complete = true, true |
| + for _, f := range snap.Files { |
| if f.Placement != to && f.Placement != from { |
| continue |
| } |
| @@ -1044,7 +1067,7 @@ func moveTargetBox(ctx context.Context, s release.Store, to, from string) (box s |
| box = f.Box |
| } |
| } |
| - return box, floorOK |
| + return box, floorOK, complete |
| } |
| ``` |
| @@ -1157,6 +1180,17 @@ func TestMirrorPrefixRecordsDestinationVersions(t *testing.T) { |
| if r2.BytesCopied != 0 || r2.Versions["0000/a.ltx"] != r1.Versions["0000/a.ltx"] || len(r2.Versions) != 2 { |
| t.Fatalf("round 2: %+v", r2) |
| } |
| + // A crash between the copy and the receipt write: the next round sees |
| + // the destination's ETag equal to the source's but no receipt (or a |
| + // receipt for an older version of the key). It recopies rather than |
| + // sealing a version it cannot vouch for. |
| + f.put("src", "instances/h/0000/a.ltx", "e1-new") |
| + f.n++ // a copy whose receipt was never persisted |
| + f.store("dst", "moves/h/mv/replica/0000/a.ltx", fakeObject{MirrorObject{ETag: "e1-new"}, "e1-new"}) |
| + r4, err := MirrorPrefix(context.Background(), f, "src", "instances/h/", "dst", "moves/h/mv/replica/", rc) |
| + if err != nil || r4.Versions["0000/a.ltx"] == r1.Versions["0000/a.ltx"] || r4.BytesCopied == 0 { |
| + t.Fatalf("stale receipt sealed: %+v %v", r4, err) |
| + } |
| // A plain MirrorStore records no versions and still works. |
| g := newFakeMirror() |
| g.put("src", "instances/h/0000/a.ltx", "e1") |
| @@ -1243,7 +1277,7 @@ type VersionedPutter interface { |
| } |
| ``` |
| -`MirrorRound` gains `Versions map[string]string` (doc: relative key → destination version id for every current source key with a known copy). `copyReceipt` gains `DstVersionID string \`json:"dst_version_id,omitempty"\``. In `MirrorPrefix`: initialise `round.Versions = map[string]string{}`; in the `staged && d.ETag == s.ETag` early-continue and the receipt-match continue, look up the receipt and, when `rcpt.DstVersionID != ""`, record `round.Versions[r] = rcpt.DstVersionID` (so the first branch now also reads the receipt: `if rcpt, ok, err := book.get(ctx, r); err == nil && ok && rcpt.DstVersionID != "" { round.Versions[r] = rcpt.DstVersionID }`); replace the copy call with: |
| +`MirrorRound` gains `Versions map[string]string` (doc: relative key → destination version id for every current source key with a known copy). `copyReceipt` gains `DstVersionID string \`json:"dst_version_id,omitempty"\``. In `MirrorPrefix`: initialise `round.Versions = map[string]string{}`. A receipt vouches for a version only when it matches the **current** source and destination completely: `rcpt.SrcETag == s.ETag && rcpt.SrcSize == s.Size && rcpt.DstETag == d.ETag && d.Size == s.Size && rcpt.DstVersionID != ""`. The `staged && d.ETag == s.ETag` early-continue therefore becomes: read the receipt; if it matches completely, record `round.Versions[r] = rcpt.DstVersionID` and continue; otherwise fall through to the copy below (a copy whose receipt was lost, or a receipt from an earlier version of the key, is recopied, which yields a fresh version id; the stale version stays unreferenced). The existing receipt-match branch records the version the same way. Replace the copy call with: |
| ```go |
| var etag, ver string |
| @@ -1279,8 +1313,8 @@ Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| - Create: `internal/moves/lease.go`, `internal/moves/lease_test.go` |
| **Interfaces:** |
| -- Consumes: `lease.Acquire(ctx, bucket, prefix, region) (*lease.Held, error)`, `(*lease.Held).SafeUntil() (time.Time, error)`, `Release`. |
| -- Produces: `moves.StagingLeasePrefix(targetRegion, host string) = "control/staging-leases/"+targetRegion+"/"+host`; `moves.SafeUntilGuard interface{ SafeUntil() (time.Time, error) }`; `moves.Guarded{Lease SafeUntilGuard; Now func() time.Time; Skew time.Duration}` with `(g Guarded) Before(fn func() error) error` returning `moves.ErrLeaseUnsafe` when `Now() >= SafeUntil() - Skew - 15 min`? No: `lease.Held.SafeUntil` already subtracts the renewal margin; the §7.3 rule subtracts the signature's whole life. `Before` refuses when `Now().Add(SignatureLife).After(safe)` with `SignatureLife = 15 * time.Minute` and `Skew` (default 1 min). |
| +- Consumes: `lease.acquireTTL(ctx, bucket, prefix, region, ttl)` (`lease.go:129`, exported in this task as `lease.AcquireTTL`), `(*lease.Held).SafeUntil() (time.Time, error)`, `Release`; `lease.NewS3Mirror` (`mirror.go:388`), extended in this task with `lease.NewStagingMirror(ctx, srcRegion, dstRegion string) (MirrorStore, error)`: the same client pair built with `o.RetryMaxAttempts = 1` (no SDK retries, so a request is signed once) and an `http.Client{Timeout: 120 * time.Second}` (the request limit), so a signature is computed immediately after the caller's check and never recomputed. |
| +- Produces: `moves.StagingLeaseTTL = 45 * time.Minute` (D3: the ordinary 15-minute `LeaseTTL` cannot leave the signature's life in front of `SafeUntil`); `moves.StagingLeasePrefix(targetRegion, host string) = "control/staging-leases/"+targetRegion+"/"+host`; `moves.SafeUntilGuard interface{ SafeUntil() (time.Time, error) }`; `moves.Guarded{Lease SafeUntilGuard; Now func() time.Time; Skew time.Duration}` with `(g Guarded) Before(fn func() error) error`: `lease.Held.SafeUntil` already subtracts the renewal margin; the §7.3 rule subtracts the signature's whole life, so `Before` refuses with `moves.ErrLeaseUnsafe` when `Now().Add(SignatureLife + Skew).After(safe)` (`SignatureLife = 15 * time.Minute`, `Skew` default 1 min) and otherwise calls `fn` at once in the same goroutine. The lane's retry loop (Task 14) calls `Before` again for each retry; nothing retries inside `fn`. |
| - [ ] **Step 1: Write the failing test** |
| @@ -1313,6 +1347,37 @@ func TestGuardedRefusesWithinTheSignatureLife(t *testing.T) { |
| if StagingLeasePrefix("uk", "a.go.example.com") != "control/staging-leases/uk/a.go.example.com" { |
| t.Fatal(StagingLeasePrefix("uk", "a.go.example.com")) |
| } |
| + // The real settings: a fresh staging lease (45 min TTL) is usable for |
| + // its first 29 minutes minus the renewal margin; the ordinary 15 min |
| + // lease would never pass the guard, which is why StagingLeaseTTL exists. |
| + if StagingLeaseTTL <= SignatureLife+DefaultSkew+15*time.Minute { |
| + t.Fatalf("StagingLeaseTTL %v leaves no room for the signature life", StagingLeaseTTL) |
| + } |
| +} |
| + |
| +// With the lease package's local test leaser (internal/lease/leasetest), a |
| +// lease acquired with StagingLeaseTTL passes the guard right after |
| +// acquisition and a lease acquired with lease.LeaseTTL does not. |
| +func TestGuardAgainstRealLeaseSettings(t *testing.T) { |
| + ctx := context.Background() |
| + store, bucket, region := leasetest.LocalStore(t) // the package's existing local S3-compatible fixture; name per leasetest |
| + held, err := lease.AcquireTTL(ctx, bucket, "control/staging-leases/uk/h", region, StagingLeaseTTL) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + defer held.Release(ctx) |
| + if err := (Guarded{Lease: held, Now: time.Now}).Before(func() error { return nil }); err != nil { |
| + t.Fatalf("staging lease refused at acquisition: %v", err) |
| + } |
| + short, err := lease.AcquireTTL(ctx, bucket, "control/staging-leases/uk/h2", region, lease.LeaseTTL) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + defer short.Release(ctx) |
| + if err := (Guarded{Lease: short, Now: time.Now}).Before(func() error { return nil }); !errors.Is(err, ErrLeaseUnsafe) { |
| + t.Fatalf("a 15 minute lease passed the guard: %v", err) |
| + } |
| + _ = store |
| } |
| ``` |
| @@ -1337,6 +1402,12 @@ import ( |
| // safe for that whole life plus the clock skew (§7.3, model 14, 18). |
| const SignatureLife = 15 * time.Minute |
| +// StagingLeaseTTL is the staging-work lease's term (D3). The ordinary |
| +// lease.LeaseTTL is 15 minutes, which can never leave SignatureLife plus |
| +// the skew in front of SafeUntil; 45 minutes gives every renewal cycle |
| +// room, and the renewer keeps the bound ahead while the lane works. |
| +const StagingLeaseTTL = 45 * time.Minute |
| + |
| // DefaultSkew is the clock-skew allowance the guard subtracts. |
| const DefaultSkew = time.Minute |
| @@ -1355,7 +1426,9 @@ type SafeUntilGuard interface{ SafeUntil() (time.Time, error) } |
| // Guarded wraps every staging request: the check runs immediately before |
| // fn signs, after any pause, and a request that cannot complete within the |
| -// signature's life before the lease ends is dropped. |
| +// signature's life before the lease ends is dropped. fn must issue exactly |
| +// one request through a client that signs once and never retries |
| +// (lease.NewStagingMirror); a caller that wants a retry calls Before again. |
| type Guarded struct { |
| Lease SafeUntilGuard |
| Now func() time.Time |
| @@ -1387,7 +1460,7 @@ Expected: PASS |
| ```bash |
| git add internal/moves/lease.go internal/moves/lease_test.go |
| -git commit -m "moves: the staging-work lease prefix and the SafeUntil guard (§7.3) |
| +git commit -m "moves: the staging-work lease (45 min TTL) and the SafeUntil guard; lease: AcquireTTL and the single-signature staging mirror (§7.3) |
| Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| ``` |
| @@ -1430,6 +1503,11 @@ func filesFixture(t *testing.T) (dir, env string) { |
| os.WriteFile(filepath.Join(dir, "a.txt"), []byte("alpha"), 0o644) |
| os.WriteFile(filepath.Join(dir, "sub", "b.bin"), []byte{1, 2, 3}, 0o600) |
| os.WriteFile(filepath.Join(dir, ".files-gen"), []byte(`{"op":"x"}`), 0o644) |
| + // A legal application file named env, and a directory named env with |
| + // a file in it: both must survive beside the <host>.env overlay. |
| + os.WriteFile(filepath.Join(dir, "env"), []byte("app-env"), 0o644) |
| + os.MkdirAll(filepath.Join(dir, "sub", "env"), 0o755) |
| + os.WriteFile(filepath.Join(dir, "sub", "env", "x"), []byte("x"), 0o644) |
| env = filepath.Join(t.TempDir(), "h.env") |
| os.WriteFile(env, []byte("CARLOS_ADMIN_TOKEN=t\n"), 0o600) |
| return dir, env |
| @@ -1452,9 +1530,12 @@ func TestBuildIsDeterministicAndExcludesTheMarker(t *testing.T) { |
| for _, e := range entries { |
| paths[e.Path] = true |
| } |
| - if paths[".files-gen"] || !paths["a.txt"] || !paths["sub/b.bin"] || !paths["env"] || n != 5+3+len("CARLOS_ADMIN_TOKEN=t\n") { |
| + if paths["files/.files-gen"] || !paths["files/a.txt"] || !paths["files/sub/b.bin"] || !paths["env"] || !paths["files/env"] || !paths["files/sub/env/x"] || n != 5+3+7+1+len("CARLOS_ADMIN_TOKEN=t\n") { |
| t.Fatalf("entries %v bytes %d", entries, n) |
| } |
| + // An application file and directory named "env" are legal and carried |
| + // under files/, never confused with the overlay (round-tripped in |
| + // TestSealOpenExtractRoundTrip through the extra fixture entries). |
| plain, bytesN, count, err := PlainDigest(dir, env) |
| if err != nil || plain != p1 || bytesN != n || count != len(entries) { |
| t.Fatalf("PlainDigest %q %d %d %v, want %q %d %d", plain, bytesN, count, err, p1, n, len(entries)) |
| @@ -1525,7 +1606,9 @@ func TestSealOpenExtractRoundTrip(t *testing.T) { |
| } |
| got, _ := os.ReadFile(filepath.Join(out, "sub", "b.bin")) |
| envGot, _ := os.ReadFile(outEnv) |
| - if string(got) != "\x01\x02\x03" || string(envGot) != "CARLOS_ADMIN_TOKEN=t\n" { |
| + appEnv, _ := os.ReadFile(filepath.Join(out, "env")) |
| + nested, _ := os.ReadFile(filepath.Join(out, "sub", "env", "x")) |
| + if string(got) != "\x01\x02\x03" || string(envGot) != "CARLOS_ADMIN_TOKEN=t\n" || string(appEnv) != "app-env" || string(nested) != "x" { |
| t.Fatal("extract content") |
| } |
| p2, _, _, _ := PlainDigest(out, outEnv) |
| @@ -1581,8 +1664,12 @@ const Cap int64 = 256 << 20 |
| // archived or digested (§5.1, §6.1 F5). |
| const MarkerName = ".files-gen" |
| -// EnvEntry is <host>.env's path inside the archive. |
| -const EnvEntry = "env" |
| +// EnvEntry is <host>.env's path inside the archive; FilesPrefix is the |
| +// files directory's namespace. The two never overlap. |
| +const ( |
| + EnvEntry = "env" |
| + FilesPrefix = "files/" |
| +) |
| var ( |
| ErrSpecialFile = errors.New("archive: only regular files and directories move") |
| @@ -1637,7 +1724,7 @@ func walk(filesDir, envPath string) ([]item, error) { |
| if !info.Mode().IsRegular() && !info.IsDir() { |
| return fmt.Errorf("%w: %s", ErrSpecialFile, rel) |
| } |
| - items = append(items, item{rel: rel, abs: p, info: info}) |
| + items = append(items, item{rel: FilesPrefix + rel, abs: p, info: info}) |
| return nil |
| }) |
| if err != nil { |
| @@ -1684,7 +1771,10 @@ func digestString(h interface{ Sum([]byte) []byte }) string { |
| // Build writes the archive (tar then zstd) to w and returns the plaintext |
| // digest, the entry list and the content byte count. Owners and times are |
| -// zeroed, modes kept, so equal content gives equal plaintext (§8.1). |
| +// zeroed, modes kept, so equal content gives equal plaintext (§8.1). The |
| +// files directory's entries live under "files/" and <host>.env is "env", in |
| +// separate namespaces, so an application file or directory named "env" is |
| +// carried as "files/env" and can never collide with the overlay. |
| func Build(filesDir, envPath string, w io.Writer) (plain string, entries []Entry, bytes int64, err error) { |
| items, err := walk(filesDir, envPath) |
| if err != nil { |
| @@ -1770,16 +1860,24 @@ func Extract(r io.Reader, filesDir, envPath string) error { |
| if name == "" || filepath.IsAbs(name) || name != filepath.ToSlash(filepath.Clean(name)) || strings.HasPrefix(name, "../") || name == ".." || strings.Contains(name, "/../") { |
| return fmt.Errorf("%w: %q", ErrBadPath, hdr.Name) |
| } |
| + var dst string |
| + switch { |
| + case name == EnvEntry: |
| + if hdr.Typeflag != tar.TypeReg || envPath == "" { |
| + return fmt.Errorf("%w: %q", ErrBadPath, hdr.Name) |
| + } |
| + dst = envPath |
| + case strings.HasPrefix(name, FilesPrefix) && len(name) > len(FilesPrefix): |
| + dst = filepath.Join(filesDir, filepath.FromSlash(strings.TrimPrefix(name, FilesPrefix))) |
| + default: |
| + return fmt.Errorf("%w: %q is outside files/ and env", ErrBadPath, hdr.Name) |
| + } |
| switch hdr.Typeflag { |
| case tar.TypeDir: |
| - if err := os.MkdirAll(filepath.Join(filesDir, filepath.FromSlash(name)), fs.FileMode(hdr.Mode).Perm()|0o700); err != nil { |
| + if err := os.MkdirAll(dst, fs.FileMode(hdr.Mode).Perm()|0o700); err != nil { |
| return err |
| } |
| case tar.TypeReg: |
| - dst := filepath.Join(filesDir, filepath.FromSlash(name)) |
| - if name == EnvEntry && envPath != "" { |
| - dst = envPath |
| - } |
| if err := os.MkdirAll(filepath.Dir(dst), 0o755); err != nil { |
| return err |
| } |
| @@ -2457,8 +2555,8 @@ Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| - Create: `internal/host/movesreport.go`, `internal/host/movesreport_test.go` |
| **Interfaces:** |
| -- Produces (pure): `moves.Check{OK bool; Reason string}`; `moves.TargetReport{V int; Box, Region string; MoveID string; At time.Time; Checks map[string]Check; Inputs provision.MoveInputs; AttemptsAdmitted int; CutBudgetS int}`; check ids `T1 … T12` as constants `CheckProtocol = "T1"` etc.; `moves.SourceReport{V int; Box string; MoveID string; At time.Time; Env EnvReport; S1, S2 Check; FilesPlain string}`; `moves.ReportFreshness = 10 * time.Minute`; `moves.Ready(roundDelta int64, bar int64, target *TargetReport, source *SourceReport, now time.Time) (ok bool, reason string)`; `(r TargetReport) Passing() (bool, string)`. |
| -- Produces (host): `(w *movesWorker) writeSourceReport(ctx, s, rec, env map[string]string, filesPlain string) error`; `(w *movesWorker) targetReportPass(ctx, s release.Store, recs []provision.Record) error` run by the target region's box for every record whose `Move.To` is its placement: computes T1 (own `moves_protocol` vs floor), T2 (own `secrets_key_fp` vs the console's published fingerprint in `control/edges/...`? The console publishes its fingerprint at `control/console-secrets-fp.json`; if no such object exists on main, add `moves.ConsoleFingerprintKey = "control/moves-console.json"` written by the console on startup in Task 12), T5 (the newest archive manifest under `moves/<host>/<id>/files/` opens with this box's key: list, read manifest, `archive.Open` to `io.Discard`, compare `plain`), T6 (`diskBytes(dataDir())`: free ≥ 2×replica + archive + 2 GiB and resulting use < 85%), T8 (one hosting box publishes the target placement), T9 (`hostclaim.Read`: `KindMoving`, id, holder incarnation = `move.json`'s; this region and box in `touched`), T10 (`PutIfAbsent`/`DeleteVersion` of `moves/_lane/<id>` in the source bucket through `lease.VersionedStore`), T11 (`moves.Estimate` with measured rates: `Rates` read from `CARLOS_MOVE_RATES="cross,s3,disk"` env, default `ExampleRates`, recorded in the report), T12 (`claim.QuietAfter[region]` passed), T3 (compose the flat held row's environment: `appconfig.LoadMergedHost` for the record's environment rendered to a temp config file, the carried `<host>.env` from the newest archive, the agent-owned variables, this process's inherited variables; `moves.CompareEnv`), T4 (for each store binding in the composed environment per `objstore.EnvNames`: list one key, put/get/delete `.carlos-move-probe/<move id>` through the S3 client of `internal/objstore`'s existing probe helper if one exists, else a minimal AWS SDK client), T7 (`release.MaterializeHeld`, Plan 2 Task 7: until it lands, T7 checks the pinned release's manifest exists in the store at `release.ManifestKey(acct, app, version)`); writes `TargetReportKey` with `Inputs` and `AttemptsAdmitted`. |
| +- Produces (pure): `moves.PutReplace(ctx, s release.Store, key string, body []byte) error`: `release.Store.Put` with an empty `ifMatch` is create-only (`store.go:38`), so a recurring object (a report, the console record) is written by `Get` for the current etag (absent → `""`), then `Put(key, body, etag)`, retrying once on `release.ErrCASConflict`; every mutable object the series writes to the deployment bucket goes through it (reports here, the console record in Task 12, the hold view's receipts ledger when it lives in a `release.Store`); `moves.Check{OK bool; Reason string}`; `moves.TargetReport{V int; Box, Region string; MoveID string; At time.Time; Checks map[string]Check; Inputs provision.MoveInputs; AttemptsAdmitted int; CutBudgetS int}`; check ids `T1 … T12` as constants `CheckProtocol = "T1"` etc.; `moves.SourceReport{V int; Box string; MoveID string; At time.Time; Env EnvReport; S1, S2 Check; FilesPlain string}`; `moves.ReportFreshness = 10 * time.Minute`; `moves.Ready(roundDelta int64, bar int64, target *TargetReport, source *SourceReport, now time.Time) (ok bool, reason string)`; `(r TargetReport) Passing() (bool, string)`. |
| +- Produces (host): `(w *movesWorker) writeSourceReport(ctx, s, rec, env map[string]string, filesPlain string) error`; `(w *movesWorker) targetReportPass(ctx, s release.Store, recs []provision.Record) error` run by the target region's box for every record whose `Move.To` is its placement: computes T1 (own `moves_protocol` vs floor), T2 (own `secrets_key_fp`, `appconfig.KeyFingerprint` of `CARLOS_SECRETS_KEY`'s public half, equals the console's `SecretsKeyFP` in `moves.ConsoleKey` = `control/moves-console.json`, which Task 12 has the console write at startup; no such object yet on main), T5 (the newest archive manifest under `moves/<host>/<id>/files/` opens with this box's key: list, read manifest, `archive.Open` to `io.Discard`, compare `plain`), T6 (`diskBytes(dataDir())`: free ≥ 2×replica + archive + 2 GiB and resulting use < 85%), T8 (one hosting box publishes the target placement), T9 (`hostclaim.Read`: `KindMoving`, id, holder incarnation = `move.json`'s; this region and box in `touched`), T10 (`PutIfAbsent`/`DeleteVersion` of `moves/_lane/<id>` in the source bucket through `lease.VersionedStore`), T11 (`moves.Estimate` with measured rates: `Rates` read from `CARLOS_MOVE_RATES="cross,s3,disk"` env, default `ExampleRates`, recorded in the report), T12 (`claim.QuietAfter[region]` passed), T3 (compose the flat held row's environment: `appconfig.LoadMergedHost` for the record's environment rendered to a temp config file, the carried `<host>.env` from the newest archive, the agent-owned variables, this process's inherited variables; `moves.CompareEnv`), T4 (for each store binding in the composed environment per `objstore.EnvNames`: list one key, put/get/delete `.carlos-move-probe/<move id>` through the S3 client of `internal/objstore`'s existing probe helper if one exists, else a minimal AWS SDK client), T7 (`release.MaterializeHeld`, Plan 2 Task 7: until it lands, T7 checks the pinned release's manifest exists in the store at `release.ManifestKey(acct, app, version)`); writes `TargetReportKey` with `Inputs` and `AttemptsAdmitted`. |
| Test (pure) `internal/moves/readiness_test.go`: |
| @@ -2618,14 +2716,14 @@ func Ready(roundDelta, bar int64, target *TargetReport, source *SourceReport, no |
| } |
| ``` |
| -Host side (`movesreport.go`): the two passes with the checks above, each check in its own small function returning `moves.Check`, the report written with `s.Put(ctx, moves.TargetReportKey(prefix), body, "")`. The test `TestTargetReportPassFlagsEachCheck` in `movesreport_test.go` builds the harness of Task 14 (`newMoveHarness`) with a target-side configuration (`cfg.Region = "uk"`, `cfg.Box = "uk-box"`), stages a `move.json`, a serving-then-moving claim with `touched` lacking `uk`, no archive, and asserts the written report has `T9` failing with "touched", `T5` failing with "no archive", `T1` passing, and that after `hostclaim.Touch` and an archive are added the two pass on the next pass. T3, T4, T6, T7, T10 take injectable hooks on `moveCfg` (`ComposeHeldEnv`, `ProbeStore`, `DiskFree`, `ReleaseReady`, `LaneProbe`) that the test stubs to pass. |
| +Host side (`movesreport.go`): the two passes with the checks above, each check in its own small function returning `moves.Check`, the report written with `moves.PutReplace(ctx, s, moves.TargetReportKey(prefix), body)` (a report refreshes every pass; `Put(..., "")` would refuse the second write). Each stubbed hook (`ComposeHeldEnv`, `ProbeStore`, `DiskFree`, `ReleaseReady`, `LaneProbe`) gets one failing case in `TestTargetReportPassFlagsEachCheck`: the hook returns an error and that check alone fails with the hook's reason. The test `TestTargetReportPassFlagsEachCheck` in `movesreport_test.go` builds the harness of Task 14 (`newMoveHarness`) with a target-side configuration (`cfg.Region = "uk"`, `cfg.Box = "uk-box"`), stages a `move.json`, a serving-then-moving claim with `touched` lacking `uk`, no archive, and asserts the written report has `T9` failing with "touched", `T5` failing with "no archive", `T1` passing, and that after `hostclaim.Touch` and an archive are added the two pass on the next pass. T3, T4, T6, T7, T10 take injectable hooks on `moveCfg` (`ComposeHeldEnv`, `ProbeStore`, `DiskFree`, `ReleaseReady`, `LaneProbe`) that the test stubs to pass. |
| - [ ] **Step 2: Run tests to verify they fail**, **Step 3: implement**, **Step 4: run** as above. |
| Run: `nice -n 10 go test ./internal/moves ./internal/host -run 'TestReadyNeedsFreshReports|TestTargetReportPass' -count=1 -p 1` |
| Expected: PASS after implementation. |
| -T4 against a real store (§14.1): add `TestT4StorageProbeMinio` in `internal/host/movesreport_minio_test.go`, gated the way `TestDeliveredContractMinio` is (skips without the MinIO endpoint env the `30-minio-contract` CI step sets), creating a bucket with the test's credentials and asserting the probe lists, puts, gets and deletes `.carlos-move-probe/<move id>` and reports `T4` failing with the store's error when the credentials are wrong. |
| +T4 against a real store (§14.1): add `TestT4StorageProbeMinio` in `internal/host/movesreport_minio_test.go`, gated on `CARLOS_MINIO_TEST=1` and the `MINIO_*` variables the way `TestDeliveredContractMinio` is, creating a bucket with the test's credentials and asserting the probe lists, puts, gets and deletes `.carlos-move-probe/<move id>` and reports `T4` failing with the store's error when the credentials are wrong. `.amadan/ci.d/30-minio-contract` runs only `go test ./internal/objstore/ -run TestDeliveredContractMinio` (line 330); add a second invocation `go test ./internal/host/ -run TestT4StorageProbeMinio -count=1 -v` under the same environment and assert its `PASS` line the way the step asserts the first, so the test runs in CI rather than only skipping. |
| - [ ] **Step 5: Commit** |
| @@ -2647,7 +2745,7 @@ Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| **Interfaces:** |
| - Produces: `moves.ProtocolVersion = 1`; `moves.FloorKey = "control/moves-protocol.json"`; `moves.Floor{Min int; MinRelease string}`; `moves.ReadFloor(ctx, s release.Store) (*Floor, error)` (absent → nil, nil); `moves.ReleaseAtOrAbove(version, minRelease string) (bool, error)` using `release.CompareVersions` if it exists (grep `func Compare` in `internal/release`), else `golang.org/x/mod/semver` on `v`-prefixed strings; `moves.ConsoleKey = "control/moves-console.json"`, `moves.ConsoleRecord{V int; Protocol int; SecretsKeyFP string; At time.Time}`. |
| - `edgesFile.MovesProtocol int \`json:"moves_protocol"\`` (not omitempty: absent means a box below any floor). |
| -- `systemApplier` refuses to activate a build below `Floor.MinRelease` in `install`, `rollback` and `recoverIfNeeded`'s restore (`restoreFiles`), reading the floor through the store `openSystemStore` returns; a store read error refuses nothing (the floor protects moves, not boots) and logs. |
| +- `systemApplier` refuses to activate a build below `Floor.MinRelease` in `install`, `rollback` and `recoverIfNeeded`'s restore (`restoreFiles`). The floor is **armed locally**: every successful read of `control/moves-protocol.json` (the agent's edges-file loop once an hour, and `carlos system apply` before it stages) writes `<optDir>/moves-floor.json` durably (temp file, fsync, rename); activation reads the local copy first and the store second. Boot recovery runs before networking (`systemapply.go:109`), so the local copy is what it has. With a floor in hand, below it refuses. With **no** floor anywhere: when the box's registry (`/data/registry.db`, the agent's path) holds any row with `hold_op != ""`, the box is a participant and activation refuses ("the moves floor is unknown and this box holds an operation"); otherwise it proceeds with a log line (a box holding no operation cannot harm a move). The floor is never deleted locally once armed. |
| - [ ] **Step 1: Write the failing tests** |
| @@ -2673,7 +2771,7 @@ func TestFloorReadAndCompare(t *testing.T) { |
| } |
| ``` |
| -In `internal/host/systemapply_test.go`, beside the existing applier tests: `TestSystemApplyRefusesBelowTheMovesFloor` writes `control/moves-protocol.json` with `min_release: "v0.30.0"` into the test's store and asserts an apply of a pointer at `v0.29.0` records the refusal reason containing "moves protocol floor" and leaves the binary untouched; `TestEdgesFilePublishesMovesProtocol` asserts the written file carries `"moves_protocol":1`. |
| +In `internal/host/systemapply_test.go`, beside the existing applier tests: `TestSystemApplyRefusesBelowTheMovesFloor` writes `control/moves-protocol.json` with `min_release: "v0.30.0"` into the test's store and asserts an apply of a pointer at `v0.29.0` records the refusal reason containing "moves protocol floor" and leaves the binary untouched, and that `<optDir>/moves-floor.json` now exists; `TestBootRecoveryUsesTheArmedFloorOffline` (the store returns errors for every read, the local floor file names `v0.30.0`, `carlos.prev` is `v0.29.0` → `restoreFiles` refuses); `TestUnknownFloorRefusesOnlyAParticipant` (no floor anywhere: a registry with a held row refuses, an empty one proceeds); `TestEdgesFilePublishesMovesProtocol` asserts the written file carries `"moves_protocol":1`. The door (Task 4) reads `readEdgesSnapshot(ctx, s)` and refuses with 409 "the fleet's edges files could not all be read. try again" unless `snapshot.complete()` (`instancesplacement.go:236`); `readEdgesFiles` discards completeness and is not used for the floor decision. |
| - [ ] **Step 2: Run tests to verify they fail**, **Step 3: implement**: |
| @@ -2719,7 +2817,7 @@ func ReleaseAtOrAbove(version, minRelease string) (bool, error) { |
| } |
| ``` |
| -(`golang.org/x/mod/semver`; confirm `x/mod` is in go.mod, else use `release`'s own comparison.) Edges file: add the field and `MovesProtocol: moves.ProtocolVersion` in the writer at `edgesfile.go:722-736`; `internal/console/instancesplacement.go`'s edges struct gains the same field. Console startup (`server.go`, where the edges-dependent config is logged): `s.cfg.Releases.Put(ctx, moves.ConsoleKey, json(ConsoleRecord{V:1, Protocol: moves.ProtocolVersion, SecretsKeyFP: fp, At: now}), "")` with `fp, _ := appconfig.PublicFingerprint(s.cfg.SecretsPub)` (empty when unset), best-effort, logged on failure. `systemapply.go`: a method `(a *systemApplier) floorAllows(ctx, version string) error` reading `moves.ReadFloor` from `a.store` (the store `openSystemStore` returned, stored on the applier; add the field if absent) and returning `fmt.Errorf("moves protocol floor: %s is below %s", version, f.MinRelease)` when below; called at the top of `install` for `req.Version`, in `rollback` and `restoreFiles` for `a.prevBuild(ctx)`; a refusal in `restoreFiles` leaves the journal and returns the error (the box stays on the failed build, paged by the existing result path), per §11. |
| +(`golang.org/x/mod/semver`; confirm `x/mod` is in go.mod, else use `release`'s own comparison.) Edges file: add the field and `MovesProtocol: moves.ProtocolVersion` in the writer at `edgesfile.go:722-736`; `internal/console/instancesplacement.go`'s edges struct gains the same field. Console startup (`server.go`, where the edges-dependent config is logged): `moves.PutReplace(ctx, s.cfg.Releases, moves.ConsoleKey, json(ConsoleRecord{V:1, Protocol: moves.ProtocolVersion, SecretsKeyFP: fp, At: now}))` with `fp, _ := appconfig.PublicFingerprint(s.cfg.SecretsPub)` (empty when unset), best-effort, logged on failure; a second console start replaces the record (a create-only `Put` would fail every restart after the first). `systemapply.go`: a method `(a *systemApplier) floorAllows(ctx, version string) error` reading `moves.ReadFloor` from `a.store` (the store `openSystemStore` returned, stored on the applier; add the field if absent) and returning `fmt.Errorf("moves protocol floor: %s is below %s", version, f.MinRelease)` when below; called at the top of `install` for `req.Version`, in `rollback` and `restoreFiles` for `a.prevBuild(ctx)`; a refusal in `restoreFiles` leaves the journal and returns the error (the box stays on the failed build, paged by the existing result path), per §11. |
| - [ ] **Step 4: Run**: `nice -n 10 go test ./internal/moves ./internal/host ./internal/console -count=1 -p 1` → PASS. |
| @@ -2845,10 +2943,10 @@ Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| **Interfaces:** |
| - `moveCfg` gains: `Box string` (the box label `metricsBoxLabel` gives), `DeployBucket, DeployBucketRegion string` (from `CARLOS_DEPLOYMENT_BUCKET` and `CARLOS_AWS_REGION`/the IMDS region the edges loop resolves; empty in dir-store deployments, where the lease falls back to a `lease.Acquire` on the own bucket under the same prefix), `Versioned func(ctx, bucket, region string) (lease.VersionedStore, error)`, `StagingLease func(ctx, targetRegion, host string) (moves.SafeUntilGuard, func(context.Context) error, error)`, `SecretsKey string` (`CARLOS_SECRETS_KEY`), `SecretsPub *ecdh.PublicKey`, `LaunchEnv func(host string) (map[string]string, error)`, `Timers func(host string) ([]registry.Timer, error)`, `FilesDir func(host string) (dir, envPath string)` (flat: `<DataDir>/<host>/`, `<DataDir>/<host>.env`), `ReadyProbe func(ctx, host string) moves.Check` (S1: `GET <ready path>` on the instance's socket through the activator's own request path; the activator's `ServeHTTP` is the edge's, so the probe dials the socket directly with `Host: <host>` and a 30 s timeout), `StrictNext func(host string) moves.Check` (S2: the activator reports whether the next spawn is strict; until Plan 2's strict-spawn task lands this returns OK when `cgroup` reports delegation). |
| -- The round, per in-flight record this box owns, in order (§7.3, K5, K12): refuse a legacy record (log once a minute, `return nil`); refuse a nested row (`rt.DBPath != provision.DefaultDBPath(dataDir, host)`: "nested instances cannot move yet"); refuse a source with `<DataDir>/<host>-sidecar.env` present or any refused registry column set on its row (`Caps`, `Unit`, `Backing`, `Platform`, `Tunnel`, `FleetBox`, `Pool`, `Canary`, `Addr`, `Addrs`, `Health`, `Replica`, `Replicas`; §6.2, §7.1), writing the reason into the source report's `S1` so the operator sees it in `carlos move status`; read the claim; `hostclaim.RequireMoveOp(id, KindMoving, "")` and holder incarnation = `rec.Move.Incarnation`; if `rec.Move.To` not in `Touched.Regions`, `hostclaim.Update(... Touch(id, []string{to}, nil))`; take the staging lease; `PutIfAbsent(move.json)` through `Versioned(target bucket)` when the receipts carry no `move_json` version yet (a 412 with a body that decodes to the same `MoveJSON` is fine; a different one refuses the round, Review Focus 2); `MirrorPrefix(own replica prefix → ReplicaPrefix)` under `Guarded.Before` per copy (wrap the mirror in a `guardedMirror` whose `CopyVersioned`/`Put` call `Before`); compute `archive.PlainDigest`; when it differs from the receipts' recorded plain, `archive.Build` + `archive.Seal` to a temp file, `PutVersioned` to `FilesPrefix+"files-<sealed>.sealed"` and `.json` (manifest), record in receipts; write the source report with `LaunchEnv` → `moves.ReportEnv`, S1 and S2; read the target report; `moves.Ready(round.DeltaBytes(), cfg.ReadyDelta, target, source, now)`; on ready from preparing, `MovePromoteReady` and `nudgeSerial`; in `ready`, keep mirroring. For `cut == "auto"` and `ready`, the source itself CASes `MoveMarkCutting` (§7.4 "cut door or the source (auto-cut)"); Plan 2 picks it up. |
| +- The round, per in-flight record this box owns, in order (§7.3, K5, K12): refuse a legacy record (log once a minute, `return nil`); refuse a nested row (`rt.DBPath != provision.DefaultDBPath(dataDir, host)`: "nested instances cannot move yet"); refuse a source with `<DataDir>/<host>-sidecar.env` present or any refused registry column set on its row (`Caps`, `Unit`, `Backing`, `Platform`, `Tunnel`, `FleetBox`, `Pool`, `Canary`, `Addr`, `Addrs`, `Health`, `Replica`, `Replicas`; §6.2, §7.1), writing the reason into the source report's `S1` so the operator sees it in `carlos move status`; read the claim; `hostclaim.RequireMoveOp(id, KindMoving, "")` and holder incarnation = `rec.Move.Incarnation`; if the target region or the target box (`Op.Intent["target"]`) is not in `Touched`, `hostclaim.Update(... Touch(id, []string{to}, []string{targetBox}))` (K12 and T9 both hold from the first staging write; the target's `checking` re-adds its box idempotently); take the staging lease (`lease.AcquireTTL(..., moves.StagingLeaseTTL)`); `move.json`: the ledger (own bucket) first records `move_json: "pending"`, then `PutIfAbsent(move.json)` through `Versioned(target bucket)`; success records the returned version; `lease.ErrExists` with a `pending` entry is this op's own earlier PUT whose ledger write was lost (the lane has one writer, this op's source under the staging lease, and the id is 128 random bits), recorded as `move_json: "present"`; `ErrExists` with **no** ledger entry refuses the round ("a staging lane for this move id already exists"; Review Focus 2). The source never reads the target object (M17); the target validates its incarnation itself, in T9 and at `checking`; `MirrorPrefix(own replica prefix → ReplicaPrefix)` through the staging mirror (`lease.NewStagingMirror`) wrapped in a `guardedMirror` whose `CopyVersioned`, `PutVersioned` and `Delete` each call `Guarded.Before` around exactly one SDK call, and whose caller retries a failed call only by calling again (a new check, a new signature); compute `archive.PlainDigest`; when it differs from the receipts' recorded plain, `archive.Build` + `archive.Seal` to a temp file, `PutVersioned` to `FilesPrefix+"files-<sealed>.sealed"` and `.json` (manifest), record in receipts; write the source report with `LaunchEnv` → `moves.ReportEnv`, S1 and S2; read the target report; `moves.Ready(round.DeltaBytes(), cfg.ReadyDelta, target, source, now)`; on ready from preparing, `MovePromoteReady` and `nudgeSerial`; in `ready`, keep mirroring. For `cut == "auto"` and `ready`, the source itself CASes `MoveMarkCutting` (§7.4 "cut door or the source (auto-cut)"); Plan 2 picks it up. |
| - Receipts: `lease.Receipts{Bucket: cfg.OwnBucket, Key: moves.ReceiptsKey(host, id)}`; a per-op `moveLedger` JSON beside it at `move-receipts/<host>/<id>-moves.json` records `move_json_version`, `archive_plain`, `archive_sealed`, `archive_versions`. |
| -Tests: rewrite `TestMovesPassPrepareAndPromote` for the new layout (keys under `moves/<host>/<id>/replica/`; `move.json` present; the round is not promoted until a passing fresh `target.json` exists in the harness store); add `TestStagingMoveJSONIsCreateOnly` (Review Focus 2: a `move.json` for another incarnation already there → the round returns an error naming it and copies nothing), `TestPreparingRefusesLegacyAndNested`, `TestPreparingTouchesTheTargetRegionFirst` (the claim's `touched.regions` gains `uk` before the fake mirror sees a copy), `TestSourceReportIsWrittenEachRound`, `TestArchiveIsRebuiltOnlyWhenPlainChanges`, `TestAutoCutMarksCuttingFromReady`. |
| +Tests: rewrite `TestMovesPassPrepareAndPromote` for the new layout (keys under `moves/<host>/<id>/replica/`; `move.json` present; the round is not promoted until a passing fresh `target.json` exists in the harness store); add `TestStagingMoveJSONIsCreateOnly` (Review Focus 2: a `move.json` already there and no ledger entry → the round returns an error naming it and copies nothing), `TestStagingMoveJSONReplayAfterLostLedger` (Review Focus 2: the ledger says `pending`, the fake versioned store answers `ErrExists` and refuses every `GetVersion`/`OpenVersion` of the target lane with an access-denied error; the round proceeds and records `present`), `TestPreparingRefusesLegacyNestedAndRefusedColumns`, `TestPreparingTouchesTheTargetRegionAndBoxFirst` (the claim's `touched.regions` gains `uk` and `touched.boxes` gains `uk-box` before the fake mirror sees a copy, from a claim that never touched either), `TestStagingWriterRetriesWithANewCheck` (the fake mirror fails one copy; the lane calls `Before` again before the retry; a `Now` past the bound makes the retry `ErrLeaseUnsafe` and the round ends without a second SDK call), `TestSourceReportIsWrittenEachRound` (two rounds, the second's report replaces the first: `PutReplace`), `TestArchiveIsRebuiltOnlyWhenPlainChanges`, `TestAutoCutMarksCuttingFromReady`. |
| - [ ] **Step 1: Write the failing tests**, **Step 2: run**, **Step 3: implement**, **Step 4: run** `nice -n 10 go test ./internal/host -run 'TestMovesPass|TestStaging|TestPreparing|TestSourceReport|TestArchiveIsRebuilt|TestAutoCut' -count=1`. |
| @@ -2863,11 +2961,26 @@ Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>" |
| --- |
| -### Task 15: Gate and hand-off |
| +### Task 15: The copy gate |
| + |
| +**Files:** modify `approved_copy_test.go`; the record `docs/superpowers/plans/2026-10-06-moves-phase1-copy.md` (already on the branch; this task makes the test read it). |
| + |
| +The gate exists from this plan on, so every later plan's strings are checked the day they are written (index D12). Two checks, modelled on the existing `TestApprovedCopyIsApplied` (`approved_copy_test.go:87`), which compares the record against a **separately pinned inventory** so a deleted row is caught: |
| + |
| +- `movesCopyIDs` is a literal in the test file: `M-C1` to `M-C20` and every `MV-N<n>` id the plans introduce (each plan's "Gate and copy" task appends to this literal as it appends rows to the record). `TestMovesCopyRecordIsComplete` fails when the record lacks an id the literal names, or names an id the literal does not. |
| +- `TestMovesApprovedCopyIsApplied`: every `approved` row's text appears byte for byte in the file the row names (as the existing test checks, through string literals for short texts). |
| +- `TestPendingCopyIsNotWired`: for every `pending` row, its text (and the text with its `<placeholder>` segments replaced by `%s`/`%d`/`%v` verbs, so a `fmt.Sprintf` form is caught too) does not appear as a string literal in any non-test Go file in a call to `jsonError`, `fmt.Fprint*(os.Stdout|os.Stderr, …)`, `fmt.Print*`, `emitEvent`, `PageOperator`, `log.Fatal*`, or as the argument of a helper whose name ends in `Sentence`, `Line` or `Message`; the check walks the file's AST with `go/ast`, finds call expressions whose callee matches, and inspects every string literal argument and every identifier bound to a string literal in the same file. |
| +- Negative fixtures under `testdata/moves-copy/`: a record missing an id, a row whose text changed after approval, a pending text reached through a `Sprintf` helper; each is loaded by a table test that expects the corresponding failure. |
| + |
| +- [ ] Write the tests, run them red (the record has only pending rows; the three negative fixtures fail as expected), implement, run green, commit: `git commit -m "copy gate for the moves series: the pinned inventory, the ship check (§20)" ...` |
| + |
| +--- |
| + |
| +### Task 16: Gate and hand-off |
| - [ ] Run the full gate serial and niced: `nice -n 10 go build ./... && nice -n 10 go vet ./... && nice -n 10 go test ./... -count=1 -p 1 && gofmt -l . && nice -n 10 make test`; then `GOOS=darwin go build ./... && GOOS=windows go build ./... && GOOS=linux GOARCH=arm64 go build ./...`. |
| - [ ] Confirm `go build .` on darwin links `internal/moves/archive` (no Linux-only syscalls were added). |
| -- [ ] List every new user-facing string of this plan in `docs/superpowers/plans/2026-10-06-moves-phase1-copy.md` under "Plan 1" with its spec id or "new" (the copy gate of Plan 6 Task 7 reads this file). |
| +- [ ] Every new user-facing string of this plan is in `docs/superpowers/plans/2026-10-06-moves-phase1-copy.md` under "Plan 1" and in `movesCopyIDs`. |
| - [ ] Commit: `git commit -m "moves plan 1: gate clean" ...` only if the gate changed anything. |
| ## Self-review notes |
diff --git a/docs/superpowers/plans/2026-10-06-moves-phase1-2-cut-flip-landing.md b/docs/superpowers/plans/2026-10-06-moves-phase1-2-cut-flip-landing.md| index 4bfe0dc82..82b36f572 100644 |
| --- a/docs/superpowers/plans/2026-10-06-moves-phase1-2-cut-flip-landing.md |
| +++ b/docs/superpowers/plans/2026-10-06-moves-phase1-2-cut-flip-landing.md |
| @@ -312,7 +312,7 @@ func MoveRevert(g MoveGuard, reason string) func(*Record) error { |
| - Modify: `internal/hostclaim/move.go`, `internal/hostclaim/move_test.go` |
| **Interfaces:** |
| -- Produces: journal keys `JournalSeal = "seal"`, `JournalSrcGen = "src_gen"`, `JournalSourceStamp = "source_stamp"`, `JournalCutSession = "cut_session"`, `JournalRevPhase = "rev_phase"`, `JournalEvidenceSource = "evidence_source"`, `JournalEvidenceTarget = "evidence_target"`; `SetMovePhase(opID, kind, from, to string) func(*Claim) error`; `SetJournal(opID string, kv map[string]string) func(*Claim) error` (op id only; any kind); `ExecToNone(opID, box, evidenceKey, journalKey string) func(*Claim) error` (K3: `c.Op.Exec == box` → `""`, records the evidence key under `journalKey`); `Flip(opID, targetBox string, sourceStamp uint64, srcGen string) func(*Claim) error` (K4: requires `KindMoving`, phase `sealed`, `Exec == ""`, `evidence_source` journalled; sets `Kind = KindLanding`, `Phase = "checking"`, `Exec = targetBox`, `holder.stamp = the resulting gen`? No: the stamp written by the flip is `c.Gen + 1`? `hostclaim.Update` increments `Gen` after the transition runs; look at `store.go` to see whether the transition can know the resulting gen. If `Update` sets `c.Gen` before calling `tr`, use `c.Gen`; otherwise `Update` must expose it. Read `store.go` and follow `create.go`'s serving CAS, which writes `holder.stamp` the same way ("a create's serving CAS writes the gen it produces"). Journals `source_stamp` and `src_gen`); `Revert(opID, sourceBox string) func(*Claim) error` (from `KindMoving`, `Exec == ""`, `evidence_source` journalled → `KindReverting`, `Exec = sourceBox`, `rev_phase = "evidence"` unless set). |
| +- Produces: journal keys `JournalSeal = "seal"`, `JournalSrcGen = "src_gen"`, `JournalSourceStamp = "source_stamp"`, `JournalCutSession = "cut_session"`, `JournalRevPhase = "rev_phase"`, `JournalEvidenceSource = "evidence_source"`, `JournalEvidenceTarget = "evidence_target"`; `SetMovePhase(opID, kind, from, to string) func(*Claim) error`; `SetJournal(opID string, kv map[string]string) func(*Claim) error` (op id only; any kind); `ExecToNone(opID, box, evidenceKey, journalKey string) func(*Claim) error` (K3: `c.Op.Exec == box` → `""`, records the evidence key under `journalKey`); `Flip(opID, targetBox string, sourceStamp uint64, srcGen string) func(*Claim) error` (K4: requires `KindMoving`, phase `sealed`, `Exec == ""`, `evidence_source` journalled; sets `Kind = KindLanding`, `Phase = "checking"`, `Exec = targetBox`, `holder.stamp = c.Gen + 1`, the gen this CAS produces: `hostclaim.Update` sets `gen = observed + 1` after `tr` runs (`store.go`, and `create.go:94` writes the serving stamp the same way); journals `source_stamp` and `src_gen`); `Revert(opID, sourceBox, targetRegion string, stagingScope *ParkScope, newID func() string) func(*Claim) error` (from `KindMoving`, `Exec == ""`, `evidence_source` journalled → `KindReverting`, `Exec = sourceBox`, `rev_phase = "evidence"` unless set, plus the `park-staging` obligation for the target region's lane with the writers' bound as its scope, which §7.11's "opened waits for the staging obligation" refers to; Plan 4 Task 6 waits on it and Plan 5 Task 1 dispatches it). |
| - [ ] **Step 1: Write the failing test** |
| @@ -366,12 +366,18 @@ func TestRevertNeedsEvidenceAndReturnsExecToSource(t *testing.T) { |
| t.Fatalf("revert while exec held: %v", err) |
| } |
| _ = ExecToNone(op, "ie-box", "moves/bab/app/h/mv/stopped-source.json", JournalEvidenceSource)(c) |
| - if err := Revert(op, "uk-box")(c); !errors.Is(err, ErrWrongState) { |
| + scope := ParkScope{Location: "moves/a.go.example.com/" + op, Incarnation: c.Holder.Incarnation, EvidenceAt: time.Now(), RequestLimit: 120 * time.Second} |
| + ids := 0 |
| + newID := func() string { ids++; return fmt.Sprintf("id%d", ids) } |
| + if err := Revert(op, "uk-box", "uk", &scope, newID)(c); !errors.Is(err, ErrWrongState) { |
| t.Fatalf("revert to the target: %v", err) |
| } |
| - if err := Revert(op, "ie-box")(c); err != nil || c.Op.Kind != KindReverting || c.Op.Exec != "ie-box" || c.Op.Journal[JournalRevPhase] != "evidence" { |
| + if err := Revert(op, "ie-box", "uk", &scope, newID)(c); err != nil || c.Op.Kind != KindReverting || c.Op.Exec != "ie-box" || c.Op.Journal[JournalRevPhase] != "evidence" { |
| t.Fatalf("revert: %v %+v", err, c.Op) |
| } |
| + if len(c.Obligations) != 1 || c.Obligations[0].Kind != "park-staging" || c.Obligations[0].Region != "uk" || c.Obligations[0].Park == nil { |
| + t.Fatalf("revert queued no target staging park: %+v", c.Obligations) |
| + } |
| } |
| ``` |
| @@ -464,8 +470,10 @@ func Flip(opID, targetBox string, sourceStamp uint64, srcGen string) func(*Claim |
| } |
| // Revert is §7.4 reverting: the source takes execution back with its own |
| -// evidence, before the flip. |
| -func Revert(opID, sourceBox string) func(*Claim) error { |
| +// evidence, before the flip. The target region's staging park is queued in |
| +// the same write (§7.11: the revert's `opened` waits for it), owned by the |
| +// target region's box, scoped to the lane with the writers' horizon. |
| +func Revert(opID, sourceBox, targetRegion string, stagingScope *ParkScope, newID func() string) func(*Claim) error { |
| return func(c *Claim) error { |
| if err := RequireOp(opID)(c); err != nil { |
| return err |
| @@ -476,10 +484,23 @@ func Revert(opID, sourceBox string) func(*Claim) error { |
| if len(c.Op.Participants) == 0 || c.Op.Participants[0] != sourceBox { |
| return fmt.Errorf("%w: %q is not the move's source", ErrWrongState, sourceBox) |
| } |
| + if stagingScope == nil || targetRegion == "" { |
| + return fmt.Errorf("%w: a revert names the staging lane it leaves behind", ErrMalformed) |
| + } |
| + if err := stagingScope.Validate(c.Host); err != nil { |
| + return err |
| + } |
| c.Op.Kind, c.Op.Exec = KindReverting, sourceBox |
| if c.Op.Journal[JournalRevPhase] == "" { |
| setJournal(c, JournalRevPhase, "evidence") |
| } |
| + for _, o := range c.Obligations { |
| + if o.Op == opID && o.Kind == "park-staging" && o.Region == targetRegion { |
| + return nil |
| + } |
| + } |
| + s := *stagingScope |
| + c.Obligations = append(c.Obligations, Obligation{ID: newID(), Op: opID, Kind: "park-staging", Region: targetRegion, Retirement: newID(), Owner: "region:" + targetRegion, Park: &s}) |
| return nil |
| } |
| } |
| @@ -495,12 +516,12 @@ func Revert(opID, sourceBox string) func(*Claim) error { |
| ### Task 3: `ReconstructStopEvidence` |
| **Files:** |
| -- Create: `internal/activator/stopreconstruct_linux.go`, `internal/activator/stopreconstruct_other.go`, `internal/activator/stopreconstruct_linux_test.go` |
| +- Create: `internal/stopevidence/reconstruct_linux.go`, `internal/stopevidence/reconstruct_other.go`, `internal/stopevidence/reconstruct_linux_test.go` |
| - Create: `internal/moves/evidence.go`, `internal/lease/lockstate.go`, `internal/lease/lockstate_test.go` |
| **Interfaces:** |
| -- Consumes: `cgroup.Manager.OpenCleanup(host) (*cgroup.Cleanup, error)`, `(*Cleanup).KillAndEmpty(ctx)`, `Populated()`, `Members()`, `Close()`; `stopevidence.Collect(ctx, stopevidence.Inputs{...})`, `stopevidence.FDHolders(procRoot, paths)`; `cgroup.ParseSelfCgroup`; `activator.StopEvidence`. |
| -- Produces: `moves.StoppedReport{V int; Role, Box, Incarnation string; ClaimGen uint64; Host, OpID string; Steps map[string]moves.Check; Stray []int; FinalTXID uint64; DBSHA256, FilesPlain, TimersSHA256 string; CheckedAt time.Time}` with step ids `StepTree = "tree"`, `StepStray = "stray"`, `StepFDs = "fds"`, `StepLease = "lease"`, `StepReplica = "replica"`; `(r StoppedReport) Complete() bool` (every step OK); `lease.LockState(ctx, bucket, prefix, region string) (held bool, owner string, expires time.Time, err error)` (reads `lock.json` as the leaser writes it; absent → `held=false`); `(a *Service) ReconstructStopEvidence(ctx, host string, dbPaths []string, leaseFree func(ctx) (bool, error)) (moves.StoppedReport, error)` (steps 1 to 4 of §5.3; step 5, the final checkpoint and digests, is the caller's); on `!linux`, returns `ErrStopEvidenceUnsupported`. |
| +- Consumes: `cgroup.Manager.OpenCleanup(host) (*cgroup.Cleanup, error)`, `(*Cleanup).KillAndEmpty(ctx)`, `Populated()`, `Members()`, `Close()`; `stopevidence.Collect`, `stopevidence.FDHolders(procRoot, paths)`, `stopevidence.Cleanup`; `cgroup.ParseSelfCgroup`. The function lives in `internal/stopevidence`, not `internal/activator`: `stopevidence` already imports `activator` (`evidence.go:10`, for `*activator.StopEvidence`), so the activator cannot import it back; `stopevidence` may import `moves` (which imports neither). The cgroup manager is the exec backend's `ExecBackend.Cgroups` (`backend_exec.go`), handed in explicitly by the host package at arm time (`eb.Cgroups`); `activator.Service` has no manager field. |
| +- Produces: `moves.StoppedReport{V int; Role, Box, Incarnation string; ClaimGen uint64; Host, OpID string; Steps map[string]moves.Check; Stray []int; FinalTXID uint64; DBSHA256, FilesPlain, TimersSHA256 string; CheckedAt time.Time}` with step ids `StepTree = "tree"`, `StepStray = "stray"`, `StepFDs = "fds"`, `StepLease = "lease"`, `StepReplica = "replica"`; `(r StoppedReport) Complete() bool` (every step OK, `StepReplica` included); `(r StoppedReport) TreeStopped() bool` (steps 1 to 4 OK: what `stopping` needs before the checkpoint); `lease.LockState(ctx, bucket, prefix, region string) (held bool, owner string, expires time.Time, err error)` (reads `lock.json` as the leaser writes it; absent → `held=false`); `stopevidence.ReconstructInputs{Host string; Cgroups *cgroup.Manager; DBPaths []string; ProcRoot string; LeaseFree func(context.Context) (bool, error)}`; `stopevidence.Reconstruct(ctx, in ReconstructInputs) (moves.StoppedReport, error)` (steps 1 to 4 of §5.3; step 5, the final checkpoint and digests, is the caller's, which then sets `StepReplica`); on `!linux`, returns `ErrUnsupported`. |
| - [ ] **Step 1: Write the failing tests** |
| @@ -533,14 +554,12 @@ func TestReconstructStopEvidenceKillsTheWholeTree(t *testing.T) { |
| t.Fatal(err) |
| } |
| cmd.Wait() |
| - svc := New(Config{}, nil, nil) |
| - svc.cgroups = m // the field the service keeps its manager in; name per activator.go |
| - rep, err := svc.ReconstructStopEvidence(context.Background(), host, nil, func(context.Context) (bool, error) { return true, nil }) |
| + rep, err := Reconstruct(context.Background(), ReconstructInputs{Host: host, Cgroups: m, ProcRoot: "/proc", LeaseFree: func(context.Context) (bool, error) { return true, nil }}) |
| if err != nil { |
| t.Fatal(err) |
| } |
| - if !rep.Steps[moves.StepTree].OK || !rep.Complete() { |
| - t.Fatalf("evidence incomplete after the parent exited: %+v", rep) |
| + if !rep.Steps[moves.StepTree].OK || !rep.TreeStopped() || rep.Complete() { |
| + t.Fatalf("tree evidence after the parent exited: %+v (Complete needs the caller's replica step)", rep) |
| } |
| c, _ := m.OpenCleanup(host) |
| pop, _ := c.Populated() |
| @@ -560,25 +579,28 @@ func TestReconstructStopEvidenceRefusesAStrayProcess(t *testing.T) { |
| - [ ] **Step 2: Run to verify they fail**, **Step 3: implement** |
| -`moves/evidence.go` declares the report and steps (as above, with `Complete()`). `lease/lockstate.go` reads `prefix + "/lock.json"` through the same S3 client constructor `acquireTTL` uses and decodes the leaser's lock body (owner, expires); if the leaser library (`lss3.Leaser`) exposes a read, use it. `stopreconstruct_linux.go`: |
| +`moves/evidence.go` declares the report and steps (as above, with `Complete()` and `TreeStopped()`). `lease/lockstate.go` reads `prefix + "/lock.json"` through the same S3 client constructor `acquireTTL` uses and decodes the leaser's lock body (owner, expires); if the leaser library (`lss3.Leaser`) exposes a read, use it. `reconstruct_linux.go`: |
| ```go |
| //go:build linux |
| -package activator |
| +package stopevidence |
| -// ReconstructStopEvidence is §5.3's evidence from durable, observable |
| -// state: it reads nothing from this process's memory, so a restarted agent |
| -// gives the same answer. Steps 1 to 4; the caller does step 5 (the final |
| -// checkpoint and the digests), since only it knows the paths and the |
| -// replica it closes. |
| -func (a *Service) ReconstructStopEvidence(ctx context.Context, host string, dbPaths []string, leaseFree func(context.Context) (bool, error)) (moves.StoppedReport, error) { |
| - rep := moves.StoppedReport{V: 1, Host: host, Steps: map[string]moves.Check{}, CheckedAt: time.Now().UTC()} |
| - if a == nil || a.cgroups == nil { |
| - return rep, ErrStopEvidenceUnsupported |
| +// Reconstruct is §5.3's evidence from durable, observable state: it reads |
| +// nothing from the agent's memory, so a restarted agent gives the same |
| +// answer. Steps 1 to 4; the caller does step 5 (the final checkpoint and |
| +// the digests), since only it knows the paths and the replica it closes. |
| +func Reconstruct(ctx context.Context, in ReconstructInputs) (moves.StoppedReport, error) { |
| + rep := moves.StoppedReport{V: 1, Host: in.Host, Steps: map[string]moves.Check{}, CheckedAt: time.Now().UTC()} |
| + if in.Cgroups == nil || in.LeaseFree == nil || !controlname.ValidHostShape(in.Host) { |
| + return rep, ErrInvalidInput |
| + } |
| + procRoot := in.ProcRoot |
| + if procRoot == "" { |
| + procRoot = "/proc" |
| } |
| // 1. Kill and drain the tree. |
| - c, err := a.cgroups.OpenCleanup(host) |
| + c, err := in.Cgroups.OpenCleanup(in.Host) |
| switch { |
| case errors.Is(err, cgroup.ErrNoCgroup): // the group never existed or was removed: an empty tree |
| rep.Steps[moves.StepTree] = moves.Check{OK: true, Reason: "no cgroup"} |
| @@ -586,22 +608,22 @@ func (a *Service) ReconstructStopEvidence(ctx context.Context, host string, dbPa |
| return rep, err |
| default: |
| defer c.Close() |
| - if err := stopevidence.Cleanup(ctx, c); err != nil { |
| + if err := Cleanup(ctx, c); err != nil { |
| rep.Steps[moves.StepTree] = moves.Check{Reason: err.Error()} |
| } else { |
| rep.Steps[moves.StepTree] = moves.Check{OK: true} |
| } |
| } |
| // 2. No stray process in the agent's own leaf. |
| - self, err := os.ReadFile("/proc/self/cgroup") |
| + self, err := os.ReadFile(filepath.Join(procRoot, "self", "cgroup")) |
| if err != nil { |
| return rep, err |
| } |
| leaf, ok := cgroup.ParseSelfCgroup(string(self)) |
| if !ok { |
| - return rep, fmt.Errorf("activator: own cgroup unreadable") |
| + return rep, fmt.Errorf("stopevidence: own cgroup unreadable") |
| } |
| - pids, err := readProcs(filepath.Join(a.cgroups.Root(), "..", leaf, "cgroup.procs")) |
| + pids, err := readProcs(filepath.Join("/sys/fs/cgroup", leaf, "cgroup.procs")) |
| if err != nil { |
| return rep, err |
| } |
| @@ -612,7 +634,7 @@ func (a *Service) ReconstructStopEvidence(ctx context.Context, host string, dbPa |
| } |
| } |
| // 3. No open database. |
| - holders, err := stopevidence.FDHolders("/proc", dbPathsWithSidecars(dbPaths)) |
| + holders, err := FDHolders(procRoot, dbPathsWithSidecars(in.DBPaths)) |
| if err != nil { |
| return rep, err |
| } |
| @@ -622,7 +644,7 @@ func (a *Service) ReconstructStopEvidence(ctx context.Context, host string, dbPa |
| rep.Steps[moves.StepFDs] = moves.Check{Reason: fmt.Sprintf("%d process(es) hold the database open", len(holders))} |
| } |
| // 4. Lease released. |
| - free, err := leaseFree(ctx) |
| + free, err := in.LeaseFree(ctx) |
| if err != nil { |
| return rep, err |
| } |
| @@ -652,10 +674,10 @@ func dbPathsWithSidecars(paths []string) []string { |
| } |
| ``` |
| -(`readProcs` parses one pid per line; `cgroup.ErrNoCgroup` is `cleanup.go:6`'s sentinel for a missing host cgroup. The path arithmetic for the agent's own leaf: `Manager.Root()` is the delegated subtree; the agent's leaf is the directory `ParseSelfCgroup` names under `/sys/fs/cgroup`; compute `filepath.Join("/sys/fs/cgroup", leaf, "cgroup.procs")` and test with a fixture.) `stopreconstruct_other.go` returns `ErrStopEvidenceUnsupported`. The cut's step 5 (Task 5) adds `StepReplica` after the checkpoint. |
| +(`readProcs` parses one pid per line; `cgroup.ErrNoCgroup` is `cleanup.go:6`'s sentinel for a missing host cgroup; the agent's own leaf is the directory `ParseSelfCgroup` names under `/sys/fs/cgroup`, tested with a fixture `procRoot`.) `reconstruct_other.go` returns `ErrUnsupported`. The cut's step 5 (Task 5) adds `StepReplica` after the checkpoint. The host package wires `moveCfg.Evidence` to `stopevidence.Reconstruct` with `Cgroups: eb.Cgroups` at `armMovesPass` (which gains the `eb *activator.ExecBackend` argument; `agent.go:371` has `eb` in hand). |
| -- [ ] **Step 4: Run**: `nice -n 10 go test ./internal/activator ./internal/lease ./internal/moves -count=1 -p 1; GOOS=darwin go build ./...` → PASS. |
| -- [ ] **Step 5: Commit**: `git commit -m "activator: ReconstructStopEvidence from cgroup, proc and lease state (§5.3)" ...` |
| +- [ ] **Step 4: Run**: `nice -n 10 go test ./internal/stopevidence ./internal/lease ./internal/moves -count=1 -p 1; GOOS=darwin go build ./...` → PASS. |
| +- [ ] **Step 5: Commit**: `git commit -m "stopevidence: Reconstruct, stop evidence from cgroup, proc and lease state (§5.3)" ...` |
| --- |
| @@ -688,15 +710,15 @@ func dbPathsWithSidecars(paths []string) []string { |
| - `runCut(ctx, s, w, rec, tgt) error`: the phase loop of §7.4: |
| - `requested`: fresh target and source reports (`moves.Ready` with the last round's delta) and `move.inputs` to pin from the target report → `MoveAdmit(g, now, budget, moves.CutDeadline(now, budget), inputs)`; stale → `MoveBackToReady`. |
| - `admitted`: re-check inputs against current bundle etags (`appconfig.LoadMergedHost` etag), overlay etag, release and launch report digest; mismatch → `MoveBackToReady`; else `MoveCutPhase(admitted → stopping)`. |
| - - `stopping`: `SetHold(host, "moving", id)`; fence (`cfg.Fence`); wait `TimersDrained`; `cfg.Evidence`; incomplete → retry next pass until `cut_deadline`, then `MoveRevert(g, "stop evidence budget")` and `hostclaim.Revert`; complete → write `stopped-source.json` create-only (K13: `release.Store.Put(ctx, key, body, "")` is create-only on both the S3 and the dir store, `store.go:38`, and loses with `release.ErrCASConflict`; on a conflict read the existing object, and equal `Host`, `OpID` and `Role` is a replay and fine) → `MoveCutPhase(stopping → stopped)`. |
| - - `stopped`: `Checkpoint` (final `SyncAndWait`); `N = NewestL0`; `Replication`, `NextDue` from the row; timers snapshot → `move-snapshots/<host>/<id>/timers.json` (own bucket, create-only) and `TimersPrefix` (target, `PutVersioned`); `files.plain` via `archive.PlainDigest`; `MoveRecordSource`; `hostclaim.Update(ExecToNone(id, box, stoppedKey, JournalEvidenceSource))` and `SetJournal(cut_session = holder.session)` in the same transition; `MoveCutPhase(stopped → staged)` is written only after the final round below succeeds, so the phase `stopped` repeats the mirror until it does: |
| + - `stopping`: `SetHold(host, "moving", id)`; fence (`cfg.Fence`); wait `TimersDrained`; `cfg.Evidence` (steps 1 to 4); `!TreeStopped()` → retry next pass until `cut_deadline`, then `MoveRevert(g, "stop evidence budget")` and `hostclaim.Revert`; `TreeStopped()` → the four steps are written into the op journal (`Journal.Steps["evidence"]`, local; nothing is published yet, because the report's `StepReplica`, `FinalTXID` and digests do not exist until the checkpoint) → `MoveCutPhase(stopping → stopped)`. |
| + - `stopped`: `Checkpoint` (final `SyncAndWait`); `N = NewestL0`; `StepReplica` OK with the database's sha256; `Replication`, `NextDue` from the row; timers snapshot → `move-snapshots/<host>/<id>/timers.json` (own bucket, create-only) and `TimersPrefix` (target, `PutVersioned`); `files.plain` via `archive.PlainDigest`; **now** the complete report: `stopped-source.json` create-only (K13: `release.Store.Put(ctx, key, body, "")` is create-only on both the S3 and the dir store, `store.go:38`, and loses with `release.ErrCASConflict`; on a conflict read the existing object, and equal `Host`, `OpID`, `Role` and `FinalTXID` is a replay and fine); `MoveRecordSource`; then `hostclaim.Update(ExecToNone(id, box, stoppedKey, JournalEvidenceSource))` and `SetJournal(cut_session = holder.session)` in the same transition (a crash between any two of these replays the phase: the checkpoint is idempotent on a stopped database, the report is create-only, the CAS is conditional); `MoveCutPhase(stopped → staged)` is written only after the final round below succeeds, so the phase `stopped` repeats the mirror until it does: |
| - the final round: `MirrorPrefix` into `replica/` under the lease; `Deferred > 0` → count in the journal (`moveLedger.DeferredRounds`), three → `MoveRevert`; else seal = `round.Versions` for every current key plus the files and timers versions, `SrcCovered = (N == LocalPosition)`, `FromDB = true` when the stopped database file exists; `SetJournal(seal)` and `SetMovePhase(moving, "" → sealed)` in one claim CAS; `MoveCutPhase(stopped → staged)`. |
| - - `staged`: final archive from the stopped directory: `archive.Build` + `Seal` → `move-snapshots/<host>/<id>/files-<sealed>.sealed` create-only (`PutIfAbsent`), manifest beside it, digests into `MoveSource.Files` (`MoveCutPhase(staged → archived, stamp)`); copy into `FilesPrefix` with `PutVersioned`, adding the versions to the seal (`SetJournal(seal)` again, same `N`). |
| - - `archived`: `MoveFlip(g, now, now + 15 min)`; then `hostclaim.Update(Flip(id, targetBox, holder.stamp, holder.generation))`; `nudgeSerial`. The claim half may be written by anyone who sees the record ahead (§7.5); the source tries once here and the console pass (Plan 5) finishes it. |
| + - `staged`: the final archive from the stopped directory, every step replayable and all of them **before** the phase advances: `archive.Build` + `Seal` → `move-snapshots/<host>/<id>/files-<sealed>.sealed` create-only (`PutIfAbsent`; on `ErrExists` the recorded object is reused, codex 19), the manifest beside it; copy both into `FilesPrefix` with `PutVersioned`; `SetJournal(seal)` with `Files` filled (same `N`); only then `MoveCutPhase(staged → archived, stamp MoveSource.Files digests)`. A crash anywhere inside replays `staged` from the top: the create-only put returns the same object, the copies make new versions the next `SetJournal(seal)` names, and the phase never says `archived` without the seal naming the archive. |
| + - `archived`: precondition, re-read from the claim: `seal.Files` non-empty and `seal.N == Source.FinalTXID` (else the phase is a replay of `staged`'s tail: redo the copy and the journal, never flip); `MoveFlip(g, now, now + 15 min)`; then `hostclaim.Update(Flip(id, targetBox, holder.stamp, holder.generation))`; `nudgeSerial`. The claim half may be written by anyone who sees the record ahead (§7.5); the source tries once here and the console pass (Plan 5) finishes it. |
| - every phase: `now > cut_deadline` → `MoveRevert` (phases `stopping..archived`) and `hostclaim.Revert` once exec is none. |
| - Each phase's local progress is in the op journal F12 (`moves.Journal`, Task 7's shape, written here first as `internal/moves/journal.go`). |
| -Tests (`movescut_test.go`, harness from Plan 1 Task 14 extended with a `fakeEvidence` returning a configurable report, a fake versioned store, and a claim written into the harness store): `TestCutAdmitsOnlyFreshPassingReports`, `TestCutGoesBackToReadyWhenInputsMove`, `TestCutResumesStoppingFromEvidenceAlone` (Review Focus 1: a fresh worker on the same store with the record at `stopping` calls `Evidence` and advances, with no in-memory state), `TestCutRevertsAfterThreeDeferredRounds` (Review Focus 2), `TestSealNamesEveryCurrentVersionAndN` (witness `wSealByVersion`: a stale copy landing after the seal creates a version the seal does not name), `TestCutArchiveIsCreateOnlyAndReused` (codex 19), `TestFlipWritesRecordThenClaim` (record `landing` with region `uk`; claim `landing/checking`, exec `uk-box`, `source_stamp` journalled; `holder.generation` unchanged), `TestCutDeadlinePassedReverts`. |
| +Tests (`movescut_test.go`, harness from Plan 1 Task 14 extended with a `fakeEvidence` returning a configurable report, a fake versioned store with a `failAfter(n)` hook, and a claim written into the harness store): `TestCutAdmitsOnlyFreshPassingReports`, `TestCutGoesBackToReadyWhenInputsMove`, `TestCutResumesStoppingFromEvidenceAlone` (Review Focus 1: a fresh worker on the same store with the record at `stopping` calls `Evidence` and advances, with no in-memory state), `TestStoppedReportIsCompleteBeforeExecToNone` (the report object exists with `StepReplica` and `FinalTXID` before the claim's exec is `""`; a crash injected after the checkpoint and before the report leaves exec with the source; the replay publishes one report), `TestCutRevertsAfterThreeDeferredRounds` (Review Focus 2), `TestSealNamesEveryCurrentVersionAndN` (witness `wSealByVersion`: a stale copy landing after the seal creates a version the seal does not name), `TestCutArchiveIsCreateOnlyAndReused` (codex 19), `TestArchivedNeverPrecedesTheSealedArchive` (crashes injected after the create-only put, after the first `PutVersioned`, and after the second: in every case the record is still `staged`; the replay completes and the seal names both `files/` versions before `archived`; a record hand-edited to `archived` with an empty `seal.Files` does not flip), `TestFlipWritesRecordThenClaim` (record `landing` with region `uk`; claim `landing/checking`, exec `uk-box`, `source_stamp` journalled; `holder.generation` unchanged), `TestCutDeadlinePassedReverts`. |
| - [ ] **Step 1 to 4** as above. Run: `nice -n 10 go test ./internal/host ./internal/moves -run 'TestCut|TestSeal|TestFlip' -count=1 -p 1` → PASS. |
| - [ ] **Step 5: Commit**: `git commit -m "host: the journaled cut, requested to flipped, with stop evidence and the seal by version (§7.4, §7.5)" ...` |
| @@ -713,10 +735,10 @@ Tests (`movescut_test.go`, harness from Plan 1 Task 14 extended with a `fakeEvid |
| **Interfaces:** |
| - `provision.ReconcileConfig.Hold func(rec Record) (status, op string, ok bool)`: when ok, a row minted for `rec` is `Upsert`ed with `Status = status, HoldOp = op` and never `ready`; an existing row under that op is left as is. `KeepParticipant func(host string) bool`: when true, the removal phase skips the row ("participant of a running operation"), and the orphan guard does not act on it. |
| -- `schedule.Runner.Busy(host string) bool` (the per-host claim, `runner.go:276`); the runner's `Tick` skips `registry.IsHold(rt.Status)` rows before `claim`, with no bookkeeping. `schedule.Converger.Frozen func(host string) bool`: `convergeHost` returns `(false, nil)` for a frozen host, and the prune's `alive` set includes frozen hosts so `PruneTimers` keeps their rows. |
| +- `schedule.Runner.Busy(host string) bool` (the per-host claim, `runner.go:276`); the runner's `Tick` takes `claim(host)` **first** and then re-reads the route (`r.Reg.Get`) inside the claim: a held row (`registry.IsHold(rt.Status)`) releases the claim and continues with no bookkeeping. The order matters (the cut's drain): `SetHold` then "wait until `!Busy(host)`" is correct only if every tick that observes the row **after** the hold does so while holding the claim, so a tick paused between its route read and its claim cannot slip past the drain and advance `attempts` (`runner.go:350`). The route read before `claim` at `runner.go:143` stays as a cheap pre-filter; the decision is the second read. `schedule.Converger.Frozen func(host string) bool`: `convergeHost` returns `(false, nil)` for a frozen host, and the prune's `alive` set includes frozen hosts so `PruneTimers` keeps their rows. |
| - `movesHoldView` in the host package: built from each moves pass's record list plus `hostclaim.Read` per host with a move in flight; exposes `hold(rec) (status, op, ok)` (target after the flip: `landing`; source in `reverting`/`restoring`: `restoring`; target in a rollback's release: `releasing`; source during `moving` at or after `stopping`: `moving`), `keep(host) bool` (this box in `op.participants` of a move-kind op), `frozen(host) bool` (a move-kind op holds the claim), `strict(host) bool` (same). |
| -Test `TestReconcileHoldsParticipantRows` (Review Focus 5): a record with region `uk` and a `Hold` returning `("landing", id, true)` mints the row held on the uk box and never `ready`; the ie box with `KeepParticipant(host) == true` keeps its `moving` row though the record's region is `uk`; with `KeepParticipant == false` the row is removed as today. Test `TestHeldRowsFreezeTimers` (witness `wTimerFreeze`): a held row's due once-timer is not delivered and its `attempts` does not advance; a frozen host's declared schedule is neither upserted nor pruned by `Converger.Pass`; after the hold lifts both resume. |
| +Test `TestReconcileHoldsParticipantRows` (Review Focus 5): a record with region `uk` and a `Hold` returning `("landing", id, true)` mints the row held on the uk box and never `ready`; the ie box with `KeepParticipant(host) == true` keeps its `moving` row though the record's region is `uk`; with `KeepParticipant == false` the row is removed as today. Test `TestHeldRowsFreezeTimers` (witness `wTimerFreeze`): a held row's due once-timer is not delivered and its `attempts` does not advance; a frozen host's declared schedule is neither upserted nor pruned by `Converger.Pass`; after the hold lifts both resume. Test `TestDrainRaceCannotAdvanceBookkeeping`: a tick reads the route (ready), pauses on a test hook before `claim`; the cut sets the hold and observes `Busy == false`; the tick resumes, claims, re-reads, sees the hold and releases; `attempts` is unchanged and nothing was delivered. |
| - [ ] **Steps 1 to 4**. Run: `nice -n 10 go test ./internal/provision ./internal/schedule ./internal/host -count=1 -p 1` → PASS. |
| - [ ] **Step 5: Commit**: `git commit -m "reconcile and schedule honour move holds: participant rows kept, held rows never ready, timers frozen and drained (§5.2, §9.4, §9.6, K9)" ...` |
| @@ -747,12 +769,12 @@ Tests: `RenderHeld` writes the file for a `landing` row that `SyncExec` would sk |
| **Interfaces:** |
| - `moves.ReadDBHeader(path string) (pageSize uint32, pageCount uint32, err error)` (SQLite header bytes 16-17 and 28-31). |
| -- `moves.EncodeFull(db io.ReaderAt, pageSize, pageCount uint32, min, max ltx.TXID, timestamp int64, walOffset int64) (data []byte, crc64 uint64, err error)`: `ltx.NewEncoder` with `Header{Version: ltx.Version, Flags: ltx.HeaderFlagNoChecksum, PageSize, Commit: pageCount, MinTXID: min, MaxTXID: max, Timestamp: timestamp, WALOffset: walOffset}`, every page in order, `Close`; the CRC-64 is the trailer's `FileChecksum` (read it back with `ltx.NewDecoder(bytes.NewReader(data)).Verify()` then `Trailer().FileChecksum`). |
| +- `moves.EncodeFull(db io.ReaderAt, pageSize, pageCount uint32, min, max ltx.TXID, timestamp int64, walOffset int64) (data []byte, crc64 uint64, err error)`: `ltx.NewEncoder` with `Header{Version: ltx.Version, Flags: ltx.HeaderFlagNoChecksum, PageSize, Commit: pageCount, MinTXID: min, MaxTXID: max, Timestamp: timestamp, WALOffset: walOffset}`, every page in order **except SQLite's lock page**, `ltx.LockPgno(pageSize)` (`ltx.go:494`; the encoder refuses it, `encoder.go:219`, and litestream skips it), while `Commit` keeps the full page count; `Close`; the CRC-64 is the trailer's `FileChecksum` (read it back with `ltx.NewDecoder(bytes.NewReader(data)).Verify()` then `Trailer().FileChecksum`). |
| - `moves.BuildIntent{V int; Op, Host string; N uint64; PageSize, Commit uint32; Timestamp int64}`; `moves.InstallManifest{V int; Op, Host string; N uint64; SnapshotKey, SnapshotVersion string; SnapshotCRC64 uint64; AnchorKey, AnchorVersion string; AnchorCRC64 uint64; IntegrityCheck string; DBSHA256 string; BuiltAt time.Time}`; `moves.GenerationPrefix(location, op string) string` (`<location>g-<op>/`, location with its trailing slash); `moves.SnapshotKey(prefix string, n uint64)` = `prefix + "0009/" + ltx.FormatFilename(1, n)`; `moves.AnchorKey(prefix, n)` = `prefix + "0000/" + ltx.FormatFilename(n, n)`; `BuildIntentKey(prefix) = prefix+"build-intent.json"`, `InstallManifestKey(prefix) = prefix+"install.json"`. |
| - `moves.NewSealedClient(store lease.VersionedStore, bucket, replicaPrefix string, seal Seal) *SealedClient` implementing `litestream.ReplicaClient`: `Type() "sealed"`, `Init` no-op, `LTXFiles(ctx, level, seek, _)` returns `ltx.NewFileInfoSliceIterator` over the sealed objects at that level parsed from their keys (`ltx.ParseFilename`), `OpenLTXFile` opens exactly the sealed version by `OpenVersion(ctx, bucket, key, versionID)` (ranged through `VersionRangeOpener` when offset/size are set), `WriteLTXFile`, `DeleteLTXFiles`, `DeleteAll` return `ErrSealedReadOnly`, `SetLogger` stores it. |
| -- `moves.BuildGeneration(ctx, in BuildInputs) (InstallManifest, error)` with `BuildInputs{Store lease.VersionedStore; Bucket, ReplicaPrefix string; Seal Seal; GenerationPrefix string; Op, Host string; WorkDir string; Now func() time.Time; Verify func(dbPath string) error}`: (1) read or write the intent (`PutIfAbsent`; on `lease.ErrExists` read it and use its timestamp, page size, commit); (2) restore: `litestream.NewReplicaWithClient(nil, sealedClient).Restore(ctx, opt)` with `opt.OutputPath = <WorkDir>/<host>.db`, `opt.TXID = ltx.TXID(seal.N)`; refuse unless the restored position is `N` (litestream writes `<db>-txid`? Only in follow mode; instead compare the plan's final `MaxTXID` through `litestream.CalcRestorePlan(ctx, client, ltx.TXID(N), time.Time{}, logger)` whose last file's `MaxTXID` must be `N`, and verify every planned file with `ltx.NewDecoder(...).Verify()` before the restore); `Verify(dbPath)` = `lease.CheckDB` (integrity_check); (3) `EncodeFull` snapshot `[1, N]` and `PutIfAbsent(SnapshotKey)`; on `ErrExists`, `OpenVersion` the current one and require equal bytes (identical by construction from the intent; differing bytes refuse with `ErrBuildDiverged`); (4) the anchor `[N, N]` with `WALOffset = litestream.WALHeaderSize`, same rule; (5) sha256 of the database file; (6) `InstallManifest` via `PutIfAbsent` (on exists, read and compare `N`, keys, versions). |
| +- `moves.BuildGeneration(ctx, in BuildInputs) (InstallManifest, error)` with `BuildInputs{Store lease.VersionedStore; Bucket, ReplicaPrefix string; Seal Seal; GenerationPrefix string; Op, Host string; WorkDir string; Now func() time.Time; Verify func(dbPath string) error}`: (1) read or write the intent (`PutIfAbsent`; on `lease.ErrExists` read it and use its timestamp, page size, commit); (2) restore into a fresh scratch file: remove `<WorkDir>/<host>.db`, `-wal`, `-shm` and `-txid` first (litestream's `Restore` refuses an existing output path, `replica.go:601`; the file is reproducible from the sealed versions, so a replay after any crash starts clean), then `litestream.NewReplicaWithClient(nil, sealedClient).Restore(ctx, opt)` with `opt.OutputPath = <WorkDir>/<host>.db`, `opt.TXID = ltx.TXID(seal.N)`; refuse unless the restored position is `N` (litestream writes `<db>-txid`? Only in follow mode; instead compare the plan's final `MaxTXID` through `litestream.CalcRestorePlan(ctx, client, ltx.TXID(N), time.Time{}, logger)` whose last file's `MaxTXID` must be `N`, and verify every planned file with `ltx.NewDecoder(...).Verify()` before the restore); `Verify(dbPath)` = `lease.CheckDB` (integrity_check); (3) `EncodeFull` snapshot `[1, N]` and `PutIfAbsent(SnapshotKey)`; on `ErrExists`, `OpenVersion` the current one and require equal bytes (identical by construction from the intent; differing bytes refuse with `ErrBuildDiverged`); (4) the anchor `[N, N]` with `WALOffset = litestream.WALHeaderSize`, same rule; (5) sha256 of the database file; (6) `InstallManifest` via `PutIfAbsent` (on exists, read and compare `N`, keys, versions). |
| -Tests (`build_test.go`, using `internal/lease/leasetest`'s local versioned store if it offers one, else a small in-memory `lease.VersionedStore` fake in `moves` tests): `TestSealedClientServesOnlySealedVersions` (a newer version of a sealed key exists; `OpenLTXFile` returns the sealed bytes; an unsealed key is absent); `TestBuildGenerationReachesNAndWritesSnapshotAndAnchor` (drive litestream over a real sqlite file through the file replica client as `litestreamfacts` does to produce a replica, seal its current versions, build, then assert: snapshot header `MinTXID 1, MaxTXID N, Flags NoChecksum`, anchor `[N, N]` with `WALOffset == WALHeaderSize`, both `Verify()`, manifest's `DBSHA256` equals the file's, `IntegrityCheck == "ok"`); `TestBuildReplayReusesIdenticalSnapshot` (Review Focus 3, witness `wBuildCrashes`: run the build with a `Store` that fails after the snapshot PUT; run again on the same store; the snapshot key has one version, the manifest exists, bytes identical); `TestBuildRefusesAGapOrAShortRestore` (drop a sealed middle file → `ErrBuildIncomplete`, no snapshot PUT). |
| +Tests (`build_test.go`, using `internal/lease/leasetest`'s local versioned store if it offers one, else a small in-memory `lease.VersionedStore` fake in `moves` tests): `TestSealedClientServesOnlySealedVersions` (a newer version of a sealed key exists; `OpenLTXFile` returns the sealed bytes; an unsealed key is absent); `TestBuildGenerationReachesNAndWritesSnapshotAndAnchor` (drive litestream over a real sqlite file through the file replica client as `litestreamfacts` does to produce a replica, seal its current versions, build, then assert: snapshot header `MinTXID 1, MaxTXID N, Flags NoChecksum`, anchor `[N, N]` with `WALOffset == WALHeaderSize`, both `Verify()`, manifest's `DBSHA256` equals the file's, `IntegrityCheck == "ok"`); `TestBuildReplayReusesIdenticalSnapshot` (Review Focus 3, witness `wBuildCrashes`: run the build with a `Store` that fails after the snapshot PUT, **in the same `WorkDir`**, with the half-restored database file left on disk; run again on the same store and directory; the snapshot key has one version, the manifest exists, bytes identical; a second variant fails during the restore itself, leaving a partial output file, and the replay still completes); `TestBuildRefusesAGapOrAShortRestore` (drop a sealed middle file → `ErrBuildIncomplete`, no snapshot PUT); `TestEncodeFullSkipsTheLockPage` (a sparse database file extended past 1 GiB with `Truncate`, page size 4096: the encoder succeeds, the file has `pageCount - 1` pages, `Verify()` passes, and litestream's own restore of it produces a database whose `PRAGMA page_count` equals the original). |
| - [ ] **Steps 1 to 4**. Run: `nice -n 10 go test ./internal/moves -run 'TestSealedClient|TestBuild' -count=1` → PASS. |
| - [ ] **Step 5: Commit**: `git commit -m "moves: build the generation from the sealed replica: planner restore, snapshot, anchor, intent and manifest (§5.1 steps 2 and 7, §7.6 building)" ...` |
| @@ -768,7 +790,7 @@ Tests (`build_test.go`, using `internal/lease/leasetest`'s local versioned store |
| **Interfaces:** |
| - Kinds `"park-superseded"` (a whole location: `Park.Location` is `instances/<host>` or `instances/<host>/<inc>`) and `"park-staging"` (`Park.Location` is `moves/<host>/<op>`); `archiveBase` for both: `"superseded/" + o.Op + "/" + host + "/" + o.Retirement + "/"`; `destination` appends `instances/...` or `moves/...` under it as today's `kind + "/"` rule does. |
| - `(e Engine) Parked(ctx, w Work) (bool, error)`: the manifest exists and every version in its deletion set is gone (what the landing's `parking` phase waits for, §7.6; acknowledgement still waits for the horizon in `Run`). |
| -- `hostclaim.QueueSupersededPark(opID, region, box string, scope ParkScope, newID func() string) func(*Claim) error` (appends `{Kind: "park-superseded", Region, Owner: "region:"+region, Park: &scope, Retirement: newID(), ID: newID()}` and, when `box != ""`, a `park-local` for the box; scope validated with `Validate(host)`); `hostclaim.QueueStagingPark(opID, region string, scope ParkScope, newID func() string) func(*Claim) error`. |
| +- `hostclaim.QueueSupersededPark(opID, region, box string, scope ParkScope, newID func() string) func(*Claim) error` (appends `{Kind: "park-superseded", Region, Owner: "region:"+region, Park: &scope, Retirement: newID(), ID: newID()}` and, when `box != ""`, a `park-local` for the box; scope validated with `Validate(host)`; idempotent per `(opID, region)` and `(opID, box)`); `hostclaim.QueueLocalPark(opID, box string, newID func() string) func(*Claim) error` (a `park-local` alone, for the landing's `local` phase: `park.Local.RetireArtifacts` runs only against a persisted `park-local` chore of the claim, `local.go:62`, and treats an unknown id as already completed, so the chore must exist in the claim before the call); `hostclaim.QueueStagingPark(opID, region string, scope ParkScope, newID func() string) func(*Claim) error`. `Plan 5 Task 4`'s destination patch covers `local_linux.go:196-201` (the `localWalk` segments). |
| Tests: `TestSupersededDestination` (a `park-superseded` obligation archives `instances/h/<inc>/0000/x.ltx@v1` to `superseded/<op>/h/<ret>/instances/h/<inc>/0000/x.ltx@v1`); `TestParkedReportsManifestAndDeletions`; `TestQueueSupersededParkValidatesScope`. |
| @@ -786,11 +808,12 @@ Tests: `TestSupersededDestination` (a `park-superseded` obligation archives `ins |
| **Interfaces:** |
| - `moveCfg` gains: `Guard hostguard.Guard`, `Park park.Engine` (own bucket and region; `View`, `Authorize` from Plan 5's providers until then a view that lists the location and authorizes only the landing's own op), `Local park.Local`, `RenderHeld func(ctx, rt registry.Route, op string) (appconfig.HeldRender, error)`, `MaterializeHeld func(ctx, rt registry.Route, version string) error`, `ReplaceTimers func(host string, rows []registry.Timer) error`, `SetReplication func(host, class string) error`, `SetNextDue func(host string, due int64) error`, `RemoveVerifyEntry func(ctx, host string) error` (loads `dbverify.Load`, deletes `Hosts[host]`, `Publish`), `OwnLease func(ctx, prefix string) (*lease.Held, error)`, `SecretsPriv *ecdh.PrivateKey`. |
| - `landingPass(ctx, s, w, rec) error` per phase (§7.6), each phase first `MoveLandingPhase(g, from, to)` then its side effects, journaled in `moves.Journal`: |
| - - `checking`: read `stopped-source.json` (must be `Complete()`); claim: `RequireMoveOp(id, KindLanding, "checking")`, `Exec == cfg.Box`; `hostclaim.Update(Touch(id, nil, []string{cfg.Box}))` if absent; `move.json` incarnation == `rec.Move.Incarnation` == `holder.incarnation`; `cfg.Guard.Activate(ctx, host, cfg.Box, nil)` (sets `active_stamp = holder.stamp`, the flip's); acquire `OwnLease` on `holder.location` and keep it in the worker's per-host map through `published`. |
| + - every landing phase: the record CAS (`MoveLandingPhase`) first, then the claim mirrors it (`hostclaim.Update(SetMovePhase(id, KindLanding, from, to))`), so the claim reaches `probing` and `probed` and the probe's admission observes the phase it binds to (`AdmitWriter` checks the phase and never advances it, `generation.go:35`); a replay that finds the claim already ahead of the record phase it is writing skips the claim half. |
| + - `checking`: read `stopped-source.json` (must be `Complete()`); claim: `RequireMoveOp(id, KindLanding, "checking")`, `Exec == cfg.Box`; `hostclaim.Update(Touch(id, nil, []string{cfg.Box}))` if absent (Plan 1 Task 14 added it at the first staging write; this is the idempotent K12 re-add); `move.json` (own bucket, readable here) incarnation == `rec.Move.Incarnation` == `holder.incarnation`; `cfg.Guard.Activate(ctx, host, cfg.Box, nil)` (sets `active_stamp = holder.stamp`, the flip's); `hostclaim.Update(QueueLocalPark(id, cfg.Box, newID))` and `cfg.Local.RecordInventory(ctx, host, captured, "flat", []string{dbPath, filesDir, envPath}, check)` so the `local` phase has a persisted chore to retire against; acquire `OwnLease` on `holder.location` and keep it in the worker's per-host map through `published`. |
| - `parking`: list `holder.location` in the own bucket (`Store.ListVersions`); nothing there → done; else queue `QueueSupersededPark(id, cfg.Region, "", ParkScope{Location: strings.TrimSuffix(location, "/"), Incarnation: holder.incarnation, EvidenceAt: rec.Move.FlippedAt, RequestLimit: 120 s}, newID)` once (journal step), `cfg.Park.Run(ctx, Work{Host, ID, Op})` (ErrPending expected), then `cfg.Park.Parked` must be true before advancing. |
| - `building`: `moves.BuildGeneration` with the seal from the claim journal (`DecodeSeal(obs.Claim.Op.Journal[JournalSeal])`), `ReplicaPrefix` in the own bucket, `GenerationPrefix(holder.location, id)`, `WorkDir = JournalDir`. |
| - `published`: journal only (no `generation.json`, R10-3). |
| - - `local`: `cfg.Local.RetireArtifacts(ctx, host, hostguard.Capture(dataDir, host), newID(), id, previousStamp)` for a warm database, files directory and `<host>.env` of an earlier stay (previous stamp = the guard record's retired stamp; when none, skip); `archive.Open` + `Extract` the newest archive from `FilesPrefix` (by the seal's files version) into `<DataDir>/<host>/` and `<DataDir>/<host>.env`; `ReplaceTimers(host, snapshot.Rows)` from the sealed timers version; `SetReplication(host, Source.Replication)`, `SetNextDue(host, Source.NextDue)`; `RemoveVerifyEntry`; fsync the directory; write `.files-gen` `{"op": id, "incarnation": inc}` last. |
| + - `local`: `cfg.Local.RetireArtifacts(ctx, host, hostguard.Capture(dataDir, host), <the park-local chore id queued at checking>, id, previousStamp)` for a warm database, files directory and `<host>.env` of an earlier stay (previous stamp = the guard record's retired stamp; when none, the chore is acknowledged empty with `AcknowledgeEmpty`); the test asserts on the filesystem that the old files are under `.parked/` and gone from the live paths, not on the call returning nil; `archive.Open` + `Extract` the newest archive from `FilesPrefix` (by the seal's files version) into `<DataDir>/<host>/` and `<DataDir>/<host>.env`; `ReplaceTimers(host, snapshot.Rows)` from the sealed timers version; `SetReplication(host, Source.Replication)`, `SetNextDue(host, Source.NextDue)`; `RemoveVerifyEntry`; fsync the directory; write `.files-gen` `{"op": id, "incarnation": inc}` last. |
| - `configured`: `RenderHeld`; `MaterializeHeld(rt, rec.Move.Inputs.Release)`; the ack's `BundleETags` must equal `rec.Move.Inputs.BundleETags` and its overlay etag `Inputs.OverlayETag`, else `MoveRollBack` (Plan 4 Task 1's transition; until it exists, write `Reason` and stop: `markForRollback(rec, "configuration changed during the move")` is the one call site Plan 4 replaces). |
| - `restored`: copy the built database to the row's `DBPath` (flat), sha256 == manifest's, `lease.CheckDB`, write `<dbPath>.gen` `{"op": id, "incarnation": inc, "location": holder.location}`, `MoveLandingPhase(configured → restored, stamp Landing.InstalledDBSHA256)`. |
| - `landing_deadline` passed before `restored` → the rollback marker as above. |
diff --git a/docs/superpowers/plans/2026-10-06-moves-phase1-3-probe-commit-open.md b/docs/superpowers/plans/2026-10-06-moves-phase1-3-probe-commit-open.md| index dd265f55f..99ccd71a9 100644 |
| --- a/docs/superpowers/plans/2026-10-06-moves-phase1-3-probe-commit-open.md |
| +++ b/docs/superpowers/plans/2026-10-06-moves-phase1-3-probe-commit-open.md |
| @@ -106,7 +106,7 @@ func TestProbeWindow(t *testing.T) { |
| **Files:** modify `internal/provision/provision.go`, test `internal/provision/move_test.go`. |
| **Interfaces:** |
| -- `MoveProbeAttempt(g MoveGuard, n int)` (state `landing`, phase `probing`; records `Landing.Attempts = n`; add `Attempts int \`json:"attempts,omitempty"\`` to `MoveLanding`); `MoveCommit(g, now time.Time) func(*Record) error` (`landing/probed`, `now <= *CutDeadline`, else `ErrMoveConflict` wrapping `ErrCutDeadline`; sets `State = committed`, `CommittedAt`); `MoveOpened(g, now)` (`committed`, `OpenedAt`); `MoveSourceParked(g, now)` (`committed`, `SourceParkedAt`); `MoveClose(g)` (`committed` with `OpenedAt` and `SourceParkedAt` set → `Move = nil`, region already `To`; Plan 5 calls it when both obligations are gone). |
| +- `MoveProbeAttempt(g MoveGuard, n int)` (state `landing`, phase `probing`; records `Landing.Attempts = n` on the `MoveLandingJournal`, Plan 1 Task 2); `MoveCommit(g, now time.Time) func(*Record) error` (`landing/probed`, `now <= *CutDeadline`, else `ErrMoveConflict` wrapping `ErrCutDeadline`; sets `State = committed`, `CommittedAt`); `MoveOpened(g, now)` (`committed`, `OpenedAt`); `MoveSourceParked(g, now)` (`committed`, `SourceParkedAt`); `MoveClose(g, now time.Time)` (`committed` with `OpenedAt` and `SourceParkedAt` set → `Move = nil`, region already `To`, and `rec.LastMove = &MoveOutcome{ID: g.ID, Outcome: "done", At: now, OfflineS: OpenedAt - CutAt}` in the same write, index D11; Plan 5 calls it when both obligations are gone). |
| - [ ] **Step 1: Write the failing test** |
| @@ -125,7 +125,7 @@ func TestCommitRefusedPastTheDeadline(t *testing.T) { |
| if err := MoveCommit(g, dl)(rec); !errors.Is(err, ErrMoveConflict) { |
| t.Fatal("second commit") |
| } |
| - if err := MoveClose(g)(rec); !errors.Is(err, ErrMoveConflict) { |
| + if err := MoveClose(g, now)(rec); !errors.Is(err, ErrMoveConflict) { |
| t.Fatal("close before open and park") |
| } |
| if err := MoveOpened(g, now)(rec); err != nil || rec.Move.OpenedAt == nil { |
| @@ -134,7 +134,7 @@ func TestCommitRefusedPastTheDeadline(t *testing.T) { |
| if err := MoveSourceParked(g, now)(rec); err != nil { |
| t.Fatal(err) |
| } |
| - if err := MoveClose(g)(rec); err != nil || rec.Move != nil || rec.Region != "uk" { |
| + if err := MoveClose(g, now)(rec); err != nil || rec.Move != nil || rec.Region != "uk" || rec.LastMove == nil || rec.LastMove.ID != g.ID || rec.LastMove.Outcome != "done" { |
| t.Fatalf("close: %v %+v", err, rec) |
| } |
| } |
| @@ -191,7 +191,14 @@ func TestCommitQueuesObligationsAndCoversTheSourceRegion(t *testing.T) { |
| _ = SetMovePhase(op, KindMoving, PhaseMoving, PhaseSealed)(c) |
| c.Gen = 7 |
| _ = Flip(op, "uk-box", 3, "")(c) |
| - c.Op.Phase = "probed" |
| + // The landing mirrors every record phase into the claim (Plan 2 Task |
| + // 10), so the claim reaches probed through SetMovePhase, never a |
| + // fixture edit. |
| + for _, step := range [][2]string{{"checking", "parking"}, {"parking", "building"}, {"building", "published"}, {"published", "local"}, {"local", "configured"}, {"configured", "restored"}, {"restored", "probing"}, {"probing", "probed"}} { |
| + if err := SetMovePhase(op, KindLanding, step[0], step[1])(c); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| // The target's probe admission left a record whose predecessor is in ie. |
| c.Holder.Generation = op |
| c.Holder.Admissions = []AdmissionRecord{{Region: "uk", PredecessorRegion: "ie", Session: 9, Box: "uk-box", Generation: op, Predecessor: "", Mode: "cold", Coverage: "sealed"}} |
| @@ -282,7 +289,7 @@ func Commit(opID, sourceRegion, sourceBox, targetRegion string, sourceScope, sta |
| **Files:** create `internal/moves/startcheck.go`, `internal/moves/startcheck_test.go`. |
| **Interfaces:** |
| -- `moves.GenerationMarker{Op, Incarnation, Host, Box string; InstalledAt time.Time; ManifestSHA256 string; SnapshotKey, SnapshotVersion string; SnapshotTXID, SnapshotCRC64 uint64}` (the `generation.json` shape, `genselect.Marker`'s fields); `moves.FilesGen{Op, Incarnation string}`; `moves.DBGen{Op, Incarnation, Location string}`; `moves.StartCheck(c *hostclaim.Claim, marker *GenerationMarker, filesGen *FilesGen, dbGen *DBGen) error` (§5.1 the start check: (1) `marker` absent = origin; its `Op` must be `holder.generation` or the predecessor of the latest admission or of a chain of admissions each without a marker (walk `holder.admissions` from the newest: a generation whose `Predecessor` matches is accepted); its `Incarnation` the holder's; (2) `filesGen` when present names the same generation and incarnation as the claim expects (`holder.generation`); (3) `dbGen` when present names the same generation and `holder.location`); errors `ErrStartMarker`, `ErrStartFiles`, `ErrStartDB` with member-safe reasons. |
| +- `moves.GenerationMarker{Op, Incarnation, Host, Box string; InstalledAt time.Time; ManifestSHA256 string; SnapshotKey, SnapshotVersion string; SnapshotTXID, SnapshotCRC64 uint64}` (the `generation.json` shape, `genselect.Marker`'s fields); `moves.FilesGen{Op, Incarnation string}`; `moves.DBGen{Op, Incarnation, Location string}`; `moves.PreparedCheck(prepared string, holder hostclaim.Holder, filesGen *FilesGen, dbGen *DBGen) error` (the pre-admission validation of a landing probe's inputs: both markers name `prepared` and the holder's incarnation and location; Plan 3 Task 5 step 3); `moves.StartCheck(c *hostclaim.Claim, marker *GenerationMarker, filesGen *FilesGen, dbGen *DBGen) error` (the §5.1 start check against the claim **after** the admission CAS, Task 5 step 6) (§5.1 the start check: (1) `marker` absent = origin; its `Op` must be `holder.generation` or the predecessor of the latest admission or of a chain of admissions each without a marker (walk `holder.admissions` from the newest: a generation whose `Predecessor` matches is accepted); its `Incarnation` the holder's; (2) `filesGen` when present names the same generation and incarnation as the claim expects (`holder.generation`); (3) `dbGen` when present names the same generation and `holder.location`); errors `ErrStartMarker`, `ErrStartFiles`, `ErrStartDB` with member-safe reasons. |
| - [ ] **Step 1: Write the failing test**: origin everywhere passes; marker naming the holder's generation passes; marker naming the predecessor of the latest admission passes (lagging); marker naming an unrelated generation refuses `ErrStartMarker`; `.files-gen` with another op refuses `ErrStartFiles`; `<dbPath>.gen` with another location refuses `ErrStartDB`. |
| - [ ] **Step 2: Run**, **Step 3: implement**, **Step 4: run** → PASS. |
| @@ -295,20 +302,20 @@ func Commit(opID, sourceRegion, sourceBox, targetRegion string, sourceScope, sta |
| **Files:** create `internal/activator/probewake.go`, `internal/activator/peercred_linux.go`, `internal/activator/peercred_other.go`, `internal/activator/probewake_test.go`. |
| **Interfaces:** |
| -- Consumes (claim Plan 6): `hostbox.Writer` (`Baseline`, `Admit`, `Published`), `hostbox.GenerationPrefix`, `spawnClaimed`, and Plan 6 Task 9's admitted-start sequence exposed as `(a *Service) startAdmitted(ctx, e *actEntry, admitted hostclaim.Observed, policy startPolicy) error` (index: "exact interfaces needed"). Consumes `genselect.SelectPrepared(ctx, s, req, genselect.PreparedPin{Generation, SnapshotKey, SnapshotVersion, SnapshotTXID, SnapshotCRC64})` to plan the prepared generation's restore from the install manifest. |
| +- Consumes (claim Plan 6): `hostbox.Writer` (`Baseline`, `Admit`, `Published`), `hostbox.GenerationPrefix`, `spawnClaimed(ctx, enabled, manager, host, limit, selected, identity, build) (*exec.Cmd, *cgroup.Prepared, error)` (the `*cgroup.Prepared` is kept for the process's life and, on every exit path including a failed start, `KillAndEmpty` then `Close` run before the attempt is counted), and Plan 6 Task 9's admitted-start sequence exposed as `(a *Service) startAdmitted(ctx, e *actEntry, admitted hostclaim.Observed, policy startPolicy) error` (index: "exact interfaces needed"). Consumes `genselect.SelectPrepared(ctx, s, req, genselect.PreparedPin{Generation, SnapshotKey, SnapshotVersion, SnapshotTXID, SnapshotCRC64})` to plan the prepared generation's restore from the install manifest. |
| - Produces: `activator.ProbeConfig{Host, Op string; Kind string ("landing" | "recovery"); Prepared *moves.InstallManifest (nil for a recovery probe into the source's own generation); Seal *moves.Seal; CutDeadline time.Time; Attempt moves.ProbeAttempt; RenderHeld func(ctx) error; ReadyPath string; LossID string (empty except for a lost-target return, Plan 4 Task 10, where the admission goes through hostclaim.AdmitWriterWithLoss)}`; `activator.ProbeResult{Session uint64; Generation string; AnsweredAt time.Time}`; `(a *Service) ProbeWake(ctx, cfg ProbeConfig) (ProbeResult, error)`: |
| 1. the row must be held with `hold_op == cfg.Op` and the claim must name this box `op.exec` (K1, K6), else `ErrProbeRefused`; |
| 2. `cfg.RenderHeld(ctx)` (the caller composes `RenderHeld` + `MaterializeHeld`; every probe kind, model 4); |
| - 3. the start check (Task 4) over `generation.json` (read by the lease round trip), `.files-gen`, `<dbPath>.gen`; |
| + 3. **prepared-input validation**, before any admission: for a landing probe, `.files-gen` and `<dbPath>.gen` must name the prepared generation (`cfg.Op`) and the holder's incarnation and location, and `cfg.Prepared` (the install manifest) must exist with `N == cfg.Seal.N`; this is not the §5.1 start check, which compares against `holder.generation`: before the first probe's admission `holder.generation` is still the source's (the flip preserves it, §7.5; only `AdmitWriter` moves it, `generation.go:27`), so a start check here would refuse the landing's own files. For a recovery probe, `.files-gen` and `<dbPath>.gen`, when present, must name `holder.generation` as observed (the source's own, or the return generation after a dirty install); |
| 4. `cfg.Attempt.MayStart(now, cfg.CutDeadline)`; |
| 5. the admission: `decide` per §7.14: cold; `NewGeneration = cfg.Op` when `holder.generation` is neither `cfg.Op` nor a descendant admitted from it, else a fresh id (`newID()`); `KeepStamp = holder.stamp > holder.session`; `Coverage = "sealed"` when `cfg.Seal != nil && cfg.Seal.FromDB` and the generation is the prepared one, else `""`; `Region`, `PredecessorRegion` per the last admission record; `RemoteMaxTXID` from the baseline plan (`SelectPrepared` for the prepared generation; `Writer.Baseline` for a fresh one); then `Writer.Admit`; |
| - 6. `startAdmitted` with `startPolicy{MaxStarts: moves.ProbeMaxStarts, Window: moves.ProbeWindow, Deadline: min(attempt deadline, cut deadline), Probe: true}`, spawning through `spawnClaimed` with `CARLOS_MOVE_PROBE=1` in the child's environment; the spawn is strict; |
| + 6. `startAdmitted` with `startPolicy{MaxStarts: moves.ProbeMaxStarts, Window: moves.ProbeWindow, Deadline: min(attempt deadline, cut deadline), Probe: true}`; **the §5.1 start check runs here, inside the admitted sequence**, with `moves.StartCheck(claimAfterCAS, marker, filesGen, dbGen)` against the generation the admission CAS just named (`holder.generation == cfg.Op` for a first probe; a fresh id for a restarted one, whose `.files-gen` still names `cfg.Op` and is accepted through the admission chain: Task 4's `StartCheck` takes the claim and walks `holder.admissions`); the spawn goes through `spawnClaimed` with `CARLOS_MOVE_PROBE=1` in the child's environment and is strict; |
| 7. readiness: `GET cfg.ReadyPath` over the row's socket with `Host: <host>`, 200 to 399, the peer's `SO_PEERCRED` pid a member of `carlos-exec@<host>` (`cgroup.Cleanup.Members`), `cfg.Attempt.AcceptReadiness(answeredAt, cfg.CutDeadline)`; a 20 s settle, the process still running, a second request; |
| 8. the marker and publication are `startAdmitted`'s (first upload, `generation.json` by If-Match or If-None-Match, `Writer.Published`); the entry stays held: `RetireExcept` keeps entries whose row `IsHold`. |
| A failure after the admission stops the tree (`FenceAndStop` with a fresh generation) and returns `ErrProbeFailed` with the reason; the attempt is over; the operation decides what next. |
| - `peercred_linux.go`: `peerPID(conn net.Conn) (int, error)` via `unix.GetsockoptUcred`; `peercred_other.go` returns `ErrStopEvidenceUnsupported`. |
| -Tests: `TestProbeRefusesUnheldRowOrForeignExec`; `TestProbeRendersBeforeStart` (the `RenderHeld` hook is called before any spawn, model 4); `TestProbeRefusesForeignPeer` (Review Focus 4: a fake listener answering 200 from this test process, whose pid is not in the host cgroup → `ErrProbeFailed` naming the peer); `TestProbeAttemptEndsAtTheWindow` (the fake start answers at 31 s → `ErrAttemptOver`); `TestProbeIsAColdAdmissionIntoThePreparedGeneration` (with Plan 6's fake writer: `Admit` is called once with `NewGeneration == op`, `Mode == "cold"`, `Coverage == "sealed"`, before the spawn hook runs; a second attempt admits a fresh generation, `KeepStamp` false). |
| +Tests: `TestProbeRefusesUnheldRowOrForeignExec`; `TestFirstProbeFromTheSourceGenerationIsAdmitted` (the claim as the flip left it: `holder.generation` is the source's `""`, `.files-gen` and `<dbPath>.gen` name the move id; the probe validates the prepared inputs, admits into the move id, and the start check after the CAS passes; a hand-built claim whose `holder.generation` is already the move id with a `.files-gen` naming another op refuses); `TestProbeRendersBeforeStart` (the `RenderHeld` hook is called before any spawn, model 4); `TestProbeRefusesForeignPeer` (Review Focus 4: a fake listener answering 200 from this test process, whose pid is not in the host cgroup → `ErrProbeFailed` naming the peer); `TestProbeAttemptEndsAtTheWindow` (the fake start answers at 31 s → `ErrAttemptOver`); `TestProbeIsAColdAdmissionIntoThePreparedGeneration` (with Plan 6's fake writer: `Admit` is called once with `NewGeneration == op`, `Mode == "cold"`, `Coverage == "sealed"`, before the spawn hook runs; a second attempt admits a fresh generation, `KeepStamp` false). |
| - [ ] **Steps 1 to 4**: run `nice -n 10 go test ./internal/activator -run TestProbe -count=1` → PASS. Cross-compile. |
| - [ ] **Step 5: Commit**: `git commit -m "activator: ProbeWake, a cold admission into the prepared generation with the attempt policy (§7.7, §7.14)" ...` |
| @@ -321,7 +328,7 @@ Tests: `TestProbeRefusesUnheldRowOrForeignExec`; `TestProbeRendersBeforeStart` ( |
| **Interfaces:** |
| - `moveCfg.Probe func(ctx, activator.ProbeConfig) (activator.ProbeResult, error)`; `moveCfg.RenderAndMaterialize func(ctx, rt registry.Route, op, version string) (appconfig.HeldRender, error)`. |
| -- `landingPass` phases: `restored → probing` (`MoveLandingPhase`); in `probing`, attempts from the journal (`Journal.Attempt`): `n = AttemptsFor("landing", targetReport.AttemptsAdmitted)`; for each attempt not yet recorded: `MoveProbeAttempt(g, i)`, journal `Attempt = i` and `Opened = now` before the call (a crash restarts only the probe, §7.13; the journalled attempt whose window passed counts as failed), `cfg.Probe(...)`; success → `MoveLandingPhase(probing → probed)`; `ErrAttemptOver`/`ErrProbeFailed` → next attempt, if any; attempts exhausted or `cut_deadline` passed → the rollback marker (Plan 4 Task 1's `MoveRollBack`; until it lands, `markForRollback`). |
| +- `landingPass` phases: `restored → probing` (`MoveLandingPhase`, then the claim's `SetMovePhase(id, KindLanding, "restored", "probing")`, Plan 2 Task 10's mirroring rule, so the admission's `ObservedPhase` is `probing`); in `probing`, attempts from the journal (`Journal.Attempt`): `n = AttemptsFor("landing", targetReport.AttemptsAdmitted)`; for each attempt not yet recorded: `MoveProbeAttempt(g, i)`, journal `Attempt = i` and `Opened = now` before the call (a crash restarts only the probe, §7.13; the journalled attempt whose window passed counts as failed), `cfg.Probe(...)`; success → `MoveLandingPhase(probing → probed)` then `SetMovePhase(id, KindLanding, "probing", "probed")`; `ErrAttemptOver`/`ErrProbeFailed` → next attempt, if any; attempts exhausted or `cut_deadline` passed → the rollback marker (Plan 4 Task 1's `MoveRollBack`; until it lands, `markForRollback`). |
| - Tests: `TestLandingProbesWithinTheBudget` (two admitted attempts: the first fails, the second passes → `probed`; a third is never attempted); `TestLandingProbeCrashRestartsOnlyTheProbe` (a fresh worker on a journal at `probing` with `Attempt = 1` does not rerun `local` or `restored`: their journal steps are untouched and no second `.files-gen` write happens); `TestLandingExhaustedAttemptsMarkRollback`. |
| - [ ] **Steps 1 to 5**. Commit: `git commit -m "host: the landing's probing and probed phases (§7.6, §7.7)" ...` |
| @@ -337,7 +344,7 @@ Tests: `TestProbeRefusesUnheldRowOrForeignExec`; `TestProbeRendersBeforeStart` ( |
| - The open: `cfg.ReleaseHold(host, id)` (`registry.ReleaseHold`); `changed == false` with the row already `ready` and `hold_op == ""` is a replay and fine; `MoveOpened(g, now)`; `nudgeSerial`; the activator's entry from the probe is the serving process: no restart. |
| - `registry.DeleteIfHold(host, op string) (bool, error)`: `DELETE FROM routes WHERE host = ? AND hold_op = ?` (the source's row deletion in Task 8, conditional on `hold_op = move id`, §7.9 step 3). |
| -Tests: `TestCommitThenOpenInOrder` (witness `wCommitBeforeOpen`: the row is `ready` only after the record says `committed` and the claim says `committed`); `TestOpenAfterCrashDoesNotReprobe` (Review Focus 3: a worker restarted on a record at `committed` with the claim still `landing/probed` writes the claim half, opens, and never calls `Probe`); `TestLateCommitRollsBack` (Review Focus 2). |
| +Tests: `TestCommitThenOpenInOrder` (witness `wCommitBeforeOpen`: driven from the flip through every landing phase with the real transitions, no fixture phase edits; the row is `ready` only after the record says `committed` and the claim says `committed`); `TestOpenAfterCrashDoesNotReprobe` (Review Focus 3: a worker restarted on a record at `committed` with the claim still `landing/probed` writes the claim half, opens, and never calls `Probe`); `TestLateCommitRollsBack` (Review Focus 2). |
| - [ ] **Steps 1 to 5**. Commit: `git commit -m "host: commit as a record CAS, the claim half, the conditional open (§7.5, §7.9 steps 1 and 2)" ...` |
diff --git a/docs/superpowers/plans/2026-10-06-moves-phase1-4-rollback-return-declare-lost.md b/docs/superpowers/plans/2026-10-06-moves-phase1-4-rollback-return-declare-lost.md| index 19270de60..c1794046a 100644 |
| --- a/docs/superpowers/plans/2026-10-06-moves-phase1-4-rollback-return-declare-lost.md |
| +++ b/docs/superpowers/plans/2026-10-06-moves-phase1-4-rollback-return-declare-lost.md |
| @@ -54,11 +54,12 @@ |
| **Interfaces:** |
| - `MoveRollBack(g MoveGuard, rbID, reason string, now time.Time) func(*Record) error`: from `landing` (any landing phase) → `rolling_back`; `Region = From`; `Rollback = &MoveRollback{ID, RequestedAt, Reason, TargetPhase: RbTargetReleasing, SourcePhase: RbSourceWaiting, ReleaseDeadline: now + 15 min}`; refused from `committed` (`ErrMoveConflict`): the commit won. |
| - `MoveRollbackTargetPhase(g, from, to string, stamp func(*MoveRollback))` over the target order `releasing, released, returning, returned, cleaning, cleaned` (a clean release skips `returning`/`returned`: `released → cleaning` is legal when `!Dirty`); `MoveRollbackSourcePhase(g, from, to, stamp)` over `waiting, claimed, restoring, verified | parking, building, published, local, configured, restored, probing, probed, recovered, opened` (clean: `restoring → verified → probing`; dirty: `restoring → parking → … → restored → probing`). |
| -- `MoveStuck(g, reason)`: from `rolling_back` (source `probing`) or `reverting` → `stuck`; `MoveRetry(g)`: `stuck → rolling_back` (source phase stays `probing`) or `→ reverting` (by `Rollback == nil`), clears `Reason`. |
| -- `MoveRolledBackClose(g)`: `rolling_back` with `RecoveredAt`, `OpenedAt`, `TargetParkedAt` set → `Move = nil` (region is `From` already). `MoveRevertClose(g)`: `reverting` with `rev_phase` opened (the caller checks the claim) → `Move = nil`. |
| +- `MoveStuck(g, reason)`: from `rolling_back` with the source at `probing` **or at `restoring`/`verified` (a verification mismatch, INV-4)**, or from `reverting` → `stuck`, recording `Rollback.StuckAt = SourcePhase`; `MoveRetry(g)`: `stuck → rolling_back` (the source phase returns to what `StuckAt` names: a mismatch retries verification, never bypasses it) or `→ reverting` (by `Rollback == nil`), clears `Reason`. |
| +- `MoveRollbackTargetLost(g, lossKey string, now)`: the declared-loss transition (§7.10): legal from target phase `releasing`, `returning`, or `released` with `Dirty` (the "released dirty, not returned" window); sets `TargetPhase = released`, `Dirty = false`, `Evidence = lossKey`, `LostAt = now`; the forward-only `MoveRollbackTargetPhase` never accepts `returning → released`, so the loss has its own closure. |
| +- `MoveRolledBackClose(g, now)`: `rolling_back` with `RecoveredAt`, `OpenedAt`, `TargetParkedAt` set → `Move = nil` (region is `From` already) and `LastMove = &MoveOutcome{ID, Outcome: "rolled_back", At: now, OfflineS: Rollback.OpenedAt - CutAt}` (index D11). `MoveRevertClose(g, now)`: `reverting` with `rev_phase` opened (the caller checks the claim) → `Move = nil`, `LastMove{Outcome: "rolled_back"}` with `OfflineS` from `CutAt` to the reopen. |
| - `MoveLanding` with `landing_deadline` passed is the console sweep's trigger (Plan 5) through `MoveRollBack`. |
| -- [ ] **Step 1: Write the failing test**: the commit/rollback race (`MoveCommit` then `MoveRollBack` → `ErrMoveConflict`; `MoveRollBack` then `MoveCommit` → `ErrMoveConflict`); `MoveRollBack` sets region `ie`, the deadline and both phases; target phases in order, the clean skip, a dirty attempt to skip `returning` refused; source phases clean and dirty orders; `MoveStuck`/`MoveRetry` round trip; `MoveRolledBackClose` refuses until all three stamps exist. |
| +- [ ] **Step 1: Write the failing test**: the commit/rollback race (`MoveCommit` then `MoveRollBack` → `ErrMoveConflict`; `MoveRollBack` then `MoveCommit` → `ErrMoveConflict`); `MoveRollBack` sets region `ie`, the deadline and both phases; target phases in order, the clean skip, a dirty attempt to skip `returning` refused; `MoveRollbackTargetLost` legal from `releasing`, `returning` and `released`+`Dirty`, refused from `cleaning` and from `released` clean; source phases clean and dirty orders; `MoveStuck` from `verified` (mismatch) and from `probing`, and `MoveRetry` returning the source to the phase it was stuck in; `MoveRolledBackClose` refuses until all three stamps exist and writes `LastMove`. |
| - [ ] **Steps 2 to 4**. **Step 5: Commit**: `git commit -m "provision: rollback, revert, stuck and retry transitions (§7.8, §7.10)" ...` |
| --- |
| @@ -70,12 +71,12 @@ |
| **Interfaces:** |
| - `RollBack(opID, rbID string) func(*Claim) error`: `KindLanding` (any phase but `committed`) → `KindRestoring`, phase `"releasing"`, `Intent["rollback"] = rbID`; exec unchanged (the target still owes evidence). |
| - `TargetReleased(opID, targetBox, targetRegion, evidenceKey string, scope ParkScope, newID func() string) func(*Claim) error`: `KindRestoring`, `Exec == targetBox` → `ExecToNone(... JournalEvidenceTarget)`, phase `"released"`, append `park-superseded` for `targetRegion` with `scope` (owner `region:<target>`) and `park-local` for `targetBox`, then `CoverRegionPredecessors(opID, "released", targetRegion)` (claim rev 6.14 item 6, R12-3). |
| -- `ReturnToSource(opID, sourceBox string, clean bool) func(*Claim) error` (§7.8 `claimed`, K4): `KindRestoring`, `Exec == ""`, `evidence_target` journalled, `sourceBox == participants[0]` → `Exec = sourceBox`, phase `"claimed"`, `holder.stamp = ParseUint(journal[source_stamp])`; when `clean`, `holder.generation = journal[src_gen]`; `holder.admissions` untouched (R11-4). |
| +- `ReturnToSource(opID, sourceBox string, clean bool) func(*Claim) error` (§7.8 `claimed`, K4): `KindRestoring`; `Exec == ""` **or `"none"`** (claim Plan 8's `declareLostTransition` writes the literal `"none"`, line 1511); either `evidence_target` journalled **or** a `loss.<id>` journal entry present (a declared-lost target gives no evidence; the declaration is the cited fence, §7.10); `sourceBox == participants[0]` (the lost target has been removed from `participants` by the declaration, so the source may be the only participant left) → `Exec = sourceBox`, phase `"claimed"`, `holder.stamp = ParseUint(journal[source_stamp])`; when `clean`, `holder.generation = journal[src_gen]`; `holder.admissions` untouched (R11-4). `TargetReleased` is not called for a declared-lost target: the console's resolve door queues the target region's `park-superseded` and runs `CoverRegionPredecessors` itself (Task 10), since no box can give evidence for the lost one. |
| - `ReleaseMove(opID, execBox string) func(*Claim) error` (K8): kind in `landing/committed`, `restoring/opened`, `reverting/opened` or `moving/aborting`; no obligation with `Op == opID` remains; sets `Op = Op{Kind: KindServing, Exec: execBox}` (participants cleared; journal dropped; the admissions stay in the holder). |
| - `SetRevPhase(opID, from, to string)` (`rev_phase` in the journal under `KindReverting`, order `evidence, verify, probing, probed, opened`). |
| - `QueueReturnLanePark(opID, sourceRegion string, scope ParkScope, newID)` (dirty: the lane `moves/<host>/<rb id>` in the source's region, queued at `probed`, §7.8). |
| -- [ ] **Step 1: Write the failing tests**: `TestRollBackRacesTheCommitOnTheClaim` (after `Commit`, `RollBack` is `ErrWrongState`; after `RollBack`, `Commit` is `ErrWrongState`); `TestTargetReleasedCoversTheTargetRegion` (an admission record with `PredecessorRegion == "uk"` becomes covered; the two obligations exist; exec is none); `TestReturnToSourceCleanNamesSrcGenAndKeepsAdmissions` (Review Focus 3: `holder.generation == src_gen`, `holder.stamp == source_stamp`, `holder.admissions` unchanged and its last record is the target's writer); `TestReturnToSourceDirtyLeavesGeneration`; `TestReleaseMoveWaitsForObligations` (with a pending obligation under the op → `ErrPendingChore`; without → serving with exec); `TestRevPhaseOrder`. |
| +- [ ] **Step 1: Write the failing tests**: `TestRollBackRacesTheCommitOnTheClaim` (after `Commit`, `RollBack` is `ErrWrongState`; after `RollBack`, `Commit` is `ErrWrongState`); `TestTargetReleasedCoversTheTargetRegion` (an admission record with `PredecessorRegion == "uk"` becomes covered; the two obligations exist; exec is none); `TestReturnToSourceCleanNamesSrcGenAndKeepsAdmissions` (Review Focus 3: `holder.generation == src_gen`, `holder.stamp == source_stamp`, `holder.admissions` unchanged and its last record is the target's writer); `TestReturnToSourceDirtyLeavesGeneration`; `TestReturnToSourceAfterADeclaredLoss` (the claim as Plan 8's transition leaves it: `Exec == "none"`, `participants == {source}`, journal `loss.<id>`, no `evidence_target` → accepted; the same without the loss entry → `ErrWrongState`); `TestReleaseMoveWaitsForObligations` (with a pending obligation under the op → `ErrPendingChore`; without → serving with exec); `TestRevPhaseOrder`. |
| - [ ] **Steps 2 to 4**. **Step 5: Commit**: `git commit -m "hostclaim: RollBack, TargetReleased, ReturnToSource, ReleaseMove, rev_phase (§7.8, K4, K8)" ...` |
| --- |
| @@ -121,7 +122,7 @@ Tests (harness with a target-side cfg, a `landing/probing` record and claim): `T |
| - `waiting`: until the target phase is `released` (clean) or `returned` (dirty) and the claim's exec is `""`. |
| - `claimed`: `hostclaim.Update(ReturnToSource(id, cfg.Box, clean))`; `cfg.Guard.Activate(ctx, host, cfg.Box, nil)` (the fresh read finds `holder.stamp == S`, matching `active_stamp`, or re-initialises a lost record to `S`); `MoveRollbackSourcePhase(waiting → claimed)`. |
| - `restoring`: `SetHold(host, "restoring", rbID)`. |
| - - clean `verified`: files plain and timers digest equal `Source.Files.Plain` and `Source.TimersSHA256`, the warm database's `<dbPath>.gen` names the pre-move generation (`src_gen`) and location; a mismatch → `MoveStuck(g, "recovery baseline mismatch")` (INV-4). |
| + - clean `verified`: files plain and timers digest equal `Source.Files.Plain` and `Source.TimersSHA256`, the warm database's `<dbPath>.gen` names the pre-move generation (`src_gen`) and location; a mismatch → `MoveStuck(g, "recovery baseline mismatch")` from `restoring` (legal there, Task 1; INV-4), and `--retry` reruns the verification, never skips it. |
| - dirty: `installGeneration` with `installSource{Lane: ReturnLane(host, rbID) in own bucket, Seal: return_seal, GenerationOp: rbID, Destination: rbID, PinInputs: nil, Phase: MoveRollbackSourcePhase}`. |
| - `probing` (Task 9; needs claim Plan 6): `ProbeWake` kind `recovery`, nine attempts, each a cold admission (dirty: into the return generation, `Coverage: "sealed"`; clean: warm or cold into the source's own generation, `Coverage: "clean-return"`), `recovery_inputs` recorded; a restarted probe restarts only the probe; nine failures → `MoveStuck(g, reason)`. |
| - `probed`: `MoveRollbackSourcePhase(probing → probed)`; dirty: `QueueReturnLanePark(id, cfg.Region, scope{Location: ScopeLocation(host, rbID), Incarnation, EvidenceAt: returnSeal.Bound(), RequestLimit: 120 s}, newID)`. |
| @@ -140,9 +141,9 @@ Tests: `TestSourceWaitsForTargetEvidence`; `TestCleanReturnVerifiesAndOpens` (no |
| **Files:** create `internal/host/movesrevert.go`, `internal/host/movesrevert_test.go`; modify `internal/host/movespass.go` (`reverting` records whose `From == cfg.Region`). |
| **Interfaces:** |
| -- `revertSource(ctx, s, w, rec)` (§7.4 reverting): claim `hostclaim.Revert(id, cfg.Box)` once exec is none (from `stopping` the evidence was never written: `rev_phase = evidence` computes it, writes `stopped-source.json`, `ExecToNone`, then `Revert`); `SetHold(host, "restoring", id)`; `rev_phase` from the claim journal: `evidence` (compute, start nothing until it holds), `verify` (from `stopped`, `staged`, `archived`: files and timers digests equal `Source`'s; a mismatch is `stuck`), `probing` (Task 9: `ProbeWake` kind `recovery`, warm or cold into the source's own generation; `SetRevPhase(verify → probing)` written before the first attempt; never re-runs `verify`), `probed` (`ReleaseHold(host, id)` conditional), `opened` (wait for the staging obligation of §7.11, Plan 5, then `ReleaseMove(id, cfg.Box)` and `MoveRevertClose`). Fifteen minutes of failed probes: `MoveStuck`. |
| +- `revertSource(ctx, s, w, rec)` (§7.4 reverting): claim `hostclaim.Revert(id, cfg.Box, rec.Move.To, &stagingScope, newID)` once exec is none, where `stagingScope = {Location: ScopeLocation(host, id), Incarnation, EvidenceAt: <the lane's last LeaseUntil from the receipts ledger>, RequestLimit: 120 s}` (from `stopping` the evidence was never written: `rev_phase = evidence` computes it, writes `stopped-source.json` once it is complete, `ExecToNone`, then `Revert`); `SetHold(host, "restoring", id)`; `rev_phase` from the claim journal: `evidence` (compute, start nothing until it holds), `verify` (from `stopped`, `staged`, `archived`: files and timers digests equal `Source`'s; a mismatch is `stuck`, and `--retry` reruns the verification), `probing` (Task 9: `ProbeWake` kind `recovery`, warm or cold into the source's own generation; `SetRevPhase(verify → probing)` written before the first attempt; never re-runs `verify`), `probed` (`ReleaseHold(host, id)` conditional), `opened` (wait for the `park-staging` obligation `Revert` queued, which the target region's box runs and acknowledges through its regional runner and the target-side pass dispatches exactly as Plan 5 Task 1 dispatches an abort's: a record in `reverting` whose `To == cfg.Region` runs `cfg.Park.Run` on it; then `ReleaseMove(id, cfg.Box)` and `MoveRevertClose(g, now)`). Fifteen minutes of failed probes: `MoveStuck`. |
| -Tests: `TestRevertFromStoppingComputesEvidenceFirst`; `TestRevertVerifyThenProbeNeverVerifiesAgain` (witness `wRollbackJournal` for the revert); `TestRevertOpensAndReleases`. |
| +Tests: `TestRevertFromStoppingComputesEvidenceFirst`; `TestRevertVerifyThenProbeNeverVerifiesAgain` (witness `wRollbackJournal` for the revert); `TestRevertOpensAndReleases` (the target-side pass archives the lane under `superseded/<mv>/<host>/…/moves/`, the obligation acknowledges after the horizon, and only then the claim is `serving` with exec = source and `LastMove.Outcome == "rolled_back"`). |
| - [ ] **Steps 1 to 5**. Commit: `git commit -m "host: reverting, the pre-flip recovery, journaled in rev_phase (§7.4)" ...` |
| @@ -188,11 +189,17 @@ Tests: `TestRevertFromStoppingComputesEvidenceFirst`; `TestRevertVerifyThenProbe |
| **Interfaces:** |
| - `moves.LossOwner{Record provision.Record}` implementing `hostclaim.LossOperationOwner.ValidateDeclareLost(c hostclaim.Claim, in hostclaim.LostDeclaration) error` with §7.10's table: legal only when `in.Box` is exactly one of `c.Op.Participants` and is **not** `participants[0]` (the source); the op is a move kind; the record's `State` is not `committed` (the record's commit decides, whatever the claim says); and one of: (a) the target owes stop evidence (`KindLanding`, or `KindRestoring` with `rollback.target_phase == releasing`); (b) the target released dirty and has not returned (`KindRestoring`, `target_phase == returning`); (c) the box owes only `park-local` obligations (legal after commit too: `State == committed` is allowed for this case alone). Everything else `ErrLossRefused` with the sentence. |
| - `moves.LossOwner.ValidateLossRecovery(c, loss hostclaim.LossRecord, in hostclaim.Admission) error` (`hostclaim.LossRecoveryOwner`): the admission is the source's, `in.Box == participants[0]`, `c.Op.Kind == KindRestoring`, phase `claimed`, and `in.AdditionalWritesLost` names **every** session the target was admitted into since the cut (every `holder.admissions` entry with `Box == lost box` and `Session > cut_session`) plus, when the seal's `src_covered` is false, the source's cut session `{Box: source, Generation: src_gen, Session: cut_session, RemoteMaxTXID: seal.N}`; unique by `(box, session)` (witnesses `wTwoSessionDeclaration`, `wNoDuplicateDeclarationEarly`, `wNoDuplicateDeclarationLate`; the legacy identity `{session: 0, generation: "" read as @root, box: holder.writer_box}` is valid, R16-3). |
| -- The console door: `moveOperatorGate`; body validated (`confirm` must equal the host, M-C18); `cloud` is the CLI's attestation `{instance_id, state ("stopped" | "terminated"), observed_at, region}` recorded into Plan 8's `LossRecord` as `LossCloudObservation` with `LossOperatorAttestation{Address: addr}`; `hostclaim.CreateLossRecord(ctx, s.cfg.Releases, signer, record)` (the console's root signer, as Plan 8 defines), then `hostclaim.DeclareLost(ctx, s.cfg.Releases, pub, obs, lossID, moves.LossOwner{Record: *rec})`; then the record: case (a) or (b) → `MoveRollBack` if still `landing`, then `MoveRollbackTargetPhase(… → released, Dirty: false, Evidence: <loss record key>)` (the rollback finishes as a clean one, §7.10) **and**, for case (b), `QueueReturnLanePark` for `moves/<host>/<rb>` in the source's region with `evidence_at = fenced_at` (model 27); the registered timers of the lost target are declared in the loss record and alerted (Plan 6's alert hook); the box goes on `Lost`. |
| -- The source's recovery with a lost target: `claimed` through `hostclaim.AdmitWriterWithLoss(ctx, s, pub, obs, admission, lossID, validated, moves.LossOwner{...})` at the probe instead of `Writer.Admit` (the moves pass passes `lossID` through `ProbeConfig.LossID`), with `validated` built from the admissions table as above. |
| +- The console door, in this order (the record decides, the claim follows, §7.5; a commit that wins between the two leaves the claim untouched): |
| + 1. `moveOperatorGate`; body `{"declare_lost": box, "confirm": host, "cloud": {"instance_id", "state", "observed_at", "region", "account", "request_id"}}` validated (`confirm` must equal the host, M-C18; `state` is `stopped` or `terminated`; every cloud field present). |
| + 2. **The record first**: if `landing`, `provision.SetRecordMove(MoveRollBack(g, NewRollbackID, "declared lost: "+box, now))`; `ErrMoveConflict` because the record is `committed` → 409 "this move has committed. the target is authoritative and cannot be declared lost" and nothing else happens. Then `MoveRollbackTargetLost(g, lossKey, now)` (Task 1) when the target phase is `releasing`, `returning` or `released`+`Dirty`; a target already `released` clean or past `cleaning` → 409 (nothing to declare). Case (c), a box owing only `park-local` chores, skips both record writes. |
| + 3. **The claim**: `hostclaim.Read` fresh; if `Op.Kind == KindLanding` the claim half of the rollback is still owed → `hostclaim.Update(RollBack(id, rbID))` first. Build Plan 8's `LossRecord` exactly: `{V: 1, ID: lossID, State: "prepared", Actor: addr, AcceptedAt: now, Host, HolderIncarnation, HolderStamp, HolderSession, ExpectedClaimGen: obs.Claim.Gen, ExpectedClaimETag: obs.ETag, OpID, OpKind, OpPhase: as observed, Box, Enrollment: <the box's enrollment id from the actor directory, Plan 8b; until then the box label, which Plan 8's validation accepts as a non-empty string>, QuarantineRevision: <previous LostRevisions[box] + 1>, CloudObservation: LossCloudObservation{ObservedAt, Account, Region, InstanceID, State, RequestID} from the body, OperatorAttestation: LossOperatorAttestation{MaintainComputeFence: true, ColdQuarantineStartOnly: true}, FencedAt: cloud.ObservedAt, ConfirmedConsequences: LossConsequences{RevokeExecution: obs.Claim.Op.Exec == box, RemoveParticipation: true, DischargeLocal: <every park-local of the box>, ReassignRegional: <every park-replica/park-generation owned by box:<box>>, WritesPotentiallyLost: true, TimersPotentiallyLost: true, RecoveryIntent: json({"kind":"move-rollback","rollback":rbID,"clean":true})}, Timers: TimerLoss{Knowledge: TimersUnknown}}` (`TimersPartial` with the staged timers snapshot as `Source` when the cut's `timers/` object exists: the registered timers the target held are declared, §7.10). `hostclaim.CreateLossRecord(ctx, s.cfg.Releases, signer, record)` with the console's root `release.Signer`, then `hostclaim.DeclareLost(ctx, s.cfg.Releases, pub, obs, lossID, moves.LossOwner{Record: *rec})`. |
| + 4. In one more claim CAS: `QueueSupersededPark(id, targetRegion, "", scope{Location: TrimSuffix(holder.location, "/"), Incarnation, EvidenceAt: FencedAt, RequestLimit: 120 s}, newID)` and `CoverRegionPredecessors(id, obs.Claim.Op.Phase, targetRegion)` (what `TargetReleased` does for a live target), and for case (b) `QueueReturnLanePark(id, sourceRegion, scope{Location: ScopeLocation(host, rbID), Incarnation, EvidenceAt: FencedAt, RequestLimit: 120 s}, newID)` (model 27). |
| + 5. The declared timers are alerted through `host.PageOperator` (Plan 5 Task 0); the box is on `claim.Lost`. |
| + Until claim Plan 8b lands (index), the door answers 409 "declare-lost is not enabled in this deployment yet" unless `CARLOS_MOVES_DECLARE_LOST=1` is set on the console, which only a test sets. |
| +- The source's recovery with a lost target: `waiting` ends when the claim carries the `loss.<id>` entry and `Exec == "none"` (no `released`/`returned` target phase will come; the door wrote `MoveRollbackTargetLost`); `claimed` is `ReturnToSource(id, cfg.Box, clean = true)` (Task 2's loss form); the probe admits through `hostclaim.AdmitWriterWithLoss(ctx, s, pub, obs, admission, lossID, validated, moves.LossOwner{...})` instead of `Writer.Admit` (the moves pass passes `lossID` through `ProbeConfig.LossID`), with `validated` built from the admissions table as above (`ValidatedWriteSource{Reason: "target" | "source-cut", Box, Generation, Session, Region, Baseline, RemoteMaxTXID}`); `TestLostTargetRecoveryRunsAgainstARealDeclaration` drives the source pass from the claim Plan 8's `DeclareLost` actually produced in the previous test, not a hand-written claim. |
| - A returning box (`Lost` contains it) refuses every move step and runs `Local.RetireInstallation` for the host under the guard before anything else (claim §7.2 quarantine; Plan 8's `hostquarantine` when it lands, else the `park.Local` call directly). |
| -Tests: `TestDeclareLostLegalityTable` (Review Focus 5: a table over (kind, phase, record state, box) with the expected error; the refused cases leave the claim unchanged); `TestDeclareLostRefusesWithoutACloudObservation` (witness `wDeclareLostCompute`: a body whose `cloud.state` is not `stopped` or `terminated`, or absent, is refused at the door; an expired lease is never an argument); `TestReturningLostBoxQuarantinesFirst` (witness `wLostQuarantine`: a box in `claim.Lost` that comes back retires the host's local artifacts before any move step runs and refuses the step); `TestLostTargetReturnDeclaresEverySessionOnce` (witnesses `wTwoSessionDeclaration`, `wNoDuplicateDeclarationEarly/Late`: a target with a probe session that closed cleanly and a warm continuation that died → two `additional_writes_lost`; a target destroyed before its first admission with `src_covered` false → the cut session once, as the predecessor itself, never twice); `TestDeclareLostQueuesTheReturnLanePark` (model 27, witness `wLostReturnPark`: the lane's scope has `evidence_at = fenced_at`); `TestResolveDeclareLostRefusesWithoutConfirm`. |
| +Tests: `TestDeclareLostLegalityTable` (Review Focus 5: a table over (kind, phase, record state, box) with the expected error; the refused cases leave the claim unchanged); `TestDeclareLostLosesToAWinningCommit` (the target commits the record between the door's gate and its record write: the door answers 409, writes no loss record and no claim change, and the target opens); `TestDeclareLostCreatesARealSignedLossRecord` (through Plan 8's `CreateLossRecord` and `ReadLossRecord` with a test root signer, every field above present, `DeclareLost` accepts it, the claim's `Lost` names the box, `Exec == "none"`, the target is out of `participants`, the `loss.<id>` journal entry exists); `TestDeclareLostRefusesWithoutACloudObservation` (witness `wDeclareLostCompute`: a body whose `cloud.state` is not `stopped` or `terminated`, or absent, is refused at the door; an expired lease is never an argument); `TestReturningLostBoxQuarantinesFirst` (witness `wLostQuarantine`: a box in `claim.Lost` that comes back retires the host's local artifacts before any move step runs and refuses the step); `TestLostTargetReturnDeclaresEverySessionOnce` (witnesses `wTwoSessionDeclaration`, `wNoDuplicateDeclarationEarly/Late`: a target with a probe session that closed cleanly and a warm continuation that died → two `additional_writes_lost`; a target destroyed before its first admission with `src_covered` false → the cut session once, as the predecessor itself, never twice); `TestDeclareLostQueuesTheReturnLanePark` (model 27, witness `wLostReturnPark`: the lane's scope has `evidence_at = fenced_at`); `TestResolveDeclareLostRefusesWithoutConfirm`. |
| - [ ] **Steps 1 to 5**. Commit: `git commit -m "moves: declare-lost with compute fencing only; the moves LossOperationOwner and the lost-target return (§7.10, K11)" ...` |
diff --git a/docs/superpowers/plans/2026-10-06-moves-phase1-5-cleanup-and-parks.md b/docs/superpowers/plans/2026-10-06-moves-phase1-5-cleanup-and-parks.md| index e2b571166..5f53af638 100644 |
| --- a/docs/superpowers/plans/2026-10-06-moves-phase1-5-cleanup-and-parks.md |
| +++ b/docs/superpowers/plans/2026-10-06-moves-phase1-5-cleanup-and-parks.md |
| @@ -45,6 +45,16 @@ |
| --- |
| +### Task 0: The paging primitive |
| + |
| +**Files:** modify `internal/host/alertsink.go`; create `internal/host/alertsink_page_test.go`. |
| + |
| +**Interfaces:** `(a *alertSinks) PageOperator(ctx, level, subject, body string) error` delivering through the same webhook and SNS topic as `deliver` (`alertsink.go:99`), and the package function `host.PageOperator(ctx, level, subject, body string) error` on the armed sinks (a no-op with a log line when none are configured, as today's alert delivery does). Moved here from Plan 6 (index D12) because Task 7's console pass and Plan 6's paging rules both need it, and Plan 6 depends on this plan. |
| + |
| +- [ ] Test `TestPageOperatorUsesTheArmedSinks` (a fake webhook poster records the subject and body; with no sinks the call returns nil and logs once); implement; commit: `git commit -m "host: PageOperator, the paging primitive for moves (§13)" ...` |
| + |
| +--- |
| + |
| ### Task 1: Abort with the target-owned staging park |
| **Files:** modify `internal/hostclaim/move.go` (`Abort`), `internal/host/movespass.go` (the `MoveAborting` arm), create `internal/host/movesabort.go`, tests. |
| @@ -52,7 +62,7 @@ |
| **Interfaces:** |
| - `hostclaim.Abort(opID, targetRegion string, scope ParkScope, newID func() string) func(*Claim) error`: `KindMoving` with phase `""` (before the seal; a cut past `requested` cannot abort) → phase `"aborting"`, appends `park-staging` for `targetRegion` with `scope` (owner `region:<target>`); idempotent. |
| - The source, on `aborting`: `hostclaim.Update(Abort(id, to, scope{Location: ScopeLocation(host, id), Incarnation, EvidenceAt: <the last round's lease SafeUntil the receipts recorded, i.e. the staging writers' horizon start>, RequestLimit: 120 s}, newID))`; delete its own `move-snapshots/<host>/<id>/` objects (own bucket; by version through `DeleteVersion`) and `move-receipts/<host>/<id>*.json`; then wait: `MoveAbortComplete` only when no obligation under the op remains and the claim is released (`ReleaseMove(id, cfg.Box)` first; the record clears second, §7.5 "the close is the other way round"). |
| -- The target region's box, for a record in `aborting` whose `To == cfg.Region`: `cfg.Park.Run` on the `park-staging` obligation (copies and deletes at once; acknowledgement after the horizon through the regional runner). |
| +- The target region's box, for a record in `aborting` **or `reverting`** whose `To == cfg.Region`: `cfg.Park.Run` on the op's `park-staging` obligation (copies and deletes at once; acknowledgement after the horizon through the regional runner). The reverting case's obligation is the one `hostclaim.Revert` queued (Plan 2 Task 2); Plan 4 Task 6's `opened` waits on it. |
| - The staging horizon start for the abort scope: the receipts ledger records `LeaseUntil` of the last round (Plan 1 Task 14's `moveLedger`); `evidence_at = LeaseUntil` (the time after which no staging writer that held the lease can still write; the same bound the seal records for a cut). |
| - Auto-abort: `preparing` with no promotable round for 60 minutes pages (§13) and, at 6 hours (`moves.AbortAfter`), the console sweep (Task 7) CASes `MoveAbort`. |
| - The record's `MoveAbort` stays Plan 1 Task 2's; `MoveAbortComplete` gains the guard check through `rec.Move.Legacy()` (a legacy record completes as before, with the Plan 1 `DeletePrefix` path kept **only** for legacy records; the per-op path never deletes from the source). |
| @@ -81,7 +91,7 @@ Tests: `TestAbortParksFromTheTargetAndWaits` (Review Focus 2: the source's pass |
| **Interfaces:** |
| - For a `committed` record whose `To == cfg.Region`: run the `park-staging` obligation (`cfg.Park.Run`; `ErrPending` until the horizon; the regional runner acknowledges). Its scope is the one `Commit` queued (`evidence_at = seal.Bound()`, Plan 3 Task 7). |
| -- The close (§7.9 step 5): any participant that sees `OpenedAt`, `SourceParkedAt` set and no obligation under the op: the terminal log event `done` (Plan 6's `moves.LogEvent`; until then a `log.Printf`), `hostclaim.Update(ReleaseMove(id, targetBox))`, `provision.SetRecordMove(MoveClose(g))`. The console pass also tries (Task 7). |
| +- The close (§7.9 step 5): any participant that sees `OpenedAt`, `SourceParkedAt` set and no obligation under the op: the terminal log event `done` (Plan 6's `moves.LogEvent`; until then a `log.Printf`), `hostclaim.Update(ReleaseMove(id, targetBox))`, `provision.SetRecordMove(MoveClose(g, now))` (which writes `LastMove{Outcome: "done"}`, index D11). The console pass also tries (Task 7). |
| - `park.Engine`'s `Now` is the real clock; the test drives `Now` through the harness and a fake `Leases`. |
| Tests: `TestStagingParkAcknowledgesAtSealBoundPlusH` (Review Focus 1, witness `wStagingScopeDeadline`: seal bound `T`; `Run` at `T + 16 min` is `ErrPending`; a new worker (restart) at `T + 17 min + 1 s` acknowledges; the scope's `EvidenceAt` is still `T`); `TestCloseWaitsForBothObligations` (with the source's `park-superseded` still pending → no release; after both acknowledged → claim `serving` with `exec = uk-box`, `Move == nil`, region `uk`); `TestCloseIsRecordLast` (the claim is released before the record clears; a crash between leaves a finished `move` that admits nothing, and the next pass clears it). |
| @@ -130,14 +140,15 @@ Tests: `TestSweepSkipsWhileAMoveStages` (Review Focus 3); `TestFreeHostSweepIsHo |
| **Files:** create `internal/console/movesrecovery.go`, `internal/console/movesrecovery_test.go`; modify `internal/console/server.go` (a ticker every 60 s, after startup; `s.cfg.MovesPassDisabled` for tests). |
| **Interfaces:** |
| -- `movesRecoveryPass(ctx, s release.Store, now time.Time, alert func(level, subject, body string)) error` (D8), over `provision.ListAllRecords` with `Move != nil`, each with `hostclaim.Read`: |
| +- `movesRecoveryPass(ctx, s release.Store, now time.Time, alert func(level, subject, body string)) error` (D8). Two inputs, joined by host: `provision.ListAllRecords` (records with `Move != nil`) **and** claim discovery, `s.List(ctx, "control/hosts/")` (`hostclaim.Key`'s prefix) read with `hostclaim.Read` for every claim whose `Op.Kind` is a move kind; a claim with a move kind whose record has no move, or no record at all, is an orphan (the request door failed after its claim CAS, or a record was deleted under a finished move). For each record with a move, `hostclaim.Read`: |
| - the claim half the record decided (§7.5): record `landing` + claim `moving/sealed` with exec none → `Flip` (needs `source_stamp` and `src_gen`: the flip half is written by whoever sees it; the console computes `sourceStamp` from `holder.stamp` as observed, the pre-flip stamp, and `srcGen` from `holder.generation`); record `committed` + claim `landing` not `committed` → `Commit` with the scopes from the seal (`seal.Bound()`) and the evidence (`stopped-source.json`'s `CheckedAt`); record `rolling_back` + claim `landing` → `RollBack(id, rollback.id)`; anything else untouched (Review Focus 5). |
| - - a `moving` claim with no record move (Plan 1 Task 4's door failed after the claim CAS) older than 10 minutes → `ReleaseMove(id, exec)` with a log line. |
| + - an orphan `moving/""` claim (from the discovery above) older than 10 minutes by `Op.StartedAt` → `hostclaim.Update(ReleaseOrphanMove(id))`, a guarded transition added to `hostclaim/move.go` in this task: legal only for `KindMoving` with phase `""`, `Exec == participants[0]`, no obligation under the op, and the caller's assertion that the record carries no move for this id (the pass re-reads the record inside the retry loop; `ReleaseMove` itself rejects phase `""` on purpose, so the orphan case has its own door); `KindLanding`/`KindRestoring`/`KindReverting` orphans are never released automatically: they page (an operation past the flip with no record is an incident). |
| - deadline sweeps: `landing` past `landing_deadline` → `MoveRollBack(g, NewRollbackID, "landing deadline", now)` (the box writes the claim half or this pass does on its next tick); `rolling_back` past `release_deadline` with no target evidence → `alert("incident", ...)` once per hour (M-C10's sentence for the operator view); `preparing` with `UpdatedAt` older than 60 minutes → `alert("warning", ...)`; older than `moves.AbortAfter` (6 h) → `MoveAbort()`. |
| - the finalization halves: `committed` with both stamps and no obligations → `ReleaseMove` then `MoveClose`; `rolling_back` with the three stamps and no obligations → `ReleaseMove(source)` then `MoveRolledBackClose`; `aborting` with no obligations → `ReleaseMove(source)` then `MoveAbortComplete`. |
| -- `Server.startMovesPass(ctx)`: the ticker, logging each pass's errors; `alert` is `host.PageOperator` (Plan 6 Task 5), which the console binary links already (`internal/host` is in the CLI); a plain log line in its place is not acceptable, so build Plan 6 Task 5 before this task. |
| +- `Server.startMovesPass(ctx)`: the ticker, logging each pass's errors; `alert` is `host.PageOperator` (Task 0), which the console binary links already (`internal/host` is in the CLI). |
| +- The closes write `MoveClose(g, now)`, `MoveRolledBackClose(g, now)` and `MoveAbortComplete` with `LastMove` (index D11). |
| -Tests: `TestRecoveryPassWritesOnlyDecidedHalves` (Review Focus 5); `TestRecoveryPassRollsBackAtTheLandingDeadline`; `TestRecoveryPassReleasesAnOrphanMovingClaim`; `TestRecoveryPassFinishesTheClose`. |
| +Tests: `TestRecoveryPassWritesOnlyDecidedHalves` (Review Focus 5); `TestRecoveryPassRollsBackAtTheLandingDeadline`; `TestRecoveryPassReleasesAnOrphanMovingClaim` (the whole pass after a request-door crash: a `moving` claim at `control/hosts/<host>.json`, a record with no move; after 10 minutes the claim is `serving` with exec = source; a `landing` orphan is paged and left alone); `TestRecoveryPassFinishesTheClose`. |
| - [ ] **Steps 1 to 5**. Commit: `git commit -m "console: the move pass writes decided claim halves, sweeps deadlines and finishes closes (§7.5, §13, D8)" ...` |
diff --git a/docs/superpowers/plans/2026-10-06-moves-phase1-6-cli-status-observability.md b/docs/superpowers/plans/2026-10-06-moves-phase1-6-cli-status-observability.md| index 21ed7d60a..8b16a6ca1 100644 |
| --- a/docs/superpowers/plans/2026-10-06-moves-phase1-6-cli-status-observability.md |
| +++ b/docs/superpowers/plans/2026-10-06-moves-phase1-6-cli-status-observability.md |
| @@ -37,22 +37,21 @@ |
| | `docs/site/cli.md` | `### carlos move` | |
| | `internal/console/apiinstancemove.go` | `detail=1`, `resolve` | |
| | `internal/moves/log.go` | `LogEvent` | |
| -| `internal/host/movesevents.go` | events and paging from the box | |
| -| `internal/host/alertsink.go` | `PageOperator` exported | |
| -| `approved_copy_test.go`, `docs/superpowers/plans/2026-10-06-moves-phase1-copy.md` | the copy gate | |
| +| `internal/host/movesevents.go`, `internal/host/movespaging.go` | events and paging rules from the box (the primitive is Plan 5 Task 0) | |
| +| `approved_copy_test.go`, `docs/superpowers/plans/2026-10-06-moves-phase1-copy.md` | this plan's rows in the copy record (the gate is Plan 1 Task 15) | |
| --- |
| -### Task 1: `carlos move` |
| +### Task 1: `carlos move` and its CLI page heading |
| -**Files:** create `move.go`, `move_test.go`; modify `main.go` (dispatch and usage). |
| +**Files:** create `move.go`, `move_test.go`; modify `main.go` (dispatch and usage) **and `docs/site/cli.md`** in the same task and commit: `internal/docsite`'s `TestCLICommandsAreDocumented` (`cli_test.go:58`) fails any commit that dispatches a verb the CLI page does not head (index D12). The section goes under "Your deployment" after `### carlos system`: one paragraph (operator-only; what a move does; that it rolls back on its own; the rehearsal order is Paul's), the five command lines of §12 in a `sh` fence, the exit codes, and `carlos move resolve --host <h> --declare-lost <box>` with the confirmation. `nice -n 10 go test ./internal/docsite -count=1` is part of this task's Step 4. |
| **Interfaces:** |
| - `runMove(args []string)`: `carlos move --host <h> --to <region> [--manual-cut] [--no-wait] [--app <a>] [--console <url>]`; sub-verbs `status`, `cut`, `abort`, `resolve --retry | --declare-lost <box> [--instance-id <id>] [--region <aws region>]`; `--host` resolves the account and app through `resolvedApp` when `--app` is absent (the host's record is found through `GET /api/cli/apps/{acct}/{app}/instances/{host}` under the chosen app; without `--app`, the CLI lists the account's apps and finds the one whose instance domain the host ends in, as `instances_cli.go`'s host resolution does if it exists; else `--app` is required with a sentence saying so). |
| -- `moveWait(ctx, sess *apiSession, path string, out io.Writer, poll time.Duration) int` polls `GET <path>?detail=1` every `poll` (2 s), prints a sentence on each state change (M-C1 at start, M-C2 while preparing, M-C3 when a target report appears, M-C4 per failing check, M-C5 at `cutting/admitted` with the budget, M-C6 at `landing/probing`, M-C7 at the open with `opened_at - cut_at`, M-C8 at `rolling_back`, M-C9 at `rolled_back` clean, M-C10 while the target owes evidence past the release deadline, M-C11 at `stuck`, M-C12 at `ready` with `cut == manual`, M-C13 for an abort refused after the cut) and returns the exit code of the terminal state it last saw (Review Focus 1). |
| +- `moveWait(ctx, sess *apiSession, path, moveID string, out io.Writer, poll time.Duration) int` polls `GET <path>?detail=1` every `poll` (2 s), prints a sentence on each state change (M-C1 at start, M-C2 while preparing, M-C3 when a target report appears, M-C4 per failing check, M-C5 at `cutting/admitted` with the budget, M-C6 at `landing/probing`, M-C7 at the open with `opened_at - cut_at`, M-C8 at `rolling_back`, M-C9 at `rolled_back` clean, M-C10 while the target owes evidence past the release deadline, M-C11 at `stuck`, M-C12 at `ready` with `cut == manual`, M-C13 for an abort refused after the cut). The terminal result is **never inferred from the last state seen**: when the answer is `state: none`, the CLI reads `last` (`Record.LastMove`, index D11) and requires `last.id == moveID`; `done` → 0 with M-C7 (`offline_s` from the outcome), `rolled_back` → 3 with M-C9 (or M-C8 and M-C9 when it never saw `rolling_back`), `aborted` → 3; `stuck` and `rolling_back` past the release deadline → 4; a `none` whose `last.id` is another move, or no `last`, is "this move's outcome is not recorded" and exits 4 (Review Focus 1). A disconnected CLI resumed with `carlos move status --host <h> --wait` reads the same outcome. |
| - `--declare-lost`: before calling the door, the CLI verifies the box through the operator's own cloud credentials: `ec2.DescribeInstances` for the instance id the operator passes (or the box's instance id from `GET /api/cli/status/instances` if that door lists it); the state must be `stopped` or `terminated` (§7.10, model 5); it prints M-C18 and reads the host name from stdin as the confirmation; then `POST …/move/resolve {"declare_lost": box, "confirm": host, "cloud": {"instance_id", "state", "observed_at", "region"}}`. Without AWS credentials the verb refuses: "declare-lost needs your AWS credentials to confirm the box is stopped or terminated". |
| -Tests: `TestMoveWaitExitCodes` (a fake console serving a scripted sequence of detail answers: preparing → ready → cutting → landing → rolling_back → none; the CLI exits 3 and printed M-C8 and M-C9; the done sequence exits 0 with M-C7; stuck exits 4; a 409 at the request exits 5); `TestMoveManualCutPrintsM12AndStops`; `TestMoveDeclareLostNeedsStoppedOrTerminated` (a fake EC2 answering `running` → refused before any console call). |
| +Tests: `TestMoveWaitExitCodes` (a fake console serving a scripted sequence of detail answers: preparing → ready → cutting → landing → rolling_back → none with `last{id, rolled_back}`; the CLI exits 3 and printed M-C8 and M-C9; the done sequence exits 0 with M-C7; stuck exits 4; a 409 at the request exits 5); `TestMoveWaitSurvivesMissedStates` (Review Focus 1: the sequence preparing → none with `last{id, rolled_back}`, every intermediate state skipped: exit 3 with M-C8 and M-C9 printed from the outcome; none with `last.id` of another move → exit 4 with the "not recorded" sentence); `TestMoveManualCutPrintsM12AndStops`; `TestMoveDeclareLostNeedsStoppedOrTerminated` (witness `wDeclareLostCompute`: a fake EC2 answering `running` → refused before any console call). |
| - [ ] **Steps 1 to 5**. Commit: `git commit -m "carlos move: request, status, cut, abort, resolve; --wait and the exit codes (§12)" ...` |
| @@ -63,7 +62,7 @@ Tests: `TestMoveWaitExitCodes` (a fake console serving a scripted sequence of de |
| **Files:** modify `internal/console/apiinstancemove.go`, `internal/console/apiinstancemove_test.go`. |
| **Interfaces:** |
| -- `moveDetail{moveView; Move *provision.MoveState; Claim struct{Kind, Phase, Exec string; Participants []string}; Target *moves.TargetReport; Source *moves.SourceReport (env HMACs only); Evidence struct{Source, Target string} (keys); Log []string (the last 20 log keys)}`; `GET …/move?detail=1` is operator-only beyond the flat view (an Instance-role token gets the flat view with no detail, M18). |
| +- `moveView` gains `Last *provision.MoveOutcome \`json:"last,omitempty"\`` (index D11: `{"state":"none","last":{"id":"mv_…","outcome":"rolled_back","at":"…","offline_s":41.2}}` after a close; every caller of the flat view sees it, M18's Instance-role token included, since it reveals nothing the state did not); `moveDetail{moveView; Move *provision.MoveState; Claim struct{Kind, Phase, Exec string; Participants []string}; Target *moves.TargetReport; Source *moves.SourceReport (env HMACs only); Evidence struct{Source, Target string} (keys); Log []string (the last 20 log keys)}`; `GET …/move?detail=1` is operator-only beyond the flat view (an Instance-role token gets the flat view with no detail, M18). |
| - `handleAPIInstanceMoveStatus` reads `detail`; reads `hostclaim.Read`, the two reports and `s.cfg.Releases.List(moves.ReportPrefix(...)+"log/")`. |
| Tests: `TestMoveDetailLeaksNothing` (Review Focus 2: the response body contains no value of the launch environment, no evidence body, no `CARLOS_` value; it contains the report's check ids and the evidence keys); `TestMoveDetailIsOperatorOnly`. |
| @@ -77,7 +76,7 @@ Tests: `TestMoveDetailLeaksNothing` (Review Focus 2: the response body contains |
| **Files:** create `internal/moves/log.go`, `internal/moves/log_test.go`, `internal/host/movesevents.go`, `internal/host/movesevents_test.go`; modify the phase writers in `movescut.go`, `movesland.go`, `movesrollback.go`, `movesrecover.go`, `movesrevert.go`, `movesparks.go`, `internal/console/apiinstancemove.go` (request, cut, abort, resolve) to call `w.logEvent(ctx, rec, event, fields)`. |
| **Interfaces:** |
| -- `moves.LogEvent{V int; Op, Host, Event, Actor string; At time.Time; Fields map[string]string}`; `moves.WriteLog(ctx, s release.Store, prefix string, seq int, ev LogEvent) error` (`PutIfAbsent` semantics through `s.Put(key, body, "")` are not create-only on `release.Store`; the key carries `seq` and the event name, so a replay rewrites the same event: acceptable, the log is observability, not a decision); `moves.NextLogSeq(ctx, s, prefix) (int, error)` (lists `log/`, max + 1). |
| +- `moves.LogEvent{V int; Op, Host, Event, Actor string; At time.Time; Fields map[string]string}`; `moves.WriteLog(ctx, s release.Store, prefix string, seq int, ev LogEvent) error`: `s.Put(key, body, "")` **is** create-only on `release.Store` (`store.go:38`), which is what a log wants: the key carries `seq` and the event name, so two actors racing to write the same sequence number collide with `release.ErrCASConflict` and the loser takes the next number (`NextLogSeq` again, bounded to three tries); a replay of a phase that already logged its event finds its own event under the previous sequence (the writer checks the last 20 keys for `-<event>.json` under this op before writing, so a replayed phase logs once). `moves.NextLogSeq(ctx, s, prefix) (int, error)` (lists `log/`, max + 1). |
| - Events: `requested, ready, cut_requested, admitted, stopping, stopped, staged, archived, flipped, checking, parking, building, published, local, configured, restored, probing, probe_attempt, probed, committed, opened, source_parked, done, reverting, rev_evidence, rev_verify, rev_probing, rev_opened, rolling_back, released, returning, returned, claimed, verified, recovered, cleaning, cleaned, rolled_back, stuck, retry, aborting, aborted, declared_lost`; `Fields` carry timings (`offline_s` at `opened`/`recovered`: `cut_at` to the open) and digests. |
| - The member stream: `emitEvent(sink, rt, level, msg)` per transition with the operator sentence (M-C1, M-C5, M-C6, M-C7, M-C8, M-C9, M-C11 as member-safe lines); `movesWorker` gets `Events activator.EventSink` from `agent.go:464`. |
| @@ -98,12 +97,11 @@ Tests: `TestLogEventsAreSequential`; `TestEveryTransitionEmitsOneEvent` (Review |
| --- |
| -### Task 5: Paging |
| +### Task 5: Paging rules |
| -**Files:** modify `internal/host/alertsink.go` (`PageOperator`), create `internal/host/movespaging.go`, `internal/host/movespaging_test.go`; modify `internal/console/movesrecovery.go` (Plan 5 Task 7 takes the `alert` hook from here). |
| +**Files:** create `internal/host/movespaging.go`, `internal/host/movespaging_test.go` (the primitive `host.PageOperator` is Plan 5 Task 0). |
| **Interfaces:** |
| -- `(a *alertSinks) PageOperator(ctx, level, subject, body string) error` delivering through the same webhook and SNS topic as `deliver`; `host.PageOperator(ctx, level, subject, body string) error` on the armed sinks (a no-op with a log line when none are configured, as today's alert delivery does). |
| - Rules (§13): `rolled_back` → warning once; `stuck` → incident once per entry; `rolling_back` past `release_deadline` → incident once per hour; a refused start check or a refused stop evidence (stray process, M-C20) → incident once per op; `preparing` with no promotable round for 60 minutes → warning once. State for rate limits lives in the op journal (`Journal.Steps["paged:<rule>"]`), so a restart does not page again. |
| Test `TestPagingIsRateLimited` (Review Focus 4). |
| @@ -112,28 +110,24 @@ Test `TestPagingIsRateLimited` (Review Focus 4). |
| --- |
| -### Task 6: `docs/site/cli.md` and the docsite gate |
| +### Task 6: The CLI page's fenced examples |
| -**Files:** modify `docs/site/cli.md` (a `### carlos move` section under "Your deployment", after `### carlos system`), `main.go` (usage). |
| +**Files:** modify `docs/site/cli.md` (the `### carlos move` section Task 1 added). |
| -- The section: one paragraph (operator-only; what a move does; that it rolls back on its own; the rehearsal order is Paul's), the five command lines of §12 in a `sh` fence, the exit codes, and `carlos move resolve --host <h> --declare-lost <box>` with the confirmation. |
| -- `nice -n 10 go test ./internal/docsite -count=1` passes (`TestCLICommandsAreDocumented`, `TestFencedCarlosCommandsAreReal`). |
| +- Every fenced `carlos move …` example in the section runs a real verb and real flags (`TestFencedCarlosCommandsAreReal`); the exit codes table and the `--declare-lost` confirmation paragraph are checked by reading them back in `TestMoveDocsNameEveryVerb` (a test in `move_test.go` that parses the section and asserts the five verbs and the four exit codes appear). |
| -- [ ] Commit: `git commit -m "docs: carlos move on the CLI page (docsite gate)" ...` |
| +- [ ] Commit: `git commit -m "docs: carlos move examples and exit codes on the CLI page" ...` |
| --- |
| -### Task 7: The copy gate |
| +### Task 7: Completing the copy record |
| -**Files:** modify `approved_copy_test.go`; create `docs/superpowers/plans/2026-10-06-moves-phase1-copy.md` (Plans 1 to 5 appended their strings to it under their headings; this task completes it). |
| +**Files:** modify `docs/superpowers/plans/2026-10-06-moves-phase1-copy.md`, `approved_copy_test.go` (`movesCopyIDs`). |
| -- The copy file lists every operator-facing string of the series with an id: M-C1 to M-C20 from the spec, and `MV-N1 …` for the new sentences the plans introduced (door refusals, reasons, paging subjects, CLI refusals). Each row: id, the file that carries it, the text, status `pending`. |
| -- `approved_copy_test.go` gains a second record, `movesApprovedCopy`, read from the copy file's `approved` rows: the test fails when an approved text is absent from its file byte for byte, when an approved id is missing from the file, or when a `pending` row's text appears in a non-test Go file **at a user-facing call site** (`jsonError`, `fmt.Fprintln(os.Stdout|os.Stderr`, `emitEvent`, `PageOperator`) while the row is not `approved` (the ship gate). Until Paul approves, the CLI and door strings stay behind the gate: the test is `t.Skip`-free and fails on any pending string wired to a call site, so the implementer wires strings only as they are approved. |
| -- The review itself is Paul's: this task prepares the file and the gate; it does not approve anything. |
| +- The gate itself is Plan 1 Task 15 (the pinned inventory `movesCopyIDs`, `TestMovesCopyRecordIsComplete`, `TestMovesApprovedCopyIsApplied`, `TestPendingCopyIsNotWired` and its negative fixtures). This task adds this plan's rows (the CLI sentences, the detail door's refusals, the paging subjects) to the record and to `movesCopyIDs`, and confirms every row names the file that carries it. |
| +- The review itself is Paul's: this task completes the record; it approves nothing. Until Paul approves, every string stays behind the gate, so the implementer wires strings only as they are approved (Review Focus 5 is pinned by Plan 1 Task 15's tests, which this task's rows extend). |
| -Test: the extended `TestApprovedCopyIsApplied` plus `TestPendingCopyIsNotWired` (Review Focus 5). |
| - |
| -- [ ] Commit: `git commit -m "copy gate for the moves series: the record, the ship check (§20)" ...` |
| +- [ ] Commit: `git commit -m "copy record for the moves series completed (§20)" ...` |
| --- |
diff --git a/docs/superpowers/plans/2026-10-06-moves-phase1-7-rehearsal-runbook.md b/docs/superpowers/plans/2026-10-06-moves-phase1-7-rehearsal-runbook.md| index 3bdfb6f13..e1c432115 100644 |
| --- a/docs/superpowers/plans/2026-10-06-moves-phase1-7-rehearsal-runbook.md |
| +++ b/docs/superpowers/plans/2026-10-06-moves-phase1-7-rehearsal-runbook.md |
| @@ -43,15 +43,16 @@ |
| **Files:** create `cmd/moveprobe/main.go`, `cmd/moveprobe/main_test.go`, `cmd/moveprobe/README.md`. |
| **Interfaces (the app's behaviour):** |
| -- Listens on `CARLOS_SOCKET` (unix) or `PORT`; data dir from `CARLOS_DATA_DIR` or the directory of `CARLOS_DB_PATH` (whatever the exec backend provides; read `internal/activator/backend_exec.go`'s `agentOwnedEnv` for the names). |
| +- The exec backend's launch contract is **flags, not variables** (`backend_exec.go:942`: `exec.Command(bin, "--socket", rt.Socket, "--db", dbPath)`): `moveprobe --socket <unix path> --db <sqlite path>` listens on the socket and opens the database; the files directory is `strings.TrimSuffix(dbPath, ".db")` (F4, the flat layout). The agent-owned environment adds `CARLOS_REGION`, `CARLOS_CONTROL_SOCKET` and the rest of `agentOwnedEnv`; the app reads those by name. `TestMoveprobeLaunchContract` starts the binary exactly as the backend does (`--socket`, `--db`, the agent variables) and asserts `/ready` answers on the socket. |
| - First boot: writes `files/instance.key` (32 random bytes, hex) beside the database; every boot: verifies the file exists and matches the row `meta(key='instance_key')` in the database; mismatch → the readiness path answers 500 with `key mismatch`. |
| - Requires `MOVEPROBE_BUCKET`, `MOVEPROBE_REGION`, `MOVEPROBE_ENDPOINT`, `MOVEPROBE_ACCESS_KEY`, `MOVEPROBE_SECRET_KEY` (an `objstore` binding with prefix `MOVEPROBE`) and writes `probe/<boot id>.txt` at boot, reading it back on `/ready`; requires `MOVEPROBE_SECRET` (a sealed secret) and answers 500 on `/ready` when absent. |
| - Registers, through the instance control socket (`CARLOS_CONTROL_SOCKET`), a declared schedule `every-minute` → `/tick` and, on `POST /arm-once?in=<seconds>`, a once-timer `once-<n>` → `/once`; `/tick` and `/once` insert rows `(kind, name, due_at, fired_at, region)`. |
| - A background loop inserts a row `(kind='bg', seq, region)` every 500 ms; a forked worker (`exec.Command(os.Args[0], "worker")` with `Setsid`) inserts `(kind='worker', seq, region)` every second and ignores SIGTERM for 30 s (Review Focus 1). |
| -- `/ready`: 200 when the key verifies, the object reads back and the secret is present, and `MOVEPROBE_FAIL_READY_IN` does not name `CARLOS_REGION` (with `,after-write` the failure starts only after the first row is written on this box, so a dirty rollback carries a write). |
| +- `MOVEPROBE_FAIL_READY_IN=<region>[,after-write]`: in the named region, the **before-write** mode (no suffix) suppresses every mutation on this box from the first instruction: the background loop, the forked worker, the `/tick` and `/once` handlers and the control-socket registrations all return without writing, and `/ready` answers 500, so the target's database, files and timers digests stay exactly what was installed and the rollback is clean (the backend's post-spawn readiness wait at `backend_exec.go:989` would otherwise see rows written before the refusal); the **after-write** mode writes one `bg` row (and lets the worker write) and then fails `/ready`, so the rollback is dirty with a known row to find on the source. |
| +- `/ready`: 200 when the key verifies, the object reads back and the secret is present, and the failure switch does not apply. |
| - `/report`: JSON with the row counts per kind and region, the once-timers fired (name, count, due_at), the key fingerprint, the object body, `has_secret`. |
| -Tests: `TestMoveprobeWorkerOutlivesMain` (start the binary in a temp cgroup-less dir, kill the main process, the worker's pid is alive and still writing; the test then kills it); `TestReadyFailsWhereTold`; `TestReportCountsRowsByRegion`. |
| +Tests: `TestMoveprobeWorkerOutlivesMain` (start the binary in a temp cgroup-less dir, kill the main process, the worker's pid is alive and still writing; the test then kills it); `TestReadyFailsWhereTold` (both modes: before-write leaves the database's row count and the timers table unchanged after two seconds of running; after-write adds exactly the rows the mode promises); `TestReportCountsRowsByRegion`; `TestMoveprobeLaunchContract`. |
| - [ ] **Steps 1 to 5**. Commit: `git commit -m "cmd/moveprobe: the rehearsal probe app (§14.4)" ...` |
| @@ -94,7 +95,7 @@ The runbook has one section per run in §14.4's order; each section is a checkli |
| - [ ] R-6 the move log has no `stray` page; the ie box's journal shows `ReconstructStopEvidence` killed the worker (the `tree` step OK) (Review Focus 1). |
| - [ ] R-7 (not for this run). |
| -**Run 2, dev forced clean rollback:** `carlos env set --app moveprobe MOVEPROBE_FAIL_READY_IN=ie` (the target, moving uk → ie); request and cut; expect M-C8 then M-C9, exit 3; checks R-1 to R-6 on uk (the source), plus: the uk row is `ready` again, the ie bucket's `instances/<host>/` and `moves/<host>/<mv>/` are empty after `cleaned`, `rollback.dirty == false` in the log's `released` event. |
| +**Run 2, dev forced clean rollback:** `carlos env set --app moveprobe MOVEPROBE_FAIL_READY_IN=ie` (the before-write mode on the target, moving uk → ie: nothing is written there, so the digests equal the install's); request and cut; expect M-C8 then M-C9, exit 3; checks R-1 to R-6 on uk (the source), plus: the uk row is `ready` again, the ie bucket's `instances/<host>/` and `moves/<host>/<mv>/` are empty after `cleaned`, `rollback.dirty == false` in the log's `released` event, and the `released` event's three digests equal `landing.installed_db_sha256`, `source.files.plain` and `source.timers_sha256`. |
| **Run 3, dev forced dirty rollback:** `MOVEPROBE_FAIL_READY_IN=ie,after-write`; expect exit 3; R-7: the source's `/report` holds a `bg` row with region `ie` written on the target during its probe (Review Focus 4); the log's `released` event has `dirty: true`, `returned` names the lane snapshot; `moves/<host>/<rb>/` in the uk bucket is parked after the horizon. |
diff --git a/docs/superpowers/plans/2026-10-06-moves-phase1-copy.md b/docs/superpowers/plans/2026-10-06-moves-phase1-copy.md| index 6ef71df6c..87fe3b263 100644 |
| --- a/docs/superpowers/plans/2026-10-06-moves-phase1-copy.md |
| +++ b/docs/superpowers/plans/2026-10-06-moves-phase1-copy.md |
| @@ -45,6 +45,10 @@ Columns: id, file, text, status. |
| | MV-N5 | internal/console/apiinstancemove.go | a box in <from> or <to> is below the moves protocol floor. roll the platform first | pending | |
| | MV-N6 | internal/console/apiinstancemove.go | cut must be "auto" or "manual" | pending | |
| | MV-N7 | internal/provision/provision.go | this move was started by an older console and has no id. abort it and request it again | pending | |
| +| MV-N8 | internal/console/apiinstancemove.go | the fleet's edges files could not all be read. try again | pending | |
| +| MV-N9 | internal/host/movespass.go | a staging lane for this move id already exists | pending | |
| +| MV-N10 | internal/host/systemapply.go | the moves floor is unknown and this box holds an operation | pending | |
| +| MV-N11 | internal/console/apiinstancemove.go | declare-lost is not enabled in this deployment yet | pending | |
| ## Plan 2 |
diff --git a/docs/superpowers/plans/CODEX-PLAN-MOVES-PHASE1-R1.md b/docs/superpowers/plans/CODEX-PLAN-MOVES-PHASE1-R1.md| new file mode 100644 |
| index 000000000..01fdb9a39 |
| --- /dev/null |
| +++ b/docs/superpowers/plans/CODEX-PLAN-MOVES-PHASE1-R1.md |
| @@ -0,0 +1,105 @@ |
| +Reviewed the pinned plans and code read-only. No files changed or tests ran. All 28 cited witness names exist. The snapshot/anchor header choices match `litestreamfacts`, but the implementation series has the following blockers. |
| + |
| +Code links below point to the pinned main checkout; claim-plan links identify unmerged dependencies. |
| + |
| +1. **P1 — Plan 1, Task 2: duplicate Go identifier prevents compilation.** |
| + [Plan:400](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-1-copy-and-staging.md:400) declares `type MoveLanding`; line 433 declares `const MoveLanding` in the same package. Both extend [provision.go:907](/home/paulca/.cache/moves-plan-main/internal/provision/provision.go:907). Rename the journal type, update its consumers, and compile the complete proposed declarations. |
| + |
| +2. **P1 — Plan 2, Task 3: stop reconstruction introduces an import cycle.** |
| + [Plan:589](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-2-cut-flip-landing.md:589) makes `activator` import `stopevidence`, which already imports `activator` and holds `*activator.StopEvidence` in [evidence.go:10](/home/paulca/.cache/moves-plan-main/internal/stopevidence/evidence.go:10). The proposed `a.cgroups` also is not a field of `Service`. Move reconstruction above both packages or extract shared evidence types, and explicitly wire the backend’s cgroup manager. |
| + |
| +3. **P1 — Index / Plan 3, Task 5: the quoted strict-spawn signature is wrong.** |
| + [Index:78](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-0-index.md:78) gives `(*exec.Cmd, error)`. [Claim Plan 6:878](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/replica-ownership/docs/superpowers/plans/2026-10-05-host-lifecycle-claim-6-box-wiring.md:878) returns `(*exec.Cmd, *cgroup.Prepared, error)` and requires retaining that prepared handle through process completion and checked cleanup. Correct the signature and make ownership of the handle explicit, including failed starts. |
| + |
| +4. **P1 — Plan 1, Task 5: receipt recovery can seal stale data.** |
| + [Plan:1246](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-1-copy-and-staging.md:1246) accepts a receipt’s version on ETag equality without validating the receipt against the current source. After copying A/v1, copying B/v2, and crashing before receipt persistence, the next round can seal v1 despite listing B. The relevant branches and persistence are [mirror.go:186](/home/paulca/.cache/moves-plan-main/internal/lease/mirror.go:186) and [mirror.go:342](/home/paulca/.cache/moves-plan-main/internal/lease/mirror.go:342). Require a complete matching receipt; recopy otherwise. Test crashes between copy acknowledgement and receipt persistence, including an absent receipt. |
| + |
| +5. **P1 — Plan 1, Task 6: real staging leases can never pass the guard.** |
| + [Plan:1374](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-1-copy-and-staging.md:1374) requires over 16 minutes remaining. Real acquisition uses a 15-minute TTL in [lease.go:37](/home/paulca/.cache/moves-plan-main/internal/lease/lease.go:37), with an earlier `SafeUntil`. The passing test supplies an artificial 30-minute lease. Define a staging lease duration compatible with the signature margin, propagate its actual bound into cleanup, and test real acquisition settings. |
| + |
| +6. **P1 — Plan 1, Tasks 6/14: checking time does not implement sign-at-check.** |
| + [Plan:1365](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-1-copy-and-staging.md:1365) checks time before invoking ordinary SDK calls. Those calls still sign later and may re-sign retries; see [mirror.go:474](/home/paulca/.cache/moves-plan-main/internal/lease/mirror.go:474). A paused worker can therefore sign after its lease ends, violating §7.3/K5 and the cleanup bound. Specify timestamp-bound signing and retry behavior for every staging mutation. Test pauses between check and send, and retries after expiry. |
| + |
| +7. **P1 — Plan 1, Task 14: `move.json` replay requires a read forbidden by M17.** |
| + [Plan:2848](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-1-copy-and-staging.md:2848) requires the source to decode the target object after a 412. The source cannot read target `moves/*`; [PutIfAbsent:277](/home/paulca/.cache/moves-plan-main/internal/lease/versioned.go:277) returns only `ErrExists`, without the existing body or version. Define target-owned validation or another replay protocol within the approved grant. Test a successful PUT followed by a lost ledger write with all source GETs of target staging denied. |
| + |
| +8. **P1 — Plan 1, Task 7: the archive loses a legal file named `env`.** |
| + [Plan:1585](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-1-copy-and-staging.md:1585) reserves `env` for the overlay while adding application files under bare relative names. Extraction at line 1780 redirects both to `<host>.env`, consumed by [backend_exec.go:1164](/home/paulca/.cache/moves-plan-main/internal/activator/backend_exec.go:1164). Use separate `files/…` and overlay namespaces, reject duplicate destinations, and round-trip a file and directory named `env` alongside an overlay. |
| + |
| +9. **P1 — Plan 1, Tasks 11/14: readiness has a circular target-box prerequisite.** |
| + [Task 11:2461](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-1-copy-and-staging.md:2461) requires the target box in `touched`; Task 14 adds only its region. The first target-box addition is [Plan 2:789](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-2-cut-flip-landing.md:789), after the flip. A first move to that box cannot become ready. Add the target box at the required earlier CAS and test from a claim that has never touched it. |
| + |
| +10. **P1 — Plan 1, Tasks 11/12, and Plan 6, Task 3: empty ETags do not overwrite.** |
| + [Readiness writes:2621](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-1-copy-and-staging.md:2621) use `Put(..., "")` for recurring reports; the console fingerprint does likewise. [Plan 6:80](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-6-cli-status-observability.md:80) explicitly misstates this API. [Store:38](/home/paulca/.cache/moves-plan-main/internal/release/store.go:38) defines it as create-only. Reports cannot refresh after their first write. Use read/ETag/CAS replacement for mutable reports and explicit idempotency for logs; test repeated writes through the real store. |
| + |
| +11. **P1 — Plan 1, Tasks 4/12: the protocol floor fails open.** |
| + [Plan:2650](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-1-copy-and-staging.md:2650) permits activation when reading the floor fails. Boot recovery normally runs before networking, as documented in [systemapply.go:109](/home/paulca/.cache/moves-plan-main/internal/host/systemapply.go:109), making this especially consequential. The door also uses `readEdgesFiles`, which discards completeness information: [instancesplacement.go:198](/home/paulca/.cache/moves-plan-main/internal/console/instancesplacement.go:198). Preserve an armed floor locally and refuse unknown activation eligibility; require complete protocol evidence at the door. Test offline recovery and unreadable actor reports. |
| + |
| +12. **P1 — Plan 2, Task 5: immutable stop evidence is published before its final proof exists.** |
| + [Plan:691](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-2-cut-flip-landing.md:691) creates `stopped-source.json` before the checkpoint and final TXID at line 692. Yet `Complete()` requires replica evidence, and landing requires a complete report. [checkpoint.go:20](/home/paulca/.cache/moves-plan-main/internal/lease/checkpoint.go:20) obtains that result only after synchronization. Either stopping cannot finish or the immutable report is incomplete. Separate preliminary stopping from final evidence, then publish the complete report before `ExecToNone`. Test every intervening crash. |
| + |
| +13. **P1 — Plan 2, Task 5: a crash can skip final archive publication and still flip.** |
| + [Plan:694](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-2-cut-flip-landing.md:694) persists `archived` before copying the final archive and adding its versions to the seal. The next phase immediately flips. A restart can therefore install an earlier archive, losing files written before stopping. Make archive publication and seal completion replayable prerequisites of `MoveFlip`; inject crashes after the record CAS and copy response. The proposed create-only archive test does not establish this barrier. |
| + |
| +14. **P1 — Plan 2, Task 6: timer admission can race past drain and snapshot.** |
| + [Plan:716](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-2-cut-flip-landing.md:716) checks the hold before acquiring the runner slot. In [runner.go:143](/home/paulca/.cache/moves-plan-main/internal/schedule/runner.go:143), route reading precedes `claim`; a paused tick can resume after the cut observes `Busy=false`, then advance bookkeeping at line 350. Acquire the slot before a fresh hold check, or implement an atomic freeze/drain handshake. Test that interleaving, rather than only an already-held row. |
| + |
| +15. **P1 — Plan 2, Task 8: snapshot encoding fails above SQLite’s lock-page boundary.** |
| + [Plan:750](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-2-cut-flip-landing.md:750) encodes every page. The pinned [LTX encoder:219](/home/paulca/go/pkg/mod/github.com/superfly/ltx@v0.5.1/encoder.go:219) rejects the lock page; litestream skips it. Skip `ltx.LockPgno(pageSize)` while preserving the database page count. Add a sparse or synthetic database test extending beyond 1 GiB. |
| + |
| +16. **P1 — Plan 2, Task 8: generation-build replay restores over an existing output file.** |
| + [Plan:753](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-2-cut-flip-landing.md:753) always restores into `<WorkDir>/<host>.db`. After a crash following restore or snapshot upload, [litestream replica.go:601](/home/paulca/go/pkg/mod/github.com/benbjohnson/litestream@v0.5.14/replica.go:601) refuses that existing file. Define verified reuse or fresh operation-owned scratch. The replay test must reuse the same work directory and cover an interrupted restore. |
| + |
| +17. **P1 — Plan 2, Task 10 / Plan 4, Task 3: local retirement silently does nothing.** |
| + [Plan 2:793](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-2-cut-flip-landing.md:793) passes a fresh ID to `RetireArtifacts`, but its preceding queue call creates no local obligation. [local.go:62](/home/paulca/.cache/moves-plan-main/internal/park/local.go:62) requires a matching persisted chore and treats an unknown ID as completed. Old files can survive installation. Queue the local chore and inventory first, persist its ID, and test actual filesystem retirement. Plan 5’s destination patch must also include [local_linux.go:201](/home/paulca/.cache/moves-plan-main/internal/park/local_linux.go:201). |
| + |
| +18. **P1 — Plan 3, Tasks 4/5: the first probe compares prepared files against the source generation.** |
| + [Plan:302](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-3-probe-commit-open.md:302) runs `StartCheck` before admission. Prepared markers name the move generation, while the flip intentionally preserves `holder.generation`; only [AdmitWriter:27](/home/paulca/.cache/moves-plan-main/internal/hostclaim/generation.go:27) changes it. The first probe therefore refuses its own files. Separate prepared-input validation from admitted-generation checks, and run the latter within the shared admitted-start sequence. Test from the real source-generation state. |
| + |
| +19. **P1 — Plan 3, Tasks 3/6: the claim never reaches a phase accepted by commit.** |
| + [Plan:324](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-3-probe-commit-open.md:324) advances record phases only; the claim remains `landing/checking`. `Commit` at line 253 accepts only `probing` or `probed`. [AdmitWriter:35](/home/paulca/.cache/moves-plan-main/internal/hostclaim/generation.go:35) checks, but does not advance, the phase. The test manually sets `probed`, hiding the gap. Specify replayable claim-phase synchronization and test flip through commit without fixture-only phase edits. |
| + |
| +20. **P1 — Plan 4, Task 10: declare-lost can override a winning record commit.** |
| + [Plan:191](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-4-rollback-return-declare-lost.md:191) calls `DeclareLost` using a captured record before attempting `MoveRollBack`. The target can commit the record between those operations. The claim declaration then revokes authority, while rollback refuses. [Claim Plan 8:1444](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/replica-ownership/docs/superpowers/plans/2026-10-05-host-lifecycle-claim-8-leases-floor.md:1444) cannot atomically validate that other object. Win and persist the record decision first, then apply its claim half. Add the commit-winning interleaving test. |
| + |
| +21. **P1 — Plan 4, Task 10 / Plan 6, Task 1: the loss payload does not match claim Plan 8.** |
| + [Plan 4:191](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-4-rollback-return-declare-lost.md:191) constructs `LossOperatorAttestation{Address: addr}`; [Claim Plan 8:476](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/replica-ownership/docs/superpowers/plans/2026-10-05-host-lifecycle-claim-8-leases-floor.md:476) has no `Address` field. The submitted cloud observation also omits required account/request IDs and required attestations. Specify the actual schema end to end and test creating a real accepted signed `LossRecord`, rather than a simplified fake. |
| + |
| +22. **P1 — Plan 4, Tasks 2/5/10: lost-target recovery cannot consume the declaration result.** |
| + [ReturnToSource:73](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-4-rollback-return-declare-lost.md:73) requires empty `Exec` and `evidence_target`. [Claim Plan 8:1511](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/replica-ownership/docs/superpowers/plans/2026-10-05-host-lifecycle-claim-8-leases-floor.md:1511) journals `loss.<id>`, writes `Exec="none"`, retains the phase, and removes the target participant. No adapter establishes recovery’s prerequisites. Define that transition explicitly, including coverage and obligations, and run the source recovery pass against a real declaration result. |
| + |
| +23. **P1 — Plan 5, Task 7: orphan claims are excluded from recovery.** |
| + [Plan:133](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-5-cleanup-and-parks.md:133) iterates records with `Move != nil`, then promises to release claims whose record has no move. Those claims are never visited; [ListAllRecords:1401](/home/paulca/.cache/moves-plan-main/internal/provision/reconcile.go:1401) is not claim discovery. Additionally, `ReleaseMove` rejects their `moving/""` phase. Add claim-to-record orphan discovery and a guarded orphan transition. Test the complete pass after a request-door crash. |
| + |
| +24. **P1 — Plan 6, Tasks 1/2: remembered polling state cannot determine the terminal result.** |
| + [Plan:52](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-6-cli-status-observability.md:52) derives the exit code from the last observed state. A disconnected or paused CLI can miss the whole rollback and next receive `none`, which [apiinstancemove.go:88](/home/paulca/.cache/moves-plan-main/internal/console/apiinstancemove.go:88) returns after clearing `Move`. Persist and expose an operation-bound terminal outcome. Test skipped intermediate states and reconnection after close; the scripted test that always observes `rolling_back` is insufficient. |
| + |
| +25. **P1 — Plan 7, Task 1: the probe app’s launch contract differs from the real backend.** |
| + [Plan:46](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-7-rehearsal-runbook.md:46) specifies socket/database discovery through environment variables. [backend_exec.go:942](/home/paulca/.cache/moves-plan-main/internal/activator/backend_exec.go:942) launches `--socket … --db …`; its agent-owned environment does not provide the proposed replacements. Require those flags and test launching through the actual exec-backend contract. |
| + |
| +26. **P2 — Index / Plan 7: activation depends on unlisted claim Plan 8b.** |
| + [Index:30](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-0-index.md:30) identifies Plan 8 as the loss dependency. [Claim Plan 8:5678](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/replica-ownership/docs/superpowers/plans/2026-10-05-host-lifecycle-claim-8-leases-floor.md:5678) explicitly defers authenticated floor identity, actor census, and live quarantine/loss/readmission wiring to 8b. Mark those activation prerequisites separately from library build dependencies. A direct local-retirement fallback cannot substitute for returning-box quarantine. |
| + |
| +27. **P2 — Plan 4, Tasks 1/5/10: recovery callers request forbidden transitions.** |
| + [Task 1:57](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-4-rollback-return-declare-lost.md:57) allows rollback `MoveStuck` only from probing, but Task 5 calls it on verification mismatch. Task 10 omits the legal `released + Dirty` loss window and requests `returning → released`, prohibited by the forward-only phase sequence. Define explicit mismatch and declared-loss transitions, with retry unable to bypass verification. Test them through the actual transition closures. |
| + |
| +28. **P2 — Plan 4, Task 6 / Plan 5, Task 1: reverting has no staging-cleanup producer or dispatcher.** |
| + [Plan 4:143](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-4-rollback-return-declare-lost.md:143) waits for staging cleanup attributed to Plan 5. [Plan 5:53](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-5-cleanup-and-parks.md:53) queues and dispatches it only for aborting. Explicitly queue and run the scoped target cleanup during reverting. Test residue removal and release after acknowledgement. |
| + |
| +29. **P2 — Plan 1, Task 11: the T4 MinIO test will not run in the cited CI gate.** |
| + [Plan:2628](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-1-copy-and-staging.md:2628) adds a host-package test gated by MinIO environment. [.amadan/ci.d/30-minio-contract:330](/home/paulca/.cache/moves-plan-main/.amadan/ci.d/30-minio-contract:330) runs only `internal/objstore` and `TestDeliveredContractMinio`. Add the host test to that live invocation and assert its PASS line. Also add isolated failing cases for the readiness hooks currently stubbed to pass. |
| + |
| +30. **P2 — Plans 5/6: the published task order has a dependency cycle and delayed gates.** |
| + [Plan 5:138](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-5-cleanup-and-parks.md:138) requires Plan 6 Task 5 first, while Plan 6 requires Plan 5 merged. Plan 6 Task 1 also adds dispatch before Task 6 supplies the mandatory heading, so [docsite’s gate:58](/home/paulca/.cache/moves-plan-main/internal/docsite/cli_test.go:58) fails that earlier commit. Move the paging primitive earlier and place dispatch with its documentation. Put copy approval/checking before the earlier tasks wire pending strings, rather than introducing the gate last. |
| + |
| +31. **P2 — Plan 6, Task 7: copy completeness needs an independent inventory.** |
| + [Plan:131](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-6-cli-status-observability.md:131) derives reviewed entries from approved rows, but that cannot detect a deleted required row by itself. The existing [approved_copy_test.go:87](/home/paulca/.cache/moves-plan-main/approved_copy_test.go:87) deliberately compares against a separately pinned inventory. Specify that inventory and negative fixtures for deleted IDs, changed text, and pending text reaching call sites through helpers or formatting. |
| + |
| +32. **P2 — Plan 7, Tasks 1/3: the “clean rollback” rehearsal writes before refusing readiness.** |
| + [Plan:50](/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/plans/2026-10-06-moves-phase1-7-rehearsal-runbook.md:50) starts database-writing loops unconditionally; the switch at line 51 changes only readiness. During the backend’s post-spawn readiness wait, [backend_exec.go:989](/home/paulca/.cache/moves-plan-main/internal/activator/backend_exec.go:989), those writes make the target dirty. Implement a before-write failure mode that suppresses every database/timer mutation, and assert unchanged digests separately from the after-write mode. |
| + |
| +The proposed tests do not yet establish “red before the fix” across the series: several use states impossible under the real APIs, manually supply missing transitions, omit the decisive crash interleaving, or are excluded from CI. |
| + |
| +PLAN: NOT READY |
| + |
| +The locked design can remain intact, but the plans need revision before implementation. As written, they contain compile failures, staging and readiness paths that cannot progress, crash windows that can select stale data or lose files, and recovery paths that do not compose with the actual claim APIs. Resolve those interfaces and ordering gaps, then require integration tests that use the real store, claim transitions, park engine, and admitted-start sequence at the identified crash boundaries. |
| \ No newline at end of file |