Moves Phase 1, Plan 7: Rehearsal Runbook Implementation Plan

> For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking. Nothing in this plan runs against production without Paul's go, and every rehearsal gates the next (§15 step 7).

Goal: Prove Phase 1 on a probe app that exercises every carried state, then on a disposable real instance, then on kpmg, in M12's order, with a written runbook an operator follows and a pass criterion the run must meet.

Architecture: moveprobe is a small Go app in cmd/moveprobe shaped like ips (§14.4): instance.key written at first boot and verified every boot; a required store binding and a sealed secret; a declared schedule and a once-timer; a background loop writing rows; a forked worker that outlives the main process; a switch to fail its readiness path in a named region before or after writing. The runbook is docs/superpowers/plans/2026-10-06-moves-phase1-rehearsal.md, a checklist per run with the commands, the expected log events, and the checks. Cross-version tests pin the door and the floor against pre-floor binaries.

Tech Stack: Go, net/http, database/sql with modernc sqlite, the platform's carlos CLI, tofu for nothing (no infra change in this plan beyond §15 step 5, which is applied as scoped work by Paul).

Spec: rev 7.23 §14.3, §14.4, §15. Index: 2026-10-06-moves-phase1-0-index.md. Plans 1 to 6 are merged, not rolled, and not complete (preflight N1): no run in this plan executes on main until claim Plan 12 and its moves dependants are merged, released and rolled.

Global Constraints

• Gate, cross-compile, the default Go build cache; tests serial and niced.
• Rollout order (§15): the dormant release first (consoles, then every box), M17's sibling_staging_arns and the versioned-object grants (§15 step 5) applied by Paul, then control/moves-protocol.json written create-only before any protected production rehearsal (§11, joint N12), then the rehearsals in order: dev IE to uk done; dev forced clean rollback; dev forced dirty rollback; dev declare-lost; the same on the flagship with scoped approval; a disposable ips instance in bdn; kpmg with Paul's approval.
• A run passes only if: the key verifies, the object reads back, the secret is present, every timer ran exactly once around the cut, no background row is missing, and no copy remains in a live namespace of the region that lost (§14.4).
• moveprobe's readiness-failure switch is an environment variable (MOVEPROBE_FAIL_READY_IN=<region>[,after-write]) delivered through the config bundle, so a rehearsal flips it with carlos env set, never a rebuild.

Review Focus

1. The forked worker must survive the main process's exit and still be killed by ReconstructStopEvidence; a run where the worker survives the cut must fail the run with the stray-process page, never pass silently. Task 1 (TestMoveprobeWorkerOutlivesMain) and runbook check R-6.
2. A once-timer registered seconds before the cut must fire exactly once, on the target, with its due_at preserved; firing twice or never fails the run. Task 1's timer registration and runbook check R-4.
3. The disposable bdn instance must be created with a fresh host name no previous rehearsal used, or a stale instances/<host>/ or moves/<host>/ lane makes the sweep's work part of the run. Runbook step S-1.
4. The dirty-rollback run must prove the target's writes came back: the probe app writes a row with the target's CARLOS_REGION while it runs on the target; after the rollback the source must hold that row. Runbook check R-7.
5. The declare-lost run must use a box the operator actually stopped through the cloud API (dev only, never the flagship's shared boxes without Paul's go), and the loss record must name the declared sessions. Runbook step S-9.

---

File structure

| path | responsibility | |---|---| | cmd/moveprobe/main.go, cmd/moveprobe/main_test.go | the probe app | | cmd/moveprobe/README.md | how to deploy it as a carlos app (carlos ship) | | internal/console/apiinstancemove_test.go, internal/host/systemapply_test.go | cross-version tests | | docs/superpowers/plans/2026-10-06-moves-phase1-rehearsal.md | the runbook |

---

Task 1: moveprobe

Files: create cmd/moveprobe/main.go, cmd/moveprobe/main_test.go, cmd/moveprobe/README.md.

Interfaces (the app's behaviour):

• The exec backend's launch contract is flags, not variables (internal/activator/backend_exec.go:1541, in spawnChildFrom: exec.Command(bin, "--socket", sock, "--db", dbPath)): moveprobe --socket <unix path> --db <sqlite path> listens on the socket and opens the database; the files directory is strings.TrimSuffix(dbPath, ".db") (F4, the flat layout). The agent-owned environment is composed by agentOwnedVars (backend_exec.go:1603-1626): CARLOS_CONTROL_SOCKET only when the control listener bound, CARLOS_PURGE, CARLOS_REGION only when the box has a region, and CARLOS_HOST. CARLOS_ADMIN_TOKEN and CARLOS_INSTANCE_KEY come from <host>.env, not the composer. The app reads those by name and treats a missing CARLOS_REGION as region home (preflight N7). TestMoveprobeLaunchContract starts the binary exactly as the backend does (--socket, --db, the agent variables) and asserts /ready answers on the socket.
• First boot: writes files/instance.key (32 random bytes, hex) beside the database; every boot: verifies the file exists and matches the row meta(key='instance_key') in the database; mismatch → the readiness path answers 500 with key mismatch.
• Requires MOVEPROBE_BUCKET, MOVEPROBE_REGION, MOVEPROBE_ENDPOINT, MOVEPROBE_ACCESS_KEY, MOVEPROBE_SECRET_KEY (an objstore binding with prefix MOVEPROBE) and writes probe/<boot id>.txt at boot, reading it back on /ready; requires MOVEPROBE_SECRET (a sealed secret) and answers 500 on /ready when absent.
• Registers, through the instance control socket (CARLOS_CONTROL_SOCKET), a declared schedule every-minute → /tick and, on POST /arm-once?in=<seconds>, a once-timer once-<n> → /once; /tick and /once insert rows (kind, name, due_at, fired_at, region).
• A background loop inserts a row (kind='bg', seq, region) every 500 ms; a forked worker (exec.Command(os.Args[0], "worker") with Setsid) inserts (kind='worker', seq, region) every second and ignores SIGTERM for 30 s (Review Focus 1).
• MOVEPROBE_FAIL_READY_IN=<region>[,after-write]: in the named region, the before-write mode (no suffix) suppresses every mutation on this box from the first instruction: the background loop, the forked worker, the /tick and /once handlers and the control-socket registrations all return without writing, and /ready answers 500, so the target's database, files and timers digests stay exactly what was installed and the rollback is clean (the readiness checks after a spawn, the wake's waitHealthyCtx at backend_exec.go:1335 (GET /, any status below 500, waitHealthy at :2344) and the claim probe's claimProbeSocketReady at internal/host/claimready_linux.go:23-41 (2xx only), would otherwise see rows written before the refusal); the after-write mode writes one bg row (and lets the worker write) and then fails /ready, so the rollback is dirty with a known row to find on the source.
• /ready: 200 when the key verifies, the object reads back and the secret is present, and the failure switch does not apply.
• /report: JSON with the row counts per kind and region, the once-timers fired (name, count, dueat), the key fingerprint, the object body, `hassecret`.

Tests: TestMoveprobeWorkerOutlivesMain (start the binary in a temp cgroup-less dir, kill the main process, the worker's pid is alive and still writing; the test then kills it); TestReadyFailsWhereTold (both modes: before-write leaves the database's row count and the timers table unchanged after two seconds of running; after-write adds exactly the rows the mode promises); TestReportCountsRowsByRegion; TestMoveprobeLaunchContract.

• [ ] Steps 1 to 5. Commit: git commit -m "cmd/moveprobe: the rehearsal probe app (§14.4)" ...

---

Task 2: Cross-version tests

Files: modify internal/console/apiinstancemove_test.go, internal/host/systemapply_test.go, internal/provision/move_test.go.

• TestDoorRefusesBelowTheFloor: an edges file with moves_protocol: 0 (a pre-floor box) in either region → 409 naming the floor; with the field absent → the same.
• TestSystemApplyRefusesAPreFloorBinaryOnAParticipant: a box whose registry holds a held row (hold_op set) refuses any carlos system apply or rollback (§11 "any downgrade of a box that is a participant in a held operation is refused"); the applier reads the registry at /data/registry.db through the same path the agent uses.
• TestOldConsoleAbortAllowlist: MoveAbort from a v0.22 console (the current provision.go:1027-1035 allowlist, reproduced as a test double) refuses every new state (landing, committed, rolling_back, reverting, stuck), per P0's rule.

• [ ] Commit: git commit -m "moves: cross-version tests: the floor at the door and at activation, the old abort allowlist (§14.3)" ...

---

Task 3: The runbook

Files: create docs/superpowers/plans/2026-10-06-moves-phase1-rehearsal.md.

The runbook has one section per run in §14.4's order; each section is a checklist an operator ticks:

Preconditions (once):

• [ ] P-1 The Phase 1 release is rolled dormant: consoles first, then every box (carlos status shows the build; control/edges/*.json all carry "moves_protocol":1).
• [ ] P-2 §15 step 5 applied: sibling_staging_arns and the versioned-object grants (Paul, scoped tofu apply -target); verify with T10's probe through carlos move status --host <h> after the first request (T10 passes).
• [ ] P-3 control/moves-protocol.json written create-only: {"min":1,"min_release":"<the rolled version>"} (an aws s3api put-object --if-none-match '*' from an operator shell with the deployment profile); carlos system apply of an older build on dev now refuses.
• [ ] P-4 moveprobe shipped as app moveprobe under the ops account with a store binding MOVEPROBE, a sealed secret MOVEPROBE_SECRET, a declared schedule, and moves.ready_path unset (/ serves /ready's answer).
• [ ] P-5 Every rehearsal host is adopted (claim Plan 5's adoption has run: control/hosts/<host>.json exists with op.kind: serving). This needs C5 merged and the #232 ownership backfill run for every rehearsal host: adoption happens only inside that backfill, never from the move door or the recovery pass, and merging the claim branches alone does not adopt anything.
• [ ] P-6 The switch is on for this deployment: CARLOS_MOVES_PHASE1=on on the console and every box (index D13), with the copy record approved and the default flipped in one commit before any run on the flagship. Run 4 (declare-lost) needs claim Plan 8b merged and rolled (the roster, census, release identity and guard, the termination fence, the real declare-lost and readmit) and CARLOS_MOVES_DECLARE_LOST is never set outside tests; until 8b is rolled, Run 4 is skipped and the series continues with Run 5, and the loss run is repeated on dev once 8b lands.

Run 1, dev IE → uk, done:

• [ ] S-1 carlos instances create --app moveprobe --host mp1-<date>.<domain> --region ie (a fresh name; Review Focus 3); wake it (curl https://<host>/), POST /arm-once?in=600, note /report.
• [ ] S-2 carlos move --host <host> --to uk --manual-cut; wait for M-C12; carlos move status --host <host> shows every T check passing and attempts_admitted ≥ 1.
• [ ] S-3 carlos move cut --host <host>; expect M-C5, M-C6, M-C7; exit 0.
• [ ] R-1 /report on uk: the key fingerprint equals S-1's; the object reads back; has_secret true.
• [ ] R-2 the bg and worker row counts are at least S-1's and every seq is contiguous (no missing row).
• [ ] R-3 the every-minute rows continue on uk with region uk from the first tick after the open.
• [ ] R-4 the once-timer fired exactly once, on uk, with S-1's due_at (Review Focus 2).
• [ ] R-5 aws s3api list-object-versions --bucket <ie data bucket> --prefix instances/<host>/ is empty after the source park acknowledged (carlos move status detail shows no obligation; the log has source_parked and done); parked/<inc>/<host>/<ret>/ holds the archive.
• [ ] R-6 the move log has no stray page; the ie box's journal shows ReconstructStopEvidence killed the worker (the tree step OK) (Review Focus 1).
• [ ] R-7 (not for this run).

Run 2, dev forced clean rollback: carlos env set --app moveprobe MOVEPROBE_FAIL_READY_IN=ie (the before-write mode on the target, moving uk → ie: nothing is written there, so the digests equal the install's); request and cut; expect M-C8 then M-C9, exit 3; checks R-1 to R-6 on uk (the source), plus: the uk row is ready again, the ie bucket's instances/<host>/ and moves/<host>/<mv>/ are empty after cleaned, rollback.dirty == false in the log's released event, and the released event's three digests equal landing.installed_db_sha256, source.files.plain and source.timers_sha256.

Run 3, dev forced dirty rollback: MOVEPROBE_FAIL_READY_IN=ie,after-write; expect exit 3; R-7: the source's /report holds a bg row with region ie written on the target during its probe (Review Focus 4); the log's released event has dirty: true, returned names the lane snapshot; moves/<host>/<rb>/ in the uk bucket is parked after the horizon.

Run 4, dev declare-lost (needs P-6's Plan 8b clause): request a move to a dev box Paul designates as disposable; after the flip, stop that box through the AWS console (dev only); carlos move resolve --host <host> --declare-lost <box> --instance-id <id>; type the host name at M-C18; expect the clean rollback to complete on the source, exit 3; the loss record names the box, fenced_at, the declared timers; restart the box and watch it quarantine the host's local state before anything else (S-9, Review Focus 5).

Run 5, flagship, scoped approval: runs 1 to 3 again on the flagship with a moveprobe instance, each with Paul's explicit go in the discussion thread.

Run 6, a disposable ips instance in bdn: carlos instances create of an ips instance Tony designates, run 1 only, with Tony's check that the instance serves after the open.

Run 7, kpmg: run 1 only, with Paul's approval, in a window agreed with the customer; the cut budget stays 120 s; the runbook's offline_s from the done event is the number reported back.

After every run: attach the done/rolled_back log event's timings, /report, and the bucket listings to the discussion; a failed check stops the series until its cause is fixed and the run repeated.

• [ ] Commit: git commit -m "docs: the Phase 1 rehearsal runbook in M12's order (§14.4, §15)" ...

---

Task 4: Gate

• [ ] Full gate, cross-compile (cmd/moveprobe builds on darwin and windows; the Setsid attribute is behind //go:build !windows).

Self-review notes

Coverage: §14.3 (Task 2), §14.4 (Tasks 1, 3), §15 steps 2 to 7 (the runbook's preconditions and order; step 1 and step 8 are done per the spec). The infra change of step 5 is Paul's scoped work and is a precondition, not a task.