| 1 | # Moves Phase 1, Plan 7: Rehearsal Runbook Implementation Plan |
| 2 | |
| 3 | > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. **Nothing in this plan runs against production without Paul's go**, and every rehearsal gates the next (§15 step 7). |
| 4 | |
| 5 | **Goal:** Prove Phase 1 on a probe app that exercises every carried state, then on a disposable real instance, then on kpmg, in M12's order, with a written runbook an operator follows and a pass criterion the run must meet. |
| 6 | |
| 7 | **Architecture:** `moveprobe` is a small Go app in `cmd/moveprobe` shaped like ips (§14.4): `instance.key` written at first boot and verified every boot; a required store binding and a sealed secret; a declared schedule and a once-timer; a background loop writing rows; a forked worker that outlives the main process; a switch to fail its readiness path in a named region before or after writing. The runbook is `docs/superpowers/plans/2026-10-06-moves-phase1-rehearsal.md`, a checklist per run with the commands, the expected log events, and the checks. Cross-version tests pin the door and the floor against pre-floor binaries. |
| 8 | |
| 9 | **Tech Stack:** Go, `net/http`, `database/sql` with modernc sqlite, the platform's `carlos` CLI, `tofu` for nothing (no infra change in this plan beyond §15 step 5, which is applied as scoped work by Paul). |
| 10 | |
| 11 | **Spec:** rev 7.23 §14.3, §14.4, §15. Index: `2026-10-06-moves-phase1-0-index.md`. Plans 1 to 6 are merged, not rolled, and not complete (preflight N1): no run in this plan executes on main until claim Plan 12 and its moves dependants are merged, released and rolled. |
| 12 | |
| 13 | ## Global Constraints |
| 14 | |
| 15 | - Gate, cross-compile, the default Go build cache; tests serial and niced. |
| 16 | - Rollout order (§15): the dormant release first (consoles, then every box), M17's `sibling_staging_arns` and the versioned-object grants (§15 step 5) applied by Paul, then `control/moves-protocol.json` written create-only **before** any protected production rehearsal (§11, joint N12), then the rehearsals in order: dev IE to uk done; dev forced clean rollback; dev forced dirty rollback; dev declare-lost; the same on the flagship with scoped approval; a disposable ips instance in `bdn`; kpmg with Paul's approval. |
| 17 | - A run passes only if: the key verifies, the object reads back, the secret is present, every timer ran exactly once around the cut, no background row is missing, and no copy remains in a live namespace of the region that lost (§14.4). |
| 18 | - `moveprobe`'s readiness-failure switch is an environment variable (`MOVEPROBE_FAIL_READY_IN=<region>[,after-write]`) delivered through the config bundle, so a rehearsal flips it with `carlos env set`, never a rebuild. |
| 19 | |
| 20 | ## Review Focus |
| 21 | |
| 22 | 1. The forked worker must survive the main process's exit and still be killed by `ReconstructStopEvidence`; a run where the worker survives the cut must fail the run with the stray-process page, never pass silently. Task 1 (`TestMoveprobeWorkerOutlivesMain`) and runbook check R-6. |
| 23 | 2. A once-timer registered seconds before the cut must fire exactly once, on the target, with its `due_at` preserved; firing twice or never fails the run. Task 1's timer registration and runbook check R-4. |
| 24 | 3. The disposable `bdn` instance must be created with a fresh host name no previous rehearsal used, or a stale `instances/<host>/` or `moves/<host>/` lane makes the sweep's work part of the run. Runbook step S-1. |
| 25 | 4. The dirty-rollback run must prove the target's writes came back: the probe app writes a row with the target's `CARLOS_REGION` while it runs on the target; after the rollback the source must hold that row. Runbook check R-7. |
| 26 | 5. The declare-lost run must use a box the operator actually stopped through the cloud API (dev only, never the flagship's shared boxes without Paul's go), and the loss record must name the declared sessions. Runbook step S-9. |
| 27 | |
| 28 | --- |
| 29 | |
| 30 | ## File structure |
| 31 | |
| 32 | | path | responsibility | |
| 33 | |---|---| |
| 34 | | `cmd/moveprobe/main.go`, `cmd/moveprobe/main_test.go` | the probe app | |
| 35 | | `cmd/moveprobe/README.md` | how to deploy it as a carlos app (`carlos ship`) | |
| 36 | | `internal/console/apiinstancemove_test.go`, `internal/host/systemapply_test.go` | cross-version tests | |
| 37 | | `docs/superpowers/plans/2026-10-06-moves-phase1-rehearsal.md` | the runbook | |
| 38 | |
| 39 | --- |
| 40 | |
| 41 | ### Task 1: `moveprobe` |
| 42 | |
| 43 | **Files:** create `cmd/moveprobe/main.go`, `cmd/moveprobe/main_test.go`, `cmd/moveprobe/README.md`. |
| 44 | |
| 45 | **Interfaces (the app's behaviour):** |
| 46 | - The exec backend's launch contract is **flags, not variables** (`internal/activator/backend_exec.go:1541`, in `spawnChildFrom`: `exec.Command(bin, "--socket", sock, "--db", dbPath)`): `moveprobe --socket <unix path> --db <sqlite path>` listens on the socket and opens the database; the files directory is `strings.TrimSuffix(dbPath, ".db")` (F4, the flat layout). The agent-owned environment is composed by `agentOwnedVars` (`backend_exec.go:1603-1626`): `CARLOS_CONTROL_SOCKET` only when the control listener bound, `CARLOS_PURGE`, `CARLOS_REGION` only when the box has a region, and `CARLOS_HOST`. `CARLOS_ADMIN_TOKEN` and `CARLOS_INSTANCE_KEY` come from `<host>.env`, not the composer. The app reads those by name and treats a missing `CARLOS_REGION` as region `home` (preflight N7). `TestMoveprobeLaunchContract` starts the binary exactly as the backend does (`--socket`, `--db`, the agent variables) and asserts `/ready` answers on the socket. |
| 47 | - First boot: writes `files/instance.key` (32 random bytes, hex) beside the database; every boot: verifies the file exists and matches the row `meta(key='instance_key')` in the database; mismatch → the readiness path answers 500 with `key mismatch`. |
| 48 | - Requires `MOVEPROBE_BUCKET`, `MOVEPROBE_REGION`, `MOVEPROBE_ENDPOINT`, `MOVEPROBE_ACCESS_KEY`, `MOVEPROBE_SECRET_KEY` (an `objstore` binding with prefix `MOVEPROBE`) and writes `probe/<boot id>.txt` at boot, reading it back on `/ready`; requires `MOVEPROBE_SECRET` (a sealed secret) and answers 500 on `/ready` when absent. |
| 49 | - Registers, through the instance control socket (`CARLOS_CONTROL_SOCKET`), a declared schedule `every-minute` → `/tick` and, on `POST /arm-once?in=<seconds>`, a once-timer `once-<n>` → `/once`; `/tick` and `/once` insert rows `(kind, name, due_at, fired_at, region)`. |
| 50 | - A background loop inserts a row `(kind='bg', seq, region)` every 500 ms; a forked worker (`exec.Command(os.Args[0], "worker")` with `Setsid`) inserts `(kind='worker', seq, region)` every second and ignores SIGTERM for 30 s (Review Focus 1). |
| 51 | - `MOVEPROBE_FAIL_READY_IN=<region>[,after-write]`: in the named region, the **before-write** mode (no suffix) suppresses every mutation on this box from the first instruction: the background loop, the forked worker, the `/tick` and `/once` handlers and the control-socket registrations all return without writing, and `/ready` answers 500, so the target's database, files and timers digests stay exactly what was installed and the rollback is clean (the readiness checks after a spawn, the wake's `waitHealthyCtx` at `backend_exec.go:1335` (GET `/`, any status below 500, `waitHealthy` at `:2344`) and the claim probe's `claimProbeSocketReady` at `internal/host/claimready_linux.go:23-41` (2xx only), would otherwise see rows written before the refusal); the **after-write** mode writes one `bg` row (and lets the worker write) and then fails `/ready`, so the rollback is dirty with a known row to find on the source. |
| 52 | - `/ready`: 200 when the key verifies, the object reads back and the secret is present, and the failure switch does not apply. |
| 53 | - `/report`: JSON with the row counts per kind and region, the once-timers fired (name, count, due_at), the key fingerprint, the object body, `has_secret`. |
| 54 | |
| 55 | Tests: `TestMoveprobeWorkerOutlivesMain` (start the binary in a temp cgroup-less dir, kill the main process, the worker's pid is alive and still writing; the test then kills it); `TestReadyFailsWhereTold` (both modes: before-write leaves the database's row count and the timers table unchanged after two seconds of running; after-write adds exactly the rows the mode promises); `TestReportCountsRowsByRegion`; `TestMoveprobeLaunchContract`. |
| 56 | |
| 57 | - [ ] **Steps 1 to 5**. Commit: `git commit -m "cmd/moveprobe: the rehearsal probe app (§14.4)" ...` |
| 58 | |
| 59 | --- |
| 60 | |
| 61 | ### Task 2: Cross-version tests |
| 62 | |
| 63 | **Files:** modify `internal/console/apiinstancemove_test.go`, `internal/host/systemapply_test.go`, `internal/provision/move_test.go`. |
| 64 | |
| 65 | - `TestDoorRefusesBelowTheFloor`: an edges file with `moves_protocol: 0` (a pre-floor box) in either region → 409 naming the floor; with the field absent → the same. |
| 66 | - `TestSystemApplyRefusesAPreFloorBinaryOnAParticipant`: a box whose registry holds a held row (`hold_op` set) refuses any `carlos system apply` or rollback (§11 "any downgrade of a box that is a participant in a held operation is refused"); the applier reads the registry at `/data/registry.db` through the same path the agent uses. |
| 67 | - `TestOldConsoleAbortAllowlist`: `MoveAbort` from a v0.22 console (the current `provision.go:1027-1035` allowlist, reproduced as a test double) refuses every new state (`landing`, `committed`, `rolling_back`, `reverting`, `stuck`), per P0's rule. |
| 68 | |
| 69 | - [ ] Commit: `git commit -m "moves: cross-version tests: the floor at the door and at activation, the old abort allowlist (§14.3)" ...` |
| 70 | |
| 71 | --- |
| 72 | |
| 73 | ### Task 3: The runbook |
| 74 | |
| 75 | **Files:** create `docs/superpowers/plans/2026-10-06-moves-phase1-rehearsal.md`. |
| 76 | |
| 77 | The runbook has one section per run in §14.4's order; each section is a checklist an operator ticks: |
| 78 | |
| 79 | **Preconditions (once):** |
| 80 | - [ ] P-1 The Phase 1 release is rolled dormant: consoles first, then every box (`carlos status` shows the build; `control/edges/*.json` all carry `"moves_protocol":1`). |
| 81 | - [ ] P-2 §15 step 5 applied: `sibling_staging_arns` and the versioned-object grants (Paul, scoped `tofu apply -target`); verify with T10's probe through `carlos move status --host <h>` after the first request (T10 passes). |
| 82 | - [ ] P-3 `control/moves-protocol.json` written create-only: `{"min":1,"min_release":"<the rolled version>"}` (an `aws s3api put-object --if-none-match '*'` from an operator shell with the deployment profile); `carlos system apply` of an older build on dev now refuses. |
| 83 | - [ ] P-4 `moveprobe` shipped as app `moveprobe` under the ops account with a store binding `MOVEPROBE`, a sealed secret `MOVEPROBE_SECRET`, a declared schedule, and `moves.ready_path` unset (`/` serves `/ready`'s answer). |
| 84 | - [ ] P-5 Every rehearsal host is adopted (claim Plan 5's adoption has run: `control/hosts/<host>.json` exists with `op.kind: serving`). This needs C5 merged and the #232 ownership backfill run for every rehearsal host: adoption happens only inside that backfill, never from the move door or the recovery pass, and merging the claim branches alone does not adopt anything. |
| 85 | - [ ] P-6 The switch is on for this deployment: `CARLOS_MOVES_PHASE1=on` on the console and every box (index D13), with the copy record approved and the default flipped in one commit before any run on the flagship. Run 4 (declare-lost) needs claim Plan 8b merged and rolled (the roster, census, release identity and guard, the termination fence, the real declare-lost and readmit) and `CARLOS_MOVES_DECLARE_LOST` is never set outside tests; until 8b is rolled, Run 4 is skipped and the series continues with Run 5, and the loss run is repeated on dev once 8b lands. |
| 86 | |
| 87 | **Run 1, dev IE → uk, done:** |
| 88 | - [ ] S-1 `carlos instances create --app moveprobe --host mp1-<date>.<domain> --region ie` (a fresh name; Review Focus 3); wake it (`curl https://<host>/`), `POST /arm-once?in=600`, note `/report`. |
| 89 | - [ ] S-2 `carlos move --host <host> --to uk --manual-cut`; wait for M-C12; `carlos move status --host <host>` shows every T check passing and `attempts_admitted ≥ 1`. |
| 90 | - [ ] S-3 `carlos move cut --host <host>`; expect M-C5, M-C6, M-C7; exit 0. |
| 91 | - [ ] R-1 `/report` on uk: the key fingerprint equals S-1's; the object reads back; `has_secret` true. |
| 92 | - [ ] R-2 the `bg` and `worker` row counts are at least S-1's and every `seq` is contiguous (no missing row). |
| 93 | - [ ] R-3 the `every-minute` rows continue on uk with region `uk` from the first tick after the open. |
| 94 | - [ ] R-4 the once-timer fired exactly once, on uk, with S-1's `due_at` (Review Focus 2). |
| 95 | - [ ] R-5 `aws s3api list-object-versions --bucket <ie data bucket> --prefix instances/<host>/` is empty after the source park acknowledged (`carlos move status` detail shows no obligation; the log has `source_parked` and `done`); `parked/<inc>/<host>/<ret>/` holds the archive. |
| 96 | - [ ] R-6 the move log has no `stray` page; the ie box's journal shows `ReconstructStopEvidence` killed the worker (the `tree` step OK) (Review Focus 1). |
| 97 | - [ ] R-7 (not for this run). |
| 98 | |
| 99 | **Run 2, dev forced clean rollback:** `carlos env set --app moveprobe MOVEPROBE_FAIL_READY_IN=ie` (the before-write mode on the target, moving uk → ie: nothing is written there, so the digests equal the install's); request and cut; expect M-C8 then M-C9, exit 3; checks R-1 to R-6 on uk (the source), plus: the uk row is `ready` again, the ie bucket's `instances/<host>/` and `moves/<host>/<mv>/` are empty after `cleaned`, `rollback.dirty == false` in the log's `released` event, and the `released` event's three digests equal `landing.installed_db_sha256`, `source.files.plain` and `source.timers_sha256`. |
| 100 | |
| 101 | **Run 3, dev forced dirty rollback:** `MOVEPROBE_FAIL_READY_IN=ie,after-write`; expect exit 3; R-7: the source's `/report` holds a `bg` row with region `ie` written on the target during its probe (Review Focus 4); the log's `released` event has `dirty: true`, `returned` names the lane snapshot; `moves/<host>/<rb>/` in the uk bucket is parked after the horizon. |
| 102 | |
| 103 | **Run 4, dev declare-lost (needs P-6's Plan 8b clause):** request a move to a dev box Paul designates as disposable; after the flip, stop that box through the AWS console (dev only); `carlos move resolve --host <host> --declare-lost <box> --instance-id <id>`; type the host name at M-C18; expect the clean rollback to complete on the source, exit 3; the loss record names the box, `fenced_at`, the declared timers; restart the box and watch it quarantine the host's local state before anything else (S-9, Review Focus 5). |
| 104 | |
| 105 | **Run 5, flagship, scoped approval:** runs 1 to 3 again on the flagship with a `moveprobe` instance, each with Paul's explicit go in the discussion thread. |
| 106 | |
| 107 | **Run 6, a disposable ips instance in `bdn`:** `carlos instances create` of an ips instance Tony designates, run 1 only, with Tony's check that the instance serves after the open. |
| 108 | |
| 109 | **Run 7, kpmg:** run 1 only, with Paul's approval, in a window agreed with the customer; the cut budget stays 120 s; the runbook's `offline_s` from the `done` event is the number reported back. |
| 110 | |
| 111 | **After every run:** attach the `done`/`rolled_back` log event's timings, `/report`, and the bucket listings to the discussion; a failed check stops the series until its cause is fixed and the run repeated. |
| 112 | |
| 113 | - [ ] Commit: `git commit -m "docs: the Phase 1 rehearsal runbook in M12's order (§14.4, §15)" ...` |
| 114 | |
| 115 | --- |
| 116 | |
| 117 | ### Task 4: Gate |
| 118 | |
| 119 | - [ ] Full gate, cross-compile (`cmd/moveprobe` builds on darwin and windows; the `Setsid` attribute is behind `//go:build !windows`). |
| 120 | |
| 121 | ## Self-review notes |
| 122 | |
| 123 | Coverage: §14.3 (Task 2), §14.4 (Tasks 1, 3), §15 steps 2 to 7 (the runbook's preconditions and order; step 1 and step 8 are done per the spec). The infra change of step 5 is Paul's scoped work and is a precondition, not a task. |
| 124 | |
amadan