| 1 | # Moves Phase 1: the rehearsal runbook |
| 2 | |
| 3 | This is the checklist an operator follows to rehearse Phase 1 moves (spec |
| 4 | §14.4, rollout §15). It covers eight runs in the spec's order: five on dev |
| 5 | (Runs 1, 1b, 2, 3 and 4; Run 1b is a move into home), one on the flagship |
| 6 | with the probe app, one on a disposable customer-shaped instance, and one |
| 7 | on a real customer instance. |
| 8 | |
| 9 | **No run in this runbook can execute today.** Every run waits for P-0 |
| 10 | below. Each run also has its own blockers, listed at its top. Never run one |
| 11 | without Paul's go in the amadan thread. Steps that change a deployment |
| 12 | (publishing the floor, switching moves on, stopping a box) each have their |
| 13 | own **Paul's scoped go** step just before them: a go in the amadan thread |
| 14 | that names the deployment and exactly what the step changes. A go for one |
| 15 | step or one run does not cover another. |
| 16 | |
| 17 | The probe app is `moveprobe`. How to build it, ship it and use it is in |
| 18 | [`cmd/moveprobe/README.md`](../../../cmd/moveprobe/README.md). This runbook |
| 19 | does not repeat it. |
| 20 | |
| 21 | Every command, flag, event, field and key below was checked against main, |
| 22 | and moveprobe's against this branch. |
| 23 | The Sources section at the end gives file and line for each. |
| 24 | |
| 25 | --- |
| 26 | |
| 27 | ## How to read this runbook |
| 28 | |
| 29 | ### Placeholders |
| 30 | |
| 31 | | placeholder | what it is | |
| 32 | |---|---| |
| 33 | | `<console>` | the deployment's console origin, `https://...`, as `carlos auth login` stored it | |
| 34 | | `<ops>` | the ops account's sqid, as `--account` takes it | |
| 35 | | `<account>`, `<app>` | a customer run's account sqid and app (Runs 6 and 7) | |
| 36 | | `<host>` | the run's host, `<label>.<instance domain>` | |
| 37 | | `<deploy-bucket>` | the deployment bucket (`CARLOS_DEPLOYMENT_BUCKET` in a box's `/etc/carlos/host.env`) | |
| 38 | | `<data-bucket-R>` | region R's data bucket: `data_bucket` in that region's box edges file | |
| 39 | | `<profile>` | the deployment's named AWS profile. Always name one. There are no default credentials. | |
| 40 | | `<move>` | the move id the console gave the request (`id` in the flat view) | |
| 41 | | `<inc>` | the instance incarnation (`detail.move.incarnation`) | |
| 42 | | `<pr-R>` | the region moveprobe reports on region R's box: `home` if that box has no `CARLOS_REGION` (its edges file has no `placement`), else its `CARLOS_REGION`, the placement code. `<pr-from>` and `<pr-to>` are the run's two. | |
| 43 | |
| 44 | ### The probe's region is not always the region's name |
| 45 | |
| 46 | moveprobe reports the region its box gives it, not the name the CLI and |
| 47 | the console use. On a home box, which has no `CARLOS_REGION`, it reports |
| 48 | `home`, while `carlos move` still says `ie` (or whatever |
| 49 | `CONSOLE_HOME_REGION` is). Before each deployment's first run, write down |
| 50 | `<pr-R>` for every rehearsal region: read `CARLOS_REGION` in the box's |
| 51 | `/etc/carlos/host.env`, or `placement` in its edges file, and confirm with |
| 52 | [report]'s `region` and `region_from` on any moveprobe host there. Every |
| 53 | [report] region check and every `MOVEPROBE_FAIL_READY_IN` value uses |
| 54 | `<pr-R>`, never the CLI's name. |
| 55 | |
| 56 | ### Read paths |
| 57 | |
| 58 | Each check says where to read it. |
| 59 | |
| 60 | - **[CLI]**: the sentence `carlos move` prints and its exit code. Exit codes: |
| 61 | 0 done; 3 rolled back or aborted; 4 waiting (ready for a manual cut, |
| 62 | stuck, a release past its deadline, relocating, or an outcome not |
| 63 | recorded); 5 refused; 1 the console could not be reached or read; 2 a |
| 64 | usage error. |
| 65 | - **[detail]**: the operator's detail answer. No `carlos` command prints it |
| 66 | yet, so read it with your own login token. Set `ACCT` and `APP` for the |
| 67 | run: `<ops>` and `moveprobe` for Runs 1 to 5, `<account>` and `<app>` |
| 68 | for Runs 6 and 7. |
| 69 | |
| 70 | ```sh |
| 71 | CONSOLE=<console> |
| 72 | ACCT=<ops> APP=moveprobe |
| 73 | TOKEN=$(jq -r --arg c "$CONSOLE" '.credentials[] | select(.console == $c) | .token' ~/.carlos/credentials) |
| 74 | detail() { curl -fsS -H "Authorization: Bearer $TOKEN" "$CONSOLE/api/cli/apps/$ACCT/$APP/instances/$1/move?detail=1"; } |
| 75 | detail <host> | jq . |
| 76 | ``` |
| 77 | |
| 78 | The flat fields (`state`, `phase`, `id`, `last`) sit at the top. The |
| 79 | rest is under `detail`: `move` (the record's journal), `claim`, |
| 80 | `target_report`, `source_report`, `evidence`, `log` (keys only), |
| 81 | `timings` (after the close) and `region`. **`detail.move` and |
| 82 | `detail.evidence` are gone once the move closes**, and a mutation waits |
| 83 | until it closes by default. So every run that needs them starts the |
| 84 | capture loop below before its cut and reads them from the capture file |
| 85 | afterwards. |
| 86 | - **[capture]**: every detail answer of one move, saved as it happens. |
| 87 | Start it in a second terminal once the request has reached ready, before |
| 88 | the cut. It stops by itself once no move is in flight. With a third |
| 89 | argument it also saves, each time round, the host claim's |
| 90 | `obligations` from the deployment bucket (`control/hosts/<host>.json`, |
| 91 | plain JSON). The detail answer does not carry them, and they are gone |
| 92 | once the move closes, so this is the only way to keep a park's horizon |
| 93 | and retirement id. Run 3 needs it (R-8): |
| 94 | |
| 95 | ```sh |
| 96 | capture() { # capture <host> <detail-file> [<claim-file>] |
| 97 | while :; do |
| 98 | if d=$(detail "$1"); then |
| 99 | printf '%s\n' "$d" >> "$2" |
| 100 | if [ -n "$3" ]; then |
| 101 | aws s3 cp --profile <profile> "s3://<deploy-bucket>/control/hosts/$1.json" - \ |
| 102 | | jq -c '.obligations // []' >> "$3" |
| 103 | fi |
| 104 | [ "$(printf '%s' "$d" | jq -r .state)" = none ] && break |
| 105 | fi |
| 106 | sleep 2 |
| 107 | done |
| 108 | } |
| 109 | capture <host> run-<n>-detail.jsonl # Runs 1, 2, 4, 5 |
| 110 | capture <host> run-3-detail.jsonl run-3-claim.jsonl # Run 3 |
| 111 | # later: the last rollback journal the move showed |
| 112 | jq -s '[.[] | .detail.move.rollback | select(. != null)] | last' run-<n>-detail.jsonl |
| 113 | ``` |
| 114 | |
| 115 | - **[deploy]**: the deployment bucket, read with `--profile <profile>`. |
| 116 | Each move keeps its reports, stop evidence and log under |
| 117 | `moves/$ACCT/$APP/<host>/<move>/`: `source.json`, `target.json`, |
| 118 | `stopped-source.json`, `stopped-target.json` and `log/<identity>.json`. |
| 119 | The claim is `control/hosts/<host>.json`, each box's edges file |
| 120 | `control/edges/<box>.json`, each box's identity record |
| 121 | `control/boxes/<box>.json`, and the floor `control/moves-protocol.json`. |
| 122 | |
| 123 | ```sh |
| 124 | P="moves/$ACCT/$APP/<host>/<move>/" |
| 125 | aws s3 ls --profile <profile> "s3://<deploy-bucket>/${P}log/" |
| 126 | aws s3 cp --profile <profile> "s3://<deploy-bucket>/${P}log/done.json" - | jq . |
| 127 | ``` |
| 128 | |
| 129 | The roster, `control/host-actors.json`, is a signed envelope, |
| 130 | `{"payload": <base64 JSON>, "sig": ...}`. Decode the payload to read it |
| 131 | (this does not check the signature; the console does). A box's |
| 132 | termination fence is a separate object, |
| 133 | `control/box-fences/<enrollment>.json`, using the enrollment from the |
| 134 | roster. No object means no fence: |
| 135 | |
| 136 | ```sh |
| 137 | roster() { aws s3 cp --profile <profile> s3://<deploy-bucket>/control/host-actors.json - | jq -r .payload | base64 -d; } |
| 138 | roster | jq '.boxes["<box>"] | {state, enrollment, loss_ids, quarantine_revision}' |
| 139 | aws s3api head-object --profile <profile> --bucket <deploy-bucket> \ |
| 140 | --key "control/box-fences/$(roster | jq -r '.boxes["<box>"].enrollment').json" # 404: no fence |
| 141 | ``` |
| 142 | |
| 143 | - **[data]**: a region's data bucket, read with `--profile <profile>`. The |
| 144 | instance's replica is under `instances/<host>/` (the box's |
| 145 | `CARLOS_HIBERNATE_PREFIX`, `instances` in production), a move's staging |
| 146 | lane under `moves/<host>/<move>/`, and parked objects under |
| 147 | `parked/<inc>/<host>/<retirement>/` with a `manifest.json`. These buckets |
| 148 | are versioned, so list versions, not objects: |
| 149 | |
| 150 | ```sh |
| 151 | aws s3api list-object-versions --profile <profile> --bucket <data-bucket-R> --prefix "instances/<host>/" |
| 152 | ``` |
| 153 | |
| 154 | - **[report]**: moveprobe's own report, with the secret from your shell: |
| 155 | |
| 156 | ```sh |
| 157 | curl -fsS -H "Authorization: Bearer $MOVEPROBE_SECRET" https://<host>/report | jq . |
| 158 | ``` |
| 159 | |
| 160 | - **[box]**: on a box, the agent's journal (`journalctl -u carlos-edge`), |
| 161 | `/etc/carlos/host.env`, and files under the data directory (`/data`). |
| 162 | |
| 163 | ### Log events |
| 164 | |
| 165 | Each transition writes one log object, keyed by its identity. The ones the |
| 166 | checks use: |
| 167 | |
| 168 | | identity (the key) | event (the `event` field) | when | |
| 169 | |---|---|---| |
| 170 | | `requested` | `requested` | the request was journaled | |
| 171 | | `ready` | `ready` | the target's checks passed | |
| 172 | | `cutting-requested` | `cut_requested` | the cut was asked for | |
| 173 | | `cutting-<phase>` | `<phase>` | `admitted`, `stopping`, `stopped`, `staged`, `archived` | |
| 174 | | `cutting-flipped` | `flipped` | the record now names the target | |
| 175 | | `landing-<phase>` | `<phase>` | the target's landing steps | |
| 176 | | `probe_attempt-<n>` | `probe_attempt` | the landing's n-th readiness probe | |
| 177 | | `committed` | `committed` | the landing committed | |
| 178 | | `committed-opened` | `opened` | the target opened (`offline_s` in its fields) | |
| 179 | | `committed-source_parked` | `source_parked` | the source parked what it held | |
| 180 | | `rolling_back` | `rolling_back` | a rollback started (`reason` in its fields) | |
| 181 | | `rolling_back-target-<phase>` | `<phase>` | `released`, `returning`, `returned` (a dirty rollback only), `cleaning`, `cleaned` | |
| 182 | | `rolling_back-target-lost` | `declared_lost` | the target was declared lost | |
| 183 | | `rolling_back-source-<phase>` | `<phase>` | the source's recovery, up to `recovered` and `opened` | |
| 184 | | `stuck-<epoch>`, `retry-<epoch>` | `stuck`, `retry` | a stuck move and its retry | |
| 185 | | `done`, `rolled_back`, `aborted` | the same | the close; `done` and `rolled_back` carry the timings | |
| 186 | |
| 187 | Every entry's `fields` carry `from`, `to` and, once known, `final_txid`, |
| 188 | `db_sha256`, `timers_sha256`, `files_plain`, `installed_db_sha256`, |
| 189 | `release`, `env_report` and `cut_budget_s`. The close's timings are |
| 190 | `prepare_s`, `cut_s`, `cut_budget_s`, `land_s`, `land_budget_s`, |
| 191 | `offline_s`, `rollback_s` and `release_budget_s`. There is no `stray` |
| 192 | event and no `dirty` field in the log. |
| 193 | |
| 194 | --- |
| 195 | |
| 196 | ## P-0: what every run waits for |
| 197 | |
| 198 | Nothing below P-0 runs until every item here is true. Paul confirms P-0 per |
| 199 | deployment before its first run. |
| 200 | |
| 201 | - [ ] **Claim Plan 12 merged.** Without it the landing, the target's half |
| 202 | of a rollback and the source's recovery are unarmed in production, so a |
| 203 | move with the switch on stops the source, flips, and then has nothing to |
| 204 | land it, roll it back or recover it. |
| 205 | - [ ] **Moves Plan 5 Task 9b, Plan 4 Tasks 9 and 10 (the box half), and |
| 206 | Plan 5 Task 8 merged.** They wire moves onto claim Plan 12. Run 4 also |
| 207 | needs claim Plan 12's loss-bound admission. |
| 208 | - [ ] **The moves cut guard merged.** It is on main (b04ff852, Paul's |
| 209 | ruling of 2026-10-11). The source refuses a cut unless the target |
| 210 | reports its landing armed (the `T13` check), and records why a cut did |
| 211 | not start. |
| 212 | - [ ] **Moves' post-Plan 12 branch merged** (`moves-phase1-5f`). It brings |
| 213 | the wake source, the recovery owners, the loss rules, the root held |
| 214 | release (T7 stages, `configured` publishes, through root's control |
| 215 | socket) and the C8 files-manifest provider. |
| 216 | - [ ] **For any run that touches the home region: the home-region parks fix |
| 217 | part 1 merged** (#236, on main as 359e8ac2) **and claim Plan 13's home |
| 218 | naming** (a0806e4c, follow-ups 2ad5412c). Part 1 covers moves out of |
| 219 | home. **Any run that moves into home also needs part 2** |
| 220 | (`2026-10-15-moves-home-region-part2.md`) merged and rolled, boxes |
| 221 | first, then consoles. The door refuses moves into home without host |
| 222 | claims (MV-N510). A home box arms only when the claim grants it the |
| 223 | `@home` alias, which needs the signed fleet home designation. Plan 13's |
| 224 | rollout order is binding: every roster reader on a Plan 13 build first, |
| 225 | then publish the designation, then turn the flag on. Older binaries |
| 226 | cannot read a roster that carries `home_region`. For moves into home, |
| 227 | the whole order is: |
| 228 | 1. Plan 13 on every box and console. Check each box reads `current` |
| 229 | with `carlos system status`. Plan 13's own order applies inside |
| 230 | this step. |
| 231 | 2. Part 2 merged (dormant), released, and rolled **boxes first, then |
| 232 | consoles**. A console rolled first could admit a move into a home box |
| 233 | that still fails T3, and that move waits at `preparing` until an |
| 234 | operator aborts it. |
| 235 | 3. The IAM mesh, including the home data bucket (P-2), applied by Paul |
| 236 | with a scoped `tofu apply -target`. |
| 237 | 4. Claim activation (#250), which publishes the signed designation. |
| 238 | Until then a home box never arms, and the door refuses claim-mode |
| 239 | moves with MV-N177. |
| 240 | 5. `CARLOS_MOVES_PHASE1` per run, as P-7 says. |
| 241 | - [ ] **The claim's activation checklist (#250) done for this deployment.** |
| 242 | Among its items: |
| 243 | - the one-time files-manifest backfill for every host older than the |
| 244 | files publisher. Until it has run for a host, that host refuses to wake |
| 245 | (`ErrNoEvidence`), by design; |
| 246 | - claim Plan 15's admitted-generation record, so hosts with history past |
| 247 | the eight-admission window can prove their file origins. Until then |
| 248 | those hosts refuse; |
| 249 | - claim Plan 14 (#280), so that root trusts no service-owned state; |
| 250 | - `CARLOS_RELEASE_PUBKEY` readable by instance control |
| 251 | (`/etc/carlos/host.env`), or every held release is refused; |
| 252 | - with `CARLOS_MOVES_PHASE1` on, a usable `CARLOS_SECRETS_KEY` on every |
| 253 | box. Without it the moves source refuses to open and the box serves |
| 254 | nothing. Its log names the cause once per start. |
| 255 | - [ ] **All of the above released and rolled, consoles first, then every |
| 256 | box.** Check every box with `carlos system status --console <console>`: |
| 257 | each reads `current` on the new version. |
| 258 | - [ ] **Each deployment's home region recorded:** `CONSOLE_HOME_REGION` in |
| 259 | the console's environment. A box with no `CARLOS_REGION` in its |
| 260 | `host.env` publishes no `placement` in its edges file: it is a home box. |
| 261 | - [ ] **Dev's home region has two boxes: wake rehearsal apps on one.** While |
| 262 | dev's claim is off, both home boxes (`carlos-dev-ie` and |
| 263 | `carlos-dev-ie-b`) take every unregioned record. A wake that switches box |
| 264 | can serve a stale local DB (amadan #286; `activator.go`, "only while a |
| 265 | region has one box"). So rehearsal apps in dev's home region are woken on |
| 266 | one box only, or the run waits for dev's claim. #286 ruled this a |
| 267 | dev-only hazard, accepted. During a window under the procedure in the |
| 268 | next item, `carlos-dev-ie-b` is stopped, so this holds by itself. |
| 269 | - [ ] **One placementless box per deployment.** For each deployment, list |
| 270 | `control/edges/` and record that exactly one edges file has no |
| 271 | `placement`: |
| 272 | |
| 273 | ```sh |
| 274 | for k in $(aws s3api list-objects-v2 --profile <profile> --bucket <deploy-bucket> \ |
| 275 | --prefix control/edges/ --query 'Contents[].Key' --output text); do |
| 276 | printf '%s %s\n' "$k" "$(aws s3 cp --profile <profile> s3://<deploy-bucket>/$k - | jq -r '.placement // "(none)"')" |
| 277 | done |
| 278 | ``` |
| 279 | |
| 280 | S1 and T8 refuse a move into **or out of** home while two do, whether or |
| 281 | not the second box is running, because an edges file outlives its box. |
| 282 | They count every edges file per placement, fresh or not. A retained file |
| 283 | below the floor makes the door refuse with MV-N5. |
| 284 | |
| 285 | On dev, `carlos-dev-ie-b` makes two. Ruling R2 (option a) applies to |
| 286 | every dev run that touches home, Run 1 included. Each window needs |
| 287 | **Paul's scoped go** in the amadan thread, naming dev and |
| 288 | `carlos-dev-ie-b`. The procedure, as the ruling gives it: |
| 289 | |
| 290 | 1. stop `carlos-dev-ie-b` (the instance, or at least its agent and console units, so it neither writes status nor serves); |
| 291 | 2. delete `control/edges/carlos-dev-ie-b.json` from the deployment bucket; |
| 292 | 3. wait until its last `status/boxes/carlos-dev-ie-b.json` is more than 15 minutes old by its `generated_at`; |
| 293 | 4. before **each** run, verify the fleet's evidence is complete: every box with a status snapshot fresher than 15 minutes has an edges file, every edges file's box has a status snapshot, and exactly one edges file has no `placement`. The door's own confirmation is the run's **S-2**, the move request itself. Only a move request runs `selectMoveBoxes` and the floor check: `carlos move status` (`handleAPIInstanceMoveStatus`) reads the record and claim only, and S-1 only wakes the app, so neither confirms anything here. If S-2 answers "the fleet's edges files could not all be read" or MV-N5, the window is not ready. Both refusals come before any claim read or write, so nothing was started: fix the evidence and repeat S-2; |
| 294 | 5. after the window, start the box again. Its edges pass rewrites the file on boot. No move touching home runs until the window is reopened by steps 1 to 4. |
| 295 | |
| 296 | Step 3's wait matters. If the edges file is deleted while the box's |
| 297 | status snapshot is still fresh, the door reads the fleet's evidence as |
| 298 | incomplete and refuses every move. |
| 299 | - [ ] **Rehearsal boxes keep instances flat.** Moves refuse a nested |
| 300 | instance. Set `CARLOS_LAYOUT_DEFAULT=flat` in each rehearsal box's |
| 301 | `host.env`. Pre-flight check, before any request: `carlos status |
| 302 | --layout --console <console>` (the layout each host is actually on, |
| 303 | box by box) shows every rehearsal host's row as `flat`, and no |
| 304 | rehearsal box is marked "(nested by default)". `carlos system layout |
| 305 | status` is not this check: it shows only the recorded intent. Once a |
| 306 | run's request exists, S-2 also checks `detail.source_report.s1.ok`, |
| 307 | the source's own refusal of a nested instance. |
| 308 | - [ ] **Never set `CARLOS_MOVES_PHASE1=on` anywhere before P-0 is true.** |
| 309 | |
| 310 | --- |
| 311 | |
| 312 | ## Preconditions, once per deployment |
| 313 | |
| 314 | ### P-1 The dormant release |
| 315 | |
| 316 | - [ ] The release carrying P-0 is rolled with `CARLOS_MOVES_PHASE1` unset |
| 317 | everywhere: consoles first, then boxes. |
| 318 | - [ ] [deploy] Every box's `control/edges/<box>.json` has |
| 319 | `"moves_protocol": 1`. |
| 320 | |
| 321 | ### P-2 Staging grants (§15 step 5, scoped tofu by Paul) |
| 322 | |
| 323 | - [ ] Every rehearsal region's `sibling_staging_arns` lists every other |
| 324 | rehearsal region's data bucket, and each box has the versioned-object |
| 325 | grants (`s3:ListBucketVersions`, `s3:GetObjectVersion`, |
| 326 | `s3:DeleteObjectVersion` on its own bucket; `s3:GetObjectVersion` for |
| 327 | the console). |
| 328 | - [ ] **The home data bucket is in the mesh.** Every named region's |
| 329 | `sibling_staging_arns` lists the home data bucket's ARN, and the home's |
| 330 | lists every named region's data bucket. Record the home data bucket's |
| 331 | name (`data_bucket` in the home box's edges file, the one with no |
| 332 | `placement`): `____`. Check that it is not the deployment bucket |
| 333 | (`<deploy-bucket>`). Without the mesh, a move into home fails the |
| 334 | source's lane round with `AccessDenied` (logged), or T10 fails on the |
| 335 | home box. |
| 336 | - [ ] Check after the first request reaches ready: [detail] |
| 337 | `detail.target_report.checks.T10.ok` is `true` (the lane probe). |
| 338 | |
| 339 | ### P-3 The protocol floor |
| 340 | |
| 341 | Publish the floor only after **every box and the console run a tagged |
| 342 | `vX.Y.Z` release** at or above `min_release`. Once a floor is armed, any |
| 343 | build that is not `vX.Y.Z` (a SHA build, `dev`, a prerelease) is refused at |
| 344 | activation, and the door treats that box as below the floor. A box never |
| 345 | lowers its local floor on its own. If dev keeps rolling SHA builds, dev's |
| 346 | runs go without a floor, and P-3 is done on the flagship only, before |
| 347 | Run 5 (§15 step 6). |
| 348 | |
| 349 | - [ ] **Paul's scoped go** in the amadan thread to publish the floor on |
| 350 | this deployment, naming the deployment and `min_release`. Nothing below |
| 351 | runs without it. |
| 352 | - [ ] Write it create-only from an operator shell. No `carlos` command |
| 353 | writes it (a gap to note on an amadan discussion). |
| 354 | |
| 355 | ```sh |
| 356 | printf '{"min":1,"min_release":"vX.Y.Z"}' > floor.json |
| 357 | aws s3api put-object --profile <profile> --bucket <deploy-bucket> \ |
| 358 | --key control/moves-protocol.json --body floor.json --if-none-match '*' |
| 359 | aws s3 cp --profile <profile> s3://<deploy-bucket>/control/moves-protocol.json - |
| 360 | ``` |
| 361 | |
| 362 | - [ ] [deploy] Every box's edges file shows `moves_floor_min_release` equal |
| 363 | to the floor's `min_release`, and either `"prev_release": "none"` or |
| 364 | `prev_allowed_for` equal to `min_release`. A box that does not is below |
| 365 | the floor for the door (MV-N5). |
| 366 | |
| 367 | ### P-4 Ship moveprobe |
| 368 | |
| 369 | - [ ] Follow `cmd/moveprobe/README.md` ("Build" and "Deploy") under the ops |
| 370 | account: ship, enable instances with private storage off, create the |
| 371 | store with `--env-prefix MOVEPROBE` and no `--environment`, set |
| 372 | `MOVEPROBE_SECRET`, and set the `every-minute` schedule. |
| 373 | - [ ] Keep `MOVEPROBE_SECRET`'s value in your shell. `/report` and |
| 374 | `/arm-once` need it. |
| 375 | - [ ] Leave `moves.ready_path` unset. A move then probes `/`, which answers |
| 376 | as `/ready` does. |
| 377 | |
| 378 | ### P-5 Rehearsal hosts, created before the claim |
| 379 | |
| 380 | A host created while `CARLOS_HOST_CLAIM` is on keeps its data under a |
| 381 | per-incarnation path, and the move door refuses it with MV-N167: "the |
| 382 | instance keeps its data under a per-incarnation path, which moves cannot |
| 383 | cut yet". So every rehearsal host is created **before** the claim goes on |
| 384 | for its deployment (Paul, N2(a)). |
| 385 | |
| 386 | - [ ] Before `CARLOS_HOST_CLAIM` goes on, create one moveprobe host per |
| 387 | planned run, in the region that run starts from, plus two spares: |
| 388 | |
| 389 | ```sh |
| 390 | carlos instances create --app moveprobe --account <ops> --host <label>.<instance domain> --region <region> |
| 391 | ``` |
| 392 | |
| 393 | On dev, create at least one spare in `ie`. If Run 1b fails, its repeat |
| 394 | is Run 1 and then Run 1b on a fresh host that starts in `ie`, and no |
| 395 | host can be created once the claim is on. |
| 396 | |
| 397 | On the flagship, also create Run 6's disposable customer-shaped |
| 398 | instance now. |
| 399 | - [ ] Record the list of hosts, with the run each one is for, in the |
| 400 | deployment's rehearsal record (outside this repo). **Never reuse a |
| 401 | host.** A host that has run once is spent, whatever the outcome. The one |
| 402 | exception is Run 1b, which moves Run 1's host back into home (Ruling |
| 403 | R6). |
| 404 | - [ ] After the claim is on, check each host is adopted: [deploy] |
| 405 | `control/hosts/<host>.json` exists, `holder.location` is |
| 406 | `instances/<host>` (not `instances/<host>/<inc>`), and `op.kind` is |
| 407 | `serving`. The console's claim recovery pass adopts them. |
| 408 | |
| 409 | ### P-6 The rest of the fleet's preconditions |
| 410 | |
| 411 | Paul confirms each of these per deployment before its first run. |
| 412 | |
| 413 | - [ ] **Claim on.** `CARLOS_HOST_CLAIM` is set (any non-empty value) in |
| 414 | the console's environment and every box's `host.env`. Check: [box] on |
| 415 | each box and the console. #250's activation blockers are closed. |
| 416 | - [ ] **Roster.** Each rehearsal box is an active, enrolled entry with no |
| 417 | termination fence. Check: [deploy] the decoded roster (`roster` above) |
| 418 | has `boxes.<box>.state` `active` and `boxes.<box>.enrollment` set, and |
| 419 | `control/box-fences/<enrollment>.json` does not exist. |
| 420 | - [ ] **Box identity records.** `CARLOS_REGION_HA=true` in each rehearsal |
| 421 | box's `host.env`. It takes `true`, `1` or `t`; **`on` reads as off**. |
| 422 | `CARLOS_AWS_ACCOUNT` is set in the console's environment. Check: [deploy] |
| 423 | `control/boxes/<box>.json` exists for each rehearsal box, and its edges |
| 424 | file has `"capabilities": ["region-ha-1"]`. |
| 425 | - [ ] **One eligible box per rehearsal region.** Any other box in a |
| 426 | rehearsal region runs without `CARLOS_REGION_HA`, so it publishes no |
| 427 | `region-ha-1` and is not eligible. Otherwise the door answers M-C21. |
| 428 | - [ ] **Secrets key.** `CARLOS_SECRETS_KEY` on every rehearsal box, from |
| 429 | the same key pair as the console's. Check: [deploy] each edges file's |
| 430 | `secrets_key_fp` is the same; once the switch is on, it equals |
| 431 | `secrets_key_fp` in `control/moves-console.json`; [detail] |
| 432 | `detail.target_report.checks.T2.ok` is `true`. |
| 433 | - [ ] **Staging bucket.** Each rehearsal box's edges file has |
| 434 | `data_bucket` set. |
| 435 | - [ ] **Alert sinks.** `CARLOS_ALERT_WEBHOOK` or |
| 436 | `CARLOS_ALERT_SNS_TOPIC_ARN` is set on every rehearsal box and on the |
| 437 | console. Without one, pages go nowhere. |
| 438 | - [ ] **Cgroup delegation.** Check: [box] `journalctl -u carlos-edge -b | |
| 439 | grep 'exec-child accounting'` shows `armed (delegated subtree ...)`. A |
| 440 | box without it cannot produce stop evidence and never cuts. |
| 441 | - [ ] **Flat rows and tagged releases**, as P-0 and P-3 say. |
| 442 | |
| 443 | ### P-7 The switch |
| 444 | |
| 445 | - [ ] P-0 to P-6 are done for this deployment. |
| 446 | - [ ] **The moves copy record is fully approved**: no row in |
| 447 | `docs/superpowers/plans/2026-10-06-moves-phase1-copy.md` is `pending` |
| 448 | (`grep -c '| pending |'` on it prints 0). The switch never goes on |
| 449 | while a sentence an operator could read is unapproved. |
| 450 | - [ ] **Paul's scoped go** in the amadan thread to switch moves on, naming |
| 451 | the deployment, the console and the boxes. |
| 452 | - [ ] `CARLOS_MOVES_PHASE1=on` in the environment of the console and of |
| 453 | the boxes the go names, and nowhere else, then restart them. |
| 454 | - [ ] Check: `carlos move status --app moveprobe --account <ops> --host |
| 455 | <host>` does not say "Moves are not enabled on this console." |
| 456 | |
| 457 | --- |
| 458 | |
| 459 | ## Between runs |
| 460 | |
| 461 | Before starting the next run, wait until all of these hold for the host |
| 462 | the last run used: |
| 463 | |
| 464 | - [ ] [detail] `state` is `none` (no move in flight) and `last.outcome` is |
| 465 | recorded. |
| 466 | - [ ] [detail] `detail.claim.kind` is `serving`. |
| 467 | - [ ] [deploy] the claim's `quiet_after` for the region the next move |
| 468 | goes to has passed (T12). It is set when an instance leaves a region and |
| 469 | lasts about 16 minutes. With a fresh host per run it rarely bites, but |
| 470 | check it. |
| 471 | - [ ] `MOVEPROBE_FAIL_READY_IN` is unset: `carlos env list --app moveprobe |
| 472 | --account <ops>` does not show it. |
| 473 | |
| 474 | --- |
| 475 | |
| 476 | ## The steps and checks each run uses |
| 477 | |
| 478 | Runs refer to these by number. `<from>` and `<to>` are the run's regions. |
| 479 | |
| 480 | ### Steps |
| 481 | |
| 482 | - **S-1 Wake and record.** `curl https://<host>/ready` answers `ready`. |
| 483 | Save [report] as `before-request.json`. |
| 484 | - **S-2 Request.** |
| 485 | |
| 486 | ```sh |
| 487 | carlos move --app moveprobe --account <ops> --host <host> --to <to> --manual-cut |
| 488 | ``` |
| 489 | |
| 490 | [CLI] M-C1 ("Moving <host> from <from> to <to>."), M-C2, M-C3, then |
| 491 | M-C12 ("Ready to cut. ..."), exit 4. A line "<to> cannot take this move |
| 492 | yet: <check>." is M-C4 and names the failing check. |
| 493 | [detail] at ready: every `detail.target_report.checks.T1` to `T12` has |
| 494 | `ok: true` (and `T13` once the cut guard lands); |
| 495 | `detail.target_report.attempts_admitted` is at least 1; |
| 496 | `detail.source_report.s1.ok` and `s2.ok` are `true`. Record `id` as |
| 497 | `<move>` and `detail.move.incarnation` as `<inc>`. |
| 498 | - **S-3 Record and arm.** Save [report] as `before-cut.json`. Arm the |
| 499 | once-timer so it falls due around the cut, then cut within a few |
| 500 | seconds: |
| 501 | |
| 502 | ```sh |
| 503 | curl -fsS -X POST -H "Authorization: Bearer $MOVEPROBE_SECRET" "https://<host>/arm-once?in=30" |
| 504 | ``` |
| 505 | |
| 506 | Record the answer's `name` and `due_at`. |
| 507 | - **S-4 Capture, then cut.** In a second terminal, start [capture] for |
| 508 | this run: `capture <host> run-<n>-detail.jsonl`. Then, in the first: |
| 509 | |
| 510 | ```sh |
| 511 | carlos move cut --app moveprobe --account <ops> --host <host> |
| 512 | ``` |
| 513 | |
| 514 | The cut waits until the move ends or needs an operator. The expected |
| 515 | sentences and exit code are in each run. Every run captures, so that |
| 516 | whatever the record held while the move was live (the evidence keys, |
| 517 | the rollback journal, a loss) can still be read after it closes. |
| 518 | |
| 519 | ### Checks after a commit |
| 520 | |
| 521 | - **R-1 State arrived.** [report] on the target: `region` is `<pr-to>`; |
| 522 | `file_key_sha256` and `platform_key_sha256` equal `before-cut.json`'s; |
| 523 | `object` has no `error` and its `body` names this boot and `<pr-to>`; |
| 524 | `has_secret` is `true`. `curl https://<host>/ready` answers `ready`. |
| 525 | - **R-2 No row lost.** [report] for `bg` and for `worker`: the sum of |
| 526 | `count` over all regions is at least `before-cut.json`'s, and equals the |
| 527 | highest `max_seq` (rows are numbered from 1 in one sequence per kind, so |
| 528 | any gap shows here). |
| 529 | - **R-3 The schedule continues.** [report] `rows` has `kind: tick` with |
| 530 | `region: <pr-to>`, and its `count` rises about once a minute for three |
| 531 | minutes after the open. Every `fired` entry has `count: 1`. A minute |
| 532 | that fell due while the instance was offline fires at most once after |
| 533 | the open, and minutes in between are not replayed, so a short gap |
| 534 | around the cut is expected. A due time that fired twice fails the run. |
| 535 | - **R-4 The once-timer fired once.** [report] `fired` has exactly one |
| 536 | entry with S-3's `name`, its `due_at` equals S-3's, and its `count` is 1. |
| 537 | - **R-5 The source holds nothing live.** After [deploy] |
| 538 | `log/committed-source_parked.json` and `log/done.json` exist: |
| 539 | [data] `list-object-versions` on `<data-bucket-from>` under |
| 540 | `instances/<host>/` returns no versions and no delete markers, and |
| 541 | `parked/<inc>/<host>/` holds the retired replica objects with a |
| 542 | `manifest.json`. [deploy] the claim's `obligations` is empty. |
| 543 | - **R-6 The source really stopped.** [deploy] `stopped-source.json` (the |
| 544 | key [capture] shows as `detail.evidence.source`) has `steps.tree.ok` |
| 545 | and `steps.stray.ok` both `true` and no `stray` field. [box] on the source box, the journal has no "the instance is not |
| 546 | proven stopped yet" line for `<host>` and no M-C20 line ("... is running |
| 547 | a process for <host> that this move cannot account for. ..."), and |
| 548 | `/data/.moves/.page-outbox/<host>/<move>/stray.json` does not exist. |
| 549 | The moveprobe worker outlives its app on purpose, so a worker that |
| 550 | survived the stop fails the cut's tree step and shows here, then |
| 551 | reverts at the cut's deadline. It does not page. |
| 552 | |
| 553 | ### Checks after a rollback (Runs 2, 3 and 4) |
| 554 | |
| 555 | R-1 to R-4 and R-6 apply as written, with the source (where the |
| 556 | instance is back) in place of the target: `<pr-from>` in place of |
| 557 | `<pr-to>` in R-1 and R-3. R-5 is replaced: |
| 558 | |
| 559 | - **R-5b The target holds nothing live.** After [deploy] |
| 560 | `log/rolling_back-target-cleaned.json` and `log/rolled_back.json` exist: |
| 561 | [data] `list-object-versions` on `<data-bucket-to>` under |
| 562 | `instances/<host>/` and under `moves/<host>/<move>/` returns no versions |
| 563 | and no delete markers. |
| 564 | - **R-7 The target's writes came back** (dirty rollbacks only). [report] |
| 565 | on the source: `rows` has a `bg` row with `region: <pr-to>` (the |
| 566 | target's probe region, `home` if the target is the home box), written |
| 567 | on the target while it ran there. |
| 568 | - **R-8 The return lane is parked** (dirty rollbacks only). A dirty |
| 569 | rollback returns the target's writes through its own lane, |
| 570 | `moves/<host>/<rb>/` in the **source's** data bucket, where `<rb>` is |
| 571 | the rollback's id (from [capture]: the last rollback journal's `id`). |
| 572 | The source queues the lane's park in the host claim when it has probed. |
| 573 | The park runs only after its horizon (the return seal's bound) plus the |
| 574 | signature age (15 minutes) and the request limit (2 minutes), and the |
| 575 | rollback does not close until every obligation of the move is |
| 576 | acknowledged. So expect the cut's wait to last at least that long. |
| 577 | The obligation exists only while the move is live, so read it from the |
| 578 | claim file [capture] saved, never from the bucket afterwards: |
| 579 | |
| 580 | ```sh |
| 581 | jq -s --arg loc "moves/<host>/<rb>" '[.[][] | select(.park.location == $loc)] | first' run-3-claim.jsonl |
| 582 | ``` |
| 583 | |
| 584 | It has `kind` `park-replica`, `region` set to the source box's |
| 585 | `CARLOS_REGION` (absent when the source is the home box), an `id`, a |
| 586 | `retirement`, and `park` with `location`, `incarnation`, `evidence_at` |
| 587 | and `request_limit_ns`. Record `retirement` as `<ret>`, |
| 588 | `park.incarnation` as `<pinc>` and `park.evidence_at`. An empty answer |
| 589 | means the capture missed the obligation: the check fails, and the run |
| 590 | is repeated on a spare host with the claim capture running. |
| 591 | After `park.evidence_at` plus 15 minutes plus `request_limit_ns`, and |
| 592 | once `log/rolled_back.json` exists: |
| 593 | [data] `list-object-versions` on `<data-bucket-from>` under |
| 594 | `moves/<host>/<rb>/` returns no versions and no delete markers, and |
| 595 | `parked/<pinc>/<host>/<ret>/manifest.json` exists in |
| 596 | `<data-bucket-from>`. |
| 597 | |
| 598 | A clean rollback needs two things: the stopped target's digests equal |
| 599 | what the landing installed, and `src_covered` (the source's own replica |
| 600 | reached the stopped position). The target box logs both: |
| 601 | [box] `journalctl -u carlos-edge | grep 'the target of rollback'` shows |
| 602 | "... stopped at TXID <n> (clean <bool>, src_covered <bool>)". |
| 603 | |
| 604 | --- |
| 605 | |
| 606 | ## Dev (Runs 1 to 4) |
| 607 | |
| 608 | **Dev topology (Paul, N4).** Dev's second region is applied: `uk`, |
| 609 | `eu-west-2`, box `carlos-dev-uk`, data bucket |
| 610 | `carlos-dev-uk-640012953448` (platform-infrastructure#42). Before Run 1: |
| 611 | |
| 612 | - [x] Bootstrap `carlos-dev-uk`. Done 2026-10-15: box `dennis-stenciling` |
| 613 | on v0.27.3, with `CARLOS_SWAP=overlap` and `CARLOS_REGION=uk`. |
| 614 | - [x] SSM parameters: nothing to copy. No box reads SSM at boot or at |
| 615 | runtime. Never copy `/carlos/release-key` to a region whose host role can |
| 616 | read `/carlos/*`. In `eu-west-2` the `carlos-dev-uk` role can. |
| 617 | - [x] `carlos-dev-uk` needs the dev console's `CARLOS_SECRETS_KEY` before |
| 618 | Run 1 (T2 and config sync). Done 2026-10-15: `dennis-stenciling` got the |
| 619 | dev key through SSM `eu-west-2` `/carlos/secrets-key`. Its |
| 620 | `secrets_key_fp` is `sha256:bb067582…f5fe`, the same as sophie's. The key |
| 621 | is backed up in 1Password as "CARLOS_SECRETS_KEY dev". |
| 622 | - [ ] `CARLOS_REGION_HA=true` on `carlos-dev-ie` and `carlos-dev-uk`, and |
| 623 | `CARLOS_AWS_ACCOUNT` on the dev console. |
| 624 | - [ ] Keep `carlos-dev-ie-b` ineligible: no `CARLOS_REGION_HA` in its |
| 625 | `host.env`, so `ie` has one eligible box (M-C21). |
| 626 | - [ ] Decide Runs 2 and 3's arrangement (see "Runs 2 and 3: the |
| 627 | regions"), then create the dev rehearsal hosts (P-5) after |
| 628 | `carlos-dev-uk` serves and before dev's claim goes on: Run 1's and Run |
| 629 | 4's in `ie`, Runs 2 and 3's in their arrangement's `<from>`, and two |
| 630 | spares, at least one of them in `ie` (P-5). |
| 631 | |
| 632 | **Dev's home region.** Dev Ireland is probably the home region (its box |
| 633 | has no `CARLOS_REGION`). Confirm it with the dev console's |
| 634 | `CONSOLE_HOME_REGION` and record it here: `____`. If `ie` is home: |
| 635 | |
| 636 | - Every dev run touches home, so every dev run needs P-0's |
| 637 | one-placementless-box item: Ruling R2's procedure for |
| 638 | `carlos-dev-ie-b`, Run 1 included. |
| 639 | - Run 1 (`ie` to `uk`) moves out of home and needs **home fix part 1**. |
| 640 | - Run 1b (`uk` to `ie`) moves Run 1's host back into home. It needs |
| 641 | **home fix part 2** rolled and dev's designation published. |
| 642 | - Run 4 (`ie` to `uk`) moves out of home and then rolls back into it. It |
| 643 | needs **part 1**, and part 1 must cover a rollback whose source is home. |
| 644 | It stays out of home even once part 2 is rolled (Ruling R6), because |
| 645 | dev's console runs on the home box. |
| 646 | - Runs 2 and 3 as written (`uk` to `ie`) move into home and need **home |
| 647 | fix part 2**. Once part 2 is rolled, they run as written. |
| 648 | - **Until part 2 is rolled, only out-of-home moves run.** Run 1 goes ahead on |
| 649 | part 1. Runs 2 and 3 then use the **out-of-home arrangement**: the host |
| 650 | is created in `ie`, the move goes `ie` to `uk`, and the failure switch |
| 651 | names `<pr-uk>`. Each rolls back into a home source, as Run 4 does, so |
| 652 | this needs part 1 to cover a rollback whose source is home. Part 1 is |
| 653 | on main as 359e8ac2. Check in its own tests that it covers this before |
| 654 | the first such run. If it does not, Runs 2, 3 and 4 all wait for the |
| 655 | fix that does. |
| 656 | - Run 4 runs only after Runs 1, 1b, 2 and 3 have passed, Runs 2 and 3 in |
| 657 | whichever arrangement they ran. No run is skipped. |
| 658 | - On the home box, moveprobe reports `home`, so `<pr-ie>` is `home`. Any |
| 659 | [report] check or `MOVEPROBE_FAIL_READY_IN` aimed at that box uses |
| 660 | `home`, never `ie`. |
| 661 | |
| 662 | Record the dev probe regions here: `<pr-ie>` = `____`, `<pr-uk>` = `____`. |
| 663 | |
| 664 | If `ie` is not home, the runs go as written and need no home fix. |
| 665 | |
| 666 | ### Run 1: dev, `ie` to `uk`, a commit |
| 667 | |
| 668 | **Blockers:** P-0; home fix part 1 if `ie` is home; if `ie` is home, P-0's |
| 669 | one-placementless-box item, with Ruling R2's window open for |
| 670 | `carlos-dev-ie-b`; dev topology above; P-1 to P-7 on dev; Paul's go. |
| 671 | |
| 672 | - [ ] Host used: `____` (created in `ie` before the claim). Run 1b moves |
| 673 | this host back. |
| 674 | - [ ] S-1, S-2 with `--to uk`. |
| 675 | - [ ] S-3, S-4. [CLI] M-C5 ("uk is ready. Starting the cut. <host> will be |
| 676 | offline for about <estimate>."), M-C6 ("Stopped in ie. Starting in |
| 677 | uk."), M-C7 ("Moved. <host> is served from uk. It was offline for |
| 678 | <seconds> seconds."), exit 0. |
| 679 | - [ ] R-1 to R-6. |
| 680 | - [ ] [deploy] `log/` has `cutting-flipped`, `committed`, |
| 681 | `committed-opened`, `committed-source_parked` and `done`. [detail] |
| 682 | `last.outcome` is `done`; `detail.timings` holds `offline_s` and |
| 683 | `cut_budget_s`. |
| 684 | |
| 685 | ### Run 1b: dev, `uk` to `ie`, a commit into home |
| 686 | |
| 687 | **Blockers:** Run 1 passed; Run 1's quiet window has passed (T12, see |
| 688 | "Between runs"); home fix part 2 merged and rolled, boxes first, then |
| 689 | consoles; the fleet home designation published on dev (claim activation, |
| 690 | #250); P-0's one-placementless-box item, with Ruling R2's window open for |
| 691 | `carlos-dev-ie-b`; P-2's home data bucket item; Paul's go. |
| 692 | |
| 693 | This run moves Run 1's host back into home (Ruling R6). It is the one |
| 694 | host this runbook uses twice (P-5). It has the same checks as Run 1, with |
| 695 | `<pr-ie>` = `home`. |
| 696 | |
| 697 | **Note for the operator (Ruling R7).** Where the console shares the home |
| 698 | box, as dev's does, the landing runs on the console's own box. If that box |
| 699 | dies mid-landing, the console's move pass stops with it until the box or |
| 700 | the console is back. The record and the claim are durable, so nothing is |
| 701 | lost, but recovery waits for the console's return. |
| 702 | |
| 703 | - [ ] Host used: Run 1's, `____`. |
| 704 | - [ ] Before the request, record the claim's touched regions and quiet |
| 705 | windows: |
| 706 | |
| 707 | ```sh |
| 708 | aws s3 cp --profile <profile> s3://<deploy-bucket>/control/hosts/<host>.json - | jq '{regions: .touched.regions, quiet_after}' |
| 709 | ``` |
| 710 | |
| 711 | - [ ] S-1, S-2 with `--to ie`. Name the home by its configured name. There |
| 712 | is no `home` keyword, because a named region may be called `home` |
| 713 | (Ruling R3). If S-2 refuses, the sentence says what is missing: |
| 714 | - MV-N510 ("moves into the home region need host claims. this console |
| 715 | runs without them"): the console runs without `CARLOS_HOST_CLAIM`; |
| 716 | - MV-N511 ("<box> sets CARLOS_REGION, so it cannot take a move into |
| 717 | the home region"): a box's edges file names `ie` as its placement. |
| 718 | That box is not the home box. Fix its `host.env`, or delete its |
| 719 | edges file if the box is gone; |
| 720 | - MV-N177 ("the host roster does not allow this move. the source and |
| 721 | target must be active boxes in uk and ie"): the roster has no home |
| 722 | designation, or the console's home name differs from it; |
| 723 | - "the fleet's edges files could not all be read" or MV-N5: Ruling R2's |
| 724 | window is not ready (P-0, step 4). |
| 725 | |
| 726 | Each of these refusals comes before any claim write. |
| 727 | - [ ] At ready, the staging lane is in the home data bucket: |
| 728 | [data] `list-object-versions` on `<data-bucket-ie>` (the home data |
| 729 | bucket P-2 recorded) with `--prefix "moves/<host>/<move>/"` returns |
| 730 | versions. |
| 731 | - [ ] S-3, S-4. [CLI] M-C5 ("ie is ready. Starting the cut. <host> will be |
| 732 | offline for about <estimate>."), M-C6 ("Stopped in uk. Starting in |
| 733 | ie."), M-C7 ("Moved. <host> is served from ie. It was offline for |
| 734 | <seconds> seconds."), exit 0. |
| 735 | - [ ] R-1 to R-6, with `<pr-to>` = `home`. |
| 736 | - [ ] [deploy] `log/` has `cutting-flipped`, `committed`, |
| 737 | `committed-opened`, `committed-source_parked` and `done`. [detail] |
| 738 | `last.outcome` is `done`; `detail.timings` holds `offline_s` and |
| 739 | `cut_budget_s`. |
| 740 | - [ ] [deploy] the claim's `touched.regions` holds at least one spelling |
| 741 | of home: `ie`, `@home`, or both. Touch adds `@home` when the target box |
| 742 | is new to `touched`, and never removes `ie`. Every region it held |
| 743 | before the request is still there. |
| 744 | - [ ] [deploy] the claim's `quiet_after` names `uk`. |
| 745 | - [ ] After the close: [data] `list-object-versions` on |
| 746 | `<data-bucket-ie>` under `parked/<inc>/<host>/` holds the residue of |
| 747 | Run 1's stay in home, if there was any. |
| 748 | - [ ] If any check fails, repeat Run 1 on a spare host created in `ie`, |
| 749 | then Run 1b on that host. |
| 750 | |
| 751 | ### Runs 2 and 3: the regions |
| 752 | |
| 753 | Runs 2 and 3 use one of two arrangements, chosen before the dev hosts are |
| 754 | created (see "Dev's home region"): |
| 755 | |
| 756 | - **As written**: the host is created in `uk`, and the move goes `uk` to |
| 757 | `ie`. Needs home fix part 2 if `ie` is home. Once part 2 is rolled, |
| 758 | Runs 2 and 3 run as written, and they wait for Run 1b: the first move |
| 759 | into home is Run 1b's plain commit, never a forced rollback. |
| 760 | - **Out of home** (`ie` is home and part 2 is not rolled): the host is |
| 761 | created in `ie`, and the move goes `ie` to `uk`. Needs part 1 to cover a |
| 762 | rollback whose source is home. This stays the fallback while part 2 is |
| 763 | not rolled. |
| 764 | |
| 765 | Below, `<from>` and `<to>` are the arrangement's regions as the CLI names |
| 766 | them, and `<pr-from>`, `<pr-to>` their probe regions. Record the |
| 767 | arrangement used in each run. |
| 768 | |
| 769 | ### Run 2: dev, forced clean rollback |
| 770 | |
| 771 | **Blockers:** P-0; the home fix its arrangement needs (above); if `ie` is |
| 772 | home, P-0's one-placementless-box item, with Ruling R2's window open for |
| 773 | `carlos-dev-ie-b`; Run 1 passed; in the as-written arrangement, Run 1b |
| 774 | passed (the first move into home is a plain commit); Paul's go. |
| 775 | |
| 776 | The target fails its readiness check in before-write mode, so it writes |
| 777 | nothing and the rollback should be clean. |
| 778 | |
| 779 | - [ ] Arrangement: `____`. Host used: `____` (created in `<from>` before |
| 780 | the claim). |
| 781 | - [ ] Before the request, set the failure switch to the target's probe |
| 782 | region: |
| 783 | |
| 784 | ```sh |
| 785 | carlos env set --app moveprobe --account <ops> MOVEPROBE_FAIL_READY_IN=<pr-to> |
| 786 | ``` |
| 787 | |
| 788 | The variable is app-wide. Any moveprobe instance that starts in that |
| 789 | region while it is set answers 503 and writes nothing. That is expected. |
| 790 | - [ ] S-1, S-2 with `--to <to>`, S-3, S-4 (capture first). [CLI] M-C8 |
| 791 | ("<to> could not start <host>: <reason>. Moving it back to <from>."), |
| 792 | then M-C9 ("<host> is back in <from> and serving. Nothing was lost."), |
| 793 | exit 3. |
| 794 | - [ ] From [capture], once the cut has returned and the capture has |
| 795 | stopped: the last rollback journal has `target_phase` `released` or |
| 796 | later, and `dirty` absent or `false`. Record its `id` as `<rb>`, and the |
| 797 | last `detail.evidence.target`. |
| 798 | - [ ] Digests: [deploy] `stopped-target.json`'s `db_sha256`, `files_plain` |
| 799 | and `timers_sha256` equal `log/rolling_back-target-released.json`'s |
| 800 | fields `installed_db_sha256`, `files_plain` and `timers_sha256`. |
| 801 | - [ ] [box] on the `<to>` box: "clean true, src_covered true". |
| 802 | - [ ] R-1 to R-4 and R-6 on `<from>` (probe region `<pr-from>`), R-5b on |
| 803 | `<to>`. |
| 804 | - [ ] [deploy] `log/` has `rolling_back`, `rolling_back-target-released`, |
| 805 | `rolling_back-target-cleaned`, `rolling_back-source-opened` and |
| 806 | `rolled_back`, and no `rolling_back-target-returning`. |
| 807 | - [ ] After the checks are recorded: |
| 808 | |
| 809 | ```sh |
| 810 | carlos env unset --app moveprobe --account <ops> MOVEPROBE_FAIL_READY_IN |
| 811 | ``` |
| 812 | |
| 813 | A dirty result here, with moveprobe in before-write mode, is a platform |
| 814 | finding, not a probe failure. Stop the series and report it. |
| 815 | |
| 816 | ### Run 3: dev, forced dirty rollback |
| 817 | |
| 818 | **Blockers:** P-0; the home fix its arrangement needs; if `ie` is home, |
| 819 | P-0's one-placementless-box item, with Ruling R2's window open for |
| 820 | `carlos-dev-ie-b`; Run 2 passed; Paul's go. |
| 821 | |
| 822 | The target writes one row, starts its worker, and then fails readiness, so |
| 823 | the rollback is dirty and the target's writes must come back. |
| 824 | |
| 825 | - [ ] Arrangement: `____`. Host used: `____` (created in `<from>` before |
| 826 | the claim). |
| 827 | - [ ] Before the request: |
| 828 | |
| 829 | ```sh |
| 830 | carlos env set --app moveprobe --account <ops> MOVEPROBE_FAIL_READY_IN=<pr-to>,after-write |
| 831 | ``` |
| 832 | |
| 833 | - [ ] S-1, S-2 with `--to <to>`, S-3, S-4, with the capture started with |
| 834 | its claim file: `capture <host> run-3-detail.jsonl run-3-claim.jsonl`. |
| 835 | [CLI] M-C8, then M-C9, exit 3. The cut's wait includes the return |
| 836 | lane's park horizon (R-8), about 17 minutes or more. |
| 837 | - [ ] From [capture]: the last rollback journal has `target_phase` |
| 838 | `released` or later and `dirty` `true`. Record its `id` as `<rb>` |
| 839 | before anything else; R-8 needs it. |
| 840 | - [ ] Digests: [deploy] `stopped-target.json`'s `db_sha256` differs from |
| 841 | `log/rolling_back-target-released.json`'s `installed_db_sha256` (the |
| 842 | target wrote). |
| 843 | - [ ] [box] on the `<to>` box: "clean false". |
| 844 | - [ ] [deploy] `log/` has `rolling_back-target-returning` and |
| 845 | `rolling_back-target-returned`, then `rolling_back-target-cleaned` and |
| 846 | `rolled_back`. |
| 847 | - [ ] R-1 to R-4 and R-6 on `<from>` (probe region `<pr-from>`), R-5b on |
| 848 | `<to>`, R-7 (a `bg` row with `region: <pr-to>`), and R-8 on `<from>` |
| 849 | (the return lane `moves/<host>/<rb>/` in `<data-bucket-from>`). |
| 850 | - [ ] Unset `MOVEPROBE_FAIL_READY_IN` once the checks are recorded. |
| 851 | |
| 852 | ### Run 4: dev, declare-lost |
| 853 | |
| 854 | **Blockers:** P-0, including claim Plan 12's loss-bound admission and Plan |
| 855 | 4 Task 10's box half; home fix part 1 if `ie` is home, covering a |
| 856 | rollback whose source is home; if `ie` is home, P-0's |
| 857 | one-placementless-box item, with Ruling R2's window open for |
| 858 | `carlos-dev-ie-b`; **Runs 1, 1b, 2 and 3 passed** (Runs 2 and 3 in |
| 859 | whichever arrangement they ran; Run 4 is never run ahead of them); Paul's |
| 860 | go. |
| 861 | |
| 862 | The target box is stopped mid-move and declared lost. With only two dev |
| 863 | regions, the target is `carlos-dev-uk`, and `uk` has no eligible box until |
| 864 | the readmit at the end. **Stop the box. Do not terminate it.** A terminated |
| 865 | box cannot be readmitted. |
| 866 | |
| 867 | Run 4 stays `ie` to `uk` even once part 2 is rolled (Ruling R6). It never |
| 868 | stops the home box: dev's console runs there, so a declare-lost of the |
| 869 | home box would stop the console too. It does not move into home itself. |
| 870 | |
| 871 | - [ ] Host used: `____` (created in `ie` before the claim). |
| 872 | - [ ] **Paul's scoped go** in the amadan thread, before anything below, |
| 873 | to stop `carlos-dev-uk` during this run and take it out of service |
| 874 | until the readmit. |
| 875 | - [ ] So the landing cannot commit before the box stops, set the failure |
| 876 | switch on the target first: |
| 877 | |
| 878 | ```sh |
| 879 | carlos env set --app moveprobe --account <ops> MOVEPROBE_FAIL_READY_IN=<pr-uk> |
| 880 | ``` |
| 881 | |
| 882 | Its probe then fails however long the stop takes. If the target finishes |
| 883 | its release before the box stops, the run is an ordinary clean rollback: |
| 884 | record it and repeat Run 4 on a spare host. |
| 885 | - [ ] Before the run, record the box's roster entry: [deploy] |
| 886 | `roster | jq '.boxes["carlos-dev-uk"]'` shows `state` `active`; note |
| 887 | its `enrollment` and `quarantine_revision`. |
| 888 | - [ ] S-1, S-2 with `--to uk`, S-3, S-4 (capture first). The capture runs |
| 889 | until the move closes, through the declaration below. |
| 890 | - [ ] When [capture] shows `state` `landing` (after `cutting-flipped`), |
| 891 | stop `carlos-dev-uk` through the AWS console (dev only). |
| 892 | - [ ] The cut's wait ends with M-C10 ("Waiting for carlos-dev-uk in uk to |
| 893 | confirm it has stopped <host>. <host> is offline."), exit 4. Until the |
| 894 | landing deadline passes, it keeps waiting. |
| 895 | - [ ] Declare it lost, with your own AWS credentials for dev in the shell |
| 896 | (the command reads EC2 with them and refuses to run as root): |
| 897 | |
| 898 | ```sh |
| 899 | carlos move resolve --app moveprobe --account <ops> --host <host> --declare-lost carlos-dev-uk |
| 900 | ``` |
| 901 | |
| 902 | [CLI] M-C18 ("This declares carlos-dev-uk lost. Anything <host> wrote |
| 903 | on carlos-dev-uk after <time> is lost. Type the host name to |
| 904 | confirm."). Type `<host>`. The command then waits for the outcome: |
| 905 | expect M-C9 and exit 3. |
| 906 | - [ ] Expected outcome: **rolled back, with the loss on the record**, not |
| 907 | necessarily clean. From [capture]: the last rollback journal has |
| 908 | `loss_ref` and `lost_at` set; record them and its `id`. [deploy] |
| 909 | `log/rolling_back-target-lost.json` (event `declared_lost`) and |
| 910 | `log/rolled_back.json` exist. [deploy] the decoded roster: |
| 911 | `boxes.carlos-dev-uk.state` is `quarantined`, `quarantine_revision` is |
| 912 | one more than before, and `loss_ids` holds the loss. |
| 913 | - [ ] R-1 to R-4 and R-6 on `ie`, with probe region `<pr-ie>` (`home` if |
| 914 | `ie` is home). |
| 915 | - [ ] Unset `MOVEPROBE_FAIL_READY_IN`. |
| 916 | - [ ] Start `carlos-dev-uk` again. It stays quarantined: the deployment's |
| 917 | admission gate refuses it for every host until the readmit. Watch [box] |
| 918 | its journal from boot, and record anything it does for `<host>` before |
| 919 | the readmit. Doing anything for it there is a finding. |
| 920 | - [ ] [deploy] the box retires its local state first and records it: |
| 921 | `control/host-retirement/carlos-dev-uk/<enrollment>/<quarantine_revision>.json` |
| 922 | exists with `"complete": true` and this boot's `boot_id`. Readmit |
| 923 | refuses without it. |
| 924 | - [ ] Readmit it. `carlos ops host` exists only with `CARLOS_HOST_CLAIM` |
| 925 | set in your shell: |
| 926 | |
| 927 | ```sh |
| 928 | CARLOS_HOST_CLAIM=1 carlos ops host resolve <host> --readmit carlos-dev-uk --console <console> |
| 929 | ``` |
| 930 | |
| 931 | Answer `readmit` at its prompt. The box must be running and reporting: |
| 932 | readmit refuses a box with a termination fence, and one whose private |
| 933 | report is older than 15 minutes. Before it, check [deploy] |
| 934 | `control/box-fences/<enrollment>.json` does not exist (a stopped box |
| 935 | has none; a terminated one does). |
| 936 | - [ ] [deploy] the decoded roster: `boxes.carlos-dev-uk.state` is `active` |
| 937 | again, with the same `enrollment`. |
| 938 | |
| 939 | --- |
| 940 | |
| 941 | ## Flagship (Run 5) |
| 942 | |
| 943 | ### Run 5: the flagship, Runs 1 to 3 again |
| 944 | |
| 945 | **Blockers:** P-0; P-1 to P-6 on the flagship; **P-3, the floor, published** |
| 946 | (§15 step 6: before any protected production rehearsal); the moves copy |
| 947 | record fully approved (no `pending` row); **dev Runs 1 to 4 passed** |
| 948 | (each run gates the next, spec §15 step 7; claim Plan 8b is on main, so |
| 949 | the old clause that let Run 5 go ahead with Run 4 skipped no longer |
| 950 | applies); Paul's explicit go in the thread, for each of the three. |
| 951 | |
| 952 | - **The switch** (Paul, N17): `CARLOS_MOVES_PHASE1=on` in the environment |
| 953 | of the console and of the two boxes in play, and nowhere else. The |
| 954 | compiled default stays off. It flips later, in its own commit. |
| 955 | - **Regions** (N3): the flagship's home is Ireland, so Run 5 uses two |
| 956 | non-home regions, `uk` and `us`, until the home fix parts land. Run 1's |
| 957 | shape goes `uk` to `us`. Runs 2 and 3 go `us` to `uk`, with |
| 958 | `MOVEPROBE_FAIL_READY_IN=<pr-uk>` and `<pr-uk>,after-write`. Record the |
| 959 | flagship's probe regions: `<pr-uk>` = `____`, `<pr-us>` = `____`. |
| 960 | |
| 961 | - [ ] P-3 on the flagship, after its own scoped go. |
| 962 | - [ ] P-7 on the flagship: the copy record fully approved, then Paul's |
| 963 | scoped go naming the console and the `uk` and `us` boxes, then the |
| 964 | switch on those three only. |
| 965 | - [ ] Hosts used: Run 1 shape `____` (created in `uk`), Run 2 shape `____` |
| 966 | and Run 3 shape `____` (created in `us`), all before the flagship's |
| 967 | claim went on. |
| 968 | - [ ] Each of the three as on dev, with every check. |
| 969 | - [ ] After the third, and only once **all three moves are closed**: for |
| 970 | each of the three hosts, [detail] `state` is `none` (no move in flight) |
| 971 | and `detail.claim.kind` is `serving`. If any move is unfinished (stuck, |
| 972 | a release past its deadline, relocating, or any exit 4), leave the |
| 973 | switch on and resolve it first with `carlos move resolve` (`--retry` |
| 974 | once the cause is fixed, or `--declare-lost <box>` for a lost target), |
| 975 | each with Paul's scoped go in the thread. The switch must stay on until the move is closed: with it off, a box's |
| 976 | moves pass skips every Phase 1 record, so an unfinished move would stop |
| 977 | being reconciled and stay where it is. |
| 978 | - [ ] Then remove `CARLOS_MOVES_PHASE1` from the two boxes and the |
| 979 | console unless Paul says to keep it, and restart them. Check: |
| 980 | `carlos move status --app moveprobe --account <ops> --host <host>` says |
| 981 | "Moves are not enabled on this console.", and neither box's `host.env` |
| 982 | still sets it. |
| 983 | |
| 984 | --- |
| 985 | |
| 986 | ## Customer instances (Runs 6 and 7) |
| 987 | |
| 988 | The runbook names no tenant. The real instances, their owners and the |
| 989 | agreed windows live in the flagship's deployment repo and in the amadan |
| 990 | thread where Paul gives each go. |
| 991 | |
| 992 | These runs move a customer's app, not moveprobe. **Do not call `/ready`, |
| 993 | `/arm-once` or `/report` on a customer host, and do not set |
| 994 | `MOVEPROBE_FAIL_READY_IN`.** Those are moveprobe's own, and S-1, S-3, R-1 |
| 995 | to R-4 and R-7 do not apply. Set `ACCT=<account>` and `APP=<app>` for |
| 996 | [detail], [capture] and the log prefix, and use `--app <app> --account |
| 997 | <account>` on every command. `<from>` and `<to>` are the regions Paul |
| 998 | approves for the run. |
| 999 | |
| 1000 | ### Customer steps |
| 1001 | |
| 1002 | - **C-0 Switch on for this run only.** Run 5's switch was removed after |
| 1003 | it, and this run's participants differ (the source is the home box). |
| 1004 | - The moves copy record is fully approved (P-7). |
| 1005 | - **Paul's scoped go** in the amadan thread naming the flagship, this |
| 1006 | run, the console, and this run's two participant boxes: the source |
| 1007 | box in `<from>` (the instance's box now) and the target box in |
| 1008 | `<to>`. |
| 1009 | - `CARLOS_MOVES_PHASE1=on` in the environment of the console and of |
| 1010 | those two boxes, and nowhere else; restart them. |
| 1011 | - Check: `carlos move status --app <app> --account <account> --host |
| 1012 | <host>` does not say "Moves are not enabled on this console." [box] |
| 1013 | each of the two boxes' `host.env` sets it, and no other box's does. |
| 1014 | - **C-1 Health check, before.** The owner names one health check for the |
| 1015 | instance (a URL and the answer that means it is serving) and approves |
| 1016 | it for the run. Record it in the deployment repo. Run it and record the |
| 1017 | answer. |
| 1018 | - **C-2 Request.** |
| 1019 | |
| 1020 | ```sh |
| 1021 | carlos move --app <app> --account <account> --host <host> --to <to> --manual-cut |
| 1022 | ``` |
| 1023 | |
| 1024 | [CLI] M-C1, M-C2, M-C3, then M-C12, exit 4. [detail] at ready, the same |
| 1025 | checks as S-2: every target check `ok`, `attempts_admitted` at least 1, |
| 1026 | `source_report.s1.ok` and `s2.ok` `true`. Record `<move>` and `<inc>`. |
| 1027 | - **C-3 Capture, then cut.** Start [capture] in a second terminal, then, |
| 1028 | in the window agreed with the owner: |
| 1029 | |
| 1030 | ```sh |
| 1031 | carlos move cut --app <app> --account <account> --host <host> |
| 1032 | ``` |
| 1033 | |
| 1034 | [CLI] M-C5, M-C6, M-C7, exit 0. |
| 1035 | - **C-4 Health check, after.** The owner's check from C-1 gives the |
| 1036 | serving answer once M-C7 has printed. The owner confirms it. |
| 1037 | - **C-5 Platform checks.** R-5 as written. R-6 as written, except its |
| 1038 | last two sentences, which are about moveprobe's worker. [deploy] `log/` |
| 1039 | has `cutting-flipped`, `committed`, `committed-opened`, |
| 1040 | `committed-source_parked` and `done`. |
| 1041 | |
| 1042 | If the move rolls back (M-C8 then M-C9, exit 3), the instance is back in |
| 1043 | `<from>`: run C-4 there, then R-5b and R-6, and stop the series. C-6 |
| 1044 | still applies, once the rollback has closed. |
| 1045 | |
| 1046 | - **C-6 Switch off, once the move is closed.** First confirm the move is |
| 1047 | closed: [detail] `state` is `none` (no move in flight) and |
| 1048 | `detail.claim.kind` is `serving`. If it is not (stuck, a release past |
| 1049 | its deadline, relocating, or any exit 4), leave the switch on and |
| 1050 | resolve the move first with `carlos move resolve --app <app> --account |
| 1051 | <account> --host <host>` (`--retry` once the cause is fixed, or |
| 1052 | `--declare-lost <box>` for a lost target), with Paul's scoped go in the |
| 1053 | thread, then check again. The switch must stay on until the move is closed: with it off, a box's |
| 1054 | moves pass skips every Phase 1 record, so an unfinished move would stop |
| 1055 | being reconciled and stay where it is. |
| 1056 | Once it is closed, and after the run's checks are recorded: remove |
| 1057 | `CARLOS_MOVES_PHASE1` from the console and the two boxes, unless Paul |
| 1058 | says in the thread to keep it, and restart them. Check: `carlos move |
| 1059 | status --app <app> --account <account> --host <host>` says "Moves are |
| 1060 | not enabled on this console.", and [box] neither box's `host.env` sets |
| 1061 | it. |
| 1062 | |
| 1063 | ### Run 6: a disposable customer-shaped instance |
| 1064 | |
| 1065 | **Blockers:** P-0; **the home-region parks fix, both parts, merged and |
| 1066 | rolled** (customer instances live in the home region); dev Runs 1 and 1b |
| 1067 | passed; Run 5 passed; the owner's agreement; Paul's go. |
| 1068 | |
| 1069 | A disposable instance of a customer app with the same shape as moveprobe |
| 1070 | (a key file, a store, a sealed secret), created for the run with the |
| 1071 | owner's agreement, before the flagship's claim went on (P-5). |
| 1072 | |
| 1073 | - [ ] Instance used: recorded in the deployment repo. |
| 1074 | - [ ] C-0 to C-6. |
| 1075 | |
| 1076 | ### Run 7: the first production customer instance |
| 1077 | |
| 1078 | **Blockers:** everything Run 6 needs; Run 6 passed; Paul's approval of the |
| 1079 | instance; a window agreed with its owner. |
| 1080 | |
| 1081 | The first production customer instance Paul approves, in a window agreed |
| 1082 | with its owner. |
| 1083 | |
| 1084 | - [ ] The instance was created before the flagship's claim went on, so it |
| 1085 | is adopted at `instances/<host>` (P-5). One created later cannot move. |
| 1086 | - [ ] C-0 to C-6, with C-3 inside the agreed window. Run 6's switch was |
| 1087 | removed at its C-6, so C-0 needs its own go. The cut window is the |
| 1088 | default 120 s. |
| 1089 | - [ ] Report `offline_s` back to the owner: [detail] `last.offline_s`, or |
| 1090 | `offline_s` in `log/done.json`'s fields. |
| 1091 | |
| 1092 | --- |
| 1093 | |
| 1094 | ## After every run |
| 1095 | |
| 1096 | - [ ] Attach to the amadan thread: the `done` or `rolled_back` log entry |
| 1097 | (its timings), the bucket listings, and the host used. For Runs 1 to 5, |
| 1098 | also the [report] files from before and after. For Runs 6 and 7, which |
| 1099 | never call `/report`, attach instead the owner-approved health check's |
| 1100 | answers from C-1 and C-4, and the owner's confirmation. |
| 1101 | - [ ] A failed check stops the series. Fix its cause and repeat the run on |
| 1102 | a fresh host before going on. |
| 1103 | |
| 1104 | --- |
| 1105 | |
| 1106 | ## Sources |
| 1107 | |
| 1108 | Paths are from the repo root. Line numbers are main's (`bc25674a`, the |
| 1109 | base of this branch); `cmd/moveprobe` is this branch's and not on main |
| 1110 | yet. Spec is `docs/superpowers/specs/2026-10-02-move-any-app-design.md`; |
| 1111 | copy is `docs/superpowers/plans/2026-10-06-moves-phase1-copy.md`. |
| 1112 | |
| 1113 | **`carlos move` (the CLI)** |
| 1114 | |
| 1115 | | what the runbook says | where | |
| 1116 | |---|---| |
| 1117 | | verbs `status`, `cut`, `abort`, `resolve`; flags `--host`, `--app`, `--account`, `--console` | `move.go:136-149` | |
| 1118 | | `--to`, `--manual-cut`, `--no-wait` on a request; `--wait` on status; `--retry`, `--declare-lost`, `--no-wait` on resolve | `move.go:151-163` | |
| 1119 | | exit codes 0, 1, 2, 3, 4, 5 | `move.go:40-45`, `:53-61` | |
| 1120 | | "carlos move needs --app" | `move.go:66`, `:230` | |
| 1121 | | "Moves are not enabled on this console." | `move.go:65`, `:240-243` | |
| 1122 | | refuses to run as root; declare-lost reads EC2 with your own credentials | `move.go:211-217`, `:80`, `:900-903` | |
| 1123 | | the request sends `region` and `cut` | `move.go:248-253` | |
| 1124 | | M-C2, M-C3, M-C4 | `move.go:703`, `:707`, `:717` | |
| 1125 | | M-C12 at ready with a manual cut, exit 4 | `move.go:722-726` | |
| 1126 | | M-C10, exit 4 | `move.go:740-754` | |
| 1127 | | M-C7 on done, M-C9 on rolled_back, exit 3 | `move.go:788-803` | |
| 1128 | | M-C18 and the typed host name | `move.go:860-882` | |
| 1129 | | "No move is in flight" | `move.go:595` | |
| 1130 | | M-C1, M-C5, M-C6, M-C7, M-C8, M-C9 texts | `internal/moves/operatorcopy.go:11-38`; copy record rows M-C1 to M-C22, `copy:24-45` | |
| 1131 | | the detail path, `/api/cli/apps/<acct>/<app>/instances/<host>/move?detail=1` | `move.go:230-232`, `:351-356`; `apimode.go:460-461` | |
| 1132 | | login tokens in `~/.carlos/credentials`, keyed by `console` | `auth.go:113-120`, `:139-148` | |
| 1133 | |
| 1134 | **The detail answer** |
| 1135 | |
| 1136 | | what | where | |
| 1137 | |---|---| |
| 1138 | | detail=1 only for operators, only with the switch on | `internal/console/apiinstancemove.go:312-327` | |
| 1139 | | `detail.move` is nil once no move is in flight, so the capture loop keeps it | `internal/console/apiinstancemove.go:1291-1292`, `:1416-1423` | |
| 1140 | | a mutation waits for the outcome unless `--no-wait` | `move.go:186-188` | |
| 1141 | | flat fields `state`, `target`, `id`, `last`, `protocol`, `phase` | `internal/console/apiinstancemove.go:160-170` | |
| 1142 | | `detail.move`, `claim`, `target_report`, `source_report`, `evidence`, `log`, `timings`, `from`, `region` | `internal/console/apiinstancemove.go:1290-1318` | |
| 1143 | | `evidence.source` and `evidence.target` come from the record in flight | `internal/console/apiinstancemove.go:1367-1370`, `:1416-1423` | |
| 1144 | | `last` fields `id`, `outcome`, `at`, `offline_s` | `internal/provision/provision.go:1059-1064` | |
| 1145 | | `move.incarnation` | `internal/provision/provision.go:961` | |
| 1146 | | `rollback.dirty`, `target_phase`, `evidence`, `loss_ref`, `lost_at` | `internal/provision/provision.go:1067-1088` | |
| 1147 | | `landing.installed_db_sha256` | `internal/provision/provision.go:1052` | |
| 1148 | | `target_report.checks` T1 to T12, `attempts_admitted`, `cut_budget_s` | `internal/moves/readiness.go:18-31`, `:58-67` | |
| 1149 | | `source_report.s1`, `s2`; S1 refuses a nested instance | `internal/moves/readiness.go:102-109`; `internal/host/movesphase1.go:41`, `:537` | |
| 1150 | |
| 1151 | **The log and the buckets** |
| 1152 | |
| 1153 | | what | where | |
| 1154 | |---|---| |
| 1155 | | identities and events: `requested`, `cutting-requested`/`cut_requested`, `cutting-flipped`/`flipped`, `committed-opened`/`opened`, `committed-source_parked`/`source_parked`, `rolling_back`, `rolling_back-target-<phase>`, `rolling_back-target-lost`/`declared_lost`, `rolling_back-source-<phase>`, `stuck-<epoch>`, the closes; the state as the event for `ready` and `committed` | `internal/moves/transition.go:122-253` | |
| 1156 | | `retry-<epoch>` | `internal/host/movesevents.go:109` | |
| 1157 | | the fields every entry carries | `internal/moves/transition.go:42-71` | |
| 1158 | | the timing keys | `internal/moves/log.go:126-133` | |
| 1159 | | target phases `releasing`, `released`, `returning`, `returned`, `cleaning`, `cleaned`, `lost`; source phases up to `recovered`, `opened` | `internal/provision/provision.go:1148-1172` | |
| 1160 | | released goes to returning only when dirty, else to cleaning | `internal/host/movesrollback.go:566-575` | |
| 1161 | | report prefix `moves/<acct>/<app>/<host>/<op>/`, `source.json`, `target.json`, `stopped-<role>.json`, `log/<identity>.json` | `internal/moves/staging.go:39-50` | |
| 1162 | | the staging lane `moves/<host>/<op>/` in the target's data bucket | `internal/moves/staging.go:6` | |
| 1163 | | the record's account is the sqid | `internal/console/apiinstances.go:1240` | |
| 1164 | | the replica under `<CARLOS_HIBERNATE_PREFIX>/<host>` | `internal/activator/activator.go:119`, `:144-146`; `internal/host/movespass.go:471` | |
| 1165 | | parked objects at `parked/<inc>/<host>/<retirement>/`, a `manifest.json`, the source versions deleted | `internal/park/remote.go:230-232`, `:473`, `:497` | |
| 1166 | | stopped report `steps.tree`, `steps.stray`, `stray` (omitted when empty), digests | `internal/moves/evidence.go:11-20`, `:38-53` | |
| 1167 | | a clean rollback: `src_covered` and unchanged digests; the "clean ..., src_covered ..." line; `dirty` and `evidence` written at released | `internal/host/movesrollback.go:394-398`, `:521-526` | |
| 1168 | | the rollback's `id` | `internal/provision/provision.go:1068` | |
| 1169 | | a dirty rollback's return lane is `moves/<host>/<rb>`, parked in the source's region, queued when the source has probed, horizon the return seal's bound | `internal/moves/parkscope.go:66-72`; `internal/host/movesrecover.go:658-675`; `internal/hostclaim/move.go:1093-1108` | |
| 1170 | | a claim obligation's fields: `id`, `op`, `kind`, `region`, `retirement`, and `park` with `location`, `incarnation`, `evidence_at`, `request_limit_ns`; the claim's `obligations` list | `internal/hostclaim/claim.go:116-121`, `:137-147`, `:166` | |
| 1171 | | the return lane's obligation `region` is the source box's `CARLOS_REGION` (omitted when empty) | `internal/host/movesrecover.go:675`; `internal/host/movespass.go:206`; `internal/hostclaim/claim.go:144` | |
| 1172 | | the claim at `control/hosts/<host>.json` is plain JSON, read as is | `internal/hostclaim/claim.go:178-182`; `internal/hostclaim/store.go:38-60` | |
| 1173 | | the detail answer's claim view carries no obligations | `internal/console/apiinstancemove.go:1356-1362` | |
| 1174 | | a rollback closes only when nothing is owed under its move; a finished park removes its obligation | `internal/host/movesrecover.go:805-809`; `internal/host/movesparks.go:134-142`; `internal/park/remote.go:568`; `internal/hostclaim/delete.go:83-98` | |
| 1175 | | the parked archive is `parked/<park.incarnation>/<host>/<retirement>/` with `manifest.json` | `internal/park/remote.go:230-232`, `:473` | |
| 1176 | | a park runs only after `evidence_at` plus the signature age (15 min) and the request limit (2 min) | `internal/park/remote.go:21`, `:516-519`; `internal/moves/parkscope.go:23` | |
| 1177 | |
| 1178 | **The door and the fleet** |
| 1179 | |
| 1180 | | what | where | |
| 1181 | |---|---| |
| 1182 | | moves into home: refused without host claims (MV-N510), and a box whose edges file names the home as its placement refused (MV-N511), both before any claim read; the home's staging from placementless edges files | `2026-10-15-moves-home-region-part2.md` (Task 5); merge commit a89e35ca; at that commit `internal/console/apiinstancemove.go:614`, `:665-666`, `internal/console/instancesplacement.go:199` | |
| 1183 | | one placementless box: S1 and T8 count every edges file per placement, fresh or not, before the data-bucket check | `internal/host/movesreport.go:113` (`loadMoveFleet`) | |
| 1184 | | Ruling R2's procedure, R6 (Run 4 stays out of home, Run 1b) and R7 (a home landing on the console's box) | `2026-10-15-moves-home-region-part2.md`, "Rulings needed" | |
| 1185 | | a region with no staging bucket is refused | `internal/console/apiinstancemove.go:580-583`; staging from `data_bucket`, `internal/console/instancesplacement.go:366` | |
| 1186 | | M-C21, more than one box | `internal/console/apiinstancemove.go:595-600` | |
| 1187 | | MV-N5, below the floor | `internal/console/apiinstancemove.go:613-616`; `copy:55` | |
| 1188 | | a box below the protocol, or not armed for the floor, or with an unjudged `carlos.prev`, is below the floor | `internal/console/boxselect.go:100-108` | |
| 1189 | | MV-N167 for a holder at `instances/<host>/<inc>` | `internal/console/apiinstancemove.go:143`, `:647-651`; `copy:217` | |
| 1190 | | a created holder is at `instances/<host>/<inc>`, an adopted one at `instances/<host>` | `internal/hostclaim/create.go:75`; `internal/hostclaim/adopt.go:20` | |
| 1191 | | the source must be an active, enrolled roster entry with no termination fence | `internal/console/apiinstancemove.go:657-684`, `:1267-1273` | |
| 1192 | | eligibility: fresh, enrolled, `region-ha-1`, the console's secrets fingerprint, not draining | `internal/boxchoice/boxchoice.go:21`, `:109-148` | |
| 1193 | | `region-ha-1` and the identity record only with `CARLOS_REGION_HA` | `internal/host/edgesfile.go:156-164`; `internal/host/boxrecord.go:25-28`, `:158-166` | |
| 1194 | | `CARLOS_REGION_HA` reads only ParseBool true values, so `on` is off | `internal/host/boxrecord.go:21-28` | |
| 1195 | | identity records at `control/boxes/<box>.json` | `internal/boxid/record.go:21`, `:28` | |
| 1196 | | `CARLOS_AWS_ACCOUNT` and `CONSOLE_HOME_REGION` on the console | `cmd/console/main.go:318`, `:321` | |
| 1197 | | the claim at `control/hosts/<host>.json`; `holder.location`, `obligations`, `quiet_after` | `internal/hostclaim/claim.go:56`, `:166`, `:171`, `:182` | |
| 1198 | | T12 reads `quiet_after`; H is 15 min plus 1 min skew | `internal/host/movesreport.go:756-762`; `internal/moves/lease.go:13`, `:22`; spec:2573 | |
| 1199 | | the console's claim recovery pass adopts with the claim on | `internal/console/hostclaimbackfill.go:40-48` | |
| 1200 | | the roster at `control/host-actors.json`; `boxes`, `state`, `enrollment`, `quarantine_revision`, `loss_ids`; a declaration quarantines and adds one to the revision | `internal/hostactors/roster.go:21`, `:30`, `:38-40`, `:72`, `:265-279` | |
| 1201 | | the roster is written as a root-signed envelope `{payload, sig}`, the payload base64 JSON | `internal/hostactors/roster.go:249`; `internal/release/rootenvelope.go:29-44`; `internal/release/release.go:377-379` | |
| 1202 | | a termination fence is `control/box-fences/<enrollment>.json`; no object, no fence | `internal/hostactors/fence_authority.go:14-34` | |
| 1203 | | a quarantined box is refused admission | `internal/hostquarantine/quarantine.go:17-25` | |
| 1204 | | `CARLOS_HOST_CLAIM` is any non-empty value | `cmd/console/main.go:1364-1366`; `hostclaim_cli.go:22` | |
| 1205 | | `carlos ops host` exists only with `CARLOS_HOST_CLAIM` in the shell; `--readmit`, `--console`, the `readmit` prompt | `hostclaim_cli.go:35-39`, `:61-75`, `:143` | |
| 1206 | | readmit refuses a fenced box and a private report older than 15 min | `internal/hostresolve/readmit.go:18-40` | |
| 1207 | | readmit needs the box's complete retirement receipt for this boot at `control/host-retirement/<box>/<enrollment>/<revision>.json`, then sets the box `active` | `internal/hostresolve/readmit.go:52-66`, `:134`; `internal/hostquarantine/quarantine.go:72-82`, `:91-96` | |
| 1208 | |
| 1209 | **The floor, the switch and the boxes** |
| 1210 | |
| 1211 | | what | where | |
| 1212 | |---|---| |
| 1213 | | the floor key and shape, `{"min","min_release"}` | `internal/moves/floor.go:20`, `:61-64` | |
| 1214 | | `vX.Y.Z` only; anything else fails closed | `internal/moves/floor.go:85-99` | |
| 1215 | | no code writes the floor; only reads | `internal/moves/floor.go:66-82` | |
| 1216 | | a box's floor only rises; lowering it is a manual step per box | `internal/host/movesfloor.go:31-36` | |
| 1217 | | `moves_protocol`, `moves_floor_min_release`, `prev_release`, `prev_allowed_for`, `data_bucket`, `secrets_key_fp` in the edges file at `control/edges/<box>.json` | `internal/host/edgesfile.go:218`, `:270`, `:295-304`; `internal/console/instancesplacement.go:22`, `:83-86` | |
| 1218 | | `control/moves-console.json` and its `secrets_key_fp` | `internal/moves/floor.go:25-33` | |
| 1219 | | the switch, `on`/`1`/`true`, default off | `internal/moves/switch.go:6-23` | |
| 1220 | | the copy record's `pending` rows keep the compiled default off; P-7 applies the same rule to the env switch (Paul's ruling) | `approved_moves_copy_test.go:811`; `docs/superpowers/plans/2026-10-06-moves-phase1-copy.md` | |
| 1221 | | with the switch off the CLI says "Moves are not enabled on this console." (the switch-off check in Run 5 and C-6) | `move.go:65`, `:240-243` | |
| 1222 | | with the switch off a box's moves pass skips every Phase 1 record, so the switch goes off only after the move closes | `internal/host/movespass.go:797-805` | |
| 1223 | | `carlos move resolve --retry` and `--declare-lost <box>` | `move.go:159-163`, `:258-262` | |
| 1224 | | the copy gate keeps the default off while a row is pending | `approved_moves_copy_test.go:811` | |
| 1225 | | the default cut window, 120 s | `internal/moves/budget.go:17` | |
| 1226 | | `CARLOS_LAYOUT_DEFAULT=flat` | `internal/host/layoutleg.go:68-70`; `system.go:714` | |
| 1227 | | `carlos system status` | `system.go:75-76`; `README.md:272` | |
| 1228 | | `carlos status --layout` shows each host's actual layout, box by box, and marks a box "(nested by default)"; `carlos system layout status` shows only the recorded intent | `cli.go:3528`; `status_instances.go:142-163`, `:224`; `system.go:709-716` | |
| 1229 | | the agent is `carlos-edge.service`, reading `/etc/carlos/host.env`, with a delegated cgroup | `internal/host/units/carlos-edge.service:18-19`, `:101` | |
| 1230 | | "exec-child accounting armed"; no manager, no stop evidence | `internal/cgroup/cgroup.go:181-184`; `internal/host/movespass.go:376-392` | |
| 1231 | | "not proven stopped yet ... retrying until the cut's deadline" | `internal/host/movescut.go:422-427` | |
| 1232 | | M-C20 and the page outbox `<DataDir>/.moves/.page-outbox/<host>/<move>/<rule>.json`, rule `stray` | `internal/host/movespaging.go:47-50`, `:87`, `:546-554`, `:586-588` | |
| 1233 | | the tree step's "could not be stopped" | `internal/stopevidence/reconstruct_linux.go:185` | |
| 1234 | | alert sinks `CARLOS_ALERT_WEBHOOK`, `CARLOS_ALERT_SNS_TOPIC_ARN`; the console arms them only with the claim and the switch on | `internal/alerting/alerting.go:191-209`; `cmd/console/main.go:1352-1362` | |
| 1235 | | `CARLOS_SECRETS_KEY` on the box | `internal/host/movespass.go:311-316` | |
| 1236 | | `/data` is the data directory | `internal/ops/ops.go:27` | |
| 1237 | |
| 1238 | **Timers and moveprobe** |
| 1239 | |
| 1240 | | what | where | |
| 1241 | |---|---| |
| 1242 | | a held timer row is skipped while the instance moves | `internal/schedule/runner.go:190-197` | |
| 1243 | | a periodic schedule's next fire is the next boundary after now (no replay) | `internal/schedule/spec.go:119-125` | |
| 1244 | | moveprobe's region is `home` with no `CARLOS_REGION`, else `CARLOS_REGION`; `/report` says which in `region_from` | `cmd/moveprobe/main.go:94-100` | |
| 1245 | | the failure switch matches the probe's own region, so it must name `<pr-R>` | `cmd/moveprobe/main.go:103-106`, `:137-138` | |
| 1246 | | a box's edges `placement` is its `CARLOS_REGION`, absent when unset (the home box) | `internal/host/edgesfile.go:205-211`, `:758` | |
| 1247 | | `bg` and `worker` rows each numbered from 1 in one sequence | `cmd/moveprobe/main.go:543-550`, `:158`, `:674` | |
| 1248 | | `/report`'s fields, `/arm-once`'s answer, the failure switch's modes | `cmd/moveprobe/README.md`, "Use it", "Make readiness fail on purpose" and "Strings" | |
| 1249 | |
| 1250 | **Other CLI commands** |
| 1251 | |
| 1252 | | what | where | |
| 1253 | |---|---| |
| 1254 | | `carlos env set`, `unset`, `list` with `--app` and `--account` | `appconfig_cli.go:342-350`, `:408-417`, `:462-467`; `cli.go:2045-2051` | |
| 1255 | | `carlos instances create --host --region --account` | `instances_cli.go:297-310` | |
| 1256 | |
| 1257 | **Not on main, cited by name** |
| 1258 | |
| 1259 | | what | where | |
| 1260 | |---|---| |
| 1261 | | the cut guard and T13 | branch `moves-cut-guard`, commits `083a6eaa` and `6871d2e3` | |
| 1262 | | the home parks fix part 2 (moves into home) | `2026-10-15-moves-home-region-part2.md`, merged to main as a89e35ca (branch `moves-home-part2-plan`); #236. Part 1 is on main as 359e8ac2 | |
| 1263 | | claim Plan 12; moves Plan 5 Tasks 9b and 8, Plan 4 Tasks 9 and 10 | plans only; the landing, rollback and recovery are unarmed until then, `internal/host/movespass.go:335-340` | |
| 1264 | | dev's second region | platform-infrastructure#42 | |
| 1265 | |
| 1266 | **Spec** |
| 1267 | |
| 1268 | | what | where | |
| 1269 | |---|---| |
| 1270 | | the runs and the pass rule | spec §14.4, spec:6263-6276 | |
| 1271 | | the rollout order, the floor before any protected production rehearsal | spec §15, spec:6278-6294 | |
| 1272 | | each rehearsal gates the next: Run 4 needs Runs 2 and 3, Run 5 needs Run 4 | spec:6294; controller rulings, fix round 1 and final review | |
| 1273 | | the flagship's home is Ireland | spec:1032 | |
| 1274 | |