carlos / platform Public

Clone
git clone https://amadan.net/carlos/platform

Plain git — no account needed to clone.

Download

Download this file

1# Moves Phase 1: the rehearsal runbook
2
3This is the checklist an operator follows to rehearse Phase 1 moves (spec
4§14.4, rollout §15). It covers eight runs in the spec's order: five on dev
5(Runs 1, 1b, 2, 3 and 4; Run 1b is a move into home), one on the flagship
6with the probe app, one on a disposable customer-shaped instance, and one
7on a real customer instance.
8
9**No run in this runbook can execute today.** Every run waits for P-0
10below. Each run also has its own blockers, listed at its top. Never run one
11without Paul's go in the amadan thread. Steps that change a deployment
12(publishing the floor, switching moves on, stopping a box) each have their
13own **Paul's scoped go** step just before them: a go in the amadan thread
14that names the deployment and exactly what the step changes. A go for one
15step or one run does not cover another.
16
17The probe app is `moveprobe`. How to build it, ship it and use it is in
18[`cmd/moveprobe/README.md`](../../../cmd/moveprobe/README.md). This runbook
19does not repeat it.
20
21Every command, flag, event, field and key below was checked against main,
22and moveprobe's against this branch.
23The Sources section at the end gives file and line for each.
24
25---
26
27## How to read this runbook
28
29### Placeholders
30
31| placeholder | what it is |
32|---|---|
33| `<console>` | the deployment's console origin, `https://...`, as `carlos auth login` stored it |
34| `<ops>` | the ops account's sqid, as `--account` takes it |
35| `<account>`, `<app>` | a customer run's account sqid and app (Runs 6 and 7) |
36| `<host>` | the run's host, `<label>.<instance domain>` |
37| `<deploy-bucket>` | the deployment bucket (`CARLOS_DEPLOYMENT_BUCKET` in a box's `/etc/carlos/host.env`) |
38| `<data-bucket-R>` | region R's data bucket: `data_bucket` in that region's box edges file |
39| `<profile>` | the deployment's named AWS profile. Always name one. There are no default credentials. |
40| `<move>` | the move id the console gave the request (`id` in the flat view) |
41| `<inc>` | the instance incarnation (`detail.move.incarnation`) |
42| `<pr-R>` | the region moveprobe reports on region R's box: `home` if that box has no `CARLOS_REGION` (its edges file has no `placement`), else its `CARLOS_REGION`, the placement code. `<pr-from>` and `<pr-to>` are the run's two. |
43
44### The probe's region is not always the region's name
45
46moveprobe reports the region its box gives it, not the name the CLI and
47the console use. On a home box, which has no `CARLOS_REGION`, it reports
48`home`, while `carlos move` still says `ie` (or whatever
49`CONSOLE_HOME_REGION` is). Before each deployment's first run, write down
50`<pr-R>` for every rehearsal region: read `CARLOS_REGION` in the box's
51`/etc/carlos/host.env`, or `placement` in its edges file, and confirm with
52[report]'s `region` and `region_from` on any moveprobe host there. Every
53[report] region check and every `MOVEPROBE_FAIL_READY_IN` value uses
54`<pr-R>`, never the CLI's name.
55
56### Read paths
57
58Each check says where to read it.
59
60- **[CLI]**: the sentence `carlos move` prints and its exit code. Exit codes:
61 0 done; 3 rolled back or aborted; 4 waiting (ready for a manual cut,
62 stuck, a release past its deadline, relocating, or an outcome not
63 recorded); 5 refused; 1 the console could not be reached or read; 2 a
64 usage error.
65- **[detail]**: the operator's detail answer. No `carlos` command prints it
66 yet, so read it with your own login token. Set `ACCT` and `APP` for the
67 run: `<ops>` and `moveprobe` for Runs 1 to 5, `<account>` and `<app>`
68 for Runs 6 and 7.
69
70 ```sh
71 CONSOLE=<console>
72 ACCT=<ops> APP=moveprobe
73 TOKEN=$(jq -r --arg c "$CONSOLE" '.credentials[] | select(.console == $c) | .token' ~/.carlos/credentials)
74 detail() { curl -fsS -H "Authorization: Bearer $TOKEN" "$CONSOLE/api/cli/apps/$ACCT/$APP/instances/$1/move?detail=1"; }
75 detail <host> | jq .
76 ```
77
78 The flat fields (`state`, `phase`, `id`, `last`) sit at the top. The
79 rest is under `detail`: `move` (the record's journal), `claim`,
80 `target_report`, `source_report`, `evidence`, `log` (keys only),
81 `timings` (after the close) and `region`. **`detail.move` and
82 `detail.evidence` are gone once the move closes**, and a mutation waits
83 until it closes by default. So every run that needs them starts the
84 capture loop below before its cut and reads them from the capture file
85 afterwards.
86- **[capture]**: every detail answer of one move, saved as it happens.
87 Start it in a second terminal once the request has reached ready, before
88 the cut. It stops by itself once no move is in flight. With a third
89 argument it also saves, each time round, the host claim's
90 `obligations` from the deployment bucket (`control/hosts/<host>.json`,
91 plain JSON). The detail answer does not carry them, and they are gone
92 once the move closes, so this is the only way to keep a park's horizon
93 and retirement id. Run 3 needs it (R-8):
94
95 ```sh
96 capture() { # capture <host> <detail-file> [<claim-file>]
97 while :; do
98 if d=$(detail "$1"); then
99 printf '%s\n' "$d" >> "$2"
100 if [ -n "$3" ]; then
101 aws s3 cp --profile <profile> "s3://<deploy-bucket>/control/hosts/$1.json" - \
102 | jq -c '.obligations // []' >> "$3"
103 fi
104 [ "$(printf '%s' "$d" | jq -r .state)" = none ] && break
105 fi
106 sleep 2
107 done
108 }
109 capture <host> run-<n>-detail.jsonl # Runs 1, 2, 4, 5
110 capture <host> run-3-detail.jsonl run-3-claim.jsonl # Run 3
111 # later: the last rollback journal the move showed
112 jq -s '[.[] | .detail.move.rollback | select(. != null)] | last' run-<n>-detail.jsonl
113 ```
114
115- **[deploy]**: the deployment bucket, read with `--profile <profile>`.
116 Each move keeps its reports, stop evidence and log under
117 `moves/$ACCT/$APP/<host>/<move>/`: `source.json`, `target.json`,
118 `stopped-source.json`, `stopped-target.json` and `log/<identity>.json`.
119 The claim is `control/hosts/<host>.json`, each box's edges file
120 `control/edges/<box>.json`, each box's identity record
121 `control/boxes/<box>.json`, and the floor `control/moves-protocol.json`.
122
123 ```sh
124 P="moves/$ACCT/$APP/<host>/<move>/"
125 aws s3 ls --profile <profile> "s3://<deploy-bucket>/${P}log/"
126 aws s3 cp --profile <profile> "s3://<deploy-bucket>/${P}log/done.json" - | jq .
127 ```
128
129 The roster, `control/host-actors.json`, is a signed envelope,
130 `{"payload": <base64 JSON>, "sig": ...}`. Decode the payload to read it
131 (this does not check the signature; the console does). A box's
132 termination fence is a separate object,
133 `control/box-fences/<enrollment>.json`, using the enrollment from the
134 roster. No object means no fence:
135
136 ```sh
137 roster() { aws s3 cp --profile <profile> s3://<deploy-bucket>/control/host-actors.json - | jq -r .payload | base64 -d; }
138 roster | jq '.boxes["<box>"] | {state, enrollment, loss_ids, quarantine_revision}'
139 aws s3api head-object --profile <profile> --bucket <deploy-bucket> \
140 --key "control/box-fences/$(roster | jq -r '.boxes["<box>"].enrollment').json" # 404: no fence
141 ```
142
143- **[data]**: a region's data bucket, read with `--profile <profile>`. The
144 instance's replica is under `instances/<host>/` (the box's
145 `CARLOS_HIBERNATE_PREFIX`, `instances` in production), a move's staging
146 lane under `moves/<host>/<move>/`, and parked objects under
147 `parked/<inc>/<host>/<retirement>/` with a `manifest.json`. These buckets
148 are versioned, so list versions, not objects:
149
150 ```sh
151 aws s3api list-object-versions --profile <profile> --bucket <data-bucket-R> --prefix "instances/<host>/"
152 ```
153
154- **[report]**: moveprobe's own report, with the secret from your shell:
155
156 ```sh
157 curl -fsS -H "Authorization: Bearer $MOVEPROBE_SECRET" https://<host>/report | jq .
158 ```
159
160- **[box]**: on a box, the agent's journal (`journalctl -u carlos-edge`),
161 `/etc/carlos/host.env`, and files under the data directory (`/data`).
162
163### Log events
164
165Each transition writes one log object, keyed by its identity. The ones the
166checks use:
167
168| identity (the key) | event (the `event` field) | when |
169|---|---|---|
170| `requested` | `requested` | the request was journaled |
171| `ready` | `ready` | the target's checks passed |
172| `cutting-requested` | `cut_requested` | the cut was asked for |
173| `cutting-<phase>` | `<phase>` | `admitted`, `stopping`, `stopped`, `staged`, `archived` |
174| `cutting-flipped` | `flipped` | the record now names the target |
175| `landing-<phase>` | `<phase>` | the target's landing steps |
176| `probe_attempt-<n>` | `probe_attempt` | the landing's n-th readiness probe |
177| `committed` | `committed` | the landing committed |
178| `committed-opened` | `opened` | the target opened (`offline_s` in its fields) |
179| `committed-source_parked` | `source_parked` | the source parked what it held |
180| `rolling_back` | `rolling_back` | a rollback started (`reason` in its fields) |
181| `rolling_back-target-<phase>` | `<phase>` | `released`, `returning`, `returned` (a dirty rollback only), `cleaning`, `cleaned` |
182| `rolling_back-target-lost` | `declared_lost` | the target was declared lost |
183| `rolling_back-source-<phase>` | `<phase>` | the source's recovery, up to `recovered` and `opened` |
184| `stuck-<epoch>`, `retry-<epoch>` | `stuck`, `retry` | a stuck move and its retry |
185| `done`, `rolled_back`, `aborted` | the same | the close; `done` and `rolled_back` carry the timings |
186
187Every entry's `fields` carry `from`, `to` and, once known, `final_txid`,
188`db_sha256`, `timers_sha256`, `files_plain`, `installed_db_sha256`,
189`release`, `env_report` and `cut_budget_s`. The close's timings are
190`prepare_s`, `cut_s`, `cut_budget_s`, `land_s`, `land_budget_s`,
191`offline_s`, `rollback_s` and `release_budget_s`. There is no `stray`
192event and no `dirty` field in the log.
193
194---
195
196## P-0: what every run waits for
197
198Nothing below P-0 runs until every item here is true. Paul confirms P-0 per
199deployment before its first run.
200
201- [ ] **Claim Plan 12 merged.** Without it the landing, the target's half
202 of a rollback and the source's recovery are unarmed in production, so a
203 move with the switch on stops the source, flips, and then has nothing to
204 land it, roll it back or recover it.
205- [ ] **Moves Plan 5 Task 9b, Plan 4 Tasks 9 and 10 (the box half), and
206 Plan 5 Task 8 merged.** They wire moves onto claim Plan 12. Run 4 also
207 needs claim Plan 12's loss-bound admission.
208- [ ] **The moves cut guard merged.** It is on main (b04ff852, Paul's
209 ruling of 2026-10-11). The source refuses a cut unless the target
210 reports its landing armed (the `T13` check), and records why a cut did
211 not start.
212- [ ] **Moves' post-Plan 12 branch merged** (`moves-phase1-5f`). It brings
213 the wake source, the recovery owners, the loss rules, the root held
214 release (T7 stages, `configured` publishes, through root's control
215 socket) and the C8 files-manifest provider.
216- [ ] **For any run that touches the home region: the home-region parks fix
217 part 1 merged** (#236, on main as 359e8ac2) **and claim Plan 13's home
218 naming** (a0806e4c, follow-ups 2ad5412c). Part 1 covers moves out of
219 home. **Any run that moves into home also needs part 2**
220 (`2026-10-15-moves-home-region-part2.md`) merged and rolled, boxes
221 first, then consoles. The door refuses moves into home without host
222 claims (MV-N510). A home box arms only when the claim grants it the
223 `@home` alias, which needs the signed fleet home designation. Plan 13's
224 rollout order is binding: every roster reader on a Plan 13 build first,
225 then publish the designation, then turn the flag on. Older binaries
226 cannot read a roster that carries `home_region`. For moves into home,
227 the whole order is:
228 1. Plan 13 on every box and console. Check each box reads `current`
229 with `carlos system status`. Plan 13's own order applies inside
230 this step.
231 2. Part 2 merged (dormant), released, and rolled **boxes first, then
232 consoles**. A console rolled first could admit a move into a home box
233 that still fails T3, and that move waits at `preparing` until an
234 operator aborts it.
235 3. The IAM mesh, including the home data bucket (P-2), applied by Paul
236 with a scoped `tofu apply -target`.
237 4. Claim activation (#250), which publishes the signed designation.
238 Until then a home box never arms, and the door refuses claim-mode
239 moves with MV-N177.
240 5. `CARLOS_MOVES_PHASE1` per run, as P-7 says.
241- [ ] **The claim's activation checklist (#250) done for this deployment.**
242 Among its items:
243 - the one-time files-manifest backfill for every host older than the
244 files publisher. Until it has run for a host, that host refuses to wake
245 (`ErrNoEvidence`), by design;
246 - claim Plan 15's admitted-generation record, so hosts with history past
247 the eight-admission window can prove their file origins. Until then
248 those hosts refuse;
249 - claim Plan 14 (#280), so that root trusts no service-owned state;
250 - `CARLOS_RELEASE_PUBKEY` readable by instance control
251 (`/etc/carlos/host.env`), or every held release is refused;
252 - with `CARLOS_MOVES_PHASE1` on, a usable `CARLOS_SECRETS_KEY` on every
253 box. Without it the moves source refuses to open and the box serves
254 nothing. Its log names the cause once per start.
255- [ ] **All of the above released and rolled, consoles first, then every
256 box.** Check every box with `carlos system status --console <console>`:
257 each reads `current` on the new version.
258- [ ] **Each deployment's home region recorded:** `CONSOLE_HOME_REGION` in
259 the console's environment. A box with no `CARLOS_REGION` in its
260 `host.env` publishes no `placement` in its edges file: it is a home box.
261- [ ] **Dev's home region has two boxes: wake rehearsal apps on one.** While
262 dev's claim is off, both home boxes (`carlos-dev-ie` and
263 `carlos-dev-ie-b`) take every unregioned record. A wake that switches box
264 can serve a stale local DB (amadan #286; `activator.go`, "only while a
265 region has one box"). So rehearsal apps in dev's home region are woken on
266 one box only, or the run waits for dev's claim. #286 ruled this a
267 dev-only hazard, accepted. During a window under the procedure in the
268 next item, `carlos-dev-ie-b` is stopped, so this holds by itself.
269- [ ] **One placementless box per deployment.** For each deployment, list
270 `control/edges/` and record that exactly one edges file has no
271 `placement`:
272
273 ```sh
274 for k in $(aws s3api list-objects-v2 --profile <profile> --bucket <deploy-bucket> \
275 --prefix control/edges/ --query 'Contents[].Key' --output text); do
276 printf '%s %s\n' "$k" "$(aws s3 cp --profile <profile> s3://<deploy-bucket>/$k - | jq -r '.placement // "(none)"')"
277 done
278 ```
279
280 S1 and T8 refuse a move into **or out of** home while two do, whether or
281 not the second box is running, because an edges file outlives its box.
282 They count every edges file per placement, fresh or not. A retained file
283 below the floor makes the door refuse with MV-N5.
284
285 On dev, `carlos-dev-ie-b` makes two. Ruling R2 (option a) applies to
286 every dev run that touches home, Run 1 included. Each window needs
287 **Paul's scoped go** in the amadan thread, naming dev and
288 `carlos-dev-ie-b`. The procedure, as the ruling gives it:
289
290 1. stop `carlos-dev-ie-b` (the instance, or at least its agent and console units, so it neither writes status nor serves);
291 2. delete `control/edges/carlos-dev-ie-b.json` from the deployment bucket;
292 3. wait until its last `status/boxes/carlos-dev-ie-b.json` is more than 15 minutes old by its `generated_at`;
293 4. before **each** run, verify the fleet's evidence is complete: every box with a status snapshot fresher than 15 minutes has an edges file, every edges file's box has a status snapshot, and exactly one edges file has no `placement`. The door's own confirmation is the run's **S-2**, the move request itself. Only a move request runs `selectMoveBoxes` and the floor check: `carlos move status` (`handleAPIInstanceMoveStatus`) reads the record and claim only, and S-1 only wakes the app, so neither confirms anything here. If S-2 answers "the fleet's edges files could not all be read" or MV-N5, the window is not ready. Both refusals come before any claim read or write, so nothing was started: fix the evidence and repeat S-2;
294 5. after the window, start the box again. Its edges pass rewrites the file on boot. No move touching home runs until the window is reopened by steps 1 to 4.
295
296 Step 3's wait matters. If the edges file is deleted while the box's
297 status snapshot is still fresh, the door reads the fleet's evidence as
298 incomplete and refuses every move.
299- [ ] **Rehearsal boxes keep instances flat.** Moves refuse a nested
300 instance. Set `CARLOS_LAYOUT_DEFAULT=flat` in each rehearsal box's
301 `host.env`. Pre-flight check, before any request: `carlos status
302 --layout --console <console>` (the layout each host is actually on,
303 box by box) shows every rehearsal host's row as `flat`, and no
304 rehearsal box is marked "(nested by default)". `carlos system layout
305 status` is not this check: it shows only the recorded intent. Once a
306 run's request exists, S-2 also checks `detail.source_report.s1.ok`,
307 the source's own refusal of a nested instance.
308- [ ] **Never set `CARLOS_MOVES_PHASE1=on` anywhere before P-0 is true.**
309
310---
311
312## Preconditions, once per deployment
313
314### P-1 The dormant release
315
316- [ ] The release carrying P-0 is rolled with `CARLOS_MOVES_PHASE1` unset
317 everywhere: consoles first, then boxes.
318- [ ] [deploy] Every box's `control/edges/<box>.json` has
319 `"moves_protocol": 1`.
320
321### P-2 Staging grants (§15 step 5, scoped tofu by Paul)
322
323- [ ] Every rehearsal region's `sibling_staging_arns` lists every other
324 rehearsal region's data bucket, and each box has the versioned-object
325 grants (`s3:ListBucketVersions`, `s3:GetObjectVersion`,
326 `s3:DeleteObjectVersion` on its own bucket; `s3:GetObjectVersion` for
327 the console).
328- [ ] **The home data bucket is in the mesh.** Every named region's
329 `sibling_staging_arns` lists the home data bucket's ARN, and the home's
330 lists every named region's data bucket. Record the home data bucket's
331 name (`data_bucket` in the home box's edges file, the one with no
332 `placement`): `____`. Check that it is not the deployment bucket
333 (`<deploy-bucket>`). Without the mesh, a move into home fails the
334 source's lane round with `AccessDenied` (logged), or T10 fails on the
335 home box.
336- [ ] Check after the first request reaches ready: [detail]
337 `detail.target_report.checks.T10.ok` is `true` (the lane probe).
338
339### P-3 The protocol floor
340
341Publish the floor only after **every box and the console run a tagged
342`vX.Y.Z` release** at or above `min_release`. Once a floor is armed, any
343build that is not `vX.Y.Z` (a SHA build, `dev`, a prerelease) is refused at
344activation, and the door treats that box as below the floor. A box never
345lowers its local floor on its own. If dev keeps rolling SHA builds, dev's
346runs go without a floor, and P-3 is done on the flagship only, before
347Run 5 (§15 step 6).
348
349- [ ] **Paul's scoped go** in the amadan thread to publish the floor on
350 this deployment, naming the deployment and `min_release`. Nothing below
351 runs without it.
352- [ ] Write it create-only from an operator shell. No `carlos` command
353 writes it (a gap to note on an amadan discussion).
354
355 ```sh
356 printf '{"min":1,"min_release":"vX.Y.Z"}' > floor.json
357 aws s3api put-object --profile <profile> --bucket <deploy-bucket> \
358 --key control/moves-protocol.json --body floor.json --if-none-match '*'
359 aws s3 cp --profile <profile> s3://<deploy-bucket>/control/moves-protocol.json -
360 ```
361
362- [ ] [deploy] Every box's edges file shows `moves_floor_min_release` equal
363 to the floor's `min_release`, and either `"prev_release": "none"` or
364 `prev_allowed_for` equal to `min_release`. A box that does not is below
365 the floor for the door (MV-N5).
366
367### P-4 Ship moveprobe
368
369- [ ] Follow `cmd/moveprobe/README.md` ("Build" and "Deploy") under the ops
370 account: ship, enable instances with private storage off, create the
371 store with `--env-prefix MOVEPROBE` and no `--environment`, set
372 `MOVEPROBE_SECRET`, and set the `every-minute` schedule.
373- [ ] Keep `MOVEPROBE_SECRET`'s value in your shell. `/report` and
374 `/arm-once` need it.
375- [ ] Leave `moves.ready_path` unset. A move then probes `/`, which answers
376 as `/ready` does.
377
378### P-5 Rehearsal hosts, created before the claim
379
380A host created while `CARLOS_HOST_CLAIM` is on keeps its data under a
381per-incarnation path, and the move door refuses it with MV-N167: "the
382instance keeps its data under a per-incarnation path, which moves cannot
383cut yet". So every rehearsal host is created **before** the claim goes on
384for its deployment (Paul, N2(a)).
385
386- [ ] Before `CARLOS_HOST_CLAIM` goes on, create one moveprobe host per
387 planned run, in the region that run starts from, plus two spares:
388
389 ```sh
390 carlos instances create --app moveprobe --account <ops> --host <label>.<instance domain> --region <region>
391 ```
392
393 On dev, create at least one spare in `ie`. If Run 1b fails, its repeat
394 is Run 1 and then Run 1b on a fresh host that starts in `ie`, and no
395 host can be created once the claim is on.
396
397 On the flagship, also create Run 6's disposable customer-shaped
398 instance now.
399- [ ] Record the list of hosts, with the run each one is for, in the
400 deployment's rehearsal record (outside this repo). **Never reuse a
401 host.** A host that has run once is spent, whatever the outcome. The one
402 exception is Run 1b, which moves Run 1's host back into home (Ruling
403 R6).
404- [ ] After the claim is on, check each host is adopted: [deploy]
405 `control/hosts/<host>.json` exists, `holder.location` is
406 `instances/<host>` (not `instances/<host>/<inc>`), and `op.kind` is
407 `serving`. The console's claim recovery pass adopts them.
408
409### P-6 The rest of the fleet's preconditions
410
411Paul confirms each of these per deployment before its first run.
412
413- [ ] **Claim on.** `CARLOS_HOST_CLAIM` is set (any non-empty value) in
414 the console's environment and every box's `host.env`. Check: [box] on
415 each box and the console. #250's activation blockers are closed.
416- [ ] **Roster.** Each rehearsal box is an active, enrolled entry with no
417 termination fence. Check: [deploy] the decoded roster (`roster` above)
418 has `boxes.<box>.state` `active` and `boxes.<box>.enrollment` set, and
419 `control/box-fences/<enrollment>.json` does not exist.
420- [ ] **Box identity records.** `CARLOS_REGION_HA=true` in each rehearsal
421 box's `host.env`. It takes `true`, `1` or `t`; **`on` reads as off**.
422 `CARLOS_AWS_ACCOUNT` is set in the console's environment. Check: [deploy]
423 `control/boxes/<box>.json` exists for each rehearsal box, and its edges
424 file has `"capabilities": ["region-ha-1"]`.
425- [ ] **One eligible box per rehearsal region.** Any other box in a
426 rehearsal region runs without `CARLOS_REGION_HA`, so it publishes no
427 `region-ha-1` and is not eligible. Otherwise the door answers M-C21.
428- [ ] **Secrets key.** `CARLOS_SECRETS_KEY` on every rehearsal box, from
429 the same key pair as the console's. Check: [deploy] each edges file's
430 `secrets_key_fp` is the same; once the switch is on, it equals
431 `secrets_key_fp` in `control/moves-console.json`; [detail]
432 `detail.target_report.checks.T2.ok` is `true`.
433- [ ] **Staging bucket.** Each rehearsal box's edges file has
434 `data_bucket` set.
435- [ ] **Alert sinks.** `CARLOS_ALERT_WEBHOOK` or
436 `CARLOS_ALERT_SNS_TOPIC_ARN` is set on every rehearsal box and on the
437 console. Without one, pages go nowhere.
438- [ ] **Cgroup delegation.** Check: [box] `journalctl -u carlos-edge -b |
439 grep 'exec-child accounting'` shows `armed (delegated subtree ...)`. A
440 box without it cannot produce stop evidence and never cuts.
441- [ ] **Flat rows and tagged releases**, as P-0 and P-3 say.
442
443### P-7 The switch
444
445- [ ] P-0 to P-6 are done for this deployment.
446- [ ] **The moves copy record is fully approved**: no row in
447 `docs/superpowers/plans/2026-10-06-moves-phase1-copy.md` is `pending`
448 (`grep -c '| pending |'` on it prints 0). The switch never goes on
449 while a sentence an operator could read is unapproved.
450- [ ] **Paul's scoped go** in the amadan thread to switch moves on, naming
451 the deployment, the console and the boxes.
452- [ ] `CARLOS_MOVES_PHASE1=on` in the environment of the console and of
453 the boxes the go names, and nowhere else, then restart them.
454- [ ] Check: `carlos move status --app moveprobe --account <ops> --host
455 <host>` does not say "Moves are not enabled on this console."
456
457---
458
459## Between runs
460
461Before starting the next run, wait until all of these hold for the host
462the last run used:
463
464- [ ] [detail] `state` is `none` (no move in flight) and `last.outcome` is
465 recorded.
466- [ ] [detail] `detail.claim.kind` is `serving`.
467- [ ] [deploy] the claim's `quiet_after` for the region the next move
468 goes to has passed (T12). It is set when an instance leaves a region and
469 lasts about 16 minutes. With a fresh host per run it rarely bites, but
470 check it.
471- [ ] `MOVEPROBE_FAIL_READY_IN` is unset: `carlos env list --app moveprobe
472 --account <ops>` does not show it.
473
474---
475
476## The steps and checks each run uses
477
478Runs refer to these by number. `<from>` and `<to>` are the run's regions.
479
480### Steps
481
482- **S-1 Wake and record.** `curl https://<host>/ready` answers `ready`.
483 Save [report] as `before-request.json`.
484- **S-2 Request.**
485
486 ```sh
487 carlos move --app moveprobe --account <ops> --host <host> --to <to> --manual-cut
488 ```
489
490 [CLI] M-C1 ("Moving <host> from <from> to <to>."), M-C2, M-C3, then
491 M-C12 ("Ready to cut. ..."), exit 4. A line "<to> cannot take this move
492 yet: <check>." is M-C4 and names the failing check.
493 [detail] at ready: every `detail.target_report.checks.T1` to `T12` has
494 `ok: true` (and `T13` once the cut guard lands);
495 `detail.target_report.attempts_admitted` is at least 1;
496 `detail.source_report.s1.ok` and `s2.ok` are `true`. Record `id` as
497 `<move>` and `detail.move.incarnation` as `<inc>`.
498- **S-3 Record and arm.** Save [report] as `before-cut.json`. Arm the
499 once-timer so it falls due around the cut, then cut within a few
500 seconds:
501
502 ```sh
503 curl -fsS -X POST -H "Authorization: Bearer $MOVEPROBE_SECRET" "https://<host>/arm-once?in=30"
504 ```
505
506 Record the answer's `name` and `due_at`.
507- **S-4 Capture, then cut.** In a second terminal, start [capture] for
508 this run: `capture <host> run-<n>-detail.jsonl`. Then, in the first:
509
510 ```sh
511 carlos move cut --app moveprobe --account <ops> --host <host>
512 ```
513
514 The cut waits until the move ends or needs an operator. The expected
515 sentences and exit code are in each run. Every run captures, so that
516 whatever the record held while the move was live (the evidence keys,
517 the rollback journal, a loss) can still be read after it closes.
518
519### Checks after a commit
520
521- **R-1 State arrived.** [report] on the target: `region` is `<pr-to>`;
522 `file_key_sha256` and `platform_key_sha256` equal `before-cut.json`'s;
523 `object` has no `error` and its `body` names this boot and `<pr-to>`;
524 `has_secret` is `true`. `curl https://<host>/ready` answers `ready`.
525- **R-2 No row lost.** [report] for `bg` and for `worker`: the sum of
526 `count` over all regions is at least `before-cut.json`'s, and equals the
527 highest `max_seq` (rows are numbered from 1 in one sequence per kind, so
528 any gap shows here).
529- **R-3 The schedule continues.** [report] `rows` has `kind: tick` with
530 `region: <pr-to>`, and its `count` rises about once a minute for three
531 minutes after the open. Every `fired` entry has `count: 1`. A minute
532 that fell due while the instance was offline fires at most once after
533 the open, and minutes in between are not replayed, so a short gap
534 around the cut is expected. A due time that fired twice fails the run.
535- **R-4 The once-timer fired once.** [report] `fired` has exactly one
536 entry with S-3's `name`, its `due_at` equals S-3's, and its `count` is 1.
537- **R-5 The source holds nothing live.** After [deploy]
538 `log/committed-source_parked.json` and `log/done.json` exist:
539 [data] `list-object-versions` on `<data-bucket-from>` under
540 `instances/<host>/` returns no versions and no delete markers, and
541 `parked/<inc>/<host>/` holds the retired replica objects with a
542 `manifest.json`. [deploy] the claim's `obligations` is empty.
543- **R-6 The source really stopped.** [deploy] `stopped-source.json` (the
544 key [capture] shows as `detail.evidence.source`) has `steps.tree.ok`
545 and `steps.stray.ok` both `true` and no `stray` field. [box] on the source box, the journal has no "the instance is not
546 proven stopped yet" line for `<host>` and no M-C20 line ("... is running
547 a process for <host> that this move cannot account for. ..."), and
548 `/data/.moves/.page-outbox/<host>/<move>/stray.json` does not exist.
549 The moveprobe worker outlives its app on purpose, so a worker that
550 survived the stop fails the cut's tree step and shows here, then
551 reverts at the cut's deadline. It does not page.
552
553### Checks after a rollback (Runs 2, 3 and 4)
554
555R-1 to R-4 and R-6 apply as written, with the source (where the
556instance is back) in place of the target: `<pr-from>` in place of
557`<pr-to>` in R-1 and R-3. R-5 is replaced:
558
559- **R-5b The target holds nothing live.** After [deploy]
560 `log/rolling_back-target-cleaned.json` and `log/rolled_back.json` exist:
561 [data] `list-object-versions` on `<data-bucket-to>` under
562 `instances/<host>/` and under `moves/<host>/<move>/` returns no versions
563 and no delete markers.
564- **R-7 The target's writes came back** (dirty rollbacks only). [report]
565 on the source: `rows` has a `bg` row with `region: <pr-to>` (the
566 target's probe region, `home` if the target is the home box), written
567 on the target while it ran there.
568- **R-8 The return lane is parked** (dirty rollbacks only). A dirty
569 rollback returns the target's writes through its own lane,
570 `moves/<host>/<rb>/` in the **source's** data bucket, where `<rb>` is
571 the rollback's id (from [capture]: the last rollback journal's `id`).
572 The source queues the lane's park in the host claim when it has probed.
573 The park runs only after its horizon (the return seal's bound) plus the
574 signature age (15 minutes) and the request limit (2 minutes), and the
575 rollback does not close until every obligation of the move is
576 acknowledged. So expect the cut's wait to last at least that long.
577 The obligation exists only while the move is live, so read it from the
578 claim file [capture] saved, never from the bucket afterwards:
579
580 ```sh
581 jq -s --arg loc "moves/<host>/<rb>" '[.[][] | select(.park.location == $loc)] | first' run-3-claim.jsonl
582 ```
583
584 It has `kind` `park-replica`, `region` set to the source box's
585 `CARLOS_REGION` (absent when the source is the home box), an `id`, a
586 `retirement`, and `park` with `location`, `incarnation`, `evidence_at`
587 and `request_limit_ns`. Record `retirement` as `<ret>`,
588 `park.incarnation` as `<pinc>` and `park.evidence_at`. An empty answer
589 means the capture missed the obligation: the check fails, and the run
590 is repeated on a spare host with the claim capture running.
591 After `park.evidence_at` plus 15 minutes plus `request_limit_ns`, and
592 once `log/rolled_back.json` exists:
593 [data] `list-object-versions` on `<data-bucket-from>` under
594 `moves/<host>/<rb>/` returns no versions and no delete markers, and
595 `parked/<pinc>/<host>/<ret>/manifest.json` exists in
596 `<data-bucket-from>`.
597
598A clean rollback needs two things: the stopped target's digests equal
599what the landing installed, and `src_covered` (the source's own replica
600reached the stopped position). The target box logs both:
601[box] `journalctl -u carlos-edge | grep 'the target of rollback'` shows
602"... stopped at TXID <n> (clean <bool>, src_covered <bool>)".
603
604---
605
606## Dev (Runs 1 to 4)
607
608**Dev topology (Paul, N4).** Dev's second region is applied: `uk`,
609`eu-west-2`, box `carlos-dev-uk`, data bucket
610`carlos-dev-uk-640012953448` (platform-infrastructure#42). Before Run 1:
611
612- [x] Bootstrap `carlos-dev-uk`. Done 2026-10-15: box `dennis-stenciling`
613 on v0.27.3, with `CARLOS_SWAP=overlap` and `CARLOS_REGION=uk`.
614- [x] SSM parameters: nothing to copy. No box reads SSM at boot or at
615 runtime. Never copy `/carlos/release-key` to a region whose host role can
616 read `/carlos/*`. In `eu-west-2` the `carlos-dev-uk` role can.
617- [x] `carlos-dev-uk` needs the dev console's `CARLOS_SECRETS_KEY` before
618 Run 1 (T2 and config sync). Done 2026-10-15: `dennis-stenciling` got the
619 dev key through SSM `eu-west-2` `/carlos/secrets-key`. Its
620 `secrets_key_fp` is `sha256:bb067582…f5fe`, the same as sophie's. The key
621 is backed up in 1Password as "CARLOS_SECRETS_KEY dev".
622- [ ] `CARLOS_REGION_HA=true` on `carlos-dev-ie` and `carlos-dev-uk`, and
623 `CARLOS_AWS_ACCOUNT` on the dev console.
624- [ ] Keep `carlos-dev-ie-b` ineligible: no `CARLOS_REGION_HA` in its
625 `host.env`, so `ie` has one eligible box (M-C21).
626- [ ] Decide Runs 2 and 3's arrangement (see "Runs 2 and 3: the
627 regions"), then create the dev rehearsal hosts (P-5) after
628 `carlos-dev-uk` serves and before dev's claim goes on: Run 1's and Run
629 4's in `ie`, Runs 2 and 3's in their arrangement's `<from>`, and two
630 spares, at least one of them in `ie` (P-5).
631
632**Dev's home region.** Dev Ireland is probably the home region (its box
633has no `CARLOS_REGION`). Confirm it with the dev console's
634`CONSOLE_HOME_REGION` and record it here: `____`. If `ie` is home:
635
636- Every dev run touches home, so every dev run needs P-0's
637 one-placementless-box item: Ruling R2's procedure for
638 `carlos-dev-ie-b`, Run 1 included.
639- Run 1 (`ie` to `uk`) moves out of home and needs **home fix part 1**.
640- Run 1b (`uk` to `ie`) moves Run 1's host back into home. It needs
641 **home fix part 2** rolled and dev's designation published.
642- Run 4 (`ie` to `uk`) moves out of home and then rolls back into it. It
643 needs **part 1**, and part 1 must cover a rollback whose source is home.
644 It stays out of home even once part 2 is rolled (Ruling R6), because
645 dev's console runs on the home box.
646- Runs 2 and 3 as written (`uk` to `ie`) move into home and need **home
647 fix part 2**. Once part 2 is rolled, they run as written.
648- **Until part 2 is rolled, only out-of-home moves run.** Run 1 goes ahead on
649 part 1. Runs 2 and 3 then use the **out-of-home arrangement**: the host
650 is created in `ie`, the move goes `ie` to `uk`, and the failure switch
651 names `<pr-uk>`. Each rolls back into a home source, as Run 4 does, so
652 this needs part 1 to cover a rollback whose source is home. Part 1 is
653 on main as 359e8ac2. Check in its own tests that it covers this before
654 the first such run. If it does not, Runs 2, 3 and 4 all wait for the
655 fix that does.
656- Run 4 runs only after Runs 1, 1b, 2 and 3 have passed, Runs 2 and 3 in
657 whichever arrangement they ran. No run is skipped.
658- On the home box, moveprobe reports `home`, so `<pr-ie>` is `home`. Any
659 [report] check or `MOVEPROBE_FAIL_READY_IN` aimed at that box uses
660 `home`, never `ie`.
661
662Record the dev probe regions here: `<pr-ie>` = `____`, `<pr-uk>` = `____`.
663
664If `ie` is not home, the runs go as written and need no home fix.
665
666### Run 1: dev, `ie` to `uk`, a commit
667
668**Blockers:** P-0; home fix part 1 if `ie` is home; if `ie` is home, P-0's
669one-placementless-box item, with Ruling R2's window open for
670`carlos-dev-ie-b`; dev topology above; P-1 to P-7 on dev; Paul's go.
671
672- [ ] Host used: `____` (created in `ie` before the claim). Run 1b moves
673 this host back.
674- [ ] S-1, S-2 with `--to uk`.
675- [ ] S-3, S-4. [CLI] M-C5 ("uk is ready. Starting the cut. <host> will be
676 offline for about <estimate>."), M-C6 ("Stopped in ie. Starting in
677 uk."), M-C7 ("Moved. <host> is served from uk. It was offline for
678 <seconds> seconds."), exit 0.
679- [ ] R-1 to R-6.
680- [ ] [deploy] `log/` has `cutting-flipped`, `committed`,
681 `committed-opened`, `committed-source_parked` and `done`. [detail]
682 `last.outcome` is `done`; `detail.timings` holds `offline_s` and
683 `cut_budget_s`.
684
685### Run 1b: dev, `uk` to `ie`, a commit into home
686
687**Blockers:** Run 1 passed; Run 1's quiet window has passed (T12, see
688"Between runs"); home fix part 2 merged and rolled, boxes first, then
689consoles; the fleet home designation published on dev (claim activation,
690#250); P-0's one-placementless-box item, with Ruling R2's window open for
691`carlos-dev-ie-b`; P-2's home data bucket item; Paul's go.
692
693This run moves Run 1's host back into home (Ruling R6). It is the one
694host this runbook uses twice (P-5). It has the same checks as Run 1, with
695`<pr-ie>` = `home`.
696
697**Note for the operator (Ruling R7).** Where the console shares the home
698box, as dev's does, the landing runs on the console's own box. If that box
699dies mid-landing, the console's move pass stops with it until the box or
700the console is back. The record and the claim are durable, so nothing is
701lost, but recovery waits for the console's return.
702
703- [ ] Host used: Run 1's, `____`.
704- [ ] Before the request, record the claim's touched regions and quiet
705 windows:
706
707 ```sh
708 aws s3 cp --profile <profile> s3://<deploy-bucket>/control/hosts/<host>.json - | jq '{regions: .touched.regions, quiet_after}'
709 ```
710
711- [ ] S-1, S-2 with `--to ie`. Name the home by its configured name. There
712 is no `home` keyword, because a named region may be called `home`
713 (Ruling R3). If S-2 refuses, the sentence says what is missing:
714 - MV-N510 ("moves into the home region need host claims. this console
715 runs without them"): the console runs without `CARLOS_HOST_CLAIM`;
716 - MV-N511 ("<box> sets CARLOS_REGION, so it cannot take a move into
717 the home region"): a box's edges file names `ie` as its placement.
718 That box is not the home box. Fix its `host.env`, or delete its
719 edges file if the box is gone;
720 - MV-N177 ("the host roster does not allow this move. the source and
721 target must be active boxes in uk and ie"): the roster has no home
722 designation, or the console's home name differs from it;
723 - "the fleet's edges files could not all be read" or MV-N5: Ruling R2's
724 window is not ready (P-0, step 4).
725
726 Each of these refusals comes before any claim write.
727- [ ] At ready, the staging lane is in the home data bucket:
728 [data] `list-object-versions` on `<data-bucket-ie>` (the home data
729 bucket P-2 recorded) with `--prefix "moves/<host>/<move>/"` returns
730 versions.
731- [ ] S-3, S-4. [CLI] M-C5 ("ie is ready. Starting the cut. <host> will be
732 offline for about <estimate>."), M-C6 ("Stopped in uk. Starting in
733 ie."), M-C7 ("Moved. <host> is served from ie. It was offline for
734 <seconds> seconds."), exit 0.
735- [ ] R-1 to R-6, with `<pr-to>` = `home`.
736- [ ] [deploy] `log/` has `cutting-flipped`, `committed`,
737 `committed-opened`, `committed-source_parked` and `done`. [detail]
738 `last.outcome` is `done`; `detail.timings` holds `offline_s` and
739 `cut_budget_s`.
740- [ ] [deploy] the claim's `touched.regions` holds at least one spelling
741 of home: `ie`, `@home`, or both. Touch adds `@home` when the target box
742 is new to `touched`, and never removes `ie`. Every region it held
743 before the request is still there.
744- [ ] [deploy] the claim's `quiet_after` names `uk`.
745- [ ] After the close: [data] `list-object-versions` on
746 `<data-bucket-ie>` under `parked/<inc>/<host>/` holds the residue of
747 Run 1's stay in home, if there was any.
748- [ ] If any check fails, repeat Run 1 on a spare host created in `ie`,
749 then Run 1b on that host.
750
751### Runs 2 and 3: the regions
752
753Runs 2 and 3 use one of two arrangements, chosen before the dev hosts are
754created (see "Dev's home region"):
755
756- **As written**: the host is created in `uk`, and the move goes `uk` to
757 `ie`. Needs home fix part 2 if `ie` is home. Once part 2 is rolled,
758 Runs 2 and 3 run as written, and they wait for Run 1b: the first move
759 into home is Run 1b's plain commit, never a forced rollback.
760- **Out of home** (`ie` is home and part 2 is not rolled): the host is
761 created in `ie`, and the move goes `ie` to `uk`. Needs part 1 to cover a
762 rollback whose source is home. This stays the fallback while part 2 is
763 not rolled.
764
765Below, `<from>` and `<to>` are the arrangement's regions as the CLI names
766them, and `<pr-from>`, `<pr-to>` their probe regions. Record the
767arrangement used in each run.
768
769### Run 2: dev, forced clean rollback
770
771**Blockers:** P-0; the home fix its arrangement needs (above); if `ie` is
772home, P-0's one-placementless-box item, with Ruling R2's window open for
773`carlos-dev-ie-b`; Run 1 passed; in the as-written arrangement, Run 1b
774passed (the first move into home is a plain commit); Paul's go.
775
776The target fails its readiness check in before-write mode, so it writes
777nothing and the rollback should be clean.
778
779- [ ] Arrangement: `____`. Host used: `____` (created in `<from>` before
780 the claim).
781- [ ] Before the request, set the failure switch to the target's probe
782 region:
783
784 ```sh
785 carlos env set --app moveprobe --account <ops> MOVEPROBE_FAIL_READY_IN=<pr-to>
786 ```
787
788 The variable is app-wide. Any moveprobe instance that starts in that
789 region while it is set answers 503 and writes nothing. That is expected.
790- [ ] S-1, S-2 with `--to <to>`, S-3, S-4 (capture first). [CLI] M-C8
791 ("<to> could not start <host>: <reason>. Moving it back to <from>."),
792 then M-C9 ("<host> is back in <from> and serving. Nothing was lost."),
793 exit 3.
794- [ ] From [capture], once the cut has returned and the capture has
795 stopped: the last rollback journal has `target_phase` `released` or
796 later, and `dirty` absent or `false`. Record its `id` as `<rb>`, and the
797 last `detail.evidence.target`.
798- [ ] Digests: [deploy] `stopped-target.json`'s `db_sha256`, `files_plain`
799 and `timers_sha256` equal `log/rolling_back-target-released.json`'s
800 fields `installed_db_sha256`, `files_plain` and `timers_sha256`.
801- [ ] [box] on the `<to>` box: "clean true, src_covered true".
802- [ ] R-1 to R-4 and R-6 on `<from>` (probe region `<pr-from>`), R-5b on
803 `<to>`.
804- [ ] [deploy] `log/` has `rolling_back`, `rolling_back-target-released`,
805 `rolling_back-target-cleaned`, `rolling_back-source-opened` and
806 `rolled_back`, and no `rolling_back-target-returning`.
807- [ ] After the checks are recorded:
808
809 ```sh
810 carlos env unset --app moveprobe --account <ops> MOVEPROBE_FAIL_READY_IN
811 ```
812
813A dirty result here, with moveprobe in before-write mode, is a platform
814finding, not a probe failure. Stop the series and report it.
815
816### Run 3: dev, forced dirty rollback
817
818**Blockers:** P-0; the home fix its arrangement needs; if `ie` is home,
819P-0's one-placementless-box item, with Ruling R2's window open for
820`carlos-dev-ie-b`; Run 2 passed; Paul's go.
821
822The target writes one row, starts its worker, and then fails readiness, so
823the rollback is dirty and the target's writes must come back.
824
825- [ ] Arrangement: `____`. Host used: `____` (created in `<from>` before
826 the claim).
827- [ ] Before the request:
828
829 ```sh
830 carlos env set --app moveprobe --account <ops> MOVEPROBE_FAIL_READY_IN=<pr-to>,after-write
831 ```
832
833- [ ] S-1, S-2 with `--to <to>`, S-3, S-4, with the capture started with
834 its claim file: `capture <host> run-3-detail.jsonl run-3-claim.jsonl`.
835 [CLI] M-C8, then M-C9, exit 3. The cut's wait includes the return
836 lane's park horizon (R-8), about 17 minutes or more.
837- [ ] From [capture]: the last rollback journal has `target_phase`
838 `released` or later and `dirty` `true`. Record its `id` as `<rb>`
839 before anything else; R-8 needs it.
840- [ ] Digests: [deploy] `stopped-target.json`'s `db_sha256` differs from
841 `log/rolling_back-target-released.json`'s `installed_db_sha256` (the
842 target wrote).
843- [ ] [box] on the `<to>` box: "clean false".
844- [ ] [deploy] `log/` has `rolling_back-target-returning` and
845 `rolling_back-target-returned`, then `rolling_back-target-cleaned` and
846 `rolled_back`.
847- [ ] R-1 to R-4 and R-6 on `<from>` (probe region `<pr-from>`), R-5b on
848 `<to>`, R-7 (a `bg` row with `region: <pr-to>`), and R-8 on `<from>`
849 (the return lane `moves/<host>/<rb>/` in `<data-bucket-from>`).
850- [ ] Unset `MOVEPROBE_FAIL_READY_IN` once the checks are recorded.
851
852### Run 4: dev, declare-lost
853
854**Blockers:** P-0, including claim Plan 12's loss-bound admission and Plan
8554 Task 10's box half; home fix part 1 if `ie` is home, covering a
856rollback whose source is home; if `ie` is home, P-0's
857one-placementless-box item, with Ruling R2's window open for
858`carlos-dev-ie-b`; **Runs 1, 1b, 2 and 3 passed** (Runs 2 and 3 in
859whichever arrangement they ran; Run 4 is never run ahead of them); Paul's
860go.
861
862The target box is stopped mid-move and declared lost. With only two dev
863regions, the target is `carlos-dev-uk`, and `uk` has no eligible box until
864the readmit at the end. **Stop the box. Do not terminate it.** A terminated
865box cannot be readmitted.
866
867Run 4 stays `ie` to `uk` even once part 2 is rolled (Ruling R6). It never
868stops the home box: dev's console runs there, so a declare-lost of the
869home box would stop the console too. It does not move into home itself.
870
871- [ ] Host used: `____` (created in `ie` before the claim).
872- [ ] **Paul's scoped go** in the amadan thread, before anything below,
873 to stop `carlos-dev-uk` during this run and take it out of service
874 until the readmit.
875- [ ] So the landing cannot commit before the box stops, set the failure
876 switch on the target first:
877
878 ```sh
879 carlos env set --app moveprobe --account <ops> MOVEPROBE_FAIL_READY_IN=<pr-uk>
880 ```
881
882 Its probe then fails however long the stop takes. If the target finishes
883 its release before the box stops, the run is an ordinary clean rollback:
884 record it and repeat Run 4 on a spare host.
885- [ ] Before the run, record the box's roster entry: [deploy]
886 `roster | jq '.boxes["carlos-dev-uk"]'` shows `state` `active`; note
887 its `enrollment` and `quarantine_revision`.
888- [ ] S-1, S-2 with `--to uk`, S-3, S-4 (capture first). The capture runs
889 until the move closes, through the declaration below.
890- [ ] When [capture] shows `state` `landing` (after `cutting-flipped`),
891 stop `carlos-dev-uk` through the AWS console (dev only).
892- [ ] The cut's wait ends with M-C10 ("Waiting for carlos-dev-uk in uk to
893 confirm it has stopped <host>. <host> is offline."), exit 4. Until the
894 landing deadline passes, it keeps waiting.
895- [ ] Declare it lost, with your own AWS credentials for dev in the shell
896 (the command reads EC2 with them and refuses to run as root):
897
898 ```sh
899 carlos move resolve --app moveprobe --account <ops> --host <host> --declare-lost carlos-dev-uk
900 ```
901
902 [CLI] M-C18 ("This declares carlos-dev-uk lost. Anything <host> wrote
903 on carlos-dev-uk after <time> is lost. Type the host name to
904 confirm."). Type `<host>`. The command then waits for the outcome:
905 expect M-C9 and exit 3.
906- [ ] Expected outcome: **rolled back, with the loss on the record**, not
907 necessarily clean. From [capture]: the last rollback journal has
908 `loss_ref` and `lost_at` set; record them and its `id`. [deploy]
909 `log/rolling_back-target-lost.json` (event `declared_lost`) and
910 `log/rolled_back.json` exist. [deploy] the decoded roster:
911 `boxes.carlos-dev-uk.state` is `quarantined`, `quarantine_revision` is
912 one more than before, and `loss_ids` holds the loss.
913- [ ] R-1 to R-4 and R-6 on `ie`, with probe region `<pr-ie>` (`home` if
914 `ie` is home).
915- [ ] Unset `MOVEPROBE_FAIL_READY_IN`.
916- [ ] Start `carlos-dev-uk` again. It stays quarantined: the deployment's
917 admission gate refuses it for every host until the readmit. Watch [box]
918 its journal from boot, and record anything it does for `<host>` before
919 the readmit. Doing anything for it there is a finding.
920- [ ] [deploy] the box retires its local state first and records it:
921 `control/host-retirement/carlos-dev-uk/<enrollment>/<quarantine_revision>.json`
922 exists with `"complete": true` and this boot's `boot_id`. Readmit
923 refuses without it.
924- [ ] Readmit it. `carlos ops host` exists only with `CARLOS_HOST_CLAIM`
925 set in your shell:
926
927 ```sh
928 CARLOS_HOST_CLAIM=1 carlos ops host resolve <host> --readmit carlos-dev-uk --console <console>
929 ```
930
931 Answer `readmit` at its prompt. The box must be running and reporting:
932 readmit refuses a box with a termination fence, and one whose private
933 report is older than 15 minutes. Before it, check [deploy]
934 `control/box-fences/<enrollment>.json` does not exist (a stopped box
935 has none; a terminated one does).
936- [ ] [deploy] the decoded roster: `boxes.carlos-dev-uk.state` is `active`
937 again, with the same `enrollment`.
938
939---
940
941## Flagship (Run 5)
942
943### Run 5: the flagship, Runs 1 to 3 again
944
945**Blockers:** P-0; P-1 to P-6 on the flagship; **P-3, the floor, published**
946(§15 step 6: before any protected production rehearsal); the moves copy
947record fully approved (no `pending` row); **dev Runs 1 to 4 passed**
948(each run gates the next, spec §15 step 7; claim Plan 8b is on main, so
949the old clause that let Run 5 go ahead with Run 4 skipped no longer
950applies); Paul's explicit go in the thread, for each of the three.
951
952- **The switch** (Paul, N17): `CARLOS_MOVES_PHASE1=on` in the environment
953 of the console and of the two boxes in play, and nowhere else. The
954 compiled default stays off. It flips later, in its own commit.
955- **Regions** (N3): the flagship's home is Ireland, so Run 5 uses two
956 non-home regions, `uk` and `us`, until the home fix parts land. Run 1's
957 shape goes `uk` to `us`. Runs 2 and 3 go `us` to `uk`, with
958 `MOVEPROBE_FAIL_READY_IN=<pr-uk>` and `<pr-uk>,after-write`. Record the
959 flagship's probe regions: `<pr-uk>` = `____`, `<pr-us>` = `____`.
960
961- [ ] P-3 on the flagship, after its own scoped go.
962- [ ] P-7 on the flagship: the copy record fully approved, then Paul's
963 scoped go naming the console and the `uk` and `us` boxes, then the
964 switch on those three only.
965- [ ] Hosts used: Run 1 shape `____` (created in `uk`), Run 2 shape `____`
966 and Run 3 shape `____` (created in `us`), all before the flagship's
967 claim went on.
968- [ ] Each of the three as on dev, with every check.
969- [ ] After the third, and only once **all three moves are closed**: for
970 each of the three hosts, [detail] `state` is `none` (no move in flight)
971 and `detail.claim.kind` is `serving`. If any move is unfinished (stuck,
972 a release past its deadline, relocating, or any exit 4), leave the
973 switch on and resolve it first with `carlos move resolve` (`--retry`
974 once the cause is fixed, or `--declare-lost <box>` for a lost target),
975 each with Paul's scoped go in the thread. The switch must stay on until the move is closed: with it off, a box's
976 moves pass skips every Phase 1 record, so an unfinished move would stop
977 being reconciled and stay where it is.
978- [ ] Then remove `CARLOS_MOVES_PHASE1` from the two boxes and the
979 console unless Paul says to keep it, and restart them. Check:
980 `carlos move status --app moveprobe --account <ops> --host <host>` says
981 "Moves are not enabled on this console.", and neither box's `host.env`
982 still sets it.
983
984---
985
986## Customer instances (Runs 6 and 7)
987
988The runbook names no tenant. The real instances, their owners and the
989agreed windows live in the flagship's deployment repo and in the amadan
990thread where Paul gives each go.
991
992These runs move a customer's app, not moveprobe. **Do not call `/ready`,
993`/arm-once` or `/report` on a customer host, and do not set
994`MOVEPROBE_FAIL_READY_IN`.** Those are moveprobe's own, and S-1, S-3, R-1
995to R-4 and R-7 do not apply. Set `ACCT=<account>` and `APP=<app>` for
996[detail], [capture] and the log prefix, and use `--app <app> --account
997<account>` on every command. `<from>` and `<to>` are the regions Paul
998approves for the run.
999
1000### Customer steps
1001
1002- **C-0 Switch on for this run only.** Run 5's switch was removed after
1003 it, and this run's participants differ (the source is the home box).
1004 - The moves copy record is fully approved (P-7).
1005 - **Paul's scoped go** in the amadan thread naming the flagship, this
1006 run, the console, and this run's two participant boxes: the source
1007 box in `<from>` (the instance's box now) and the target box in
1008 `<to>`.
1009 - `CARLOS_MOVES_PHASE1=on` in the environment of the console and of
1010 those two boxes, and nowhere else; restart them.
1011 - Check: `carlos move status --app <app> --account <account> --host
1012 <host>` does not say "Moves are not enabled on this console." [box]
1013 each of the two boxes' `host.env` sets it, and no other box's does.
1014- **C-1 Health check, before.** The owner names one health check for the
1015 instance (a URL and the answer that means it is serving) and approves
1016 it for the run. Record it in the deployment repo. Run it and record the
1017 answer.
1018- **C-2 Request.**
1019
1020 ```sh
1021 carlos move --app <app> --account <account> --host <host> --to <to> --manual-cut
1022 ```
1023
1024 [CLI] M-C1, M-C2, M-C3, then M-C12, exit 4. [detail] at ready, the same
1025 checks as S-2: every target check `ok`, `attempts_admitted` at least 1,
1026 `source_report.s1.ok` and `s2.ok` `true`. Record `<move>` and `<inc>`.
1027- **C-3 Capture, then cut.** Start [capture] in a second terminal, then,
1028 in the window agreed with the owner:
1029
1030 ```sh
1031 carlos move cut --app <app> --account <account> --host <host>
1032 ```
1033
1034 [CLI] M-C5, M-C6, M-C7, exit 0.
1035- **C-4 Health check, after.** The owner's check from C-1 gives the
1036 serving answer once M-C7 has printed. The owner confirms it.
1037- **C-5 Platform checks.** R-5 as written. R-6 as written, except its
1038 last two sentences, which are about moveprobe's worker. [deploy] `log/`
1039 has `cutting-flipped`, `committed`, `committed-opened`,
1040 `committed-source_parked` and `done`.
1041
1042If the move rolls back (M-C8 then M-C9, exit 3), the instance is back in
1043`<from>`: run C-4 there, then R-5b and R-6, and stop the series. C-6
1044still applies, once the rollback has closed.
1045
1046- **C-6 Switch off, once the move is closed.** First confirm the move is
1047 closed: [detail] `state` is `none` (no move in flight) and
1048 `detail.claim.kind` is `serving`. If it is not (stuck, a release past
1049 its deadline, relocating, or any exit 4), leave the switch on and
1050 resolve the move first with `carlos move resolve --app <app> --account
1051 <account> --host <host>` (`--retry` once the cause is fixed, or
1052 `--declare-lost <box>` for a lost target), with Paul's scoped go in the
1053 thread, then check again. The switch must stay on until the move is closed: with it off, a box's
1054 moves pass skips every Phase 1 record, so an unfinished move would stop
1055 being reconciled and stay where it is.
1056 Once it is closed, and after the run's checks are recorded: remove
1057 `CARLOS_MOVES_PHASE1` from the console and the two boxes, unless Paul
1058 says in the thread to keep it, and restart them. Check: `carlos move
1059 status --app <app> --account <account> --host <host>` says "Moves are
1060 not enabled on this console.", and [box] neither box's `host.env` sets
1061 it.
1062
1063### Run 6: a disposable customer-shaped instance
1064
1065**Blockers:** P-0; **the home-region parks fix, both parts, merged and
1066rolled** (customer instances live in the home region); dev Runs 1 and 1b
1067passed; Run 5 passed; the owner's agreement; Paul's go.
1068
1069A disposable instance of a customer app with the same shape as moveprobe
1070(a key file, a store, a sealed secret), created for the run with the
1071owner's agreement, before the flagship's claim went on (P-5).
1072
1073- [ ] Instance used: recorded in the deployment repo.
1074- [ ] C-0 to C-6.
1075
1076### Run 7: the first production customer instance
1077
1078**Blockers:** everything Run 6 needs; Run 6 passed; Paul's approval of the
1079instance; a window agreed with its owner.
1080
1081The first production customer instance Paul approves, in a window agreed
1082with its owner.
1083
1084- [ ] The instance was created before the flagship's claim went on, so it
1085 is adopted at `instances/<host>` (P-5). One created later cannot move.
1086- [ ] C-0 to C-6, with C-3 inside the agreed window. Run 6's switch was
1087 removed at its C-6, so C-0 needs its own go. The cut window is the
1088 default 120 s.
1089- [ ] Report `offline_s` back to the owner: [detail] `last.offline_s`, or
1090 `offline_s` in `log/done.json`'s fields.
1091
1092---
1093
1094## After every run
1095
1096- [ ] Attach to the amadan thread: the `done` or `rolled_back` log entry
1097 (its timings), the bucket listings, and the host used. For Runs 1 to 5,
1098 also the [report] files from before and after. For Runs 6 and 7, which
1099 never call `/report`, attach instead the owner-approved health check's
1100 answers from C-1 and C-4, and the owner's confirmation.
1101- [ ] A failed check stops the series. Fix its cause and repeat the run on
1102 a fresh host before going on.
1103
1104---
1105
1106## Sources
1107
1108Paths are from the repo root. Line numbers are main's (`bc25674a`, the
1109base of this branch); `cmd/moveprobe` is this branch's and not on main
1110yet. Spec is `docs/superpowers/specs/2026-10-02-move-any-app-design.md`;
1111copy is `docs/superpowers/plans/2026-10-06-moves-phase1-copy.md`.
1112
1113**`carlos move` (the CLI)**
1114
1115| what the runbook says | where |
1116|---|---|
1117| verbs `status`, `cut`, `abort`, `resolve`; flags `--host`, `--app`, `--account`, `--console` | `move.go:136-149` |
1118| `--to`, `--manual-cut`, `--no-wait` on a request; `--wait` on status; `--retry`, `--declare-lost`, `--no-wait` on resolve | `move.go:151-163` |
1119| exit codes 0, 1, 2, 3, 4, 5 | `move.go:40-45`, `:53-61` |
1120| "carlos move needs --app" | `move.go:66`, `:230` |
1121| "Moves are not enabled on this console." | `move.go:65`, `:240-243` |
1122| refuses to run as root; declare-lost reads EC2 with your own credentials | `move.go:211-217`, `:80`, `:900-903` |
1123| the request sends `region` and `cut` | `move.go:248-253` |
1124| M-C2, M-C3, M-C4 | `move.go:703`, `:707`, `:717` |
1125| M-C12 at ready with a manual cut, exit 4 | `move.go:722-726` |
1126| M-C10, exit 4 | `move.go:740-754` |
1127| M-C7 on done, M-C9 on rolled_back, exit 3 | `move.go:788-803` |
1128| M-C18 and the typed host name | `move.go:860-882` |
1129| "No move is in flight" | `move.go:595` |
1130| M-C1, M-C5, M-C6, M-C7, M-C8, M-C9 texts | `internal/moves/operatorcopy.go:11-38`; copy record rows M-C1 to M-C22, `copy:24-45` |
1131| the detail path, `/api/cli/apps/<acct>/<app>/instances/<host>/move?detail=1` | `move.go:230-232`, `:351-356`; `apimode.go:460-461` |
1132| login tokens in `~/.carlos/credentials`, keyed by `console` | `auth.go:113-120`, `:139-148` |
1133
1134**The detail answer**
1135
1136| what | where |
1137|---|---|
1138| detail=1 only for operators, only with the switch on | `internal/console/apiinstancemove.go:312-327` |
1139| `detail.move` is nil once no move is in flight, so the capture loop keeps it | `internal/console/apiinstancemove.go:1291-1292`, `:1416-1423` |
1140| a mutation waits for the outcome unless `--no-wait` | `move.go:186-188` |
1141| flat fields `state`, `target`, `id`, `last`, `protocol`, `phase` | `internal/console/apiinstancemove.go:160-170` |
1142| `detail.move`, `claim`, `target_report`, `source_report`, `evidence`, `log`, `timings`, `from`, `region` | `internal/console/apiinstancemove.go:1290-1318` |
1143| `evidence.source` and `evidence.target` come from the record in flight | `internal/console/apiinstancemove.go:1367-1370`, `:1416-1423` |
1144| `last` fields `id`, `outcome`, `at`, `offline_s` | `internal/provision/provision.go:1059-1064` |
1145| `move.incarnation` | `internal/provision/provision.go:961` |
1146| `rollback.dirty`, `target_phase`, `evidence`, `loss_ref`, `lost_at` | `internal/provision/provision.go:1067-1088` |
1147| `landing.installed_db_sha256` | `internal/provision/provision.go:1052` |
1148| `target_report.checks` T1 to T12, `attempts_admitted`, `cut_budget_s` | `internal/moves/readiness.go:18-31`, `:58-67` |
1149| `source_report.s1`, `s2`; S1 refuses a nested instance | `internal/moves/readiness.go:102-109`; `internal/host/movesphase1.go:41`, `:537` |
1150
1151**The log and the buckets**
1152
1153| what | where |
1154|---|---|
1155| identities and events: `requested`, `cutting-requested`/`cut_requested`, `cutting-flipped`/`flipped`, `committed-opened`/`opened`, `committed-source_parked`/`source_parked`, `rolling_back`, `rolling_back-target-<phase>`, `rolling_back-target-lost`/`declared_lost`, `rolling_back-source-<phase>`, `stuck-<epoch>`, the closes; the state as the event for `ready` and `committed` | `internal/moves/transition.go:122-253` |
1156| `retry-<epoch>` | `internal/host/movesevents.go:109` |
1157| the fields every entry carries | `internal/moves/transition.go:42-71` |
1158| the timing keys | `internal/moves/log.go:126-133` |
1159| target phases `releasing`, `released`, `returning`, `returned`, `cleaning`, `cleaned`, `lost`; source phases up to `recovered`, `opened` | `internal/provision/provision.go:1148-1172` |
1160| released goes to returning only when dirty, else to cleaning | `internal/host/movesrollback.go:566-575` |
1161| report prefix `moves/<acct>/<app>/<host>/<op>/`, `source.json`, `target.json`, `stopped-<role>.json`, `log/<identity>.json` | `internal/moves/staging.go:39-50` |
1162| the staging lane `moves/<host>/<op>/` in the target's data bucket | `internal/moves/staging.go:6` |
1163| the record's account is the sqid | `internal/console/apiinstances.go:1240` |
1164| the replica under `<CARLOS_HIBERNATE_PREFIX>/<host>` | `internal/activator/activator.go:119`, `:144-146`; `internal/host/movespass.go:471` |
1165| parked objects at `parked/<inc>/<host>/<retirement>/`, a `manifest.json`, the source versions deleted | `internal/park/remote.go:230-232`, `:473`, `:497` |
1166| stopped report `steps.tree`, `steps.stray`, `stray` (omitted when empty), digests | `internal/moves/evidence.go:11-20`, `:38-53` |
1167| a clean rollback: `src_covered` and unchanged digests; the "clean ..., src_covered ..." line; `dirty` and `evidence` written at released | `internal/host/movesrollback.go:394-398`, `:521-526` |
1168| the rollback's `id` | `internal/provision/provision.go:1068` |
1169| a dirty rollback's return lane is `moves/<host>/<rb>`, parked in the source's region, queued when the source has probed, horizon the return seal's bound | `internal/moves/parkscope.go:66-72`; `internal/host/movesrecover.go:658-675`; `internal/hostclaim/move.go:1093-1108` |
1170| a claim obligation's fields: `id`, `op`, `kind`, `region`, `retirement`, and `park` with `location`, `incarnation`, `evidence_at`, `request_limit_ns`; the claim's `obligations` list | `internal/hostclaim/claim.go:116-121`, `:137-147`, `:166` |
1171| the return lane's obligation `region` is the source box's `CARLOS_REGION` (omitted when empty) | `internal/host/movesrecover.go:675`; `internal/host/movespass.go:206`; `internal/hostclaim/claim.go:144` |
1172| the claim at `control/hosts/<host>.json` is plain JSON, read as is | `internal/hostclaim/claim.go:178-182`; `internal/hostclaim/store.go:38-60` |
1173| the detail answer's claim view carries no obligations | `internal/console/apiinstancemove.go:1356-1362` |
1174| a rollback closes only when nothing is owed under its move; a finished park removes its obligation | `internal/host/movesrecover.go:805-809`; `internal/host/movesparks.go:134-142`; `internal/park/remote.go:568`; `internal/hostclaim/delete.go:83-98` |
1175| the parked archive is `parked/<park.incarnation>/<host>/<retirement>/` with `manifest.json` | `internal/park/remote.go:230-232`, `:473` |
1176| a park runs only after `evidence_at` plus the signature age (15 min) and the request limit (2 min) | `internal/park/remote.go:21`, `:516-519`; `internal/moves/parkscope.go:23` |
1177
1178**The door and the fleet**
1179
1180| what | where |
1181|---|---|
1182| moves into home: refused without host claims (MV-N510), and a box whose edges file names the home as its placement refused (MV-N511), both before any claim read; the home's staging from placementless edges files | `2026-10-15-moves-home-region-part2.md` (Task 5); merge commit a89e35ca; at that commit `internal/console/apiinstancemove.go:614`, `:665-666`, `internal/console/instancesplacement.go:199` |
1183| one placementless box: S1 and T8 count every edges file per placement, fresh or not, before the data-bucket check | `internal/host/movesreport.go:113` (`loadMoveFleet`) |
1184| Ruling R2's procedure, R6 (Run 4 stays out of home, Run 1b) and R7 (a home landing on the console's box) | `2026-10-15-moves-home-region-part2.md`, "Rulings needed" |
1185| a region with no staging bucket is refused | `internal/console/apiinstancemove.go:580-583`; staging from `data_bucket`, `internal/console/instancesplacement.go:366` |
1186| M-C21, more than one box | `internal/console/apiinstancemove.go:595-600` |
1187| MV-N5, below the floor | `internal/console/apiinstancemove.go:613-616`; `copy:55` |
1188| a box below the protocol, or not armed for the floor, or with an unjudged `carlos.prev`, is below the floor | `internal/console/boxselect.go:100-108` |
1189| MV-N167 for a holder at `instances/<host>/<inc>` | `internal/console/apiinstancemove.go:143`, `:647-651`; `copy:217` |
1190| a created holder is at `instances/<host>/<inc>`, an adopted one at `instances/<host>` | `internal/hostclaim/create.go:75`; `internal/hostclaim/adopt.go:20` |
1191| the source must be an active, enrolled roster entry with no termination fence | `internal/console/apiinstancemove.go:657-684`, `:1267-1273` |
1192| eligibility: fresh, enrolled, `region-ha-1`, the console's secrets fingerprint, not draining | `internal/boxchoice/boxchoice.go:21`, `:109-148` |
1193| `region-ha-1` and the identity record only with `CARLOS_REGION_HA` | `internal/host/edgesfile.go:156-164`; `internal/host/boxrecord.go:25-28`, `:158-166` |
1194| `CARLOS_REGION_HA` reads only ParseBool true values, so `on` is off | `internal/host/boxrecord.go:21-28` |
1195| identity records at `control/boxes/<box>.json` | `internal/boxid/record.go:21`, `:28` |
1196| `CARLOS_AWS_ACCOUNT` and `CONSOLE_HOME_REGION` on the console | `cmd/console/main.go:318`, `:321` |
1197| the claim at `control/hosts/<host>.json`; `holder.location`, `obligations`, `quiet_after` | `internal/hostclaim/claim.go:56`, `:166`, `:171`, `:182` |
1198| T12 reads `quiet_after`; H is 15 min plus 1 min skew | `internal/host/movesreport.go:756-762`; `internal/moves/lease.go:13`, `:22`; spec:2573 |
1199| the console's claim recovery pass adopts with the claim on | `internal/console/hostclaimbackfill.go:40-48` |
1200| the roster at `control/host-actors.json`; `boxes`, `state`, `enrollment`, `quarantine_revision`, `loss_ids`; a declaration quarantines and adds one to the revision | `internal/hostactors/roster.go:21`, `:30`, `:38-40`, `:72`, `:265-279` |
1201| the roster is written as a root-signed envelope `{payload, sig}`, the payload base64 JSON | `internal/hostactors/roster.go:249`; `internal/release/rootenvelope.go:29-44`; `internal/release/release.go:377-379` |
1202| a termination fence is `control/box-fences/<enrollment>.json`; no object, no fence | `internal/hostactors/fence_authority.go:14-34` |
1203| a quarantined box is refused admission | `internal/hostquarantine/quarantine.go:17-25` |
1204| `CARLOS_HOST_CLAIM` is any non-empty value | `cmd/console/main.go:1364-1366`; `hostclaim_cli.go:22` |
1205| `carlos ops host` exists only with `CARLOS_HOST_CLAIM` in the shell; `--readmit`, `--console`, the `readmit` prompt | `hostclaim_cli.go:35-39`, `:61-75`, `:143` |
1206| readmit refuses a fenced box and a private report older than 15 min | `internal/hostresolve/readmit.go:18-40` |
1207| readmit needs the box's complete retirement receipt for this boot at `control/host-retirement/<box>/<enrollment>/<revision>.json`, then sets the box `active` | `internal/hostresolve/readmit.go:52-66`, `:134`; `internal/hostquarantine/quarantine.go:72-82`, `:91-96` |
1208
1209**The floor, the switch and the boxes**
1210
1211| what | where |
1212|---|---|
1213| the floor key and shape, `{"min","min_release"}` | `internal/moves/floor.go:20`, `:61-64` |
1214| `vX.Y.Z` only; anything else fails closed | `internal/moves/floor.go:85-99` |
1215| no code writes the floor; only reads | `internal/moves/floor.go:66-82` |
1216| a box's floor only rises; lowering it is a manual step per box | `internal/host/movesfloor.go:31-36` |
1217| `moves_protocol`, `moves_floor_min_release`, `prev_release`, `prev_allowed_for`, `data_bucket`, `secrets_key_fp` in the edges file at `control/edges/<box>.json` | `internal/host/edgesfile.go:218`, `:270`, `:295-304`; `internal/console/instancesplacement.go:22`, `:83-86` |
1218| `control/moves-console.json` and its `secrets_key_fp` | `internal/moves/floor.go:25-33` |
1219| the switch, `on`/`1`/`true`, default off | `internal/moves/switch.go:6-23` |
1220| the copy record's `pending` rows keep the compiled default off; P-7 applies the same rule to the env switch (Paul's ruling) | `approved_moves_copy_test.go:811`; `docs/superpowers/plans/2026-10-06-moves-phase1-copy.md` |
1221| with the switch off the CLI says "Moves are not enabled on this console." (the switch-off check in Run 5 and C-6) | `move.go:65`, `:240-243` |
1222| with the switch off a box's moves pass skips every Phase 1 record, so the switch goes off only after the move closes | `internal/host/movespass.go:797-805` |
1223| `carlos move resolve --retry` and `--declare-lost <box>` | `move.go:159-163`, `:258-262` |
1224| the copy gate keeps the default off while a row is pending | `approved_moves_copy_test.go:811` |
1225| the default cut window, 120 s | `internal/moves/budget.go:17` |
1226| `CARLOS_LAYOUT_DEFAULT=flat` | `internal/host/layoutleg.go:68-70`; `system.go:714` |
1227| `carlos system status` | `system.go:75-76`; `README.md:272` |
1228| `carlos status --layout` shows each host's actual layout, box by box, and marks a box "(nested by default)"; `carlos system layout status` shows only the recorded intent | `cli.go:3528`; `status_instances.go:142-163`, `:224`; `system.go:709-716` |
1229| the agent is `carlos-edge.service`, reading `/etc/carlos/host.env`, with a delegated cgroup | `internal/host/units/carlos-edge.service:18-19`, `:101` |
1230| "exec-child accounting armed"; no manager, no stop evidence | `internal/cgroup/cgroup.go:181-184`; `internal/host/movespass.go:376-392` |
1231| "not proven stopped yet ... retrying until the cut's deadline" | `internal/host/movescut.go:422-427` |
1232| M-C20 and the page outbox `<DataDir>/.moves/.page-outbox/<host>/<move>/<rule>.json`, rule `stray` | `internal/host/movespaging.go:47-50`, `:87`, `:546-554`, `:586-588` |
1233| the tree step's "could not be stopped" | `internal/stopevidence/reconstruct_linux.go:185` |
1234| alert sinks `CARLOS_ALERT_WEBHOOK`, `CARLOS_ALERT_SNS_TOPIC_ARN`; the console arms them only with the claim and the switch on | `internal/alerting/alerting.go:191-209`; `cmd/console/main.go:1352-1362` |
1235| `CARLOS_SECRETS_KEY` on the box | `internal/host/movespass.go:311-316` |
1236| `/data` is the data directory | `internal/ops/ops.go:27` |
1237
1238**Timers and moveprobe**
1239
1240| what | where |
1241|---|---|
1242| a held timer row is skipped while the instance moves | `internal/schedule/runner.go:190-197` |
1243| a periodic schedule's next fire is the next boundary after now (no replay) | `internal/schedule/spec.go:119-125` |
1244| moveprobe's region is `home` with no `CARLOS_REGION`, else `CARLOS_REGION`; `/report` says which in `region_from` | `cmd/moveprobe/main.go:94-100` |
1245| the failure switch matches the probe's own region, so it must name `<pr-R>` | `cmd/moveprobe/main.go:103-106`, `:137-138` |
1246| a box's edges `placement` is its `CARLOS_REGION`, absent when unset (the home box) | `internal/host/edgesfile.go:205-211`, `:758` |
1247| `bg` and `worker` rows each numbered from 1 in one sequence | `cmd/moveprobe/main.go:543-550`, `:158`, `:674` |
1248| `/report`'s fields, `/arm-once`'s answer, the failure switch's modes | `cmd/moveprobe/README.md`, "Use it", "Make readiness fail on purpose" and "Strings" |
1249
1250**Other CLI commands**
1251
1252| what | where |
1253|---|---|
1254| `carlos env set`, `unset`, `list` with `--app` and `--account` | `appconfig_cli.go:342-350`, `:408-417`, `:462-467`; `cli.go:2045-2051` |
1255| `carlos instances create --host --region --account` | `instances_cli.go:297-310` |
1256
1257**Not on main, cited by name**
1258
1259| what | where |
1260|---|---|
1261| the cut guard and T13 | branch `moves-cut-guard`, commits `083a6eaa` and `6871d2e3` |
1262| the home parks fix part 2 (moves into home) | `2026-10-15-moves-home-region-part2.md`, merged to main as a89e35ca (branch `moves-home-part2-plan`); #236. Part 1 is on main as 359e8ac2 |
1263| claim Plan 12; moves Plan 5 Tasks 9b and 8, Plan 4 Tasks 9 and 10 | plans only; the landing, rollback and recovery are unarmed until then, `internal/host/movespass.go:335-340` |
1264| dev's second region | platform-infrastructure#42 |
1265
1266**Spec**
1267
1268| what | where |
1269|---|---|
1270| the runs and the pass rule | spec §14.4, spec:6263-6276 |
1271| the rollout order, the floor before any protected production rehearsal | spec §15, spec:6278-6294 |
1272| each rehearsal gates the next: Run 4 needs Runs 2 and 3, Run 5 needs Run 4 | spec:6294; controller rulings, fix round 1 and final review |
1273| the flagship's home is Ireland | spec:1032 |
1274