carlos / platform Public
Moves Phase 1: the rehearsal runbook
This is the checklist an operator follows to rehearse Phase 1 moves (spec §14.4, rollout §15). It covers eight runs in the spec's order: five on dev (Runs 1, 1b, 2, 3 and 4; Run 1b is a move into home), one on the flagship with the probe app, one on a disposable customer-shaped instance, and one on a real customer instance.
No run in this runbook can execute today. Every run waits for P-0 below. Each run also has its own blockers, listed at its top. Never run one without Paul's go in the amadan thread. Steps that change a deployment (publishing the floor, switching moves on, stopping a box) each have their own Paul's scoped go step just before them: a go in the amadan thread that names the deployment and exactly what the step changes. A go for one step or one run does not cover another.
The probe app is moveprobe. How to build it, ship it and use it is in cmd/moveprobe/README.md. This runbook does not repeat it.
Every command, flag, event, field and key below was checked against main, and moveprobe's against this branch. The Sources section at the end gives file and line for each.
---
How to read this runbook
Placeholders
| placeholder | what it is | |---|---| | <console> | the deployment's console origin, https://..., as carlos auth login stored it | | <ops> | the ops account's sqid, as --account takes it | | <account>, <app> | a customer run's account sqid and app (Runs 6 and 7) | | <host> | the run's host, <label>.<instance domain> | | <deploy-bucket> | the deployment bucket (CARLOS_DEPLOYMENT_BUCKET in a box's /etc/carlos/host.env) | | <data-bucket-R> | region R's data bucket: data_bucket in that region's box edges file | | <profile> | the deployment's named AWS profile. Always name one. There are no default credentials. | | <move> | the move id the console gave the request (id in the flat view) | | <inc> | the instance incarnation (detail.move.incarnation) | | <pr-R> | the region moveprobe reports on region R's box: home if that box has no CARLOS_REGION (its edges file has no placement), else its CARLOS_REGION, the placement code. <pr-from> and <pr-to> are the run's two. |
The probe's region is not always the region's name
moveprobe reports the region its box gives it, not the name the CLI and the console use. On a home box, which has no CARLOS_REGION, it reports home, while carlos move still says ie (or whatever CONSOLE_HOME_REGION is). Before each deployment's first run, write down <pr-R> for every rehearsal region: read CARLOS_REGION in the box's /etc/carlos/host.env, or placement in its edges file, and confirm with [report]'s region and region_from on any moveprobe host there. Every [report] region check and every MOVEPROBE_FAIL_READY_IN value uses <pr-R>, never the CLI's name.
Read paths
Each check says where to read it.
- [CLI]: the sentence
carlos moveprints and its exit code. Exit codes: 0 done; 3 rolled back or aborted; 4 waiting (ready for a manual cut, stuck, a release past its deadline, relocating, or an outcome not recorded); 5 refused; 1 the console could not be reached or read; 2 a usage error. - [detail]: the operator's detail answer. No
carloscommand prints it yet, so read it with your own login token. SetACCTandAPPfor the run:<ops>andmoveprobefor Runs 1 to 5,<account>and<app>for Runs 6 and 7.
``sh CONSOLE=<console> ACCT=<ops> APP=moveprobe TOKEN=$(jq -r --arg c "$CONSOLE" '.credentials[] | select(.console == $c) | .token' ~/.carlos/credentials) detail() { curl -fsS -H "Authorization: Bearer $TOKEN" "$CONSOLE/api/cli/apps/$ACCT/$APP/instances/$1/move?detail=1"; } detail <host> | jq . ``
The flat fields (state, phase, id, last) sit at the top. The rest is under detail: move (the record's journal), claim, target_report, source_report, evidence, log (keys only), timings (after the close) and region. detail.move and detail.evidence are gone once the move closes, and a mutation waits until it closes by default. So every run that needs them starts the capture loop below before its cut and reads them from the capture file afterwards.
- [capture]: every detail answer of one move, saved as it happens. Start it in a second terminal once the request has reached ready, before the cut. It stops by itself once no move is in flight. With a third argument it also saves, each time round, the host claim's
obligationsfrom the deployment bucket (control/hosts/<host>.json, plain JSON). The detail answer does not carry them, and they are gone once the move closes, so this is the only way to keep a park's horizon and retirement id. Run 3 needs it (R-8):
``sh capture() { # capture <host> <detail-file> [<claim-file>] while :; do if d=$(detail "$1"); then printf '%s\n' "$d" >> "$2" if [ -n "$3" ]; then aws s3 cp --profile <profile> "s3://<deploy-bucket>/control/hosts/$1.json" - \ | jq -c '.obligations // []' >> "$3" fi [ "$(printf '%s' "$d" | jq -r .state)" = none ] && break fi sleep 2 done } capture <host> run-<n>-detail.jsonl # Runs 1, 2, 4, 5 capture <host> run-3-detail.jsonl run-3-claim.jsonl # Run 3 # later: the last rollback journal the move showed jq -s '[.[] | .detail.move.rollback | select(. != null)] | last' run-<n>-detail.jsonl ``
- [deploy]: the deployment bucket, read with
--profile <profile>. Each move keeps its reports, stop evidence and log undermoves/$ACCT/$APP/<host>/<move>/:source.json,target.json,stopped-source.json,stopped-target.jsonandlog/<identity>.json. The claim iscontrol/hosts/<host>.json, each box's edges filecontrol/edges/<box>.json, each box's identity recordcontrol/boxes/<box>.json, and the floorcontrol/moves-protocol.json.
``sh P="moves/$ACCT/$APP/<host>/<move>/" aws s3 ls --profile <profile> "s3://<deploy-bucket>/${P}log/" aws s3 cp --profile <profile> "s3://<deploy-bucket>/${P}log/done.json" - | jq . ``
The roster, control/host-actors.json, is a signed envelope, {"payload": <base64 JSON>, "sig": ...}. Decode the payload to read it (this does not check the signature; the console does). A box's termination fence is a separate object, control/box-fences/<enrollment>.json, using the enrollment from the roster. No object means no fence:
``sh roster() { aws s3 cp --profile <profile> s3://<deploy-bucket>/control/host-actors.json - | jq -r .payload | base64 -d; } roster | jq '.boxes["<box>"] | {state, enrollment, loss_ids, quarantine_revision}' aws s3api head-object --profile <profile> --bucket <deploy-bucket> \ --key "control/box-fences/$(roster | jq -r '.boxes["<box>"].enrollment').json" # 404: no fence ``
- [data]: a region's data bucket, read with
--profile <profile>. The instance's replica is underinstances/<host>/(the box'sCARLOS_HIBERNATE_PREFIX,instancesin production), a move's staging lane undermoves/<host>/<move>/, and parked objects underparked/<inc>/<host>/<retirement>/with amanifest.json. These buckets are versioned, so list versions, not objects:
``sh aws s3api list-object-versions --profile <profile> --bucket <data-bucket-R> --prefix "instances/<host>/" ``
- [report]: moveprobe's own report, with the secret from your shell:
``sh curl -fsS -H "Authorization: Bearer $MOVEPROBE_SECRET" https://<host>/report | jq . ``
- [box]: on a box, the agent's journal (
journalctl -u carlos-edge),/etc/carlos/host.env, and files under the data directory (/data).
Log events
Each transition writes one log object, keyed by its identity. The ones the checks use:
| identity (the key) | event (the event field) | when | |---|---|---| | requested | requested | the request was journaled | | ready | ready | the target's checks passed | | cutting-requested | cut_requested | the cut was asked for | | cutting-<phase> | <phase> | admitted, stopping, stopped, staged, archived | | cutting-flipped | flipped | the record now names the target | | landing-<phase> | <phase> | the target's landing steps | | probe_attempt-<n> | probe_attempt | the landing's n-th readiness probe | | committed | committed | the landing committed | | committed-opened | opened | the target opened (offline_s in its fields) | | committed-source_parked | source_parked | the source parked what it held | | rolling_back | rolling_back | a rollback started (reason in its fields) | | rolling_back-target-<phase> | <phase> | released, returning, returned (a dirty rollback only), cleaning, cleaned | | rolling_back-target-lost | declared_lost | the target was declared lost | | rolling_back-source-<phase> | <phase> | the source's recovery, up to recovered and opened | | stuck-<epoch>, retry-<epoch> | stuck, retry | a stuck move and its retry | | done, rolled_back, aborted | the same | the close; done and rolled_back carry the timings |
Every entry's fields carry from, to and, once known, final_txid, db_sha256, timers_sha256, files_plain, installed_db_sha256, release, env_report and cut_budget_s. The close's timings are prepare_s, cut_s, cut_budget_s, land_s, land_budget_s, offline_s, rollback_s and release_budget_s. There is no stray event and no dirty field in the log.
---
P-0: what every run waits for
Nothing below P-0 runs until every item here is true. Paul confirms P-0 per deployment before its first run.
- [ ] Claim Plan 12 merged. Without it the landing, the target's half of a rollback and the source's recovery are unarmed in production, so a move with the switch on stops the source, flips, and then has nothing to land it, roll it back or recover it.
- [ ] Moves Plan 5 Task 9b, Plan 4 Tasks 9 and 10 (the box half), and Plan 5 Task 8 merged. They wire moves onto claim Plan 12. Run 4 also needs claim Plan 12's loss-bound admission.
- [ ] The moves cut guard merged. It is on main (b04ff852, Paul's ruling of 2026-10-11). The source refuses a cut unless the target reports its landing armed (the
T13check), and records why a cut did not start. - [ ] Moves' post-Plan 12 branch merged (
moves-phase1-5f). It brings the wake source, the recovery owners, the loss rules, the root held release (T7 stages,configuredpublishes, through root's control socket) and the C8 files-manifest provider. - [ ] For any run that touches the home region: the home-region parks fix part 1 merged (#236, on main as 359e8ac2) and claim Plan 13's home naming (a0806e4c, follow-ups 2ad5412c). Part 1 covers moves out of home. Any run that moves into home also needs part 2 (
2026-10-15-moves-home-region-part2.md) merged and rolled, boxes first, then consoles. The door refuses moves into home without host claims (MV-N510). A home box arms only when the claim grants it the@homealias, which needs the signed fleet home designation. Plan 13's rollout order is binding: every roster reader on a Plan 13 build first, then publish the designation, then turn the flag on. Older binaries cannot read a roster that carrieshome_region. For moves into home, the whole order is: 1. Plan 13 on every box and console. Check each box readscurrentwithcarlos system status. Plan 13's own order applies inside this step. 2. Part 2 merged (dormant), released, and rolled boxes first, then consoles. A console rolled first could admit a move into a home box that still fails T3, and that move waits atpreparinguntil an operator aborts it. 3. The IAM mesh, including the home data bucket (P-2), applied by Paul with a scopedtofu apply -target. 4. Claim activation (#250), which publishes the signed designation. Until then a home box never arms, and the door refuses claim-mode moves with MV-N177. 5.CARLOS_MOVES_PHASE1per run, as P-7 says. - [ ] The claim's activation checklist (#250) done for this deployment. Among its items: - the one-time files-manifest backfill for every host older than the files publisher. Until it has run for a host, that host refuses to wake (
ErrNoEvidence), by design; - claim Plan 15's admitted-generation record, so hosts with history past the eight-admission window can prove their file origins. Until then those hosts refuse; - claim Plan 14 (#280), so that root trusts no service-owned state; -CARLOS_RELEASE_PUBKEYreadable by instance control (/etc/carlos/host.env), or every held release is refused; - withCARLOS_MOVES_PHASE1on, a usableCARLOS_SECRETS_KEYon every box. Without it the moves source refuses to open and the box serves nothing. Its log names the cause once per start. - [ ] All of the above released and rolled, consoles first, then every box. Check every box with
carlos system status --console <console>: each readscurrenton the new version. - [ ] Each deployment's home region recorded:
CONSOLE_HOME_REGIONin the console's environment. A box with noCARLOS_REGIONin itshost.envpublishes noplacementin its edges file: it is a home box. - [ ] Dev's home region has two boxes: wake rehearsal apps on one. While dev's claim is off, both home boxes (
carlos-dev-ieandcarlos-dev-ie-b) take every unregioned record. A wake that switches box can serve a stale local DB (amadan #286;activator.go, "only while a region has one box"). So rehearsal apps in dev's home region are woken on one box only, or the run waits for dev's claim. #286 ruled this a dev-only hazard, accepted. During a window under the procedure in the next item,carlos-dev-ie-bis stopped, so this holds by itself. - [ ] One placementless box per deployment. For each deployment, list
control/edges/and record that exactly one edges file has noplacement:
``sh for k in $(aws s3api list-objects-v2 --profile <profile> --bucket <deploy-bucket> \ --prefix control/edges/ --query 'Contents[].Key' --output text); do printf '%s %s\n' "$k" "$(aws s3 cp --profile <profile> s3://<deploy-bucket>/$k - | jq -r '.placement // "(none)"')" done ``
S1 and T8 refuse a move into or out of home while two do, whether or not the second box is running, because an edges file outlives its box. They count every edges file per placement, fresh or not. A retained file below the floor makes the door refuse with MV-N5.
On dev, carlos-dev-ie-b makes two. Ruling R2 (option a) applies to every dev run that touches home, Run 1 included. Each window needs Paul's scoped go in the amadan thread, naming dev and carlos-dev-ie-b. The procedure, as the ruling gives it:
1. stop carlos-dev-ie-b (the instance, or at least its agent and console units, so it neither writes status nor serves); 2. delete control/edges/carlos-dev-ie-b.json from the deployment bucket; 3. wait until its last status/boxes/carlos-dev-ie-b.json is more than 15 minutes old by its generated_at; 4. before each run, verify the fleet's evidence is complete: every box with a status snapshot fresher than 15 minutes has an edges file, every edges file's box has a status snapshot, and exactly one edges file has no placement. The door's own confirmation is the run's S-2, the move request itself. Only a move request runs selectMoveBoxes and the floor check: carlos move status (handleAPIInstanceMoveStatus) reads the record and claim only, and S-1 only wakes the app, so neither confirms anything here. If S-2 answers "the fleet's edges files could not all be read" or MV-N5, the window is not ready. Both refusals come before any claim read or write, so nothing was started: fix the evidence and repeat S-2; 5. after the window, start the box again. Its edges pass rewrites the file on boot. No move touching home runs until the window is reopened by steps 1 to 4.
Step 3's wait matters. If the edges file is deleted while the box's status snapshot is still fresh, the door reads the fleet's evidence as incomplete and refuses every move.
- [ ] Rehearsal boxes keep instances flat. Moves refuse a nested instance. Set
CARLOS_LAYOUT_DEFAULT=flatin each rehearsal box'shost.env. Pre-flight check, before any request:carlos status --layout --console <console>(the layout each host is actually on, box by box) shows every rehearsal host's row asflat, and no rehearsal box is marked "(nested by default)".carlos system layout statusis not this check: it shows only the recorded intent. Once a run's request exists, S-2 also checksdetail.source_report.s1.ok, the source's own refusal of a nested instance. - [ ] Never set
CARLOS_MOVES_PHASE1=onanywhere before P-0 is true.
---
Preconditions, once per deployment
P-1 The dormant release
- [ ] The release carrying P-0 is rolled with
CARLOS_MOVES_PHASE1unset everywhere: consoles first, then boxes. - [ ] [deploy] Every box's
control/edges/<box>.jsonhas"moves_protocol": 1.
P-2 Staging grants (§15 step 5, scoped tofu by Paul)
- [ ] Every rehearsal region's
sibling_staging_arnslists every other rehearsal region's data bucket, and each box has the versioned-object grants (s3:ListBucketVersions,s3:GetObjectVersion,s3:DeleteObjectVersionon its own bucket;s3:GetObjectVersionfor the console). - [ ] The home data bucket is in the mesh. Every named region's
sibling_staging_arnslists the home data bucket's ARN, and the home's lists every named region's data bucket. Record the home data bucket's name (data_bucketin the home box's edges file, the one with noplacement):____. Check that it is not the deployment bucket (<deploy-bucket>). Without the mesh, a move into home fails the source's lane round withAccessDenied(logged), or T10 fails on the home box. - [ ] Check after the first request reaches ready: [detail]
detail.target_report.checks.T10.okistrue(the lane probe).
P-3 The protocol floor
Publish the floor only after every box and the console run a tagged vX.Y.Z release at or above min_release. Once a floor is armed, any build that is not vX.Y.Z (a SHA build, dev, a prerelease) is refused at activation, and the door treats that box as below the floor. A box never lowers its local floor on its own. If dev keeps rolling SHA builds, dev's runs go without a floor, and P-3 is done on the flagship only, before Run 5 (§15 step 6).
- [ ] Paul's scoped go in the amadan thread to publish the floor on this deployment, naming the deployment and
min_release. Nothing below runs without it. - [ ] Write it create-only from an operator shell. No
carloscommand writes it (a gap to note on an amadan discussion).
``sh printf '{"min":1,"min_release":"vX.Y.Z"}' > floor.json aws s3api put-object --profile <profile> --bucket <deploy-bucket> \ --key control/moves-protocol.json --body floor.json --if-none-match '*' aws s3 cp --profile <profile> s3://<deploy-bucket>/control/moves-protocol.json - ``
- [ ] [deploy] Every box's edges file shows
moves_floor_min_releaseequal to the floor'smin_release, and either"prev_release": "none"orprev_allowed_forequal tomin_release. A box that does not is below the floor for the door (MV-N5).
P-4 Ship moveprobe
- [ ] Follow
cmd/moveprobe/README.md("Build" and "Deploy") under the ops account: ship, enable instances with private storage off, create the store with--env-prefix MOVEPROBEand no--environment, setMOVEPROBE_SECRET, and set theevery-minuteschedule. - [ ] Keep
MOVEPROBE_SECRET's value in your shell./reportand/arm-onceneed it. - [ ] Leave
moves.ready_pathunset. A move then probes/, which answers as/readydoes.
P-5 Rehearsal hosts, created before the claim
A host created while CARLOS_HOST_CLAIM is on keeps its data under a per-incarnation path, and the move door refuses it with MV-N167: "the instance keeps its data under a per-incarnation path, which moves cannot cut yet". So every rehearsal host is created before the claim goes on for its deployment (Paul, N2(a)).
- [ ] Before
CARLOS_HOST_CLAIMgoes on, create one moveprobe host per planned run, in the region that run starts from, plus two spares:
``sh carlos instances create --app moveprobe --account <ops> --host <label>.<instance domain> --region <region> ``
On dev, create at least one spare in ie. If Run 1b fails, its repeat is Run 1 and then Run 1b on a fresh host that starts in ie, and no host can be created once the claim is on.
On the flagship, also create Run 6's disposable customer-shaped instance now.
- [ ] Record the list of hosts, with the run each one is for, in the deployment's rehearsal record (outside this repo). Never reuse a host. A host that has run once is spent, whatever the outcome. The one exception is Run 1b, which moves Run 1's host back into home (Ruling R6).
- [ ] After the claim is on, check each host is adopted: [deploy]
control/hosts/<host>.jsonexists,holder.locationisinstances/<host>(notinstances/<host>/<inc>), andop.kindisserving. The console's claim recovery pass adopts them.
P-6 The rest of the fleet's preconditions
Paul confirms each of these per deployment before its first run.
- [ ] Claim on.
CARLOS_HOST_CLAIMis set (any non-empty value) in the console's environment and every box'shost.env. Check: [box] on each box and the console. #250's activation blockers are closed. - [ ] Roster. Each rehearsal box is an active, enrolled entry with no termination fence. Check: [deploy] the decoded roster (
rosterabove) hasboxes.<box>.stateactiveandboxes.<box>.enrollmentset, andcontrol/box-fences/<enrollment>.jsondoes not exist. - [ ] Box identity records.
CARLOS_REGION_HA=truein each rehearsal box'shost.env. It takestrue,1ort;onreads as off.CARLOS_AWS_ACCOUNTis set in the console's environment. Check: [deploy]control/boxes/<box>.jsonexists for each rehearsal box, and its edges file has"capabilities": ["region-ha-1"]. - [ ] One eligible box per rehearsal region. Any other box in a rehearsal region runs without
CARLOS_REGION_HA, so it publishes noregion-ha-1and is not eligible. Otherwise the door answers M-C21. - [ ] Secrets key.
CARLOS_SECRETS_KEYon every rehearsal box, from the same key pair as the console's. Check: [deploy] each edges file'ssecrets_key_fpis the same; once the switch is on, it equalssecrets_key_fpincontrol/moves-console.json; [detail]detail.target_report.checks.T2.okistrue. - [ ] Staging bucket. Each rehearsal box's edges file has
data_bucketset. - [ ] Alert sinks.
CARLOS_ALERT_WEBHOOKorCARLOS_ALERT_SNS_TOPIC_ARNis set on every rehearsal box and on the console. Without one, pages go nowhere. - [ ] Cgroup delegation. Check: [box]
journalctl -u carlos-edge -b | grep 'exec-child accounting'showsarmed (delegated subtree ...). A box without it cannot produce stop evidence and never cuts. - [ ] Flat rows and tagged releases, as P-0 and P-3 say.
P-7 The switch
- [ ] P-0 to P-6 are done for this deployment.
- [ ] The moves copy record is fully approved: no row in
docs/superpowers/plans/2026-10-06-moves-phase1-copy.mdispending(grep -c '| pending |'on it prints 0). The switch never goes on while a sentence an operator could read is unapproved. - [ ] Paul's scoped go in the amadan thread to switch moves on, naming the deployment, the console and the boxes.
- [ ]
CARLOS_MOVES_PHASE1=onin the environment of the console and of the boxes the go names, and nowhere else, then restart them. - [ ] Check:
carlos move status --app moveprobe --account <ops> --host <host>does not say "Moves are not enabled on this console."
---
Between runs
Before starting the next run, wait until all of these hold for the host the last run used:
- [ ] [detail]
stateisnone(no move in flight) andlast.outcomeis recorded. - [ ] [detail]
detail.claim.kindisserving. - [ ] [deploy] the claim's
quiet_afterfor the region the next move goes to has passed (T12). It is set when an instance leaves a region and lasts about 16 minutes. With a fresh host per run it rarely bites, but check it. - [ ]
MOVEPROBE_FAIL_READY_INis unset:carlos env list --app moveprobe --account <ops>does not show it.
---
The steps and checks each run uses
Runs refer to these by number. <from> and <to> are the run's regions.
Steps
- S-1 Wake and record.
curl https://<host>/readyanswersready. Save [report] asbefore-request.json. - S-2 Request.
``sh carlos move --app moveprobe --account <ops> --host <host> --to <to> --manual-cut ``
[CLI] M-C1 ("Moving <host> from <from> to <to>."), M-C2, M-C3, then M-C12 ("Ready to cut. ..."), exit 4. A line "<to> cannot take this move yet: <check>." is M-C4 and names the failing check. [detail] at ready: every detail.target_report.checks.T1 to T12 has ok: true (and T13 once the cut guard lands); detail.target_report.attempts_admitted is at least 1; detail.source_report.s1.ok and s2.ok are true. Record id as <move> and detail.move.incarnation as <inc>.
- S-3 Record and arm. Save [report] as
before-cut.json. Arm the once-timer so it falls due around the cut, then cut within a few seconds:
``sh curl -fsS -X POST -H "Authorization: Bearer $MOVEPROBE_SECRET" "https://<host>/arm-once?in=30" ``
Record the answer's name and due_at.
- S-4 Capture, then cut. In a second terminal, start [capture] for this run:
capture <host> run-<n>-detail.jsonl. Then, in the first:
``sh carlos move cut --app moveprobe --account <ops> --host <host> ``
The cut waits until the move ends or needs an operator. The expected sentences and exit code are in each run. Every run captures, so that whatever the record held while the move was live (the evidence keys, the rollback journal, a loss) can still be read after it closes.
Checks after a commit
- R-1 State arrived. [report] on the target:
regionis<pr-to>;file_key_sha256andplatform_key_sha256equalbefore-cut.json's;objecthas noerrorand itsbodynames this boot and<pr-to>;has_secretistrue.curl https://<host>/readyanswersready. - R-2 No row lost. [report] for
bgand forworker: the sum ofcountover all regions is at leastbefore-cut.json's, and equals the highestmax_seq(rows are numbered from 1 in one sequence per kind, so any gap shows here). - R-3 The schedule continues. [report]
rowshaskind: tickwithregion: <pr-to>, and itscountrises about once a minute for three minutes after the open. Everyfiredentry hascount: 1. A minute that fell due while the instance was offline fires at most once after the open, and minutes in between are not replayed, so a short gap around the cut is expected. A due time that fired twice fails the run. - R-4 The once-timer fired once. [report]
firedhas exactly one entry with S-3'sname, itsdue_atequals S-3's, and itscountis 1. - R-5 The source holds nothing live. After [deploy]
log/committed-source_parked.jsonandlog/done.jsonexist: [data]list-object-versionson<data-bucket-from>underinstances/<host>/returns no versions and no delete markers, andparked/<inc>/<host>/holds the retired replica objects with amanifest.json. [deploy] the claim'sobligationsis empty. - R-6 The source really stopped. [deploy]
stopped-source.json(the key [capture] shows asdetail.evidence.source) hassteps.tree.okandsteps.stray.okbothtrueand nostrayfield. [box] on the source box, the journal has no "the instance is not proven stopped yet" line for<host>and no M-C20 line ("... is running a process for <host> that this move cannot account for. ..."), and/data/.moves/.page-outbox/<host>/<move>/stray.jsondoes not exist. The moveprobe worker outlives its app on purpose, so a worker that survived the stop fails the cut's tree step and shows here, then reverts at the cut's deadline. It does not page.
Checks after a rollback (Runs 2, 3 and 4)
R-1 to R-4 and R-6 apply as written, with the source (where the instance is back) in place of the target: <pr-from> in place of <pr-to> in R-1 and R-3. R-5 is replaced:
- R-5b The target holds nothing live. After [deploy]
log/rolling_back-target-cleaned.jsonandlog/rolled_back.jsonexist: [data]list-object-versionson<data-bucket-to>underinstances/<host>/and undermoves/<host>/<move>/returns no versions and no delete markers. - R-7 The target's writes came back (dirty rollbacks only). [report] on the source:
rowshas abgrow withregion: <pr-to>(the target's probe region,homeif the target is the home box), written on the target while it ran there. - R-8 The return lane is parked (dirty rollbacks only). A dirty rollback returns the target's writes through its own lane,
moves/<host>/<rb>/in the source's data bucket, where<rb>is the rollback's id (from [capture]: the last rollback journal'sid). The source queues the lane's park in the host claim when it has probed. The park runs only after its horizon (the return seal's bound) plus the signature age (15 minutes) and the request limit (2 minutes), and the rollback does not close until every obligation of the move is acknowledged. So expect the cut's wait to last at least that long. The obligation exists only while the move is live, so read it from the claim file [capture] saved, never from the bucket afterwards:
``sh jq -s --arg loc "moves/<host>/<rb>" '[.[][] | select(.park.location == $loc)] | first' run-3-claim.jsonl ``
It has kind park-replica, region set to the source box's CARLOS_REGION (absent when the source is the home box), an id, a retirement, and park with location, incarnation, evidence_at and request_limit_ns. Record retirement as <ret>, park.incarnation as <pinc> and park.evidence_at. An empty answer means the capture missed the obligation: the check fails, and the run is repeated on a spare host with the claim capture running. After park.evidence_at plus 15 minutes plus request_limit_ns, and once log/rolled_back.json exists: [data] list-object-versions on <data-bucket-from> under moves/<host>/<rb>/ returns no versions and no delete markers, and parked/<pinc>/<host>/<ret>/manifest.json exists in <data-bucket-from>.
A clean rollback needs two things: the stopped target's digests equal what the landing installed, and src_covered (the source's own replica reached the stopped position). The target box logs both: [box] journalctl -u carlos-edge | grep 'the target of rollback' shows "... stopped at TXID <n> (clean <bool>, src_covered <bool>)".
---
Dev (Runs 1 to 4)
Dev topology (Paul, N4). Dev's second region is applied: uk, eu-west-2, box carlos-dev-uk, data bucket carlos-dev-uk-640012953448 (platform-infrastructure#42). Before Run 1:
- [x] Bootstrap
carlos-dev-uk. Done 2026-10-15: boxdennis-stencilingon v0.27.3, withCARLOS_SWAP=overlapandCARLOS_REGION=uk. - [x] SSM parameters: nothing to copy. No box reads SSM at boot or at runtime. Never copy
/carlos/release-keyto a region whose host role can read/carlos/*. Ineu-west-2thecarlos-dev-ukrole can. - [x]
carlos-dev-ukneeds the dev console'sCARLOS_SECRETS_KEYbefore Run 1 (T2 and config sync). Done 2026-10-15:dennis-stencilinggot the dev key through SSMeu-west-2/carlos/secrets-key. Itssecrets_key_fpissha256:bb067582…f5fe, the same as sophie's. The key is backed up in 1Password as "CARLOSSECRETSKEY dev". - [ ]
CARLOS_REGION_HA=trueoncarlos-dev-ieandcarlos-dev-uk, andCARLOS_AWS_ACCOUNTon the dev console. - [ ] Keep
carlos-dev-ie-bineligible: noCARLOS_REGION_HAin itshost.env, soiehas one eligible box (M-C21). - [ ] Decide Runs 2 and 3's arrangement (see "Runs 2 and 3: the regions"), then create the dev rehearsal hosts (P-5) after
carlos-dev-ukserves and before dev's claim goes on: Run 1's and Run 4's inie, Runs 2 and 3's in their arrangement's<from>, and two spares, at least one of them inie(P-5).
Dev's home region. Dev Ireland is probably the home region (its box has no CARLOS_REGION). Confirm it with the dev console's CONSOLE_HOME_REGION and record it here: ____. If ie is home:
- Every dev run touches home, so every dev run needs P-0's one-placementless-box item: Ruling R2's procedure for
carlos-dev-ie-b, Run 1 included. - Run 1 (
ietouk) moves out of home and needs home fix part 1. - Run 1b (
uktoie) moves Run 1's host back into home. It needs home fix part 2 rolled and dev's designation published. - Run 4 (
ietouk) moves out of home and then rolls back into it. It needs part 1, and part 1 must cover a rollback whose source is home. It stays out of home even once part 2 is rolled (Ruling R6), because dev's console runs on the home box. - Runs 2 and 3 as written (
uktoie) move into home and need home fix part 2. Once part 2 is rolled, they run as written. - Until part 2 is rolled, only out-of-home moves run. Run 1 goes ahead on part 1. Runs 2 and 3 then use the out-of-home arrangement: the host is created in
ie, the move goesietouk, and the failure switch names<pr-uk>. Each rolls back into a home source, as Run 4 does, so this needs part 1 to cover a rollback whose source is home. Part 1 is on main as 359e8ac2. Check in its own tests that it covers this before the first such run. If it does not, Runs 2, 3 and 4 all wait for the fix that does. - Run 4 runs only after Runs 1, 1b, 2 and 3 have passed, Runs 2 and 3 in whichever arrangement they ran. No run is skipped.
- On the home box, moveprobe reports
home, so<pr-ie>ishome. Any [report] check orMOVEPROBE_FAIL_READY_INaimed at that box useshome, neverie.
Record the dev probe regions here: <pr-ie> = ____, <pr-uk> = ____.
If ie is not home, the runs go as written and need no home fix.
Run 1: dev, ie to uk, a commit
Blockers: P-0; home fix part 1 if ie is home; if ie is home, P-0's one-placementless-box item, with Ruling R2's window open for carlos-dev-ie-b; dev topology above; P-1 to P-7 on dev; Paul's go.
- [ ] Host used:
____(created iniebefore the claim). Run 1b moves this host back. - [ ] S-1, S-2 with
--to uk. - [ ] S-3, S-4. [CLI] M-C5 ("uk is ready. Starting the cut. <host> will be offline for about <estimate>."), M-C6 ("Stopped in ie. Starting in uk."), M-C7 ("Moved. <host> is served from uk. It was offline for <seconds> seconds."), exit 0.
- [ ] R-1 to R-6.
- [ ] [deploy]
log/hascutting-flipped,committed,committed-opened,committed-source_parkedanddone. [detail]last.outcomeisdone;detail.timingsholdsoffline_sandcut_budget_s.
Run 1b: dev, uk to ie, a commit into home
Blockers: Run 1 passed; Run 1's quiet window has passed (T12, see "Between runs"); home fix part 2 merged and rolled, boxes first, then consoles; the fleet home designation published on dev (claim activation, #250); P-0's one-placementless-box item, with Ruling R2's window open for carlos-dev-ie-b; P-2's home data bucket item; Paul's go.
This run moves Run 1's host back into home (Ruling R6). It is the one host this runbook uses twice (P-5). It has the same checks as Run 1, with <pr-ie> = home.
Note for the operator (Ruling R7). Where the console shares the home box, as dev's does, the landing runs on the console's own box. If that box dies mid-landing, the console's move pass stops with it until the box or the console is back. The record and the claim are durable, so nothing is lost, but recovery waits for the console's return.
- [ ] Host used: Run 1's,
____. - [ ] Before the request, record the claim's touched regions and quiet windows:
``sh aws s3 cp --profile <profile> s3://<deploy-bucket>/control/hosts/<host>.json - | jq '{regions: .touched.regions, quiet_after}' ``
- [ ] S-1, S-2 with
--to ie. Name the home by its configured name. There is nohomekeyword, because a named region may be calledhome(Ruling R3). If S-2 refuses, the sentence says what is missing: - MV-N510 ("moves into the home region need host claims. this console runs without them"): the console runs withoutCARLOS_HOST_CLAIM; - MV-N511 ("<box> sets CARLOS_REGION, so it cannot take a move into the home region"): a box's edges file namesieas its placement. That box is not the home box. Fix itshost.env, or delete its edges file if the box is gone; - MV-N177 ("the host roster does not allow this move. the source and target must be active boxes in uk and ie"): the roster has no home designation, or the console's home name differs from it; - "the fleet's edges files could not all be read" or MV-N5: Ruling R2's window is not ready (P-0, step 4).
Each of these refusals comes before any claim write.
- [ ] At ready, the staging lane is in the home data bucket: [data]
list-object-versionson<data-bucket-ie>(the home data bucket P-2 recorded) with--prefix "moves/<host>/<move>/"returns versions. - [ ] S-3, S-4. [CLI] M-C5 ("ie is ready. Starting the cut. <host> will be offline for about <estimate>."), M-C6 ("Stopped in uk. Starting in ie."), M-C7 ("Moved. <host> is served from ie. It was offline for <seconds> seconds."), exit 0.
- [ ] R-1 to R-6, with
<pr-to>=home. - [ ] [deploy]
log/hascutting-flipped,committed,committed-opened,committed-source_parkedanddone. [detail]last.outcomeisdone;detail.timingsholdsoffline_sandcut_budget_s. - [ ] [deploy] the claim's
touched.regionsholds at least one spelling of home:ie,@home, or both. Touch adds@homewhen the target box is new totouched, and never removesie. Every region it held before the request is still there. - [ ] [deploy] the claim's
quiet_afternamesuk. - [ ] After the close: [data]
list-object-versionson<data-bucket-ie>underparked/<inc>/<host>/holds the residue of Run 1's stay in home, if there was any. - [ ] If any check fails, repeat Run 1 on a spare host created in
ie, then Run 1b on that host.
Runs 2 and 3: the regions
Runs 2 and 3 use one of two arrangements, chosen before the dev hosts are created (see "Dev's home region"):
- As written: the host is created in
uk, and the move goesuktoie. Needs home fix part 2 ifieis home. Once part 2 is rolled, Runs 2 and 3 run as written, and they wait for Run 1b: the first move into home is Run 1b's plain commit, never a forced rollback. - Out of home (
ieis home and part 2 is not rolled): the host is created inie, and the move goesietouk. Needs part 1 to cover a rollback whose source is home. This stays the fallback while part 2 is not rolled.
Below, <from> and <to> are the arrangement's regions as the CLI names them, and <pr-from>, <pr-to> their probe regions. Record the arrangement used in each run.
Run 2: dev, forced clean rollback
Blockers: P-0; the home fix its arrangement needs (above); if ie is home, P-0's one-placementless-box item, with Ruling R2's window open for carlos-dev-ie-b; Run 1 passed; in the as-written arrangement, Run 1b passed (the first move into home is a plain commit); Paul's go.
The target fails its readiness check in before-write mode, so it writes nothing and the rollback should be clean.
- [ ] Arrangement:
____. Host used:____(created in<from>before the claim). - [ ] Before the request, set the failure switch to the target's probe region:
``sh carlos env set --app moveprobe --account <ops> MOVEPROBE_FAIL_READY_IN=<pr-to> ``
The variable is app-wide. Any moveprobe instance that starts in that region while it is set answers 503 and writes nothing. That is expected.
- [ ] S-1, S-2 with
--to <to>, S-3, S-4 (capture first). [CLI] M-C8 ("<to> could not start <host>: <reason>. Moving it back to <from>."), then M-C9 ("<host> is back in <from> and serving. Nothing was lost."), exit 3. - [ ] From [capture], once the cut has returned and the capture has stopped: the last rollback journal has
target_phasereleasedor later, anddirtyabsent orfalse. Record itsidas<rb>, and the lastdetail.evidence.target. - [ ] Digests: [deploy]
stopped-target.json'sdb_sha256,files_plainandtimers_sha256equallog/rolling_back-target-released.json's fieldsinstalled_db_sha256,files_plainandtimers_sha256. - [ ] [box] on the
<to>box: "clean true, src_covered true". - [ ] R-1 to R-4 and R-6 on
<from>(probe region<pr-from>), R-5b on<to>. - [ ] [deploy]
log/hasrolling_back,rolling_back-target-released,rolling_back-target-cleaned,rolling_back-source-openedandrolled_back, and norolling_back-target-returning. - [ ] After the checks are recorded:
``sh carlos env unset --app moveprobe --account <ops> MOVEPROBE_FAIL_READY_IN ``
A dirty result here, with moveprobe in before-write mode, is a platform finding, not a probe failure. Stop the series and report it.
Run 3: dev, forced dirty rollback
Blockers: P-0; the home fix its arrangement needs; if ie is home, P-0's one-placementless-box item, with Ruling R2's window open for carlos-dev-ie-b; Run 2 passed; Paul's go.
The target writes one row, starts its worker, and then fails readiness, so the rollback is dirty and the target's writes must come back.
- [ ] Arrangement:
____. Host used:____(created in<from>before the claim). - [ ] Before the request:
``sh carlos env set --app moveprobe --account <ops> MOVEPROBE_FAIL_READY_IN=<pr-to>,after-write ``
- [ ] S-1, S-2 with
--to <to>, S-3, S-4, with the capture started with its claim file:capture <host> run-3-detail.jsonl run-3-claim.jsonl. [CLI] M-C8, then M-C9, exit 3. The cut's wait includes the return lane's park horizon (R-8), about 17 minutes or more. - [ ] From [capture]: the last rollback journal has
target_phasereleasedor later anddirtytrue. Record itsidas<rb>before anything else; R-8 needs it. - [ ] Digests: [deploy]
stopped-target.json'sdb_sha256differs fromlog/rolling_back-target-released.json'sinstalled_db_sha256(the target wrote). - [ ] [box] on the
<to>box: "clean false". - [ ] [deploy]
log/hasrolling_back-target-returningandrolling_back-target-returned, thenrolling_back-target-cleanedandrolled_back. - [ ] R-1 to R-4 and R-6 on
<from>(probe region<pr-from>), R-5b on<to>, R-7 (abgrow withregion: <pr-to>), and R-8 on<from>(the return lanemoves/<host>/<rb>/in<data-bucket-from>). - [ ] Unset
MOVEPROBE_FAIL_READY_INonce the checks are recorded.
Run 4: dev, declare-lost
Blockers: P-0, including claim Plan 12's loss-bound admission and Plan 4 Task 10's box half; home fix part 1 if ie is home, covering a rollback whose source is home; if ie is home, P-0's one-placementless-box item, with Ruling R2's window open for carlos-dev-ie-b; Runs 1, 1b, 2 and 3 passed (Runs 2 and 3 in whichever arrangement they ran; Run 4 is never run ahead of them); Paul's go.
The target box is stopped mid-move and declared lost. With only two dev regions, the target is carlos-dev-uk, and uk has no eligible box until the readmit at the end. Stop the box. Do not terminate it. A terminated box cannot be readmitted.
Run 4 stays ie to uk even once part 2 is rolled (Ruling R6). It never stops the home box: dev's console runs there, so a declare-lost of the home box would stop the console too. It does not move into home itself.
- [ ] Host used:
____(created iniebefore the claim). - [ ] Paul's scoped go in the amadan thread, before anything below, to stop
carlos-dev-ukduring this run and take it out of service until the readmit. - [ ] So the landing cannot commit before the box stops, set the failure switch on the target first:
``sh carlos env set --app moveprobe --account <ops> MOVEPROBE_FAIL_READY_IN=<pr-uk> ``
Its probe then fails however long the stop takes. If the target finishes its release before the box stops, the run is an ordinary clean rollback: record it and repeat Run 4 on a spare host.
- [ ] Before the run, record the box's roster entry: [deploy]
roster | jq '.boxes["carlos-dev-uk"]'showsstateactive; note itsenrollmentandquarantine_revision. - [ ] S-1, S-2 with
--to uk, S-3, S-4 (capture first). The capture runs until the move closes, through the declaration below. - [ ] When [capture] shows
statelanding(aftercutting-flipped), stopcarlos-dev-ukthrough the AWS console (dev only). - [ ] The cut's wait ends with M-C10 ("Waiting for carlos-dev-uk in uk to confirm it has stopped <host>. <host> is offline."), exit 4. Until the landing deadline passes, it keeps waiting.
- [ ] Declare it lost, with your own AWS credentials for dev in the shell (the command reads EC2 with them and refuses to run as root):
``sh carlos move resolve --app moveprobe --account <ops> --host <host> --declare-lost carlos-dev-uk ``
[CLI] M-C18 ("This declares carlos-dev-uk lost. Anything <host> wrote on carlos-dev-uk after <time> is lost. Type the host name to confirm."). Type <host>. The command then waits for the outcome: expect M-C9 and exit 3.
- [ ] Expected outcome: rolled back, with the loss on the record, not necessarily clean. From [capture]: the last rollback journal has
loss_refandlost_atset; record them and itsid. [deploy]log/rolling_back-target-lost.json(eventdeclared_lost) andlog/rolled_back.jsonexist. [deploy] the decoded roster:boxes.carlos-dev-uk.stateisquarantined,quarantine_revisionis one more than before, andloss_idsholds the loss. - [ ] R-1 to R-4 and R-6 on
ie, with probe region<pr-ie>(homeifieis home). - [ ] Unset
MOVEPROBE_FAIL_READY_IN. - [ ] Start
carlos-dev-ukagain. It stays quarantined: the deployment's admission gate refuses it for every host until the readmit. Watch [box] its journal from boot, and record anything it does for<host>before the readmit. Doing anything for it there is a finding. - [ ] [deploy] the box retires its local state first and records it:
control/host-retirement/carlos-dev-uk/<enrollment>/<quarantine_revision>.jsonexists with"complete": trueand this boot'sboot_id. Readmit refuses without it. - [ ] Readmit it.
carlos ops hostexists only withCARLOS_HOST_CLAIMset in your shell:
``sh CARLOS_HOST_CLAIM=1 carlos ops host resolve <host> --readmit carlos-dev-uk --console <console> ``
Answer readmit at its prompt. The box must be running and reporting: readmit refuses a box with a termination fence, and one whose private report is older than 15 minutes. Before it, check [deploy] control/box-fences/<enrollment>.json does not exist (a stopped box has none; a terminated one does).
- [ ] [deploy] the decoded roster:
boxes.carlos-dev-uk.stateisactiveagain, with the sameenrollment.
---
Flagship (Run 5)
Run 5: the flagship, Runs 1 to 3 again
Blockers: P-0; P-1 to P-6 on the flagship; P-3, the floor, published (§15 step 6: before any protected production rehearsal); the moves copy record fully approved (no pending row); dev Runs 1 to 4 passed (each run gates the next, spec §15 step 7; claim Plan 8b is on main, so the old clause that let Run 5 go ahead with Run 4 skipped no longer applies); Paul's explicit go in the thread, for each of the three.
- The switch (Paul, N17):
CARLOS_MOVES_PHASE1=onin the environment of the console and of the two boxes in play, and nowhere else. The compiled default stays off. It flips later, in its own commit. - Regions (N3): the flagship's home is Ireland, so Run 5 uses two non-home regions,
ukandus, until the home fix parts land. Run 1's shape goesuktous. Runs 2 and 3 goustouk, withMOVEPROBE_FAIL_READY_IN=<pr-uk>and<pr-uk>,after-write. Record the flagship's probe regions:<pr-uk>=____,<pr-us>=____.
- [ ] P-3 on the flagship, after its own scoped go.
- [ ] P-7 on the flagship: the copy record fully approved, then Paul's scoped go naming the console and the
ukandusboxes, then the switch on those three only. - [ ] Hosts used: Run 1 shape
____(created inuk), Run 2 shape____and Run 3 shape____(created inus), all before the flagship's claim went on. - [ ] Each of the three as on dev, with every check.
- [ ] After the third, and only once all three moves are closed: for each of the three hosts, [detail]
stateisnone(no move in flight) anddetail.claim.kindisserving. If any move is unfinished (stuck, a release past its deadline, relocating, or any exit 4), leave the switch on and resolve it first withcarlos move resolve(--retryonce the cause is fixed, or--declare-lost <box>for a lost target), each with Paul's scoped go in the thread. The switch must stay on until the move is closed: with it off, a box's moves pass skips every Phase 1 record, so an unfinished move would stop being reconciled and stay where it is. - [ ] Then remove
CARLOS_MOVES_PHASE1from the two boxes and the console unless Paul says to keep it, and restart them. Check:carlos move status --app moveprobe --account <ops> --host <host>says "Moves are not enabled on this console.", and neither box'shost.envstill sets it.
---
Customer instances (Runs 6 and 7)
The runbook names no tenant. The real instances, their owners and the agreed windows live in the flagship's deployment repo and in the amadan thread where Paul gives each go.
These runs move a customer's app, not moveprobe. Do not call /ready, /arm-once or /report on a customer host, and do not set MOVEPROBE_FAIL_READY_IN. Those are moveprobe's own, and S-1, S-3, R-1 to R-4 and R-7 do not apply. Set ACCT=<account> and APP=<app> for [detail], [capture] and the log prefix, and use --app <app> --account <account> on every command. <from> and <to> are the regions Paul approves for the run.
Customer steps
- C-0 Switch on for this run only. Run 5's switch was removed after it, and this run's participants differ (the source is the home box). - The moves copy record is fully approved (P-7). - Paul's scoped go in the amadan thread naming the flagship, this run, the console, and this run's two participant boxes: the source box in
<from>(the instance's box now) and the target box in<to>. -CARLOS_MOVES_PHASE1=onin the environment of the console and of those two boxes, and nowhere else; restart them. - Check:carlos move status --app <app> --account <account> --host <host>does not say "Moves are not enabled on this console." [box] each of the two boxes'host.envsets it, and no other box's does. - C-1 Health check, before. The owner names one health check for the instance (a URL and the answer that means it is serving) and approves it for the run. Record it in the deployment repo. Run it and record the answer.
- C-2 Request.
``sh carlos move --app <app> --account <account> --host <host> --to <to> --manual-cut ``
[CLI] M-C1, M-C2, M-C3, then M-C12, exit 4. [detail] at ready, the same checks as S-2: every target check ok, attempts_admitted at least 1, source_report.s1.ok and s2.ok true. Record <move> and <inc>.
- C-3 Capture, then cut. Start [capture] in a second terminal, then, in the window agreed with the owner:
``sh carlos move cut --app <app> --account <account> --host <host> ``
[CLI] M-C5, M-C6, M-C7, exit 0.
- C-4 Health check, after. The owner's check from C-1 gives the serving answer once M-C7 has printed. The owner confirms it.
- C-5 Platform checks. R-5 as written. R-6 as written, except its last two sentences, which are about moveprobe's worker. [deploy]
log/hascutting-flipped,committed,committed-opened,committed-source_parkedanddone.
If the move rolls back (M-C8 then M-C9, exit 3), the instance is back in <from>: run C-4 there, then R-5b and R-6, and stop the series. C-6 still applies, once the rollback has closed.
- C-6 Switch off, once the move is closed. First confirm the move is closed: [detail]
stateisnone(no move in flight) anddetail.claim.kindisserving. If it is not (stuck, a release past its deadline, relocating, or any exit 4), leave the switch on and resolve the move first withcarlos move resolve --app <app> --account <account> --host <host>(--retryonce the cause is fixed, or--declare-lost <box>for a lost target), with Paul's scoped go in the thread, then check again. The switch must stay on until the move is closed: with it off, a box's moves pass skips every Phase 1 record, so an unfinished move would stop being reconciled and stay where it is. Once it is closed, and after the run's checks are recorded: removeCARLOS_MOVES_PHASE1from the console and the two boxes, unless Paul says in the thread to keep it, and restart them. Check:carlos move status --app <app> --account <account> --host <host>says "Moves are not enabled on this console.", and [box] neither box'shost.envsets it.
Run 6: a disposable customer-shaped instance
Blockers: P-0; the home-region parks fix, both parts, merged and rolled (customer instances live in the home region); dev Runs 1 and 1b passed; Run 5 passed; the owner's agreement; Paul's go.
A disposable instance of a customer app with the same shape as moveprobe (a key file, a store, a sealed secret), created for the run with the owner's agreement, before the flagship's claim went on (P-5).
- [ ] Instance used: recorded in the deployment repo.
- [ ] C-0 to C-6.
Run 7: the first production customer instance
Blockers: everything Run 6 needs; Run 6 passed; Paul's approval of the instance; a window agreed with its owner.
The first production customer instance Paul approves, in a window agreed with its owner.
- [ ] The instance was created before the flagship's claim went on, so it is adopted at
instances/<host>(P-5). One created later cannot move. - [ ] C-0 to C-6, with C-3 inside the agreed window. Run 6's switch was removed at its C-6, so C-0 needs its own go. The cut window is the default 120 s.
- [ ] Report
offline_sback to the owner: [detail]last.offline_s, oroffline_sinlog/done.json's fields.
---
After every run
- [ ] Attach to the amadan thread: the
doneorrolled_backlog entry (its timings), the bucket listings, and the host used. For Runs 1 to 5, also the [report] files from before and after. For Runs 6 and 7, which never call/report, attach instead the owner-approved health check's answers from C-1 and C-4, and the owner's confirmation. - [ ] A failed check stops the series. Fix its cause and repeat the run on a fresh host before going on.
---
Sources
Paths are from the repo root. Line numbers are main's (bc25674a, the base of this branch); cmd/moveprobe is this branch's and not on main yet. Spec is docs/superpowers/specs/2026-10-02-move-any-app-design.md; copy is docs/superpowers/plans/2026-10-06-moves-phase1-copy.md.
carlos move (the CLI)
| what the runbook says | where | |---|---| | verbs status, cut, abort, resolve; flags --host, --app, --account, --console | move.go:136-149 | | --to, --manual-cut, --no-wait on a request; --wait on status; --retry, --declare-lost, --no-wait on resolve | move.go:151-163 | | exit codes 0, 1, 2, 3, 4, 5 | move.go:40-45, :53-61 | | "carlos move needs --app" | move.go:66, :230 | | "Moves are not enabled on this console." | move.go:65, :240-243 | | refuses to run as root; declare-lost reads EC2 with your own credentials | move.go:211-217, :80, :900-903 | | the request sends region and cut | move.go:248-253 | | M-C2, M-C3, M-C4 | move.go:703, :707, :717 | | M-C12 at ready with a manual cut, exit 4 | move.go:722-726 | | M-C10, exit 4 | move.go:740-754 | | M-C7 on done, M-C9 on rolled_back, exit 3 | move.go:788-803 | | M-C18 and the typed host name | move.go:860-882 | | "No move is in flight" | move.go:595 | | M-C1, M-C5, M-C6, M-C7, M-C8, M-C9 texts | internal/moves/operatorcopy.go:11-38; copy record rows M-C1 to M-C22, copy:24-45 | | the detail path, /api/cli/apps/<acct>/<app>/instances/<host>/move?detail=1 | move.go:230-232, :351-356; apimode.go:460-461 | | login tokens in ~/.carlos/credentials, keyed by console | auth.go:113-120, :139-148 |
The detail answer
| what | where | |---|---| | detail=1 only for operators, only with the switch on | internal/console/apiinstancemove.go:312-327 | | detail.move is nil once no move is in flight, so the capture loop keeps it | internal/console/apiinstancemove.go:1291-1292, :1416-1423 | | a mutation waits for the outcome unless --no-wait | move.go:186-188 | | flat fields state, target, id, last, protocol, phase | internal/console/apiinstancemove.go:160-170 | | detail.move, claim, target_report, source_report, evidence, log, timings, from, region | internal/console/apiinstancemove.go:1290-1318 | | evidence.source and evidence.target come from the record in flight | internal/console/apiinstancemove.go:1367-1370, :1416-1423 | | last fields id, outcome, at, offline_s | internal/provision/provision.go:1059-1064 | | move.incarnation | internal/provision/provision.go:961 | | rollback.dirty, target_phase, evidence, loss_ref, lost_at | internal/provision/provision.go:1067-1088 | | landing.installed_db_sha256 | internal/provision/provision.go:1052 | | target_report.checks T1 to T12, attempts_admitted, cut_budget_s | internal/moves/readiness.go:18-31, :58-67 | | source_report.s1, s2; S1 refuses a nested instance | internal/moves/readiness.go:102-109; internal/host/movesphase1.go:41, :537 |
The log and the buckets
| what | where | |---|---| | identities and events: requested, cutting-requested/cut_requested, cutting-flipped/flipped, committed-opened/opened, committed-source_parked/source_parked, rolling_back, rolling_back-target-<phase>, rolling_back-target-lost/declared_lost, rolling_back-source-<phase>, stuck-<epoch>, the closes; the state as the event for ready and committed | internal/moves/transition.go:122-253 | | retry-<epoch> | internal/host/movesevents.go:109 | | the fields every entry carries | internal/moves/transition.go:42-71 | | the timing keys | internal/moves/log.go:126-133 | | target phases releasing, released, returning, returned, cleaning, cleaned, lost; source phases up to recovered, opened | internal/provision/provision.go:1148-1172 | | released goes to returning only when dirty, else to cleaning | internal/host/movesrollback.go:566-575 | | report prefix moves/<acct>/<app>/<host>/<op>/, source.json, target.json, stopped-<role>.json, log/<identity>.json | internal/moves/staging.go:39-50 | | the staging lane moves/<host>/<op>/ in the target's data bucket | internal/moves/staging.go:6 | | the record's account is the sqid | internal/console/apiinstances.go:1240 | | the replica under <CARLOS_HIBERNATE_PREFIX>/<host> | internal/activator/activator.go:119, :144-146; internal/host/movespass.go:471 | | parked objects at parked/<inc>/<host>/<retirement>/, a manifest.json, the source versions deleted | internal/park/remote.go:230-232, :473, :497 | | stopped report steps.tree, steps.stray, stray (omitted when empty), digests | internal/moves/evidence.go:11-20, :38-53 | | a clean rollback: src_covered and unchanged digests; the "clean ..., srccovered ..." line; dirty and evidence written at released | internal/host/movesrollback.go:394-398, :521-526 | | the rollback's id | internal/provision/provision.go:1068 | | a dirty rollback's return lane is moves/<host>/<rb>, parked in the source's region, queued when the source has probed, horizon the return seal's bound | internal/moves/parkscope.go:66-72; internal/host/movesrecover.go:658-675; internal/hostclaim/move.go:1093-1108 | | a claim obligation's fields: id, op, kind, region, retirement, and park with location, incarnation, `evidenceat, requestlimitns; the claim's obligations list | internal/hostclaim/claim.go:116-121, :137-147, :166 | | the return lane's obligation region is the source box's CARLOSREGION (omitted when empty) | internal/host/movesrecover.go:675; internal/host/movespass.go:206; internal/hostclaim/claim.go:144 | | the claim at control/hosts/<host>.json is plain JSON, read as is | internal/hostclaim/claim.go:178-182; internal/hostclaim/store.go:38-60 | | the detail answer's claim view carries no obligations | internal/console/apiinstancemove.go:1356-1362 | | a rollback closes only when nothing is owed under its move; a finished park removes its obligation | internal/host/movesrecover.go:805-809; internal/host/movesparks.go:134-142; internal/park/remote.go:568; internal/hostclaim/delete.go:83-98 | | the parked archive is parked/<park.incarnation>/<host>/<retirement>/ with manifest.json | internal/park/remote.go:230-232, :473 | | a park runs only after evidenceat plus the signature age (15 min) and the request limit (2 min) | internal/park/remote.go:21, :516-519; internal/moves/parkscope.go:23` |
The door and the fleet
| what | where | |---|---| | moves into home: refused without host claims (MV-N510), and a box whose edges file names the home as its placement refused (MV-N511), both before any claim read; the home's staging from placementless edges files | 2026-10-15-moves-home-region-part2.md (Task 5); merge commit a89e35ca; at that commit internal/console/apiinstancemove.go:614, :665-666, internal/console/instancesplacement.go:199 | | one placementless box: S1 and T8 count every edges file per placement, fresh or not, before the data-bucket check | internal/host/movesreport.go:113 (loadMoveFleet) | | Ruling R2's procedure, R6 (Run 4 stays out of home, Run 1b) and R7 (a home landing on the console's box) | 2026-10-15-moves-home-region-part2.md, "Rulings needed" | | a region with no staging bucket is refused | internal/console/apiinstancemove.go:580-583; staging from data_bucket, internal/console/instancesplacement.go:366 | | M-C21, more than one box | internal/console/apiinstancemove.go:595-600 | | MV-N5, below the floor | internal/console/apiinstancemove.go:613-616; copy:55 | | a box below the protocol, or not armed for the floor, or with an unjudged carlos.prev, is below the floor | internal/console/boxselect.go:100-108 | | MV-N167 for a holder at instances/<host>/<inc> | internal/console/apiinstancemove.go:143, :647-651; copy:217 | | a created holder is at instances/<host>/<inc>, an adopted one at instances/<host> | internal/hostclaim/create.go:75; internal/hostclaim/adopt.go:20 | | the source must be an active, enrolled roster entry with no termination fence | internal/console/apiinstancemove.go:657-684, :1267-1273 | | eligibility: fresh, enrolled, region-ha-1, the console's secrets fingerprint, not draining | internal/boxchoice/boxchoice.go:21, :109-148 | | region-ha-1 and the identity record only with CARLOS_REGION_HA | internal/host/edgesfile.go:156-164; internal/host/boxrecord.go:25-28, :158-166 | | CARLOS_REGION_HA reads only ParseBool true values, so on is off | internal/host/boxrecord.go:21-28 | | identity records at control/boxes/<box>.json | internal/boxid/record.go:21, :28 | | CARLOS_AWS_ACCOUNT and CONSOLE_HOME_REGION on the console | cmd/console/main.go:318, :321 | | the claim at control/hosts/<host>.json; holder.location, obligations, quiet_after | internal/hostclaim/claim.go:56, :166, :171, :182 | | T12 reads quiet_after; H is 15 min plus 1 min skew | internal/host/movesreport.go:756-762; internal/moves/lease.go:13, :22; spec:2573 | | the console's claim recovery pass adopts with the claim on | internal/console/hostclaimbackfill.go:40-48 | | the roster at control/host-actors.json; boxes, state, enrollment, quarantine_revision, loss_ids; a declaration quarantines and adds one to the revision | internal/hostactors/roster.go:21, :30, :38-40, :72, :265-279 | | the roster is written as a root-signed envelope {payload, sig}, the payload base64 JSON | internal/hostactors/roster.go:249; internal/release/rootenvelope.go:29-44; internal/release/release.go:377-379 | | a termination fence is control/box-fences/<enrollment>.json; no object, no fence | internal/hostactors/fence_authority.go:14-34 | | a quarantined box is refused admission | internal/hostquarantine/quarantine.go:17-25 | | CARLOS_HOST_CLAIM is any non-empty value | cmd/console/main.go:1364-1366; hostclaim_cli.go:22 | | carlos ops host exists only with CARLOS_HOST_CLAIM in the shell; --readmit, --console, the readmit prompt | hostclaim_cli.go:35-39, :61-75, :143 | | readmit refuses a fenced box and a private report older than 15 min | internal/hostresolve/readmit.go:18-40 | | readmit needs the box's complete retirement receipt for this boot at control/host-retirement/<box>/<enrollment>/<revision>.json, then sets the box active | internal/hostresolve/readmit.go:52-66, :134; internal/hostquarantine/quarantine.go:72-82, :91-96 |
The floor, the switch and the boxes
| what | where | |---|---| | the floor key and shape, {"min","min_release"} | internal/moves/floor.go:20, :61-64 | | vX.Y.Z only; anything else fails closed | internal/moves/floor.go:85-99 | | no code writes the floor; only reads | internal/moves/floor.go:66-82 | | a box's floor only rises; lowering it is a manual step per box | internal/host/movesfloor.go:31-36 | | moves_protocol, moves_floor_min_release, prev_release, prev_allowed_for, data_bucket, secrets_key_fp in the edges file at control/edges/<box>.json | internal/host/edgesfile.go:218, :270, :295-304; internal/console/instancesplacement.go:22, :83-86 | | control/moves-console.json and its secrets_key_fp | internal/moves/floor.go:25-33 | | the switch, on/1/true, default off | internal/moves/switch.go:6-23 | | the copy record's pending rows keep the compiled default off; P-7 applies the same rule to the env switch (Paul's ruling) | approved_moves_copy_test.go:811; docs/superpowers/plans/2026-10-06-moves-phase1-copy.md | | with the switch off the CLI says "Moves are not enabled on this console." (the switch-off check in Run 5 and C-6) | move.go:65, :240-243 | | with the switch off a box's moves pass skips every Phase 1 record, so the switch goes off only after the move closes | internal/host/movespass.go:797-805 | | carlos move resolve --retry and --declare-lost <box> | move.go:159-163, :258-262 | | the copy gate keeps the default off while a row is pending | approved_moves_copy_test.go:811 | | the default cut window, 120 s | internal/moves/budget.go:17 | | CARLOS_LAYOUT_DEFAULT=flat | internal/host/layoutleg.go:68-70; system.go:714 | | carlos system status | system.go:75-76; README.md:272 | | carlos status --layout shows each host's actual layout, box by box, and marks a box "(nested by default)"; carlos system layout status shows only the recorded intent | cli.go:3528; status_instances.go:142-163, :224; system.go:709-716 | | the agent is carlos-edge.service, reading /etc/carlos/host.env, with a delegated cgroup | internal/host/units/carlos-edge.service:18-19, :101 | | "exec-child accounting armed"; no manager, no stop evidence | internal/cgroup/cgroup.go:181-184; internal/host/movespass.go:376-392 | | "not proven stopped yet ... retrying until the cut's deadline" | internal/host/movescut.go:422-427 | | M-C20 and the page outbox <DataDir>/.moves/.page-outbox/<host>/<move>/<rule>.json, rule stray | internal/host/movespaging.go:47-50, :87, :546-554, :586-588 | | the tree step's "could not be stopped" | internal/stopevidence/reconstruct_linux.go:185 | | alert sinks CARLOS_ALERT_WEBHOOK, CARLOS_ALERT_SNS_TOPIC_ARN; the console arms them only with the claim and the switch on | internal/alerting/alerting.go:191-209; cmd/console/main.go:1352-1362 | | CARLOS_SECRETS_KEY on the box | internal/host/movespass.go:311-316 | | /data is the data directory | internal/ops/ops.go:27 |
Timers and moveprobe
| what | where | |---|---| | a held timer row is skipped while the instance moves | internal/schedule/runner.go:190-197 | | a periodic schedule's next fire is the next boundary after now (no replay) | internal/schedule/spec.go:119-125 | | moveprobe's region is home with no CARLOS_REGION, else CARLOS_REGION; /report says which in region_from | cmd/moveprobe/main.go:94-100 | | the failure switch matches the probe's own region, so it must name <pr-R> | cmd/moveprobe/main.go:103-106, :137-138 | | a box's edges placement is its CARLOS_REGION, absent when unset (the home box) | internal/host/edgesfile.go:205-211, :758 | | bg and worker rows each numbered from 1 in one sequence | cmd/moveprobe/main.go:543-550, :158, :674 | | /report's fields, /arm-once's answer, the failure switch's modes | cmd/moveprobe/README.md, "Use it", "Make readiness fail on purpose" and "Strings" |
Other CLI commands
| what | where | |---|---| | carlos env set, unset, list with --app and --account | appconfig_cli.go:342-350, :408-417, :462-467; cli.go:2045-2051 | | carlos instances create --host --region --account | instances_cli.go:297-310 |
Not on main, cited by name
| what | where | |---|---| | the cut guard and T13 | branch moves-cut-guard, commits 083a6eaa and 6871d2e3 | | the home parks fix part 2 (moves into home) | 2026-10-15-moves-home-region-part2.md, merged to main as a89e35ca (branch moves-home-part2-plan); #236. Part 1 is on main as 359e8ac2 | | claim Plan 12; moves Plan 5 Tasks 9b and 8, Plan 4 Tasks 9 and 10 | plans only; the landing, rollback and recovery are unarmed until then, internal/host/movespass.go:335-340 | | dev's second region | platform-infrastructure#42 |
Spec
| what | where | |---|---| | the runs and the pass rule | spec §14.4, spec:6263-6276 | | the rollout order, the floor before any protected production rehearsal | spec §15, spec:6278-6294 | | each rehearsal gates the next: Run 4 needs Runs 2 and 3, Run 5 needs Run 4 | spec:6294; controller rulings, fix round 1 and final review | | the flagship's home is Ireland | spec:1032 |