carlos / platform Public

branch:

Clone

git clone https://amadan.net/carlos/platform

Plain git — no account needed to clone.

Download

Download this file

1	# Offline move runbook: moving one instance to another region with `carlos move offline` (v0.28.7 and later)
2
3	This is the operator runbook for moving one instance from one region to another with planned downtime, using the pre-copy (`stage`) that v0.28.7 added. It was rewritten on 2026-10-21 from what the first real moves taught. Its commands are the ones that worked on dev and on the flagship on 2026-10-20 and 2026-10-21, with the instance names replaced by placeholders.
4
5	The file runs nothing and authorizes nothing. Moving a customer's instance needs Paul's go for that instance, and anything that writes to a box as root needs its own approval (AGENTS.md, "Production infrastructure needs scoped approval"). A step that writes says so in a line that starts Go:. Everything else is read-only.
6
7	Sources: harness `RUNLOG.md` Runs 9 and 10 with corrections M2 to M15; the dev rehearsals on v0.28.6 and v0.28.7; the flagship moves of 2026-10-21, without and with `stage` (all in the session's `own/after236/` notes). The history at the end says what happened when.
8
9	## The short version
10
11	1. Check the fleet once (section 2) and the instance and the target box before each move (section 3). The target check is the one that matters most: a stale `/opt/carlos/live/<host>` link on the target makes `flip` fail only after the instance is already offline.
12	2. Run `stage` while the instance serves, and run it again until a round copies almost nothing (section 4).
13	3. Run `move.sh`. It stages once more, records a baseline, then runs `start`, `seal`, `flip`, a check of the target and `open` back to back, judging each step by its exit code. Before `open`, any surprise aborts the move. Expect about 2.5 to 3 minutes offline for an instance with a 1.6 to 3.7 GB replica (section 5).
14	4. Measure the offline time yourself, from just before `start` to the first 2xx through the target. The console's figure read 40 to 75 seconds low on the flagship (section 6).
15	5. Verify on the target with the row fingerprint (section 7), then run `finish` (section 8).
16	6. About an hour after `open`, the target stops the instance briefly to nest it. Check it (section 9).
17	7. Once everything checks out, delete the source region's copy of the replica by exact key, keeping `lock.json` (section 10).
18	8. Before `open`, a problem means `abort`, and nothing is lost. After `open` there is no move back: fix forward on the target (section 11).
19
20	Write every step to the log as it runs, and after any restart, read `carlos move offline status` before anything else (section 1).
21
22	## Contents
23
24	- 0. Before you start
25	- 1. Logging discipline
26	- 2. Fleet pre-checks, once per session
27	- 3. Per-instance pre-checks, including the target box
28	- 4. Staging
29	- 5. The offline doors
30	- 6. Measuring the downtime
31	- 7. Verification
32	- 8. finish
33	- 9. The nesting stop, about an hour after open
34	- 10. Replica cleanup
35	- 11. Rollback
36	- 12. Known gaps
37	- 13. Rehearsing on dev
38	- 14. Telling the customer
39	- Appendix A. Setup and helper scripts
40	- History
41
42	---
43
44	## 0. Before you start
45
46	Placeholders. `shop.example.com` is the instance's host, `<acct>` its account sqid, `<app>` its app. The source is "the IE box" (the home region, record region empty, console name `ie`). The target is "the UK box" (region `uk`). Swap the values in `env.sh` (Appendix A) for another pair of regions.
47
48	What has to be true.
49
50	- Every box in both regions and the console run v0.28.7 or later. `stage` refuses while any box in either region lacks the `offline-stage-1` capability.
51	- Your login is a system operator on the console and a member of the instance's account. The doors check both and answer a byte-identical 404 otherwise.
52	- Paul has given his go for this instance, and the customer has been told the window (section 14).
53	- No other instance in the deployment is staged or moving. There is one offline-move lock per deployment, and a staged move holds it. Stage, move and finish one instance, then start on the next.
54	- Nobody promotes the app from the first `stage` until `finish`. The landing's `release` item compares the build the seal recorded with what the target holds.
55	- `carlos move` refuses to run as root. Run it as yourself on DOCS-T1.
56
57	Getting the fleet onto the release. That is the release recipe in the README ("Releasing the platform") and each release's operator notes, not this file. Read the notes for every release between the fleet's version and the one you roll: [v0.28.5](https://github.com/carlosframework/releases/releases/tag/v0.28.5), [v0.28.6](https://github.com/carlosframework/releases/releases/tag/v0.28.6), [v0.28.7](https://github.com/carlosframework/releases/releases/tag/v0.28.7). The parts that matter for moves:
58
59	- Roll dev first, then flagship. Boxes before their console.
60	- From v0.28.5 on, every box takes the next release through its ordinary `carlos system update`, armed or not. The hand re-arm of the v0.28.5 roll is history.
61	- A guard-armed console box (dev's) upgrades its console with `carlos system claims install-console --upgrade --from <sha256 of the running console>` as root, after the box's router is on the new release (the v0.28.6 notes). The flagship console is unguarded and rolls by ship and promote.
62	- After the roll, check that every box's edges file lists both `offline-move-1` and `offline-stage-1` (2.2).
63
64	Set up the scratch directory, `env.sh` and the helpers in Appendix A before anything else.
65
66	---
67
68	## 1. Logging discipline
69
70	On 2026-10-21 a session restart killed a move script in the middle of a move. Its log wrapper wrote a step's output only when the step finished, so the log showed nothing, and the resumed session believed the move had not started. The instance had been started and sealed, and stayed offline for 12.7 minutes until someone read `status` and aborted it. Nothing was lost, but the outage was avoidable.
71
72	The rules:
73
74	1. Write each step to the log as it runs, not when it ends. `L` in Appendix A pipes output through `tee -a` line by line, and `move.sh` and `stage.sh` print as they go.
75	2. Before each door, log a line `ABOUT TO RUN <door> <host> <UTC time>`. `move.sh` and `stage.sh` do it. If you run a door by hand, write the line first.
76	3. After any restart (your session, the agent, a lost terminal), run `MV status` for every host that might be in play before reading the log or doing anything else. "No move is in flight", "is done" and "is aborted" are safe. Anything else is a live move: decide from `status`, not from the log.
77	4. Write a short note into the log before each block saying what you are about to do and what you expect.
78	5. Never print a secret. The helpers print names, sha256 prefixes and counts only.
79
80	---
81
82	## 2. Fleet pre-checks, once per session
83
84	All read-only. Run them through `L` so they land in the log.
85
86	### 2.1 Versions and the doors
87
88	```bash
89	carlos version; sha256sum "$(command -v carlos)" \| cut -c1-16
90	curl -sS -o /dev/null -D - "$C/healthz" \| grep -i -E '^HTTP\|x-carlos-version'
91	carlos system status --console "$C"
92	MV status
93	```
94
95	Pass: the CLI is v0.28.7 or later; the console answers 200; every box reads `current` on the release; `MV status` answers "No move is in flight for shop.example.com." (or the outcome of an earlier move), not a 404. A 404 means an older console or that you are not an operator and member.
96
97	### 2.2 Edges files: capabilities, buckets, secrets key
98
99	```bash
100	aws --profile "$PROFILE" s3 ls "s3://$DEPB/control/edges/"
101	for k in $(aws --profile "$PROFILE" s3 ls "s3://$DEPB/control/edges/" \| awk '{print $4}'); do
102	echo "== $k"; aws --profile "$PROFILE" s3 cp --quiet "s3://$DEPB/control/edges/$k" - \| jq -c '{region, placement, caps: .capabilities, secrets_key_fp: (.secrets_key_fp // "" \| .[0:20]), data_bucket, data_bucket_region, hibernation_configured, hosts: (.hosts \| length)}'
103	done
104	```
105
106	Pass:
107
108	- one file per live box, and no file for a box that no longer exists (a stale file naming the home placement blocks `seal`; removing it needs Paul);
109	- every box in both regions lists `offline-move-1` and `offline-stage-1`;
110	- the target's `placement` is the region you move to, and it has a `data_bucket` (that is `$TGTDATA`);
111	- `hibernation_configured` true;
112	- the target's `secrets_key_fp` equals the source's. The start door checks only the target's fingerprint against the console. A box with no `CARLOS_SECRETS_KEY` publishes no fingerprint at all; on dev, leo-climbing is such a box, and it is not a blocker unless it is the target.
113
114	### 2.3 The boxes and the console
115
116	```bash
117	src "$(cat "$W/block.box.sh")"
118	tgt "$(cat "$W/block.box.sh")"
119	src 'P=$(systemctl show -p MainPID --value console.service 2>/dev/null); [ -n "$P" ] && [ "$P" != 0 ] && { tr "\0" "\n" < /proc/$P/environ \| grep -E "^CARLOS_(HOST_CLAIM\|MOVES_PHASE1)=" ; echo "console exe sha16 $(sha256sum < /proc/$P/exe \| cut -c1-16)"; } \|\| echo "no console on this box"'
120	```
121
122	Pass on both boxes: `claim lines: 0`, `phase1 lines: 0`, `edge env claim/phase1 vars: 0`; `User=carlos` (any other value is a STOP: the move reads bucket and tenant-reachable inputs, so it never runs where the agent runs as root); `Delegate=yes`; `ActiveState=active`; `stage.path: active enabled`; the release version; the same `secrets key sha8` on both. An armed guard is allowed. The console, if it runs on the source box, shows no `CARLOS_HOST_CLAIM` value and no `CARLOS_MOVES_PHASE1`. Check every other box in the source region the same way: `start` waits for every box in the source region to confirm it has stopped the host.
123
124	### 2.4 The staging lane
125
126	The source box's role must be able to write into the target data bucket's `moves/` prefix. The boxes have no `aws` CLI, so simulate the policy from DOCS-T1:
127
128	```bash
129	P=$(aws --profile "$PROFILE" --region "$SRC_REGION" ec2 describe-instances --instance-ids "$SRC_ID" --query 'Reservations[0].Instances[0].IamInstanceProfile.Arn' --output text)
130	R=$(aws --profile "$PROFILE" iam get-instance-profile --instance-profile-name "${P##*/}" --query 'InstanceProfile.Roles[0].Arn' --output text); echo "role $R"
131	aws --profile "$PROFILE" iam simulate-principal-policy --policy-source-arn "$R" --action-names s3:PutObject s3:GetObject s3:DeleteObject --resource-arns "arn:aws:s3:::$TGTDATA/moves/x/y" --query 'EvaluationResults[].[EvalActionName,EvalDecision]' --output text
132	aws --profile "$PROFILE" iam simulate-principal-policy --policy-source-arn "$R" --action-names s3:ListBucket --resource-arns "arn:aws:s3:::$TGTDATA" --context-entries ContextKeyName=s3:prefix,ContextKeyValues=moves/,ContextKeyType=string --query 'EvaluationResults[].[EvalActionName,EvalDecision]' --output text
133	```
134
135	Pass: `PutObject allowed`, `DeleteObject allowed`, `ListBucket allowed`, and `GetObject implicitDeny`. The deny is the designed grant (Put and Delete on `moves/`, a `moves/`-conditioned List, never `instances/` and never Get). `implicitDeny` on Put, Delete or List means the lane is missing: STOP, it is an infra change.
136
137	### 2.5 Leftovers in the control plane
138
139	```bash
140	aws --profile "$PROFILE" s3api head-object --bucket "$DEPB" --key control/hosts-backfill.json >/dev/null 2>&1 && echo "hosts-backfill PRESENT" \|\| echo "control/hosts-backfill.json absent"
141	carlos status --layout --console "$C" \| grep -E "$H\|WARNING"
142	```
143
144	Pass: `absent`. The layout row for the host is `nested` on the source. A warning that the source box "runs nested instances beside flat, foreign or external ones" is about the box, not this move, and does not block it.
145
146	---
147
148	## 3. Per-instance pre-checks
149
150	### 3.1 The record and the doors
151
152	```bash
153	MV status
154	aws --profile "$PROFILE" s3 cp --quiet "s3://$DEPB/apps/$ACCT/$APP/instances/$H.json" - \| jq -c .
155	aws --profile "$PROFILE" s3api head-object --bucket "$DEPB" --key "control/hosts/$H.json" >/dev/null 2>&1 && echo "control/hosts/$H.json PRESENT" \|\| echo "control/hosts/$H.json absent"
156	aws --profile "$PROFILE" s3 ls "s3://$DEPB/control/certs/" \| grep -c "$H"
157	carlos schedule ls --app "$APP" --account "$ACCT" --console "$C"
158	```
159
160	Pass: no move in flight; the record has no seed, replica or replicas, steering, unit, fleet, pool or open run (the door refuses those classes, so a surprise is a STOP, not a workaround); `control/hosts/<host>.json` absent. For certificates, either an object under `control/certs/` for the host or a wildcard that covers it; confirm TLS through the target before the move with `curl -sS -o /dev/null -w '%{http_code} ssl_verify=%{ssl_verify_result}\n' --resolve $H:443:$TGT_IP https://$H$PROBE_PATH` (any HTTP code is fine, `ssl_verify=0` is the pass). `schedule ls` names the timers that must arrive on the target. A schedule that "fires more often than the idle window" means the instance never sleeps, so plan for the nesting stop in section 9.
161
162	### 3.2 Sizes on the source
163
164	```bash
165	src "set -- $H; $(cat "$W/sizes.box.sh")"
166	aws --profile "$PROFILE" s3 ls --recursive --summarize "s3://$SRCDATA/instances/$H/" \| tail -2
167	```
168
169	Pass: the files tree is under 268435456 bytes (256 MiB; over that the seal refuses safely); no sidecar file. Record the database size and the replica's object count and total size: staging time and the size of the target's copy scale with the replica, not the database. A legacy flat `/data/<host>/` directory holding old key copies needs nothing: the seal passes and the retire parks it with the instance.
170
171	### 3.3 The target box
172
173	This check exists because of the first staged move on the flagship. An earlier attempt had left a root-owned link `/opt/carlos/live/<host>` on the UK box, naming a build from weeks before. Nothing reports it before `flip`, and `flip` fails the landing's `release` item only after `start` has taken the instance offline: `flip` exited 4 with "uk could not take shop.example.com: the target box still holds an older build of this instance", the move was aborted, and the instance was offline for about two minutes for nothing (amadan #339).
174
175	```bash
176	tgt "set -- $H; $(cat "$W/target.box.sh")"
177	src "readlink /opt/carlos/live/$H"
178	curl -sS -m 30 -D - -o /dev/null --resolve "$H:443:$SRC_IP" "https://$H$PROBE_PATH" \| grep -i -E '^HTTP\|x-carlos-version'
179	for p in instances moves; do echo "target $p/$H/ current objects: $(aws --profile "$PROFILE" s3 ls --recursive "s3://$TGTDATA/$p/$H/" \| wc -l)"; done
180	echo "target lock: $(aws --profile "$PROFILE" s3 cp --quiet "s3://$TGTDATA/instances/$H/lock.json" - 2>&1 \| head -c 200)"
181	tgt "df -B1 --output=avail,pcent / \| tail -1"
182	```
183
184	Pass, every item:
185
186	- Live link: `live link: absent` on the target. If it is present and names a different build from the one the source's `readlink` shows (and the `x-carlos-version` header confirms), STOP before staging. Clearing it is a root action on the target box and needs Paul's approval; the recipe is below.
187	- `/data` leftovers: every path `absent`: `/data/instances/<host>`, `/data/.instances/<host>`, `/data/<host>`, `/data/programs/<host>.json`, `/data/<host>.env`, `/data/<host>.config.env`, `/data/<host>.db`.
188	- The route: `registry routes [] timers 0`.
189	- A parked directory: `/data/.offline-moves/<host>` absent, or holding only `<move-id>/target/` from an earlier aborted move. That one does not block: the retry on 2026-10-21 passed with it in place. Anything else under it is a STOP.
190	- The UK bucket prefixes: 0 current objects under `instances/<host>/` and `moves/<host>/` in the target data bucket, and no `lock.json`. Versions and delete markers left by an earlier aborted move are fine.
191	- Disk: at least twice the replica's total size free on the target. It holds the staged copy and then the live replica.
192	- Capabilities and switches, again for this move: run 2.2 and 2.3 for the source and the target before each instance, not only once per session. Both boxes list `offline-move-1` and `offline-stage-1`, and neither has a claim or Phase 1 switch (`claim lines: 0`, `phase1 lines: 0`, `edge env claim/phase1 vars: 0`).
193
194	Go (Paul, root on the target box): clearing a stale live link. Only when the link names a build the instance does not run. Confirm that no other link names the same version directory and no process runs from it, then remove the link and that version directory by exact path, never a glob:
195
196	```bash
197	tgt "ls -la /opt/carlos/live/ \| grep -F '/opt/carlos/versions/<old-version>/' ; grep -l '/opt/carlos/versions/<old-version>/' /proc/[0-9]*/maps 2>/dev/null \| head -3; du -sh /opt/carlos/versions/<old-version>"
198	tgt "rm -- /opt/carlos/live/$H && rm -r -- /opt/carlos/versions/<old-version> && echo removed"
199	```
200
201	Then run 3.3 again.
202
203	### 3.4 Baseline and the row fingerprint
204
205	```bash
206	src "set -- $H; $(cat "$W/facts.box.sh")"
207	tgt "set -- $H; $(cat "$W/facts.box.sh")"
208	echo "source lock: $(aws --profile "$PROFILE" s3 cp --quiet "s3://$SRCDATA/instances/$H/lock.json" - \| jq -c .)"
209	src "set -- $H; $(cat "$W/schema.box.sh")"
210	```
211
212	`facts` on the source prints the env file's sha, the instance key's digest, `config.env`'s sha and names, the files tree's count and digest, the database, the row, the timers and anything parked. On the target it must print `env file: absent`, `row: []` and `timers: []`.
213
214	The row fingerprint compares the database before and after. `schema.box.sh` lists each table with its row count and time columns. Pick two or three tables the app only appends to (an events or audit log, for example), with plenty of rows. A table whose old rows the app updates will show a different digest with nothing wrong, so leave those out. Then read their highest rowids:
215
216	```bash
217	src "set -- $H <table1> <table2>; $(cat "$W/rows.box.sh")"
218	```
219
220	and set `ROWS_ARGS` in `env.sh` to `"<table1>:<max_rowid1> <table2>:<max_rowid2>"`. With a limit, `rows.box.sh` also prints a digest of every row up to that rowid, which must be identical on the target. It never prints row content.
221
222	---
223
224	## 4. Staging
225
226	`stage` copies the instance's replica into the target data bucket's `moves/<host>/` prefix while the instance keeps serving. Each run copies only what changed since the last. On the flagship, the first round copied 1.6 GB in 76 to 79 seconds and 3.7 GB in 142 seconds, with no failed requests through the source while it ran. Later rounds took 15 to 18 seconds.
227
228	Run it until it has caught up.
229
230	```bash
231	L "stage 1" "bash $W/stage.sh"
232	L "stage 2" "bash $W/stage.sh"
233	```
234
235	`stage.sh` logs `ABOUT TO RUN stage`, runs the door (it waits for the round to finish), then prints `status`, the record's `move` and the stager's report. Caught up means `status` says "Staged n of n objects" with the two numbers equal, and the stager report's `round_bytes` is small (a few KB). The first round of a large replica can end at "Staged 237 of 240" because Litestream compacts while it copies; the next round closes the gap.
236
237	Where to read what a round did: the stager report `control/offline-moves/<host>/<move-id>/stager/<box>.json` (`round_objects`, `round_bytes`, `staged_objects`, `replica_objects`) and the source journal line "staged round: N object(s) changed, B bytes copied; n of m objects staged". `status` shows only the totals. `round_objects` counts removals too: a round that copies 2 KB can report 124 objects changed when stale compacted objects are dropped from the staged prefix.
238
239	Run it once more right before `start`. Litestream writes a new object about every 10 seconds while the instance takes writes, and whatever is new at `start` is copied by the seal while the instance is offline. `move.sh` does this final stage itself, about 25 seconds before `start`. On the flagship the seal then copied 0 to 2 objects.
240
241	While a move is staged:
242
243	- the instance serves as normal;
244	- it refuses `carlos run` and deletion;
245	- no other instance in the deployment can stage or move offline;
246	- `abort` removes the staged copy and ends the move with the instance never offline. `abort` and the `finish` after it both exit 3, the normal "aborted" code, with "The staged copy of shop.example.com is removed." and "shop.example.com is back in ie and serving. Nothing was lost." Afterwards `status` prints "Moving shop.example.com from ie to uk. Moving shop.example.com back to ie." and a list of items "waiting". Ignore that; the abort's own lines are the record (#339).
247
248	---
249
250	## 5. The offline doors
251
252	Go (Paul): this instance is offline from `start` until it first answers on the target.
253
254	```bash
255	L "move" "bash $W/move.sh 1200"
256	```
257
258	What `move.sh` does, in order (Appendix A has the script):
259
260	1. Final stage, as in section 4.
261	2. Baseline on the source: `facts`, kept for the gate, and the row fingerprint with `ROWS_ARGS`. If the baseline is incomplete it stops with nothing offline.
262	3. T0, then the doors back to back, each preceded by an `ABOUT TO RUN` line:
263	- `start --to uk` takes the instance offline and waits until every box in the source region confirms it stopped it;
264	- `seal` copies whatever is newer than the last stage and checks the copy;
265	- the source slice's process count (it should be 0);
266	- `flip` stops the instance in the source region for good and lands it on the target;
267	- the gate: `facts` on the target must show the row `landing` in `uk` at `/data/<host>.db` (flat), and the same env file sha and instance key digest, the same `config.env` sha, the same files count and digest and the same timer names as the baseline;
268	- `open` serves it from the target.
269	4. The probe: requests `$PROBE_PATH` through the target every 2 seconds until the first 2xx, prints the offline seconds since T0, then checks the host through the source box and through DNS, and prints `status`.
270
271	Judge each door by its exit code, never by its text. From `docs/site/cli.md`: `0` done, `3` rolled back or aborted, `4` needs an operator (stuck, waiting for a box, or the landing failed an item), `5` refused, `1` the console could not be reached. Several successful doors print lines that read like trouble:
272
273	- `start` prints "Waiting for <box> in ie to confirm it has stopped shop.example.com." before it succeeds;
274	- `open` starts with "this move is offline-open. run carlos move offline status ... to see what it waits for", then "Waiting for <box> to clean up", and exits 0;
275	- after `open`, `status` may say "Its report is not current" for every box. From `open` on, no door reads those reports.
276
277	The one wait the script retries is "the report from <box> is not current" (exit 5, MV-O178): it tries again every 15 seconds, up to 8 times. It is never a process.
278
279	Before `open`, any other failure aborts. If `seal`, `flip` or the gate fails, `move.sh` logs `STOP`, runs `abort` (expect exit 3) and `finish` (expect exit 3, "Nothing was lost"), prints `status` and stops. If `start` itself fails, it stops and leaves the decision to you: read `status`, and if a move is in flight, abort it the same way. If `open` fails, it stops without aborting; read `status` (section 11).
280
281	What it looks like on the flagship, v0.28.7, staged (UTC seconds, all exit 0):
282
283	\| Step \| 1.6 GB replica, 305 objects \| 3.7 GB replica, 270 objects \|
284	\|---\|---\|---\|
285	\| final stage \| 15.8 \| 18.5 \|
286	\| start \| 23.0 \| 23.3 \|
287	\| seal \| 28.4 (copied 2 objects, 487 bytes) \| 30.6 (copied 0) \|
288	\| flip \| 44.2 \| 57.7 \|
289	\| gate (two SSM reads) \| about 2 \| about 3 \|
290	\| open \| 7.3 \| 15.9 \|
291	\| open returned to first 2xx through the target \| 40.2 \| 57.0 \|
292	\| of which: promoting the staged objects into the live replica and restoring \| 29 \| 52 \|
293	\| offline, client-measured \| 147.6 \| 190.4 \|
294	\| console's `offline_s` \| 99.9 \| 117.2 \|
295
296	So: about 2.5 to 3 minutes offline at 1.6 to 3.7 GB. Without `stage` (v0.28.6), the seal of the same 1.6 GB replica took about 6 minutes on its own, and a 27 MB instance with 288 replica objects was offline 164 seconds. On dev, an instance of under a megabyte is offline 30 to 40 seconds.
297
298	After `open`, expect the target to be slow to answer, and do not abort (abort is refused after `open` anyway). The first request wakes the instance, and the wake first copies every staged object from `moves/<host>/` into `instances/<host>/` in the target bucket (the journal says "promoting 270 staged object(s) from moves/shop.example.com/ (residence move landing)") and then restores the database from it. That is 30 to 50 seconds at 1.6 to 3.7 GB, and it sits inside the offline window: `stage` does not shorten it. Meanwhile the target answers 503 at the 30 second wake bound, and the probe's 10 second requests time out (`000`). For the first 10 to 15 seconds after `open` the target may also answer 404, because it does not route the host until the row is ready. Requests that reach a source-region edge can 404 for up to about 20 seconds after the target first serves, until the target's published host list reaches that edge.
299
300	---
301
302	## 6. Measuring the downtime
303
304	Report the offline time measured from the client: T0 (logged just before `start`) to the first 2xx through the target. `move.sh` prints it as "FIRST 2xx through the target at <time>: offline <n> s since start".
305
306	Do not report the console's figure. `last_move.offline_s` on the record, and the "It was offline for N seconds" that `finish` prints, both end at `open`, so they leave out the first wake. On 2026-10-21 they read 42 to 73 seconds low (99.9 against 147.6, 117.2 against 190.4, 122.5 against 164.1).
307
308	Two more numbers worth recording for each move:
309
310	- What each stage round copied: from the stager report and journal (section 4).
311	- What the seal copied, which no output reports: list the object versions written into the target's `moves/<host>/` during the seal window. Take the window from the `seal` line in the log (seconds are enough):
312
313	```bash
314	bash "$W/sealcount.sh" 2026-10-21T15:34:38 2026-10-21T15:35:06
315	```
316
317	The versions survive the promotion, so this works any time after the move.
318
319	---
320
321	## 7. Verification
322
323	Run these after the probe has had its first 2xx, before `finish`. Every item must pass.
324
325	```bash
326	tgt "set -- $H; $(cat "$W/facts.box.sh")"
327	tgt "set -- $H $ROWS_ARGS; $(cat "$W/rows.box.sh")"
328	tgt "python3 -c \"import sqlite3; c=sqlite3.connect('file:/data/$H.db?mode=ro',uri=True); print('quick_check', c.execute('pragma quick_check').fetchone()[0])\""
329	tgt "journalctl -u carlos-edge --since '-15 min' --no-pager \| grep -F $H \| grep -v 'replica sync' \| grep -i -E 'landed\|opened\|promot\|restor\|layout\|woke\|config\|refused\|extracted' \| tail -15"
330	for ip in "$TGT_IP" "$SRC_IP"; do curl -sS -m 30 -o /dev/null -w "$ip %{http_code} %{time_total}s\n" --resolve "$H:443:$ip" "https://$H$PROBE_PATH"; done
331	curl -sS -m 30 -o /dev/null -w "DNS %{http_code}\n" "https://$H$PROBE_PATH"
332	```
333
334	\| Check \| Pass \|
335	\|---\|---\|
336	\| The row on the target \| `ready`, region `uk`, `/data/<host>.db` (flat until the nesting stop) \|
337	\| Env file and instance key \| the same sha and key digest as the baseline \|
338	\| `config.env` \| the same sha and names; it was written at landing, before the first request (the journal's "config updated" comes before "opened") \|
339	\| Files \| the same file count and digest \|
340	\| Rows \| the per-table digests up to the recorded rowids are identical to the source's just before `start`. Table count, total rows and the counts digest also match if the app has written nothing since it woke; if it has, they grow, and only the digests have to match \|
341	\| Database \| `quick_check ok` \|
342	\| Timers \| the same timer names on the target; a schedule tick in its journal or the app's own tick rows after `open`; `carlos schedule ls` shows the instance on the target box. Ticks due while the instance was offline were refused ("maintenance fence active") and do not run later; a once-timer that came due during the move fires once at the first wake \|
343	\| Serving \| 2xx through the target, through the source box (it backhauls to the target) and through DNS, with the expected `x-carlos-version`. If the app has a sign-in, Paul or the customer's contact tries it \|
344
345	A failed item after `open` is fixed forward on the target (section 11).
346
347	---
348
349	## 8. finish
350
351	```bash
352	L "finish" 'echo "ABOUT TO RUN finish $H $(date -u +%FT%T.%3NZ)"; MV finish; echo "finish exit $?"; MV status'
353	```
354
355	`finish` waits for the source box to clean up, then prints "Moved. shop.example.com is served from uk. It was offline for N seconds." and exits 0. It took 1.5 to 4 minutes on 2026-10-21; the instance serves throughout. If it answers that a report "is not current", run it again after 15 seconds. `status` then says "Move <id> of shop.example.com is done."
356
357	After `finish`:
358
359	```bash
360	src "set -- $H; $(cat "$W/stray.box.sh")"
361	src "set -- $H; $(cat "$W/facts.box.sh")"
362	echo "source lock: $(aws --profile "$PROFILE" s3 cp --quiet "s3://$SRCDATA/instances/$H/lock.json" - \| jq -c .)"
363	echo "target lock: $(aws --profile "$PROFILE" s3 cp --quiet "s3://$TGTDATA/instances/$H/lock.json" - \| jq -c .)"
364	aws --profile "$PROFILE" s3 cp --quiet "s3://$DEPB/apps/$ACCT/$APP/instances/$H.json" - \| jq -c '{region, move, last_move}'
365	echo "target moves/$H/ current: $(aws --profile "$PROFILE" s3 ls --recursive "s3://$TGTDATA/moves/$H/" \| wc -l)"
366	carlos status --layout --console "$C" \| grep -F "$H"
367	```
368
369	Pass:
370
371	- the source's `/data/.instances/<host>/` is empty (no stray env file);
372	- the source has no row and no timers for the host, and its journal says "removed retired instance route" and then "tombstone lifted";
373	- parked on the source, under `/data/.offline-moves/<host>/<move-id>/`: `sealer/`, `source/instance/` (the old nested directory, database included) and, if there was one, `source/<host>/` (the legacy flat directory). The tool never deletes these. Add them to the record for the claim census;
374	- left in place on the source by design: `/data/instances/<host>` and `/data/.instances/<host>` (empty), `/opt/carlos/live/<host>` (root-owned link) and `/data/programs/<host>.json` (root-owned, see section 12). `status` lists each with "left ... in place";
375	- the source `lock.json` reads `owner moved:<move-id>` with `expires_at 1970-01-01T00:00:00Z`. That expired object is the normal end state, not an abandoned move. Before `finish` it carries a 7-day expiry, renewed on each moves pass;
376	- the target lock is the target box's own lease;
377	- the record: `region uk`, `move null`, `last_move` outcome `done`;
378	- 0 current objects under the target's `moves/<host>/` (the promotion moved them; versions and delete markers remain);
379	- the layout row is on the target box, `flat`.
380
381	---
382
383	## 9. The nesting stop, about an hour after open
384
385	After `open` the instance runs flat on the target. The layout pass nests it when it is asleep, or, if it stays awake, stops it once the 1-hour grace is up and nests it. The target journal says so right after `open`: "layout: deferred until asleep -> shop.example.com (awake (stopped for the move in 1h0m0s unless it sleeps first))". An instance with a schedule shorter than its idle window never sleeps, so expect the stop. Tell the customer's contact there may be a stop of a few seconds about an hour after the move.
386
387	About 65 minutes after `open`:
388
389	```bash
390	tgt "journalctl -u carlos-edge --since '-80 min' --no-pager \| grep -F $H \| grep -E 'layout' \| tail -8"
391	carlos status --layout --console "$C" \| grep -E "$H\|WARNING"
392	carlos logs --app "$APP" --account "$ACCT" --console "$C" --since 90m \| grep -E 'instance (stopping\|exited\|woke)' \| tail -5
393	tgt "set -- $H $ROWS_ARGS; $(cat "$W/rows.box.sh")"
394	curl -sS -m 30 -o /dev/null -w "target after nesting: %{http_code}\n" --resolve "$H:443:$TGT_IP" "https://$H$PROBE_PATH"
395	```
396
397	Pass: the journal's "awake past its 1h0m0s grace -> stopping it for the move" and "migrated"; `status --layout` shows the host `nested` with "migrated at <time>: moved with it"; the digests unchanged; a 2xx. Measured stops: about 2 seconds on dev (the wake took 0.9 s), 24 seconds in the harness. The stop for a large flagship instance was not timed.
398
399	Then check the instance's custom domains and aliases, if it has any: amadan #338 found an alias row that kept the flat socket after nesting and answered 502.
400
401	Do not move the next instance until this stop is done, so two instances are never nested at once on the target.
402
403	---
404
405	## 10. Replica cleanup
406
407	A finished move leaves the source region's whole replica in the source data bucket, and the parked local Litestream directory on the source box. For a large instance that is several GB in the source region until someone removes it. Remove both, per instance, once it has passed section 7 and `finish`, and only then.
408
409	Go (Paul): deletes the source region's replica of one moved instance.
410
411	```bash
412	L "cleanup" "bash $W/cleanup.sh /data/.offline-moves/$H/<move-id>/source/instance <open time, e.g. 2026-10-21T15:36:02+00:00>"
413	```
414
415	`cleanup.sh` (Appendix A):
416
417	1. refuses unless the source `lock.json` is the lifted tombstone (`moved:<id>`, expiring in 1970), so a move still in flight is never touched;
418	2. lists the current objects under `instances/<host>/` in the source data bucket and refuses if any of them, other than `lock.json`, was written after the open time you give it. A write after `open` would mean something still writes there;
419	3. removes the parked local replica `.../source/instance/.activator-<host>.db-litestream/ltx` on the source box, by exact path (a few hundred KB);
420	4. deletes every object under `instances/<host>/` by exact key, keeping `lock.json` (the lifted tombstone stays as the record), and logs each key with the version id it removed and its size.
421
422	The data buckets are versioned, and noncurrent versions expire 7 days after they are replaced (platform #336). A delete only adds a delete marker; the bytes stay as noncurrent versions for 7 days, are billed until then, and can be brought back from the logged version id within that window. The parked database, env and `config.env` under `source/instance/` stay too.
423
424	Never run this for an instance whose move was aborted: its source replica is the live one.
425
426	On 2026-10-21 this removed 290 objects (72 MB) for the 27 MB instance, 305 objects (1.58 GB) for the 1.6 GB instance and 270 objects (3.66 GB) for the 3.7 GB instance, about 4 minutes each.
427
428	---
429
430	## 11. Rollback
431
432	Before `open`: abort. At any doubt, a failed item, a target missing `config.env`, or a door that refuses for a real reason. `move.sh` does it on its own for `seal`, `flip` and the gate. By hand:
433
434	```bash
435	echo "ABOUT TO RUN abort $H $(date -u +%FT%T.%3NZ)"; MV abort; echo "abort exit $?"
436	echo "ABOUT TO RUN finish $H $(date -u +%FT%T.%3NZ)"; MV finish; echo "finish exit $?"
437	```
438
439	Both exit 3. `abort` prints "Moving shop.example.com back to ie." then "Every box has put shop.example.com back in ie."; `finish` prints "shop.example.com is back in ie and serving. Nothing was lost." The source's database was never removed, so nothing is lost. It took 25 seconds for the abort and under a second for the finish on the flagship, and the instance answered 2xx on the source 2 seconds later.
440
441	Check after an abort:
442
443	- `facts` and the row fingerprint on the source equal the baseline, and `quick_check ok`;
444	- the source row is `ready` and nested; the source lease is the source box's own, at `generation 2`;
445	- the source journal shows "wake fence lifted" and "tombstone lifted on abort";
446	- the target has no row and no timers, and parked what it had extracted under `/data/.offline-moves/<host>/<move-id>/target/`. Leave it: it does not block a retry;
447	- the target bucket has 0 current objects under `instances/<host>/` and `moves/<host>/`;
448	- the record has no region, `move null`, `last_move` outcome `aborted`;
449	- 2xx through the source box, through the target box (backhaul) and through DNS.
450
451	A retry is a new move, staged from scratch, after section 3 passes again.
452
453	After `open`: there is no move back. `abort` is refused once the move is open (MV-O27), the target holds the only live copy and may already have taken writes. Fix forward on the target. Most problems after `open` (an app that cannot boot there, a certificate, reaching its store from the new region) show up in the first minute of section 7. A move back to the source region would be a new offline move in the other direction. It has never been run, it needs the target region's staging grant on the source data bucket, and the source box would need the section 3.3 checks, including its own stale `/opt/carlos/live/<host>` link, which the move leaves in place. Treat it as a new piece of work with its own approval, not as a rollback.
454
455	---
456
457	## 12. Known gaps
458
459	- amadan #339: nothing checks the target before the instance goes offline. A stale root-owned `/opt/carlos/live/<host>` on the target fails `flip` (exit 4, the `release` item, MV-O177) only after `start`. Section 3.3 is the workaround. The same discussion covers two smaller things: after an abort from staging, `status` reads as if the instance moved and came back; and no output says how many objects the seal copied (section 6 has the count).
460	- amadan #338: an alias row can keep the flat socket after nesting and answer 502. Check an instance's aliases after the nesting stop (section 9).
461	- The source keeps root-owned files. `/data/programs/<host>.json` (root, 0644, in a root-owned directory) cannot be parked by the agent, which runs as `carlos`; `status` says "left /data/programs/<host>.json in place: cannot be renamed: ... permission denied". The root-owned `/opt/carlos/live/<host>` link stays too, by design. Both are listed for the claim census, and both are the kind of leftover that section 3.3 catches if that box is ever a move target.
462	- The console's offline figure ends at `open` and leaves out the first wake (section 6).
463	- One offline-move lock per deployment. A staged or moving instance blocks every other one.
464	- The source region keeps the replica until the cleanup in section 10, and the parked database and directories indefinitely.
465	- A new nested instance can start a few seconds before its `config.env` (seen on dev on v0.28.6: about 12 seconds, then a config-change restart). That is the instance-create path, not the move: the move writes `config.env` at landing, before the first request.
466
467	---
468
469	## 13. Rehearsing on dev
470
471	Rehearse each new release's move path on dev before using it on the flagship. Dev's pair is sophie-knitting (home, `ie`, the console box, guard armed) to dennis-stenciling (`uk`); leo-climbing is a second home-region box and takes part as a source box. The values are in Appendix A.
472
473	The rehearsals of 2026-10-21 used one app, `moverehearsal` in account `bab`, built from wakecheck's bytes, with four settings including store-shaped ones, one secret and a schedule `every-minute --every 1m --path /tick`. Each scenario uses a fresh instance of that app (`moverehearsal-a.dev.carlosframework.com` and so on), created with `carlos instances create` and woken through DNS so that it lands on sophie, its nested home. The probe path is `/`.
474
475	Scenarios worth running:
476
477	1. A move with stage: two stage rounds with writes between them, then `move.sh`. Expect about 40 seconds offline.
478	2. Abort from staging: stage, then `abort` and `finish` (both exit 3); the instance never goes offline and the same process keeps serving.
479	3. Abort after flip, before open: about 30 seconds offline, nothing lost.
480	4. The nesting stop an hour after the first move's `open`.
481
482	Afterwards: `carlos instances delete` each rehearsal host (the route retires on the next pass; parked data stays), then the replica cleanup of section 10 for the moved ones. Record what is left on each box and in each bucket.
483
484	---
485
486	## 14. Telling the customer
487
488	A draft for Paul to edit and send, one per instance, at least a day ahead:
489
490	> We will move `<host>` to our UK region on `<date>` at `<time>` UTC. It will be offline for about 3 minutes, and we have booked 15 minutes for it. Your data moves with it, and if anything looks wrong before we reopen it, we put everything back as it was. Scheduled jobs that fall due while it is offline will not run. About an hour later it may pause again for a few seconds while we finish settling it in.
491
492	Tell the customer's contact when each move starts and when it is done.
493
494	---
495
496	## Appendix A. Setup and helper scripts
497
498	### A.1 Scratch and values
499
500	Make a scratch directory under `$TMPDIR` (never `/tmp` directly) and keep its exact path; it is removed by that path at the end (A.4). Keep the log somewhere that outlives the scratch.
501
502	```bash
503	W=$(mktemp -d "$TMPDIR/offline-move.XXXXXX"); echo "$W"
504	```
505
506	Write `$W/env.sh`. These are the flagship's values for IE to UK; dev's are in the comments.
507
508	```bash
509	cat > "$W/env.sh" <<'EOF'
510	export W=<the scratch path printed above>
511	export LOG=<the move log, e.g. $HOME/.cache/<task>/moves-<date>.md>
512	# Deployment. Dev: C=https://platform.dev.carlosframework.com PROFILE=kass DEPB=carlos-dev-640012953448
513	export C=https://console.carloku.com PROFILE=keymail DEPB=carlos-flagship-271376211898
514	# Source box (flagship IE, annie-welding). Dev: sophie-knitting i-05498eada6f0afd23, eu-west-1
515	export SRC_REGION=eu-west-1 SRC_ID=i-092c0c1eea75723cb
516	# Target box (flagship UK, iris-clockmaking). Dev: dennis-stenciling i-009d593f451cfd562, eu-west-2
517	export TGT_REGION=eu-west-2 TGT_ID=i-03c310ceaf50a9bab
518	# Data buckets: each box's CARLOS_HIBERNATE_BUCKET, also data_bucket in its edges file. Dev UK: carlos-dev-uk-640012953448
519	export SRCDATA=carlos-data-ie-271376211898 TGTDATA=carlos-data-uk-271376211898
520	export TO=uk
521	# The instance
522	export H=shop.example.com APP=<app> ACCT=<account sqid>
523	export PROBE_PATH=/signin # a path that answers 2xx without signing in
524	export ROWS_ARGS="" # "<table>:<max_rowid> ...", from 3.4
525	export SRC_IP=<source public IP> TGT_IP=<target public IP>
526	ts() { date -u +%T.%3NZ; }
527	MV() { carlos move offline "$1" --host "$H" --app "$APP" --account "$ACCT" --console "$C" ${2:-}; }
528	src() { "$W/ssm.sh" "$PROFILE" "$SRC_REGION" "$SRC_ID" "$1"; }
529	tgt() { "$W/ssm.sh" "$PROFILE" "$TGT_REGION" "$TGT_ID" "$1"; }
530	# L <label> <command>: logs the label and command, then the output line by line as it runs, then the exit
531	L() { local label=$1; shift
532	printf '\n#### %s (started %s)\n```\n$ %s\n' "$label" "$(date -u +%FT%T.%3NZ)" "$*" >> "$LOG"
533	bash -c "source $W/env.sh; $*" 2>&1 \| tee -a "$LOG"; local rc=${PIPESTATUS[0]}
534	printf '```\n(ended %s, exit %s)\n' "$(date -u +%FT%T.%3NZ)" "$rc" >> "$LOG"; return $rc; }
535	# note <text>: a timestamped line in the log, e.g. what you are about to run and what you expect
536	note() { printf -- '- %s %s\n' "$(date -u +%FT%T.%3NZ)" "$*" \| tee -a "$LOG"; }
537	EOF
538	source "$W/env.sh"
539	```
540
541	The two addresses, for `--resolve`:
542
543	```bash
544	aws --profile "$PROFILE" --region "$SRC_REGION" ec2 describe-instances --instance-ids "$SRC_ID" --query 'Reservations[0].Instances[0].PublicIpAddress' --output text
545	aws --profile "$PROFILE" --region "$TGT_REGION" ec2 describe-instances --instance-ids "$TGT_ID" --query 'Reservations[0].Instances[0].PublicIpAddress' --output text
546	```
547
548	Put them in `env.sh` and source it again. Do the same whenever you change `H` or `ROWS_ARGS`.
549
550	### A.2 ssm.sh
551
552	Runs one command as root on a box over SSM and prints its stdout, with stderr kept apart and a non-zero exit unless the command succeeded. The command goes into a temporary file and runs with stdin closed. SSM returns at most 24,000 characters.
553
554	```bash
555	cat > "$W/ssm.sh" <<'EOF'
556	#!/bin/bash
557	# usage: ssm.sh <profile> <region> <instance-id> '<shell command>'
558	set -euo pipefail
559	profile=$1 region=$2 inst=$3 cmd=$4
560	b64=$(printf '%s' "$cmd" \| base64 -w0)
561	params=$(python3 -I -c 'import json,sys;print(json.dumps({"commands":["f=$(mktemp) && echo %s \| base64 -d > $f && bash $f </dev/null; rc=$?; rm -f $f; exit $rc" % sys.argv[1]]}))' "$b64")
562	id=$(aws --profile "$profile" --region "$region" ssm send-command --instance-ids "$inst" \
563	--document-name AWS-RunShellScript --parameters "$params" --query Command.CommandId --output text)
564	until s=$(aws --profile "$profile" --region "$region" ssm get-command-invocation --command-id "$id" \
565	--instance-id "$inst" --query Status --output text 2>/dev/null) && [ "$s" != InProgress ] && [ "$s" != Pending ]; do sleep 2; done
566	aws --profile "$profile" --region "$region" ssm get-command-invocation --command-id "$id" --instance-id "$inst" \
567	--query StandardOutputContent --output text
568	aws --profile "$profile" --region "$region" ssm get-command-invocation --command-id "$id" --instance-id "$inst" \
569	--query StandardErrorContent --output text >&2
570	[ "$s" = Success ]
571	EOF
572	chmod 700 "$W/ssm.sh"
573	```
574
575	### A.3 Box scripts
576
577	Each is sent with `src` or `tgt`, taking the host as `$1` where it needs one: `src "set -- $H; $(cat "$W/facts.box.sh")"`. All are read-only.
578
579	```bash
580	# block.box.sh: claim and Phase 1 switches, the edge unit, the stage path, version, guard, secrets key, region, disk
581	cat > "$W/block.box.sh" <<'EOF'
582	echo "== $(hostname)"
583	echo "claim lines: $(grep -c "^CARLOS_HOST_CLAIM=." /etc/carlos/host.env \|\| true)"
584	echo "phase1 lines: $(grep -c "^CARLOS_MOVES_PHASE1=" /etc/carlos/host.env \|\| true)"
585	P=$(systemctl show -p MainPID --value carlos-edge)
586	echo "edge env claim/phase1 vars: $(tr "\0" "\n" < /proc/$P/environ \| grep -cE "^CARLOS_(HOST_CLAIM=.\|MOVES_PHASE1=)" \|\| true)"
587	systemctl show carlos-edge -p User -p Delegate -p ActiveState
588	echo "stage.path: $(systemctl is-active carlos-stage.path) $(systemctl is-enabled carlos-stage.path)"
589	/opt/carlos/carlos version 2>&1 \| tail -1
590	if [ -f /opt/carlos/system/guard/policy.json ]; then echo "guard policy present"; else echo "no guard policy"; fi
591	echo "secrets key lines: $(grep -c "^CARLOS_SECRETS_KEY=" /etc/carlos/host.env \|\| true)"
592	echo "secrets key sha8: $(grep "^CARLOS_SECRETS_KEY=" /etc/carlos/host.env \| cut -d= -f2- \| tr -d "\"\n" \| sha256sum \| cut -c1-8)"
593	echo "region/pools: $(grep -E "^CARLOS_(REGION\|POOLS)=" /etc/carlos/host.env \| tr "\n" " ")"
594	echo "disk avail bytes, use: $(df -B1 --output=avail,pcent / \| tail -1)"
595	EOF
596
597	# target.box.sh <host>: what the target box already holds for the host (3.3)
598	cat > "$W/target.box.sh" <<'EOF'
599	H=$1
600	L=/opt/carlos/live/$H
601	if [ -L "$L" ] \|\| [ -e "$L" ]; then echo "live link: PRESENT -> $(readlink "$L") (dated $(stat -c %y "$L" \| cut -c1-19))"; else echo "live link: absent"; fi
602	for p in /data/instances/$H /data/.instances/$H /data/$H /data/programs/$H.json /data/.offline-moves/$H /data/$H.env /data/$H.config.env /data/$H.db; do
603	if [ -e "$p" ]; then echo "PRESENT $p"; else echo "absent $p"; fi
604	done
605	[ -d /data/.offline-moves/$H ] && find /data/.offline-moves/$H -maxdepth 2 \| sed 's/^/ parked: /'
606	python3 - "$H" <<'PY'
607	import sqlite3,sys
608	h=sys.argv[1]; c=sqlite3.connect('file:/data/registry.db?mode=ro',uri=True)
609	print('registry routes', c.execute('select host,status from routes where host=?',(h,)).fetchall(),
610	'timers', c.execute('select count(*) from timers where host=?',(h,)).fetchone()[0])
611	PY
612	EOF
613
614	# facts.box.sh <host>: env and key digests, config.env, files, db, row, timers, parked (never values)
615	cat > "$W/facts.box.sh" <<'EOF'
616	H=$1
617	E=""; for d in /data/instances/$H /data; do [ -f $d/$H.env ] && { E=$d/$H.env; break; }; done
618	if [ -n "$E" ]; then
619	echo " env file: $E $(sha256sum < $E \| cut -c1-16) key-sha $(grep ^CARLOS_INSTANCE_KEY= $E \| cut -d= -f2- \| tr -d '"\n' \| sha256sum \| cut -c1-16) names: $(cut -d= -f1 $E \| sort \| tr '\n' ' ')"
620	else echo " env file: absent"; fi
621	for f in /data/instances/$H/$H.config.env /data/$H.config.env; do [ -f $f ] && echo " config.env $f ($(stat -c %y $f)) sha $(sha256sum < $f \| cut -c1-16): $(cut -d= -f1 $f \| sort \| tr '\n' ' ')"; done
622	FD=""; for d in /data/instances/$H/$H /data/$H; do [ -d $d ] && { FD=$d; break; }; done
623	if [ -n "$FD" ]; then
624	echo " files: $FD $(cd $FD && find . -type f \| wc -l) files, digest $(cd $FD && find . -type f -print0 \| sort -z \| xargs -0 -r sha256sum \| sha256sum \| cut -c1-16)"
625	else echo " files: none 0 files, digest none"; fi
626	for f in /data/instances/$H/$H.db /data/$H.db; do [ -f $f ] && echo " db: $f $(stat -c %s $f) bytes"; done
627	ls /data/instances/$H 2>/dev/null \| tr '\n' ' ' \| sed 's/^/ instance dir: /'; echo
628	python3 - $H <<'PY'
629	import sqlite3,sys
630	h=sys.argv[1]; c=sqlite3.connect('file:/data/registry.db?mode=ro',uri=True)
631	print(' row:', c.execute('select status,last_error,region,db_path,next_due,replication from routes where host=?',(h,)).fetchall())
632	try:
633	t=c.execute('select * from timers where host=?',(h,)).fetchall()
634	print(' timers:', [(x[1],x[2],x[3],x[4],x[7]) for x in t])
635	print(' timer names:', ' '.join(sorted(x[1] for x in t)))
636	except Exception as e: print(' timers: ?', e)
637	PY
638	ls -d /data/.offline-moves/$H// 2>/dev/null \| sed 's/^/ parked: /'
639	EOF
640
641	# sizes.box.sh <host>: files tree (limit 256 MiB), nested dir, db, local Litestream dir, sidecars, legacy flat dir
642	cat > "$W/sizes.box.sh" <<'EOF'
643	H=$1; D=/data/.instances/$H
644	echo "== $H"
645	echo "files tree bytes: $(du -sb $D/$H 2>/dev/null \| cut -f1)"
646	echo "nested dir: $(du -sh $D \| cut -f1)"
647	ls -l $D/$H.db 2>&1 \| awk '{print "db bytes:", $5}'
648	echo "local ltx: $(du -sh $D/.activator-$H.db-litestream 2>/dev/null \| cut -f1)"
649	ls $D/sidecar /data/$H-sidecar.env 2>&1 \| head -2
650	ls -la /data/$H 2>&1 \| head -6
651	EOF
652
653	# procs.box.sh <host>: processes in the host's cgroup slice (0 before flip)
654	cat > "$W/procs.box.sh" <<'EOF'
655	H=$1
656	f=$(find /sys/fs/cgroup/carlos.slice/carlos-instances.slice -maxdepth 1 -name "carlos-instances-$H.slice" 2>/dev/null \| head -1)
657	echo "slice: ${f:-none}"
658	if [ -n "$f" ]; then echo "processes: $(cat $f/cgroup.procs \| wc -l)"; else echo "processes: 0 (no slice)"; fi
659	d=$(find /sys/fs/cgroup/carlos.slice/carlos-edge.service -maxdepth 1 -name "carlos-exec@$H" 2>/dev/null \| head -1)
660	if [ -n "$d" ]; then echo "exec cgroup $d processes: $(cat $d/cgroup.procs \| wc -l)"; fi
661	EOF
662
663	# stray.box.sh <host>: what the source keeps after finish
664	cat > "$W/stray.box.sh" <<'EOF'
665	H=$1
666	echo "== /data/.instances/$H"; ls -la /data/.instances/$H/ 2>&1
667	echo "== parked"; ls -d /data/.offline-moves/$H// /data/.offline-moves/$H//source/ 2>&1
668	echo "== legacy /data/$H"; ls -la /data/$H 2>&1 \| head -5
669	echo "== programs"; ls -la /data/programs/$H.json 2>&1
670	echo "== live link"; ls -la /opt/carlos/live/$H 2>&1
671	EOF
672
673	# schema.box.sh <host>: tables, row counts and time columns, to choose the fingerprint tables
674	cat > "$W/schema.box.sh" <<'EOF'
675	H=$1
676	DB=""; for f in /data/instances/$H/$H.db /data/$H.db; do [ -f $f ] && { DB=$f; break; }; done
677	echo "db: ${DB:-none}"; [ -z "$DB" ] && exit 0
678	python3 - "$DB" <<'PY'
679	import sqlite3,sys
680	c=sqlite3.connect('file:%s?mode=ro'%sys.argv[1],uri=True)
681	for (n,) in c.execute("select name from sqlite_master where type='table' order by name"):
682	cols=[r[1] for r in c.execute('pragma table_info("%s")'%n)]
683	k=c.execute('select count(*) from "%s"'%n).fetchone()[0]
684	tcols=[x for x in cols if x in ('created_at','updated_at','at','ts','timestamp','occurred_at','sent_at','inserted_at')]
685	print(n, k, 'timecols=%s'%tcols)
686	PY
687	EOF
688
689	# rows.box.sh <host> [table[:max_rowid] ...]: read-only row fingerprint (counts, max rowids, digests; never row content)
690	cat > "$W/rows.box.sh" <<'EOF'
691	H=$1; shift
692	DB=""; for f in /data/instances/$H/$H.db /data/$H.db; do [ -f $f ] && { DB=$f; break; }; done
693	echo "db: ${DB:-none}"; [ -z "$DB" ] && exit 0
694	python3 - "$DB" "$@" <<'PY'
695	import sqlite3,sys,hashlib
696	c=sqlite3.connect('file:%s?mode=ro'%sys.argv[1],uri=True)
697	c.execute('begin')
698	tabs=[n for (n,) in c.execute("select name from sqlite_master where type='table' and name not like 'sqlite_%' and name not like '_litestream%' order by name")]
699	counts={n:c.execute('select count(*) from "%s"'%n).fetchone()[0] for n in tabs}
700	print('tables %d, total rows %d, nonempty %d' % (len(tabs), sum(counts.values()), sum(1 for v in counts.values() if v)))
701	print('counts digest', hashlib.sha256(''.join('%s:%d\n'%(n,counts[n]) for n in tabs).encode()).hexdigest()[:16])
702	for arg in sys.argv[2:]:
703	t,_,lim=arg.partition(':')
704	r=c.execute('select count(*),max(rowid) from "%s"'%t).fetchone()
705	print('%s: count %d max_rowid %s' % (t,r[0],r[1]))
706	if lim:
707	h=hashlib.sha256(); n=0
708	for row in c.execute('select rowid,* from "%s" where rowid<=? order by rowid'%t,(int(lim),)):
709	h.update(repr(row).encode()); n+=1
710	print('%s rowid<=%s: %d rows digest %s' % (t,lim,n,h.hexdigest()[:16]))
711	c.execute('rollback')
712	PY
713	EOF
714	```
715
716	### A.4 Operator scripts
717
718	```bash
719	# stage.sh: one staging round for $H, then status, the record's move and the stager report
720	cat > "$W/stage.sh" <<'EOF'
721	#!/bin/bash
722	source "$(dirname "$0")/env.sh"
723	now() { date -u +%FT%T.%3NZ; }
724	echo "=== before $(now): source instances/$H/ $(aws --profile $PROFILE s3 ls --recursive s3://$SRCDATA/instances/$H/ \| wc -l) objects; target moves/$H/ $(aws --profile $PROFILE s3 ls --recursive s3://$TGTDATA/moves/$H/ \| wc -l) objects"
725	echo "ABOUT TO RUN stage $H $(now)"
726	s=$(ts); out=$(MV stage "--to $TO" 2>&1); rc=$?
727	echo "--- stage $s -> $(ts) exit=$rc"; echo "$out" \| sed 's/^/ /'
728	echo "--- status $(ts)"; MV status \| sed 's/^/ /'
729	M=$(aws --profile $PROFILE s3 cp --quiet s3://$DEPB/apps/$ACCT/$APP/instances/$H.json - \| jq -c '.move')
730	echo "record move: $M"
731	ID=$(echo "$M" \| jq -r '.id // empty')
732	if [ -n "$ID" ]; then
733	for k in $(aws --profile $PROFILE s3 ls --recursive "s3://$DEPB/control/offline-moves/$H/$ID/stager/" \| awk '{print $4}'); do
734	echo "stager report $k"
735	aws --profile $PROFILE s3 cp --quiet "s3://$DEPB/$k" - \| jq -c '{box, phase, at, replica_objects, replica_bytes, staged_objects, staged_bytes, round_objects, round_bytes, stage_error}'
736	done
737	fi
738	echo "END stage.sh rc=$rc $(now)"
739	exit $rc
740	EOF
741
742	# move.sh [probe-seconds]: final stage, baseline, then start, seal, flip, the gate and open back to back.
743	# Before open, a failed seal, flip or gate aborts the move and runs finish. After open it only reports.
744	cat > "$W/move.sh" <<'EOF'
745	#!/bin/bash
746	source "$(dirname "$0")/env.sh"
747	PROBE_FOR=${1:-1200}
748	now() { date -u +%FT%T.%3NZ; }
749	door() { # door <verb> [args]: retries only a report that is not current (MV-O178)
750	local i out rc s
751	for i in 1 2 3 4 5 6 7 8; do
752	echo "ABOUT TO RUN $1 $H $(now) (try $i)"
753	s=$(ts); out=$(MV "$1" "${2:-}" 2>&1); rc=$?
754	echo "--- $1 try $i $s -> $(ts) exit=$rc"; echo "$out" \| sed 's/^/ /'
755	if [ $rc -ne 0 ] && echo "$out" \| grep -q "is not current"; then sleep 15; continue; fi
756	return $rc
757	done; return $rc; }
758	back_out() { # back_out <reason>: abort and finish, both expected to exit 3
759	local s out rc
760	echo "STOP: $1 $(now). Aborting (before open)."
761	echo "ABOUT TO RUN abort $H $(now)"; s=$(ts); out=$(MV abort 2>&1); rc=$?
762	echo "--- abort $s -> $(ts) exit=$rc"; echo "$out" \| sed 's/^/ /'
763	echo "ABOUT TO RUN finish $H $(now)"; s=$(ts); out=$(MV finish 2>&1); rc=$?
764	echo "--- finish $s -> $(ts) exit=$rc"; echo "$out" \| sed 's/^/ /'
765	for i in $(seq 1 20); do c=$(curl -s -m 20 -o /dev/null -w '%{http_code}' --resolve $H:443:$SRC_IP https://$H$PROBE_PATH); echo "source $c $(ts)"; case "$c" in 2*) break;; esac; sleep 3; done
766	MV status; echo "END move.sh STOPPED $(now)"; exit 12; }
767
768	echo "=== final stage right before start $(now)"
769	door stage "--to $TO" \|\| { echo "STOP: final stage exit $? $(now). Nothing is offline."; MV status; exit 10; }
770	echo "=== baseline on the source $(now)"
771	A=$(src "set -- $H; $(cat $W/facts.box.sh)"); echo "$A"
772	ENVSHA=$(echo "$A" \| sed -n -E 's/^ env file: \S+ ([0-9a-f]{16}) key-sha .*/\1/p')
773	KEYSHA=$(echo "$A" \| sed -n -E 's/^ env file: .* key-sha ([0-9a-f]{16}) .*/\1/p')
774	CFGSHA=$(echo "$A" \| sed -n -E 's/^ config.env .* sha ([0-9a-f]{16}): .*/\1/p')
775	FILES=$(echo "$A" \| sed -n -E 's/^ files: \S+ (.*)$/\1/p')
776	TIMERS=$(echo "$A" \| sed -n -E 's/^ timer names: (.*)$/\1/p')
777	echo "expect env $ENVSHA key $KEYSHA cfg ${CFGSHA:-none} files $FILES timers $TIMERS"
778	for v in "$ENVSHA" "$KEYSHA" "$FILES"; do [ -n "$v" ] \|\| { echo "STOP: baseline incomplete. Nothing is offline."; exit 9; }; done
779	echo "=== rows fingerprint on the source, just before start $(now)"
780	src "set -- $H $ROWS_ARGS; $(cat $W/rows.box.sh)"
781
782	T0=$(date +%s.%N); echo "=== T0 $(now)"
783	door start "--to $TO" \|\| { echo "STOP: start exit $? $(now). Read status: if a move is in flight, abort it and run finish."; MV status; exit 10; }
784	door seal \|\| back_out "seal exit $?"
785	echo "--- source slice $(now)"; src "set -- $H; $(cat $W/procs.box.sh)"
786	door flip \|\| back_out "flip exit $?"
787
788	echo "--- target facts (gate before open) $(now)"
789	B=$(tgt "set -- $H; $(cat $W/facts.box.sh)"); echo "$B"
790	fail=""
791	echo "$B" \| grep -q -F "row: [('landing', '', '$TO', '/data/$H.db'" \|\| fail="$fail row"
792	echo "$B" \| grep -q -E "^ env file: /data/$H.env $ENVSHA key-sha $KEYSHA " \|\| fail="$fail env/key"
793	[ "$(echo "$B" \| sed -n -E 's/^ config.env .* sha ([0-9a-f]{16}): .*/\1/p')" = "$CFGSHA" ] \|\| fail="$fail config"
794	[ "$(echo "$B" \| sed -n -E 's/^ files: \S+ (.*)$/\1/p')" = "$FILES" ] \|\| fail="$fail files"
795	[ "$(echo "$B" \| sed -n -E 's/^ timer names: (.*)$/\1/p')" = "$TIMERS" ] \|\| fail="$fail timers"
796	[ -z "$fail" ] \|\| back_out "gate failed:$fail"
797	echo "gate before open: ok $(now)"
798
799	door open \|\| { echo "STOP: open exit $? $(now). Do not abort blindly: read status (runbook section 11)."; MV status; exit 14; }
800	echo "--- probe $PROBE_PATH through the target until a 2xx (up to $PROBE_FOR s) $(now)"
801	prev=""; first=""; end=$(( $(date +%s) + PROBE_FOR ))
802	while [ $(date +%s) -lt $end ]; do
803	c=$(curl -s -m 10 -o /dev/null -w '%{http_code}' --resolve $H:443:$TGT_IP https://$H$PROBE_PATH)
804	[ "$c" != "$prev" ] && { echo "target $c $(ts)"; prev=$c; }
805	case "$c" in 2*) first=$(date +%s.%N); break;; esac; sleep 2; done
806	[ -n "$first" ] \|\| { echo "NO 2xx through the target within $PROBE_FOR s $(now)"; exit 15; }
807	echo "FIRST 2xx through the target at $(ts): offline $(python3 -c "print(round($first-$T0,1))") s since start"
808	curl -sS -m 30 -D - -o /dev/null --resolve $H:443:$TGT_IP https://$H$PROBE_PATH \| grep -i -E "^HTTP\|x-carlos-version"
809	for i in $(seq 1 25); do a=$(curl -s -m 20 -o /dev/null -w '%{http_code}' --resolve $H:443:$SRC_IP https://$H$PROBE_PATH); d=$(curl -s -m 20 -o /dev/null -w '%{http_code}' https://$H$PROBE_PATH); echo "$(ts) via source $a \| via DNS $d"; [ "${a:0:1}" = 2 ] && [ "${d:0:1}" = 2 ] && break; sleep 3; done
810	echo "--- status after open $(now)"; MV status
811	echo "END move.sh OK $(now)"
812	EOF
813
814	# sealcount.sh <seal-start> <seal-end> (UTC, to the second, e.g. 2026-10-21T15:34:38): versions the seal wrote
815	cat > "$W/sealcount.sh" <<'EOF'
816	#!/bin/bash
817	source "$(dirname "$0")/env.sh"
818	aws --profile $PROFILE s3api list-object-versions --bucket $TGTDATA --prefix moves/$H/ --query 'Versions[].[LastModified,Size]' --output text \|
819	awk -v a="$1" -v b="$2" '{t=substr($1,1,19)} t<a {pre++} t>=a && t<=b {n++; s+=$2} t>b {post++} END {print "before the seal (stage rounds):", pre+0; print "during the seal window:", n+0, "objects,", s+0, "bytes"; print "after the seal window:", post+0}'
820	EOF
821
822	# cleanup.sh <parked source/instance dir> <open time, e.g. 2026-10-21T15:36:02+00:00>: section 10
823	cat > "$W/cleanup.sh" <<'EOF'
824	#!/bin/bash
825	source "$(dirname "$0")/env.sh"
826	P=$1 OPENED=$2 PFX=instances/$H/
827	[ -n "$P" ] && [ -n "$OPENED" ] \|\| { echo "usage: cleanup.sh <parked source/instance dir> <open time>"; exit 2; }
828	LOCK=$(aws --profile $PROFILE s3 cp --quiet "s3://$SRCDATA/${PFX}lock.json" -); echo "lock.json: $LOCK"
829	echo "$LOCK" \| jq -e '(.owner\|startswith("moved:")) and .expires_at=="1970-01-01T00:00:00Z"' >/dev/null \|\| { echo "STOP: the tombstone is not lifted, so the move is not finished"; exit 2; }
830	aws --profile $PROFILE s3api list-objects-v2 --bucket $SRCDATA --prefix $PFX --query 'Contents[].[Key,LastModified,Size]' --output text > "$W/cleanup-$H.keys"
831	LATEST=$(grep -v -F "${PFX}lock.json" "$W/cleanup-$H.keys" \| cut -f2 \| sort \| tail -1)
832	echo "current objects: $(wc -l < "$W/cleanup-$H.keys"); latest write excluding lock.json: $LATEST; opened: $OPENED"
833	[[ "$LATEST" < "$OPENED" ]] \|\| { echo "STOP: something wrote to the source replica after open"; exit 2; }
834	L2=$P/.activator-$H.db-litestream/ltx
835	echo "== parked local replica: $L2"
836	src "test -d $L2 && echo \"size: \$(du -sh $L2 \| cut -f1), files: \$(find $L2 -type f \| wc -l)\" && rm -r -- $L2 && echo removed $L2; ls -la $(dirname $L2)"
837	echo "== source replica objects by exact key (key, removed version, size), keeping lock.json"
838	n=0; bytes=0
839	while IFS=$'\t' read -r k lm size; do
840	case "$k" in
841	"${PFX}lock.json") echo "keep $k"; continue;;
842	"$PFX"*) ;;
843	*) echo "SKIP unexpected $k"; continue;;
844	esac
845	v=$(aws --profile $PROFILE s3api head-object --bucket $SRCDATA --key "$k" --query VersionId --output text)
846	aws --profile $PROFILE s3api delete-object --bucket $SRCDATA --key "$k" >/dev/null && echo "deleted ${k#$PFX} version $v size $size" && n=$((n+1)) && bytes=$((bytes+size))
847	done < "$W/cleanup-$H.keys"
848	echo "deleted $n source replica objects of $H ($bytes bytes)"
849	echo "== after"; aws --profile $PROFILE s3 ls --recursive "s3://$SRCDATA/$PFX"
850	aws --profile $PROFILE s3 cp --quiet "s3://$SRCDATA/${PFX}lock.json" - \| jq -c .
851	EOF
852	chmod 700 "$W"/*.sh
853	```
854
855	At the end of the session, check nothing you started is still running, then remove the scratch by its exact path (never a glob): `rm -r -- <the path printed in A.1>`. Keep the log.
856
857	---
858
859	## History
860
861	### v0.28.5 roll, 2026-10-20
862
863	The first release with the offline-move doors. Dev's guard-armed boxes could not take it through their own updater (the v0.28.4 updater read a candidate's stderr as part of its answer), so leo-climbing and sophie-knitting took it by a ruled hand re-arm, and dennis-stenciling by its normal `system update`. Sophie's guarded console could not be replaced by `install-console` (it refused an occupied slot), so it was upgraded by an attended recovery, "E1": both slots moved aside, then `install-console`. All of it was rehearsed first in the harness (Run 9), which also ran the first offline move of a 4 MB instance: 366 seconds offline, most of it waiting for stale box reports. The full roll runbook with its signing session, re-arm and E1 is this file as it stood at commit b1464900 on branch v0285-roll-runbook.
864
865	Run 9 found four defects, all fixed in v0.28.6:
866
867	\| Defect \| What it was \|
868	\|---\|---\|
869	\| 5 \| `install-console` could not replace an installed protected console. Fixed by `install-console --upgrade --from <sha256>` \|
870	\| 6 \| a new nested instance could run without its `config.env` for up to about 70 minutes \|
871	\| 7 \| `flip` and `open` trusted a box report for 2 minutes while boxes rewrote theirs every 5 to 6, so the doors refused with "still runs a process" for most of each cycle. Fixed by a 15-second report refresh during a move and a "not current" refusal \|
872	\| 8 \| the source's scheduler re-created the parked instance's env file after `open` \|
873
874	### v0.28.6 roll and the first moves, 2026-10-20 and 21
875
876	The harness (Run 10) rolled the v0.28.6 candidate by plain `system update` on every box and upgraded the guarded console with `--upgrade`, and moved a small instance in about 20 seconds and a 636 MB instance with a 3.5 GB replica history in 296 seconds. It found that the console's offline figure ends at `open`, that a large replica's first wake on the target takes minutes, and that a finished move leaves the whole replica in the source region.
877
878	v0.28.6 went to dev, then flagship (boxes 08:39 to 08:43Z, console 08:45Z on 2026-10-21).
879
880	- Dev rehearsal on v0.28.6 (08:02 to 09:22Z): a move of a small instance from sophie to dennis, 30.6 seconds offline (the console said 23.5); an abort after flip, 32.8 seconds; the nesting stop at the hour, about 2 seconds; the cleanup. Its corrections: the staging lane's `GetObject implicitDeny` is the designed grant; only the target's secrets key fingerprint matters; `open` prints a refusal-like first line while succeeding; the source cannot park the root-owned `/data/programs/<host>.json`.
881	- The first flagship move, a 27 MB instance without `stage`: seal 82 seconds for 288 replica objects on real S3, 164 seconds offline (the console said 123). Verified by the row fingerprint, nested at the hour.
882	- The aborted move, a 1.6 GB instance without `stage`: the seal alone took about 6 minutes. A session restart killed the move script after the seal; its log wrapper had written nothing, so the move was believed not to have started. Found by `status` 12 minutes later, aborted, back on the source with nothing lost after 12.7 minutes offline. That is where section 1 comes from.
883
884	Paul then approved building the pre-copy and parallel object copies.
885
886	### v0.28.7 and the moves with stage, 2026-10-21
887
888	v0.28.7 added `carlos move offline stage` and eight object copies at once. It went to dev (14:30Z), then flagship (14:41Z).
889
890	- Dev pre-copy rehearsal (14:31 to 14:54Z): a move from staging, 40.6 seconds offline (the console said 22.3), the seal copying the 4 objects written since the last stage; an abort from staging with the instance never offline. Its corrections are sections 4 and 6: stage once more right before `start`; read round counts from the stager report; `stage` does not shorten the first wake on the target; the target can 404 for about 8 seconds after `open`; report client-measured time; abort from staging exits 3; `status` after it reads oddly; one lock per deployment.
891	- The 1.6 GB instance, first attempt with stage: staged 1.58 GB in 79 seconds while serving with no failed requests, seal copied nothing, then `flip` exited 4 on the target's stale `/opt/carlos/live/<host>` link from an earlier attempt. Aborted automatically, about 2 minutes offline, nothing lost. Paul approved clearing the link and its old version directory on the target. Section 3.3 is the check that would have caught it before staging, and amadan #339 the fix.
892	- The 3.7 GB instance, with the new target pre-checks: staged in 142 seconds, 190 seconds offline (the console said 117), 52 of them promoting the staged copy into the live replica on the first wake.
893	- The 1.6 GB instance, retry: 148 seconds offline (the console said 100).
894	- Cleanup: the source region's replicas of all three moved instances deleted by exact key with their version ids logged, `lock.json` kept, the parked Litestream directories removed.
895