amadan

carlos / platform Public

Clone 
git clone https://amadan.net/carlos/platform

Plain git — no account needed to clone.

Download

Download this file

Offline move runbook: moving one instance to another region with carlos move offline (v0.28.7 and later)

This is the operator runbook for moving one instance from one region to another with planned downtime, using the pre-copy (stage) that v0.28.7 added. It was rewritten on 2026-10-21 from what the first real moves taught. Its commands are the ones that worked on dev and on the flagship on 2026-10-20 and 2026-10-21, with the instance names replaced by placeholders.

The file runs nothing and authorizes nothing. Moving a customer's instance needs Paul's go for that instance, and anything that writes to a box as root needs its own approval (AGENTS.md, "Production infrastructure needs scoped approval"). A step that writes says so in a line that starts Go:. Everything else is read-only.

Sources: harness RUNLOG.md Runs 9 and 10 with corrections M2 to M15; the dev rehearsals on v0.28.6 and v0.28.7; the flagship moves of 2026-10-21, without and with stage (all in the session's own/after236/ notes). The history at the end says what happened when.

The short version

  1. Check the fleet once (section 2) and the instance and the target box before each move (section 3). The target check is the one that matters most: a stale /opt/carlos/live/<host> link on the target makes flip fail only after the instance is already offline.
  2. Run stage while the instance serves, and run it again until a round copies almost nothing (section 4).
  3. Run move.sh. It stages once more, records a baseline, then runs start, seal, flip, a check of the target and open back to back, judging each step by its exit code. Before open, any surprise aborts the move. Expect about 2.5 to 3 minutes offline for an instance with a 1.6 to 3.7 GB replica (section 5).
  4. Measure the offline time yourself, from just before start to the first 2xx through the target. The console's figure read 40 to 75 seconds low on the flagship (section 6).
  5. Verify on the target with the row fingerprint (section 7), then run finish (section 8).
  6. About an hour after open, the target stops the instance briefly to nest it. Check it (section 9).
  7. Once everything checks out, delete the source region's copy of the replica by exact key, keeping lock.json (section 10).
  8. Before open, a problem means abort, and nothing is lost. After open there is no move back: fix forward on the target (section 11).

Write every step to the log as it runs, and after any restart, read carlos move offline status before anything else (section 1).

Contents

  • 0. Before you start
  • 1. Logging discipline
  • 2. Fleet pre-checks, once per session
  • 3. Per-instance pre-checks, including the target box
  • 4. Staging
  • 5. The offline doors
  • 6. Measuring the downtime
  • 7. Verification
  • 8. finish
  • 9. The nesting stop, about an hour after open
  • 10. Replica cleanup
  • 11. Rollback
  • 12. Known gaps
  • 13. Rehearsing on dev
  • 14. Telling the customer
  • Appendix A. Setup and helper scripts
  • History

---

0. Before you start

Placeholders. shop.example.com is the instance's host, <acct> its account sqid, <app> its app. The source is "the IE box" (the home region, record region empty, console name ie). The target is "the UK box" (region uk). Swap the values in env.sh (Appendix A) for another pair of regions.

What has to be true.

  • Every box in both regions and the console run v0.28.7 or later. stage refuses while any box in either region lacks the offline-stage-1 capability.
  • Your login is a system operator on the console and a member of the instance's account. The doors check both and answer a byte-identical 404 otherwise.
  • Paul has given his go for this instance, and the customer has been told the window (section 14).
  • No other instance in the deployment is staged or moving. There is one offline-move lock per deployment, and a staged move holds it. Stage, move and finish one instance, then start on the next.
  • Nobody promotes the app from the first stage until finish. The landing's release item compares the build the seal recorded with what the target holds.
  • carlos move refuses to run as root. Run it as yourself on DOCS-T1.

Getting the fleet onto the release. That is the release recipe in the README ("Releasing the platform") and each release's operator notes, not this file. Read the notes for every release between the fleet's version and the one you roll: v0.28.5, v0.28.6, v0.28.7. The parts that matter for moves:

  • Roll dev first, then flagship. Boxes before their console.
  • From v0.28.5 on, every box takes the next release through its ordinary carlos system update, armed or not. The hand re-arm of the v0.28.5 roll is history.
  • A guard-armed console box (dev's) upgrades its console with carlos system claims install-console --upgrade --from <sha256 of the running console> as root, after the box's router is on the new release (the v0.28.6 notes). The flagship console is unguarded and rolls by ship and promote.
  • After the roll, check that every box's edges file lists both offline-move-1 and offline-stage-1 (2.2).

Set up the scratch directory, env.sh and the helpers in Appendix A before anything else.

---

1. Logging discipline

On 2026-10-21 a session restart killed a move script in the middle of a move. Its log wrapper wrote a step's output only when the step finished, so the log showed nothing, and the resumed session believed the move had not started. The instance had been started and sealed, and stayed offline for 12.7 minutes until someone read status and aborted it. Nothing was lost, but the outage was avoidable.

The rules:

  1. Write each step to the log as it runs, not when it ends. L in Appendix A pipes output through tee -a line by line, and move.sh and stage.sh print as they go.
  2. Before each door, log a line ABOUT TO RUN <door> <host> <UTC time>. move.sh and stage.sh do it. If you run a door by hand, write the line first.
  3. After any restart (your session, the agent, a lost terminal), run MV status for every host that might be in play before reading the log or doing anything else. "No move is in flight", "is done" and "is aborted" are safe. Anything else is a live move: decide from status, not from the log.
  4. Write a short note into the log before each block saying what you are about to do and what you expect.
  5. Never print a secret. The helpers print names, sha256 prefixes and counts only.

---

2. Fleet pre-checks, once per session

All read-only. Run them through L so they land in the log.

2.1 Versions and the doors

carlos version; sha256sum "$(command -v carlos)" | cut -c1-16
curl -sS -o /dev/null -D - "$C/healthz" | grep -i -E '^HTTP|x-carlos-version'
carlos system status --console "$C"
MV status

Pass: the CLI is v0.28.7 or later; the console answers 200; every box reads current on the release; MV status answers "No move is in flight for shop.example.com." (or the outcome of an earlier move), not a 404. A 404 means an older console or that you are not an operator and member.

2.2 Edges files: capabilities, buckets, secrets key

aws --profile "$PROFILE" s3 ls "s3://$DEPB/control/edges/"
for k in $(aws --profile "$PROFILE" s3 ls "s3://$DEPB/control/edges/" | awk '{print $4}'); do
  echo "== $k"; aws --profile "$PROFILE" s3 cp --quiet "s3://$DEPB/control/edges/$k" - | jq -c '{region, placement, caps: .capabilities, secrets_key_fp: (.secrets_key_fp // "" | .[0:20]), data_bucket, data_bucket_region, hibernation_configured, hosts: (.hosts | length)}'
done

Pass:

  • one file per live box, and no file for a box that no longer exists (a stale file naming the home placement blocks seal; removing it needs Paul);
  • every box in both regions lists offline-move-1 and offline-stage-1;
  • the target's placement is the region you move to, and it has a data_bucket (that is $TGTDATA);
  • hibernation_configured true;
  • the target's secrets_key_fp equals the source's. The start door checks only the target's fingerprint against the console. A box with no CARLOS_SECRETS_KEY publishes no fingerprint at all; on dev, leo-climbing is such a box, and it is not a blocker unless it is the target.

2.3 The boxes and the console

src "$(cat "$W/block.box.sh")"
tgt "$(cat "$W/block.box.sh")"
src 'P=$(systemctl show -p MainPID --value console.service 2>/dev/null); [ -n "$P" ] && [ "$P" != 0 ] && { tr "\0" "\n" < /proc/$P/environ | grep -E "^CARLOS_(HOST_CLAIM|MOVES_PHASE1)=" ; echo "console exe sha16 $(sha256sum < /proc/$P/exe | cut -c1-16)"; } || echo "no console on this box"'

Pass on both boxes: claim lines: 0, phase1 lines: 0, edge env claim/phase1 vars: 0; User=carlos (any other value is a STOP: the move reads bucket and tenant-reachable inputs, so it never runs where the agent runs as root); Delegate=yes; ActiveState=active; stage.path: active enabled; the release version; the same secrets key sha8 on both. An armed guard is allowed. The console, if it runs on the source box, shows no CARLOS_HOST_CLAIM value and no CARLOS_MOVES_PHASE1. Check every other box in the source region the same way: start waits for every box in the source region to confirm it has stopped the host.

2.4 The staging lane

The source box's role must be able to write into the target data bucket's moves/ prefix. The boxes have no aws CLI, so simulate the policy from DOCS-T1:

P=$(aws --profile "$PROFILE" --region "$SRC_REGION" ec2 describe-instances --instance-ids "$SRC_ID" --query 'Reservations[0].Instances[0].IamInstanceProfile.Arn' --output text)
R=$(aws --profile "$PROFILE" iam get-instance-profile --instance-profile-name "${P##*/}" --query 'InstanceProfile.Roles[0].Arn' --output text); echo "role $R"
aws --profile "$PROFILE" iam simulate-principal-policy --policy-source-arn "$R" --action-names s3:PutObject s3:GetObject s3:DeleteObject --resource-arns "arn:aws:s3:::$TGTDATA/moves/x/y" --query 'EvaluationResults[].[EvalActionName,EvalDecision]' --output text
aws --profile "$PROFILE" iam simulate-principal-policy --policy-source-arn "$R" --action-names s3:ListBucket --resource-arns "arn:aws:s3:::$TGTDATA" --context-entries ContextKeyName=s3:prefix,ContextKeyValues=moves/,ContextKeyType=string --query 'EvaluationResults[].[EvalActionName,EvalDecision]' --output text

Pass: PutObject allowed, DeleteObject allowed, ListBucket allowed, and GetObject implicitDeny. The deny is the designed grant (Put and Delete on moves/*, a moves/-conditioned List, never instances/* and never Get). implicitDeny on Put, Delete or List means the lane is missing: STOP, it is an infra change.

2.5 Leftovers in the control plane

aws --profile "$PROFILE" s3api head-object --bucket "$DEPB" --key control/hosts-backfill.json >/dev/null 2>&1 && echo "hosts-backfill PRESENT" || echo "control/hosts-backfill.json absent"
carlos status --layout --console "$C" | grep -E "$H|WARNING"

Pass: absent. The layout row for the host is nested on the source. A warning that the source box "runs nested instances beside flat, foreign or external ones" is about the box, not this move, and does not block it.

---

3. Per-instance pre-checks

3.1 The record and the doors

MV status
aws --profile "$PROFILE" s3 cp --quiet "s3://$DEPB/apps/$ACCT/$APP/instances/$H.json" - | jq -c .
aws --profile "$PROFILE" s3api head-object --bucket "$DEPB" --key "control/hosts/$H.json" >/dev/null 2>&1 && echo "control/hosts/$H.json PRESENT" || echo "control/hosts/$H.json absent"
aws --profile "$PROFILE" s3 ls "s3://$DEPB/control/certs/" | grep -c "$H"
carlos schedule ls --app "$APP" --account "$ACCT" --console "$C"

Pass: no move in flight; the record has no seed, replica or replicas, steering, unit, fleet, pool or open run (the door refuses those classes, so a surprise is a STOP, not a workaround); control/hosts/<host>.json absent. For certificates, either an object under control/certs/ for the host or a wildcard that covers it; confirm TLS through the target before the move with curl -sS -o /dev/null -w '%{http_code} ssl_verify=%{ssl_verify_result}\n' --resolve $H:443:$TGT_IP https://$H$PROBE_PATH (any HTTP code is fine, ssl_verify=0 is the pass). schedule ls names the timers that must arrive on the target. A schedule that "fires more often than the idle window" means the instance never sleeps, so plan for the nesting stop in section 9.

3.2 Sizes on the source

src "set -- $H; $(cat "$W/sizes.box.sh")"
aws --profile "$PROFILE" s3 ls --recursive --summarize "s3://$SRCDATA/instances/$H/" | tail -2

Pass: the files tree is under 268435456 bytes (256 MiB; over that the seal refuses safely); no sidecar file. Record the database size and the replica's object count and total size: staging time and the size of the target's copy scale with the replica, not the database. A legacy flat /data/<host>/ directory holding old key copies needs nothing: the seal passes and the retire parks it with the instance.

3.3 The target box

This check exists because of the first staged move on the flagship. An earlier attempt had left a root-owned link /opt/carlos/live/<host> on the UK box, naming a build from weeks before. Nothing reports it before flip, and flip fails the landing's release item only after start has taken the instance offline: flip exited 4 with "uk could not take shop.example.com: the target box still holds an older build of this instance", the move was aborted, and the instance was offline for about two minutes for nothing (amadan #339).

tgt "set -- $H; $(cat "$W/target.box.sh")"
src "readlink /opt/carlos/live/$H"
curl -sS -m 30 -D - -o /dev/null --resolve "$H:443:$SRC_IP" "https://$H$PROBE_PATH" | grep -i -E '^HTTP|x-carlos-version'
for p in instances moves; do echo "target $p/$H/ current objects: $(aws --profile "$PROFILE" s3 ls --recursive "s3://$TGTDATA/$p/$H/" | wc -l)"; done
echo "target lock: $(aws --profile "$PROFILE" s3 cp --quiet "s3://$TGTDATA/instances/$H/lock.json" - 2>&1 | head -c 200)"
tgt "df -B1 --output=avail,pcent / | tail -1"

Pass, every item:

  • Live link: live link: absent on the target. If it is present and names a different build from the one the source's readlink shows (and the x-carlos-version header confirms), STOP before staging. Clearing it is a root action on the target box and needs Paul's approval; the recipe is below.
  • /data leftovers: every path absent: /data/instances/<host>, /data/.instances/<host>, /data/<host>, /data/programs/<host>.json, /data/<host>.env, /data/<host>.config.env, /data/<host>.db.
  • The route: registry routes [] timers 0.
  • A parked directory: /data/.offline-moves/<host> absent, or holding only <move-id>/target/ from an earlier aborted move. That one does not block: the retry on 2026-10-21 passed with it in place. Anything else under it is a STOP.
  • The UK bucket prefixes: 0 current objects under instances/<host>/ and moves/<host>/ in the target data bucket, and no lock.json. Versions and delete markers left by an earlier aborted move are fine.
  • Disk: at least twice the replica's total size free on the target. It holds the staged copy and then the live replica.
  • Capabilities and switches, again for this move: run 2.2 and 2.3 for the source and the target before each instance, not only once per session. Both boxes list offline-move-1 and offline-stage-1, and neither has a claim or Phase 1 switch (claim lines: 0, phase1 lines: 0, edge env claim/phase1 vars: 0).

Go (Paul, root on the target box): clearing a stale live link. Only when the link names a build the instance does not run. Confirm that no other link names the same version directory and no process runs from it, then remove the link and that version directory by exact path, never a glob:

tgt "ls -la /opt/carlos/live/ | grep -F '/opt/carlos/versions/<old-version>/' ; grep -l '/opt/carlos/versions/<old-version>/' /proc/[0-9]*/maps 2>/dev/null | head -3; du -sh /opt/carlos/versions/<old-version>"
tgt "rm -- /opt/carlos/live/$H && rm -r -- /opt/carlos/versions/<old-version> && echo removed"

Then run 3.3 again.

3.4 Baseline and the row fingerprint

src "set -- $H; $(cat "$W/facts.box.sh")"
tgt "set -- $H; $(cat "$W/facts.box.sh")"
echo "source lock: $(aws --profile "$PROFILE" s3 cp --quiet "s3://$SRCDATA/instances/$H/lock.json" - | jq -c .)"
src "set -- $H; $(cat "$W/schema.box.sh")"

facts on the source prints the env file's sha, the instance key's digest, config.env's sha and names, the files tree's count and digest, the database, the row, the timers and anything parked. On the target it must print env file: absent, row: [] and timers: [].

The row fingerprint compares the database before and after. schema.box.sh lists each table with its row count and time columns. Pick two or three tables the app only appends to (an events or audit log, for example), with plenty of rows. A table whose old rows the app updates will show a different digest with nothing wrong, so leave those out. Then read their highest rowids:

src "set -- $H <table1> <table2>; $(cat "$W/rows.box.sh")"

and set ROWS_ARGS in env.sh to "<table1>:<max_rowid1> <table2>:<max_rowid2>". With a limit, rows.box.sh also prints a digest of every row up to that rowid, which must be identical on the target. It never prints row content.

---

4. Staging

stage copies the instance's replica into the target data bucket's moves/<host>/ prefix while the instance keeps serving. Each run copies only what changed since the last. On the flagship, the first round copied 1.6 GB in 76 to 79 seconds and 3.7 GB in 142 seconds, with no failed requests through the source while it ran. Later rounds took 15 to 18 seconds.

Run it until it has caught up.

L "stage 1" "bash $W/stage.sh"
L "stage 2" "bash $W/stage.sh"

stage.sh logs ABOUT TO RUN stage, runs the door (it waits for the round to finish), then prints status, the record's move and the stager's report. Caught up means status says "Staged n of n objects" with the two numbers equal, and the stager report's round_bytes is small (a few KB). The first round of a large replica can end at "Staged 237 of 240" because Litestream compacts while it copies; the next round closes the gap.

Where to read what a round did: the stager report control/offline-moves/<host>/<move-id>/stager/<box>.json (round_objects, round_bytes, staged_objects, replica_objects) and the source journal line "staged round: N object(s) changed, B bytes copied; n of m objects staged". status shows only the totals. round_objects counts removals too: a round that copies 2 KB can report 124 objects changed when stale compacted objects are dropped from the staged prefix.

Run it once more right before start. Litestream writes a new object about every 10 seconds while the instance takes writes, and whatever is new at start is copied by the seal while the instance is offline. move.sh does this final stage itself, about 25 seconds before start. On the flagship the seal then copied 0 to 2 objects.

While a move is staged:

  • the instance serves as normal;
  • it refuses carlos run and deletion;
  • no other instance in the deployment can stage or move offline;
  • abort removes the staged copy and ends the move with the instance never offline. abort and the finish after it both exit 3, the normal "aborted" code, with "The staged copy of shop.example.com is removed." and "shop.example.com is back in ie and serving. Nothing was lost." Afterwards status prints "Moving shop.example.com from ie to uk. Moving shop.example.com back to ie." and a list of items "waiting". Ignore that; the abort's own lines are the record (#339).

---

5. The offline doors

Go (Paul): this instance is offline from start until it first answers on the target.

L "move" "bash $W/move.sh 1200"

What move.sh does, in order (Appendix A has the script):

  1. Final stage, as in section 4.
  2. Baseline on the source: facts, kept for the gate, and the row fingerprint with ROWS_ARGS. If the baseline is incomplete it stops with nothing offline.
  3. T0, then the doors back to back, each preceded by an ABOUT TO RUN line: - start --to uk takes the instance offline and waits until every box in the source region confirms it stopped it; - seal copies whatever is newer than the last stage and checks the copy; - the source slice's process count (it should be 0); - flip stops the instance in the source region for good and lands it on the target; - the gate: facts on the target must show the row landing in uk at /data/<host>.db (flat), and the same env file sha and instance key digest, the same config.env sha, the same files count and digest and the same timer names as the baseline; - open serves it from the target.
  4. The probe: requests $PROBE_PATH through the target every 2 seconds until the first 2xx, prints the offline seconds since T0, then checks the host through the source box and through DNS, and prints status.

Judge each door by its exit code, never by its text. From docs/site/cli.md: 0 done, 3 rolled back or aborted, 4 needs an operator (stuck, waiting for a box, or the landing failed an item), 5 refused, 1 the console could not be reached. Several successful doors print lines that read like trouble:

  • start prints "Waiting for <box> in ie to confirm it has stopped shop.example.com." before it succeeds;
  • open starts with "this move is offline-open. run carlos move offline status ... to see what it waits for", then "Waiting for <box> to clean up", and exits 0;
  • after open, status may say "Its report is not current" for every box. From open on, no door reads those reports.

The one wait the script retries is "the report from <box> is not current" (exit 5, MV-O178): it tries again every 15 seconds, up to 8 times. It is never a process.

Before open, any other failure aborts. If seal, flip or the gate fails, move.sh logs STOP, runs abort (expect exit 3) and finish (expect exit 3, "Nothing was lost"), prints status and stops. If start itself fails, it stops and leaves the decision to you: read status, and if a move is in flight, abort it the same way. If open fails, it stops without aborting; read status (section 11).

What it looks like on the flagship, v0.28.7, staged (UTC seconds, all exit 0):

| Step | 1.6 GB replica, 305 objects | 3.7 GB replica, 270 objects | |---|---|---| | final stage | 15.8 | 18.5 | | start | 23.0 | 23.3 | | seal | 28.4 (copied 2 objects, 487 bytes) | 30.6 (copied 0) | | flip | 44.2 | 57.7 | | gate (two SSM reads) | about 2 | about 3 | | open | 7.3 | 15.9 | | open returned to first 2xx through the target | 40.2 | 57.0 | | of which: promoting the staged objects into the live replica and restoring | 29 | 52 | | offline, client-measured | 147.6 | 190.4 | | console's offline_s | 99.9 | 117.2 |

So: about 2.5 to 3 minutes offline at 1.6 to 3.7 GB. Without stage (v0.28.6), the seal of the same 1.6 GB replica took about 6 minutes on its own, and a 27 MB instance with 288 replica objects was offline 164 seconds. On dev, an instance of under a megabyte is offline 30 to 40 seconds.

After open, expect the target to be slow to answer, and do not abort (abort is refused after open anyway). The first request wakes the instance, and the wake first copies every staged object from moves/<host>/ into instances/<host>/ in the target bucket (the journal says "promoting 270 staged object(s) from moves/shop.example.com/ (residence move landing)") and then restores the database from it. That is 30 to 50 seconds at 1.6 to 3.7 GB, and it sits inside the offline window: stage does not shorten it. Meanwhile the target answers 503 at the 30 second wake bound, and the probe's 10 second requests time out (000). For the first 10 to 15 seconds after open the target may also answer 404, because it does not route the host until the row is ready. Requests that reach a source-region edge can 404 for up to about 20 seconds after the target first serves, until the target's published host list reaches that edge.

---

6. Measuring the downtime

Report the offline time measured from the client: T0 (logged just before start) to the first 2xx through the target. move.sh prints it as "FIRST 2xx through the target at <time>: offline <n> s since start".

Do not report the console's figure. last_move.offline_s on the record, and the "It was offline for N seconds" that finish prints, both end at open, so they leave out the first wake. On 2026-10-21 they read 42 to 73 seconds low (99.9 against 147.6, 117.2 against 190.4, 122.5 against 164.1).

Two more numbers worth recording for each move:

  • What each stage round copied: from the stager report and journal (section 4).
  • What the seal copied, which no output reports: list the object versions written into the target's moves/<host>/ during the seal window. Take the window from the seal line in the log (seconds are enough):

``bash bash "$W/sealcount.sh" 2026-10-21T15:34:38 2026-10-21T15:35:06 ``

The versions survive the promotion, so this works any time after the move.

---

7. Verification

Run these after the probe has had its first 2xx, before finish. Every item must pass.

tgt "set -- $H; $(cat "$W/facts.box.sh")"
tgt "set -- $H $ROWS_ARGS; $(cat "$W/rows.box.sh")"
tgt "python3 -c \"import sqlite3; c=sqlite3.connect('file:/data/$H.db?mode=ro',uri=True); print('quick_check', c.execute('pragma quick_check').fetchone()[0])\""
tgt "journalctl -u carlos-edge --since '-15 min' --no-pager | grep -F $H | grep -v 'replica sync' | grep -i -E 'landed|opened|promot|restor|layout|woke|config|refused|extracted' | tail -15"
for ip in "$TGT_IP" "$SRC_IP"; do curl -sS -m 30 -o /dev/null -w "$ip %{http_code} %{time_total}s\n" --resolve "$H:443:$ip" "https://$H$PROBE_PATH"; done
curl -sS -m 30 -o /dev/null -w "DNS %{http_code}\n" "https://$H$PROBE_PATH"

| Check | Pass | |---|---| | The row on the target | ready, region uk, /data/<host>.db (flat until the nesting stop) | | Env file and instance key | the same sha and key digest as the baseline | | config.env | the same sha and names; it was written at landing, before the first request (the journal's "config updated" comes before "opened") | | Files | the same file count and digest | | Rows | the per-table digests up to the recorded rowids are identical to the source's just before start. Table count, total rows and the counts digest also match if the app has written nothing since it woke; if it has, they grow, and only the digests have to match | | Database | quick_check ok | | Timers | the same timer names on the target; a schedule tick in its journal or the app's own tick rows after open; carlos schedule ls shows the instance on the target box. Ticks due while the instance was offline were refused ("maintenance fence active") and do not run later; a once-timer that came due during the move fires once at the first wake | | Serving | 2xx through the target, through the source box (it backhauls to the target) and through DNS, with the expected x-carlos-version. If the app has a sign-in, Paul or the customer's contact tries it |

A failed item after open is fixed forward on the target (section 11).

---

8. finish

L "finish" 'echo "ABOUT TO RUN finish $H $(date -u +%FT%T.%3NZ)"; MV finish; echo "finish exit $?"; MV status'

finish waits for the source box to clean up, then prints "Moved. shop.example.com is served from uk. It was offline for N seconds." and exits 0. It took 1.5 to 4 minutes on 2026-10-21; the instance serves throughout. If it answers that a report "is not current", run it again after 15 seconds. status then says "Move <id> of shop.example.com is done."

After finish:

src "set -- $H; $(cat "$W/stray.box.sh")"
src "set -- $H; $(cat "$W/facts.box.sh")"
echo "source lock: $(aws --profile "$PROFILE" s3 cp --quiet "s3://$SRCDATA/instances/$H/lock.json" - | jq -c .)"
echo "target lock: $(aws --profile "$PROFILE" s3 cp --quiet "s3://$TGTDATA/instances/$H/lock.json" - | jq -c .)"
aws --profile "$PROFILE" s3 cp --quiet "s3://$DEPB/apps/$ACCT/$APP/instances/$H.json" - | jq -c '{region, move, last_move}'
echo "target moves/$H/ current: $(aws --profile "$PROFILE" s3 ls --recursive "s3://$TGTDATA/moves/$H/" | wc -l)"
carlos status --layout --console "$C" | grep -F "$H"

Pass:

  • the source's /data/.instances/<host>/ is empty (no stray env file);
  • the source has no row and no timers for the host, and its journal says "removed retired instance route" and then "tombstone lifted";
  • parked on the source, under /data/.offline-moves/<host>/<move-id>/: sealer/, source/instance/ (the old nested directory, database included) and, if there was one, source/<host>/ (the legacy flat directory). The tool never deletes these. Add them to the record for the claim census;
  • left in place on the source by design: /data/instances/<host> and /data/.instances/<host> (empty), /opt/carlos/live/<host> (root-owned link) and /data/programs/<host>.json (root-owned, see section 12). status lists each with "left ... in place";
  • the source lock.json reads owner moved:<move-id> with expires_at 1970-01-01T00:00:00Z. That expired object is the normal end state, not an abandoned move. Before finish it carries a 7-day expiry, renewed on each moves pass;
  • the target lock is the target box's own lease;
  • the record: region uk, move null, last_move outcome done;
  • 0 current objects under the target's moves/<host>/ (the promotion moved them; versions and delete markers remain);
  • the layout row is on the target box, flat.

---

9. The nesting stop, about an hour after open

After open the instance runs flat on the target. The layout pass nests it when it is asleep, or, if it stays awake, stops it once the 1-hour grace is up and nests it. The target journal says so right after open: "layout: deferred until asleep -> shop.example.com (awake (stopped for the move in 1h0m0s unless it sleeps first))". An instance with a schedule shorter than its idle window never sleeps, so expect the stop. Tell the customer's contact there may be a stop of a few seconds about an hour after the move.

About 65 minutes after open:

tgt "journalctl -u carlos-edge --since '-80 min' --no-pager | grep -F $H | grep -E 'layout' | tail -8"
carlos status --layout --console "$C" | grep -E "$H|WARNING"
carlos logs --app "$APP" --account "$ACCT" --console "$C" --since 90m | grep -E 'instance (stopping|exited|woke)' | tail -5
tgt "set -- $H $ROWS_ARGS; $(cat "$W/rows.box.sh")"
curl -sS -m 30 -o /dev/null -w "target after nesting: %{http_code}\n" --resolve "$H:443:$TGT_IP" "https://$H$PROBE_PATH"

Pass: the journal's "awake past its 1h0m0s grace -> stopping it for the move" and "migrated"; status --layout shows the host nested with "migrated at <time>: moved with it"; the digests unchanged; a 2xx. Measured stops: about 2 seconds on dev (the wake took 0.9 s), 24 seconds in the harness. The stop for a large flagship instance was not timed.

Then check the instance's custom domains and aliases, if it has any: amadan #338 found an alias row that kept the flat socket after nesting and answered 502.

Do not move the next instance until this stop is done, so two instances are never nested at once on the target.

---

10. Replica cleanup

A finished move leaves the source region's whole replica in the source data bucket, and the parked local Litestream directory on the source box. For a large instance that is several GB in the source region until someone removes it. Remove both, per instance, once it has passed section 7 and finish, and only then.

Go (Paul): deletes the source region's replica of one moved instance.

L "cleanup" "bash $W/cleanup.sh /data/.offline-moves/$H/<move-id>/source/instance <open time, e.g. 2026-10-21T15:36:02+00:00>"

cleanup.sh (Appendix A):

  1. refuses unless the source lock.json is the lifted tombstone (moved:<id>, expiring in 1970), so a move still in flight is never touched;
  2. lists the current objects under instances/<host>/ in the source data bucket and refuses if any of them, other than lock.json, was written after the open time you give it. A write after open would mean something still writes there;
  3. removes the parked local replica .../source/instance/.activator-<host>.db-litestream/ltx on the source box, by exact path (a few hundred KB);
  4. deletes every object under instances/<host>/ by exact key, keeping lock.json (the lifted tombstone stays as the record), and logs each key with the version id it removed and its size.

The data buckets are versioned, and noncurrent versions expire 7 days after they are replaced (platform #336). A delete only adds a delete marker; the bytes stay as noncurrent versions for 7 days, are billed until then, and can be brought back from the logged version id within that window. The parked database, env and config.env under source/instance/ stay too.

Never run this for an instance whose move was aborted: its source replica is the live one.

On 2026-10-21 this removed 290 objects (72 MB) for the 27 MB instance, 305 objects (1.58 GB) for the 1.6 GB instance and 270 objects (3.66 GB) for the 3.7 GB instance, about 4 minutes each.

---

11. Rollback

Before open: abort. At any doubt, a failed item, a target missing config.env, or a door that refuses for a real reason. move.sh does it on its own for seal, flip and the gate. By hand:

echo "ABOUT TO RUN abort $H $(date -u +%FT%T.%3NZ)"; MV abort; echo "abort exit $?"
echo "ABOUT TO RUN finish $H $(date -u +%FT%T.%3NZ)"; MV finish; echo "finish exit $?"

Both exit 3. abort prints "Moving shop.example.com back to ie." then "Every box has put shop.example.com back in ie."; finish prints "shop.example.com is back in ie and serving. Nothing was lost." The source's database was never removed, so nothing is lost. It took 25 seconds for the abort and under a second for the finish on the flagship, and the instance answered 2xx on the source 2 seconds later.

Check after an abort:

  • facts and the row fingerprint on the source equal the baseline, and quick_check ok;
  • the source row is ready and nested; the source lease is the source box's own, at generation 2;
  • the source journal shows "wake fence lifted" and "tombstone lifted on abort";
  • the target has no row and no timers, and parked what it had extracted under /data/.offline-moves/<host>/<move-id>/target/. Leave it: it does not block a retry;
  • the target bucket has 0 current objects under instances/<host>/ and moves/<host>/;
  • the record has no region, move null, last_move outcome aborted;
  • 2xx through the source box, through the target box (backhaul) and through DNS.

A retry is a new move, staged from scratch, after section 3 passes again.

After open: there is no move back. abort is refused once the move is open (MV-O27), the target holds the only live copy and may already have taken writes. Fix forward on the target. Most problems after open (an app that cannot boot there, a certificate, reaching its store from the new region) show up in the first minute of section 7. A move back to the source region would be a new offline move in the other direction. It has never been run, it needs the target region's staging grant on the source data bucket, and the source box would need the section 3.3 checks, including its own stale /opt/carlos/live/<host> link, which the move leaves in place. Treat it as a new piece of work with its own approval, not as a rollback.

---

12. Known gaps

  • amadan #339: nothing checks the target before the instance goes offline. A stale root-owned /opt/carlos/live/<host> on the target fails flip (exit 4, the release item, MV-O177) only after start. Section 3.3 is the workaround. The same discussion covers two smaller things: after an abort from staging, status reads as if the instance moved and came back; and no output says how many objects the seal copied (section 6 has the count).
  • amadan #338: an alias row can keep the flat socket after nesting and answer 502. Check an instance's aliases after the nesting stop (section 9).
  • The source keeps root-owned files. /data/programs/<host>.json (root, 0644, in a root-owned directory) cannot be parked by the agent, which runs as carlos; status says "left /data/programs/<host>.json in place: cannot be renamed: ... permission denied". The root-owned /opt/carlos/live/<host> link stays too, by design. Both are listed for the claim census, and both are the kind of leftover that section 3.3 catches if that box is ever a move target.
  • The console's offline figure ends at open and leaves out the first wake (section 6).
  • One offline-move lock per deployment. A staged or moving instance blocks every other one.
  • The source region keeps the replica until the cleanup in section 10, and the parked database and directories indefinitely.
  • A new nested instance can start a few seconds before its config.env (seen on dev on v0.28.6: about 12 seconds, then a config-change restart). That is the instance-create path, not the move: the move writes config.env at landing, before the first request.

---

13. Rehearsing on dev

Rehearse each new release's move path on dev before using it on the flagship. Dev's pair is sophie-knitting (home, ie, the console box, guard armed) to dennis-stenciling (uk); leo-climbing is a second home-region box and takes part as a source box. The values are in Appendix A.

The rehearsals of 2026-10-21 used one app, moverehearsal in account bab, built from wakecheck's bytes, with four settings including store-shaped ones, one secret and a schedule every-minute --every 1m --path /tick. Each scenario uses a fresh instance of that app (moverehearsal-a.dev.carlosframework.com and so on), created with carlos instances create and woken through DNS so that it lands on sophie, its nested home. The probe path is /.

Scenarios worth running:

  1. A move with stage: two stage rounds with writes between them, then move.sh. Expect about 40 seconds offline.
  2. Abort from staging: stage, then abort and finish (both exit 3); the instance never goes offline and the same process keeps serving.
  3. Abort after flip, before open: about 30 seconds offline, nothing lost.
  4. The nesting stop an hour after the first move's open.

Afterwards: carlos instances delete each rehearsal host (the route retires on the next pass; parked data stays), then the replica cleanup of section 10 for the moved ones. Record what is left on each box and in each bucket.

---

14. Telling the customer

A draft for Paul to edit and send, one per instance, at least a day ahead:

> We will move <host> to our UK region on <date> at <time> UTC. It will be offline for about 3 minutes, and we have booked 15 minutes for it. Your data moves with it, and if anything looks wrong before we reopen it, we put everything back as it was. Scheduled jobs that fall due while it is offline will not run. About an hour later it may pause again for a few seconds while we finish settling it in.

Tell the customer's contact when each move starts and when it is done.

---

Appendix A. Setup and helper scripts

A.1 Scratch and values

Make a scratch directory under $TMPDIR (never /tmp directly) and keep its exact path; it is removed by that path at the end (A.4). Keep the log somewhere that outlives the scratch.

W=$(mktemp -d "$TMPDIR/offline-move.XXXXXX"); echo "$W"

Write $W/env.sh. These are the flagship's values for IE to UK; dev's are in the comments.

cat > "$W/env.sh" <<'EOF'
export W=<the scratch path printed above>
export LOG=<the move log, e.g. $HOME/.cache/<task>/moves-<date>.md>
# Deployment. Dev: C=https://platform.dev.carlosframework.com PROFILE=kass DEPB=carlos-dev-640012953448
export C=https://console.carloku.com PROFILE=keymail DEPB=carlos-flagship-271376211898
# Source box (flagship IE, annie-welding). Dev: sophie-knitting i-05498eada6f0afd23, eu-west-1
export SRC_REGION=eu-west-1 SRC_ID=i-092c0c1eea75723cb
# Target box (flagship UK, iris-clockmaking). Dev: dennis-stenciling i-009d593f451cfd562, eu-west-2
export TGT_REGION=eu-west-2 TGT_ID=i-03c310ceaf50a9bab
# Data buckets: each box's CARLOS_HIBERNATE_BUCKET, also data_bucket in its edges file. Dev UK: carlos-dev-uk-640012953448
export SRCDATA=carlos-data-ie-271376211898 TGTDATA=carlos-data-uk-271376211898
export TO=uk
# The instance
export H=shop.example.com APP=<app> ACCT=<account sqid>
export PROBE_PATH=/signin          # a path that answers 2xx without signing in
export ROWS_ARGS=""                # "<table>:<max_rowid> ...", from 3.4
export SRC_IP=<source public IP> TGT_IP=<target public IP>
ts()  { date -u +%T.%3NZ; }
MV()  { carlos move offline "$1" --host "$H" --app "$APP" --account "$ACCT" --console "$C" ${2:-}; }
src() { "$W/ssm.sh" "$PROFILE" "$SRC_REGION" "$SRC_ID" "$1"; }
tgt() { "$W/ssm.sh" "$PROFILE" "$TGT_REGION" "$TGT_ID" "$1"; }
# L <label> <command>: logs the label and command, then the output line by line as it runs, then the exit
L() { local label=$1; shift
  printf '\n#### %s  (started %s)\n```\n$ %s\n' "$label" "$(date -u +%FT%T.%3NZ)" "$*" >> "$LOG"
  bash -c "source $W/env.sh; $*" 2>&1 | tee -a "$LOG"; local rc=${PIPESTATUS[0]}
  printf '```\n(ended %s, exit %s)\n' "$(date -u +%FT%T.%3NZ)" "$rc" >> "$LOG"; return $rc; }
# note <text>: a timestamped line in the log, e.g. what you are about to run and what you expect
note() { printf -- '- %s %s\n' "$(date -u +%FT%T.%3NZ)" "$*" | tee -a "$LOG"; }
EOF
source "$W/env.sh"

The two addresses, for --resolve:

aws --profile "$PROFILE" --region "$SRC_REGION" ec2 describe-instances --instance-ids "$SRC_ID" --query 'Reservations[0].Instances[0].PublicIpAddress' --output text
aws --profile "$PROFILE" --region "$TGT_REGION" ec2 describe-instances --instance-ids "$TGT_ID" --query 'Reservations[0].Instances[0].PublicIpAddress' --output text

Put them in env.sh and source it again. Do the same whenever you change H or ROWS_ARGS.

A.2 ssm.sh

Runs one command as root on a box over SSM and prints its stdout, with stderr kept apart and a non-zero exit unless the command succeeded. The command goes into a temporary file and runs with stdin closed. SSM returns at most 24,000 characters.

cat > "$W/ssm.sh" <<'EOF'
#!/bin/bash
# usage: ssm.sh <profile> <region> <instance-id> '<shell command>'
set -euo pipefail
profile=$1 region=$2 inst=$3 cmd=$4
b64=$(printf '%s' "$cmd" | base64 -w0)
params=$(python3 -I -c 'import json,sys;print(json.dumps({"commands":["f=$(mktemp) && echo %s | base64 -d > $f && bash $f </dev/null; rc=$?; rm -f $f; exit $rc" % sys.argv[1]]}))' "$b64")
id=$(aws --profile "$profile" --region "$region" ssm send-command --instance-ids "$inst" \
  --document-name AWS-RunShellScript --parameters "$params" --query Command.CommandId --output text)
until s=$(aws --profile "$profile" --region "$region" ssm get-command-invocation --command-id "$id" \
  --instance-id "$inst" --query Status --output text 2>/dev/null) && [ "$s" != InProgress ] && [ "$s" != Pending ]; do sleep 2; done
aws --profile "$profile" --region "$region" ssm get-command-invocation --command-id "$id" --instance-id "$inst" \
  --query StandardOutputContent --output text
aws --profile "$profile" --region "$region" ssm get-command-invocation --command-id "$id" --instance-id "$inst" \
  --query StandardErrorContent --output text >&2
[ "$s" = Success ]
EOF
chmod 700 "$W/ssm.sh"

A.3 Box scripts

Each is sent with src or tgt, taking the host as $1 where it needs one: src "set -- $H; $(cat "$W/facts.box.sh")". All are read-only.

# block.box.sh: claim and Phase 1 switches, the edge unit, the stage path, version, guard, secrets key, region, disk
cat > "$W/block.box.sh" <<'EOF'
echo "== $(hostname)"
echo "claim lines: $(grep -c "^CARLOS_HOST_CLAIM=." /etc/carlos/host.env || true)"
echo "phase1 lines: $(grep -c "^CARLOS_MOVES_PHASE1=" /etc/carlos/host.env || true)"
P=$(systemctl show -p MainPID --value carlos-edge)
echo "edge env claim/phase1 vars: $(tr "\0" "\n" < /proc/$P/environ | grep -cE "^CARLOS_(HOST_CLAIM=.|MOVES_PHASE1=)" || true)"
systemctl show carlos-edge -p User -p Delegate -p ActiveState
echo "stage.path: $(systemctl is-active carlos-stage.path) $(systemctl is-enabled carlos-stage.path)"
/opt/carlos/carlos version 2>&1 | tail -1
if [ -f /opt/carlos/system/guard/policy.json ]; then echo "guard policy present"; else echo "no guard policy"; fi
echo "secrets key lines: $(grep -c "^CARLOS_SECRETS_KEY=" /etc/carlos/host.env || true)"
echo "secrets key sha8: $(grep "^CARLOS_SECRETS_KEY=" /etc/carlos/host.env | cut -d= -f2- | tr -d "\"\n" | sha256sum | cut -c1-8)"
echo "region/pools: $(grep -E "^CARLOS_(REGION|POOLS)=" /etc/carlos/host.env | tr "\n" " ")"
echo "disk avail bytes, use: $(df -B1 --output=avail,pcent / | tail -1)"
EOF

# target.box.sh <host>: what the target box already holds for the host (3.3)
cat > "$W/target.box.sh" <<'EOF'
H=$1
L=/opt/carlos/live/$H
if [ -L "$L" ] || [ -e "$L" ]; then echo "live link: PRESENT -> $(readlink "$L") (dated $(stat -c %y "$L" | cut -c1-19))"; else echo "live link: absent"; fi
for p in /data/instances/$H /data/.instances/$H /data/$H /data/programs/$H.json /data/.offline-moves/$H /data/$H.env /data/$H.config.env /data/$H.db; do
  if [ -e "$p" ]; then echo "PRESENT $p"; else echo "absent $p"; fi
done
[ -d /data/.offline-moves/$H ] && find /data/.offline-moves/$H -maxdepth 2 | sed 's/^/  parked: /'
python3 - "$H" <<'PY'
import sqlite3,sys
h=sys.argv[1]; c=sqlite3.connect('file:/data/registry.db?mode=ro',uri=True)
print('registry routes', c.execute('select host,status from routes where host=?',(h,)).fetchall(),
      'timers', c.execute('select count(*) from timers where host=?',(h,)).fetchone()[0])
PY
EOF

# facts.box.sh <host>: env and key digests, config.env, files, db, row, timers, parked (never values)
cat > "$W/facts.box.sh" <<'EOF'
H=$1
E=""; for d in /data/instances/$H /data; do [ -f $d/$H.env ] && { E=$d/$H.env; break; }; done
if [ -n "$E" ]; then
  echo "  env file: $E $(sha256sum < $E | cut -c1-16) key-sha $(grep ^CARLOS_INSTANCE_KEY= $E | cut -d= -f2- | tr -d '"\n' | sha256sum | cut -c1-16) names: $(cut -d= -f1 $E | sort | tr '\n' ' ')"
else echo "  env file: absent"; fi
for f in /data/instances/$H/$H.config.env /data/$H.config.env; do [ -f $f ] && echo "  config.env $f ($(stat -c %y $f)) sha $(sha256sum < $f | cut -c1-16): $(cut -d= -f1 $f | sort | tr '\n' ' ')"; done
FD=""; for d in /data/instances/$H/$H /data/$H; do [ -d $d ] && { FD=$d; break; }; done
if [ -n "$FD" ]; then
  echo "  files: $FD $(cd $FD && find . -type f | wc -l) files, digest $(cd $FD && find . -type f -print0 | sort -z | xargs -0 -r sha256sum | sha256sum | cut -c1-16)"
else echo "  files: none 0 files, digest none"; fi
for f in /data/instances/$H/$H.db /data/$H.db; do [ -f $f ] && echo "  db: $f $(stat -c %s $f) bytes"; done
ls /data/instances/$H 2>/dev/null | tr '\n' ' ' | sed 's/^/  instance dir: /'; echo
python3 - $H <<'PY'
import sqlite3,sys
h=sys.argv[1]; c=sqlite3.connect('file:/data/registry.db?mode=ro',uri=True)
print('  row:', c.execute('select status,last_error,region,db_path,next_due,replication from routes where host=?',(h,)).fetchall())
try:
    t=c.execute('select * from timers where host=?',(h,)).fetchall()
    print('  timers:', [(x[1],x[2],x[3],x[4],x[7]) for x in t])
    print('  timer names:', ' '.join(sorted(x[1] for x in t)))
except Exception as e: print('  timers: ?', e)
PY
ls -d /data/.offline-moves/$H/*/* 2>/dev/null | sed 's/^/  parked: /'
EOF

# sizes.box.sh <host>: files tree (limit 256 MiB), nested dir, db, local Litestream dir, sidecars, legacy flat dir
cat > "$W/sizes.box.sh" <<'EOF'
H=$1; D=/data/.instances/$H
echo "== $H"
echo "files tree bytes: $(du -sb $D/$H 2>/dev/null | cut -f1)"
echo "nested dir: $(du -sh $D | cut -f1)"
ls -l $D/$H.db 2>&1 | awk '{print "db bytes:", $5}'
echo "local ltx: $(du -sh $D/.activator-$H.db-litestream 2>/dev/null | cut -f1)"
ls $D/*sidecar* /data/$H-sidecar.env 2>&1 | head -2
ls -la /data/$H 2>&1 | head -6
EOF

# procs.box.sh <host>: processes in the host's cgroup slice (0 before flip)
cat > "$W/procs.box.sh" <<'EOF'
H=$1
f=$(find /sys/fs/cgroup/carlos.slice/carlos-instances.slice -maxdepth 1 -name "carlos-instances-$H.slice" 2>/dev/null | head -1)
echo "slice: ${f:-none}"
if [ -n "$f" ]; then echo "processes: $(cat $f/cgroup.procs | wc -l)"; else echo "processes: 0 (no slice)"; fi
d=$(find /sys/fs/cgroup/carlos.slice/carlos-edge.service -maxdepth 1 -name "carlos-exec@$H" 2>/dev/null | head -1)
if [ -n "$d" ]; then echo "exec cgroup $d processes: $(cat $d/cgroup.procs | wc -l)"; fi
EOF

# stray.box.sh <host>: what the source keeps after finish
cat > "$W/stray.box.sh" <<'EOF'
H=$1
echo "== /data/.instances/$H"; ls -la /data/.instances/$H/ 2>&1
echo "== parked"; ls -d /data/.offline-moves/$H/*/* /data/.offline-moves/$H/*/source/* 2>&1
echo "== legacy /data/$H"; ls -la /data/$H 2>&1 | head -5
echo "== programs"; ls -la /data/programs/$H.json 2>&1
echo "== live link"; ls -la /opt/carlos/live/$H 2>&1
EOF

# schema.box.sh <host>: tables, row counts and time columns, to choose the fingerprint tables
cat > "$W/schema.box.sh" <<'EOF'
H=$1
DB=""; for f in /data/instances/$H/$H.db /data/$H.db; do [ -f $f ] && { DB=$f; break; }; done
echo "db: ${DB:-none}"; [ -z "$DB" ] && exit 0
python3 - "$DB" <<'PY'
import sqlite3,sys
c=sqlite3.connect('file:%s?mode=ro'%sys.argv[1],uri=True)
for (n,) in c.execute("select name from sqlite_master where type='table' order by name"):
    cols=[r[1] for r in c.execute('pragma table_info("%s")'%n)]
    k=c.execute('select count(*) from "%s"'%n).fetchone()[0]
    tcols=[x for x in cols if x in ('created_at','updated_at','at','ts','timestamp','occurred_at','sent_at','inserted_at')]
    print(n, k, 'timecols=%s'%tcols)
PY
EOF

# rows.box.sh <host> [table[:max_rowid] ...]: read-only row fingerprint (counts, max rowids, digests; never row content)
cat > "$W/rows.box.sh" <<'EOF'
H=$1; shift
DB=""; for f in /data/instances/$H/$H.db /data/$H.db; do [ -f $f ] && { DB=$f; break; }; done
echo "db: ${DB:-none}"; [ -z "$DB" ] && exit 0
python3 - "$DB" "$@" <<'PY'
import sqlite3,sys,hashlib
c=sqlite3.connect('file:%s?mode=ro'%sys.argv[1],uri=True)
c.execute('begin')
tabs=[n for (n,) in c.execute("select name from sqlite_master where type='table' and name not like 'sqlite_%' and name not like '_litestream%' order by name")]
counts={n:c.execute('select count(*) from "%s"'%n).fetchone()[0] for n in tabs}
print('tables %d, total rows %d, nonempty %d' % (len(tabs), sum(counts.values()), sum(1 for v in counts.values() if v)))
print('counts digest', hashlib.sha256(''.join('%s:%d\n'%(n,counts[n]) for n in tabs).encode()).hexdigest()[:16])
for arg in sys.argv[2:]:
    t,_,lim=arg.partition(':')
    r=c.execute('select count(*),max(rowid) from "%s"'%t).fetchone()
    print('%s: count %d max_rowid %s' % (t,r[0],r[1]))
    if lim:
        h=hashlib.sha256(); n=0
        for row in c.execute('select rowid,* from "%s" where rowid<=? order by rowid'%t,(int(lim),)):
            h.update(repr(row).encode()); n+=1
        print('%s rowid<=%s: %d rows digest %s' % (t,lim,n,h.hexdigest()[:16]))
c.execute('rollback')
PY
EOF

A.4 Operator scripts

# stage.sh: one staging round for $H, then status, the record's move and the stager report
cat > "$W/stage.sh" <<'EOF'
#!/bin/bash
source "$(dirname "$0")/env.sh"
now() { date -u +%FT%T.%3NZ; }
echo "=== before $(now): source instances/$H/ $(aws --profile $PROFILE s3 ls --recursive s3://$SRCDATA/instances/$H/ | wc -l) objects; target moves/$H/ $(aws --profile $PROFILE s3 ls --recursive s3://$TGTDATA/moves/$H/ | wc -l) objects"
echo "ABOUT TO RUN stage $H $(now)"
s=$(ts); out=$(MV stage "--to $TO" 2>&1); rc=$?
echo "--- stage $s -> $(ts) exit=$rc"; echo "$out" | sed 's/^/    /'
echo "--- status $(ts)"; MV status | sed 's/^/    /'
M=$(aws --profile $PROFILE s3 cp --quiet s3://$DEPB/apps/$ACCT/$APP/instances/$H.json - | jq -c '.move')
echo "record move: $M"
ID=$(echo "$M" | jq -r '.id // empty')
if [ -n "$ID" ]; then
  for k in $(aws --profile $PROFILE s3 ls --recursive "s3://$DEPB/control/offline-moves/$H/$ID/stager/" | awk '{print $4}'); do
    echo "stager report $k"
    aws --profile $PROFILE s3 cp --quiet "s3://$DEPB/$k" - | jq -c '{box, phase, at, replica_objects, replica_bytes, staged_objects, staged_bytes, round_objects, round_bytes, stage_error}'
  done
fi
echo "END stage.sh rc=$rc $(now)"
exit $rc
EOF

# move.sh [probe-seconds]: final stage, baseline, then start, seal, flip, the gate and open back to back.
# Before open, a failed seal, flip or gate aborts the move and runs finish. After open it only reports.
cat > "$W/move.sh" <<'EOF'
#!/bin/bash
source "$(dirname "$0")/env.sh"
PROBE_FOR=${1:-1200}
now() { date -u +%FT%T.%3NZ; }
door() { # door <verb> [args]: retries only a report that is not current (MV-O178)
  local i out rc s
  for i in 1 2 3 4 5 6 7 8; do
    echo "ABOUT TO RUN $1 $H $(now) (try $i)"
    s=$(ts); out=$(MV "$1" "${2:-}" 2>&1); rc=$?
    echo "--- $1 try $i $s -> $(ts) exit=$rc"; echo "$out" | sed 's/^/    /'
    if [ $rc -ne 0 ] && echo "$out" | grep -q "is not current"; then sleep 15; continue; fi
    return $rc
  done; return $rc; }
back_out() { # back_out <reason>: abort and finish, both expected to exit 3
  local s out rc
  echo "STOP: $1 $(now). Aborting (before open)."
  echo "ABOUT TO RUN abort $H $(now)"; s=$(ts); out=$(MV abort 2>&1); rc=$?
  echo "--- abort $s -> $(ts) exit=$rc"; echo "$out" | sed 's/^/    /'
  echo "ABOUT TO RUN finish $H $(now)"; s=$(ts); out=$(MV finish 2>&1); rc=$?
  echo "--- finish $s -> $(ts) exit=$rc"; echo "$out" | sed 's/^/    /'
  for i in $(seq 1 20); do c=$(curl -s -m 20 -o /dev/null -w '%{http_code}' --resolve $H:443:$SRC_IP https://$H$PROBE_PATH); echo "source $c $(ts)"; case "$c" in 2*) break;; esac; sleep 3; done
  MV status; echo "END move.sh STOPPED $(now)"; exit 12; }

echo "=== final stage right before start $(now)"
door stage "--to $TO" || { echo "STOP: final stage exit $? $(now). Nothing is offline."; MV status; exit 10; }
echo "=== baseline on the source $(now)"
A=$(src "set -- $H; $(cat $W/facts.box.sh)"); echo "$A"
ENVSHA=$(echo "$A" | sed -n -E 's/^  env file: \S+ ([0-9a-f]{16}) key-sha .*/\1/p')
KEYSHA=$(echo "$A" | sed -n -E 's/^  env file: .* key-sha ([0-9a-f]{16}) .*/\1/p')
CFGSHA=$(echo "$A" | sed -n -E 's/^  config.env .* sha ([0-9a-f]{16}): .*/\1/p')
FILES=$(echo "$A" | sed -n -E 's/^  files: \S+ (.*)$/\1/p')
TIMERS=$(echo "$A" | sed -n -E 's/^  timer names: (.*)$/\1/p')
echo "expect env $ENVSHA key $KEYSHA cfg ${CFGSHA:-none} files $FILES timers $TIMERS"
for v in "$ENVSHA" "$KEYSHA" "$FILES"; do [ -n "$v" ] || { echo "STOP: baseline incomplete. Nothing is offline."; exit 9; }; done
echo "=== rows fingerprint on the source, just before start $(now)"
src "set -- $H $ROWS_ARGS; $(cat $W/rows.box.sh)"

T0=$(date +%s.%N); echo "=== T0 $(now)"
door start "--to $TO" || { echo "STOP: start exit $? $(now). Read status: if a move is in flight, abort it and run finish."; MV status; exit 10; }
door seal || back_out "seal exit $?"
echo "--- source slice $(now)"; src "set -- $H; $(cat $W/procs.box.sh)"
door flip || back_out "flip exit $?"

echo "--- target facts (gate before open) $(now)"
B=$(tgt "set -- $H; $(cat $W/facts.box.sh)"); echo "$B"
fail=""
echo "$B" | grep -q -F "row: [('landing', '', '$TO', '/data/$H.db'" || fail="$fail row"
echo "$B" | grep -q -E "^  env file: /data/$H.env $ENVSHA key-sha $KEYSHA " || fail="$fail env/key"
[ "$(echo "$B" | sed -n -E 's/^  config.env .* sha ([0-9a-f]{16}): .*/\1/p')" = "$CFGSHA" ] || fail="$fail config"
[ "$(echo "$B" | sed -n -E 's/^  files: \S+ (.*)$/\1/p')" = "$FILES" ] || fail="$fail files"
[ "$(echo "$B" | sed -n -E 's/^  timer names: (.*)$/\1/p')" = "$TIMERS" ] || fail="$fail timers"
[ -z "$fail" ] || back_out "gate failed:$fail"
echo "gate before open: ok $(now)"

door open || { echo "STOP: open exit $? $(now). Do not abort blindly: read status (runbook section 11)."; MV status; exit 14; }
echo "--- probe $PROBE_PATH through the target until a 2xx (up to $PROBE_FOR s) $(now)"
prev=""; first=""; end=$(( $(date +%s) + PROBE_FOR ))
while [ $(date +%s) -lt $end ]; do
  c=$(curl -s -m 10 -o /dev/null -w '%{http_code}' --resolve $H:443:$TGT_IP https://$H$PROBE_PATH)
  [ "$c" != "$prev" ] && { echo "target $c $(ts)"; prev=$c; }
  case "$c" in 2*) first=$(date +%s.%N); break;; esac; sleep 2; done
[ -n "$first" ] || { echo "NO 2xx through the target within $PROBE_FOR s $(now)"; exit 15; }
echo "FIRST 2xx through the target at $(ts): offline $(python3 -c "print(round($first-$T0,1))") s since start"
curl -sS -m 30 -D - -o /dev/null --resolve $H:443:$TGT_IP https://$H$PROBE_PATH | grep -i -E "^HTTP|x-carlos-version"
for i in $(seq 1 25); do a=$(curl -s -m 20 -o /dev/null -w '%{http_code}' --resolve $H:443:$SRC_IP https://$H$PROBE_PATH); d=$(curl -s -m 20 -o /dev/null -w '%{http_code}' https://$H$PROBE_PATH); echo "$(ts) via source $a | via DNS $d"; [ "${a:0:1}" = 2 ] && [ "${d:0:1}" = 2 ] && break; sleep 3; done
echo "--- status after open $(now)"; MV status
echo "END move.sh OK $(now)"
EOF

# sealcount.sh <seal-start> <seal-end> (UTC, to the second, e.g. 2026-10-21T15:34:38): versions the seal wrote
cat > "$W/sealcount.sh" <<'EOF'
#!/bin/bash
source "$(dirname "$0")/env.sh"
aws --profile $PROFILE s3api list-object-versions --bucket $TGTDATA --prefix moves/$H/ --query 'Versions[].[LastModified,Size]' --output text |
  awk -v a="$1" -v b="$2" '{t=substr($1,1,19)} t<a {pre++} t>=a && t<=b {n++; s+=$2} t>b {post++} END {print "before the seal (stage rounds):", pre+0; print "during the seal window:", n+0, "objects,", s+0, "bytes"; print "after the seal window:", post+0}'
EOF

# cleanup.sh <parked source/instance dir> <open time, e.g. 2026-10-21T15:36:02+00:00>: section 10
cat > "$W/cleanup.sh" <<'EOF'
#!/bin/bash
source "$(dirname "$0")/env.sh"
P=$1 OPENED=$2 PFX=instances/$H/
[ -n "$P" ] && [ -n "$OPENED" ] || { echo "usage: cleanup.sh <parked source/instance dir> <open time>"; exit 2; }
LOCK=$(aws --profile $PROFILE s3 cp --quiet "s3://$SRCDATA/${PFX}lock.json" -); echo "lock.json: $LOCK"
echo "$LOCK" | jq -e '(.owner|startswith("moved:")) and .expires_at=="1970-01-01T00:00:00Z"' >/dev/null || { echo "STOP: the tombstone is not lifted, so the move is not finished"; exit 2; }
aws --profile $PROFILE s3api list-objects-v2 --bucket $SRCDATA --prefix $PFX --query 'Contents[].[Key,LastModified,Size]' --output text > "$W/cleanup-$H.keys"
LATEST=$(grep -v -F "${PFX}lock.json" "$W/cleanup-$H.keys" | cut -f2 | sort | tail -1)
echo "current objects: $(wc -l < "$W/cleanup-$H.keys"); latest write excluding lock.json: $LATEST; opened: $OPENED"
[[ "$LATEST" < "$OPENED" ]] || { echo "STOP: something wrote to the source replica after open"; exit 2; }
L2=$P/.activator-$H.db-litestream/ltx
echo "== parked local replica: $L2"
src "test -d $L2 && echo \"size: \$(du -sh $L2 | cut -f1), files: \$(find $L2 -type f | wc -l)\" && rm -r -- $L2 && echo removed $L2; ls -la $(dirname $L2)"
echo "== source replica objects by exact key (key, removed version, size), keeping lock.json"
n=0; bytes=0
while IFS=$'\t' read -r k lm size; do
  case "$k" in
    "${PFX}lock.json") echo "keep $k"; continue;;
    "$PFX"*) ;;
    *) echo "SKIP unexpected $k"; continue;;
  esac
  v=$(aws --profile $PROFILE s3api head-object --bucket $SRCDATA --key "$k" --query VersionId --output text)
  aws --profile $PROFILE s3api delete-object --bucket $SRCDATA --key "$k" >/dev/null && echo "deleted ${k#$PFX} version $v size $size" && n=$((n+1)) && bytes=$((bytes+size))
done < "$W/cleanup-$H.keys"
echo "deleted $n source replica objects of $H ($bytes bytes)"
echo "== after"; aws --profile $PROFILE s3 ls --recursive "s3://$SRCDATA/$PFX"
aws --profile $PROFILE s3 cp --quiet "s3://$SRCDATA/${PFX}lock.json" - | jq -c .
EOF
chmod 700 "$W"/*.sh

At the end of the session, check nothing you started is still running, then remove the scratch by its exact path (never a glob): rm -r -- <the path printed in A.1>. Keep the log.

---

History

v0.28.5 roll, 2026-10-20

The first release with the offline-move doors. Dev's guard-armed boxes could not take it through their own updater (the v0.28.4 updater read a candidate's stderr as part of its answer), so leo-climbing and sophie-knitting took it by a ruled hand re-arm, and dennis-stenciling by its normal system update. Sophie's guarded console could not be replaced by install-console (it refused an occupied slot), so it was upgraded by an attended recovery, "E1": both slots moved aside, then install-console. All of it was rehearsed first in the harness (Run 9), which also ran the first offline move of a 4 MB instance: 366 seconds offline, most of it waiting for stale box reports. The full roll runbook with its signing session, re-arm and E1 is this file as it stood at commit b1464900 on branch v0285-roll-runbook.

Run 9 found four defects, all fixed in v0.28.6:

| Defect | What it was | |---|---| | 5 | install-console could not replace an installed protected console. Fixed by install-console --upgrade --from <sha256> | | 6 | a new nested instance could run without its config.env for up to about 70 minutes | | 7 | flip and open trusted a box report for 2 minutes while boxes rewrote theirs every 5 to 6, so the doors refused with "still runs a process" for most of each cycle. Fixed by a 15-second report refresh during a move and a "not current" refusal | | 8 | the source's scheduler re-created the parked instance's env file after open |

v0.28.6 roll and the first moves, 2026-10-20 and 21

The harness (Run 10) rolled the v0.28.6 candidate by plain system update on every box and upgraded the guarded console with --upgrade, and moved a small instance in about 20 seconds and a 636 MB instance with a 3.5 GB replica history in 296 seconds. It found that the console's offline figure ends at open, that a large replica's first wake on the target takes minutes, and that a finished move leaves the whole replica in the source region.

v0.28.6 went to dev, then flagship (boxes 08:39 to 08:43Z, console 08:45Z on 2026-10-21).

  • Dev rehearsal on v0.28.6 (08:02 to 09:22Z): a move of a small instance from sophie to dennis, 30.6 seconds offline (the console said 23.5); an abort after flip, 32.8 seconds; the nesting stop at the hour, about 2 seconds; the cleanup. Its corrections: the staging lane's GetObject implicitDeny is the designed grant; only the target's secrets key fingerprint matters; open prints a refusal-like first line while succeeding; the source cannot park the root-owned /data/programs/<host>.json.
  • The first flagship move, a 27 MB instance without stage: seal 82 seconds for 288 replica objects on real S3, 164 seconds offline (the console said 123). Verified by the row fingerprint, nested at the hour.
  • The aborted move, a 1.6 GB instance without stage: the seal alone took about 6 minutes. A session restart killed the move script after the seal; its log wrapper had written nothing, so the move was believed not to have started. Found by status 12 minutes later, aborted, back on the source with nothing lost after 12.7 minutes offline. That is where section 1 comes from.

Paul then approved building the pre-copy and parallel object copies.

v0.28.7 and the moves with stage, 2026-10-21

v0.28.7 added carlos move offline stage and eight object copies at once. It went to dev (14:30Z), then flagship (14:41Z).

  • Dev pre-copy rehearsal (14:31 to 14:54Z): a move from staging, 40.6 seconds offline (the console said 22.3), the seal copying the 4 objects written since the last stage; an abort from staging with the instance never offline. Its corrections are sections 4 and 6: stage once more right before start; read round counts from the stager report; stage does not shorten the first wake on the target; the target can 404 for about 8 seconds after open; report client-measured time; abort from staging exits 3; status after it reads oddly; one lock per deployment.
  • The 1.6 GB instance, first attempt with stage: staged 1.58 GB in 79 seconds while serving with no failed requests, seal copied nothing, then flip exited 4 on the target's stale /opt/carlos/live/<host> link from an earlier attempt. Aborted automatically, about 2 minutes offline, nothing lost. Paul approved clearing the link and its old version directory on the target. Section 3.3 is the check that would have caught it before staging, and amadan #339 the fix.
  • The 3.7 GB instance, with the new target pre-checks: staged in 142 seconds, 190 seconds offline (the console said 117), 52 of them promoting the staged copy into the live replica on the first wake.
  • The 1.6 GB instance, retry: 148 seconds offline (the console said 100).
  • Cleanup: the source region's replicas of all three moved instances deleted by exact key with their version ids logged, lock.json kept, the parked Litestream directories removed.