carlos / platform Public

spec: move-any-app rev 2, Phase 1 only, after Astra round 1 (#175)

Answers all twenty round-1 findings (dispositions in §20) and takes three
outside changes: Paul's rulings M4-M15 (rev 1's recommendations), #199
rev 3 (no record storage fields; tombstones and parking), and the
confirmed root cause (only Ireland holds CARLOS_SECRETS_KEY).

Phase 1 is now fully specified as an operator-driven single-instance move
built on a handoff protocol this spec owns: generation markers outside the
record, holds released only by the exact operation, and an execution right
that moves only on stop evidence. Journaled cut and landing phases,
pre-flip recovery, commit-then-open finalization, clean and dirty
rollback (target writes carried back), timers frozen and replaced exactly,
every registry column classified, a sealed archive with separate content
and ciphertext identity, a move-protocol floor, the shared secrets_key_fp
field for nearest-region R9, and a crash simulation with mutation checks.
Phases 2-5 become follow-on specs with invariants only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

claude 3 weeks ago 45b7769194357dcfbfac83a830674ace7d8d701b parent 9fbf4af

1 file changed, +1059 −1418

docs/superpowers/specs/2026-10-02-move-any-app-design.md +1059 −1418

diff --git a/docs/superpowers/specs/2026-10-02-move-any-app-design.md b/docs/superpowers/specs/2026-10-02-move-any-app-design.md

index be8336431..5619052a9 100644

--- a/docs/superpowers/specs/2026-10-02-move-any-app-design.md

+++ b/docs/superpowers/specs/2026-10-02-move-any-app-design.md

@@ -1,1470 +1,1111 @@

-# Move any app or instance to another region, safely

+# Move any app or instance to another region: Phase 1, a safe single-instance move

**Date:** 2026-10-02

+**Revision:** 2.

+- Rev 1 (e41c4fef) went to Astra (GPT-6, via codex). The findings are in

+ `CODEX-FINDINGS-MOVES-R1.md` beside this file, unedited (9fbf4afe):

+ 12 P1, 7 P2, 1 P3, "not ready for Phase 1 planning".

+- Rev 2 answers every finding. Changes are marked "(codex N)", and §20

+ records each disposition.

+- Rev 2 also takes three changes from outside the review:

+ - Paul accepted every recommendation in rev 1's §23. They are rulings M4

+ to M15 (§0.1).

+ - #199 rev 2 was dropped. #199 rev 3 parks a deleted instance's data at

+ delete time and changes no record field. This spec now **owns** the

+ generation handoff, the cross-region execution fence, the cleanup of

+ what a completed move leaves behind, and the carriage of the instance's

+ files (§0.3, §17).

+ - The cause of the missing config on the UK box is confirmed (§1.2).

+- **Scope cut.** Rev 2 specifies **Phase 1 only**: a safe, operator-driven

+ move of one instance. Everything else (owner self-serve, moves to home,

+ residency deletion, stores, app moves, hooks) is reduced to a list of

+ follow-on specs, each with the invariants it must meet and no design

+ (§16).

**Status:** DRAFT for review. Nothing here is built.

-**Discussion:** amadan `carlos/platform` #175 (instance moves). Its last

-four posts are the production findings this spec answers.

+**Discussion:** amadan `carlos/platform` #175.

**Goal (Paul, 2026-10-02):** "I want region-default AND the ability to

-easily move any app or instance to a different region."

-**Companion:** `docs/superpowers/specs/2026-10-01-nearest-region-default-design.md`

-on branch `design-nearest-region` (approved). That spec gives every app a

-region at creation. This one moves apps and instances afterwards.

+easily move any app or instance to a different region." The region default

+is `docs/superpowers/specs/2026-10-01-nearest-region-default-design.md` on

+branch `design-nearest-region`.

**Builds on:**

-- #331, `docs/superpowers/specs/2026-08-28-instance-move-design.md`: the

- two-phase, data-following move. Its never-two-writers cut ordering

- stands and is extended, not reopened.

-- The moves design record, `docs/superpowers/specs/2026-09-24-instance-moves-for-every-app-design.md`

- on branch `docs-instance-moves-design` (called "the 09-24 record" below),

- with its §0 rulings and slices P0 to P9, and

- `CODEX-RULINGS-INSTANCE-MOVES.md` beside it.

-- On `main` (7edd4b33): P0 (cut fence) and P7 (edge door), merged and

- live; the move-ready fix (a round promotes at a delta of at most 16 MiB,

- and copy receipts handle multipart ETags).

-- **Replica ownership (#199)**, `docs/superpowers/specs/2026-10-02-replica-ownership-and-db-push-design.md`

- on branch `design-replica-ownership` (rev 2 at ef0b7e45, being

- rewritten; called "the #199 spec" below). Approved as a constraint on

- this spec by Paul on 2026-10-02. §0.3 says what it changes here, and §26

- lists every interface point between the two specs, so they can be

- reviewed together.

-Citations are to this worktree at 7edd4b33 unless they name another

-branch or repo.

-## 0. Rulings in force

-### 0.1 Paul's rulings for this spec (2026-10-02). Not reopened here.

-| id | ruling |

-|---|---|

-| M1 | **Who:** owners move their own apps and instances, self-serve, from the console and the CLI. Operators can do the same for any account. The platform refuses unsafe moves. |

-| M2 | **Downtime:** a short cut window (tens of seconds of 503), an automatic post-cut health check, and **automatic rollback to the source if the target will not start**. Not live migration. |

-| M3 | **Stores move with the app:** an app's object store is copied to the target region and rebound, so residency follows the app (the 09-24 record's P8). |

+- #331, `docs/superpowers/specs/2026-08-28-instance-move-design.md`.

+- The moves design record (the "09-24 record"),

+ `docs/superpowers/specs/2026-09-24-instance-moves-for-every-app-design.md`

+ on branch `docs-instance-moves-design`, with `CODEX-RULINGS-INSTANCE-MOVES.md`.

+- On `main` (7edd4b33): P0 (the cut fence), P7 (the edge door), and the

+ move-ready fix (16 MiB readiness bar, copy receipts for multipart ETags).

+- #199 rev 3, `docs/superpowers/specs/2026-10-02-replica-ownership-design.md`

+ at 66f1dad3 on `design-replica-ownership` (the "#199 spec"): host

+ tombstones and parking at delete time.

+Citations are to this worktree at 7edd4b33 unless they name another branch

+or repo.

-### 0.2 The 09-24 record's §0 rulings, as they apply here

+## 0. Rulings

-They bind unless M1 to M3 override them. Two are overridden, and in a

-narrow way:

+### 0.1 Paul's rulings for this work (2026-10-02). Not reopened here.

+| id | ruling | where Phase 1 spends it |

+|---|---|---|

+| M1 | **Who:** owners move their own apps and instances, self-serve, from the console and the CLI. Operators can do the same for any account. The platform refuses unsafe moves. | Phase 1 is operator-only (§12); owners come in F1 (§16) |

+| M2 | **Downtime:** a short cut window (tens of seconds of 503), an automatic health check after the cut, and **automatic rollback to the source if the target will not start**. Not live migration. | §7 |

+| M3 | **Stores move with the app:** an app's object store is copied to the target region and rebound. | F4; Phase 1 leaves the app store where it is and refuses private stores (§7.1) |

+| M4 | The files the move carries are a fixed set (no app-declared paths). | §6.1 |

+| M5 | The files archive is capped at 256 MiB, enforced while streaming and again at the cut. | §8.1 |

+| M6 | Person-started moves cut automatically once ready and validated; the API default stays manual; `--manual-cut` exists. | §12 |

+| M7 | The source copy is held for 7 days after a move; owners may end the hold early. | F3 |

+| M8 | An app move pauses when one of its instances rolls back, and keeps the target app region. | F5 |

+| M9 | An app store's final copy holds every consumer of the store at 503, behind a durable barrier that covers concurrent creates. | F4, F5 |

+| M10 | An app with sidecars is refused an app move until sidecar moves exist. | F5 |

+| M11 | Target pre-warming is deferred; move eligibility is gated on measured restore and cut duration instead. | §7.3 (T11) |

+| M12 | Rehearsal order: a probe app, then a disposable ips instance, then kpmg, each with scoped approval. | §14.4 |

+| M13 | Owners only, not members; the member-minted instance-token bypass is closed. | Phase 1 is operator-only (§12); F1 |

+| M14 | One landing path in production. (Ruled as "require #199's epoch". #199 rev 3 has no epoch; the intent stands: Phase 1 has exactly one landing path.) | §7.6 |

+| M15 | A landed copy is installed into the target's `instances/`, never served from `moves/` (Astra's I-1 verdict). | §5.1, §7.6 |

+### 0.2 The 09-24 record's rulings

+They bind unless M1 to M15 override them:

- **R4 ("rollback is a move back, never an automatic undo") is narrowed by

- M2.** Before the target has admitted any public traffic, a failed landing

- is undone automatically by flipping the record back (§7.6). Nothing of

- value is lost, because no user write reached the target. After the

- target opens, R4 stands exactly: recovery is a move back, and nothing

- ever restores an older source copy over newer target data (the 09-24

- record's M2).

-- **Q5 ("other accounts are operator-only until #171") is superseded by M1

- for an owner's own apps.** §14 says how the residual #171 risk is

- bounded: the console's own move ledger, not the record, authorizes every

- privileged act a move performs.

-Everything else binds as written: R1 to R3 and R5, Q1 to Q4 and Q6 to Q8,

-M1 and M2 of that record (write fence for stores, recovery path for a stuck

-landing), P4 joining the minimum path, and P7-trust.

-### 0.3 The #199 constraint (Paul, 2026-10-02)

-This spec assumes the #199 spec's storage model and is written against it:

-- **Replica data is keyed by incarnation and generation.** A record's

- replica lives at `<Root>/<host>/<incarnation>/<Gen>/`, and the record's

- `DB` pointer names the current generation (#199 §2.2, §2.3). The local

- database carries a `.lineage` file naming the storage key it belongs to

- (#199 §2.5).

-- **Records that exist when #199 rolls out keep the legacy host-only

- layout** (`instances/<host>`) until their first v2 event. A record is

- legacy while it has no `DB` pointer and was created before the

- deployment's v2 epoch (`control/storage-v2.json`, #199 §2.3).

-- **A move lands its copy as a new generation in the target, and the flip

- sets `Region` and `DB` in one CAS** (#199 §4). A stale copy from an

- earlier stay in the target has an older generation and is never read.

- "Staging wins" (`backend_exec.go:1581-1588`) goes away for v2 records.

-- **The source still writes only `moves/*` in other regions.** No new IAM.

-- **Still undecided in #199:** whether landing copies the staged

- generation into `instances/`, or the pointer names the staging root

- (`Root: "moves"`). This spec works with either and recommends one (§26,

- I-1).

-Where this spec says "generation" it means #199's `Gen`. Where it gives a

-legacy path, that path applies only to records #199 classifies as legacy.

-## 1. What happened in production, and why

-### 1.1 Finding 1: moves never reached `ready` (fixed on main)

-`bdn/ips` `kpmg.bdn.oncarlos.com` sat in `preparing` for 80 minutes. The

-cause was multipart ETags: litestream uploads large LTX files in parts,

-`CopyObject` gives the copy a single-part ETag, and the mirror's ETag

-comparison never matched. Fixed in 2cbf6196 and 167d721b (merged as

-7edd4b33): copy receipts in the source's own bucket at

-`move-receipts/<host>.json` (`internal/activator/backend_exec.go:1557`)

-and a 16 MiB readiness bar (`internal/host/movespass.go:215-235`, applied

-at `:387`). Nothing more is needed for it here.

-### 1.2 Finding 2: a completed move left the instance unable to start for 1h54m

-kpmg moved IE (home) to uk cleanly and then failed every wake on the UK

-box with `instance not healthy within 30s`

-(`internal/activator/backend_exec.go:1443`). Two causes.

-**(a) The per-host data directory does not move.** The move carries the

-litestream replica and nothing else. The cut mirrors

-`instances/<host>/` into the target's `moves/<host>/`

-(`movespass.go:476-487`), purges the local database artifacts

-(`:492`, `PurgeLocalDBArtifacts` at `backend_exec.go:1566-1575`, which

-removes the DB, WAL, SHM and the two litestream meta directories and

-nothing else), and flips the record (`:494`). The app keeps its

-`instance.key` and `report.token` in `/data/<host>/`, beside its database.

-No platform code knows that directory exists: no path in this repo is

-built from it, nothing backs it up, and nothing deletes it. On the UK box

-the app found no key, minted a new one, and could not read data sealed

-under the old one.

-**(b) `<host>.config.env` was never rendered on the target.** Exec

-config is rendered by the agent's in-process `SyncExec`

-(`internal/appconfig/syncexec.go`, `ConfigEnvPath` at `:71-76`), which runs

-only when the box holds `CARLOS_SECRETS_KEY`: both config loops log

-"disabled" and return without it (`internal/host/agent.go:881-886`,

-`:1279`), and the fast converge skips the config leg when the key is absent

-(`agent.go:1812`). The spawn treats a missing config file as an empty one

-(`backend_exec.go:1146-1149`), so the app started with no store binding

-and logged "build app: File storage is unavailable." The UK box had

-`<host>.env` (the activator mints it at first wake,

-`internal/activator/prepare.go:49`) but no `<host>.config.env`.

-**Which of three causes stopped the render is not yet confirmed.** In

-order of likelihood: the UK box has no `CARLOS_SECRETS_KEY` in

-`/etc/carlos/host.env` (the loops are then inert, see §6); it holds a

-different key, so every sealed value fails to open; or `SyncExec` ran and

-skipped the route. The UK agent's journal answers it in one line (the

-"config sync loop disabled" message at `agent.go:884`). Phase 1's first

-task reads it. The design below does not depend on the answer: the

-target-readiness report (§7.3) catches all three before anything is

-stopped.

-Recovery was a hand edit of the record's region back to home plus a serial

-bump. It worked because the source still had the key in `/data/<host>/`

-and its own bucket still had the replica. Had anything deleted

-`/data/<host>/` on the source, the data would have been unrecoverable.

+ M2.** A landing that fails before the target admits public traffic is

+ undone automatically (§7.8). After the target opens, R4 stands: recovery

+ is a move back, and nothing restores an older source copy over newer

+ target data.

+- **Q5 (other accounts are operator-only until #171)** is consistent with

+ Phase 1, which is operator-only.

+- Everything else binds as written.

+### 0.3 What #199 rev 3 changes here

+- **No record field carries storage identity.** Replica data stays at

+ `instances/<host>` (`internal/activator/activator.go:138`). This spec

+ therefore keys a move's generation by the move id, in object names and in

+ marker files, never in a persistent record field (§5.1).

+- **Deletes write a host tombstone first**, at

+ `control/host-tombstones/<host>/<incarnation>.json`, and the box in the

+ tombstone's region parks `instances/<host>/` and `moves/<host>/` into

+ `parked/<incarnation>/<host>/`. Reconcile defers a recreated host's row

+ until its park is done (#199 §2, §3). A move must compose with all three

+ (§7.12).

+- **The incarnation** is `provision.Incarnation(rec)`: `RecordIdentity` over

+ account, app, host and CreatedAt, or a domain-separated legacy identity

+ when CreatedAt is zero (#199 §2). Every move artifact is bound to it.

+- **The source-region lineage a completed move leaves behind, and the

+ instance's working directory, are this spec's** (#199 §2.5b of rev 2 and

+ §5 of rev 3).

+## 1. What happened in production

+### 1.1 Finding 1: busy instances never reached `ready` (fixed on main)

+Multipart ETags never matched after `CopyObject`, so every round re-copied

+everything. Fixed in 2cbf6196 and 167d721b (merged as 7edd4b33): copy

+receipts (`internal/activator/backend_exec.go:1557`) and a 16 MiB

+readiness bar (`internal/host/movespass.go:215-235`, applied at `:387`).

+### 1.2 Finding 2: a completed move could not start for 1h54m

+`kpmg.bdn.oncarlos.com` (account bdn, app ips) moved IE (home) to uk. Every

+wake on the UK box then failed with `instance not healthy within 30s`

+(`backend_exec.go:1443`). Two causes, both now confirmed.

+**(a) The per-host data directory does not move.** The cut mirrors the

+replica (`movespass.go:476-490`), purges database files only

+(`backend_exec.go:1566-1575`), and flips the record (`movespass.go:494`).

+The app keeps `instance.key` and `report.token` in `/data/<host>/`. No

+platform code names that directory. On the UK box the app minted a new

+key and could not read data sealed under the old one.

+**(b) Config never rendered, because only the Ireland box holds

+`CARLOS_SECRETS_KEY` (codex 20).** The UK, US and AU journals all log

+"config sync loop disabled: CARLOS_SECRETS_KEY not set"

+(`internal/host/agent.go:884`). Without the key both config loops return

+at once (`agent.go:881-886`, `:1279`), the fast converge skips the config

+leg (`agent.go:1812`), and the spawn treats a missing config file as empty

+(`backend_exec.go:1146-1149`). So the app started with no store binding.

+**Remediation (Paul approved, ops in progress):** distribute the key to

+every region box. **Verification** is the fingerprint every box publishes

+(§10): a region is ready when its boxes publish the console's fingerprint.

+Readiness check T2 refuses any target that does not (§7.3).

### 1.3 Finding 3: the rehearsal proved nothing

-`bdf/hello` moved IE to uk in about 6 minutes with about 50 s of 503. It

-has no key file, no store, no config and no scheduled jobs, so it

-exercised none of the state that failed. §20.4 specifies a rehearsal app

-shaped like ips.

+`bdf/hello` has no key file, store, config or timers. §14.4 rehearses with

+an app shaped like ips.

### 1.4 Finding 4: SSM Parameter Store is regional

-The carloku-infrastructure root says so plainly: "SSM Parameter Store is

-regional: this host reads /carlos/* in ITS OWN region, so every parameter

-it needs (release key, app secrets) must be created in eu-west-2. Nothing

-replicates them from eu-west-1." (`region-uk.tf:61-64`; the same note is in

-`region-us.tf` and `region-au.tf`). §6 traces what that means for an

-instance.

-## 2. Goals and non-goals

-Goals:

-- **Any eligible instance moves safely.** Every piece of per-host state is

- carried or provably reproduced on the target, checked before anything

- is stopped, and checked again before the target serves.

-- **A target that cannot run the instance gives it back.** The source

- serves again automatically, with the data it had at the cut.

-- **An app moves as a unit.** Its region changes, then its instances move

- one at a time, then its object store follows (M3).

-- **Owners do it themselves** from the console and `carlos move` (M1).

-- **The source region is emptied** after a hold period (P5), so a move

- delivers the residency it was bought for.

-Non-goals:

-- Live migration, zero-downtime cuts, or pre-warming the target (§23 Q8).

-- Moving pooled, fleet-placed, addr-backed, unit-backed or non-hibernating

- instances, replicas, or homes with replicas (§13).

-- The freeze/thaw/verify hooks (the 09-24 record's P1). They make a cut

- graceful; this spec makes it safe without them. They slot in at Phase 5.

-- Adoption of externally run instances (P6) and the isolated layout pass

- (P9). They stay on the 09-24 record. §8.5 records one requirement P9

- must now meet.

-- Latency routing for moved instances. Phase L of the nearest-region spec

- derives it from the record's region, which a move already updates.

+"SSM Parameter Store is regional: this host reads /carlos/* in ITS OWN

+region ... Nothing replicates them from eu-west-1" (carloku-infrastructure

+`region-uk.tf:61-64`). §10 traces what an instance needs from it: only the

+box's secrets key.

+## 2. Phase 1: what it does and what it refuses

+Phase 1 moves **one** exec-backed, hibernating `kind:"instance"` record from

+one non-fleet placement to another, driven by an operator through the

+console API or `carlos move`. It carries every piece of per-host state the

+instance needs, proves the target can run it before the source stops,

+starts it on the target with no public traffic, and opens only after a

+commit. If the target cannot start it, the instance goes back to the

+source automatically, carrying anything the target wrote.

+Phase 1 refuses everything §7.1 lists, including moves into home (F2),

+private stores (F4), and any caller that is not an operator (F1).

+Phase 1 does **not** delete anything. What a completed move leaves in the

+source region is parked under the move id, so it is enumerable, never read

+again, and ready for the residency cleanup in F3.

## 3. Words

-- **Instance move:** one record's `region` changes. The app's region does

- not.

-- **App move:** the app's region changes (the console `apps.region`

- column, nearest-region spec §5.1), then each of its instances that lives

- in the old region moves, then the app's store follows.

- **Source / target:** the box serving the instance before the move, and

- the box that will serve it after. "Box" means the one box whose registry

- owns the route (`OwnsRoute`, `movespass.go:180-185`).

-- **The flip:** the single record write that sets `region` to the target.

-- **Landing:** the interval after the flip during which the target

- restores, checks and probes the instance with no public traffic.

-- **Open:** the target admits public traffic (route status `ready`).

-- **Authoritative copy:** the one copy of the instance's data that a

- later restore must reproduce. §4 defines where it is in every state.

-- **Hold:** the period after a successful move during which the source

- keeps its copy (§10).

-- **Incarnation, generation, storage key, epoch:** #199's terms (§2.1 to

- §2.3 there). The **source generation** is the one the record points at

- when the move is requested (`move.from_db`). The **move generation** is

- the new one the move allocates for the target (`move.to_gen`).

+ the box serving it after. "Box" means the one box whose registry owns the

+ route (`OwnsRoute`, `movespass.go:180-185`).

+- **Move id:** console-minted, unique, `mv_` plus 26 lowercase base32

+ characters. It names the move's **generation**.

+- **Generation:** the identity of one installed copy of an instance's state

+ (database, files, timers). An instance that has never moved is in the

+ **origin** generation. A landing installs generation `<move id>`. A dirty

+ rollback installs generation `<rollback id>` (§7.8).

+- **Handoff:** the protocol that transfers the right to run an instance

+ from one box to another, with a new generation (§5). A move is a handoff.

+ A dirty rollback is a handoff back. A future `carlos db push` is a

+ handoff on one box.

+- **Hold:** a registry row status that admits no traffic, no ticks and no

+ ordinary wakes (§5.2).

+- **The flip:** the record CAS that sets `region` to the target.

+- **Commit:** the record CAS after which the target is authoritative.

+- **Open:** the target's row becomes `ready`.

## 4. Invariants

-Review rounds on earlier move work kept finding new interleavings of one

-fault. So the properties are stated once, here, and §20.2 tests them by

-randomized simulation across crashes, not by example.

-- **INV-1 One writer.** At every instant at most one box runs a process

- for the host. While the target runs one during landing, the source's

- route is fenced and its child is proven stopped.

-- **INV-2 No open before commit.** The target sets its route to `ready`

- only after its compare-and-set from `landing` to at-rest has succeeded.

- A box that loses that CAS never opens.

-- **INV-3 No lost acknowledged write.** Any write the instance acknowledged

- to a client (a request admitted by an edge, or a tick) is present in the

- authoritative copy at every terminal state: done, aborted, rolled back,

- or stuck.

-- **INV-4 Authority.** Before the flip the authoritative copy is the

- source (its local DB while running, the source generation once

- stopped). From the flip until the target's commit it is the source

- generation, which is byte-identical to the final staged copy. After the

- commit it is the move generation. A rollback restores from the source

- generation and nothing else.

-- **INV-4b One pointer.** `region` and the `DB` pointer change only

- together, in one CAS: at the flip (to the target and the move

- generation) and at a rollback (back to the source and the source

- generation). No restore on any box reads a location the record does not

- point at, so an older generation from an earlier stay is never read.

-- **INV-5 Files continuity.** Every start of the instance, on any box,

- sees the files and `<host>.env` of the authoritative state. No start

- ever runs where a carried file should exist and does not.

-- **INV-6 Config presence.** No start during landing runs without the

- instance's config rendered on that box, whenever the app has a config

- bundle.

-- **INV-7 Termination.** Every move reaches `done`, `aborted`,

- `rolled_back` or `stuck` in bounded time, provided crashed actors

- eventually restart. `stuck` happens only when the source also fails to

- start after a rollback.

-- **INV-8 Residency.** Nothing deletes source state before `done`. After

- the hold, nothing of the instance remains in the source region except

- what §10 lists. A deletion never removes a lineage that is live at the

- moment it fires.

-## 5. Every piece of per-host state, and what happens to it

-"Per-host" means keyed by the instance's hostname or its instance id

-(`provision.InstanceID`, `internal/provision/provision.go:1259`). The

-flat layout keeps files under `<DataDir>` (`/data`); the nested

-(isolated-exec) layout keeps them under `InstanceDir` =

-`<DataDir>/instances/<iid>/` (`provision.go:1270`;

-`internal/activator/paths.go:33-54`).

-|---|---|---|---|---|---|

-| S1 | SQLite DB, WAL, SHM | `<DataDir>/<host>.db` (`provision.go:1219`) or `InstanceDir/<host>.db` | app | carried as the replica; source copy purged at cut (`movespass.go:492`) | unchanged |

-| S3 | Replica lineage | legacy: `instances/<host>` (`activator.go:138`); v2: `<Root>/<host>/<incarnation>/<Gen>/` in the box's own data bucket (#199 §2.3) | activator | mirrored to target staging `moves/<host>/` (`backend_exec.go:1549`) and promoted over the target's `instances/<host>` (`:1580-1606`); source copy kept forever | mirrored into the **move generation** `moves/<host>/<incarnation>/<to_gen>/` in the target bucket; the flip points the record at it (#199 §4); the source generation is deleted after the hold (§10) |

-| S4b | Local lineage stamp | `<dbPath>.lineage` (#199 §2.5) | activator | n/a | removed first by the cut's purge (#199 §5.6); written on the target after its first sync |

-| S5 | **Instance files** | flat: `<DataDir>/<host>/` (app-created: `instance.key`, `report.token`); nested: files the app writes in `InstanceDir` that are not platform-owned | app | **not carried, not backed up, never deleted** | **carried** in the sealed files archive (§8) |

-| S6 | **`<host>.env`** | `<DataDir>/<host>.env` (`paths.go:50`) | activator mints `CARLOS_ADMIN_TOKEN` (`admintoken.go:115`, `prepare.go:49`); operators add box-local overrides, which win over the bucket (README "App config & secrets") | **not carried**: target mints a new token, overrides are lost | **carried** in the files archive |

-| S7 | **`<host>.config.env`** | `ConfigEnvPath` (`syncexec.go:71-76`) | `SyncExec`, only with `CARLOS_SECRETS_KEY` (`agent.go:881-886`, `:1812`) | rendered only if the target holds the key; absence is silently accepted at spawn (`backend_exec.go:1146-1149`) | reproduced from the bucket; **proven** on the target before the cut (§7.3) and again before the probe (§7.5) |

-| S11 | Timers: declared schedules | `timers` table, `kind=schedule` (`internal/registry/registry.go:565-579`) | converger from `schedules.json` (`internal/schedule/converge.go:17-20`) | rows re-created on the target with `next_at`/`last_at`/`attempts` reset | **carried** with their state; the converger keeps an imported row whose spec and path match (`converge.go:223-241`) |

-| S12 | Timers: once-timers | `timers` table, `kind=once` (`internal/schedule/spec.go:19`), registered by the app over its control socket (`control.go:41`) | app | **lost** (09-24 G15) | **carried** |

-| S15 | Custom domains and certs | aliases derived per box from claims (`internal/domains/aliases.go`); certificate cache in the deployment bucket under `control/certs/` (`internal/edge/certsweep.go:4`) | domains pass, edge | reproduced; 404 between the source's retire and the target's open | reproduced; the 404 becomes a 503 with P2 (Phase 2) |

-**S5 needs a definition the platform can act on.** No code in this repo

-names `/data/<host>/`. It is the app's own convention: a directory named

-for the host beside its database. The design makes it a platform contract:

-- **Flat layout:** the instance files are the directory

- `strings.TrimSuffix(dbPath, ".db")`, which is `<DataDir>/<host>/` for

- every row whose `DBPath` is the default.

-- **Nested layout:** every regular file and directory under `InstanceDir`

- except the platform's own names (S1, S2, S4b, S6 is carried separately,

- S7, S17, S19, and #199's `.incarnation` marker).

-- Anything an app writes elsewhere under `/data` is not carried.

- `docs/site` says so, next to the existing rule that the app must read

- its store coordinates from env.

-**The files belong to the incarnation, and a move hands them to a new

-generation.** The #199 spec gives the nested working directory an

-`.incarnation` marker and moves a mismatched directory aside to

-`<DataDir>/.retired/<old-incarnation>/` (#199 §2.5b). A push replaces only

-the database, so `instance.key` survives a push. A move is different: it

-creates a new generation **in another box**, and the files have to arrive

-with it. So the files are keyed the same way the replica is, in both

-places they live:

-- **In the bucket**, the archive sits beside the move generation, at

- `moves/_files/<host>/<incarnation>/<to_gen>.{archive,json}` (§8). A

- stale archive from an earlier stay has another generation in its key and

- is never read.

-- **On a box**, the directory carries `.incarnation` (#199) and a second

- marker this spec adds, `.files-gen`, naming the generation whose landing

- wrote it. A landing never trusts a directory it did not just write: it

- moves whatever is there aside and extracts fresh (§7.5, L2). A push

- leaves `.files-gen` alone, because the files did not change.

-- **Flat rows**, which #199 cannot attribute (§2.5b there), become

- attributable for this one directory, because this spec makes

- `<DataDir>/<host>/` a platform-defined path. #199's `.incarnation` check

- should extend to it (§26, I-6).

-## 6. SSM, secrets, and what a target box needs

-The platform's Go code reads no SSM parameter at runtime. There is no SSM

-client in this repo. The region-host module only grants a box

-`ssm:GetParameter` on one ARN pattern (`infra/modules/region-host/main.tf:179-180`)

-and denies every other parameter read (`:435-450`). So SSM reaches an

-instance in exactly two ways, and only one of them matters to a move:

-1. **Box provisioning.** `/etc/carlos/host.env` is staged per box, and its

- values come from the deployment's secret store, which on carloku is SSM

- in the box's own region. The value that matters is

- **`CARLOS_SECRETS_KEY`**: it decrypts every sealed config value, and the

- README says it "belongs in exactly two places, your secret manager and

- every host's `/etc/carlos/host.env`". A box without it renders no exec

- config at all (§1.2b).

-2. **Apps that read SSM themselves.** Today these are unit-backed adopted

- apps whose deployment scripts read `/carlos/kass/*` and

- `/carlos/keymail/*` (carloku `README.md:16-18`, `scripts/copy-kass-secrets.sh`).

- Unit-backed routes never move (`internal/console/apiinstancemove.go:53-60`).

-An exec instance's own secrets come from its config bundle in the

-deployment bucket, which lives at home (Q4) and which every box can read.

-**So nothing per-app or per-instance needs to be replicated between SSM

-regions. What has to be true is that every target box holds the

-deployment's secrets key.** That is a box property, so the design checks

-it as one:

-- Each box publishes, in its edges file, the fingerprint of the public half

- of its `CARLOS_SECRETS_KEY` (`secrets_key_fp`, additive and omitempty,

- beside `placement` and `data_bucket` at `internal/host/edgesfile.go:603-605`),

- or `""` when it has none.

-- The console publishes its own `CARLOS_SECRETS_PUBKEY` fingerprint.

-- The move door refuses a target whose fingerprint is absent or different

- (§7.3, check T2). That one refusal would have stopped the kpmg move

- before the cut.

-Operationally, the roll recipe gains one line: staging a new region means

-creating its SSM copy of the secrets key, and the edges file proves it

-landed. A follow-up for carloku-infrastructure (not this repo) is a small

-tofu resource that copies `/carlos/secrets-key`, or whatever the

-deployment names it, into each region from one source of truth.

-## 7. The instance move

-### 7.1 States

-The record keeps #331's in-flight object (`provision.MoveState`,

-`provision.go:907-917`) and its CAS (`SetRecordMove`, `:949`). New states

-and fields are additive. A record written by today's console still parses.

-```

-preparing ─► ready ─► cutting ─► landing ─► (at rest in target; log: done)

- │ │ │ │

- │ │ │ └─► rolling_back ─► (at rest in source; log: rolled_back)

- │ │ │ └─► stuck (source failed too; operator)

- └──────────┴─────────┴─► aborting ─► (at rest in source; log: aborted)

- (cutting only before the source stops anything: cut_refused ─► ready)

+Stated first, designed to, and proved by the simulation in §14.2.

+- **INV-1 One writer.** At any instant at most one process, on any box in

+ any region, runs the instance with write access to its database, files

+ or store. A box starts the instance only while holding the execution

+ right (§5.3), and the right moves between boxes only on **stop

+ evidence** from the box giving it up. A timeout never moves it (codex 1).

+- **INV-2 No lost acknowledged write.** No write the instance acknowledged,

+ to a client, to a tick, or to its own background work or an external

+ system, is lost by any outcome: done, aborted, rolled back, or stuck. In

+ particular a rollback carries back whatever the target wrote (codex 2).

+- **INV-3 Convergence without a human.** A crash of any actor (a console

+ or agent process restart, or a box reboot) at any step leaves a state

+ from which the move reaches a terminal outcome with no human action. The

+ one exception is stated, not hidden: a box that **never returns**

+ (destroyed) during a handoff it holds; that leaves a paged, recoverable

+ state and an operator verb (§7.10).

+- **INV-4 Files match the database.** Every start of the instance, on any

+ box, uses a files directory whose generation equals the generation of the

+ database it starts with. A mismatch refuses the start (§5.1).

+Supporting invariants:

+- **INV-5 No admission before commit.** A row leaves a hold only by a

+ conditional write naming the exact operation that holds it, and the

+ target's hold is released only after the commit (codex 3, 6).

+- **INV-6 The environment that runs is the environment that was checked.**

+ The readiness checks evaluate the final merged launch environment, and

+ the probe binds health to the spawned process (codex 14).

+- **INV-7 Install is never replayed over writes.** Once a probe has started

+ a process on an installed generation, nothing re-copies, re-extracts or

+ re-imports into it (codex 5).

+- **INV-8 Phase 1 deletes nothing a recovery might need.** Data a move

+ supersedes is parked under its move id, never deleted (codex 8).

+- **INV-9 Bounded.** Every non-terminal state has a deadline after which it

+ either advances, recovers, or pages in a state that still satisfies

+ INV-1 and INV-2.

+## 5. The handoff protocol

+The two primitives this spec owns, and that a later `carlos db push` builds

+on: the generation handoff and the execution fence.

+### 5.1 Generation markers

+A generation is named in three places. None is a record field, so no old

+writer can erase one (#199 rev 3's lesson).

+|---|---|---|---|

+| `generation.json` `{"gen","incarnation","host","installed_at","box"}` | `instances/<host>/generation.json` in the box's own data bucket, beside the existing `lock.json` | the installing box | last step of an install (§7.6, I3), after every object is verified |

+Absent everywhere means **origin**, so every instance that has never moved

+is unchanged.

+**The start check (INV-4).** Before every spawn (`Start`, `Seed`, and the

+probe), the activator compares `<dbPath>.gen` with `.files-gen`, when the

+files directory exists. Unequal refuses the wake with a member-safe reason

+("this instance's files and database come from different copies; an

+operator has been paged") and pages. A cold restore writes `<dbPath>.gen`

+from the restored prefix's `generation.json` before the comparison.

+Where the database is warm on a box but its generation differs from the

+bucket's `generation.json`, the warm file is from an older copy (an earlier

+stay on this box). The activator parks it (§6.3) and restores cold. This

+replaces "staging wins" (`backend_exec.go:1581-1588`): staging is never

+restored from directly (M15), and `promoteStagedReplica` is deleted.

+### 5.2 Holds

+A hold is a registry row status that the edge does not serve

+(`internal/edge/edge.go:595` serves only `ready`), that ticks refuse

+(`activator.go:2785-2789` refuses any non-`ready` row), and that no

+ordinary wake reaches. Phase 1 has three hold statuses: `moving` (source,

+from the cut until the move ends), `landing` (target, until it opens), and

+`restoring` (source, during a rollback).

+**Every hold names its operation.** The `routes` table gains an additive

+column, `hold_op`, set to the move id (or rollback id) whenever a hold

+status is written. Leaving a hold is one conditional statement:

+```sql

+UPDATE routes SET status='ready', hold_op='' WHERE host=? AND hold_op=?

```

+A delayed or duplicate open from a finished operation matches nothing, so

+it can never reopen a newer operation's hold (codex 6). Reconcile, which

+today mints every row `ready` (`internal/provision/reconcile.go:364-376`),

+mints a row whose record is mid-move in the hold its role requires, with

+`hold_op` set (§9.6). A row for a record in `rolling_back` or `stuck` is

+never minted `ready` (codex 3).

+### 5.3 The execution right and stop evidence

+The right to run the instance belongs to exactly one box, named by the move

+state:

+| move state | holder |

+|---|---|

+| no move, `preparing`, `ready`, `cutting` (every phase), `reverting` | source. After `stopping` it holds the right without using it, so a pre-flip recovery needs nobody else's evidence |

+| `landing`, `committed` | target. The flip requires the source's stop evidence (§7.4), and the target reads and checks it before it starts anything (§7.6) |

+| `rolling_back` | nobody until the target's stop evidence; then source |

+**Stop evidence** is what the activator's `FenceAndStop` already produces

+(`internal/activator/maintenance.go:50`, `StopEvidence.Matches`): the main

+process stopped and exited, its sidecar stopped, its replica closed, its

+lease released. The box that gives up the right writes it to the

+deployment bucket at

+`moves/<acct>/<app>/<host>/<move id>/stopped-<role>.json` with its box

+label, the incarnation, the database file's sha256 after a final

+checkpoint, the files digest and the timers digest (§7.4, §7.8). The box

+that takes the right reads it and checks the move id, incarnation and role

+before it starts anything.

+**Why this survives a crash.** The agent runs as `carlos-edge.service`

+with systemd's default `KillMode=control-group` (no `KillMode` line in

+`internal/host/units/carlos-edge.service`), so a flat exec child dies with

+the agent. A nested child runs in its own `carlos-instance@` unit

+(`internal/host/units/carlos-instance@.service`) and survives, so a

+restarted agent stops it through the instance-control channel before it

+writes evidence. Either way a restarted box can produce evidence. A box

+that never comes back cannot, and that is INV-3's stated exception

+(§7.10).

+**What does not count as evidence:** a deadline, a lost CAS, a silent box,

+or a lease in another bucket. Leases are per bucket and prefix

+(`backend_exec.go:228-231`), so the source's and target's leases never

+exclude each other (codex 1).

+## 6. State inventory

+### 6.1 Files on a box

|---|---|---|---|

+| F4 | **files directory** | flat: `strings.TrimSuffix(dbPath, ".db")` = `<DataDir>/<host>/` (the app's `instance.key`, `report.token`); nested: every entry in `InstanceDir` except F1 to F3, F5, F6, F8, F9, F10 and #199's markers | **carried** in the archive (§8) |

+Anything else an app writes under `/data` is not carried (M4). `docs/site`

+says so.

+### 6.2 Registry state (codex 16)

-New fields on `MoveState`, all omitempty:

+Every `routes` column (`internal/registry/registry.go:34`), classified:

+| column(s) | class | Phase 1 |

+|---|---|---|

+| Host, Slug, Kind, App, Account, Region, Channel, Environment, Hibernate, StatusNamed, Steering, Gate, Seed | reproduced | from the record by `routeFor` (`reconcile.go:364-376`) |

+| Socket, DBPath | reproduced | derived on the target for its own layout |

+| Provisioned | reproduced | true on mint |

+| Status | protocol | the hold statuses of §5.2 |

+| hold_op (new) | protocol | §5.2 |

+| **Replication** | **carried** | box-local, absent from records, preserved by every path today (`registry.go:138-145`, `reconcile.go:1100-1113`); read from the source row at the cut, written to the target row at install |

+| **NextDue** | **carried** | the app's own next scheduled wake, probed at stop (`activator.go:2019-2030`); read from the source row after its stop, written to the target row at install, so the due-wake sweep still wakes it |

+| Version, PrevVersion | reset | the target materializes the channel head itself (§9.2) |

+| LastError, DNSStatus, DNSCheckedAt, CertExpiresAt, CreatedAt | reset | recomputed by their own passes |

+| Caps | **refused** if non-empty | box-local authority; not carried implicitly |

+| Unit, Backing (unit), Platform, Tunnel, FleetBox, Pool, Canary | **refused** if set | not the platform's to move |

+| Addr, Addrs, Health | **refused** if set | addr-backed |

+| Replica, Replicas | **refused** | a replica, or a home with replicas (Q2) |

+| IdleSeconds, MaxRunSeconds, Args | n/a | sidecar columns |

+The `timers` table (`registry.go:565-579`) is carried as a snapshot (§7.4,

+§9.4).

+### 6.3 Parking on a box and in a bucket

+Phase 1 reuses #199's two park mechanisms with a move-scoped destination:

+- **Bucket park** of a prefix: per-key copy into the destination, then a

+ manifest of every key and its size, then delete the source keys named in

+ the manifest (#199 §3.3). Never `MirrorPrefix`, which deletes

+ destination-only objects (`internal/lease/mirror.go:222`).

+- **Local park:** `rename` into `<DataDir>/.parked/<name>/<host>/` (#199

+ §3.4).

+Destinations: `superseded/<move id>/<host>/` in the bucket and

+`.parked/superseded-<move id>/<host>/` on the box. A move id is unique, so

+two parks can never collide, and no park destination is the ancestor of a

+live prefix (codex 8).

+## 7. The move

+### 7.1 Request and refusals

+`POST /api/cli/apps/{account}/{app}/instances/{host}/move`

+`{"region":"uk","cut":"auto"|"manual"}`, operator-only in Phase 1 (§12).

+Refused (400 or 409, with the sentence), at the door and again by every box

+step, since records have other writers:

+- not exec-backed `kind:"instance"`, addr-backed, pooled, fleet-placed,

+ non-hibernating (`internal/console/apiinstancemove.go:53-72`);

+- a replica (`:187-190`, `provision.go:978-980`), or **a home with

+ replicas** (new; neither the door nor `MoveRequest` checks `Replicas`

+ today);

+- a target of home (`apiinstancemove.go:203-209`; F2), or the current

+ region, or one outside the manifest's `regions` allow-list

+ (`apiinstances.go:1797`; also how an app that needs out-of-band

+ host-role grants is pinned to its region);

+- a legacy `<host>-sidecar.env` on the source, or any refused registry

+ column (§6.2);

+- **a private store** bound to the instance's environment (#412): its

+ cutover needs a joint transaction (codex 12; F4). An **app-level store**

+ stays where it is and the moved instance reads it across regions; T4

+ proves it can;

+- a move already in flight, or a host tombstone for this host in either

+ region that is not yet `parked` (§7.12);

+- any box in the source or target region below the protocol floor (§11).

+`MoveRequest` (`provision.go:966-983`) writes `move` with the fields of

+§7.2 and state `preparing`.

+### 7.2 Record fields

+All additive under the existing `move` object (`provision.go:907-917`).

+Old binaries are kept away by the protocol floor (§11).

```json

"move": {

- "id": "mv_01J9…", // console-minted; the generation nonce, the log, the ledger

- "from": "", // record region before the move ("" = home)

- "from_db": {"gen": "g3-…", "root": "instances"}, // the source generation; null = legacy

- "to_gen": "g4-mv_01J9…", // the move generation (#199 §4)

- "to": "uk",

- "state": "landing",

- "cut": "auto", // "" = manual cut door (today's behaviour)

- "requested_at": "…", "ready_at": "…", "updated_at": "…",

- "cut_at": "…", // source stamps when it sets the route "moving"

- "flipped_at": "…",

- "landing_deadline": "…", // flipped_at + 10 min

- "files": {"digest": "sha256:…", "bytes": 18234, "entries": 4},

- "reason": "UK could not start the instance: …" // rollback, refusal or stuck

+ "id": "mv_…", "incarnation": "…", "from": "", "to": "uk",

+ "state": "landing", "cut": "auto",

+ "requested_by": "operator:paul@…", "requested_at": "…", "updated_at": "…",

+ "ready_at": "…",

+ "cut_phase": "flipped", "cut_at": "…",

+ "source": {"replication": "stream", "next_due": 0,

+ "db_sha256": "…", "files": {"plain": "sha256:…", "sealed": "sha256:…", "bytes": 1824, "entries": 4},

+ "timers_sha256": "…"},

+ "flipped_at": "…", "landing_deadline": "…",

+ "landing_phase": "probing",

+ "landing": {"installed_db_sha256": "…", "probe_attempts": 1},

+ "committed_at": "…", "opened_at": "…", "source_parked_at": "…",

+ "rollback": {"id": "rb_…", "requested_at": "…", "reason": "…", "dirty": true,

+ "phase": "returned", "release_deadline": "…"},

+ "reason": "…"

}

```

-`MoveRequest` (`provision.go:966-983`) records `from`, `from_db` (the

-record's `DB` at that moment, or null for a legacy record) and allocates

-`to_gen = g<seq+1>-<move id>`, where `seq` is the source generation's

-sequence (0 for a legacy or pointerless record, so a legacy record's first

-move lands at `g1-<move id>`, #199 §4). The move id is the nonce, so no two

-moves, and no move and push, can share a location.

-`MoveComplete` (`provision.go:1056-1065`) changes in two ways. In one CAS

-it writes `region = to`, `DB = {Gen: to_gen, Root: <I-1>}` and

-`state = landing`, instead of clearing `move`. The target clears `move` at

-its commit. Region and generation flip together in exactly one write

-(#199 §4), and the source has nothing to do after it. This is the 09-24

-record's §6.1 `landing`, with #199's pointer added.

-Consoles roll first (§21). P0's allowlist in `MoveAbort`

-(`provision.go:1027-1035`) already refuses any state it does not know, so a

-console from v0.22 cannot abort a `landing`, `rolling_back` or `stuck`

-record.

-### 7.2 Preparing

-The replica rounds are main's delta rounds of `lease.MirrorPrefix`

-(`movespass.go:372-394`) with #199's keys: from the source storage key

-(`StorageKey(rec)`, #199 §2.3) into the target's

-`moves/<host>/<incarnation>/<to_gen>/`, with receipts at the source-keyed

-receipt path (S4). Two additions run in the same pass:

-1. **Files-archive rounds (source).** Each round, the source builds the

- instance files archive (§8) from the live directory, and uploads it only

- when its digest has changed since the last upload. It goes to the

- target's data bucket at `moves/_files/<host>/<incarnation>/<to_gen>.archive`

- with a manifest at `….json`. The `_` makes the key impossible to

- collide with a host (no valid host contains one, `provision.go:1252-1258`),

- and it sits inside the existing push-only `moves/*` grant

- (`infra/modules/region-host/main.tf:217-227`). No new IAM.

- This archive is a rehearsal copy taken from a running instance, so it

- may be inconsistent. It proves the path and the key, not the content.

- The final archive is taken at the cut.

-2. **Readiness reports (target).** §7.3.

-`ready` keeps today's rule (delta at most the bar) and gains one: the

-newest target report for this move id passes and is under 10 minutes old.

-### 7.3 Target readiness (the pre-flip check)

-The target box gains a move-worker role. Its pass already lists every

-record (`movespass.go:242`). For each record whose `move.to` is this box's

-placement and whose state is `preparing` or `ready`, it runs the checks

-below and writes the result to the deployment bucket at

-`moves/<acct>/<app>/<host>/<move-id>/target.json`. Every check runs while

-the source keeps serving.

-| check | passes when | catches |

+Every box-side transition is a CAS guarded by `move.id` and

+`move.incarnation`, so a recreated host's record (a new incarnation) never

+matches an old move's step (§7.12).

+### 7.3 Preparing and readiness

+**Replica rounds (source)**, as on main (`movespass.go:372-394`), into the

+target's `moves/<host>/<move id>/`, with receipts at

+`move-receipts/<host>/<move id>.json` in the source bucket. Keying staging

+and receipts by move id is the generation handoff's bucket half: a stale

+staging prefix or receipt from another move is never listed.

+**Snapshot rounds (source).** Each round the source builds the files

+archive (§8) from the live directory and uploads it when its plaintext

+digest changes, and publishes its own **launch report** to

+`moves/<acct>/<app>/<host>/<move id>/source.json`: the names of every

+variable in its final merged launch environment

+(`ExecBackend.instanceEnv`, `backend_exec.go:1135-1153`, plus the agent-owned

+variables), each with an HMAC-SHA256 of its value under a key derived (HKDF)

+from `CARLOS_SECRETS_KEY`. The report reveals names and equality, never

+values, and only a box holding the key can compute a matching digest.

+**Readiness report (target).** The target box's move worker runs these

+checks for every record whose `move.to` is its placement, while the source

+keeps serving, and writes the result to

+`moves/<acct>/<app>/<host>/<move id>/target.json`:

+| check | passes when | finding |

|---|---|---|

-| T1 capability | this box runs a build that understands `landing` and the files archive, and its edges file publishes `moves_schema >= 2` | an old target minting a `ready` route with no checks: today's failure mode |

-| T2 secrets key | this box holds `CARLOS_SECRETS_KEY` and its public fingerprint equals the console's | §1.2b, every variant |

-| T3 config render | a dry run of `SyncExec`'s resolution for this host's (account, app, environment) plus the instance overlay decrypts every sealed value; the report lists the variable **names** and the bundle etags, never a value | a bundle the box cannot open, a missing environment |

-| T4 store reach | for every store bound in that render (`<PREFIX>_BUCKET`, `_REGION`, `_ENDPOINT`, `_ACCESS_KEY`, `_SECRET_KEY`, `internal/objstore/objstore.go:214-218`), a `HeadBucket` with the bound credential from this box succeeds | "File storage is unavailable" |

-| T5 files archive | the newest staged archive opens with this box's key and its digest matches its manifest | the key path for S5 and S6 |

-| T6 disk | free space on the data volume (`readDiskFrac`, `internal/host/probes.go:209`) covers twice the replica size, plus the archive, plus 2 GiB, and stays under 85% full afterwards | the 91% Ireland disk of 2026-10-01 |

-| T7 release | the instance's channel head materializes on this box (the check a wake would make; "no adoptable head", `internal/activator/logsink.go:64`) | a channel with nothing promoted for this box's architecture |

-| T8 one owner | exactly one hosting box publishes the target placement | a second same-region box minting the route (the `OwnsRoute` reasoning at `movespass.go:180-185`, from the other side) |

-The console's cut door refuses with the failing check's sentence (409)

-unless the newest report passes. The source re-reads the report as the

-first step of every cut pass and refuses before it stops anything

-(`cutting` back to `ready`, with `reason`), because records have other

-writers. An auto-cut (§16) is the source taking this same step itself.

-### 7.4 The cut (source)

-#331's ordering (`movespass.go:43-51`), with three steps added (marked

-NEW):

-1. NEW: re-read the target report; refuse the cut if it no longer passes.

-2. Route status `moving`; stamp `cut_at` once (`:418`).

-3. Wake fence (`:424-431`); wait until `Quiesced` (`:432`).

-4. Wait out a layout journal (`:447`).

-5. Final checkpoint into the source generation (`:470`).

-6. Byte-final mirror round into the move generation (`:476-490`).

-7. NEW: **final files archive**: build it from the stopped instance's

- directory, upload it, record its digest. Export the host's `timers` rows

- into it (§8.1).

-8. Local DB purge (`:492`), with #199's error-returning purge, which

- removes `.lineage` first and fails the pass on any error other than

- not-exist (#199 §5.6). **The instance files and `<host>.env` are not

- touched.** They are the rollback's material until the hold ends.

-9. NEW in content: the flip writes `region = to`, `DB = {to_gen, Root}`,

- `state = landing`, `flipped_at`, `landing_deadline` and `files` in one

- CAS, guarded by the move id and `from_db`.

-Every step stays idempotent, and a box that dies mid-cut resumes at the

-top. The archive build is deterministic (§8.1), so a resumed cut re-uploads

-a byte-identical object.

-### 7.5 Landing (target)

-The target's reconcile mints the route when the record's region names it,

-as today, with #199's `incarnation`, `storage_key` and `storage_gen`

-columns rendered from the record (#199 §2.4), and with one change: a

-record in `landing` is minted with status `landing` instead of `ready`

-(`reconcile.go:374`). That status is the durable fence. It survives an agent restart, which the in-memory activator

-fence does not (`movespass.go:294-300`):

-- the edge serves only `ready` rows (`internal/edge/edge.go:595`), so no

- visitor can wake it;

-- ticks refuse any non-`ready` row (`internal/activator/activator.go:2785-2789`).

-The target's move worker then runs, in order, each step idempotent:

-| step | action | on failure |

+| T1 protocol | this box runs at or above the protocol floor (§11) | codex 7 |

+| T2 secrets key | this box publishes `secrets_key_fp` equal to the console's (§10) | §1.2b |

+| T3 environment | the target renders the bundle for the instance's environment and overlay through the held-render API (§9.1), opens the newest archive, merges exactly as `instanceEnv` will at spawn (`<host>.env` over the rendered config, `backend_exec.go:1231-1246`), adds the agent-owned variables, and every name in the source's launch report is present with an equal value digest, except `CARLOS_REGION`. A variable from `<host>.env` that shadows a rendered one is listed in the report | codex 14 |

+| T4 storage | for each store bound in that **merged** environment (`<PREFIX>_BUCKET/_REGION/_ENDPOINT/_ACCESS_KEY/_SECRET_KEY`, `internal/objstore/objstore.go:214-218`), using the merged credentials: list one key, put, get and delete a probe object at `.carlos-move-probe/<move id>` | codex 14 |

+| T5 archive | the newest archive opens with this box's key, its plaintext digest matches its manifest, and it holds no special file and is under the cap (M5) | |

+| T6 disk | free space (`readDiskFrac`, `internal/host/probes.go:209`) covers twice the replica, plus the archive, plus 2 GiB, and stays under 85% full | |

+| T7 release | the channel head materializes for the held row (§9.2) | codex 13 |

+| T8 one owner | exactly one hosting box publishes the target placement | |

+| T9 no tombstone | no host tombstone for this host is `pending` in this region (§7.12) | |

+| T10 return lane | this box can put and delete `moves/_lane/<move id>` in the **source's** data bucket, so a dirty rollback can carry its writes back (§7.8) | codex 2 |

+| T11 duration | the measured restore time for this replica size, plus the source's measured stop time, is under the operator's window (default 120 s) (M11) | |

+**Source readiness (codex 14).** The source proves the probe contract on

+the instance as it runs today: a probe request (§7.7) to its readiness path

+answers 200 to 399 from the instance's own process. An app that fails this

+on its home box is refused with "declare `moves.ready_path`" rather than

+moved and rolled back.

+`ready` requires a round at most the 16 MiB bar **and** a passing target

+report and source probe, each under 10 minutes old.

+### 7.4 The cut, journaled (codex 4)

+The cut is a sequence of phases. **Each phase is written to the record

+(`cut_phase`) before its side effects start**, so a restart resumes

+exactly, and a failure has a defined recovery.

+|---|---|---|---|

+| `admitted` | source CAS `requested` → `admitted`, stamping `cut_at`, **only if** the target report and source probe are still fresh | none yet | not fresh: CAS back to `ready` with `reason`. Abort is legal until `admitted` and refused after it, and both are CASes on the same field, so exactly one wins |

+| `stopped` | source | final checkpoint into its own `instances/<host>`; read `Replication` and `NextDue` from its row; write `stopped-source.json` (§5.3); snapshot the `timers` rows into `move-snapshots/<host>/<move id>/timers.json` in its own bucket | |

+| `archived` | source | build the final archive from the stopped directory, seal it, write it **create-only** to its own bucket, record both digests in `source.files` (§8.3); then copy it and the timers snapshot into the target's staging | over the cap, a special file, or 10 min of upload failures: `reverting` |

+The source does **not** purge its local database or files at the cut (rev 1

+did, `movespass.go:492`). They stay, held, until finalization parks them

+(§7.9), so a rollback restarts from them warm, and nothing on the source

+can be read after a completed move because the park removes it from every

+path a wake looks at.

+**`reverting` (pre-flip recovery).** The record still names the source, and

+the source holds the execution right. The source keeps the row held as

+`restoring`, checks its files against `source.files.plain` (unchanged, since

+nothing ran), runs the probe (§7.7), and on success opens with the

+conditional write and CASes to `aborting` with the reason. On probe failure

+it retries for 15 minutes, then `stuck` (paged, still held). Staging is

+then deleted by the abort arm.

+### 7.5 The flip

+One CAS, guarded by move id, incarnation and `cut_phase = archived`.

+Execution right: the source has written stop evidence, so the right passes

+to the target as soon as it reads it. The source row stays held (`moving`):

+reconcile no longer removes a row whose record is mid-move with this box as

+`from` (§9.6), so its timers, `Replication` and local files remain for a

+rollback.

+### 7.6 Landing and install (codex 5, 13)

+The target's reconcile mints the row `landing` with `hold_op = move id`

+(§9.6), unless a pending tombstone for the host is in this region (§7.12).

+The target's move worker then runs the landing phases, each written to

+`landing_phase` before its side effects:

+| phase | action | replay rule |

|---|---|---|

-| L1 | fetch the archive at `moves/_files/<host>/<incarnation>/<to_gen>`; verify it against `move.files.digest`; open it | roll back: `files archive missing or damaged` |

-| L2 | move any existing files directory aside to `<DataDir>/.retired/<incarnation>-<old .files-gen or "unknown">/` (#199 §2.5b's convention, never a delete); extract fresh (§8.3); write `.incarnation` and `.files-gen = to_gen`; write `<host>.env`; import the timers | roll back |

-| L3 | run `SyncExec` for this host; require a config ack from this box for this host written after `flipped_at` (`configack.go:116`), whenever the app has a bundle; require every name in T3's list to be in the file | roll back: `configuration did not render on <target>` |

-| L4 | repeat T4 with the rendered file | roll back: `<target> cannot reach the app's storage` |

-| L5 | **probe wake**: a new activator entry, `ProbeWake(host, moveID)`, legal only on a `landing` row. It restores cold from the row's `storage_key`, which the record's pointer names (#199 §2.4; no `.lineage` exists yet, so no warm file can be used), runs `CheckDB` (`backend_exec.go:313`), spawns, and requires the existing health answer (an HTTP response under 500 on the socket within 30 s, `backend_exec.go:36`, `:1426-1444`). Then it waits a 20 s settle window and asks again. Up to three attempts. Whether the restore reads the staging root directly or a copy under `instances/` is I-1 (§26) | roll back: the member-safe wake reason (`logsink.go:49`) |

-| L6 | **commit**: CAS `landing` → at rest (clear `move`), conditional on the record still saying `landing` with this move id | lost CAS: never open; follow the record (it was rolled back) |

-| L7 | **open**: route status `ready`; bump the serial; log `done` with the timings | retried every pass until true |

-INV-2 lives in L6 before L7. A target that crashes between them finds a

-record at rest in its region and a `landing` row on restart, and finishes

-L7. Nothing else writes route status, so this repair is the worker's job

-(#175's P0 notes: "nothing writes a `moving` row back to `ready`").

-**What visitors see.** From the source's `moving` until L7, the host has no

-`ready` row anywhere. A host under a platform apps domain gets the 503

-pending page (#174, `internal/edge/pending.go`). A custom domain gets 404

-until P2's holding route lands (Phase 2). That is the window M2 calls

-"tens of seconds of 503". §20.4 measures it on an ips-shaped instance.

-### 7.6 Automatic rollback

-A failed landing check, or a passed `landing_deadline`, rolls the move

-back. This is the M2 promise and it needs no staging lane, so it works when

-the source is the home region (kpmg's case) without the home placement of

-§9.

-**Who writes it.**

-- The target, on any L1 to L5 failure: first stop its child and purge its

- local DB (error-returning, `.lineage` first) and config, and move its

- freshly extracted files aside; then CAS `landing` → `rolling_back`,

- setting `region = move.from`, `DB = move.from_db` and `reason`.

-- The source, when `now > landing_deadline` and the record still says

- `landing`: the same CAS. The target may be dead or wedged.

-Both are CASes from `landing`, guarded by the move id and

-`DB.Gen == to_gen`, so exactly one of commit (L6) and rollback wins. A

-target that loses to the source's rollback never opened (INV-2), so it

-holds no acknowledged write (INV-3). Region and pointer go back together

-(INV-4b): the rollback is the flip, reversed, in one write. This is why

-rollback needs neither a staging lane into the source nor any copy.

-**What the source does** in `rolling_back` (its placement matches the

-record region again):

-1. If its `moving` row still exists: set it `ready` and lift its fence.

- Otherwise reconcile re-mints the route. Either way the row ends up

- `ready`, with the source storage key. If the source row had been

- retired and re-minted, #199's retire-on-identity-change (§2.6 there)

- does not fire, because the incarnation and storage key are the ones the

- row was minted with.

-2. **Files check.** The files directory and `<host>.env` are where the cut

- left them (§7.4 step 8). The source re-hashes them against the

- archive's per-file manifest (§8.2), which it wrote at the cut. A

- mismatch means something changed the directory after the cut: go to

- `stuck`, never start (INV-5). The config file is re-rendered by the

- same pass.

-3. Probe wake. The restore comes from the source generation, which equals

- the final staged copy (INV-4). The cut removed the local database and

- its `.lineage`, so the warm-cache path cannot win.

-4. Probe passes: CAS `rolling_back` → at rest; log `rolled_back` with the

- target's reason; page as a warning.

-5. Probe fails three times: CAS → `stuck`; page as an incident.

-The source admits traffic as soon as its row is `ready`. That is safe: the

-source copy is the authoritative one in this state.

-**What the target leaves** once the record has left its scope: reconcile

-retires the `landing` row (its incarnation is unchanged but the record no

-longer names this region). The move generation

-`moves/<host>/<incarnation>/<to_gen>/` and its archive are **unreferenced**:

-nothing points at them, and no later move can reuse the key, because the

-move id is in it. They are garbage, not a hazard (#199 §4), so residency

-cleanup deletes them on the next pass rather than as part of the

-rollback. A denied delete (`movespass.go:509-512`) is logged as residue.

-The target's moved-aside files directory goes the same way.

-**`stuck`** keeps `region = from` and the full `move` object as evidence.

-New moves are refused. An operator resolves it with

-`carlos move resolve --host <h>` after fixing the box (§16). The verb

-re-runs the source probe, and on success clears the move. It never touches

-data. This is the 09-24 record's M2 recovery path.

-**Side effects.** An app that does work at boot (sends mail, calls a

-webhook) will have done it on the target during L5, and will do it again on

-the source after a rollback. `docs/site` says so. Apps that care can read

-`CARLOS_MOVE_PROBE=1`, which the probe wake sets, and skip boot-time side

-effects.

-### 7.7 Abort

-Legal in `preparing`, `ready`, and in `cutting` before the source has set

-the route `moving` (`cut_at` unset). The source's abort arm

-(`movespass.go:396-413`) deletes the move generation and its archive

-before it clears the move. Under v2 that delete is residency hygiene, not

-safety (an unreferenced generation is never read), so a failed delete no

-longer has to hold the abort open; it is logged as residue. For a legacy

-record the abort keeps today's two-phase rule, because legacy staging

-still wins promotion (§7.9). Abort after `cut_at` is refused. Past that point the move

-either completes or rolls back on its own, and both are faster than an

-abort would be.

-The 09-24 record's "abort until the purge" (cutting before the purge) is

-not adopted. Automatic rollback covers the same ground with one mechanism

-instead of two.

-### 7.8 Crash at any step

-| actor crashes in | on restart |

-|---|---|

-| source, `preparing`/`ready` | next round resumes; receipts make it cheap |

-| source, `cutting` before the flip | resumes at step 1 (#331); re-uploads a byte-identical archive |

-| source, after the flip | nothing to do until `rolling_back` or the deadline |

-| target, before L6 | re-runs L1 onwards; extraction and import are idempotent; the probe counts attempts per process, and the deadline bounds the total |

-| target, between L6 and L7 | sees record at rest plus a `landing` row; does L7 |

-| target, during its rollback | record already `rolling_back`: finishes cleanup; or not yet written: re-runs the failed check, fails again, writes it |

-| source, during `rolling_back` | re-runs §7.6 steps 1 to 5 |

-| console, any state | stateless for instance moves; the ledger row is re-read |

-| both boxes dead | deadline passes with no writer; the console's move sweep pages at deadline + 5 min, shows `landing (overdue)`, and offers `carlos move resolve` |

-### 7.9 Legacy records

-A record is legacy under #199 §2.3 (no `DB` pointer, created before the v2

-epoch). Two cases.

-**After the epoch exists**, a move is the record's first v2 event (#199

-§4). The source reads its legacy location, `instances/<host>`, and keeps

-the legacy receipt key, `move-receipts/<host>.json`. Everything it writes

-in the target is v2: the move generation is `g1-<move id>`, staged at

-`moves/<host>/<incarnation>/g1-<move id>/`, with the archive beside it.

-The flip writes the record's first `DB` pointer. `from_db` is null, and a

-rollback writes `DB = null` back, which #199 reads as the legacy location

-again, because CreatedAt is still before the epoch. **The incarnation of a

-legacy record whose CreatedAt is zero** is whatever

-`provision.Incarnation` returns for it, as long as source and target

-compute it identically. §26 I-5 asks the #199 spec to pin that.

-**Before the epoch exists**, every record is legacy and #199 writes no

-pointer. The move then keeps main's storage flow exactly: staging at

-`moves/<host>/`, the target promotes it over `instances/<host>` on first

-wake ("staging wins", `backend_exec.go:1581-1588`), and receipts at

-`move-receipts/<host>.json`. Everything else in this spec still applies:

-the archive (at `moves/_files/<host>/<move id>.*`), readiness, `landing`,

-the probe, and rollback. Rollback is still a region flip, since the source

-location never changed. What the legacy flow cannot do is make the

-target's promoted copy unreferenced: after a rollback the target bucket

-holds a full `instances/<host>` lineage that a later restore on that box

-would trust. So, pre-epoch, the target deletes it during the rollback,

-before writing `rolling_back`. If the delete is denied, the rollback goes

-ahead anyway (the instance must not stay offline over cleanup), and the

-residue is logged and shown to operators as a hazard, not as garbage. A

-later move to that box is still safe, because its fresh staging wins

-promotion. What is not safe is anyone pointing the record at that region

-by hand. §23 Q11 asks whether to support this flow at all, or to refuse

-moves until the epoch exists.

-## 8. The instance files archive

-### 8.1 Format

-- A tar stream of the instance files (§5, S5) plus `<host>.env` (S6) plus

- `timers.json` (S11, S12), in sorted path order, with mtimes and owners

- zeroed and modes kept. The output is byte-deterministic for one input.

-- Only regular files and directories. A symlink, device, FIFO or socket in

- the directory refuses the move with its name. The nested layout already

- refuses special files on tenant ground (34beb1b3), and the archive holds

- the same line on both layouts.

-- Size cap: **256 MiB** before compression (§23 Q2). Larger refuses the

- move and points at object storage.

-- zstd, then sealed.

+| `installing` | read `stopped-source.json` (evidence check); bucket-park whatever is at `instances/<host>/` in its own bucket into `superseded/<move id>/<host>/` (an older stay); per-key copy `moves/<host>/<move id>/` into `instances/<host>/`, verifying each size; write `instances/<host>/generation.json` with `gen = move id` last | replayable until `generation.json` exists: the staging is immutable after the flip and the copy is per-key with no destination deletes |

+| `installed` | local-park any warm DB and any existing files directory into `.parked/superseded-<move id>/`; open the archive, verify the plaintext digest, extract; write `.files-gen = move id` last by rename; write `<host>.env`; replace the host's timers with the snapshot (§9.4); set `Replication` and `NextDue` on the row | replayable until `.files-gen` exists |

+| `configured` | render through the held-render API (§9.1) and verify the move-bound ack | replayable |

+| `restored` | restore the database cold from `instances/<host>` (now generation `<move id>`), write `<dbPath>.gen`, `CheckDB` (`backend_exec.go:313`); record `landing.installed_db_sha256` | replayable until `probing` |

+| `probing` | the probe (§7.7) | **never replays an earlier phase** (INV-7). A crash here restarts only the probe, on the same installed files and database, which may now hold target writes |

+| `probed` | probe passed | |

+Any failure in `installing` to `probing` triggers a rollback (§7.8). A

+landing past `landing_deadline` triggers one too.

+### 7.7 The probe (codex 14)

+A new activator entry, `ProbeWake(host, op)`, legal only on a row held with

+`hold_op = op`:

+- spawns the instance through the normal path, with `CARLOS_MOVE_PROBE=1`

+ set (advisory; INV-2 does not rely on it);

+- **binds the answer to the process:** the socket's peer credentials

+ (`SO_PEERCRED`) must name a process in the spawned child's cgroup, so

+ another process squatting on the socket cannot pass;

+- requests the app's readiness path, `moves.ready_path` from the instances

+ manifest when declared, `/` otherwise, and requires 200 to 399 within 30 s

+ (the same budget as a wake, `backend_exec.go:36`);

+- waits a 20 s settle window, requires the process to still be running,

+ and asks again;

+- up to three attempts.

+The probe's child is registered as **held** in the activator, so an edge

+reload's `RetireExcept` (`internal/edge/edge.go:924`,

+`activator.go:2624`) does not retire it for being absent from the ready set,

+and the edge still never admits a request to it (codex 13).

+### 7.8 Rollback (codex 1, 2, 3)

+**Triggers:** any landing failure (the target writes it), or

+`landing_deadline` passing (the source writes it, or the console's sweep).

+Both are a CAS `landing` → `rolling_back`, guarded by move id. The commit is

+a CAS from `probed` on the same record, so exactly one of commit and

+rollback wins.

+The rollback CAS sets `region = from`, `rollback.id`, `reason` and

+`release_deadline = now + 15 min`. Nothing starts anywhere yet.

+**Target release.** The target's worker, on seeing `rolling_back`:

+1. `FenceAndStop` until stop evidence (retry; a crashed agent restarts and

+ does the same).

+2. Final checkpoint; sha256 of the database file after a `TRUNCATE`

+ checkpoint; plaintext digest of the files directory; digest of the

+ host's timers rows.

+3. **Clean or dirty.** Clean when all three equal what was installed

+ (`landing.installed_db_sha256`, `source.files.plain`,

+ `source.timers_sha256`) or when no probe ever started. Otherwise dirty.

+4. Write `stopped-target.json` with the digests and the verdict, and CAS

+ `rollback.phase = released`, `rollback.dirty`.

+5. **Dirty only:** carry the writes back. This is the handoff, reversed:

+ mirror its `instances/<host>` into the source bucket's

+ `moves/<host>/<rollback id>/` (T10 proved the lane), build and seal an

+ archive of its files directory, snapshot its timers, publish all three,

+ then CAS `rollback.phase = returned`.

+6. Hold its row until reconcile retires it (the record no longer names its

+ region), then local-park its database and files into

+ `.parked/superseded-<rollback id>/`.

+**Source recovery**, on `released` (clean) or `returned` (dirty):

+1. Its row is still held (`moving`, `hold_op = move id`, §7.5). It switches

+ it to `restoring` with `hold_op = rollback id`.

+2. Clean: its own database and files are what it stopped with. It checks

+ the files against `source.files.plain`; a mismatch means something

+ touched them, and goes to `stuck` (INV-4).

+3. Dirty: it installs generation `<rollback id>` exactly as §7.6 does

+ (park its own lineage and local files into `superseded/<rollback id>/`,

+ copy, extract, timers, config), because the target's copy is now the

+ authoritative one (INV-2).

+4. The probe (§7.7), still held.

+5. Pass: open with the conditional write on `hold_op = rollback id`; log

+ `rolled_back`; clear `move` (§7.9's last step). Page as a warning.

+6. Three failed probes: `stuck`, still held. Paged as an incident.

+The source never opens before its own probe passes, and never starts before

+the target's stop evidence (codex 1, 3).

+**Waiting for the target.** If `release_deadline` passes with no evidence,

+the move stays in `rolling_back`, shown as "waiting for uk to release", and

+pages. It is still safe (nothing runs, the source is held) and it converges

+as soon as the target box returns (INV-3). §7.10 covers a box that never

+returns.

+### 7.9 Commit and finalization (codex 6)

+1. **Commit** (target): CAS `probed` → `committed`, stamping `committed_at`.

+ The `move` object stays, so no new move and no push can start, and all

+ recovery information is kept.

+2. **Open** (target): the conditional write on `hold_op = move id`; stamp

+ `opened_at` (CAS). A restarted target finds `committed` and a held row,

+ and opens it.

+3. **Source park** (source): bucket-park its `instances/<host>/` into

+ `superseded/<move id>/<host>/`; local-park its database, files directory

+ and `<host>.env` into `.parked/superseded-<move id>/`; delete its row

+ (with `hold_op = move id` as the condition); stamp `source_parked_at`.

+ Its timers rows go with the row, and the snapshot in

+ `move-snapshots/<host>/<move id>/` remains.

+4. **Close** (whichever of the two stamps lands second): write the terminal

+ log event with every timing and digest, then CAS-clear `move`, guarded by

+ move id and both stamps.

+Until step 4 the record still says `committed`, so every crash between

+steps resumes from the record.

+### 7.10 Stuck, held, and a box that never returns

+- `stuck`: the source's probe failed on recovery. The row stays held. The

+ operator fixes the cause and runs `carlos move resolve --host <h>

+ --retry`, which reruns the source probe. Nothing is deleted.

+- `rolling_back` past `release_deadline`: as §7.8, waiting. Converges when

+ the target returns.

+- **A box destroyed mid-handoff.** Its stop evidence can never arrive.

+ `carlos move resolve --host <h> --declare-lost <box>` is the one

+ operator act in Phase 1 that may lose data: it records the operator's

+ statement that the box's processes are gone, with a confirmation that

+ names what is lost (target writes since `flipped_at`, if the lost box is

+ the target), and then proceeds as a clean rollback. §18 asks Paul to

+ confirm this.

+### 7.11 Abort

+Legal in `preparing`, `ready`, and `cutting` while `cut_phase = requested`

+(CAS). The source deletes `moves/<host>/<move id>/` and the archive, then

+clears `move`. A failed delete is logged and retried; the prefix is

+unreferenced (its key names the move id), so it is never read.

+### 7.12 Tombstones (#199 rev 3)

+- **The instance is deleted mid-move.** The delete door writes the

+ tombstone with the record's **current** region, then deletes the record.

+ Every move worker binds each step to `move.incarnation`; a record that is

+ authoritatively absent (or present with another incarnation) ends the

+ move for both boxes: each stops anything it runs (stop evidence as usual)

+ and **parks its own region's artifacts of this move into the tombstone's

+ `parked/<incarnation>/<host>/`**: the source its `instances/<host>/` (if

+ the tombstone is in the other region) and its receipts and snapshots; the

+ target its staging, `instances/<host>/` if installed, and local files. The

+ box in the tombstone's region parks as #199 §3.3 says. Phase 1 adds the

+ other region's half, because #199's park covers only its own region.

+- **Recreated mid-move.** The new record has a new incarnation and no

+ `move`. No old step matches it. #199 §3.5 defers the new row until the

+ parks are done, and the landing mint obeys the same deferral.

+- **#199's park of `moves/<host>/`.** Staging is

+ `moves/<host>/<move id>/` with a `move.json` naming the incarnation. The

+ tombstone park takes only keys whose `move.json` names the tombstoned

+ incarnation, or legacy keys with none. A live move of a newer incarnation

+ is never swept (§17, I-3). T9 also refuses to start a move into a region

+ with a pending tombstone for the host.

+### 7.13 Crash at every step

+Every row assumes the crashed actor restarts. "Resumes" means the actor

+re-reads the record and runs the phase it names.

+|---|---|---|---|

+| cutting: stopping | flat child died with the agent, nested unit stopped on restart; evidence written | n/a | |

+| cutting: stopped to archived | resumes; the sealed archive is reused from its create-only key (§8.3) | n/a | |

+| reverting | resumes the recovery, still held | n/a | |

+| landing: probing | n/a | restarts the probe only; the child died with the agent or is stopped first | |

+## 8. The archive (codex 19)

+### 8.1 Content

+A tar of the files directory (F4) and `<host>.env` (F5), in sorted path

+order, with owners and times zeroed and modes kept, so equal content gives

+equal plaintext bytes. Only regular files and directories; a symlink,

+device, FIFO or socket refuses with its name. The 256 MiB cap is enforced

+while streaming the tar, so an oversized directory stops early (M5). Then

+zstd.

### 8.2 Sealing

-The archive carries the very key that protects the app's data, so it must

-not sit in a bucket beside that data in the clear. It is sealed to the

-deployment's config keypair, the same one config bundles are sealed to:

-- a fresh 256-bit data key; AES-256-GCM over 1 MiB chunks, each chunk's

- nonce derived from its index, with the final chunk flagged so the stream

- cannot be truncated;

-- the data key sealed with ephemeral X25519 to the public half of

- `CARLOS_SECRETS_KEY` (which `appconfig.ParsePrivateKey` already parses),

- exactly as a sealed config value is;

-- the manifest (the `.json` beside the archive) holds the sealed data key,

- the sha256 of the sealed object (the `files.digest` in the record), the

- plaintext entry list (paths, sizes, sha256 of each file), the move id,

- the incarnation and `to_gen`. The target refuses a manifest whose

- incarnation or generation differs from the record's.

-- The source keeps its own copy of the manifest beside the receipts

- (`move-receipts/<host>/<incarnation>/<from_gen>.files.json`), because it

- cannot read the target's bucket and the rollback's files check (§7.6

- step 2) needs it.

-Only a box holding `CARLOS_SECRETS_KEY` can open it. T2 proves the target

-is one.

-### 8.3 Extraction on the target

-Into the path §5 defines for the target row's layout, after moving any

-existing directory aside (§7.5, L2). The target row may be flat when the

-source was nested, or the reverse. The two markers, `.incarnation` and

-`.files-gen`, are written last, by atomic rename, so a crash mid-extraction

-leaves a directory that the next pass recognises as not yet landed and

-replaces. For a nested row,

-writes go through `instancefs` (no symlink follow), as every other write on

-tenant ground does. Every extracted file is checked against the manifest's

-per-file hash. `<host>.env` is written 0600. Timers are upserted with their

-saved `next_at`, `due_at`, `last_*` and `attempts`. Once-timers that came

-due during the cut fire on the first tick after open.

-### 8.4 What it does not carry

-Any path outside §5's definition, and process state. An app that keeps

-files elsewhere under `/data` loses them on a move. `docs/site` says so

-plainly.

-### 8.5 A requirement for P9

-The isolated-exec design moves `<DataDir>/<host>.*` into the instance

-directory (isolated-exec design §2 item 1). That glob does not match the

-directory `<DataDir>/<host>/`, and the unit's `TemporaryFileSystem=/data:ro`

-hides everything outside the instance directory (§4, "load-bearing"

-notes). So the layout pass, as designed, would strand every flat app's

-`instance.key` the same way this move did. P9 must carry `<DataDir>/<host>/`

-into `InstanceDir`, and stamp it with #199's `.incarnation` when it does.

-This is recorded here because it is the same fault, and is to be filed on

-#126.

-## 9. Moves to and from home (P3)

-Automatic rollback (§7.6) does not need this section. Two things do: a

-deliberate move to home (moving back after a successful move away, or an

-app whose region is home), and any move whose source is home and whose

-target needs to stage into home (never; the source stages into the target).

-So P3 is about home as a **target**.

-Today a move to home is refused (`apiinstancemove.go:203-209`), because the

-home box publishes no placement: it runs with `CARLOS_REGION` unset, and

-records store `""` for home. `moveTargets` drops any edges file without a

-placement (`movespass.go:345`), and the home's children receive no

-`CARLOS_REGION` (`backend_exec.go:989-993`).

-The fix is the 09-24 record's §6.6, which R4 ruled on:

-1. **`CARLOS_HOME_PLACEMENT`** on the home box: the console's

- `CONSOLE_HOME_REGION` name (`ie` on carloku). The box publishes it as

- `placement` with `"home": true` in its edges file. Its own

- `CARLOS_REGION` stays unset, so record scope (`""`) and every reconcile

- that compares by equality are unchanged.

-2. **`moveTargets`** maps `move.to = ""` to the edges file marked home.

- **`runMoveRound`** looks up `targets[""]` through that map.

-3. **The target's role** (§7.3 and §7.5) matches `move.to == ""` on the box

- marked home.

-4. **Home children** get `CARLOS_REGION = CARLOS_HOME_PLACEMENT`, so an app

- sees one variable in every region. This is a behaviour change for every

- app at home, so it ships behind `CARLOS_HOME_REGION_ENV=on` on the home

- box, and the roll notes say what it changes.

-5. **The door** (`apiinstancemove.go:203-209`) accepts home once some edges

- file is marked home with staging coordinates. Until then its refusal

- says a move to home is not available yet.

-6. **Infra:** every sibling's `sibling_staging_arns` lists the home data

- bucket. That is production IAM and needs scoped approval per deployment.

-7. **Under #199**, a move to home is an ordinary v2 move: a new generation

- staged in the home bucket and pointed at by the flip. A boomerang

- (ie→uk→ie) lands in a fresh `g<n>-<move id>` and never touches the

- generation the instance left behind in ie, so the 09-24 record's

- staging-wins reasoning for this case is no longer needed. Until that

- earlier generation's hold ends, the home bucket holds both; P5 deletes

- the old one on schedule.

-**Rollback and P3 are separate.** Automatic rollback (§7.6) restores the

-source generation where it already is, in the source's own bucket, and

-works today for a move out of home. P3 is needed only to move **into**

-home deliberately: `carlos move back` after a successful move away, or an

-app move whose target is home.

-## 10. After a move: the hold, then residency cleanup (P5)

-On `done`, the target writes a cleanup task into the move log: "delete the

-source's copy of `<host>` in `<from>` after `<done + hold>`." The hold is 7

-days by default (`CARLOS_MOVE_HOLD`, §23 Q4).

-**During the hold** the source keeps:

-- the source generation (`from_db`'s location, or `instances/<host>` for

- a record that was legacy), and the source-keyed receipts and files

- manifest beside it (S4);

-- the instance files directory and `<host>.env` (box disk);

-- its quarantine file, if any (S19);

-- its config ack object (S9);

-- for an app move, the old store, write-fenced (§11).

-What the hold buys, stated without overselling it. After `done`, the target

-has taken user writes. A **move back** (`carlos move back`, §16) is the

-supported way to return, and it carries the target's data. The kept source

-copy is last-resort evidence for an operator if the target region loses

-data. It is never restored automatically (R4; the 09-24 record's M2).

-**After the hold** the source box deletes all of it and logs the count of

-objects and files removed. The 09-24 record's §6.7 fire-time re-check

-applies, and #199 makes it much simpler. A v2 deletion is keyed to one

-storage key, `<Root>/<host>/<incarnation>/<from_gen>/`, so it runs only if,

-at that moment, the record's `DB` pointer does not name that generation

-(and, for a legacy source, the record has a `DB` pointer at all). A

-boomerang can no longer make the deleted prefix live again, because a

-return lands in a new generation. The files directory is deleted only if

-its `.files-gen` names the source generation's era (it was not replaced by

-a later landing on this box) and this box owns no route for the host. Any

-failure cancels the deletion, with a log line.

-**#199's generation retention.** #199 keeps the newest five generations and

-deletes older ones after a push commits (§5.4 there). After a move, the

-earlier generations live in the source region's bucket, which the target

-box cannot list or delete. So:

-- the retention GC on a box considers only generations in its own bucket;

-- generations a move left in another region are P5's to delete, on P5's

- schedule, never the GC's;

-- `carlos db rollback --to <gen>` refuses a generation that lives in

- another region (#199 R3: push and rollback are in-region only), and

- `carlos db backups` lists such a generation as "in <region>, kept until

- <date>, not restorable here".

-**#171 guard.** A box role can forge records (§14), so a forged record must

-not be able to schedule a deletion. A cleanup task is honoured only with a

-**console-signed warrant**: the console signs `(move id, host, from, not

-before)` with its release key, which boxes already verify with

-`CARLOS_RELEASE_PUBKEY` and do not hold (`cmd/console/main.go:236-250`).

-The console signs only for a move in its own ledger whose log shows the

-target's commit.

-`carlos move status` and the console show "Ireland copy kept until 9

-October 2026" during the hold and "Ireland copy deleted on …" after it.

-## 11. Stores follow their owner (M3, P8)

-The rule that composes the two kinds of move:

-- **An app-level store follows the app.** It moves in an app move, after

- the app's instances (§12). An instance move leaves it where it is, and

- the moved instance reads it across regions (T4 proves it can).

-- **A private store (#412) follows its instance.** It moves inside that

- instance's move.

-That is M3, and it is consistent with "moving one instance does not change

-the app's region".

-**Prerequisite: a store backend per placement.** `CARLOS_STORE_REGION` is

-read once, console-wide (`cmd/console/main.go:550-560`). Each armed box

-publishes its store region and bucket prefix in its edges file, and the

-provisioner creates a twin in the target's store region. This is the 09-24

-record's §7.1, unchanged.

-**The copy, per store:**

-1. **Provision a twin** in the target store region: same entitlement, new

- bucket, new credentials (`internal/objstore/provisioner.go`).

-2. **Pre-copy** in delta rounds while everything serves, with copy receipts

- as the replica mirror uses. The copier is the **move-scoped IAM user**

- ruled in Q6: Get and List on the one source bucket, Put, Delete and

- List on the one twin, an `aws:CurrentTime` condition that ends its

- authority 24 hours after minting, and a reaper for abandoned users. The

- console mints it only for a move in its own ledger (§14).

-3. **Write fence (the 09-24 record's M1).** A bucket-policy `Deny` on

- `s3:PutObject` and `s3:DeleteObject` for every principal except the

- copier. A policy is evaluated per request, so it also stops presigned

- uploads already handed out. It is the only fence that covers every

- credential holder.

-4. **Final delta**, then **rebind**: `appconfig.ApplyEnvironment` rewrites

- the bundle's store coordinates to the twin (default bundle for an app

- store, the instance's environment for a private store). Config sync

- bounces running instances, and hibernating ones read it on wake.

-5. **Lift nothing on the old bucket.** It stays write-fenced, read-only,

- until the hold ends, then P5 deletes it.

-**Where the downtime falls.** For a private store, steps 3 and 4 run inside

-the instance's own cut, between §7.4 steps 6 and 7. The instance is already

-down, so the fence costs nothing extra. For an app store, every instance of

-the app writes it, so the final delta needs all of them held. The app move

-holds the whole app at 503 for the store cut (§23 Q6), measured and shown

-to the owner before they confirm.

-**Rollback.** Before the rebind, a failed store step lifts the fence and

-leaves the binding alone. After the rebind, the store moves forward only.

-## 12. The app move

-### 12.1 Shape

-`carlos move --app ips --to uk`, or the app's Region section in the

-console. The console runs it as a **sequencer**: a console loop over a new

-`app_moves` table in the console DB. The console owns that DB, so the

-sequencer's state survives restarts, and every step is idempotent.

-1. **Validate.** The target is valid, armed, hosting and staging-capable

- (`apiinstancemove.go:227-235`); eligible for the app (nearest-region

- §6.1); allowed by the manifest's `regions` (`apiinstances.go:1797`); and

- not the app's current region. The app is not fleet-placed (the

- placement column governs its instances' regions, nearest-region §12)

- and not pooled. The nearest-region switch is effectively on

- (`CONSOLE_APP_REGIONS=on` and a named home, nearest-region §5.6),

- because otherwise the app region cannot hold a non-home value.

-2. **Plan.** List the app's records. The instances that move are those

- whose `region` equals the app's current region and that pass every

- per-instance rail (§13). Everything else is listed with why it stays.

- The owner sees the plan before confirming.

-3. **Change the app region** (`apps.region = to`, `region_source = moved`,

- a new label beside nearest-region §5.1's values). From this moment, new

- instances created without a region start in the target (the fill,

- nearest-region §7). That is what "the app region changes, then every

- instance moves" means in order.

-4. **Move instances one at a time**, each a full instance move with

- `cut: auto`. One at a time bounds the target's load and keeps every cut

- window separate.

-5. **Move the app store** (§11), if it has one.

-6. **Done.** The app move's log is the list of its instance moves.

-### 12.2 Partial failure

-- **An instance rolls back.** That instance is back in the old region and

- serving. The sequencer **pauses** the app move (`paused`, with the

- reason). The app region stays at the target, so new instances keep going

- there. The owner chooses: `carlos move resume` (retry the failed one),

- `carlos move skip --host <h>` (leave it in the old region and continue),

- or `carlos move --app ips --to <old>` (move the app back). §23 Q5.

-- **An instance goes `stuck`.** The app move pauses and pages, as for

- `stuck` in §7.6.

-- **The store cut fails before the rebind.** The fence lifts, the store

- stays where it was, and the app move pauses at "store not moved".

-- **Abort** (`carlos move abort --app ips`) aborts the in-flight instance

- move if it is still abortable, and stops the sequencer. Instances already

- moved stay moved. The app region stays at the target. Abort is not undo.

-- **The console restarts mid-move.** The sequencer re-reads `app_moves` and

- the in-flight instance's record, and continues.

-### 12.3 Composition with the nearest-region spec

-- That spec's R5 says changing an existing app's region is a follow-up,

- "an owner-only verb that affects future creates only". The app move is

- that verb. An app with nothing to move (no instances, or all of them

- elsewhere) is a region change that only affects future creates.

-- An app move writes a non-home `apps.region`, so nearest-region §5.7's

- rollback floor applies to it unchanged.

-- **An instance move never touches `apps.region`** (nearest-region §12,

- "Moves").

-- Explicit per-instance placements survive an app move: an instance whose

- region differs from the app's is not moved, and the plan says so.

-## 13. What does not move, and why

-Each refusal is checked at the door (fast answer) and again by the box

-(records have other writers).

-| shape | answer | why |

-|---|---|---|

-| pooled record (`Pool != ""`) | refused (`apiinstancemove.go:65-68`) | its data lives on the pool's box, not a region box |

-| fleet-placed record or app | refused (`:61-64`; nearest-region §12) | fleets keep state-does-not-follow on purpose |

-| non-hibernating | refused (`:69-72`) | runs outside the activator; nothing can quiesce it |

-| addr-backed, unit-backed, static, status kinds | refused (`:53-60`) | no platform-owned process or data. Static routes are served from the deployment bucket by every edge, so an app move leaves them and has nothing to carry |

-| replica record | refused (`:187-190`, `provision.go:978-980`) | the replica role is placed, not moved (G14) |

-| **home with replicas** | **refused (new)** | Q2. Neither the door nor `MoveRequest` checks `Replicas` today, only `Role`. Phase 1 adds the rail |

-| native sidecar (kind `sidecar`) | refused (`:53-56`) | sidecars carry their own replicated DB (sidecar-members design §4) and need their own move. An app move with sidecars in the source region is refused by default (§23 Q7) |

-| legacy `<host>-sidecar.env` | **refused (new)** | hand-staged on one box; a move would silently drop the sidecar |

-| target not in the manifest's `regions` | **refused (new)** | the app said where it may run. This is also how an operator pins an app that depends on out-of-band host-role grants (S24): set `regions` to its one region |

-| target fails a readiness check (T1 to T8) | cut refused with the check's sentence | §7.3 |

-| instance files over the cap, or containing a special file | refused with the size or the name | §8.1 |

-| another move in flight for the host, or an app move for its app | refused (`provision.go:968-970`) | one at a time |

-## 14. Who may move, and the #171 residue

-- **Owners** of the account (`RoleOwner`), from a session or a person's CLI

- token. The doors return the family's byte-identical 404 to anyone else,

- as `handleAppMoveAccount` does for its owner check

- (`internal/console/appsettings.go:1258-1261`). Members are not owners.

- Today's move doors admit any member, so this narrows them.

-- **Instance-role service credentials** keep request, cut, abort and status

- on instance moves (`internal/console/authcli.go:625-628`), unchanged:

- Tito's router uses them. They cannot start an app move.

-- **Operators** (`Store.IsOperator`, `internal/console/store.go:1858`) can do

- any of it for any account through an operator route family,

- `/api/cli/ops/accounts/{sqid}/apps/{app}/…`. Non-operators get 404 there.

- Every operator act is written to the target account's audit trail.

-**The #171 residue.** Any region box's role can write any record (#171,

-open). A forged `move.to` can already move an instance today, whatever the

-door allows, so M1 does not widen that. What a forged record must not be

-able to do is drive the two privileged acts this spec adds. So the console

-keeps a **move ledger** (console DB: move id, account, app, host, from,

-to, requested by, app move id, outcome) and:

-- mints a store-copy credential (§11) only for a ledger move;

-- signs a P5 deletion warrant (§10) only for a ledger move whose log shows

- the target's commit.

-A record whose `move.id` is not in the ledger is shown in the console as

-"not started from this console", and nothing privileged follows from it.

-## 15. API

-All additive. Old clients see today's answers.

-| door | change |

-|---|---|

-| `POST …/instances/{host}/move` | body gains `"cut": "auto"`; response gains `move_id`, `from`, and `not_moved` (for example `["app store stays in Ireland with the app"]`). Owners only for people (§14) |

-| `POST …/instances/{host}/move/cut` | refused (409) unless the newest target report passes; the body names the failing check |

-| `DELETE …/instances/{host}/move` | legal until `cut_at`, then 409 "the cut has started; it completes or rolls back on its own" |

-| `GET …/instances/{host}/move` | unchanged by default (titogo depends on it, 09-24 §6.2). `?history=1` adds the newest terminal outcome for 30 days, the target report, timings and `reason` |

-| `POST …/apps/{app}/move` (new) | `{"region":"uk","dry_run":true}` returns the plan; without `dry_run` it starts the app move. People only, owners only |

-| `GET …/apps/{app}/move` (new) | sequencer state, plan, per-instance outcomes |

-| `POST …/apps/{app}/move/resume`, `…/skip`, `DELETE …/apps/{app}/move` (new) | §12.2 |

-| `POST …/instances/{host}/move/resolve` (new) | operator only; §7.6 `stuck` |

-| `/api/cli/ops/accounts/{sqid}/…` (new) | operator mirrors of the above |

-## 16. The CLI (G8)

-One top-level verb, `carlos move`. It is free in `main.go`'s dispatch, and

-`docs/site/cli.md` gains its heading in the same change (`internal/docsite`

-fails the gate otherwise).

+Sealed for the deployment's config keypair, the one config bundles use

+(P-256 ECDH, `internal/appconfig/crypto.go:44-59`):

+- a fresh 256-bit data key; AES-256-GCM over 1 MiB chunks, each nonce the

+ chunk index, the last chunk flagged so truncation is detected;

+- the data key sealed with `appconfig.Seal` (`crypto.go:86`), which uses a

+ fresh ephemeral key and nonce;

+- a manifest: the sealed data key, the sealed object's sha256, the

+ plaintext's sha256, the per-file list (path, size, sha256), the move id

+ and the incarnation.

+Sealing is randomized on purpose. Encryption is never made deterministic by

+reusing a key or nonce.

+### 8.3 Identity, publication, retention

+- **Two identities.** `plain` (the plaintext digest) says what the files

+ are. `sealed` (the ciphertext digest) names one immutable object. Checks

+ of content (T5, rollback, finalization) use `plain`. Fetches use

+ `sealed`.

+- **One sealed object per cut.** In `archived` the source writes the sealed

+ archive and manifest **create-only** to its own bucket at

+ `move-snapshots/<host>/<move id>/files-<sealed>.{sealed,json}`, then

+ records both digests in the record, then copies them to the target's

+ staging. A crash before the record write leaves an unreferenced object

+ and the next attempt builds a new one; a crash after it reuses the

+ recorded one and never rebuilds. So a replay never changes

+ `source.files.sealed` (codex 19).

+- **Retention.** The source's snapshots (archive, manifest, timers) stay in

+ `move-snapshots/<host>/<move id>/` until F3's cleanup, because a rollback

+ and an audit both need them. The target's staged copies are parked with

+ staging at finalization.

+## 9. Changes to existing code (codex 13, 15)

+### 9.1 Config render for a held row

+`SyncExec` skips any row that is not `ready` (`execRouteBase`,

+`internal/appconfig/syncexec.go:643-647`), and writes an ack only when a

+file changes (`syncexec.go:1159`). A held row needs neither behaviour. New:

+`appconfig.RenderHeld(ctx, rt, op)` renders exactly the file `SyncExec`

+would, writes it, and writes a **move-bound ack**

+`control/config-ack/<box>/<host>.json` with `op`, the file's sha256 and the

+bundle etags, whether or not the file changed. The landing reads that ack.

+`SyncExec` continues to skip held rows, so the two never race on one file;

+once the row opens, `SyncExec` owns it again.

+### 9.2 Release materialization for a held row

+`release.Adoptable` excludes non-`ready` rows (`internal/release/adopt.go:816-819`).

+New: `release.MaterializeHeld(rt, op)` stages the channel head for a held

+row without adopting traffic, used by T7 and before the probe.

+### 9.3 The activator

+- `ProbeWake` (§7.7).

+- `RetireExcept` keeps held entries (§7.7).

+- The start check (§5.1) in `Start`, `Seed` and `ProbeWake`.

+- Restore writes `<dbPath>.gen` and parks a warm file of another

+ generation (§5.1). `promoteStagedReplica` and its call in `Restore`

+ (`backend_exec.go:290`) are deleted.

+- `FenceAndStop` returns evidence the worker can publish.

+### 9.4 Timers frozen and restored

+- **Frozen:** the runner skips, with no bookkeeping at all, any row whose

+ status is a hold. Today a blocked delivery still advances `attempts` and

+ state (`internal/schedule/runner.go:345-352`).

+- **Not pruned:** a held row is a route, so the converger's prune

+ (`internal/schedule/converge.go:114`) leaves its timers alone. The source

+ row is held through the whole move (§7.5).

+- **Replaced exactly:** new `Registry.ReplaceTimers(host, rows)` deletes the

+ host's rows and inserts the snapshot in one transaction, preserving

+ `next_at`, `due_at`, `attempts` and `registration`. `UpsertTimer` resets

+ `attempts` and `registration` when it rearms

+ (`internal/registry/timers.go:46-55`), so it is not used for this. The

+ target replaces at install; the source replaces at a dirty rollback.

+- **Declared schedules** then converge as today; an imported row whose spec

+ and path match is kept (`converge.go:223-241`).

+### 9.5 Edge

+None in Phase 1. A held row is not served (`edge.go:595`). Platform-domain

+hosts get the 503 pending page (#174, `internal/edge/pending.go`); custom

+domains get 404 during the window, until F1's holding route.

+### 9.6 Reconcile

+- Mint (`routeFor`, `reconcile.go:364-376`): a record in `landing` mints

+ `landing` with `hold_op = move id` on the target. A record in

+ `rolling_back` or `stuck` never mints `ready` (it mints `restoring` on the

+ source if the row was lost).

+- Removal (`reconcile.go:1114-1159`): a row whose host has a record with a

+ `move` naming this box's placement as `from` and not yet at

+ `source_parked_at` is not removed. The source's move worker deletes it at

+ finalization.

+- Deferral: #199 §3.5's tombstone deferral applies to landing mints.

+## 10. The secrets key, SSM, and the shared fingerprint

+The platform reads no SSM parameter at runtime. An exec instance's secrets

+reach it through its config bundle in the home deployment bucket, opened

+with the box's `CARLOS_SECRETS_KEY`. So a target needs the deployment's key,

+and nothing per app or per instance needs replicating between SSM regions.

+SSM matters only for staging that key on each box, which is Paul's approved

+remediation.

+**Shared with nearest-region R9.** Both specs use one edges-file field, so

+the region-default spec can adopt it as written:

+- **name:** `secrets_key_fp`, in `edgesFile`

+ (`internal/host/edgesfile.go:155-176`, written at `:603-605`);

+- **format:** `"sha256:"` followed by 64 lowercase hex digits of SHA-256

+ over the public key's bytes, `(*ecdh.PublicKey).Bytes()` (the uncompressed

+ P-256 point) of the box's `CARLOS_SECRETS_KEY`;

+- **not omitempty**, following `hibernation_configured`: absent means the

+ writer predates the field (unknown); `""` means the box has no key;

+- **the console** computes the same over its `CARLOS_SECRETS_PUBKEY`.

+A box is key-ready when its value equals the console's. Unknown is never

+ready.

+## 11. Versions and the protocol floor (codex 7)

+The new states, fields and holds are meaningless to an older binary, and an

+older target would mint a `ready` row for a landing record

+(`reconcile.go:374`). So:

+- Every box publishes `moves_protocol: 1` in its edges file; the console

+ reports its own. The request door, and T1, refuse unless every box in the

+ source and target regions and the console are at or above it.

+- Once any Phase 1 move has started, the deployment writes

+ `control/moves-protocol.json` `{"min": 1, "min_release": "vX.Y.Z"}`

+ create-only. `carlos system rollback` and `carlos system apply` (run by

+ the newer binary) refuse to install a release below `min_release`, and

+ refuse any downgrade on a box with an in-flight move it is part of. The

+ console roll recipe states the same floor.

+- The simulation (§14.2) models a target downgraded to a pre-floor binary

+ at every state and asserts the floor stops it.

+## 12. The operator surface

+**API.** The existing doors (`internal/console/authcli.go:625-628`) with the

+§7.1 body, the cut door's freshness check, and abort per §7.11. In Phase 1

+every move mutation requires an operator (`Store.IsOperator`,

+`internal/console/store.go:1858`); a non-operator gets the family's 404.

+Instance-role service credentials keep the status GET only. This closes the

+member-minted token path (`apiinstances.go:1554`) for moves until F1

+designs owner delegation (codex 11). New: `POST …/move/resolve`

+(§7.10). `GET …/move` is unchanged by default; `?detail=1` adds the phases,

+reports, digests and reasons.

+**CLI.** One top-level verb; `docs/site/cli.md` gets its heading in the same

+change (`internal/docsite` fails the gate otherwise).

```

-carlos move --app ips --to uk # app move: shows the plan, asks, then runs

-carlos move --host kpmg.bdn.oncarlos.com --to uk # instance move

-carlos move back --host kpmg.bdn.oncarlos.com # instance move to the region it came from

-carlos move status [--app ips | --host <h>]

-carlos move abort [--app ips | --host <h>]

-carlos move resume --app ips

-carlos move skip --app ips --host <h>

-carlos move resolve --host <h> # operators; a stuck move

+carlos move --host kpmg.bdn.oncarlos.com --to uk [--manual-cut] [--no-wait]

+carlos move status --host <h>

+carlos move cut --host <h>

+carlos move abort --host <h>

+carlos move resolve --host <h> --retry | --declare-lost <box>

```

-- Region names are codes or the home's name (`homeRegionIn`,

- `apiinstancemove.go:136-141`). Output uses place names (nearest-region

- §14).

-- `--wait` is the default for a terminal: it prints each transition with

- its time and exits 0 on `done`, 3 on `rolled_back`, 4 on `stuck`, 5 on

- refusal. `--no-wait` returns after the request.

-- `--manual-cut` stops at `ready`. `carlos move cut --host <h>` cuts.

- Without it, the CLI asks for `cut: auto`.

-- `--yes` skips the app move's confirmation, for scripts.

-- Every fact it prints comes from the console (the 2026-08-08 rule). No

- step needs box access.

-## 17. The console

-**Instance page** (`internal/console/templates/app_instance.html`) gains a

-**Region** section above Danger: the region's place name, "Move to another

-region" with a region select (eligible targets only, each with a reason

-when greyed out), and a confirmation page (M-C10 to M-C13) that says what

-moves, what stays, and the expected 503 window.

-**App settings** (`templates/app_settings.html`) gains a **Region** section

-beside the existing Account section. Its button says "Change region", never

-"Move", because the Account section's "Move to another account"

-(`server.go:1209`) already uses that word.

-**During a move**, both pages show a progress list (M-C20 to M-C27):

-Copying data (with bytes copied), Checking UK (each check, green or the

-failing sentence), Stopping in Ireland, Starting in UK, Done. Each step has

-its time. The Instances table (`app_instances.html:86-91`) shows

-"Moving to UK" in the Region column. The page polls `GET …/move?history=1`.

-**After a move:** "Moved from Ireland to UK on 2 October 2026. 48 seconds

-offline." and the hold line from §10. After a rollback: the reason and

-"Back in Ireland and serving. Nothing was lost." After `stuck`: an

-incident banner for the owner, and the `resolve` button for an operator.

-## 18. Observability (P4)

-- **Move log.** One object per event under

- `moves/<acct>/<app>/<host>/<move-id>/` in the deployment bucket, named

- `<unix-nanos>-<actor>.json`. One object per event means three writers

- (console, source, target) never contend. The target report (§7.3) lives

- beside them. Events: requested, round (sampled one in ten, plus the

- last), ready, cut-refused, cut stages, flipped, each landing step,

- committed, opened, rollback (with the failing step), rolled_back, stuck,

- cleanup scheduled, cleanup done.

-- **Timings** in the terminal event: prepare, cut (from `cut_at` to the

- flip), land (flip to open), and **offline**, from `cut_at` to open (or to

- the source's `ready` after a rollback). Offline is the number owners

- care about.

-- **Paging:** `rolled_back` is a warning; `stuck` and `landing (overdue)`

- are incidents; `preparing` with no promotable round for 60 minutes is a

- warning (finding 1's shape).

-- **Box logs** keep the `moves: <host>: …` shape (`movespass.go`) and add

- one line per step with its duration.

-- **Member log stream:** each transition is an event on the app's platform

- stream (`emitEvent`, `internal/activator/logsink.go:31-36`), so `carlos

- logs` shows the move beside the app's own output.

-## 19. Failure modes

-| failure | detected by | outcome |

+`--wait` is the default: it prints each phase with its time, and exits 0

+on done, 3 on rolled back, 4 on stuck or waiting, 5 on refusal.

+## 13. Observability

+- **Move log:** one object per event under

+ `moves/<acct>/<app>/<host>/<move id>/` in the deployment bucket

+ (`<unix-nanos>-<actor>.json`), beside the reports and evidence. Every

+ phase entry and exit, every check result, every reason.

+- **Timings:** prepare, cut (`cut_at` to `flipped_at`), land (`flipped_at`

+ to `opened_at`), and **offline** (`cut_at` to `opened_at`, or to the

+ source's reopen after a rollback).

+- **Paging:** `rolled_back` (warning); `stuck`, `rolling_back` past

+ `release_deadline`, and any start refused by the start check (incident);

+ `preparing` with no promotable round for 60 minutes (warning).

+- **Member stream:** each transition is an event on the app's platform log

+ (`emitEvent`, `internal/activator/logsink.go:31-36`).

+## 14. Tests

+### 14.1 Unit and integration

+- Every transition and its guard, including each illegal edge and a stale

+ move id or incarnation.

+- Archive: equal plaintext for equal content; randomized sealed bytes;

+ truncation and reordering detected; wrong key refused; special files

+ refused; cap enforced mid-stream.

+- Each readiness check failing alone, against fakes, and T4 against MinIO.

+- Each landing phase crashing and replaying; `probing` never replaying an

+ earlier phase.

+- Held render, held materialization, `RetireExcept` keeping a held entry,

+ the runner's freeze, `ReplaceTimers` exactness.

+- The start check: mismatched markers refuse; absent markers are origin.

+- Reconcile: landing and restoring mints; source row not removed mid-move.

+### 14.2 The randomized crash simulation

+A Go test driving the **real** transition functions and the real worker

+phase functions through injected interfaces, never a re-implementation.

+Actors: console, source agent, target agent, deadline sweep, and **the

+application process** on whichever box runs it.

+The model includes what rev 1's model missed (codex findings, "simulation"

+note):

+- the app's **own background work**: it writes the database and an

+ external ledger on its own schedule while running, including during a

+ probe, and an external acknowledgement counts as acknowledged (INV-2);

+- **stalled and surviving processes**: a worker can stall indefinitely

+ while its child keeps running; an agent crash kills flat children and

+ leaves nested ones running;

+- **scheduler bookkeeping**: timers fire, advance and are refused;

+- **config, adoption and retirement**: held render, `RetireExcept` on edge

+ reloads, reconcile mints and removals;

+- **versioned buckets**: deletes leave noncurrent versions;

+- **forged metadata**: another box writes a fake evidence or report object

+ with a wrong move id or incarnation;

+- **tombstones**: delete and recreate at any step;

+- **downgrades**: a box replaced by a pre-floor binary at any step.

+A seeded scheduler picks the next atomic step, injects crashes (losing

+memory, keeping disk and buckets), stalls, and the faults of §7. After

+every step it asserts INV-1 and INV-5; at the end of every run INV-2,

+INV-3 (every run with only restarting crashes terminates), INV-4 (every

+start's markers matched), INV-7, INV-8 and INV-9.

+2,000 seeds in the gate; 200,000 nightly.

+**Mutation checks**, run in a tree copy, each of which must fail the

+simulation within the gate's seeds: start the source on a deadline without

+evidence; open before commit; open without the `hold_op` condition; treat a

+lost CAS as stop evidence; discard target writes on rollback; replay

+install after `probing`; re-extract files after `probing`; skip the start

+check; let the runner advance a held row's timers; restore from staging;

+re-seal the archive on replay; let reconcile remove the source row mid-move;

+let a tombstone park sweep another incarnation's staging; skip the protocol

+floor. A mutation the simulation cannot catch is a defect in the

+simulation.

+### 14.3 Cross-version

+The request door refuses below the floor; a pre-floor target cannot be

+installed during a move; abort on every new state is refused by v0.22

+consoles (P0's allowlist, `provision.go:1027-1035`).

+### 14.4 Rehearsal (M12)

+`moveprobe`, a test app in this repo, is shaped like ips:

+- writes `instance.key` into its files directory at first boot, seals a

+ row with it, and refuses to start on a mismatch;

+- requires a store binding and a sealed secret at boot, and writes and

+ reads an object every boot;

+- declares a schedule and registers a once-timer, logging each run;

+- writes a row from a background loop every few seconds (INV-2's case);

+- fails its readiness path in a named region on request, before or after

+ writing (to force clean and dirty rollbacks).

+Runs, each recording every timing: dev, IE to uk, done; dev, forced clean

+rollback; dev, forced dirty rollback; then the same on the flagship with

+scoped approval; then a disposable ips instance in `bdn` with scoped

+approval; then kpmg with Paul's approval. A run passes only if the key

+verifies, the object reads back, the secret is present, every timer ran

+exactly once around the cut, and no background row is missing.

+## 15. Rollout and gates

+1. **Ops (in progress):** `CARLOS_SECRETS_KEY` on every region box.

+2. **Release with `secrets_key_fp`** (shared with nearest-region R9).

+ Verify every box's value equals the console's.

+3. **Phase 1 release, dormant.** Consoles first, then every box. The door

+ refuses until every box in both regions publishes `moves_protocol: 1`.

+4. **Infra, scoped approval:** each target region's `sibling_staging_arns`

+ lists each source region's data bucket (T10's return lane). For kpmg

+ that is uk listing ie's bucket.

+5. **Rehearsals** in §14.4's order. Each gates the next.

+6. **Write `control/moves-protocol.json`** before the first production

+ move.

+## 16. Follow-on specs

+Each is its own spec. No design here, only what it must satisfy.

+| spec | scope | must meet |

|---|---|---|

-| target lacks the secrets key | T2 | cut refused; source never stops |

-| bundle cannot be opened on target | T3 | cut refused |

-| store unreachable from target | T4, L4 | cut refused, or rollback |

-| target disk too full | T6 | cut refused |

-| target has nothing promoted for its architecture | T7 | cut refused |

-| old target build | T1 (capability in edges file) | door refuses the request |

-| files archive damaged or missing at landing | L1 | rollback |

-| app starts, crashes after binding | L5 settle window | rollback |

-| app needs more than 30 s to start | L5 | rollback. The bar is the same one every wake meets, so the app would fail ordinary wakes too |

-| target box dies during landing | source, at `landing_deadline` | rollback |

-| source dies after the flip, landing succeeds | none needed | done; P5 waits for the box, or for an operator if the box is gone |

-| source dies before rollback finishes | sweep at deadline + 5 min | `landing (overdue)` incident; `resolve` once the box is back |

-| both fail | §7.6 | `stuck`; incident; data intact in the source prefix |

-| staging delete denied on target | cleanup | v2: unreferenced garbage, logged; legacy pre-epoch: logged as a hazard (§7.9) |

-| a stale generation or files directory from an earlier stay on the target | landing | never read: the pointer names the move generation, and L2 moves the old directory aside |

-| a db push requested during a move, or a move during a push | door and box | refused both ways (§26, I-7) |

-| a forged record requests a move | ledger mismatch | no store credential, no deletion warrant; the console flags it |

-| app writes outside its files dir | not detectable | documented loss (§8.4) |

-| app does boot-time side effects | not preventable | they may run twice on a rollback; `CARLOS_MOVE_PROBE=1` lets the app skip them |

-## 20. Tests

-### 20.1 Unit and integration

-- `MoveState` transitions, including the new states, every illegal edge

- (409 matrix), and old-record parsing.

-- Archive: deterministic bytes for one input; special files refused;

- truncation and chunk-reorder detected; wrong key refused; per-file hashes

- checked on extraction; flat-to-nested and nested-to-flat extraction.

-- Target report: each of T1 to T8, failing and passing, against fakes and

- against MinIO for T4.

-- Landing: L1 to L7 each failing in turn leads to rollback, never to open.

- An L6 lost to a rollback never opens.

-- Reconcile mints `landing` for a landing record, and `ready` otherwise.

-- The source's `rolling_back` arm with the `moving` row present and with it

- retired.

-- Timers carried: a once-timer due during the cut fires once after open; a

- declared schedule keeps its `next_at`.

-- Door: owner-only, the operator family, home with replicas, `regions`

- allow-list, legacy sidecar env.

-### 20.2 The state-machine simulation

-A Go test that models the console, the source box, the target box, the

-sequencer, the record store (with CAS), the source and target buckets, a

-client issuing writes through whichever edge serves, and a clock. Each step

-of each actor is an atomic action. A seeded scheduler picks the next

-action, and can crash any actor (losing its memory, keeping its disk and

-buckets) or delay it. Faults are injected at the same points as §19: each

-landing check failing, box death, deadline expiry, denied deletes.

-After every action it asserts INV-1, INV-2, INV-4, INV-4b and INV-6. At

-the end of every run it asserts INV-3 (every acknowledged client write is

-in the authoritative copy), INV-5 (every start saw the authoritative files

-version), INV-7 (terminal within the bound) and INV-8.

-The model carries #199's storage: generations keyed by incarnation, the

-record's `DB` pointer, `.lineage` and the two directory markers, and the

-epoch, which the scheduler can write at any moment, so legacy, first-v2

-and v2 moves all occur. Its runs include moves interleaved with a

-`carlos db push` request, and boomerangs (ie→uk→ie, including with an

-earlier move's generation and files directory still in place).

-- 2,000 seeds in the ordinary gate; 200,000 in a nightly job.

-- **Mutation checks**, run in a copy of the tree: open before commit; skip

- the cut's report re-check; flip `region` and `DB` in two writes; let the

- target restore from a generation the pointer does not name; let rollback

- leave `DB` at the move generation; let a landing reuse an existing files

- directory; extract files after the probe; let the source purge the files

- directory at the cut. Each mutation must

- make the simulation fail within the gate's seed count. A mutation it

- cannot catch is a defect in the simulation.

-- The simulation drives the real transition functions

- (`provision.Move*`) and the real worker step functions through injected

- interfaces, not a re-implementation of them. A fake that agrees with

- itself proves nothing.

-### 20.3 Cross-version

-An old target (no capability) is refused at the door. A v0.22 console

-refuses abort on every new state (P0's allowlist). A new console against

-old boxes refuses every move.

-### 20.4 Rehearsal with an ips-shaped app

-The `bdf/hello` rehearsal passed because hello has none of the state that

-failed. The rehearsal app, `moveprobe` (a test app in this repo), has every

-piece of it:

-- at first boot it writes `instance.key` into its files directory and seals

- a row with it; every boot verifies the seal and **refuses to start** on a

- mismatch, so a missing or regenerated key fails the health check;

-- a store binding it requires at boot (refuses to start without it), and an

- object it writes and reads back on every boot;

-- a sealed config secret it requires;

-- a declared schedule and a once-timer, each logging its runs to the DB;

-- a config switch, `MOVEPROBE_FAIL_IN=<placement>`, that makes it fail

- health in one region, to force a rollback.

-Rehearsals, each with timings recorded in the move log:

-1. dev deployment: moveprobe IE to uk, done; then forced failure, rolled

- back; then uk to home after Phase 3.

-2. flagship, with scoped approval: a disposable moveprobe instance under the

- operator's account, the same three runs.

-3. flagship, with scoped approval: a disposable ips instance in `bdn`,

- seeded through the app's own seed, with a key and a store. This is the

- shape that failed. (§23 Q9.)

-4. Only then kpmg, at a time Paul chooses.

-Each run passes only if the key verifies, the object reads back, the

-secret is present, every timer ran exactly once around the cut, and the

-offline time is recorded.

-## 21. Rollout and gates

-0. **#199's release N first** (its §7): every box and console understands

- `DB`, `StorageKey`, `.lineage` and the epoch. Phase 1's code reads and

- writes storage only through `StorageKey` (§26, I-2), so it is built on

- top of N, not beside it. Its own rollback floor (#199 §7 step 4) then

- covers moved records too: a binary older than N would drop a moved

- record's `DB` pointer and send it back to its first generation.

-1. **Consoles first.** They learn the new states, the cut door's report

- check, owner-only doors, and the ledger. Moves keep working as today

- against old boxes, because the door refuses any target that does not

- publish `moves_schema >= 2`. In practice that refuses every move until

- step 2.

-2. **Boxes, every region.** The source and the target roles ship together.

- A box publishes `moves_schema: 2` only when it runs the code. Mixed

- fleets are safe: the door checks both boxes.

-3. **Infra check, per deployment:** every region box publishes a

- `secrets_key_fp` equal to the console's. Where not, stage the key (the

- SSM note in §6). This is production configuration and needs scoped

- approval.

-4. **Rehearsals** (§20.4) in order. Each one gates the next.

-5. **Owner self-serve** turns on with `CONSOLE_MOVES_SELF_SERVE=on`, and

- only after Phase 3 (P5 deletion) is live, so a move that promises

- residency delivers it. Until then the doors serve operators and

- instance-role services only.

-6. **App moves** turn on after Phase 4 and its rehearsal: an app with two

- moveprobe instances and an app store.

-## 22. Phases

-Each phase lands dormant and is useful alone.

+| **F1 Owner surface** | owner self-serve from the console and `carlos move`; auto-cut for persons (M6); holding route for custom domains (P2); move history | owners only (M13); **move-capable service authority is explicitly owner-delegated**, including the legacy mint path (`apiinstances.go:1554`), with a stated treatment of existing member-minted tokens that keeps approved router credentials working (codex 11) |

+| **F2 Moves into home (P3)** | `CARLOS_HOME_PLACEMENT`, home as a target, `carlos move back` | the handoff of §5 unchanged; home children's `CARLOS_REGION` change gated |

+| **F3 Residency cleanup (P5)** | delete `superseded/<move id>/`, `move-snapshots/`, `.parked/superseded-*`, receipts, after the 7-day hold (M7), with owner early deletion | **no deletion of a prefix that is, or is an ancestor of, a live location**, and selection is excluded atomically before deletion, not by a fire-time read (codex 8); **the complete residue inventory**, including noncurrent object versions and delete markers in versioned data buckets (`infra/modules/deployment-buckets/main.tf:27-30`), earlier parks and local parks, with scoped version-delete authority and verification of actual removal before reporting residency complete (codex 9); **a deletion warrant that binds the exact scope**: account, app, incarnation, move id, bucket and prefixes, issued from an immutable console ledger and never from box-writable records or logs (codex 10) |

+| **F4 Stores** | private stores with their instance, app stores with their app (M3), consumer hold (M9) | **a joint instance and store transaction** with binding versions on both sides, health checks and compensation, so a rollback never leaves the source bound to an unusable twin (codex 12); **the store record transitions with config** (`walk.go:427`, `:448`), and grant, rotate, revoke and purge are serialized against the move, with security policy preserved and a named principal deleting the fenced old bucket (codex 17); copy credentials issued under F3's ledger binding (codex 10) |

+| **F5 App moves** | the sequencer, app region change, pause (M8), sidecar refusal (M10) | **durable child move ids persisted before dispatch**, with recovery of their outcomes (codex 18); **an app and store cut barrier** covering every consumer, including concurrent creates, skipped and explicitly placed instances (codex 18, M9); defined transitions for pause, abort and move-back of the app operation |

+| **F6 Graceful cuts** | freeze, thaw, verify hooks (the 09-24 P1); sidecar moves | hooks never replace stop evidence |

+| **F7 `carlos db push`** | #199's deferred push, on §5's handoff with source and target the same box | the same four invariants |

+## 17. Interface points with #199 rev 3

|---|---|---|---|

-| **1. Safe instance moves** | journal check of §1.2b's cause; files archive (§8) with `<host>.env` and timers, keyed by incarnation and generation; target readiness T1 to T8 and the cut's re-check (§7.3); `landing` with the durable fence, L1 to L7, automatic rollback and `stuck` (§7.5, §7.6); `MoveState` fields and the generation-carrying flip and rollback (§7.1, with #199 §4); the legacy path (§7.9) if Q11 keeps it; edges file `moves_schema` and `secrets_key_fp`; new rails (home with replicas, legacy sidecar env, `regions`, no move during a push); the minimal move log (events and `reason`); the simulation (§20.2) | #199 release N built (rolled before Phase 1 rolls); the §26 interface points agreed by both specs | rehearsals 1 to 3 pass; kpmg moves with Paul's approval |

-| **2. Owner surface** | `carlos move` (G8); console Region sections and progress; auto-cut; owner-only and operator doors; the ledger; P2 holding route (503 for custom domains and the source's holding answer); full P4 log, history, paging; copy review | Phase 1 | an owner can move an instance without help, with `self-serve` still off |

-| **3. Home and residency** | P3 (§9); `carlos move back`; P5 hold and deletion with signed warrants (§10) | Phase 2; infra approval for the home staging grant | rehearsal 1's uk-to-home run passes; a hold expires and deletes; `self-serve` on |

-| **4. App moves and stores** | the sequencer (§12); `apps.region` writes; store backend per placement; private and app store moves with the write fence and the move-scoped credential (§11) | Phase 3; nearest-region spec built | the two-instance app rehearsal passes |

-Phase 1 is the smallest set that makes an ips-shaped move safe. Without

-the archive the key is lost (finding 2a). Without T2 and L3 the config is

-missing (finding 2b). Without L5 and §7.6 a bad target is an outage rather

-than a 503 and a rollback (M2). Without the simulation the next

-interleaving is found in production. Everything else is surface,

-convenience or residency, and none of it is needed for a move to be safe.

-## 23. Open questions for Paul

-Each is a choice, with a recommendation.

-1. **What the archive carries.** (a) Only the fixed set in §5 (the files

- directory, `<host>.env`, timers). (b) Also paths an app declares in its

- manifest. **Recommend (a).** One rule is easy to document and to

- verify, and every app we run fits it. (b) can be added later without

- changing the format.

-2. **Archive size cap.** (a) 256 MiB, refused above it. (b) 1 GiB. (c) No

- cap. **Recommend (a).** Key files and tokens are kilobytes. Anything

- large belongs in object storage, and the cap keeps the cut window

- predictable.

-3. **Cut timing for person-started moves.** (a) Auto-cut when ready and the

- target passes, the default in the CLI and the console. (b) Always ask.

- **Recommend (a)**, with `--manual-cut`. The API default stays manual,

- because titogo holds its own freeze before cutting.

-4. **Hold period.** (a) 7 days. (b) 30 days. Separately, may an owner end

- the hold early ("delete the Ireland copy now")? **Recommend 7 days, and

- yes to early deletion for owners**, as an explicit, confirmed act.

-5. **An app move when an instance rolls back.** (a) Pause, keep the app

- region at the target, owner decides. (b) Continue with the rest and

- report. (c) Revert the app region. **Recommend (a).** It never flaps the

- app region, and the owner sees the failure before more cuts happen.

-6. **Downtime for an app store's final copy.** (a) Hold every instance of

- the app at 503 for the store's final delta. (b) Keep instances up with

- the bucket write-fenced, so uploads fail with storage errors for that

- window. **Recommend (a).** A 503 is honest and retried; (b) shows users

- app errors such as "File storage is unavailable". The plan shows the

- expected length before the owner confirms.

-7. **Apps with native sidecars in the source region.** (a) Refuse the app

- move until sidecar moves exist (Phase 5). (b) Move the instances and

- leave the sidecars, listed in the plan. **Recommend (a).** A sidecar

- that talks to its app across regions is a surprise nobody chose.

-8. **Pre-warming the target** (restore staging into a scratch DB during

- `ready`, so the cut only applies the last delta). (a) Later, if

- rehearsals show the window above about a minute. (b) In Phase 1.

- **Recommend (a).** It adds a second restore path to Phase 1, which is

- exactly where the risk should not grow.

-9. **Rehearsal subject on the flagship.** (a) moveprobe, then a disposable

- ips instance in `bdn`, then kpmg. (b) moveprobe, then kpmg. **Recommend

- (a).** Finding 3 is that a rehearsal must have the moved app's shape,

- and moveprobe has the shape we know about but not ips's own startup.

-10. **Members.** Today's move doors admit any member of the account. M1

- names owners. (a) Owners only, as written in §14. (b) Owners plus a new

- "can move" grant for members. **Recommend (a)**, matching app delete

- and the account move.

-11. **Moves before #199's epoch exists.** (a) Refuse every move until the

- epoch is written, so production only ever runs the v2 landing. (b)

- Support the pre-epoch legacy flow of §7.9 as well. **Recommend (a).**

- #199's rollout writes the epoch as soon as every box and console runs

- release N, which Phase 1 needs anyway. (b) keeps "staging wins" and a

- rollback that has to delete a live-looking lineage in the target, which

- is the hazard the generations exist to remove. Choose (b) only if kpmg

- must move before #199's epoch can be written. Either way, a legacy

- record moved after the epoch takes the first-v2 path of §7.9.

-12. **Where a landed generation lives** (#199 §4's open point, I-1 in

- §26). (a) The pointer names the staging root (`Root: "moves"`), with no

- copy. (b) Landing copies the staged generation into

- `instances/<host>/<incarnation>/<to_gen>/` before the probe, and the

- flip points there. **Recommend (b)**, for the reason in I-1: under (a)

- every moved instance's live replica sits in the one prefix that every

- other region's box role may write and delete.

-## 24. Copy (pending copy review)

-Drafts. None of these is approved. Every string goes through the

-copy-review gate before it is written into code or templates. Place names

-come from nearest-region §14. `<…>` marks a value.

-### CLI

+| I-1 | **Incarnation.** Every move artifact and guard uses `provision.Incarnation(rec)`, including the legacy identity for a zero CreatedAt (#199 §2). | #199 | one shared function; this spec calls it |

+| I-2 | **Tombstone region during a move.** The delete door writes the record's current `region`. | #199 | confirm; this spec adds the other region's park (§7.12) |

+| I-4 | **Park destinations.** #199 parks to `parked/<incarnation>/<host>/` and `.parked/<incarnation>/<host>/`; this spec to `superseded/<move id>/<host>/` and `.parked/superseded-<move id>/<host>/`. | shared | both use the same copy-manifest-delete and rename helpers; the two namespaces never overlap |

+| I-5 | **Deferral.** #199 §3.5 defers a recreated host's row while a park is pending. | #199 | the landing mint (§9.6) calls the same predicate |

+| I-6 | **Stale-row retire.** #199 §4 retires a row whose `(account, app, host)` record is absent. | #199 | its retire stops the child and deletes the row only; it must never park or purge the files directory or database of a row held by a move (`hold_op` set) while the move's record exists |

+| I-7 | **Local parks and the files directory.** #199 §3.4 parks the nested `InstanceDir`; this spec defines the files directory for both layouts (§6.1). | shared | #199 parks the flat `<DataDir>/<host>/` too, by this spec's definition, so a recreated flat host does not inherit an `instance.key` |

+| I-8 | **`generation.json` and `lock.json`.** #199 §3.3 copies every key except `lock.json`. | #199 | `generation.json` is copied like any other key; a park never deletes `lock.json` early |

+## 18. Question for Paul

+**Declaring a box lost (§7.10).** If a box is destroyed during a handoff it

+holds, its stop evidence can never arrive, and the move waits, paged. Choose

+one:

+- **(a)** An operator verb, `carlos move resolve --declare-lost <box>`,

+ records that the box's processes are gone and proceeds as a clean

+ rollback. If the lost box was the target, anything it wrote after the

+ flip is lost, and the confirmation says so.

+- **(b)** No verb. Recovery from a destroyed box is a manual, below-the-API

+ procedure written in the runbook.

+**Recommend (a).** It is the only exception to INV-2 and INV-3, it needs a

+person, and it keeps the recovery inside the API (the 2026-08-06 rule). The

+loss it can cause is bounded to a probe that never served traffic, unless

+the box was destroyed after the commit, in which case the verb is refused

+(the target is authoritative and the source is parked).

+## 19. Copy (pending copy review)

+Drafts for Phase 1's operator CLI. None is approved; every string goes

+through the copy-review gate before it is written into code. Place names come

+from nearest-region §14. `<…>` marks a value. No em dashes.

| id | string |

|---|---|

| M-C1 | Moving <host> from <from> to <to>. |

| M-C2 | Copying data. <host> keeps serving while this runs. |

-| M-C3 | Copied <bytes>. Waiting for the copy to catch up. |

-| M-C4 | Checking that <to> can run <host>. |

-| M-C5 | <to> is ready. Starting the cut. <host> will answer 503 for about <estimate>. |

+| M-C3 | Checking that <to> can run <host>. |

+| M-C4 | <to> cannot take this move yet: <check>. |

+| M-C5 | <to> is ready. Starting the cut. <host> will be offline for about <estimate>. |

| M-C6 | Stopped in <from>. Starting in <to>. |

| M-C7 | Moved. <host> is served from <to>. It was offline for <seconds> seconds. |

-| M-C8 | The <from> copy is kept until <date>, then deleted. |

-| M-C9 | <to> could not start <host>: <reason>. It is back in <from> and serving. Nothing was lost. |

-| M-C10 | <host> could not start in <to> or in <from>. It is offline. An operator has been paged. |

-| M-C11 | Ready to cut. Run carlos move cut --host <host> when you are ready. |

-| M-C12 | This move has started its cut. It will finish or move back on its own. |

-### App move

-| id | string |

-|---|---|

-| M-C13 | Moving app <app> to <to>. New instances start in <to> from now on. |

-| M-C14 | <n> instances will move, one at a time. Each is offline for about a minute. |

-| M-C15 | <n> stay where they are: |

-| M-C16 | <host>, already in <region>. |

-| M-C17 | Its files will move to <to> after the instances. The whole app is offline for about <estimate> while they finish copying. |

-| M-C18 | Paused. <host> could not move: <reason>. It is serving in <from>. Run carlos move resume --app <app> to try again, or carlos move skip --app <app> --host <host> to leave it there. |

-| M-C19 | Moved app <app> to <to>. |

-### Refusals

-| id | string |

-|---|---|

-| M-C30 | <to> cannot take this move yet: <check>. |

-| M-C31 | <to> cannot read this app's configuration. An operator needs to install the deployment's secrets key there. |

-| M-C32 | <to> cannot reach this app's storage. |

-| M-C33 | <to> does not have enough free disk for <host>. |

-| M-C34 | This instance has replicas. Instances with replicas cannot move yet. |

-| M-C35 | This app only runs in <regions>. Change that in its instances settings first. |

-| M-C36 | <host> keeps a file this move cannot copy: <path>. Only regular files and folders move. |

-| M-C37 | <host> keeps <size> of files beside its database. Moves carry up to 256 MiB. Keep large files in object storage. |

-| M-C38 | <host> is already moving. Wait for it to finish, or run carlos move abort --host <host>. |

-| M-C39 | Moves into <home> are not available yet. |

-| M-C40 | This app has sidecars in <from>. Apps with sidecars cannot move yet. |

-### Console

-| id | string |

-|---|---|

-| M-C50 | Region |

-| M-C51 | Served from <region>. |

-| M-C52 | Move to another region |

-| M-C53 | Change region |

-| M-C54 | Moving copies this instance's data to <to>, stops it here, and starts it there. It is offline for about a minute. If it cannot start in <to>, it comes back here on its own. |

-| M-C55 | What moves: the database, the files beside it, its settings and its scheduled jobs. |

-| M-C56 | What stays: the app's files storage stays in <region> with the app. |

-| M-C57 | Move <host> to <to> |

-| M-C58 | Moving to <to> |

-| M-C59 | Copying data |

-| M-C60 | Checking <to> |

-| M-C61 | Stopping in <from> |

-| M-C62 | Starting in <to> |

-| M-C63 | Done |

-| M-C64 | Moved from <from> to <to> on <date>. Offline for <seconds> seconds. |

-| M-C65 | The <from> copy is kept until <date>. |

-| M-C66 | The <from> copy was deleted on <date>. |

-| M-C67 | Back in <from> and serving. Nothing was lost. <to> could not start it: <reason>. |

-| M-C68 | This move needs an operator. The instance is offline. |

-| M-C69 | Not started from this console. |

-## 25. Gaps this spec closes, against today's code

-| # | gap | where |

+| M-C8 | <to> could not start <host>: <reason>. Moving it back to <from>. |

+| M-C9 | <host> is back in <from> and serving. Nothing was lost. |

+| M-C10 | Waiting for <box> in <to> to confirm it has stopped <host>. <host> is offline. |

+| M-C11 | <host> could not start in <to> or in <from>. It is offline. Run carlos move resolve --host <host> --retry once the cause is fixed. |

+| M-C12 | Ready to cut. Run carlos move cut --host <host> when you are ready. |

+| M-C13 | This move has started its cut. It will finish or move back on its own. |

+| M-C14 | <host> keeps a file this move cannot copy: <path>. Only regular files and folders move. |

+| M-C15 | <host> keeps more than 256 MiB of files beside its database. Keep large files in object storage. |

+| M-C16 | <host> does not answer its readiness path. Declare moves.ready_path for this app. |

+| M-C17 | <host> uses private storage. Instances with private storage cannot move yet. |

+| M-C18 | This declares <box> lost. Anything <host> wrote there after <time> is lost. Type the host name to confirm. |

+## 20. Round 1 dispositions

+| # | finding | disposition |

|---|---|---|

-| X1 | Instance files directory not carried, not backed up | `movespass.go:476-494`; no reference anywhere in the repo |

-| X2 | `<host>.env` not carried: new admin token, box-local overrides lost | `prepare.go:49`; `admintoken.go:115` |

-| X3 | Config rendered only where `CARLOS_SECRETS_KEY` is staged, and absence is silent at spawn | `agent.go:881-886`, `:1812`; `backend_exec.go:1146-1149` |

-| X4 | Nothing checks the target before the source stops | `movespass.go:415-505` |

-| X5 | Target opens at mint, with no health gate | `reconcile.go:374`; `edge.go:595` |

-| X6 | No automatic rollback; recovery was a hand edit of the record | #175, 2026-10-02 post |

-| X7 | Home with replicas passes the move gate (Q2 unimplemented) | `apiinstancemove.go:187-190`; `provision.go:966-983` |

-| X8 | The manifest's `regions` allow-list is not applied to moves | `apiinstancemove.go:179-237` reads no manifest |

-| X9 | Timers lost or reset (G15) | `registry.go:565-579`; `converge.go:223-241` |

-| X10 | Any member can move | `authcli.go:625-628` |

-| X11 | The isolated layout pass, as designed, strands `<DataDir>/<host>/` | isolated-exec design §2 item 1 |

-| X12 | Moves to home refused | `apiinstancemove.go:203-209`; `movespass.go:345`; `backend_exec.go:989-993` |

-| X13 | Source replica, files and env never deleted | `movespass.go:492` (DB artifacts only) |

-## 26. Interface points with #199 (replica ownership)

-The #199 session will not change `movespass.go`, `promoteStagedReplica`

-or `MoveComplete` until the two specs agree. These are every point where

-they meet. Each says who owns it and what this spec needs. Both specs

-should be reviewed against this list together.

-|---|---|---|---|

-| I-1 | **Where a landed generation lives.** #199 §4 says `DB = {Gen, Root: "moves"}` and no copy; the coordinator reports it as still undecided. | #199 | **Recommend a copy into `instances/<host>/<incarnation>/<to_gen>/` on landing**, as a step between L4 and L5, with the flip writing `Root: "instances"`. The reason is IAM. Every sibling region's box role holds `s3:PutObject` and `s3:DeleteObject` on `moves/*` in this bucket (`infra/modules/region-host/main.tf:217-221`), so under `Root: "moves"` a moved instance's **live** replica sits in the one prefix that every other box, and anything that obtains a box role (#171), may overwrite or delete. The copy is an in-bucket server-side copy into a key nothing else uses (the move id is in it), so it reintroduces none of the staging-wins hazards, and it costs what today's promotion costs (`backend_exec.go:1580-1606`). If #199 keeps `Root: "moves"`, this spec works unchanged (L5 restores whatever the pointer names), and the IAM exposure is recorded as accepted. |

-| I-2 | **One function for every location.** `movespass.go:135`, `:364`, `:365` and `:369` build the checkpoint prefix, the mirror source, the staging destination and the receipts key from the host alone. | #199 defines; this spec calls | The moves code computes the source with `StorageKey(rec, epoch)` (#199 §2.3) and the destination with a new `provision.MoveStagingKey(rec, toGen)` that #199 defines beside it, so the layout lives in one package. The cut's checkpoint goes into the source storage key. |

-| I-3 | **The flip, the commit and the rollback CAS.** | shared | `MoveComplete` writes `Region = to`, `DB = {to_gen, Root}`, `state = landing` in one CAS, guarded by move id and `DB == from_db`. A new `MoveCommit` (target) clears `move`, guarded by move id and `DB.Gen == to_gen`. A new `MoveRollback` writes `Region = from`, `DB = from_db`, `state = rolling_back`, with the same guard. #199's push commit is guarded by request id, incarnation and `FromGen` (#199 §5.4 step 5). The two families must never both succeed on one record: see I-7. |

-| I-4 | **Generation allocation for a move.** #199 §4 says a move allocates `g<seq+1>-<moveID>` but not who. | this spec | The console allocates `to_gen` in `MoveRequest` from `from_db`'s sequence (0 for legacy or pointerless), with the console-minted move id as the nonce. The move id's alphabet must be valid in #199's `Gen` grammar; #199 states that grammar. |

-| I-5 | **Incarnation of a legacy record with zero CreatedAt.** | #199 | `provision.Incarnation(rec)` must return the same value on every box for such a record, or #199 must say that a zero-CreatedAt record is refused a move until it is backfilled. The first-v2 landing (§7.9) keys the staged copy, the archive and the files markers by it. |

-| I-6 | **The files directory and its markers.** #199 §2.5b gives the nested `InstanceDir` an `.incarnation` marker and moves a mismatch aside to `.retired/<old-incarnation>/`. | shared | (a) Extend the same check to the flat `<DataDir>/<host>/`, which this spec makes a platform-defined path (§5); a recreated flat host otherwise inherits the deleted instance's `instance.key`. (b) This spec adds `.files-gen`; #199's push must leave it unchanged. (c) `.retired/` names need the generation too, because one incarnation can be moved aside more than once by successive landings on one box: `.retired/<incarnation>-<files-gen>/`. (d) #199's wake check must treat a directory without markers as "not landed yet" during `landing`, not as a mismatch to retire, because L2 writes the markers last. |

-| I-7 | **A push and a move never overlap.** | shared | `MoveRequest` refuses while the record has an in-flight push request; #199's push request refuses while `move != nil`, in any state including `stuck`. The route statuses stay distinct: `db-push` is #199's, `moving` and `landing` are this spec's. #199's restart rule ("a route left at `db-push` with no in-flight request is released") never touches a `landing` row, and the landing worker's repair (§7.5, L7) never touches a `db-push` row. |

-| I-8 | **Receipts and the source's files manifest.** | this spec, on #199's key | `move-receipts/<host>/<incarnation>/<from_gen>.json` (#199 §4) and `….files.json` beside it (§8.2); legacy records keep `move-receipts/<host>.json`. Both are cleared at a terminal state, and P5 deletes anything left with the source generation. |

-| I-9 | **Purge with errors.** | #199 | The cut (`movespass.go:492`) and the target's rollback use `PurgeLocalDBArtifactsErr` (#199 §5.6), which removes `.lineage` first; an error fails the pass and is retried. Neither purge touches the files directory. |

-| I-10 | **Registry columns and retire-on-identity-change.** | #199 | The landing row is minted with #199's `incarnation`, `storage_key` and `storage_gen` and status `landing`. After the flip, the source's row can be retired twice over (out of scope by region, and a storage-key mismatch under #199 §2.6). Both are idempotent, and **#199 §2.6's retire must purge only the database, never the files directory**, which is the rollback's material on the source (§7.4 step 8). On rollback the source row's storage key equals the record's again, so no retire fires. |

-| I-11 | **Generation retention across regions.** | #199 | The keep-five GC (#199 §5.4) considers only its own bucket. Generations a move left in another region are P5's (§10). `carlos db rollback` and `carlos db pull --generation` refuse an out-of-region generation (#199 R3) and `carlos db backups` labels it. |

-| I-12 | **The epoch and moves.** | shared | Q11: refuse moves until `control/storage-v2.json` exists, or support §7.9's pre-epoch flow. Either way #199 §7's rollback floor (never below release N once the epoch exists) also protects moved records. |

-| I-13 | **One record listing per pass.** | shared | The moves pass's source and target roles share #199 §5.7's single `ListAllRecords`. Every destructive step re-reads its own record, as both specs already require. |

-| I-14 | **Reads during a move.** | #199 | The pull door (`resolveReplica`) refuses an instance in `landing` or `rolling_back`, because the pointer names a generation that is not yet authoritative (INV-4). In other states it serves the pointer, as #199 specifies. |

-| I-15 | **Staging wins, retired.** `promoteStagedReplica` (`backend_exec.go:1580`) is called from `Restore` (`:290`). | #199 removes; this spec agrees | It runs only for a legacy row on a pre-epoch deployment (§7.9). For v2 rows `Restore` never lists `moves/<host>/`. |

+| 1 | deadline does not prove the target stopped | Accepted. Execution right moves only on stop evidence (§5.3); a deadline only requests release (§7.8) |

+| 2 | probe writes are lost on rollback | Accepted. Clean/dirty verdict by digests; a dirty rollback carries the target's state back over a return lane checked by T10 (§7.8); INV-2 covers background and external work |

+| 3 | rollback opens before checking | Accepted. Recovery runs held as `restoring`, opens only after its probe; `rolling_back`/`stuck` never mint `ready` (§5.2, §7.8) |

+| 4 | no recovery between stop and flip | Accepted. Journaled `cut_phase`, `reverting`, bounded phases; admission and abort are one CAS field (§7.4) |

+| 5 | replayed landing mixes old files with a newer DB | Accepted. Journaled `landing_phase`, no replay after `probing`, generation markers and the start check, unique park destinations (§5.1, §7.6) |

+| 6 | clearing `move` before finalization | Accepted. `committed` keeps `move` through open and source park; opens are conditional on `hold_op` (§5.2, §7.9) |

+| 7 | rollback floor does not cover the landing fence | Accepted. A move-protocol floor with a box-enforced downgrade refusal (§11) |

+| 8 | P5 can delete a live generation | Phase 1 deletes nothing (INV-8) and parks under unique move ids; the deletion rules are F3's invariants (§16) |

+| 9 | P5 leaves versions behind | F3 invariant (§16) |

+| 10 | warrant does not bind its scope | F3 and F4 invariant (§16) |

+| 11 | member-minted token bypasses the owner gate | Phase 1 move mutations are operator-only (§12); delegation is F1's invariant (§16) |

+| 12 | private-store and rollback commit points conflict | Phase 1 refuses private stores (§7.1); the joint transaction is F4's invariant (§16) |

+| 13 | helpers exclude held rows | Accepted. Held render with move-bound acks, held materialization, `RetireExcept` exemption (§7.7, §9.1 to §9.3) |

+| 14 | readiness checks the wrong environment and process | Accepted. Merged-environment equality by HMAC digests, read/write store probe, process-bound health with an explicit readiness path, source probed too (§7.3, §7.7) |

+| 15 | timers neither frozen nor restored | Accepted. Runner freeze, no prune of held rows, source-readable snapshot, exact `ReplaceTimers` (§9.4) |

+| 16 | inventory misses registry state | Accepted. Every column classified; `Replication` and `NextDue` carried; `Caps` refused (§6.2) |

+| 17 | store record omitted | F4 invariant (§16) |

+| 18 | app-move child identity and store barrier | F5 invariant (§16) |

+| 19 | archive replay contradicts encryption | Accepted. Plaintext and sealed identities; one create-only sealed object per cut, reused on replay; retention stated (§8.3) |

+| 20 | cause still unconfirmed | Accepted. Confirmed from the UK, US and AU journals; remediation and fingerprint verification (§1.2, §10, §15) |

+Astra's interface dispositions on rev 1's §26 are superseded by #199 rev 3,

+which dropped storage keys, generations as record fields, the epoch and

+`Root: "moves"`. Their substance carries into rev 2: one key function

+(I-1 above), exclusion through finalization (§7.9), blocking reads during

+transitional states (F7 inherits it), no "staging wins" (§5.1), and the

+landed copy installed into `instances/` (M15).

+Astra's §23 recommendations matched rev 1's and Paul accepted them as M4 to

+M15, with Astra's additions folded in: M5's streaming enforcement (§8.1),

+M11's duration gate (T11), and M14's single landing path.