carlos / platform Public

spec: move-any-app rev 3, on the host lifecycle claim, after Astra round 2 (#175)

Answers round 2's six P1s and four P2s (dispositions in §22) and takes
rulings M16-M18 (declare-lost, the approved return lane, Tito waits).
Builds on the host lifecycle claim spec (653ad4b1 on
design-replica-ownership) and says how moves use it: op kinds and
per-phase admission, the execution holder in the claim journal, touched
before first write, obligations for every superseded copy, and eight
points where the claim spec needs to change or confirm (§19.3).

Phase 1 changes: stop evidence recomputed from system state with strict
spawns (no descendant can outlive it), a durable source-recovery journal
that never reinstalls after its probe, participant rows retained through
rollback, the rolled-back target's replica parked out of the live
namespace, install phases split so each marker is published only after
what it vouches for, staging subprefixes, a composed launch environment
with pinned input versions, every marker and the restore verdict
classified, and a whole-cut duration budget. The simulation is specified
to run jointly with the claim's model.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

claude 3 weeks ago 6e4d3a0e175f34614d4fa7f492e0590fe72ff636 parent e5b6751

1 file changed, +946 −848

docs/superpowers/specs/2026-10-02-move-any-app-design.md +946 −848

diff --git a/docs/superpowers/specs/2026-10-02-move-any-app-design.md b/docs/superpowers/specs/2026-10-02-move-any-app-design.md

index 5619052a9..492c62423 100644

--- a/docs/superpowers/specs/2026-10-02-move-any-app-design.md

+++ b/docs/superpowers/specs/2026-10-02-move-any-app-design.md

@@ -1,27 +1,24 @@

# Move any app or instance to another region: Phase 1, a safe single-instance move

**Date:** 2026-10-02

-**Revision:** 2.

-- Rev 1 (e41c4fef) went to Astra (GPT-6, via codex). The findings are in

- `CODEX-FINDINGS-MOVES-R1.md` beside this file, unedited (9fbf4afe):

- 12 P1, 7 P2, 1 P3, "not ready for Phase 1 planning".

-- Rev 2 answers every finding. Changes are marked "(codex N)", and §20

- records each disposition.

-- Rev 2 also takes three changes from outside the review:

- - Paul accepted every recommendation in rev 1's §23. They are rulings M4

- to M15 (§0.1).

- - #199 rev 2 was dropped. #199 rev 3 parks a deleted instance's data at

- delete time and changes no record field. This spec now **owns** the

- generation handoff, the cross-region execution fence, the cleanup of

- what a completed move leaves behind, and the carriage of the instance's

- files (§0.3, §17).

- - The cause of the missing config on the UK box is confirmed (§1.2).

-- **Scope cut.** Rev 2 specifies **Phase 1 only**: a safe, operator-driven

- move of one instance. Everything else (owner self-serve, moves to home,

- residency deletion, stores, app moves, hooks) is reduced to a list of

- follow-on specs, each with the invariants it must meet and no design

- (§16).

+**Revision:** 3.

+- **Rev 1** (e41c4fef) went to Astra (GPT-6, via codex): 12 P1, 7 P2, 1 P3,

+ "not ready". The findings are in `CODEX-FINDINGS-MOVES-R1.md` (9fbf4afe).

+- **Rev 2** (45b77691) answered them, cut the scope to Phase 1, took Paul's

+ rulings M4 to M15, and rebuilt on #199 rev 3. Round 2,

+ `CODEX-FINDINGS-MOVES-R2.md` (e5b67514), resolved or accepted the

+ deferral of most of round 1. It found 6 new P1s and 4 P2s: "not ready".

+- **Rev 3** answers round 2. Changes are marked "(codex R2-N)"; rev 2's

+ marks "(codex N)" stay. §22 records every disposition from both rounds.

+ Rev 3 also:

+ - takes Paul's rulings M16 to M18 (§0.1);

+ - builds on **the host lifecycle claim** that Paul ruled on 2026-10-02:

+ one object per host, updated by CAS, that serializes delete, park,

+ move, push, the db door and wake. It is specified in

+ `docs/superpowers/specs/2026-10-02-host-lifecycle-claim-design.md`

+ (rev 1 at 653ad4b1 on `design-replica-ownership`; "the claim spec").

+ §19 says how moves use it and where this spec asks it to change.

**Status:** DRAFT for review. Nothing here is built.

**Discussion:** amadan `carlos/platform` #175.

@@ -38,100 +35,113 @@ branch `design-nearest-region`.

- On `main` (7edd4b33): P0 (the cut fence), P7 (the edge door), and the

move-ready fix (16 MiB readiness bar, copy receipts for multipart ETags).

- #199 rev 3, `docs/superpowers/specs/2026-10-02-replica-ownership-design.md`

- at 66f1dad3 on `design-replica-ownership` (the "#199 spec"): host

- tombstones and parking at delete time.

+ at 66f1dad3 on `design-replica-ownership` (the "#199 spec"). It failed

+ review for the same reason as round 2's P1s; the claim spec supersedes it

+ from its §2 onward (tombstones become the claim's `deleting` and

+ `parking` operations).

+- The claim spec, `docs/superpowers/specs/2026-10-02-host-lifecycle-claim-design.md`

+ at 653ad4b1 on `design-replica-ownership`.

Citations are to this worktree at 7edd4b33 unless they name another branch

or repo.

-## 0. Rulings

+## 0. Rulings and the shared primitive

-### 0.1 Paul's rulings for this work (2026-10-02). Not reopened here.

+### 0.1 Paul's rulings (2026-10-02). Not reopened here.

| id | ruling | where Phase 1 spends it |

|---|---|---|

| M1 | **Who:** owners move their own apps and instances, self-serve, from the console and the CLI. Operators can do the same for any account. The platform refuses unsafe moves. | Phase 1 is operator-only (§12); owners come in F1 (§16) |

| M2 | **Downtime:** a short cut window (tens of seconds of 503), an automatic health check after the cut, and **automatic rollback to the source if the target will not start**. Not live migration. | §7 |

-| M3 | **Stores move with the app:** an app's object store is copied to the target region and rebound. | F4; Phase 1 leaves the app store where it is and refuses private stores (§7.1) |

-| M4 | The files the move carries are a fixed set (no app-declared paths). | §6.1 |

+| M3 | **Stores move with the app.** | F4; Phase 1 leaves the app store in place and refuses private stores (§7.1) |

+| M4 | The files the move carries are a fixed set. | §6.1 |

| M5 | The files archive is capped at 256 MiB, enforced while streaming and again at the cut. | §8.1 |

| M6 | Person-started moves cut automatically once ready and validated; the API default stays manual; `--manual-cut` exists. | §12 |

| M7 | The source copy is held for 7 days after a move; owners may end the hold early. | F3 |

| M8 | An app move pauses when one of its instances rolls back, and keeps the target app region. | F5 |

-| M9 | An app store's final copy holds every consumer of the store at 503, behind a durable barrier that covers concurrent creates. | F4, F5 |

+| M9 | An app store's final copy holds every consumer at 503, behind a durable barrier covering concurrent creates. | F4, F5 |

| M10 | An app with sidecars is refused an app move until sidecar moves exist. | F5 |

-| M11 | Target pre-warming is deferred; move eligibility is gated on measured restore and cut duration instead. | §7.3 (T11) |

+| M11 | Target pre-warming is deferred; eligibility is gated on the measured duration of the whole cut. | §7.3 (T11) |

| M12 | Rehearsal order: a probe app, then a disposable ips instance, then kpmg, each with scoped approval. | §14.4 |

-| M13 | Owners only, not members; the member-minted instance-token bypass is closed. | Phase 1 is operator-only (§12); F1 |

-| M14 | One landing path in production. (Ruled as "require #199's epoch". #199 rev 3 has no epoch; the intent stands: Phase 1 has exactly one landing path.) | §7.6 |

-| M15 | A landed copy is installed into the target's `instances/`, never served from `moves/` (Astra's I-1 verdict). | §5.1, §7.6 |

+| M13 | Owners only, not members; the member-minted instance-token bypass is closed. | §12; F1 |

+| M14 | One landing path in production. | §7.6 |

+| M15 | A landed copy is installed into the target's `instances/`, never served from `moves/`. | §5.1, §7.6 |

+| **M16** | **An operator-only `carlos move resolve --declare-lost <box>`, refused after commit, proceeds as a clean rollback after a confirmation naming what is lost.** | §7.10 |

+| **M17** | **The return lane is approved.** Every region's `sibling_staging_arns` lists every other region's data bucket, with `moves/*` Put and Delete and prefix-conditioned List only. | §7.3 (T10), §15 |

+| **M18** | **Tito waits.** Move mutations are operator-only; instance-role tokens keep only the status read. | §12 |

### 0.2 The 09-24 record's rulings

-They bind unless M1 to M15 override them:

-- **R4 ("rollback is a move back, never an automatic undo") is narrowed by

- M2.** A landing that fails before the target admits public traffic is

- undone automatically (§7.8). After the target opens, R4 stands: recovery

- is a move back, and nothing restores an older source copy over newer

- target data.

-- **Q5 (other accounts are operator-only until #171)** is consistent with

- Phase 1, which is operator-only.

-- Everything else binds as written.

-### 0.3 What #199 rev 3 changes here

-- **No record field carries storage identity.** Replica data stays at

- `instances/<host>` (`internal/activator/activator.go:138`). This spec

- therefore keys a move's generation by the move id, in object names and in

- marker files, never in a persistent record field (§5.1).

-- **Deletes write a host tombstone first**, at

- `control/host-tombstones/<host>/<incarnation>.json`, and the box in the

- tombstone's region parks `instances/<host>/` and `moves/<host>/` into

- `parked/<incarnation>/<host>/`. Reconcile defers a recreated host's row

- until its park is done (#199 §2, §3). A move must compose with all three

- (§7.12).

-- **The incarnation** is `provision.Incarnation(rec)`: `RecordIdentity` over

- account, app, host and CreatedAt, or a domain-separated legacy identity

- when CreatedAt is zero (#199 §2). Every move artifact is bound to it.

-- **The source-region lineage a completed move leaves behind, and the

- instance's working directory, are this spec's** (#199 §2.5b of rev 2 and

- §5 of rev 3).

+They bind unless M1 to M18 override them. **R4** ("rollback is a move

+back") is narrowed by M2: a landing that fails before the target admits

+traffic is undone automatically (§7.8); after the target opens, R4 stands.

+**Q5** (other accounts operator-only until #171) is consistent with Phase 1.

+### 0.3 What #199 contributes

+- Replica data stays at `instances/<host>`

+ (`internal/activator/activator.go:138`). No record field carries storage

+ identity, so this spec names a generation by the operation id that

+ installed it, in object names and marker files (§5.1).

+- The incarnation, `provision.Incarnation(rec)`, including the

+ zero-CreatedAt legacy identity, and the park helpers (copy with a manifest

+ then delete; local rename), now specified in the claim spec §2 and §5.

+### 0.4 The host lifecycle claim

+From the claim spec (§2 to §9 there), the parts this spec relies on:

+- **The object** `control/hosts/<host>.json`, rewritten whole by CAS: a

+ per-write version `gen`; a `holder` (family, record key, incarnation); an

+ `op` (`kind`, `id`, `phase`, `journal`); `touched` regions and boxes,

+ appended **before** the first write there; and `obligations[]`, durable

+ chores that must finish before the host can become free.

+- **Its invariants:** C1 one holder; C2 claim before record on create,

+ `DeleteIfMatch` under the claim on delete; C3 destructive steps only by

+ the current op id, holding the location's lease, before `SafeUntil`

+ (§3.1 there states the clock assumption); C4 admission reads the claim

+ fresh; C5 revocation needs stop evidence; C6 a read error never acts.

+- **Op kinds** (§4 there): `moving`, `landing` and `reverting`/`restoring`

+ belong to this spec, including the admission rule for each phase. One

+ operation runs at a time, so a delete and a move refuse each other.

+- **Stop evidence** (§6 there) recomputed from system state.

+- **Retained participant rows** (§8 there) and the protocol floor (§9

+ there, which reuses §11's object).

+This spec's rules for using the claim are K1 to K13 in §19, named "K" so

+they never read as the claim spec's own C1 to C6. Where this spec needs

+the claim spec to change, §19.3 says so. "Execution holder" below means

+the claim's `op.journal.exec` (K1).

## 1. What happened in production

### 1.1 Finding 1: busy instances never reached `ready` (fixed on main)

-Multipart ETags never matched after `CopyObject`, so every round re-copied

-everything. Fixed in 2cbf6196 and 167d721b (merged as 7edd4b33): copy

-receipts (`internal/activator/backend_exec.go:1557`) and a 16 MiB

-readiness bar (`internal/host/movespass.go:215-235`, applied at `:387`).

+Multipart ETags never matched after `CopyObject`. Fixed in 2cbf6196 and

+167d721b (merged as 7edd4b33): copy receipts

+(`internal/activator/backend_exec.go:1557`) and a 16 MiB readiness bar

+(`internal/host/movespass.go:215-235`, applied at `:387`).

### 1.2 Finding 2: a completed move could not start for 1h54m

-`kpmg.bdn.oncarlos.com` (account bdn, app ips) moved IE (home) to uk. Every

-wake on the UK box then failed with `instance not healthy within 30s`

-(`backend_exec.go:1443`). Two causes, both now confirmed.

+`kpmg.bdn.oncarlos.com` (account bdn, app ips) moved IE (home) to uk, and

+every wake on the UK box then failed with `instance not healthy within 30s`

+(`backend_exec.go:1443`). Both causes are confirmed.

**(a) The per-host data directory does not move.** The cut mirrors the

replica (`movespass.go:476-490`), purges database files only

(`backend_exec.go:1566-1575`), and flips the record (`movespass.go:494`).

-The app keeps `instance.key` and `report.token` in `/data/<host>/`. No

-platform code names that directory. On the UK box the app minted a new

-key and could not read data sealed under the old one.

-**(b) Config never rendered, because only the Ireland box holds

-`CARLOS_SECRETS_KEY` (codex 20).** The UK, US and AU journals all log

-"config sync loop disabled: CARLOS_SECRETS_KEY not set"

-(`internal/host/agent.go:884`). Without the key both config loops return

-at once (`agent.go:881-886`, `:1279`), the fast converge skips the config

-leg (`agent.go:1812`), and the spawn treats a missing config file as empty

-(`backend_exec.go:1146-1149`). So the app started with no store binding.

-**Remediation (Paul approved, ops in progress):** distribute the key to

-every region box. **Verification** is the fingerprint every box publishes

-(§10): a region is ready when its boxes publish the console's fingerprint.

-Readiness check T2 refuses any target that does not (§7.3).

+The app keeps `instance.key` and `report.token` in `/data/<host>/`, which no

+platform code names. On the UK box the app minted a new key.

+**(b) Only the Ireland box holds `CARLOS_SECRETS_KEY` (codex 20).** The UK,

+US and AU journals log "config sync loop disabled: CARLOS_SECRETS_KEY not

+set" (`internal/host/agent.go:884`). Both config loops return at once

+(`agent.go:881-886`, `:1279`), the fast converge skips its config leg

+(`agent.go:1812`), and the spawn treats a missing config file as empty

+(`backend_exec.go:1146-1149`). Paul approved distributing the key to every

+region box (ops, in progress). The fingerprint every box publishes (§10)

+verifies it, and T2 refuses any target without it.

### 1.3 Finding 3: the rehearsal proved nothing

@@ -140,761 +150,784 @@ an app shaped like ips.

### 1.4 Finding 4: SSM Parameter Store is regional

-"SSM Parameter Store is regional: this host reads /carlos/* in ITS OWN

-region ... Nothing replicates them from eu-west-1" (carloku-infrastructure

-`region-uk.tf:61-64`). §10 traces what an instance needs from it: only the

-box's secrets key.

+carloku-infrastructure `region-uk.tf:61-64`: nothing replicates `/carlos/*`

+between regions. §10 shows that an instance needs only the box's secrets

+key from it.

## 2. Phase 1: what it does and what it refuses

-Phase 1 moves **one** exec-backed, hibernating `kind:"instance"` record from

-one non-fleet placement to another, driven by an operator through the

-console API or `carlos move`. It carries every piece of per-host state the

-instance needs, proves the target can run it before the source stops,

-starts it on the target with no public traffic, and opens only after a

-commit. If the target cannot start it, the instance goes back to the

-source automatically, carrying anything the target wrote.

-Phase 1 refuses everything §7.1 lists, including moves into home (F2),

-private stores (F4), and any caller that is not an operator (F1).

-Phase 1 does **not** delete anything. What a completed move leaves in the

-source region is parked under the move id, so it is enumerable, never read

-again, and ready for the residency cleanup in F3.

+Phase 1 moves **one** exec-backed, hibernating `kind:"instance"` record

+between two non-home, non-fleet placements or out of home, driven by an

+operator. It carries every piece of per-host state, proves the target can

+run it before the source stops, starts it on the target with no public

+traffic, and opens only after a commit. If the target cannot start it, the

+instance returns to the source automatically, carrying anything the target

+wrote. Phase 1 refuses everything §7.1 lists. It deletes nothing: what a

+move supersedes is parked under the operation's id, out of every live

+namespace, ready for F3's cleanup.

## 3. Words

-- **Source / target:** the box serving the instance before the move, and

- the box serving it after. "Box" means the one box whose registry owns the

- route (`OwnsRoute`, `movespass.go:180-185`).

-- **Move id:** console-minted, unique, `mv_` plus 26 lowercase base32

- characters. It names the move's **generation**.

-- **Generation:** the identity of one installed copy of an instance's state

- (database, files, timers). An instance that has never moved is in the

- **origin** generation. A landing installs generation `<move id>`. A dirty

- rollback installs generation `<rollback id>` (§7.8).

-- **Handoff:** the protocol that transfers the right to run an instance

- from one box to another, with a new generation (§5). A move is a handoff.

- A dirty rollback is a handoff back. A future `carlos db push` is a

- handoff on one box.

-- **Hold:** a registry row status that admits no traffic, no ticks and no

- ordinary wakes (§5.2).

-- **The flip:** the record CAS that sets `region` to the target.

-- **Commit:** the record CAS after which the target is authoritative.

-- **Open:** the target's row becomes `ready`.

+- **Source / target:** the box serving before the move, and after. "Box"

+ means the one box whose registry owns the route (`OwnsRoute`,

+ `movespass.go:180-185`).

+- **Move id / rollback id:** console-minted, unique, `mv_` or `rb_` plus 26

+ lowercase base32 characters. Each names one **operation**.

+- **Generation:** one installed copy of an instance's state (database,

+ files, timers), identified by `(op id, incarnation)`. An instance that has

+ never moved is in the **origin** generation (no markers anywhere).

+- **Handoff:** the transfer of the execution right from one box to another,

+ with a new generation (§5). A move is a handoff; a dirty rollback is a

+ handoff back; a future `carlos db push` is a handoff on one box.

+- **Hold:** a registry row status that admits no traffic, ticks or ordinary

+ wakes (§5.2).

+- **Flip / commit / open:** the CAS that sets `region` to the target; the

+ CAS after which the target is authoritative; the target row becoming

+ `ready`.

## 4. Invariants

Stated first, designed to, and proved by the simulation in §14.2.

-- **INV-1 One writer.** At any instant at most one process, on any box in

- any region, runs the instance with write access to its database, files

- or store. A box starts the instance only while holding the execution

- right (§5.3), and the right moves between boxes only on **stop

- evidence** from the box giving it up. A timeout never moves it (codex 1).

+- **INV-1 One writer.** At any instant at most one process tree, on any box

+ in any region, runs the instance with write access to its database, files

+ or store. A box starts the instance only while the claim names it as the

+ execution holder, and the right moves only on **stop evidence** proving

+ the previous holder's whole process tree is gone (§5.3). A timeout never

+ moves it (codex 1, R2-1).

- **INV-2 No lost acknowledged write.** No write the instance acknowledged,

- to a client, to a tick, or to its own background work or an external

- system, is lost by any outcome: done, aborted, rolled back, or stuck. In

- particular a rollback carries back whatever the target wrote (codex 2).

-- **INV-3 Convergence without a human.** A crash of any actor (a console

- or agent process restart, or a box reboot) at any step leaves a state

- from which the move reaches a terminal outcome with no human action. The

- one exception is stated, not hidden: a box that **never returns**

- (destroyed) during a handoff it holds; that leaves a paged, recoverable

- state and an operator verb (§7.10).

-- **INV-4 Files match the database.** Every start of the instance, on any

- box, uses a files directory whose generation equals the generation of the

- database it starts with. A mismatch refuses the start (§5.1).

+ to a client, a tick, its own background work or an external system, is

+ lost by any outcome. A rollback carries back whatever the target wrote,

+ and a source recovery never reinstalls over its own new writes (codex 2,

+ R2-3). M16's `--declare-lost` is the one disclosed exception.

+- **INV-3 Convergence without a human.** A crash of any actor (a console or

+ agent process restart, or a box reboot) at any step leaves a state from

+ which the move reaches a terminal outcome with no human action. This

+ holds even when the crashed activator has lost all memory (codex R2-2).

+ The exception is a box that never returns, handled by M16 (§7.10).

+- **INV-4 Files match the database.** Every start, on any box, uses a files

+ directory whose generation equals its database's generation, which

+ equals the generation the claim says is installed on that box. Otherwise

+ the start is refused (§5.1).

Supporting invariants:

- **INV-5 No admission before commit.** A row leaves a hold only by a

- conditional write naming the exact operation that holds it, and the

- target's hold is released only after the commit (codex 3, 6).

-- **INV-6 The environment that runs is the environment that was checked.**

- The readiness checks evaluate the final merged launch environment, and

- the probe binds health to the spawned process (codex 14).

-- **INV-7 Install is never replayed over writes.** Once a probe has started

- a process on an installed generation, nothing re-copies, re-extracts or

- re-imports into it (codex 5).

-- **INV-8 Phase 1 deletes nothing a recovery might need.** Data a move

- supersedes is parked under its move id, never deleted (codex 8).

+ conditional write naming the operation that holds it (codex 3, 6).

+- **INV-6 The environment that runs is the environment that was checked,**

+ from inputs whose versions were pinned at readiness (codex 14, R2-8).

+- **INV-7 Install is never replayed over writes,** on the target or the

+ source; and a completion marker is published only after everything it

+ vouches for is durable (codex 5, R2-3, R2-6).

+- **INV-8 Phase 1 deletes nothing a recovery might need** (codex 8).

- **INV-9 Bounded.** Every non-terminal state has a deadline after which it

- either advances, recovers, or pages in a state that still satisfies

- INV-1 and INV-2.

+ advances, recovers, or pages in a state that still satisfies INV-1 and

+ INV-2. Recovery deadlines are separate from the promised cut window

+ (codex R2-10).

+- **INV-10 Incarnation isolation.** When a move or rollback ends, no copy

+ of the instance's data remains in a live host namespace

+ (`instances/<host>/`, `moves/<host>/`, the flat DB path, the files

+ directory) on any box or bucket except the one the claim names as

+ current. Everything else is parked under the operation id (codex R2-5).

## 5. The handoff protocol

-The two primitives this spec owns, and that a later `carlos db push` builds

-on: the generation handoff and the execution fence.

+### 5.1 Generations and markers (codex R2-6, R2-9)

-### 5.1 Generation markers

+A generation is named in three markers. None is a record field.

-A generation is named in three places. None is a record field, so no old

-writer can erase one (#199 rev 3's lesson).

|---|---|---|---|

-| `generation.json` `{"gen","incarnation","host","installed_at","box"}` | `instances/<host>/generation.json` in the box's own data bucket, beside the existing `lock.json` | the installing box | last step of an install (§7.6, I3), after every object is verified |

-Absent everywhere means **origin**, so every instance that has never moved

-is unchanged.

-**The start check (INV-4).** Before every spawn (`Start`, `Seed`, and the

-probe), the activator compares `<dbPath>.gen` with `.files-gen`, when the

-files directory exists. Unequal refuses the wake with a member-safe reason

-("this instance's files and database come from different copies; an

-operator has been paged") and pages. A cold restore writes `<dbPath>.gen`

-from the restored prefix's `generation.json` before the comparison.

-Where the database is warm on a box but its generation differs from the

-bucket's `generation.json`, the warm file is from an older copy (an earlier

-stay on this box). The activator parks it (§6.3) and restores cold. This

-replaces "staging wins" (`backend_exec.go:1581-1588`): staging is never

-restored from directly (M15), and `promoteStagedReplica` is deleted.

+| `generation.json` | `instances/<host>/generation.json`, beside `lock.json` | **last**, after the install manifest is complete and verified (§7.6) | `{"op","incarnation","host","box","installed_at","manifest_sha256"}` |

+| `<dbPath>.gen` | beside the local database | after a restore completes, from `generation.json` | `{"op","incarnation"}` |

+| `.files-gen` | inside the files directory | **last**, after files, `<host>.env`, timers and the carried registry columns are all durable (§7.6) | `{"op","incarnation"}` |

+All three are **protocol metadata**: excluded from the archive, from every

+content digest, and from F4's definition of the files directory (§6.1).

+Absent everywhere means origin. The claim's `gen` is a per-write version

+(claim spec §2), not a generation, so it never appears in a marker.

+**The start check (INV-4).** Before every spawn (`Start`, `Seed`,

+`ProbeWake`), the activator requires `<dbPath>.gen`, `.files-gen` (when

+the files directory exists) and the box's own `instances/<host>/generation.json`

+to name the same op and incarnation, and the incarnation to be the claim's

+holder (claim spec C4). Absent in all three is origin. A marker that

+merely exists proves nothing (codex R2-6). A mismatch refuses the wake

+with a member-safe reason and pages. The bucket read rides the wake's

+existing lease round trip to the same prefix.

+A warm local database whose `<dbPath>.gen` differs from the bucket's

+`generation.json` is from an earlier stay on this box. The activator parks

+it (§6.3) and restores cold. This replaces "staging wins"

+(`backend_exec.go:1581-1588`); `promoteStagedReplica` is deleted (M15).

### 5.2 Holds

-A hold is a registry row status that the edge does not serve

-(`internal/edge/edge.go:595` serves only `ready`), that ticks refuse

-(`activator.go:2785-2789` refuses any non-`ready` row), and that no

-ordinary wake reaches. Phase 1 has three hold statuses: `moving` (source,

-from the cut until the move ends), `landing` (target, until it opens), and

-`restoring` (source, during a rollback).

-**Every hold names its operation.** The `routes` table gains an additive

-column, `hold_op`, set to the move id (or rollback id) whenever a hold

-status is written. Leaving a hold is one conditional statement:

+A hold status is one the edge does not serve (`internal/edge/edge.go:595`),

+ticks refuse (`activator.go:2785-2789`), and no ordinary wake reaches:

+`moving` (source), `landing` (target), `restoring` (source, recovering),

+`releasing` (target, during a rollback). The `routes` table gains an

+additive column, `hold_op`, set whenever a hold is written. Leaving a hold

+is one conditional statement:

```sql

UPDATE routes SET status='ready', hold_op='' WHERE host=? AND hold_op=?

```

-A delayed or duplicate open from a finished operation matches nothing, so

-it can never reopen a newer operation's hold (codex 6). Reconcile, which

-today mints every row `ready` (`internal/provision/reconcile.go:364-376`),

-mints a row whose record is mid-move in the hold its role requires, with

-`hold_op` set (§9.6). A row for a record in `rolling_back` or `stuck` is

-never minted `ready` (codex 3).

-### 5.3 The execution right and stop evidence

-The right to run the instance belongs to exactly one box, named by the move

-state:

-| move state | holder |

-|---|---|

-| no move, `preparing`, `ready`, `cutting` (every phase), `reverting` | source. After `stopping` it holds the right without using it, so a pre-flip recovery needs nobody else's evidence |

-| `landing`, `committed` | target. The flip requires the source's stop evidence (§7.4), and the target reads and checks it before it starts anything (§7.6) |

-| `rolling_back` | nobody until the target's stop evidence; then source |

-**Stop evidence** is what the activator's `FenceAndStop` already produces

-(`internal/activator/maintenance.go:50`, `StopEvidence.Matches`): the main

-process stopped and exited, its sidecar stopped, its replica closed, its

-lease released. The box that gives up the right writes it to the

-deployment bucket at

-`moves/<acct>/<app>/<host>/<move id>/stopped-<role>.json` with its box

-label, the incarnation, the database file's sha256 after a final

-checkpoint, the files digest and the timers digest (§7.4, §7.8). The box

-that takes the right reads it and checks the move id, incarnation and role

-before it starts anything.

-**Why this survives a crash.** The agent runs as `carlos-edge.service`

-with systemd's default `KillMode=control-group` (no `KillMode` line in

-`internal/host/units/carlos-edge.service`), so a flat exec child dies with

-the agent. A nested child runs in its own `carlos-instance@` unit

-(`internal/host/units/carlos-instance@.service`) and survives, so a

-restarted agent stops it through the instance-control channel before it

-writes evidence. Either way a restarted box can produce evidence. A box

-that never comes back cannot, and that is INV-3's stated exception

-(§7.10).

-**What does not count as evidence:** a deadline, a lost CAS, a silent box,

-or a lease in another bucket. Leases are per bucket and prefix

-(`backend_exec.go:228-231`), so the source's and target's leases never

-exclude each other (codex 1).

+A late write from a finished operation matches nothing (codex 6). Reconcile

+mints held rows for every participant the claim names and never mints

+`ready` while the claim holds a move operation (`moving`, `reverting`,

+`landing` before the commit, or `restoring`) (§9.6, §19 K9).

+### 5.3 Stop evidence (codex 1, R2-1, R2-2)

+The execution right moves only on stop evidence. Rev 2 relied on the

+activator's in-memory `FenceAndStop` (`internal/activator/maintenance.go:50`).

+Round 2 showed two gaps: a flat child's descendant that outlives it

+(`childProc.Stop` waits for the main process only, `backend_exec.go:1502-1543`),

+and evidence that cannot be produced after an agent restart, because

+`FenceAndStop` returns `ErrStopUnproven` when it has neither an entry nor a

+remembered stop history (`maintenance.go:90-100`). The claim spec §6

+defines evidence the same way this section does, from system state; this

+section adds the strict-spawn and stray-process conditions it still lacks

+(§19.3 D3). Rev 3 defines evidence

+from **durable, observable state**, never from memory.

+**Every spawn under a move is strict.** Flat children already run in a

+per-host cgroup, `carlos-exec@<host>` under the agent's delegated subtree

+(`internal/cgroup/cgroup.go:37`, `:243`; `Delegate=` in

+`internal/host/units/carlos-edge.service`), but placement is best-effort:

+a failed placement spawns "unaccounted" in the agent's own cgroup

+(`cgroup.go:248-252`). The strict path exists

+(`internal/cgroup/strict_linux.go`: `PrepareStrict`, `Populated`,

+`KillAndEmpty`). Under a move, every spawn of the host (the source's

+recovery, the target's probe, and the source's ordinary wakes while the

+claim says `moving`) uses `PrepareStrict` and refuses to spawn if it

+cannot. A nested child already runs in its own `carlos-instance@<iid>`

+unit.

+**The source's pre-move child.** Its cgroup placement was best-effort. The

+source's readiness (§7.3, S2) therefore requires its current or next child

+to be strict: before `ready`, it stops an awake non-strict child (an

+ordinary idle stop) so the next wake is strict.

+**Evidence** is `ReconstructStopEvidence(host)`, a new activator call that

+reads only the box's current state, so it gives the same answer after an

+agent restart as before:

+1. **Kill and drain the tree.** Flat: write `cgroup.kill` on

+ `carlos-exec@<host>` and wait until `cgroup.events` says

+ `populated 0` (`KillAndEmpty`). Nested: stop `carlos-instance@<iid>`

+ (and its seed and sidecar units) through the instance-control channel

+ and require each unit inactive and its cgroup unpopulated.

+2. **No stray process.** The agent's own cgroup (`cgroup.procs` of the

+ service's leaf) holds only the agent's own process. A strict spawn never

+ leaves anything there; anything else is unattributable and refuses

+ evidence with an alert.

+3. **No open database.** No process on the box holds the instance's

+ database, WAL or SHM open (a scan of `/proc/*/fd` for the paths,

+ restricted to the `carlos` uid).

+4. **Lease released.** The host's `lock.json` lease

+ (`internal/lease/lease.go:86`) is absent, released, or expired by more

+ than the lease's renewal margin.

+5. **Replica final.** After 1 to 4, a final checkpoint of the local

+ database (if any) into the box's own `instances/<host>`, and its

+ sha256.

+**A durably never-spawned operation.** If the box's journal for the

+operation shows no spawn was ever attempted (the target's `landing_phase`

+before `probing`), steps 1 to 4 still run: an empty tree proves the same

+thing whether or not a spawn happened. "No entry in memory" is never

+evidence on its own (codex R2-2).

+The box writes the result to the deployment bucket at

+`moves/<acct>/<app>/<host>/<op id>/stopped-<role>.json`: box label,

+incarnation, the claim `gen` it read, each step's observation, and the database, files

+and timers digests. It then CASes the claim's execution holder to `none`

+(§19 K3). The next holder reads the evidence and the claim before it starts

+anything.

## 6. State inventory

-### 6.1 Files on a box

+### 6.1 Files on a box (codex R2-9)

|---|---|---|---|

-| F4 | **files directory** | flat: `strings.TrimSuffix(dbPath, ".db")` = `<DataDir>/<host>/` (the app's `instance.key`, `report.token`); nested: every entry in `InstanceDir` except F1 to F3, F5, F6, F8, F9, F10 and #199's markers | **carried** in the archive (§8) |

-Anything else an app writes under `/data` is not carried (M4). `docs/site`

-says so.

+| F4 | **files directory** | flat: `strings.TrimSuffix(dbPath, ".db")` = `<DataDir>/<host>/`; nested: every entry in `InstanceDir` except F1 to F3, F5 to F11 and #199's markers | **carried** (§8), excluding `.files-gen` |

### 6.2 Registry state (codex 16)

-Every `routes` column (`internal/registry/registry.go:34`), classified:

+Every `routes` column (`internal/registry/registry.go:34`):

| column(s) | class | Phase 1 |

|---|---|---|

-| Host, Slug, Kind, App, Account, Region, Channel, Environment, Hibernate, StatusNamed, Steering, Gate, Seed | reproduced | from the record by `routeFor` (`reconcile.go:364-376`) |

-| Socket, DBPath | reproduced | derived on the target for its own layout |

-| Provisioned | reproduced | true on mint |

-| Status | protocol | the hold statuses of §5.2 |

-| hold_op (new) | protocol | §5.2 |

-| **Replication** | **carried** | box-local, absent from records, preserved by every path today (`registry.go:138-145`, `reconcile.go:1100-1113`); read from the source row at the cut, written to the target row at install |

-| **NextDue** | **carried** | the app's own next scheduled wake, probed at stop (`activator.go:2019-2030`); read from the source row after its stop, written to the target row at install, so the due-wake sweep still wakes it |

-| Version, PrevVersion | reset | the target materializes the channel head itself (§9.2) |

-| LastError, DNSStatus, DNSCheckedAt, CertExpiresAt, CreatedAt | reset | recomputed by their own passes |

-| Caps | **refused** if non-empty | box-local authority; not carried implicitly |

-| Unit, Backing (unit), Platform, Tunnel, FleetBox, Pool, Canary | **refused** if set | not the platform's to move |

-| Addr, Addrs, Health | **refused** if set | addr-backed |

+| Host, Slug, Kind, App, Account, Region, Channel, Environment, Hibernate, StatusNamed, Steering, Gate, Seed | reproduced | from the record by `routeFor` (`internal/provision/reconcile.go:364-376`) |

+| Socket, DBPath | reproduced | derived for the target row's layout; **Phase 1 lands flat** (`routeFor` mints flat) |

+| Provisioned | reproduced | true |

+| Status, hold_op (new) | protocol | §5.2 |

+| **Replication** | **carried** | box-local (`registry.go:138-145`, `reconcile.go:1100-1113`) |

+| **NextDue** | **carried** | the app's own next wake, probed at stop (`activator.go:2019-2030`) |

+| Version, PrevVersion | reset | materialized for the held row (§9.2) |

+| LastError, DNSStatus, DNSCheckedAt, CertExpiresAt, CreatedAt | reset | recomputed by their passes |

+| Caps, Unit, Backing (unit), Platform, Tunnel, FleetBox, Pool, Canary, Addr, Addrs, Health | **refused** if set | not the platform's to move |

| Replica, Replicas | **refused** | a replica, or a home with replicas (Q2) |

| IdleSeconds, MaxRunSeconds, Args | n/a | sidecar columns |

-The `timers` table (`registry.go:565-579`) is carried as a snapshot (§7.4,

-§9.4).

+The `timers` table (`registry.go:565-579`) is carried as a snapshot (§9.4).

+### 6.3 Other host-keyed state (codex R2-9)

+| state | where | disposition |

+|---|---|---|

+| restore-verification verdict | `control/restore-verify.json`, `hosts[<host>]` (`internal/dbverify/dbverify.go:10`, `:33`) | the host's entry is **removed** at every install (landing and dirty rollback), so an earlier generation's "ok" never vouches for a new one; the next drill writes a fresh verdict |

+| config ack | `control/config-ack/<box>/<host>.json` (`internal/appconfig/configack.go:111-118`) | move-bound acks (§9.1); the source's parked at finalization |

+| copy receipts | `move-receipts/<host>/<op id>.json` (source bucket) | keyed by operation |

+| snapshots | `move-snapshots/<host>/<op id>/` (source bucket) | §8.3 |

-### 6.3 Parking on a box and in a bucket

+### 6.4 Parking

-Phase 1 reuses #199's two park mechanisms with a move-scoped destination:

+Phase 1 reuses #199's park helpers with an operation-scoped destination:

-- **Bucket park** of a prefix: per-key copy into the destination, then a

- manifest of every key and its size, then delete the source keys named in

- the manifest (#199 §3.3). Never `MirrorPrefix`, which deletes

- destination-only objects (`internal/lease/mirror.go:222`).

-- **Local park:** `rename` into `<DataDir>/.parked/<name>/<host>/` (#199

+- **bucket park:** per-key copy, then a manifest of every key and size,

+ then delete the keys named in the manifest (#199 §3.3). Never

+ `MirrorPrefix`, which deletes destination-only objects

+ (`internal/lease/mirror.go:222`).

+- **local park:** `rename` into `<DataDir>/.parked/<name>/<host>/` (#199

§3.4).

-Destinations: `superseded/<move id>/<host>/` in the bucket and

-`.parked/superseded-<move id>/<host>/` on the box. A move id is unique, so

-two parks can never collide, and no park destination is the ancestor of a

-live prefix (codex 8).

+Destinations are `superseded/<op id>/<host>/` and

+`.parked/superseded-<op id>/<host>/`. An op id is unique, so no two parks

+collide and no park destination is an ancestor of a live prefix (codex 8).

+**A park is its own journaled phase**, completed (manifest written and

+every named key deleted) before any install into the vacated namespace

+begins (codex R2-6).

## 7. The move

### 7.1 Request and refusals

`POST /api/cli/apps/{account}/{app}/instances/{host}/move`

-`{"region":"uk","cut":"auto"|"manual"}`, operator-only in Phase 1 (§12).

-Refused (400 or 409, with the sentence), at the door and again by every box

-step, since records have other writers:

+`{"region":"uk","cut":"auto"|"manual"}`, operator-only (§12). The door

+CASes the claim from `serving` (holder = this incarnation, execution

+holder = the source box) to `moving` with op id = the move id (§19 K2). It

+is refused (400 or 409, with the sentence), and every box step re-checks:

- not exec-backed `kind:"instance"`, addr-backed, pooled, fleet-placed,

non-hibernating (`internal/console/apiinstancemove.go:53-72`);

-- a replica (`:187-190`, `provision.go:978-980`), or **a home with

- replicas** (new; neither the door nor `MoveRequest` checks `Replicas`

- today);

-- a target of home (`apiinstancemove.go:203-209`; F2), or the current

- region, or one outside the manifest's `regions` allow-list

- (`apiinstances.go:1797`; also how an app that needs out-of-band

- host-role grants is pinned to its region);

-- a legacy `<host>-sidecar.env` on the source, or any refused registry

- column (§6.2);

-- **a private store** bound to the instance's environment (#412): its

- cutover needs a joint transaction (codex 12; F4). An **app-level store**

- stays where it is and the moved instance reads it across regions; T4

- proves it can;

-- a move already in flight, or a host tombstone for this host in either

- region that is not yet `parked` (§7.12);

-- any box in the source or target region below the protocol floor (§11).

-`MoveRequest` (`provision.go:966-983`) writes `move` with the fields of

-§7.2 and state `preparing`.

+- a replica (`:187-190`, `provision.go:978-980`), or a home with replicas

+ (new);

+- a target of home (`apiinstancemove.go:203-209`; F2), the current region,

+ or one outside the manifest's `regions` (`apiinstances.go:1797`);

+- a source `<host>-sidecar.env`, or any refused registry column (§6.2);

+- a private store bound to the instance's environment (codex 12; F4);

+- the claim not `serving`, or any box in either region below the protocol

+ floor (§11).

### 7.2 Record fields

-All additive under the existing `move` object (`provision.go:907-917`).

-Old binaries are kept away by the protocol floor (§11).

+Additive under `move` (`provision.go:907-917`). The claim carries the

+cross-operation serialization (§19); the record carries this move's phase

+journal, because only a move reads it.

```json

"move": {

- "id": "mv_…", "incarnation": "…", "from": "", "to": "uk",

- "state": "landing", "cut": "auto",

- "requested_by": "operator:paul@…", "requested_at": "…", "updated_at": "…",

+ "id": "mv_…", "incarnation": "…", "from": "", "to": "uk", "cut": "auto",

+ "state": "landing", "requested_by": "…", "requested_at": "…", "updated_at": "…",

"ready_at": "…",

- "cut_phase": "flipped", "cut_at": "…",

- "source": {"replication": "stream", "next_due": 0,

- "db_sha256": "…", "files": {"plain": "sha256:…", "sealed": "sha256:…", "bytes": 1824, "entries": 4},

- "timers_sha256": "…"},

+ "inputs": {"bundle_etags": {"…": "…"}, "overlay_etag": "…", "release": "…",

+ "files_plain": "sha256:…", "env_report": "sha256:…"},

+ "cut_phase": "flipped", "cut_at": "…", "cut_budget_s": 95,

+ "source": {"replication": "stream", "next_due": 0, "db_sha256": "…",

+ "files": {"plain": "sha256:…", "sealed": "sha256:…", "bytes": 1824, "entries": 4},

+ "timers_sha256": "…", "evidence": "…/stopped-source.json"},

"flipped_at": "…", "landing_deadline": "…",

- "landing_phase": "probing",

- "landing": {"installed_db_sha256": "…", "probe_attempts": 1},

+ "landing_phase": "probing", "landing": {"installed_db_sha256": "…"},

"committed_at": "…", "opened_at": "…", "source_parked_at": "…",

- "rollback": {"id": "rb_…", "requested_at": "…", "reason": "…", "dirty": true,

- "phase": "returned", "release_deadline": "…"},

+ "rollback": {"id": "rb_…", "requested_at": "…", "reason": "…",

+ "target_phase": "returned", "dirty": true, "evidence": "…",

+ "source_phase": "probing", "recovered_at": "…", "opened_at": "…",

+ "target_parked_at": "…", "release_deadline": "…"},

"reason": "…"

}

```

-Every box-side transition is a CAS guarded by `move.id` and

-`move.incarnation`, so a recreated host's record (a new incarnation) never

-matches an old move's step (§7.12).

+Every box transition is a CAS on the record guarded by `move.id` and

+`move.incarnation`, **and** revalidates the claim's op id, kind and phase

+first (§19 K5).

### 7.3 Preparing and readiness

-**Replica rounds (source)**, as on main (`movespass.go:372-394`), into the

-target's `moves/<host>/<move id>/`, with receipts at

-`move-receipts/<host>/<move id>.json` in the source bucket. Keying staging

-and receipts by move id is the generation handoff's bucket half: a stale

-staging prefix or receipt from another move is never listed.

+**Staging layout (codex R2-7).** Each artifact class has its own

+subprefix under `moves/<host>/<move id>/` in the target bucket:

-**Snapshot rounds (source).** Each round the source builds the files

-archive (§8) from the live directory and uploads it when its plaintext

-digest changes, and publishes its own **launch report** to

-`moves/<acct>/<app>/<host>/<move id>/source.json`: the names of every

-variable in its final merged launch environment

-(`ExecBackend.instanceEnv`, `backend_exec.go:1135-1153`, plus the agent-owned

-variables), each with an HMAC-SHA256 of its value under a key derived (HKDF)

-from `CARLOS_SECRETS_KEY`. The report reveals names and equality, never

-values, and only a box holding the key can compute a matching digest.

-**Readiness report (target).** The target box's move worker runs these

-checks for every record whose `move.to` is its placement, while the source

-keeps serving, and writes the result to

-`moves/<acct>/<app>/<host>/<move id>/target.json`:

-| check | passes when | finding |

+| subprefix | content | writer |

|---|---|---|

-| T1 protocol | this box runs at or above the protocol floor (§11) | codex 7 |

-| T2 secrets key | this box publishes `secrets_key_fp` equal to the console's (§10) | §1.2b |

-| T3 environment | the target renders the bundle for the instance's environment and overlay through the held-render API (§9.1), opens the newest archive, merges exactly as `instanceEnv` will at spawn (`<host>.env` over the rendered config, `backend_exec.go:1231-1246`), adds the agent-owned variables, and every name in the source's launch report is present with an equal value digest, except `CARLOS_REGION`. A variable from `<host>.env` that shadows a rendered one is listed in the report | codex 14 |

-| T4 storage | for each store bound in that **merged** environment (`<PREFIX>_BUCKET/_REGION/_ENDPOINT/_ACCESS_KEY/_SECRET_KEY`, `internal/objstore/objstore.go:214-218`), using the merged credentials: list one key, put, get and delete a probe object at `.carlos-move-probe/<move id>` | codex 14 |

-| T5 archive | the newest archive opens with this box's key, its plaintext digest matches its manifest, and it holds no special file and is under the cap (M5) | |

-| T6 disk | free space (`readDiskFrac`, `internal/host/probes.go:209`) covers twice the replica, plus the archive, plus 2 GiB, and stays under 85% full | |

-| T7 release | the channel head materializes for the held row (§9.2) | codex 13 |

-| T8 one owner | exactly one hosting box publishes the target placement | |

-| T9 no tombstone | no host tombstone for this host is `pending` in this region (§7.12) | |

-| T10 return lane | this box can put and delete `moves/_lane/<move id>` in the **source's** data bucket, so a dirty rollback can carry its writes back (§7.8) | codex 2 |

-| T11 duration | the measured restore time for this replica size, plus the source's measured stop time, is under the operator's window (default 120 s) (M11) | |

-**Source readiness (codex 14).** The source proves the probe contract on

-the instance as it runs today: a probe request (§7.7) to its readiness path

-answers 200 to 399 from the instance's own process. An app that fails this

-on its home box is refused with "declare `moves.ready_path`" rather than

-moved and rolled back.

-`ready` requires a round at most the 16 MiB bar **and** a passing target

-report and source probe, each under 10 minutes old.

-### 7.4 The cut, journaled (codex 4)

-The cut is a sequence of phases. **Each phase is written to the record

-(`cut_phase`) before its side effects start**, so a restart resumes

-exactly, and a failure has a defined recovery.

+| `replica/` | the mirrored litestream replica; the only subtree `MirrorPrefix` touches | source rounds |

+| `files/` | sealed archive and manifest | source |

+| `timers/` | the timers snapshot | source |

+| `move.json` (outside every subtree) | `{"move_id","incarnation","host","from","to"}`, create-only | source, before the first round |

+`MirrorPrefix` runs on `replica/` only, so it can never delete an archive,

+a snapshot or `move.json`. Landing installs only `replica/` into

+`instances/<host>/` (§7.6). Receipts are `move-receipts/<host>/<move id>.json`

+in the source bucket.

+**Snapshot rounds (source).** Each round the source builds the files

+archive (§8) when the plaintext digest changes, and publishes its **launch

+report** `moves/<acct>/<app>/<host>/<move id>/source.json`: every variable

+of the **composed** launch environment its next spawn would get, which is

+`childEnv` (`backend_exec.go:1158-1196`) over `instanceEnv`

+(`:1135-1153`) plus the agent-owned variables (`:975-993`), including the

+non-`CARLOS_*` variables `childEnv` inherits from the agent (codex R2-8).

+Each value is reported as an HMAC-SHA256 under a key derived (HKDF) from

+`CARLOS_SECRETS_KEY`, so the report reveals names and equality only.

+**Readiness report (target).** For every record whose `move.to` is its

+placement, the target writes `moves/<acct>/<app>/<host>/<move id>/target.json`:

+| check | passes when |

+|---|---|

+| T1 protocol | this box is at or above the protocol floor (§11) (codex 7) |

+| T2 secrets key | this box publishes `secrets_key_fp` equal to the console's (§10) |

+| T3 environment | the target composes the environment it would launch with for a **flat** held row on this box (rendered bundle through §9.1, the carried `<host>.env` from the newest archive, the agent-owned variables, and its own inherited agent variables), and every source variable matches by HMAC, with these classes: **region** (`CARLOS_REGION`): must differ as expected; **layout** (`CARLOS_CONTROL_SOCKET`, `CARLOS_PURGE`): must be present iff the flat layout provides it, value not compared; **box** (an allow-list of host-specific inherited variables such as `INVOCATION_ID`, `JOURNAL_STREAM`): not compared; **everything else**: equal. A `<host>.env` variable that shadows a rendered one is reported. The report records the **input versions** it used: every bundle etag, the overlay etag, the archive's plaintext digest, the release version and the launch report's digest (codex R2-8) |

+| T4 storage | for each store bound in that composed environment (`internal/objstore/objstore.go:214-218`), with its credentials: list one key, put, get and delete `.carlos-move-probe/<move id>` |

+| T5 archive | the newest archive opens with this box's key, matches its manifest, has no special file, and is under the cap |

+| T6 disk | free space (`readDiskFrac`, `internal/host/probes.go:209`) covers twice the replica, plus the archive, plus 2 GiB, and stays under 85% |

+| T7 release | the pinned release materializes for the held row (§9.2) |

+| T8 one owner | exactly one hosting box publishes the target placement |

+| T9 claim | the claim is `moving` with this move id, its holder is the record's incarnation, and this region and box are in `touched` (added by CAS before the first staging write, claim spec §2) |

+| T10 return lane | this box can put and delete `moves/_lane/<move id>` in the source's bucket (M17) |

+| T11 budget | the **whole** stop-to-open estimate fits the operator's cut window (default 120 s) (codex R2-10, M11), from measured rates: stop + final checkpoint + final mirror (at most the 16 MiB bar at the measured cross-region copy rate) + archive build and upload + install copy (replica size at the measured in-bucket copy rate) + restore (replica size at the measured restore rate) + render + one probe attempt (30 s + the 20 s settle). The estimate is recorded per phase as `cut_budget_s` |

+**Source readiness.** S1: a probe request (§7.7) to the readiness path on

+the instance as it runs today answers 200 to 399 from its own process

+(codex 14); an app that fails is refused with "declare `moves.ready_path`".

+S2: its next spawn is strict (§5.3).

+`ready` requires a round at most the 16 MiB bar, a passing target report

+and source readiness, each under 10 minutes old.

+### 7.4 The cut, journaled (codex 4, R2-8, R2-10)

+Each phase is written to `cut_phase` before its side effects.

|---|---|---|---|

-| `admitted` | source CAS `requested` → `admitted`, stamping `cut_at`, **only if** the target report and source probe are still fresh | none yet | not fresh: CAS back to `ready` with `reason`. Abort is legal until `admitted` and refused after it, and both are CASes on the same field, so exactly one wins |

-| `stopped` | source | final checkpoint into its own `instances/<host>`; read `Replication` and `NextDue` from its row; write `stopped-source.json` (§5.3); snapshot the `timers` rows into `move-snapshots/<host>/<move id>/timers.json` in its own bucket | |

-| `archived` | source | build the final archive from the stopped directory, seal it, write it **create-only** to its own bucket, record both digests in `source.files` (§8.3); then copy it and the timers snapshot into the target's staging | over the cap, a special file, or 10 min of upload failures: `reverting` |

-The source does **not** purge its local database or files at the cut (rev 1

-did, `movespass.go:492`). They stay, held, until finalization parks them

-(§7.9), so a rollback restarts from them warm, and nothing on the source

-can be read after a completed move because the park removes it from every

-path a wake looks at.

-**`reverting` (pre-flip recovery).** The record still names the source, and

-the source holds the execution right. The source keeps the row held as

-`restoring`, checks its files against `source.files.plain` (unchanged, since

-nothing ran), runs the probe (§7.7), and on success opens with the

-conditional write and CASes to `aborting` with the reason. On probe failure

-it retries for 15 minutes, then `stuck` (paged, still held). Staging is

-then deleted by the abort arm.

+| `admitted` | source CAS, stamping `cut_at` | **only if** the report and source readiness are fresh **and** every pinned input version (`move.inputs`) still matches the current bundle etags, overlay etag, release and launch report; otherwise CAS back to `ready` so the checks rerun (codex R2-8) | abort is legal until `admitted` and refused after; both CAS the same field |

+| `stopped` | source | read `Replication` and `NextDue`; snapshot the timers rows to `move-snapshots/<host>/<move id>/timers.json` **and** `…/timers/` in staging; write `stopped-source.json`; CAS the claim's execution holder to `none` (§19 K3) | |

+| `archived` | source | build the final archive from the stopped directory, seal, write create-only to `move-snapshots/`, record both digests, copy into `files/` | over the cap, a special file, or its budget: `reverting` |

+Every phase's budget comes from `cut_budget_s`. Exceeding a phase's budget

+starts its recovery; the recovery's own deadlines (`reverting` 15 min,

+`landing_deadline` 10 min) are separate from the promised window (codex

+R2-10). The source does not purge its database or files at the cut;

+finalization parks them (§7.9).

+**`reverting` (pre-flip recovery, codex 4).** The record still names the

+source. The source CASes the claim to `op.kind = reverting` with itself as

+execution holder, switches the row to `restoring` with

+`hold_op = move id`, and then:

+- **from `stopping`:** stop evidence was never obtained, so something may

+ still run. The source keeps computing evidence (§5.3) and starts nothing

+ until it has it. There is no files baseline to check, because the move

+ has not touched the files; the app's own last writes stand. If evidence

+ stays unobtainable for 15 minutes (a stray process), `stuck`, held,

+ paged;

+- **from `stopped`, `staged` or `archived`:** it checks its files and timers

+ against the digests it recorded at `stopped` (rev 2 used the archive

+ digest, which did not exist yet when `staged` failed: codex R2, R1-4

+ disposition).

+It then probes (§7.7), opens with the conditional write, and CASes the

+record to `aborting`. Fifteen minutes of failed probes: `stuck`, held.

+**Source digests at `stopped`.** The source records `files.plain` (§8.3) and

+`timers_sha256` at `stopped`, before any phase that can fail, so every

+recovery has a baseline.

### 7.5 The flip

-One CAS, guarded by move id, incarnation and `cut_phase = archived`.

-Execution right: the source has written stop evidence, so the right passes

-to the target as soon as it reads it. The source row stays held (`moving`):

-reconcile no longer removes a row whose record is mid-move with this box as

-`from` (§9.6), so its timers, `Replication` and local files remain for a

-rollback.

+One CAS on the record, guarded by move id, incarnation and

+`cut_phase = archived`, followed by the claim CAS (§19 K4). If the record

+CAS lands and the claim CAS does not (a crash between them), the source's

+restart finds `flipped` in the record and completes the claim CAS; until

+then nobody holds the execution right, which is safe. The source row stays

+held (`moving`); reconcile keeps it because the claim names this box as a

+participant (§9.6).

-### 7.6 Landing and install (codex 5, 13)

+### 7.6 Landing and install (codex 5, 13, R2-6)

-The target's reconcile mints the row `landing` with `hold_op = move id`

-(§9.6), unless a pending tombstone for the host is in this region (§7.12).

-The target's move worker then runs the landing phases, each written to

-`landing_phase` before its side effects:

+The target's reconcile mints the row held as `landing` with

+`hold_op = move id` (§9.6). The target's worker runs these phases. Each is

+written to `landing_phase` before its side effects, and its local work is

+journaled in `<DataDir>/.moves/<host>/<move id>/` (F12).

-| phase | action | replay rule |

-|---|---|---|

-| `installing` | read `stopped-source.json` (evidence check); bucket-park whatever is at `instances/<host>/` in its own bucket into `superseded/<move id>/<host>/` (an older stay); per-key copy `moves/<host>/<move id>/` into `instances/<host>/`, verifying each size; write `instances/<host>/generation.json` with `gen = move id` last | replayable until `generation.json` exists: the staging is immutable after the flip and the copy is per-key with no destination deletes |

-| `installed` | local-park any warm DB and any existing files directory into `.parked/superseded-<move id>/`; open the archive, verify the plaintext digest, extract; write `.files-gen = move id` last by rename; write `<host>.env`; replace the host's timers with the snapshot (§9.4); set `Replication` and `NextDue` on the row | replayable until `.files-gen` exists |

-| `configured` | render through the held-render API (§9.1) and verify the move-bound ack | replayable |

-| `restored` | restore the database cold from `instances/<host>` (now generation `<move id>`), write `<dbPath>.gen`, `CheckDB` (`backend_exec.go:313`); record `landing.installed_db_sha256` | replayable until `probing` |

-| `probing` | the probe (§7.7) | **never replays an earlier phase** (INV-7). A crash here restarts only the probe, on the same installed files and database, which may now hold target writes |

-| `probed` | probe passed | |

-Any failure in `installing` to `probing` triggers a rollback (§7.8). A

-landing past `landing_deadline` triggers one too.

+|---|---|---|---|

+| `checking` | read `stopped-source.json`; require the claim's `op.kind = landing` with this move id and execution holder = this box; require this box in `touched.boxes` (CAS-add before any local write); read `move.json` and require its incarnation; acquire the lease on `instances/<host>/` in its own bucket (claim spec C3) and hold it through `published` | | replayable |

+| `parking` | bucket-park whatever is at `instances/<host>/` in its own bucket into `superseded/<move id>/<host>/` (an earlier stay) | the park manifest exists **and** every named key is deleted | replayable by #199's rule (manifest present: skip copy, delete the rest) |

+| `copying` | per-key copy of `replica/` into `instances/<host>/`, verifying each size; then write the install manifest `instances/<host>/install-<move id>.json` (every key and size) | the install manifest exists | replayable: staging is immutable after the flip, the destination was emptied by `parking`, and the copy never deletes |

+| `local` | local-park any warm database (with `<dbPath>.gen`), files directory and `<host>.env` into `.parked/superseded-<move id>/`; extract the archive's files; write `<host>.env` from the archive; `ReplaceTimers` from the snapshot (§9.4); set `Replication` and `NextDue`; remove the host's restore-verification entry (§6.3); fsync all; **then** write `.files-gen` | `.files-gen` names this op and incarnation, and the journal records each sub-step | each sub-step is idempotent and journaled; `.files-gen` is written last, so its presence vouches for all of them (codex R2-6) |

+| `restored` | restore cold from `instances/<host>` (generation `<move id>`), write `<dbPath>.gen`, `CheckDB` (`backend_exec.go:313`), record `landing.installed_db_sha256` | | replayable until `probing` |

+Any failure from `checking` to `probing`, or `landing_deadline`, triggers a

+rollback (§7.8).

### 7.7 The probe (codex 14)

-A new activator entry, `ProbeWake(host, op)`, legal only on a row held with

-`hold_op = op`:

-- spawns the instance through the normal path, with `CARLOS_MOVE_PROBE=1`

- set (advisory; INV-2 does not rely on it);

-- **binds the answer to the process:** the socket's peer credentials

- (`SO_PEERCRED`) must name a process in the spawned child's cgroup, so

- another process squatting on the socket cannot pass;

-- requests the app's readiness path, `moves.ready_path` from the instances

- manifest when declared, `/` otherwise, and requires 200 to 399 within 30 s

- (the same budget as a wake, `backend_exec.go:36`);

-- waits a 20 s settle window, requires the process to still be running,

- and asks again;

+`ProbeWake(host, op)`, legal only on a row held with `hold_op = op` and

+only while the claim names this box as execution holder:

+- a strict spawn (§5.3), with `CARLOS_MOVE_PROBE=1` (advisory);

+- the answer must come from the spawned tree: the socket peer's

+ credentials (`SO_PEERCRED`) name a process in the host's cgroup;

+- the readiness path (`moves.ready_path` from the instances manifest, `/`

+ otherwise) answers 200 to 399 within 30 s (`backend_exec.go:36`);

+- a 20 s settle, the process still running, and a second request;

- up to three attempts.

-The probe's child is registered as **held** in the activator, so an edge

-reload's `RetireExcept` (`internal/edge/edge.go:924`,

-`activator.go:2624`) does not retire it for being absent from the ready set,

-and the edge still never admits a request to it (codex 13).

-### 7.8 Rollback (codex 1, 2, 3)

-**Triggers:** any landing failure (the target writes it), or

-`landing_deadline` passing (the source writes it, or the console's sweep).

-Both are a CAS `landing` → `rolling_back`, guarded by move id. The commit is

-a CAS from `probed` on the same record, so exactly one of commit and

-rollback wins.

-The rollback CAS sets `region = from`, `rollback.id`, `reason` and

-`release_deadline = now + 15 min`. Nothing starts anywhere yet.

-**Target release.** The target's worker, on seeing `rolling_back`:

-1. `FenceAndStop` until stop evidence (retry; a crashed agent restarts and

- does the same).

-2. Final checkpoint; sha256 of the database file after a `TRUNCATE`

- checkpoint; plaintext digest of the files directory; digest of the

- host's timers rows.

-3. **Clean or dirty.** Clean when all three equal what was installed

- (`landing.installed_db_sha256`, `source.files.plain`,

- `source.timers_sha256`) or when no probe ever started. Otherwise dirty.

-4. Write `stopped-target.json` with the digests and the verdict, and CAS

- `rollback.phase = released`, `rollback.dirty`.

-5. **Dirty only:** carry the writes back. This is the handoff, reversed:

- mirror its `instances/<host>` into the source bucket's

- `moves/<host>/<rollback id>/` (T10 proved the lane), build and seal an

- archive of its files directory, snapshot its timers, publish all three,

- then CAS `rollback.phase = returned`.

-6. Hold its row until reconcile retires it (the record no longer names its

- region), then local-park its database and files into

- `.parked/superseded-<rollback id>/`.

-**Source recovery**, on `released` (clean) or `returned` (dirty):

-1. Its row is still held (`moving`, `hold_op = move id`, §7.5). It switches

- it to `restoring` with `hold_op = rollback id`.

-2. Clean: its own database and files are what it stopped with. It checks

- the files against `source.files.plain`; a mismatch means something

- touched them, and goes to `stuck` (INV-4).

-3. Dirty: it installs generation `<rollback id>` exactly as §7.6 does

- (park its own lineage and local files into `superseded/<rollback id>/`,

- copy, extract, timers, config), because the target's copy is now the

- authoritative one (INV-2).

-4. The probe (§7.7), still held.

-5. Pass: open with the conditional write on `hold_op = rollback id`; log

- `rolled_back`; clear `move` (§7.9's last step). Page as a warning.

-6. Three failed probes: `stuck`, still held. Paged as an incident.

-The source never opens before its own probe passes, and never starts before

-the target's stop evidence (codex 1, 3).

-**Waiting for the target.** If `release_deadline` passes with no evidence,

-the move stays in `rolling_back`, shown as "waiting for uk to release", and

-pages. It is still safe (nothing runs, the source is held) and it converges

-as soon as the target box returns (INV-3). §7.10 covers a box that never

-returns.

+The probe's activator entry is **held**: `RetireExcept`

+(`internal/edge/edge.go:924`, `activator.go:2624`) keeps it, and the edge

+never admits a request to it (codex 13).

+### 7.8 Rollback (codex 1, 2, 3, R2-3, R2-4, R2-5)

+**Triggers:** a landing failure (the target), or `landing_deadline` (the

+source or the console's sweep). Both are a record CAS `landing` →

+`rolling_back`, guarded by move id; the commit is a CAS from `probed` on

+the same record, so exactly one wins. The CAS sets `region = from`,

+`rollback.id`, `reason`, `release_deadline = now + 15 min`, and the claim

+moves to `op.kind = restoring` under the same op id, with the execution

+holder still the target until its evidence arrives. Both boxes stay

+participants, so **reconcile keeps both rows

+held** (§9.6, §19 K9): the target row is not retired by the region change

+(codex R2-4).

+**Target release**, journaled as `rollback.target_phase`:

+| phase | action |

+|---|---|

+| `releasing` | row `releasing` with `hold_op = rollback id`; `ReconstructStopEvidence` |

+| `released` | digests after a final checkpoint: database sha256, files plaintext, timers. **Clean** when all three equal what was installed (`landing.installed_db_sha256`, `source.files.plain`, `source.timers_sha256`) or `landing_phase` never reached `probing`; otherwise **dirty**. Write `stopped-target.json`; CAS the claim's execution holder to `none` |

+| `returning` (dirty only) | the handoff back: mirror its `instances/<host>` into the source bucket's `moves/<host>/<rollback id>/replica/`, seal and publish its files archive and timers snapshot under `files/` and `timers/`, write `move.json` |

+| `returned` (dirty only) | every artifact published |

+| `cleaning` | only after the source records `recovered_at` (below): **bucket-park its `instances/<host>/` and its staging `moves/<host>/<move id>/` into `superseded/<move id>/<host>/`**, local-park its database, `<dbPath>.gen`, files and `<host>.env`, delete its row (conditional on `hold_op = rollback id`), clear its timers rows |

+| `cleaned` | CAS `rollback.target_parked_at` |

+The `cleaning` phase is what leaves no copy of the incarnation in the

+target's live namespace, so a later delete and recreate there cannot

+restore it (codex R2-5, INV-10). It waits for the source's recovery

+because, until then, a source failure might need the target's state again.

+**Source recovery**, journaled as `rollback.source_phase` and in

+`<DataDir>/.moves/<host>/<rollback id>/` (codex R2-3):

+| phase | action | replay |

+|---|---|---|

+| `waiting` | until `released` (clean) or `returned` (dirty) and the claim's execution holder is `none` | |

+| `claimed` | CAS the claim: `op.kind = restoring`, execution holder = this box, citing the target's evidence (§19 K4) | |

+| `restoring` | row `restoring`, `hold_op = rollback id` | replayable |

+| clean: `verified` | files and timers digests equal its `stopped` digests (a mismatch is `stuck`, INV-4); its warm database's `<dbPath>.gen` equals its pre-move generation | replayable |

+| dirty: `parking`, `copying`, `published`, `local`, `configured`, `restored` | exactly the target's install phases of §7.6, on the source, from `moves/<host>/<rollback id>/`, with destination `superseded/<rollback id>/<host>/` for what it supersedes | as §7.6 |

+| `probing` | the probe | **never replays an earlier phase**: once its recovery process has started, its own new writes are authoritative (INV-2, INV-7) |

+| `probed` | | |

+| `recovered` | CAS `rollback.recovered_at` | |

+| `opened` | the conditional open on `hold_op = rollback id`; CAS `rollback.opened_at` | |

+**Obligations.** The target's `released` CAS also adds to the claim, in the

+same write, a `park-superseded` obligation for the target box and region

+(its replica, staging and local state; destination

+`superseded/<move id>/<host>/`), which its `cleaning` phase discharges. A

+dirty source recovery adds one for its own region and box for what its

+install supersedes. These are the claim spec's durable chores (§2 there);

+§19.3 asks for the kind.

+**Finalization:** when `recovered_at`, `opened_at` and `target_parked_at`

+are all stamped, and no obligation this op created remains, whichever

+actor stamps last writes the terminal log event (`rolled_back`, with every

+timing, digest and reason), releases the claim to `serving` (§19 K8), and

+CAS-clears `move`. Until then `move` and the claim stay, so nothing else

+can start on the host (codex 6). The source serves from `opened` on; the

+held claim blocks only other operations.

+Three failed recovery probes: `stuck`, held (§7.10).

+**Waiting.** If `release_deadline` passes with no target evidence, the move

+stays in `rolling_back` ("waiting for uk to release"), paged, safe: nothing

+runs anywhere. It converges when the target returns (INV-3).

### 7.9 Commit and finalization (codex 6)

-1. **Commit** (target): CAS `probed` → `committed`, stamping `committed_at`.

- The `move` object stays, so no new move and no push can start, and all

- recovery information is kept.

-2. **Open** (target): the conditional write on `hold_op = move id`; stamp

- `opened_at` (CAS). A restarted target finds `committed` and a held row,

- and opens it.

-3. **Source park** (source): bucket-park its `instances/<host>/` into

- `superseded/<move id>/<host>/`; local-park its database, files directory

- and `<host>.env` into `.parked/superseded-<move id>/`; delete its row

- (with `hold_op = move id` as the condition); stamp `source_parked_at`.

- Its timers rows go with the row, and the snapshot in

- `move-snapshots/<host>/<move id>/` remains.

-4. **Close** (whichever of the two stamps lands second): write the terminal

- log event with every timing and digest, then CAS-clear `move`, guarded by

- move id and both stamps.

-Until step 4 the record still says `committed`, so every crash between

-steps resumes from the record.

-### 7.10 Stuck, held, and a box that never returns

-- `stuck`: the source's probe failed on recovery. The row stays held. The

- operator fixes the cause and runs `carlos move resolve --host <h>

- --retry`, which reruns the source probe. Nothing is deleted.

-- `rolling_back` past `release_deadline`: as §7.8, waiting. Converges when

- the target returns.

-- **A box destroyed mid-handoff.** Its stop evidence can never arrive.

- `carlos move resolve --host <h> --declare-lost <box>` is the one

- operator act in Phase 1 that may lose data: it records the operator's

- statement that the box's processes are gone, with a confirmation that

- names what is lost (target writes since `flipped_at`, if the lost box is

- the target), and then proceeds as a clean rollback. §18 asks Paul to

- confirm this.

+1. **Commit** (target): record CAS `probed` → `committed`, then the claim:

+ phase `committed`, plus two obligations in the same claim write,

+ `park-superseded` for the source box and region and `park-superseded`

+ for the target's staging (§19.3).

+2. **Open** (target): conditional open on `hold_op = move id`; CAS

+ `opened_at`. From here the target serves; the claim's `landing` op with

+ phase `committed` admits it (§19 K6).

+3. **Source park** (source, holding the lease on its own

+ `instances/<host>/`, claim spec C3): bucket-park its `instances/<host>/` into

+ `superseded/<move id>/<host>/`; local-park its database (with

+ `<dbPath>.gen`), files, `<host>.env`, quarantine and journals; delete its

+ row (conditional on `hold_op = move id`) and its timers rows; CAS

+ `source_parked_at` and remove its obligation.

+4. **Target staging park** (target, holding the lease on its own

+ `moves/<host>/`): `moves/<host>/<move id>/` into

+ `superseded/<move id>/<host>/`; remove its obligation.

+5. **Close** (when both obligations are gone): terminal log event `done`;

+ release the claim to `serving` with the target as execution holder

+ (§19 K8); CAS-clear `move`.

+### 7.10 Stuck and declare-lost (M16)

+- **`stuck`:** the source's recovery probe failed. Held. `carlos move

+ resolve --host <h> --retry` reruns the recovery's `probing` phase.

+- **`--declare-lost <box>` (M16)**, operator-only. `carlos move resolve

+ --host <h> --declare-lost <box>` is the move-facing form of the claim

+ spec's `carlos ops host resolve <host> --declare-lost <box>` (§4 there),

+ which removes the box's local obligations; for a move it also does the

+ following, with Astra's round-2 conditions:

+ - **refused after `committed`**, and refused unless the named box is the

+ exact participant that owes the missing evidence for this move;

+ - **execution fencing is established independently first:** the verb

+ CASes the claim's execution holder away from the lost box to `none`,

+ with a `lost` entry naming it. Every box checks the claim before any

+ spawn (§19 K6) and, on reading a claim that does not name it, kills the

+ host's tree (§5.3 step 1) before doing anything else. So a box that was

+ only partitioned cannot resume writing when it returns;

+ - **recover durable data first:** if the target had reached `returned`,

+ the dirty return is used and nothing is lost; if the source is the

+ lost box, the verb is refused (its copy is the only authority);

+ - otherwise it proceeds as a clean rollback, after a confirmation that

+ names the box, the flip time, and that anything the target wrote since

+ then is lost (M-C18).

### 7.11 Abort

-Legal in `preparing`, `ready`, and `cutting` while `cut_phase = requested`

-(CAS). The source deletes `moves/<host>/<move id>/` and the archive, then

-clears `move`. A failed delete is logged and retried; the prefix is

-unreferenced (its key names the move id), so it is never read.

-### 7.12 Tombstones (#199 rev 3)

-- **The instance is deleted mid-move.** The delete door writes the

- tombstone with the record's **current** region, then deletes the record.

- Every move worker binds each step to `move.incarnation`; a record that is

- authoritatively absent (or present with another incarnation) ends the

- move for both boxes: each stops anything it runs (stop evidence as usual)

- and **parks its own region's artifacts of this move into the tombstone's

- `parked/<incarnation>/<host>/`**: the source its `instances/<host>/` (if

- the tombstone is in the other region) and its receipts and snapshots; the

- target its staging, `instances/<host>/` if installed, and local files. The

- box in the tombstone's region parks as #199 §3.3 says. Phase 1 adds the

- other region's half, because #199's park covers only its own region.

-- **Recreated mid-move.** The new record has a new incarnation and no

- `move`. No old step matches it. #199 §3.5 defers the new row until the

- parks are done, and the landing mint obeys the same deferral.

-- **#199's park of `moves/<host>/`.** Staging is

- `moves/<host>/<move id>/` with a `move.json` naming the incarnation. The

- tombstone park takes only keys whose `move.json` names the tombstoned

- incarnation, or legacy keys with none. A live move of a newer incarnation

- is never swept (§17, I-3). T9 also refuses to start a move into a region

- with a pending tombstone for the host.

+Legal in `preparing`, `ready`, and `cutting` with `cut_phase = requested`

+(CAS). The source parks `moves/<host>/<move id>/` and its snapshots into

+`superseded/<move id>/` (unreferenced, so a failed park is retried, never

+dangerous), clears `move`, and releases the claim to `serving`.

+### 7.12 Deletes and parks under the claim

+- **Delete during a move is refused** (409): the delete doors and

+ `PurgeApp` CAS the claim from `serving` to `deleting` (claim spec §5), so

+ they cannot start while a move op holds it (§19 K7). A purge of an app

+ with a moving instance waits. An operator who must delete aborts first,

+ or waits for the commit or rollback to finish.

+- **Recreate during a move** cannot happen: create requires the claim free

+ or absent (claim spec C2).

+- **A delete after a move** parks every region in `touched.regions`

+ (claim spec §5). The move added each region and box to `touched` before

+ writing there, so nothing it left is missed, and what it superseded is

+ already out of the live namespace (§7.8, §7.9, INV-10). The claim's

+ delete park copies `moves/<host>/` and `instances/<host>/` as separate

+ subtrees, which is compatible with per-op staging (§7.3).

### 7.13 Crash at every step

-Every row assumes the crashed actor restarts. "Resumes" means the actor

-re-reads the record and runs the phase it names.

+The crashed actor restarts with **no activator memory**; everything it

+needs is in the record, the claim, its journals (F12) and the box's actual

+process state (§5.3).

|---|---|---|---|

-| cutting: stopping | flat child died with the agent, nested unit stopped on restart; evidence written | n/a | |

-| cutting: stopped to archived | resumes; the sealed archive is reused from its create-only key (§8.3) | n/a | |

-| reverting | resumes the recovery, still held | n/a | |

-| landing: probing | n/a | restarts the probe only; the child died with the agent or is stopped first | |

+| cutting: requested, admitted | resumes the phase | n/a | |

+| cutting: stopping | `ReconstructStopEvidence` from cgroup and unit state | n/a | |

+| cutting: stopped to archived | resumes; the sealed archive is reused from its create-only key | n/a | |

+| reverting | resumes the recovery, held | n/a | |

## 8. The archive (codex 19)

### 8.1 Content

-A tar of the files directory (F4) and `<host>.env` (F5), in sorted path

-order, with owners and times zeroed and modes kept, so equal content gives

-equal plaintext bytes. Only regular files and directories; a symlink,

-device, FIFO or socket refuses with its name. The 256 MiB cap is enforced

-while streaming the tar, so an oversized directory stops early (M5). Then

-zstd.

+A tar of the files directory (F4, without `.files-gen`) and `<host>.env`

+(F6), in sorted path order, with owners and times zeroed and modes kept, so

+equal content gives equal plaintext. Regular files and directories only; a

+symlink, device, FIFO or socket refuses with its name. The 256 MiB cap is

+enforced while streaming (M5). Then zstd.

### 8.2 Sealing

-Sealed for the deployment's config keypair, the one config bundles use

-(P-256 ECDH, `internal/appconfig/crypto.go:44-59`):

-- a fresh 256-bit data key; AES-256-GCM over 1 MiB chunks, each nonce the

- chunk index, the last chunk flagged so truncation is detected;

-- the data key sealed with `appconfig.Seal` (`crypto.go:86`), which uses a

- fresh ephemeral key and nonce;

-- a manifest: the sealed data key, the sealed object's sha256, the

- plaintext's sha256, the per-file list (path, size, sha256), the move id

- and the incarnation.

-Sealing is randomized on purpose. Encryption is never made deterministic by

-reusing a key or nonce.

+For the deployment's config keypair (P-256 ECDH,

+`internal/appconfig/crypto.go:44-59`): a fresh 256-bit data key; AES-256-GCM

+over 1 MiB chunks, each nonce the chunk index, the last chunk flagged; the

+data key sealed with `appconfig.Seal` (`crypto.go:86`, fresh ephemeral key

+and nonce); a manifest with the sealed data key, the sealed object's

+sha256, the plaintext sha256, the per-file list (path, size, sha256), the op

+id and the incarnation. Encryption is randomized and never made

+deterministic.

### 8.3 Identity, publication, retention

-- **Two identities.** `plain` (the plaintext digest) says what the files

- are. `sealed` (the ciphertext digest) names one immutable object. Checks

- of content (T5, rollback, finalization) use `plain`. Fetches use

- `sealed`.

-- **One sealed object per cut.** In `archived` the source writes the sealed

- archive and manifest **create-only** to its own bucket at

- `move-snapshots/<host>/<move id>/files-<sealed>.{sealed,json}`, then

- records both digests in the record, then copies them to the target's

- staging. A crash before the record write leaves an unreferenced object

- and the next attempt builds a new one; a crash after it reuses the

- recorded one and never rebuilds. So a replay never changes

- `source.files.sealed` (codex 19).

-- **Retention.** The source's snapshots (archive, manifest, timers) stay in

- `move-snapshots/<host>/<move id>/` until F3's cleanup, because a rollback

- and an audit both need them. The target's staged copies are parked with

- staging at finalization.

+- **`plain`** (plaintext digest) identifies content; **`sealed`**

+ (ciphertext digest) identifies one immutable object. Content checks use

+ `plain`; fetches use `sealed`.

+- The `plain` digest of the files directory as stopped is computed at

+ `stopped` without sealing, so recoveries have a baseline before any

+ archive exists (§7.4).

+- **One sealed object per cut:** written create-only to

+ `move-snapshots/<host>/<move id>/files-<sealed>.{sealed,json}`, then its

+ digests recorded in the record, then copied to staging `files/`. A crash

+ before the record write leaves an unreferenced object; after it, replay

+ reuses the recorded object (codex 19).

+- **Retention:** the source's snapshots stay until F3; staging is parked at

+ finalization (§7.9, §7.8).

## 9. Changes to existing code (codex 13, 15)

### 9.1 Config render for a held row

-`SyncExec` skips any row that is not `ready` (`execRouteBase`,

-`internal/appconfig/syncexec.go:643-647`), and writes an ack only when a

-file changes (`syncexec.go:1159`). A held row needs neither behaviour. New:

-`appconfig.RenderHeld(ctx, rt, op)` renders exactly the file `SyncExec`

-would, writes it, and writes a **move-bound ack**

-`control/config-ack/<box>/<host>.json` with `op`, the file's sha256 and the

-bundle etags, whether or not the file changed. The landing reads that ack.

-`SyncExec` continues to skip held rows, so the two never race on one file;

-once the row opens, `SyncExec` owns it again.

+`SyncExec` skips non-`ready` rows (`internal/appconfig/syncexec.go:643-647`)

+and writes an ack only when a file changes (`:1159`). New:

+`appconfig.RenderHeld(ctx, rt, op)` renders the same file, writes it, and

+writes a **move-bound ack** with the op, the file's sha256 and the bundle

+etags, changed or not. `SyncExec` keeps skipping held rows, so the two never

+race on one file.

### 9.2 Release materialization for a held row

-`release.Adoptable` excludes non-`ready` rows (`internal/release/adopt.go:816-819`).

-New: `release.MaterializeHeld(rt, op)` stages the channel head for a held

-row without adopting traffic, used by T7 and before the probe.

+`release.Adoptable` excludes non-`ready` rows

+(`internal/release/adopt.go:816-819`). New: `release.MaterializeHeld(rt, op,

+version)` stages exactly the pinned release (`move.inputs.release`).

### 9.3 The activator

-- `ProbeWake` (§7.7).

-- `RetireExcept` keeps held entries (§7.7).

-- The start check (§5.1) in `Start`, `Seed` and `ProbeWake`.

-- Restore writes `<dbPath>.gen` and parks a warm file of another

- generation (§5.1). `promoteStagedReplica` and its call in `Restore`

- (`backend_exec.go:290`) are deleted.

-- `FenceAndStop` returns evidence the worker can publish.

-### 9.4 Timers frozen and restored

-- **Frozen:** the runner skips, with no bookkeeping at all, any row whose

- status is a hold. Today a blocked delivery still advances `attempts` and

- state (`internal/schedule/runner.go:345-352`).

-- **Not pruned:** a held row is a route, so the converger's prune

- (`internal/schedule/converge.go:114`) leaves its timers alone. The source

- row is held through the whole move (§7.5).

-- **Replaced exactly:** new `Registry.ReplaceTimers(host, rows)` deletes the

- host's rows and inserts the snapshot in one transaction, preserving

- `next_at`, `due_at`, `attempts` and `registration`. `UpsertTimer` resets

- `attempts` and `registration` when it rearms

- (`internal/registry/timers.go:46-55`), so it is not used for this. The

- target replaces at install; the source replaces at a dirty rollback.

-- **Declared schedules** then converge as today; an imported row whose spec

- and path match is kept (`converge.go:223-241`).

+- `ProbeWake` (§7.7); `RetireExcept` keeps held entries.

+- `ReconstructStopEvidence` (§5.3); strict spawns under a move.

+- The start check (§5.1) in `Start`, `Seed` and `ProbeWake`, reading the

+ claim.

+- Restore writes `<dbPath>.gen` and parks a warm file of another generation.

+ `promoteStagedReplica` and its call (`backend_exec.go:290`) are deleted.

+### 9.4 Timers frozen, drained and restored (codex 15, R1-15 partial)

+- **Drained:** entering `stopping`, the source waits until the runner has

+ no delivery in flight for the host (the runner's per-host claim,

+ `internal/schedule/runner.go:154`) before taking evidence, so a delivery

+ that started before the hold cannot advance a row after the snapshot.

+- **Frozen:** the runner skips, with no bookkeeping, any held row (today a

+ blocked delivery still advances `attempts`, `runner.go:345-351`), and the

+ converger neither upserts nor prunes timers for a host whose claim op is

+ `moving` (`internal/schedule/converge.go:114`, `:223-241`).

+- **Replaced exactly:** `Registry.ReplaceTimers(host, rows)` deletes and

+ inserts in one transaction, preserving `next_at`, `due_at`, `attempts`

+ and `registration`. `UpsertTimer` resets them on rearm

+ (`internal/registry/timers.go:46-55`), so it is not used here.

+- Declared schedules converge again once the claim returns to `serving`.

### 9.5 Edge

-None in Phase 1. A held row is not served (`edge.go:595`). Platform-domain

-hosts get the 503 pending page (#174, `internal/edge/pending.go`); custom

-domains get 404 during the window, until F1's holding route.

+None in Phase 1. Platform-domain hosts get the 503 pending page during the

+window (#174, `internal/edge/pending.go`); custom domains get 404 until F1.

-### 9.6 Reconcile

+### 9.6 Reconcile (codex R2-4)

-- Mint (`routeFor`, `reconcile.go:364-376`): a record in `landing` mints

- `landing` with `hold_op = move id` on the target. A record in

- `rolling_back` or `stuck` never mints `ready` (it mints `restoring` on the

- source if the row was lost).

-- Removal (`reconcile.go:1114-1159`): a row whose host has a record with a

- `move` naming this box's placement as `from` and not yet at

- `source_parked_at` is not removed. The source's move worker deletes it at

- finalization.

-- Deferral: #199 §3.5's tombstone deferral applies to landing mints.

+- **Mint:** while the claim holds a move operation, a participant box mints the

+ host's row held (`landing` on the target after the flip, `restoring` on

+ the source during recovery), never `ready`.

+- **Removal:** never removes a row of a host whose claim names this box as a

+ participant, whatever the record's region (`reconcile.go:1114-1159`;

+ claim spec §8, "rows that a running operation lists as participants are

+ retained"). The move worker deletes it at its own finalization step.

+- **Orphan retirement** (claim spec §8, a row whose incarnation is not the

+ claim's holder) does not act on a participant row of a move op; a move

+ never changes the holder, so the case cannot arise for a live move, and

+ the guard makes that explicit (§19 K9).

-## 10. The secrets key, SSM, and the shared fingerprint

+## 10. The secrets key and the shared fingerprint

The platform reads no SSM parameter at runtime. An exec instance's secrets

-reach it through its config bundle in the home deployment bucket, opened

-with the box's `CARLOS_SECRETS_KEY`. So a target needs the deployment's key,

-and nothing per app or per instance needs replicating between SSM regions.

-SSM matters only for staging that key on each box, which is Paul's approved

-remediation.

+reach it through its config bundle, opened with the box's

+`CARLOS_SECRETS_KEY`, so a target needs only the deployment's key.

-**Shared with nearest-region R9.** Both specs use one edges-file field, so

-the region-default spec can adopt it as written:

+**Shared with nearest-region R9.** One edges-file field for both specs:

- **name:** `secrets_key_fp`, in `edgesFile`

(`internal/host/edgesfile.go:155-176`, written at `:603-605`);

-- **format:** `"sha256:"` followed by 64 lowercase hex digits of SHA-256

- over the public key's bytes, `(*ecdh.PublicKey).Bytes()` (the uncompressed

- P-256 point) of the box's `CARLOS_SECRETS_KEY`;

-- **not omitempty**, following `hibernation_configured`: absent means the

- writer predates the field (unknown); `""` means the box has no key;

-- **the console** computes the same over its `CARLOS_SECRETS_PUBKEY`.

-A box is key-ready when its value equals the console's. Unknown is never

-ready.

+- **format:** `"sha256:"` and 64 lowercase hex digits of SHA-256 over

+ `(*ecdh.PublicKey).Bytes()` (the uncompressed P-256 point) of the box's

+ `CARLOS_SECRETS_KEY`;

+- **not omitempty**, like `hibernation_configured`: absent means unknown,

+ `""` means no key;

+- the console computes the same over `CARLOS_SECRETS_PUBKEY`. A box is

+ key-ready when its value equals the console's.

## 11. Versions and the protocol floor (codex 7)

-The new states, fields and holds are meaningless to an older binary, and an

-older target would mint a `ready` row for a landing record

-(`reconcile.go:374`). So:

-- Every box publishes `moves_protocol: 1` in its edges file; the console

- reports its own. The request door, and T1, refuse unless every box in the

- source and target regions and the console are at or above it.

-- Once any Phase 1 move has started, the deployment writes

- `control/moves-protocol.json` `{"min": 1, "min_release": "vX.Y.Z"}`

- create-only. `carlos system rollback` and `carlos system apply` (run by

- the newer binary) refuse to install a release below `min_release`, and

- refuse any downgrade on a box with an in-flight move it is part of. The

- console roll recipe states the same floor.

-- The simulation (§14.2) models a target downgraded to a pre-floor binary

- at every state and asserts the floor stops it.

-## 12. The operator surface

-**API.** The existing doors (`internal/console/authcli.go:625-628`) with the

-§7.1 body, the cut door's freshness check, and abort per §7.11. In Phase 1

-every move mutation requires an operator (`Store.IsOperator`,

-`internal/console/store.go:1858`); a non-operator gets the family's 404.

-Instance-role service credentials keep the status GET only. This closes the

-member-minted token path (`apiinstances.go:1554`) for moves until F1

-designs owner delegation (codex 11). New: `POST …/move/resolve`

-(§7.10). `GET …/move` is unchanged by default; `?detail=1` adds the phases,

-reports, digests and reasons.

-**CLI.** One top-level verb; `docs/site/cli.md` gets its heading in the same

-change (`internal/docsite` fails the gate otherwise).

+- Every box publishes `moves_protocol: 1`; the console reports its own. The

+ request door and T1 refuse unless every box in both regions and the

+ console are at or above it, and every box is claim-aware (§19 K10).

+- Before the first production move the deployment writes

+ `control/moves-protocol.json` `{"min": 1, "min_release": "vX.Y.Z"}`,

+ create-only. `carlos system apply` and `carlos system rollback` refuse a

+ release below it, and any downgrade on a box that is a participant in a

+ claim held as `moving`. The console roll recipe states the same floor.

+## 12. The operator surface (M16, M18)

+**API.** The existing doors (`internal/console/authcli.go:625-628`). Every

+move mutation requires an operator (`Store.IsOperator`,

+`internal/console/store.go:1858`); others get the family's 404. Instance-role

+tokens keep the status GET only (M18), which closes the member-minted token

+path (`apiinstances.go:1554`) for moves (codex 11). New:

+`POST …/move/resolve` (§7.10). `GET …/move?detail=1` adds phases, reports,

+evidence and reasons.

+**CLI.** `docs/site/cli.md` gets the heading in the same change

+(`internal/docsite` fails the gate otherwise).

```

-carlos move --host kpmg.bdn.oncarlos.com --to uk [--manual-cut] [--no-wait]

-carlos move status --host <h>

-carlos move cut --host <h>

-carlos move abort --host <h>

+carlos move --host <h> --to <region> [--manual-cut] [--no-wait]

+carlos move status --host <h>

+carlos move cut --host <h>

+carlos move abort --host <h>

carlos move resolve --host <h> --retry | --declare-lost <box>

```

-`--wait` is the default: it prints each phase with its time, and exits 0

-on done, 3 on rolled back, 4 on stuck or waiting, 5 on refusal.

+`--wait` is the default; exit 0 done, 3 rolled back, 4 stuck or waiting,

+5 refused.

## 13. Observability

- **Move log:** one object per event under

- `moves/<acct>/<app>/<host>/<move id>/` in the deployment bucket

- (`<unix-nanos>-<actor>.json`), beside the reports and evidence. Every

- phase entry and exit, every check result, every reason.

-- **Timings:** prepare, cut (`cut_at` to `flipped_at`), land (`flipped_at`

- to `opened_at`), and **offline** (`cut_at` to `opened_at`, or to the

- source's reopen after a rollback).

+ `moves/<acct>/<app>/<host>/<op id>/` in the deployment bucket, beside the

+ reports and evidence.

+- **Timings:** prepare, cut, land, and **offline** (`cut_at` to the open, or

+ to the source's reopen after a rollback), each against its budget.

- **Paging:** `rolled_back` (warning); `stuck`, `rolling_back` past

- `release_deadline`, and any start refused by the start check (incident);

- `preparing` with no promotable round for 60 minutes (warning).

+ `release_deadline`, a refused start check, and a refused stop evidence

+ (stray process) (incident); `preparing` with no promotable round for 60

+ minutes (warning).

- **Member stream:** each transition is an event on the app's platform log

(`emitEvent`, `internal/activator/logsink.go:31-36`).

@@ -902,156 +935,200 @@ on done, 3 on rolled back, 4 on stuck or waiting, 5 on refusal.

### 14.1 Unit and integration

-- Every transition and its guard, including each illegal edge and a stale

- move id or incarnation.

-- Archive: equal plaintext for equal content; randomized sealed bytes;

- truncation and reordering detected; wrong key refused; special files

- refused; cap enforced mid-stream.

-- Each readiness check failing alone, against fakes, and T4 against MinIO.

-- Each landing phase crashing and replaying; `probing` never replaying an

- earlier phase.

-- Held render, held materialization, `RetireExcept` keeping a held entry,

- the runner's freeze, `ReplaceTimers` exactness.

-- The start check: mismatched markers refuse; absent markers are origin.

-- Reconcile: landing and restoring mints; source row not removed mid-move.

+Every transition and guard; each readiness check failing alone (T4 against

+MinIO); every journaled phase crashing between its individual side effects

+and replaying; `ReconstructStopEvidence` against a real cgroup v2 hierarchy

+with a forked descendant that closes its descriptors and outlives its

+parent; the start check; `RenderHeld`, `MaterializeHeld`, `ReplaceTimers`;

+reconcile keeping participant rows.

### 14.2 The randomized crash simulation

-A Go test driving the **real** transition functions and the real worker

-phase functions through injected interfaces, never a re-implementation.

-Actors: console, source agent, target agent, deadline sweep, and **the

-application process** on whichever box runs it.

-The model includes what rev 1's model missed (codex findings, "simulation"

-note):

-- the app's **own background work**: it writes the database and an

- external ledger on its own schedule while running, including during a

- probe, and an external acknowledgement counts as acknowledged (INV-2);

-- **stalled and surviving processes**: a worker can stall indefinitely

- while its child keeps running; an agent crash kills flat children and

- leaves nested ones running;

-- **scheduler bookkeeping**: timers fire, advance and are refused;

-- **config, adoption and retirement**: held render, `RetireExcept` on edge

- reloads, reconcile mints and removals;

-- **versioned buckets**: deletes leave noncurrent versions;

-- **forged metadata**: another box writes a fake evidence or report object

- with a wrong move id or incarnation;

-- **tombstones**: delete and recreate at any step;

-- **downgrades**: a box replaced by a pre-floor binary at any step.

-A seeded scheduler picks the next atomic step, injects crashes (losing

-memory, keeping disk and buckets), stalls, and the faults of §7. After

-every step it asserts INV-1 and INV-5; at the end of every run INV-2,

-INV-3 (every run with only restarting crashes terminates), INV-4 (every

-start's markers matched), INV-7, INV-8 and INV-9.

-2,000 seeds in the gate; 200,000 nightly.

-**Mutation checks**, run in a tree copy, each of which must fail the

-simulation within the gate's seeds: start the source on a deadline without

-evidence; open before commit; open without the `hold_op` condition; treat a

-lost CAS as stop evidence; discard target writes on rollback; replay

-install after `probing`; re-extract files after `probing`; skip the start

-check; let the runner advance a held row's timers; restore from staging;

-re-seal the archive on replay; let reconcile remove the source row mid-move;

-let a tombstone park sweep another incarnation's staging; skip the protocol

-floor. A mutation the simulation cannot catch is a defect in the

-simulation.

+A Go test that drives the **real** transition functions and worker phase

+functions through injected interfaces, at the granularity of **individual

+side effects**, never whole phases (codex R2, simulation note). Actors:

+console, the claim, source and target agents, the deadline sweep, and the

+application's process tree on whichever box runs it.

+The model contains:

+- the app's own background work, writing the database and an external

+ ledger while running, including during probes and source recovery;

+- **descendants that outlive the main process** (codex R2-1);

+- **activator restarts with empty memory** (codex R2-2);

+- stalled workers whose children keep running;

+- scheduler deliveries in flight across the hold;

+- config, adoption, edge reloads with `RetireExcept`, reconcile mints and

+ removals, **including reconcile running before the target's release**

+ (codex R2-4);

+- versioned buckets;

+- forged evidence and reports with a wrong op id or incarnation;

+- delete and recreate attempts at every step, and **after a rollback**

+ (codex R2-5);

+- **crashes between a marker's publication and the next write** (codex

+ R2-6);

+- downgrades to a pre-floor binary.

+Assertions: INV-1 and INV-5 after every step; INV-2, INV-3, INV-4, INV-7,

+INV-8, INV-9 and INV-10 at the end of every run. 2,000 seeds in the gate;

+200,000 nightly.

+**Mutation checks** (each must fail within the gate's seeds): start on a

+deadline without evidence; accept evidence while a descendant lives; treat

+a missing entry as stopped; open before commit; open without the `hold_op`

+condition; discard target writes; reinstall during source recovery after

+its probe; let reconcile retire the target row before release; skip the

+target's replica park on rollback; write `.files-gen` before the timers;

+replay install after `probing`; skip the start check; let the runner

+advance a held row; mirror over `files/`; re-seal on replay; skip the

+pinned-input check at admission; skip the protocol floor; let a delete

+proceed while the claim is `moving`.

### 14.3 Cross-version

-The request door refuses below the floor; a pre-floor target cannot be

-installed during a move; abort on every new state is refused by v0.22

-consoles (P0's allowlist, `provision.go:1027-1035`).

+The door refuses below the floor; a pre-floor binary cannot be installed on

+a participant; v0.22 consoles refuse abort on every new state (P0's

+allowlist, `provision.go:1027-1035`).

### 14.4 Rehearsal (M12)

-`moveprobe`, a test app in this repo, is shaped like ips:

-- writes `instance.key` into its files directory at first boot, seals a

- row with it, and refuses to start on a mismatch;

-- requires a store binding and a sealed secret at boot, and writes and

- reads an object every boot;

-- declares a schedule and registers a once-timer, logging each run;

-- writes a row from a background loop every few seconds (INV-2's case);

-- fails its readiness path in a named region on request, before or after

- writing (to force clean and dirty rollbacks).

-Runs, each recording every timing: dev, IE to uk, done; dev, forced clean

-rollback; dev, forced dirty rollback; then the same on the flagship with

-scoped approval; then a disposable ips instance in `bdn` with scoped

-approval; then kpmg with Paul's approval. A run passes only if the key

-verifies, the object reads back, the secret is present, every timer ran

-exactly once around the cut, and no background row is missing.

+`moveprobe`, a test app in this repo, shaped like ips: `instance.key`

+written at first boot and verified every boot; a required store binding and

+sealed secret; a declared schedule and a once-timer; a background loop

+writing rows and **a forked worker that outlives the main process**; a

+switch to fail its readiness path in a named region, before or after

+writing. Runs: dev IE to uk, done; dev forced clean rollback; dev forced

+dirty rollback; dev declare-lost; then the same on the flagship with scoped

+approval; then a disposable ips instance in `bdn`; then kpmg with Paul's

+approval. A run passes only if the key verifies, the object reads back, the

+secret is present, every timer ran exactly once around the cut, no

+background row is missing, and no copy remains in a live namespace of the

+region that lost.

## 15. Rollout and gates

1. **Ops (in progress):** `CARLOS_SECRETS_KEY` on every region box.

-2. **Release with `secrets_key_fp`** (shared with nearest-region R9).

- Verify every box's value equals the console's.

-3. **Phase 1 release, dormant.** Consoles first, then every box. The door

- refuses until every box in both regions publishes `moves_protocol: 1`.

-4. **Infra, scoped approval:** each target region's `sibling_staging_arns`

- lists each source region's data bucket (T10's return lane). For kpmg

- that is uk listing ie's bucket.

-5. **Rehearsals** in §14.4's order. Each gates the next.

-6. **Write `control/moves-protocol.json`** before the first production

- move.

+2. **Release with `secrets_key_fp`.** Verify every box against the console.

+3. **The host lifecycle claim** (claim spec §9) ships and every box and

+ console honours it, with §19.3's differences settled.

+4. **Phase 1 release, dormant.** Consoles first, then every box.

+5. **Infra (M17, approved):** every region's `sibling_staging_arns` lists

+ every other region's data bucket. Applied per deployment as scoped work.

+6. **Rehearsals** in §14.4's order; each gates the next.

+7. **Write `control/moves-protocol.json`** before the first production move.

## 16. Follow-on specs

-Each is its own spec. No design here, only what it must satisfy.

+No design here, only what each must satisfy.

| spec | scope | must meet |

|---|---|---|

-| **F1 Owner surface** | owner self-serve from the console and `carlos move`; auto-cut for persons (M6); holding route for custom domains (P2); move history | owners only (M13); **move-capable service authority is explicitly owner-delegated**, including the legacy mint path (`apiinstances.go:1554`), with a stated treatment of existing member-minted tokens that keeps approved router credentials working (codex 11) |

-| **F2 Moves into home (P3)** | `CARLOS_HOME_PLACEMENT`, home as a target, `carlos move back` | the handoff of §5 unchanged; home children's `CARLOS_REGION` change gated |

-| **F3 Residency cleanup (P5)** | delete `superseded/<move id>/`, `move-snapshots/`, `.parked/superseded-*`, receipts, after the 7-day hold (M7), with owner early deletion | **no deletion of a prefix that is, or is an ancestor of, a live location**, and selection is excluded atomically before deletion, not by a fire-time read (codex 8); **the complete residue inventory**, including noncurrent object versions and delete markers in versioned data buckets (`infra/modules/deployment-buckets/main.tf:27-30`), earlier parks and local parks, with scoped version-delete authority and verification of actual removal before reporting residency complete (codex 9); **a deletion warrant that binds the exact scope**: account, app, incarnation, move id, bucket and prefixes, issued from an immutable console ledger and never from box-writable records or logs (codex 10) |

-| **F4 Stores** | private stores with their instance, app stores with their app (M3), consumer hold (M9) | **a joint instance and store transaction** with binding versions on both sides, health checks and compensation, so a rollback never leaves the source bound to an unusable twin (codex 12); **the store record transitions with config** (`walk.go:427`, `:448`), and grant, rotate, revoke and purge are serialized against the move, with security policy preserved and a named principal deleting the fenced old bucket (codex 17); copy credentials issued under F3's ledger binding (codex 10) |

-| **F5 App moves** | the sequencer, app region change, pause (M8), sidecar refusal (M10) | **durable child move ids persisted before dispatch**, with recovery of their outcomes (codex 18); **an app and store cut barrier** covering every consumer, including concurrent creates, skipped and explicitly placed instances (codex 18, M9); defined transitions for pause, abort and move-back of the app operation |

-| **F6 Graceful cuts** | freeze, thaw, verify hooks (the 09-24 P1); sidecar moves | hooks never replace stop evidence |

-| **F7 `carlos db push`** | #199's deferred push, on §5's handoff with source and target the same box | the same four invariants |

+| **F1 Owner surface** | owner self-serve; person auto-cut (M6); holding route for custom domains | owners only (M13); move-capable service authority explicitly owner-delegated, including the legacy mint path (`apiinstances.go:1554`), with a stated treatment of existing member-minted tokens (codex 11); Tito's router credentials (M18) |

+| **F2 Moves into home** | `CARLOS_HOME_PLACEMENT`, home as a target, `carlos move back` | §5's handoff unchanged |

+| **F3 Residency cleanup** | delete `superseded/<op id>/`, `move-snapshots/`, `.parked/superseded-*`, receipts after the hold (M7) | no deletion of a live location or its ancestor, selection excluded atomically through the claim (codex 8); the complete residue inventory including noncurrent versions and delete markers (`infra/modules/deployment-buckets/main.tf:27-30`) with scoped version-delete authority and verified removal (codex 9); a warrant binding the exact scope from an immutable console ledger (codex 10) |

+| **F4 Stores** | private and app stores (M3, M9) | a joint instance and store transaction with binding versions and compensation (codex 12); store record transitions with config (`internal/objstore/walk.go:427`, `:448`), lifecycle operations serialized against the move (codex 17) |

+| **F5 App moves** | sequencer, pause (M8), sidecar refusal (M10) | durable child move ids before dispatch; an app and store cut barrier covering every consumer and concurrent create (codex 18) |

+| **F6 Graceful cuts** | freeze, thaw, verify hooks; sidecar moves | hooks never replace stop evidence |

+| **F7 `carlos db push`** | push on §5's handoff, source and target the same box | the four core invariants; the claim's `pushing` op |

+## 17. Interface points with #199 and the claim spec

+The claim spec supersedes #199 rev 3 from its §2 onward, so most of rev 2's

+#199 interface points now go through the claim (§19).

+| id | point | status |

+|---|---|---|

+| I-1 | **Incarnation**, including the zero-CreatedAt legacy identity | agreed: one `provision.Incarnation` (claim spec §2) |

+| I-2 | **Where an incarnation's data lives** | agreed: `touched.regions` and `touched.boxes`, appended before the first write (claim spec §2); every move write site adds first (T9, §7.6 `checking`) |

+| I-3 | **Parking `moves/<host>/`** | agreed: the claim's delete park copies `instances/` and `moves/` as separate subtrees (claim spec §5 step 4.3), and a delete cannot run during a move, so a live move's staging is never swept |

+| I-4 | **Park helpers and namespaces** | agreed in shape; the destination for a move's superseded data is `superseded/<op id>/`, never `parked/<incarnation>/` (§19.3 D5) |

+| I-5 | **Recreate** | agreed: create requires the claim free or absent (claim spec C2) |

+| I-6 | **Stale or orphan row retirement** | agreed: participant rows retained (claim spec §8); §9.6 |

+| I-7 | **Flat files** | **open**: the claim spec's local park inventory (§5 step 4.4) names the working directory for nested rows; it should take the flat `<DataDir>/<host>/` too, as this spec defines it (§6.1), or a recreated flat host inherits an `instance.key` |

+| I-8 | **`generation.json`, `install-<op>.json`, `lock.json`** | agreed: copied like any other key; `lock.json` deleted last by its own holder |

+| I-9 | **Destructive steps** | agreed: serialized by the claim and the location's lease (claim spec C3), with the cross-region exception in §19.3 D4 |

+| I-10 | **Push** | agreed: F7, on §5 and the claim's `pushing` op |

+## 18. Questions for Paul

+None. Rev 2's question became ruling M16, and round 2's verdicts on the

+former questions match M4 to M15. Where this spec and the claim spec

+differ (§19.3), the difference is for the two specs' authors and reviewer

+to settle, not a product choice.

-## 17. Interface points with #199 rev 3

+## 19. How moves use the host lifecycle claim

+### 19.1 Op kinds and admission, per phase (claim spec §4 assigns these here)

+One op id (the move id) runs through every kind below. The kind changes by

+CAS; the record's state is the move's own detailed journal.

|---|---|---|---|

-| I-1 | **Incarnation.** Every move artifact and guard uses `provision.Incarnation(rec)`, including the legacy identity for a zero CreatedAt (#199 §2). | #199 | one shared function; this spec calls it |

-| I-2 | **Tombstone region during a move.** The delete door writes the record's current `region`. | #199 | confirm; this spec adds the other region's park (§7.12) |

-| I-4 | **Park destinations.** #199 parks to `parked/<incarnation>/<host>/` and `.parked/<incarnation>/<host>/`; this spec to `superseded/<move id>/<host>/` and `.parked/superseded-<move id>/<host>/`. | shared | both use the same copy-manifest-delete and rename helpers; the two namespaces never overlap |

-| I-5 | **Deferral.** #199 §3.5 defers a recreated host's row while a park is pending. | #199 | the landing mint (§9.6) calls the same predicate |

-| I-6 | **Stale-row retire.** #199 §4 retires a row whose `(account, app, host)` record is absent. | #199 | its retire stops the child and deletes the row only; it must never park or purge the files directory or database of a row held by a move (`hold_op` set) while the move's record exists |

-| I-7 | **Local parks and the files directory.** #199 §3.4 parks the nested `InstanceDir`; this spec defines the files directory for both layouts (§6.1). | shared | #199 parks the flat `<DataDir>/<host>/` too, by this spec's definition, so a recreated flat host does not inherit an `instance.key` |

-| I-8 | **`generation.json` and `lock.json`.** #199 §3.3 copies every key except `lock.json`. | #199 | `generation.json` is copied like any other key; a park never deletes `lock.json` early |

-## 18. Question for Paul

-**Declaring a box lost (§7.10).** If a box is destroyed during a handoff it

-holds, its stop evidence can never arrive, and the move waits, paged. Choose

-one:

-- **(a)** An operator verb, `carlos move resolve --declare-lost <box>`,

- records that the box's processes are gone and proceeds as a clean

- rollback. If the lost box was the target, anything it wrote after the

- flip is lost, and the confirmation says so.

-- **(b)** No verb. Recovery from a destroyed box is a manual, below-the-API

- procedure written in the runbook.

-**Recommend (a).** It is the only exception to INV-2 and INV-3, it needs a

-person, and it keeps the recovery inside the API (the 2026-08-06 rule). The

-loss it can cause is bounded to a probe that never served traffic, unless

-the box was destroyed after the commit, in which case the verb is refused

-(the target is authoritative and the source is parked).

-## 19. Copy (pending copy review)

-Drafts for Phase 1's operator CLI. None is approved; every string goes

-through the copy-review gate before it is written into code. Place names come

-from nearest-region §14. `<…>` marks a value. No em dashes.

+The db door refuses on every move kind (claim spec §8).

+### 19.2 Rules K1 to K13

+| id | rule |

+|---|---|

+| **K1 Execution holder** | The move keeps `op.journal.exec` = a box label or `none`, and `op.journal.participants` = {source box, target box}. Admission (claim spec C4) for a move kind is the table in §19.1 **and** `exec` naming this box. |

+| **K2 Acquire** | The request door CASes `serving` → `moving` only when the holder incarnation equals the record's and no other op holds the claim; it sets op id, participants and `exec` = source box. |

+| **K3 Execution to none** | A participant that has written stop evidence for this op id (§5.3) may CAS `exec` from itself to `none`. Nobody moves `exec` from one box to another directly. |

+| **K4 Execution handoff** | From `none` to a participant, by that participant, citing the previous holder's evidence object for this op id: at the flip (to target), at source recovery (to source), and at `reverting` (the source, citing its own evidence). |

+| **K5 Revalidation** | Before each destructive step the actor re-reads the claim and requires op id, kind, phase and `exec` to be what the step expects (claim spec C3); it holds the lease of every location it writes, acting before `SafeUntil`, with the exception in §19.3 D4. |

+| **K6 Wake admission** | `doWake` reads the claim fresh (claim spec §8) and applies K1. A box that reads a claim whose `exec` does not name it kills that host's process tree (§5.3 step 1) before anything else. |

+| **K7 Other operations** | Delete, purge, park and push start only from `serving` (claim spec §4); while a move op holds the claim they are refused. Create requires free or absent. |

+| **K8 Release** | Only the move releases to `serving`, naming `exec` (target after `done`; source after `rolled_back` or `aborted`), and only when every obligation this op created is gone (§7.8, §7.9). |

+| **K9 Other passes while a move op holds the claim** | Reconcile mints participant rows held and never `ready`, and never removes or orphan-retires a participant row; the schedule converger neither upserts nor prunes the host's timers; `SyncExec` and release adoption skip the held rows (they already skip non-`ready`). |

+| **K10 Version floor** | The claim spec's floor (§9 there) and `moves_protocol` (§11) are one object and one check. |

+| **K11 Declare-lost** | §7.10; §19.3 D6. |

+| **K12 Touched** | Every move write site CAS-adds its region or box to `touched` before its first write: the source adds the target region before the first staging round; the target adds its box at `checking`; a dirty return adds nothing new (the source region and box are already there). |

+| **K13 Evidence durability** | Evidence objects are create-only per (op id, role) in the deployment bucket, recomputable from system state after a restart (§5.3), and referenced from the claim journal, never inlined. |

+### 19.3 Where this spec asks the claim spec to change

+| id | claim spec says | moves need | why |

+|---|---|---|---|

+| D1 | admission is "`op.kind` permits serving" (§3 C4, §8) | admission for a move kind is per box and per phase, from `op.journal.exec` and the §19.1 table | the source serves during `moving` until the cut, the target only from `committed`; a kind-level answer cannot say which box |

+| D3 | evidence is an empty per-host cgroup, no live lease, no open replicator (§6) | also: (a) every spawn of a host under a lifecycle op is **strict** (`cgroup.PrepareStrict`), and a non-strict pre-move child is stopped before the cut; (b) the agent's own cgroup holds no process but the agent; (c) no process holds the database open | `cgroup.Manager.Start` falls back to spawning **unaccounted** in the agent's cgroup, or to a PID move that leaves already-forked grandchildren behind (`internal/cgroup/cgroup.go:225-252`). In either case the per-host cgroup reads `populated 0` while a descendant still writes (round 2, R2-1) |

+| D4 | destructive steps hold the replica lease of every location written (C3) | an exception for a **per-op prefix written only by that op**: `moves/<host>/<op id>/` in another region's bucket | the source writes staging cross-region with push-only IAM (Put, Delete, prefix-scoped List, no Get: `infra/modules/region-host/main.tf:217-227`, M17), so it cannot read `lock.json` there and cannot take a lease. Exclusion there comes from the op id in the key, the claim's revalidation, and the target never reading staging until the flip |

+| D5 | rollback adds a `park-replica` obligation for the target region (§5) | an obligation kind `park-superseded {region or box, op id}` with destination `superseded/<op id>/<host>/` | `park-replica` parks into `parked/<incarnation>/<host>/`, and "a manifest that is present is never rewritten". The incarnation is still live; if it later moves back to that region and is deleted, the delete's park finds the rollback's manifest, skips the copy, and then stops on unlisted keys. A per-op destination never collides |

+| D6 | `--declare-lost` removes the box's `park-local` obligations and its `touched.boxes` entry (§4) | also: append the box to `op.journal.lost`; a box in `lost` never CAS-adds itself back to `touched` and never spawns the host until the claim is released; refused after commit; refused if the lost box is the source | `doWake` CAS-adds a missing box to `touched` (§8 there). A box that was only partitioned would re-add itself on return and wake, beside the source |

+### 19.4 Joint simulation (claim spec §10)

+§14.2's simulation is built to run with the claim's model, not beside it:

+- **One claim model.** The claim object, its CAS, `touched`, obligations,

+ leases with `SafeUntil` and the clock assumption of claim spec §3.1 come

+ from the claim spec's model. This spec's actors use only its interface:

+ read, CAS, lease acquire and renew.

+- **Shared actors.** Delete-park, create, reconcile, wake and the db door

+ are the claim spec's actors; source, target, the deadline sweep and the

+ application process tree are this spec's. Every actor exposes the same

+ `Step(rng) (effect, done)` contract at side-effect granularity.

+- **Shared fault menu.** Crash with memory loss, stall, partition (a box

+ that keeps running but cannot reach the buckets), surviving descendants,

+ versioned objects, forged objects, old binaries.

+- **Union of invariants.** The claim spec's C1 to C6 and this spec's INV-1

+ to INV-10, asserted together after every step and at the end of every

+ run.

+- **Union of mutations.** Each spec's mutation list runs against the joint

+ model; the gate passes only if every mutation from both lists is caught.

+## 20. Copy (pending copy review)

+Drafts for the operator CLI. None is approved; every string goes through the

+copy-review gate. No em dashes.

| id | string |

|---|---|

@@ -1072,40 +1149,61 @@ from nearest-region §14. `<…>` marks a value. No em dashes.

| M-C15 | <host> keeps more than 256 MiB of files beside its database. Keep large files in object storage. |

| M-C16 | <host> does not answer its readiness path. Declare moves.ready_path for this app. |

| M-C17 | <host> uses private storage. Instances with private storage cannot move yet. |

-| M-C18 | This declares <box> lost. Anything <host> wrote there after <time> is lost. Type the host name to confirm. |

-## 20. Round 1 dispositions

+| M-C18 | This declares <box> lost. Anything <host> wrote on <box> after <time> is lost. Type the host name to confirm. |

+| M-C19 | <host> is being moved. Delete it after the move finishes, or abort the move first. |

+| M-C20 | <box> is running a process for <host> that this move cannot account for. An operator has been paged. |

+## 21. Out of scope for Phase 1, restated

+Owners, stores, app moves, moves into home, residency deletion, hooks and

+sidecars are F1 to F7. Phase 1 lands flat; a nested target is refused until

+the layout pass (P9) and the claim both cover it.

+## 22. Dispositions

+### 22.1 Round 1 (on e41c4fef), as of rev 3

+| # | finding | rev 2 | round 2 verdict | rev 3 |

+|---|---|---|---|---|

+| 8 to 10 | P5 deletion | F3 | deferred, acceptable | |

+| 11 | member token bypass | operator-only | resolved for Phase 1 | M18 |

+| 17, 18 | stores, app moves | F4, F5 | deferred, acceptable | |

+### 22.2 Round 2 (on 45b77691)

| # | finding | disposition |

|---|---|---|

-| 1 | deadline does not prove the target stopped | Accepted. Execution right moves only on stop evidence (§5.3); a deadline only requests release (§7.8) |

-| 2 | probe writes are lost on rollback | Accepted. Clean/dirty verdict by digests; a dirty rollback carries the target's state back over a return lane checked by T10 (§7.8); INV-2 covers background and external work |

-| 3 | rollback opens before checking | Accepted. Recovery runs held as `restoring`, opens only after its probe; `rolling_back`/`stuck` never mint `ready` (§5.2, §7.8) |

-| 4 | no recovery between stop and flip | Accepted. Journaled `cut_phase`, `reverting`, bounded phases; admission and abort are one CAS field (§7.4) |

-| 5 | replayed landing mixes old files with a newer DB | Accepted. Journaled `landing_phase`, no replay after `probing`, generation markers and the start check, unique park destinations (§5.1, §7.6) |

-| 6 | clearing `move` before finalization | Accepted. `committed` keeps `move` through open and source park; opens are conditional on `hold_op` (§5.2, §7.9) |

-| 7 | rollback floor does not cover the landing fence | Accepted. A move-protocol floor with a box-enforced downgrade refusal (§11) |

-| 8 | P5 can delete a live generation | Phase 1 deletes nothing (INV-8) and parks under unique move ids; the deletion rules are F3's invariants (§16) |

-| 9 | P5 leaves versions behind | F3 invariant (§16) |

-| 10 | warrant does not bind its scope | F3 and F4 invariant (§16) |

-| 11 | member-minted token bypasses the owner gate | Phase 1 move mutations are operator-only (§12); delegation is F1's invariant (§16) |

-| 12 | private-store and rollback commit points conflict | Phase 1 refuses private stores (§7.1); the joint transaction is F4's invariant (§16) |

-| 13 | helpers exclude held rows | Accepted. Held render with move-bound acks, held materialization, `RetireExcept` exemption (§7.7, §9.1 to §9.3) |

-| 14 | readiness checks the wrong environment and process | Accepted. Merged-environment equality by HMAC digests, read/write store probe, process-bound health with an explicit readiness path, source probed too (§7.3, §7.7) |

-| 15 | timers neither frozen nor restored | Accepted. Runner freeze, no prune of held rows, source-readable snapshot, exact `ReplaceTimers` (§9.4) |

-| 16 | inventory misses registry state | Accepted. Every column classified; `Replication` and `NextDue` carried; `Caps` refused (§6.2) |

-| 17 | store record omitted | F4 invariant (§16) |

-| 18 | app-move child identity and store barrier | F5 invariant (§16) |

-| 19 | archive replay contradicts encryption | Accepted. Plaintext and sealed identities; one create-only sealed object per cut, reused on replay; retention stated (§8.3) |

-| 20 | cause still unconfirmed | Accepted. Confirmed from the UK, US and AU journals; remediation and fingerprint verification (§1.2, §10, §15) |

-Astra's interface dispositions on rev 1's §26 are superseded by #199 rev 3,

-which dropped storage keys, generations as record fields, the epoch and

-`Root: "moves"`. Their substance carries into rev 2: one key function

-(I-1 above), exclusion through finalization (§7.9), blocking reads during

-transitional states (F7 inherits it), no "staging wins" (§5.1), and the

-landed copy installed into `instances/` (M15).

-Astra's §23 recommendations matched rev 1's and Paul accepted them as M4 to

-M15, with Astra's additions folded in: M5's streaming enforcement (§8.1),

-M11's duration gate (T11), and M14's single landing path.

+| R2-1 | flat-exec descendant outlives evidence | **Accepted.** Strict per-host cgroup for every spawn under a move; evidence requires `cgroup.kill` and an unpopulated cgroup, no stray process in the agent's cgroup, no open database (§5.3); the simulation models surviving descendants |

+| R2-2 | evidence lost with activator memory | **Accepted.** `ReconstructStopEvidence` reads only actual state (cgroup, unit, `/proc`, lease) and covers a never-spawned operation; a missing entry is never evidence (§5.3) |

+| R2-3 | no durable source-recovery journal | **Accepted.** `rollback.source_phase` with its own install phases through `probing`, recovery commit and open; never reinstalls after `probing`; `move` kept until recovery and target cleaning are stamped (§7.8) |

+| R2-4 | reconcile removes the rollback target | **Accepted.** The claim keeps a move op (`restoring`) with both participants, and the claim spec §8 retains participant rows; reconcile never removes a participant's row; the target deletes its own row only in `cleaning` (§7.8, §9.6, §19 K9) |

+| R2-5 | rolled-back target leaves a live replica | **Accepted.** The target's `cleaning` parks its `instances/<host>/` and staging out of the live namespace before the move closes; INV-10 (§4, §7.8) |

+| R2-6 | markers published before install completes | **Accepted.** Park, copy and publish are separate journaled phases; `.files-gen` is written after env, timers and columns; markers carry op and incarnation, and the start check compares values, not existence (§5.1, §6.4, §7.6) |

+| R2-7 | mirroring deletes co-located metadata | **Accepted.** `replica/`, `files/`, `timers/` subprefixes and `move.json` outside them; `MirrorPrefix` touches `replica/` only (§7.3) |

+| R2-8 | T3 misses inherited variables, layouts and input pinning | **Accepted.** The report covers `childEnv`'s inherited variables; variables are classed (region, layout, box, equal); input versions are pinned at readiness and re-verified at admission and at `configured` (§7.3, §7.4, §7.6) |

+| R2-9 | markers and restore verdict missing from the inventory | **Accepted.** Markers classed as protocol metadata and excluded from archives and digests; `<dbPath>.gen` local-parked; the host's restore verdict removed at every install (§5.1, §6.1, §6.3) |

+| R2-10 | T11 omits most of the cut | **Accepted.** T11 estimates every phase from measured rates; per-phase budgets; recovery deadlines separate from the cut window (§7.3, §7.4) |

+**Interfaces (round 2's table).** I-3 and I-7 are recorded as #199

+amendments (§17). I-5, I-6 and I-9 are now the claim's job (§19). I-2's

+other-region obligation disappears because a delete cannot run mid-move

+(§7.12).

+**Former §23 questions.** Round 2's recommendations match M4 to M15,

+including "require #199's epoch" now read as M14's single landing path.

+Rev 2's §18 question is M16, with Astra's three conditions built in

+(§7.10).