Instance moves: two-phase, data-following (issue #331)

Status: DRAFT for review — 2026-08-28 Companion issues: #329 (region on create — shipped separately), #330 (replica_regions — shape only, untouched here), titogo residence design §5/§8.3 (the consumer contract this serves).

Decisions taken in conversation (Paul, 2026-08-28)

• Cross-bucket IAM is push-only into a staging prefix (moves/), never into instances/ — a box can stage state toward a sibling region but can never touch another region's live replica lineage.
• The platform moves the instance DB and record only. The uploads bulk-copy stays with the tenant (titogo's router), which owns the bucket, the credentials, and the knowledge of which keys belong to which instance. The move state machine exposes the phases so the tenant can sequence its own copy inside the frozen window. If a per-instance uploads prefix ever becomes a declared platform concept (a record field), the move widens to carry it without unwinding this design.

The invariant

Residence is a property of a route; the record's region field is the residence carrier (ingress spec §Anticipated #2). This spec never adds a second arbiter: at every instant exactly one region's box is in scope for the record, and the flip of region is the LAST act of the source. Never two writers falls out of that plus the freeze the tenant holds across the cut.

Wire shape

The record gains one additive, omitempty object — present only while a move is in flight:

"move": {
  "to": "uk",
  "state": "preparing" | "ready" | "cutting" | "aborting",
  "requested_at": "<RFC3339>",
  "ready_at": "<RFC3339, when state >= ready>"
}

Records written before this field parse unchanged (wire-shape rule). move is written by the console (the two verbs below) and by the SOURCE box (the preparing → ready promotion and the terminal clear-and-flip); each write is the narrow CAS SetRecordRegion already uses. The record's region stays the SOURCE region until the cut completes — every reader that knows nothing about moves keeps a consistent answer throughout.

The two verbs

Console API, on one instance, gated exactly like every other instance door (app bearer; the tenant's router holds the token — never the instance itself):

• POST .../instances/{host}/move body {"region":"uk"} — validated against the armed-placement set (#329's gate, mandatory here: a move to an unarmed region is always a 400, dormant never applies because a move NEEDS a live target box). Refused 409 if a move is already in flight, 400 if to equals the current region. Writes move:{to,state:"preparing"}.
• POST .../instances/{host}/move/cut — refused 409 unless move.state == "ready". Writes move.state:"cutting". The tenant calls this only after freezing writes; the platform does not verify the freeze (it cannot — the freeze is app-level), it documents it.
• DELETE .../instances/{host}/move — abort, legal in preparing or ready. TWO-PHASE (review round 1): the door marks aborting; the SOURCE box deletes the staged copy from the target bucket and then clears move — a bare clear would leave staging behind to win a later landing on the target (staging-wins below). Refused 409 in cutting (completes forward) and in aborting (already under way).
• State is read, not pushed: the existing single-record GET already returns the record verbatim, so move.state is pollable today with zero new read surface.
• Every move write bumps control/serial (the config fast-convergence nudge the env door already uses), so the source box notices in seconds, not on the hourly floor. Between states the source's own move worker self-paces its delta rounds.

What the source box does

The reconcile/activator pass on the box whose CARLOS_REGION matches region (the source — the record still points at it):

1. preparing (AMENDED during planning, 2026-08-28: the draft said "a second litestream replica of the live DB" — two litestream processes cannot tail one WAL, both want to own checkpointing. The fallback IS the design, and it is better): the move copies the REPLICA, never the database. The activator already keeps instances/<host>/ in the source's own bucket continuously synced (running child) or final-checkpointed (parked child) — the move worker mirrors that prefix into s3://<target-data-bucket>/moves/<host>/ with server-side CopyObject (its IAM: read own bucket, push sibling staging), in delta rounds every ~30s — List both sides, copy new/changed keys, delete keys the source pruned (a true mirror, so litestream retention holds in staging too). Uniform for running and parked instances; no wake, no litestream internals touched. Target bucket name and AWS region are derived, not configured per-move: each box publishes its data bucket coordinates as two new additive fields in its edges file (data_bucket, data_bucket_region — same additive rule as placement), so the source reads the target's coordinates from the same file set everything else reads.
2. ready promotion: once a delta round completes with zero differences against a fresh listing, the source CASes move.state: "preparing" → "ready", stamping ready_at. Rounds keep running — ready is a level, not an edge; writes after ready keep flowing into staging on the next round.
3. cutting (order amended, review round 1): on seeing state:"cutting", the source (in order): set the route's Status to "moving" (a non-ready status: the edge stops serving it, the edges file drops it, and the activator's RetireExcept sweep stops the child through the ordinary stop path — which final-checkpoints to the source's OWN bucket — but SWALLOWS a checkpoint failure, safe only while the warm cache stood behind it) → wait for the child to be down → re-checkpoint the local DB with the lease and wait (proving the own replica carries the tail — review round 1, the failed-final-checkpoint data-loss hole) → one final delta round into staging (now byte-identical to the own replica) → delete the local DB, WAL/SHM, and litestream meta dirs (the warm-cache purge; without it a later move back would resurrect a stale local file — the activator's warm-cache trust is exactly right for crashes and exactly wrong here) → one CAS that sets region: <to> and clears move → remove its own route row on its next pass (the record has left its scope, the existing retirement path does this unprompted). If the box dies mid-cut, the record still says cutting with the old region: the source resumes the sequence on restart (every step is idempotent — sync again, stop a stopped child, delete absent files, CAS already-flipped is a no-op conflict it re-reads).

What the target box does

Nothing move-specific until the flip — then its ordinary pass sees a record newly in scope and mints the route. One addition in the activator's Restore: when the local DB is absent AND instances/<host>/ in its own bucket has no replica, it checks moves/<host>/ (own bucket, own IAM); if present, server-side copy moves/<host>/ → instances/<host>/, restore from there, CheckDB as always, and delete moves/<host>/ on success. A corrupt staged replica quarantines exactly like a corrupt own one — and the abort story still stands, because the source's last own-bucket checkpoint is intact and region can be pointed back by hand (SetRecordRegion — the operator recovery door that already exists).

IAM (the infra half — platform-infrastructure + tito repo applies)

region-host (and region-remote-host) gain an optional sibling_staging_arns list; when set, the role gains s3:PutObject on <each sibling data bucket>/moves/* — and nothing else: no Get, no List, no Delete, never instances/*. Residency-at-rest holds (cross-region reads remain impossible at IAM); the write blast radius is confined to staging keys no box ever trusts as live state without its own restore-time CheckDB. The mirror needs three actions on the sibling, all moves/*-scoped: s3:PutObject and s3:DeleteObject on the object ARNs, and s3:ListBucket on the bucket ARN with an s3:prefix condition of moves/* — still no reads of sibling data, still nothing outside staging.

Instance env

Already shipped with #329: every exec child gets CARLOS_REGION (the box's placement code), agent-owned in both directions. An app that lands on the target box after a move reads the new value at boot — titogo's boot-clears-freeze rule needs exactly this.

Out of scope

• Uploads (tenant-side, by decision above).
• replica_regions behavior (#330 stays shape-only).
• Moving unit-backed, addr-backed, static, or everywhere-class routes: the verbs refuse anything but an exec-backed kind:"instance" route (400 naming the kind) — the only class whose state the platform owns.
• Fleet placements (fleet/<name>): refused; fleets keep D6 state-does-not-follow semantics on purpose.
• Pooled records (pool != ""): refused in v1 — their data lives on the pool's box, not the platform region box that would drive the move. Non-hibernating instances (hibernate:false): refused — they run outside the activator, so nothing manages (or could quiesce) their data.
• The worker acts only on routes its OWN registry serves: region equality alone would let a standby or second same-region box drive a cut whose child runs on its sibling (per-box registries make its Quiesced trivially true).
• Automatic freeze verification; region-aware unit economics; the member-facing residence door (titogo builds on these verbs).

Testing

• Record CAS: move-field transitions (prepare/ready/cut/abort/conflict) against the fake store, including the 409 matrix.
• Source pass: fake-store + fake-litestream harness proving the cut ordering (final sync BEFORE stop BEFORE local purge BEFORE flip), the parked-instance prepare (no wake), resume-from-cutting idempotence, and that abort leaves the source serving with its own replication intact.
• Target restore: staging fallback (copy, restore, CheckDB, cleanup), corrupt-staging quarantine, and precedence (own instances/ replica beats staging when both exist — a re-landed instance must not regress to an older staged copy).
• Console: verb gates (bearer, kind, armed region, state machine), poll-through-GET.
• The never-two-writers argument pinned by an ordering test: a request hitting the source during cutting after stop is a 503 from the edge (route gone on next pass) while the target has not yet minted. The length of that window is the target's reconcile latency, NOT assumed: measure it during build (the adopt path/timer cadence is the floor) and record it in the runbook — the tenant's freeze must be budgeted to cover it, and if it measures in minutes rather than seconds, a serial-bump nudge on the record write is the follow-up, not a redesign.