amadan
spec: instance moves — two-phase, data-following (issue #331, DRAFT for review)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PEypt7de2X1G84aQ9rNkCf
1 file changed,
+187
−0
docs/superpowers/specs/2026-08-28-instance-move-design.md+187 −0
diff --git a/docs/superpowers/specs/2026-08-28-instance-move-design.md b/docs/superpowers/specs/2026-08-28-instance-move-design.md| new file mode 100644 |
| index 000000000..f283f8a97 |
| --- /dev/null |
| +++ b/docs/superpowers/specs/2026-08-28-instance-move-design.md |
| @@ -0,0 +1,187 @@ |
| +# Instance moves: two-phase, data-following (issue #331) |
| + |
| +Status: DRAFT for review — 2026-08-28 |
| +Companion issues: #329 (region on create — shipped separately), #330 |
| +(replica_regions — shape only, untouched here), titogo residence design |
| +§5/§8.3 (the consumer contract this serves). |
| + |
| +## Decisions taken in conversation (Paul, 2026-08-28) |
| + |
| +- Cross-bucket IAM is **push-only into a staging prefix** (`moves/`), |
| + never into `instances/` — a box can stage state toward a sibling |
| + region but can never touch another region's live replica lineage. |
| +- The platform moves **the instance DB and record only**. The uploads |
| + bulk-copy stays with the tenant (titogo's router), which owns the |
| + bucket, the credentials, and the knowledge of which keys belong to |
| + which instance. The move state machine exposes the phases so the |
| + tenant can sequence its own copy inside the frozen window. If a |
| + per-instance uploads prefix ever becomes a declared platform concept |
| + (a record field), the move widens to carry it without unwinding this |
| + design. |
| + |
| +## The invariant |
| + |
| +*Residence is a property of a route; the record's `region` field is the |
| +residence carrier* (ingress spec §Anticipated #2). This spec never adds |
| +a second arbiter: at every instant exactly one region's box is in scope |
| +for the record, and the flip of `region` is the LAST act of the source. |
| +Never two writers falls out of that plus the freeze the tenant holds |
| +across the cut. |
| + |
| +## Wire shape |
| + |
| +The record gains one additive, omitempty object — present only while a |
| +move is in flight: |
| + |
| +```json |
| +"move": { |
| + "to": "uk", |
| + "state": "preparing" | "ready" | "cutting", |
| + "requested_at": "<RFC3339>", |
| + "ready_at": "<RFC3339, when state >= ready>" |
| +} |
| +``` |
| + |
| +Records written before this field parse unchanged (wire-shape rule). |
| +`move` is written by the console (the two verbs below) and by the |
| +SOURCE box (the `preparing → ready` promotion and the terminal |
| +clear-and-flip); each write is the narrow CAS `SetRecordRegion` already |
| +uses. The record's `region` stays the SOURCE region until the cut |
| +completes — every reader that knows nothing about moves keeps a |
| +consistent answer throughout. |
| + |
| +## The two verbs |
| + |
| +Console API, on one instance, gated exactly like every other instance |
| +door (app bearer; the tenant's router holds the token — never the |
| +instance itself): |
| + |
| +- `POST .../instances/{host}/move` body `{"region":"uk"}` — validated |
| + against the armed-placement set (#329's gate, mandatory here: a move |
| + to an unarmed region is always a 400, dormant never applies because a |
| + move NEEDS a live target box). Refused 409 if a move is already in |
| + flight, 400 if `to` equals the current region. Writes |
| + `move:{to,state:"preparing"}`. |
| +- `POST .../instances/{host}/move/cut` — refused 409 unless |
| + `move.state == "ready"`. Writes `move.state:"cutting"`. The tenant |
| + calls this only after freezing writes; the platform does not verify |
| + the freeze (it cannot — the freeze is app-level), it documents it. |
| +- `DELETE .../instances/{host}/move` — abort, legal in `preparing` or |
| + `ready` (clears `move`; the source drops its second replica; staged |
| + bytes in the target bucket are garbage that the next prepare |
| + overwrites and the target's post-restore cleanup deletes). Refused |
| + 409 in `cutting`: past that point the move completes forward. |
| +- State is read, not pushed: the existing single-record GET already |
| + returns the record verbatim, so `move.state` is pollable today with |
| + zero new read surface. |
| + |
| +## What the source box does |
| + |
| +The reconcile/activator pass on the box whose `CARLOS_REGION` matches |
| +`region` (the source — the record still points at it): |
| + |
| +1. **preparing**: alongside its normal own-bucket replication, the |
| + activator opens a SECOND litestream replica of the instance DB to |
| + `s3://<target-data-bucket>/moves/<host>/` — the lease package's |
| + `ReplicateWithLease` pointed at the sibling bucket's staging prefix. |
| + Target bucket name and AWS region are derived, not configured |
| + per-move: each box publishes its data bucket coordinates as two new |
| + additive fields in its edges file (`data_bucket`, `data_bucket_region` |
| + — same additive rule as `placement`), so the source reads the |
| + target's coordinates from the same file set everything else reads. |
| + A wake is forced if the instance is asleep? NO — a parked instance's |
| + state is already fully in the source bucket; for a parked instance |
| + the "second replica" is a bucket-to-staging copy performed by the |
| + source box (its IAM can read its own bucket and push to the sibling's |
| + staging), and prepare completes without ever waking the child. |
| +2. **ready promotion**: once the staging replica reports caught up |
| + (litestream generation synced; for the parked case, copy complete), |
| + the source CASes `move.state: "preparing" → "ready"`, stamping |
| + `ready_at`. The second replica keeps streaming — ready is a level, |
| + not an edge; writes after ready keep flowing to staging. |
| +3. **cutting**: on seeing `state:"cutting"`, the source (in order): |
| + final `SyncAndWait` on the staging replica → stop the child (normal |
| + stopping path, which also checkpoints to the source's OWN bucket — |
| + the abort-recovery copy) → close both replicators → **delete the |
| + local DB, WAL/SHM, and litestream meta dirs** (the warm-cache purge; |
| + without it a later move back would resurrect a stale local file — the |
| + activator's warm-cache trust is exactly right for crashes and exactly |
| + wrong here) → one CAS that sets `region: <to>` and clears `move` → |
| + remove its own route row on its next pass (the record has left its |
| + scope, the existing retirement path does this unprompted). |
| + If the box dies mid-cut, the record still says `cutting` with the old |
| + region: the source resumes the sequence on restart (every step is |
| + idempotent — sync again, stop a stopped child, delete absent files, |
| + CAS already-flipped is a no-op conflict it re-reads). |
| + |
| +## What the target box does |
| + |
| +Nothing move-specific until the flip — then its ordinary pass sees a |
| +record newly in scope and mints the route. One addition in the |
| +activator's `Restore`: when the local DB is absent AND |
| +`instances/<host>/` in its own bucket has no replica, it checks |
| +`moves/<host>/` (own bucket, own IAM); if present, server-side copy |
| +`moves/<host>/ → instances/<host>/`, restore from there, `CheckDB` as |
| +always, and delete `moves/<host>/` on success. A corrupt staged replica |
| +quarantines exactly like a corrupt own one — and the abort story still |
| +stands, because the source's last own-bucket checkpoint is intact and |
| +`region` can be pointed back by hand (`SetRecordRegion` — the operator |
| +recovery door that already exists). |
| + |
| +## IAM (the infra half — platform-infrastructure + tito repo applies) |
| + |
| +`region-host` (and `region-remote-host`) gain an optional |
| +`sibling_staging_arns` list; when set, the role gains `s3:PutObject` on |
| +`<each sibling data bucket>/moves/*` — and nothing else: no Get, no |
| +List, no Delete, never `instances/*`. Residency-at-rest holds |
| +(cross-region reads remain impossible at IAM); the write blast radius |
| +is confined to staging keys no box ever trusts as live state without |
| +its own restore-time `CheckDB`. Litestream against a push-only prefix: |
| +the lease/generation dance needs List on its own prefix — checked |
| +during build; if litestream cannot run push-only, prepare falls back to |
| +periodic own-checkpoint + copy-into-staging (same convergence, chunkier |
| +cadence), and the spec's contract (`ready` = staged copy caught up) |
| +does not change. |
| + |
| +## Instance env |
| + |
| +Already shipped with #329: every exec child gets `CARLOS_REGION` (the |
| +box's placement code), agent-owned in both directions. An app that |
| +lands on the target box after a move reads the new value at boot — |
| +titogo's boot-clears-freeze rule needs exactly this. |
| + |
| +## Out of scope |
| + |
| +- Uploads (tenant-side, by decision above). |
| +- `replica_regions` behavior (#330 stays shape-only). |
| +- Moving unit-backed, addr-backed, static, or everywhere-class routes: |
| + the verbs refuse anything but an exec-backed `kind:"instance"` route |
| + (400 naming the kind) — the only class whose state the platform owns. |
| +- Fleet placements (`fleet/<name>`): refused; fleets keep D6 |
| + state-does-not-follow semantics on purpose. |
| +- Automatic freeze verification; region-aware unit economics; the |
| + member-facing residence door (titogo builds on these verbs). |
| + |
| +## Testing |
| + |
| +- Record CAS: move-field transitions (prepare/ready/cut/abort/conflict) |
| + against the fake store, including the 409 matrix. |
| +- Source pass: fake-store + fake-litestream harness proving the cut |
| + ordering (final sync BEFORE stop BEFORE local purge BEFORE flip), the |
| + parked-instance prepare (no wake), resume-from-`cutting` idempotence, |
| + and that abort leaves the source serving with its own replication |
| + intact. |
| +- Target restore: staging fallback (copy, restore, CheckDB, cleanup), |
| + corrupt-staging quarantine, and precedence (own `instances/` replica |
| + beats staging when both exist — a re-landed instance must not regress |
| + to an older staged copy). |
| +- Console: verb gates (bearer, kind, armed region, state machine), |
| + poll-through-GET. |
| +- The never-two-writers argument pinned by an ordering test: a request |
| + hitting the source during `cutting` after stop is a 503 from the edge |
| + (route gone on next pass) while the target has not yet minted. The |
| + length of that window is the target's reconcile latency, NOT assumed: |
| + measure it during build (the adopt path/timer cadence is the floor) |
| + and record it in the runbook — the tenant's freeze must be budgeted |
| + to cover it, and if it measures in minutes rather than seconds, a |
| + serial-bump nudge on the record write is the follow-up, not a redesign. |