amadan
spec: instance moves — prepare copies the replica prefix, never tails the DB twice (planning amendment)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PEypt7de2X1G84aQ9rNkCf
1 file changed,
+41
−29
docs/superpowers/specs/2026-08-28-instance-move-design.md+41 −29
diff --git a/docs/superpowers/specs/2026-08-28-instance-move-design.md b/docs/superpowers/specs/2026-08-28-instance-move-design.md| index f283f8a97..bc9ca5a82 100644 |
| --- a/docs/superpowers/specs/2026-08-28-instance-move-design.md |
| +++ b/docs/superpowers/specs/2026-08-28-instance-move-design.md |
| @@ -74,41 +74,54 @@ instance itself): |
| - State is read, not pushed: the existing single-record GET already |
| returns the record verbatim, so `move.state` is pollable today with |
| zero new read surface. |
| +- Every move write bumps `control/serial` (the config fast-convergence |
| + nudge the env door already uses), so the source box notices in |
| + seconds, not on the hourly floor. Between states the source's own |
| + move worker self-paces its delta rounds. |
| ## What the source box does |
| The reconcile/activator pass on the box whose `CARLOS_REGION` matches |
| `region` (the source — the record still points at it): |
| -1. **preparing**: alongside its normal own-bucket replication, the |
| - activator opens a SECOND litestream replica of the instance DB to |
| - `s3://<target-data-bucket>/moves/<host>/` — the lease package's |
| - `ReplicateWithLease` pointed at the sibling bucket's staging prefix. |
| +1. **preparing** (AMENDED during planning, 2026-08-28: the draft said |
| + "a second litestream replica of the live DB" — two litestream |
| + processes cannot tail one WAL, both want to own checkpointing. The |
| + fallback IS the design, and it is better): the move copies the |
| + REPLICA, never the database. The activator already keeps |
| + `instances/<host>/` in the source's own bucket continuously synced |
| + (running child) or final-checkpointed (parked child) — the move |
| + worker mirrors that prefix into |
| + `s3://<target-data-bucket>/moves/<host>/` with server-side |
| + CopyObject (its IAM: read own bucket, push sibling staging), in |
| + delta rounds every ~30s — List both sides, copy new/changed keys, |
| + delete keys the source pruned (a true mirror, so litestream |
| + retention holds in staging too). Uniform for running and parked |
| + instances; no wake, no litestream internals touched. |
| Target bucket name and AWS region are derived, not configured |
| per-move: each box publishes its data bucket coordinates as two new |
| additive fields in its edges file (`data_bucket`, `data_bucket_region` |
| — same additive rule as `placement`), so the source reads the |
| target's coordinates from the same file set everything else reads. |
| - A wake is forced if the instance is asleep? NO — a parked instance's |
| - state is already fully in the source bucket; for a parked instance |
| - the "second replica" is a bucket-to-staging copy performed by the |
| - source box (its IAM can read its own bucket and push to the sibling's |
| - staging), and prepare completes without ever waking the child. |
| -2. **ready promotion**: once the staging replica reports caught up |
| - (litestream generation synced; for the parked case, copy complete), |
| - the source CASes `move.state: "preparing" → "ready"`, stamping |
| - `ready_at`. The second replica keeps streaming — ready is a level, |
| - not an edge; writes after ready keep flowing to staging. |
| +2. **ready promotion**: once a delta round completes with zero |
| + differences against a fresh listing, the source CASes |
| + `move.state: "preparing" → "ready"`, stamping `ready_at`. Rounds |
| + keep running — ready is a level, not an edge; writes after ready |
| + keep flowing into staging on the next round. |
| 3. **cutting**: on seeing `state:"cutting"`, the source (in order): |
| - final `SyncAndWait` on the staging replica → stop the child (normal |
| - stopping path, which also checkpoints to the source's OWN bucket — |
| - the abort-recovery copy) → close both replicators → **delete the |
| - local DB, WAL/SHM, and litestream meta dirs** (the warm-cache purge; |
| - without it a later move back would resurrect a stale local file — the |
| - activator's warm-cache trust is exactly right for crashes and exactly |
| - wrong here) → one CAS that sets `region: <to>` and clears `move` → |
| - remove its own route row on its next pass (the record has left its |
| - scope, the existing retirement path does this unprompted). |
| + set the route's Status to "moving" (a non-ready status: the edge |
| + stops serving it, the edges file drops it, and the activator's |
| + RetireExcept sweep stops the child through the ordinary stop path — |
| + which final-checkpoints to the source's OWN bucket, the |
| + abort-recovery copy) → wait for the child to be down → one final |
| + delta round into staging (now byte-identical to the own replica) → |
| + **delete the local DB, WAL/SHM, and litestream meta dirs** (the |
| + warm-cache purge; without it a later move back would resurrect a |
| + stale local file — the activator's warm-cache trust is exactly right |
| + for crashes and exactly wrong here) → one CAS that sets |
| + `region: <to>` and clears `move` → remove its own route row on its |
| + next pass (the record has left its scope, the existing retirement |
| + path does this unprompted). |
| If the box dies mid-cut, the record still says `cutting` with the old |
| region: the source resumes the sequence on restart (every step is |
| idempotent — sync again, stop a stopped child, delete absent files, |
| @@ -136,12 +149,11 @@ recovery door that already exists). |
| List, no Delete, never `instances/*`. Residency-at-rest holds |
| (cross-region reads remain impossible at IAM); the write blast radius |
| is confined to staging keys no box ever trusts as live state without |
| -its own restore-time `CheckDB`. Litestream against a push-only prefix: |
| -the lease/generation dance needs List on its own prefix — checked |
| -during build; if litestream cannot run push-only, prepare falls back to |
| -periodic own-checkpoint + copy-into-staging (same convergence, chunkier |
| -cadence), and the spec's contract (`ready` = staged copy caught up) |
| -does not change. |
| +its own restore-time `CheckDB`. The mirror needs three actions on the |
| +sibling, all `moves/*`-scoped: `s3:PutObject` and `s3:DeleteObject` on |
| +the object ARNs, and `s3:ListBucket` on the bucket ARN with an |
| +`s3:prefix` condition of `moves/*` — still no reads of sibling data, |
| +still nothing outside staging. |
| ## Instance env |