amadan
plan: instance moves (#331)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PEypt7de2X1G84aQ9rNkCf
1 file changed,
+109
−0
docs/superpowers/plans/2026-08-28-instance-moves.md+109 −0
diff --git a/docs/superpowers/plans/2026-08-28-instance-moves.md b/docs/superpowers/plans/2026-08-28-instance-moves.md| new file mode 100644 |
| index 000000000..3da28981a |
| --- /dev/null |
| +++ b/docs/superpowers/plans/2026-08-28-instance-moves.md |
| @@ -0,0 +1,109 @@ |
| +# Instance Moves Implementation Plan |
| + |
| +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| + |
| +**Goal:** Two-phase, data-following instance moves between regions (issue #331): prepare mirrors the instance's replica prefix into the target region's staging area; cut stops, final-syncs, purges, and flips the record's region. |
| + |
| +**Architecture:** The record carries `move:{to,state}`; the console writes requests (+ serial bump), the SOURCE box's move worker does everything else in delta rounds; the target box needs only a staging fallback in the activator's Restore. Never two writers: the region flip is the source's last act. |
| + |
| +**Tech Stack:** Go; S3 server-side CopyObject via aws-sdk-go-v2 (already a dependency through litestream/lss3); the repo's DirStore + fake patterns for tests. |
| + |
| +**Spec:** `docs/superpowers/specs/2026-08-28-instance-move-design.md` (as amended: prepare copies the REPLICA prefix, never tails the DB twice). |
| + |
| +## Global Constraints |
| + |
| +- Wire-shape rule: every field additive + omitempty; pre-change records/edges files parse unchanged. |
| +- Move verbs accept ONLY exec-backed `kind:"instance"` (empty kind = instance) routes; fleet placements (`fleet/<name>`) refused. |
| +- Target region must be in the armed-placement set (`armedPlacements`, #329) — no dormant path for moves. |
| +- Sibling-bucket IAM stays `moves/*`-scoped: Put/Delete on objects, List with prefix condition. Never `instances/*`, never Get. |
| +- Cut ordering is sacred: route Status "moving" → child down → final mirror round → local purge → CAS flip+clear. Idempotent on resume. |
| +- `TMPDIR=/tmp go test ./...` green before any commit claims done. |
| + |
| +--- |
| + |
| +### Task 1: Record move shape + CAS transitions (internal/provision) |
| + |
| +**Files:** Modify `internal/provision/provision.go`; test `internal/provision/provision_test.go`. |
| + |
| +**Produces:** |
| +- `type MoveState struct { To string `json:"to"`; State string `json:"state"`; RequestedAt time.Time `json:"requested_at"`; ReadyAt *time.Time `json:"ready_at,omitempty"` }` |
| +- `Record.Move *MoveState `json:"move,omitempty"`` |
| +- `const MovePreparing/MoveReady/MoveCutting = "preparing"/"ready"/"cutting"` |
| +- `func SetRecordMove(ctx, s, acct, app, host string, mut func(*Record) error) error` — one narrow read-CAS-write with one retry, `SetRecordRegion`'s exact shape, generalized: every transition (request, ready-promotion, abort-clear, cut-mark, flip+clear) is a `mut` that validates the current state and returns a typed error on an illegal transition (`ErrMoveConflict`). |
| + |
| +**Steps:** |
| +- [ ] Failing tests: request-on-clean sets preparing; request-on-inflight → ErrMoveConflict; ready only from preparing; cut only from ready; abort clears preparing/ready, refuses cutting; flip sets Region=To and clears Move in ONE write; lost-CAS retried once; pre-move JSON (no `move` key) round-trips byte-identical. |
| +- [ ] Implement; `TMPDIR=/tmp go test ./internal/provision`. |
| +- [ ] Commit. |
| + |
| +### Task 2: Edges files carry data-bucket coordinates (internal/host + console reader) |
| + |
| +**Files:** Modify `internal/host/edgesfile.go` (fields `DataBucket string `json:"data_bucket,omitempty"``, `DataBucketRegion string `json:"data_bucket_region,omitempty"`` on `edgesFile`, sourced in `edgesFileLoop` from `CARLOS_HIBERNATE_BUCKET`/`CARLOS_HIBERNATE_REGION`, two new `edgesFilePass` params like `placement`); `internal/console/instancesplacement.go` (same fields on `edgesFilePlacementView`). Tests: `internal/host/edgesfile_test.go` raw-wire body with/without. |
| + |
| +- [ ] Failing raw-wire test (set → present between placement and generation; unset → absent) → implement → test → commit. |
| + |
| +### Task 3: Console move verbs |
| + |
| +**Files:** Create `internal/console/apiinstancemove.go`; modify `internal/console/server.go` (routes `POST .../instances/{host}/move`, `POST .../instances/{host}/move/cut`, `DELETE .../instances/{host}/move`); test `internal/console/apiinstancemove_test.go`. |
| + |
| +**Consumes:** Task 1 helpers; `armedPlacements` (#329); `bumpSerial` (domainattach.go's helper — reuse, don't copy); `instanceGate`. |
| + |
| +**Behaviour matrix (each a test):** 202 + record shows preparing on happy path; 400 unarmed target naming armed set (armed==false is ALSO a 400 for moves — "this deployment has no armed regions"); 400 to==current region; 400 non-exec kind (static/service/addr) and fleet placement; 409 move in flight; cut: 409 unless ready; abort: 200 clears preparing/ready, 409 cutting; every successful write bumps control/serial; 404 matrix byte-identical to the instance GET's. |
| + |
| +- [ ] Failing tests → implement → test → commit. |
| + |
| +### Task 4: Prefix mirror (internal/lease/mirror.go) |
| + |
| +**Files:** Create `internal/lease/mirror.go`, `internal/lease/mirror_test.go`. |
| + |
| +**Produces:** |
| +- `type MirrorStore interface { List(ctx, bucket, prefix string) (map[string]string, error) /* key→etag */; Copy(ctx, srcBucket, srcKey, dstBucket, dstKey string) error; Delete(ctx, bucket, key string) error }` |
| +- `func NewS3Mirror(region, dstRegion string) MirrorStore` — aws-sdk-go-v2 client(s); CopyObject with URL-encoded CopySource; dst client in the sibling bucket's region. |
| +- `func MirrorPrefix(ctx context.Context, m MirrorStore, srcBucket, srcPrefix, dstBucket, dstPrefix string) (changed int, err error)` — list both, copy new/etag-mismatched, delete dst keys absent at src; returns changed==0 ⇒ caught up. |
| + |
| +- [ ] Failing tests against an in-memory fake MirrorStore: initial full copy; delta only; deletion mirrored; changed==0 when identical; error propagation → implement → test → commit. (NewS3Mirror itself is exercised live in Task 8's rehearsal, not unit-tested.) |
| + |
| +### Task 5: The source-box move worker (internal/host/movespass.go) |
| + |
| +**Files:** Create `internal/host/movespass.go`, `internal/host/movespass_test.go`. |
| + |
| +**Consumes:** Task 1 transitions, Task 4 `MirrorPrefix`, `readEdgesFiles`-equivalent host-side parse of `control/edges/` for target coordinates (host has its own edges reader in steering_loop.go — `loadEdgesFileBodies` + a small parse), `registry.SetStatus` (verify exact setter name at build time; the registry has route mutators), activator quiesce via the keep-set (Task 6). |
| + |
| +**Produces:** `func movesPass(ctx context.Context, s release.Store, reg *registry.Registry, cfg moveCfg) error` called from the agent loop (Task 7); `moveCfg{Region, OwnBucket, OwnBucketRegion, Prefix string; Mirror func(dstRegion string) lease.MirrorStore; DataDir string; Quiesced func(host string) bool}`. |
| + |
| +**Semantics (each pinned by a table-driven test with DirStore + fake mirror):** |
| +- Scans records in this box's scope with `Move != nil`. |
| +- preparing: resolve target coords from edges files (placement match; missing/ambiguous → log, skip — NOT a state change); one `MirrorPrefix` round `instances/<host>/ → moves/<host>/`; if changed==0 promote to ready (CAS). Abort seen (Move cleared by console) ⇒ nothing to do — rounds simply stop. |
| +- ready: keep mirroring each pass (writes after ready still flow). |
| +- cutting, in order: set route Status "moving" (edge stops serving; RetireExcept stops the child); if child still up (Quiesced false) → return, next pass retries; final MirrorPrefix round; delete local `<DataDir>/<host>.db{,-wal,-shm}` + litestream meta dirs (reuse the exact list from `ExecBackend.Restore`'s cleanup — extract a shared helper in Task 6); CAS flip Region=To + clear Move; route retirement left to the existing reconcile pass. |
| +- Resume idempotence: every cutting step re-runnable; a flip that lost the CAS because it already happened is terminal success. |
| +- The pass NEVER acts on a record whose Region ≠ cfg.Region (post-flip: source stops seeing it; target never runs movespass for it). |
| + |
| +- [ ] Failing tests (state table: prepare-not-caught-up no-op, prepare-caught-up promotes, ready keeps mirroring, cut full sequence ordering asserted via fake-mirror/op log, cut waits for quiesce, resume-from-cutting, missing target coords skip) → implement → test → commit. |
| + |
| +### Task 6: Activator: staging-fallback restore + shared purge + quiesce visibility |
| + |
| +**Files:** Modify `internal/activator/backend_exec.go`, `internal/activator/activator.go`; test in `internal/activator/backend_exec_test.go`. |
| + |
| +**Produces:** |
| +- `func PurgeLocalDB(dataDir, host, dbPath string)` — the one list of local artifacts (db, -wal, -shm, both litestream meta dirs), used by Restore's pre-restore cleanup AND Task 5's cut. |
| +- `ExecBackend.Restore` fallback: own `instances/<host>/` has no replica AND `moves/<host>/` exists in own bucket → server-side copy `moves/→instances/` (via a `Mirror lease.MirrorStore` field on ExecBackend, nil = no fallback), then restore as normal; delete `moves/<host>/` only after `CheckDB` passes; corrupt staged replica quarantines exactly like a corrupt own one and leaves staging in place (evidence). |
| +- `Service.Idle(host) bool` (or reuse an existing accessor if one exists — check `entries` map) for Task 5's `Quiesced`. |
| + |
| +- [ ] Failing tests (fallback copy+restore+cleanup; own-replica-beats-staging precedence; corrupt staging quarantines, staging kept; nil Mirror = old behaviour) → implement → test → commit. |
| + |
| +### Task 7: Agent wiring |
| + |
| +**Files:** Modify `internal/host/agent.go` (arm movesPass on the same trigger as the reconcile pass, gated like every duty: only when deployment bucket + CARLOS_REGION set — a box with no placement code can still be a move SOURCE? NO: a record with `region:""` lives on the home box which has no code; moves FROM the home box need cfg.Region==record.Region — "" == "" matches, so arm whenever the deployment bucket exists and pass cfg.Region verbatim, empty included); `ExecBackend.Mirror = lease.NewS3Mirror(...)` when hibernate bucket configured. |
| + |
| +- [ ] Wire, `go vet ./...`, full `TMPDIR=/tmp go test ./...`, commit. |
| + |
| +### Task 8: IAM module + rollout collateral |
| + |
| +**Files:** Modify `infra/modules/region-host/variables.tf` + `main.tf` (`variable "sibling_staging" { type = list(object({bucket_arn=string})) , default = [] }` → one policy statement: PutObject+DeleteObject on `${arn}/moves/*`, ListBucket on `${arn}` with `s3:prefix` condition `moves/*`); same for `region-remote-host`. README env/di contract notes (CARLOS_REGION on the home box optional, movesPass duty, staging prefix). |
| + |
| +- [ ] `tofu fmt` + `tofu validate` in the module dir (no apply — applies are per-deployment, tito's are tito-side per the 2026-08-28 ruling); commit. |
| + |
| +### Task 9: Gate + PR |
| + |
| +- [ ] Full suite + vet; self-review diff; PR referencing #331 with the amendment called out; live-rehearsal plan (flagship: create a canary instance in `us`, move it `us→uk`, prove serve + data survival, move back) listed in the PR as the deploy-time verification — NOT run until the fleet carries the binary and the IAM grant is applied. |