| 1 | # Instance Moves Implementation Plan |
| 2 | |
| 3 | > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| 4 | |
| 5 | **Goal:** Two-phase, data-following instance moves between regions (issue #331): prepare mirrors the instance's replica prefix into the target region's staging area; cut stops, final-syncs, purges, and flips the record's region. |
| 6 | |
| 7 | **Architecture:** The record carries `move:{to,state}`; the console writes requests (+ serial bump), the SOURCE box's move worker does everything else in delta rounds; the target box needs only a staging fallback in the activator's Restore. Never two writers: the region flip is the source's last act. |
| 8 | |
| 9 | **Tech Stack:** Go; S3 server-side CopyObject via aws-sdk-go-v2 (already a dependency through litestream/lss3); the repo's DirStore + fake patterns for tests. |
| 10 | |
| 11 | **Spec:** `docs/superpowers/specs/2026-08-28-instance-move-design.md` (as amended: prepare copies the REPLICA prefix, never tails the DB twice). |
| 12 | |
| 13 | ## Global Constraints |
| 14 | |
| 15 | - Wire-shape rule: every field additive + omitempty; pre-change records/edges files parse unchanged. |
| 16 | - Move verbs accept ONLY exec-backed `kind:"instance"` (empty kind = instance) routes; fleet placements (`fleet/<name>`) refused. |
| 17 | - Target region must be in the armed-placement set (`armedPlacements`, #329) — no dormant path for moves. |
| 18 | - Sibling-bucket IAM stays `moves/*`-scoped: Put/Delete on objects, List with prefix condition. Never `instances/*`, never Get. |
| 19 | - Cut ordering is sacred: route Status "moving" → child down → final mirror round → local purge → CAS flip+clear. Idempotent on resume. |
| 20 | - `TMPDIR=/tmp go test ./...` green before any commit claims done. |
| 21 | |
| 22 | --- |
| 23 | |
| 24 | ### Task 1: Record move shape + CAS transitions (internal/provision) |
| 25 | |
| 26 | **Files:** Modify `internal/provision/provision.go`; test `internal/provision/provision_test.go`. |
| 27 | |
| 28 | **Produces:** |
| 29 | - `type MoveState struct { To string `json:"to"`; State string `json:"state"`; RequestedAt time.Time `json:"requested_at"`; ReadyAt *time.Time `json:"ready_at,omitempty"` }` |
| 30 | - `Record.Move *MoveState `json:"move,omitempty"`` |
| 31 | - `const MovePreparing/MoveReady/MoveCutting = "preparing"/"ready"/"cutting"` |
| 32 | - `func SetRecordMove(ctx, s, acct, app, host string, mut func(*Record) error) error` — one narrow read-CAS-write with one retry, `SetRecordRegion`'s exact shape, generalized: every transition (request, ready-promotion, abort-clear, cut-mark, flip+clear) is a `mut` that validates the current state and returns a typed error on an illegal transition (`ErrMoveConflict`). |
| 33 | |
| 34 | **Steps:** |
| 35 | - [ ] Failing tests: request-on-clean sets preparing; request-on-inflight → ErrMoveConflict; ready only from preparing; cut only from ready; abort clears preparing/ready, refuses cutting; flip sets Region=To and clears Move in ONE write; lost-CAS retried once; pre-move JSON (no `move` key) round-trips byte-identical. |
| 36 | - [ ] Implement; `TMPDIR=/tmp go test ./internal/provision`. |
| 37 | - [ ] Commit. |
| 38 | |
| 39 | ### Task 2: Edges files carry data-bucket coordinates (internal/host + console reader) |
| 40 | |
| 41 | **Files:** Modify `internal/host/edgesfile.go` (fields `DataBucket string `json:"data_bucket,omitempty"``, `DataBucketRegion string `json:"data_bucket_region,omitempty"`` on `edgesFile`, sourced in `edgesFileLoop` from `CARLOS_HIBERNATE_BUCKET`/`CARLOS_HIBERNATE_REGION`, two new `edgesFilePass` params like `placement`); `internal/console/instancesplacement.go` (same fields on `edgesFilePlacementView`). Tests: `internal/host/edgesfile_test.go` raw-wire body with/without. |
| 42 | |
| 43 | - [ ] Failing raw-wire test (set → present between placement and generation; unset → absent) → implement → test → commit. |
| 44 | |
| 45 | ### Task 3: Console move verbs |
| 46 | |
| 47 | **Files:** Create `internal/console/apiinstancemove.go`; modify `internal/console/server.go` (routes `POST .../instances/{host}/move`, `POST .../instances/{host}/move/cut`, `DELETE .../instances/{host}/move`); test `internal/console/apiinstancemove_test.go`. |
| 48 | |
| 49 | **Consumes:** Task 1 helpers; `armedPlacements` (#329); `bumpSerial` (domainattach.go's helper — reuse, don't copy); `instanceGate`. |
| 50 | |
| 51 | **Behaviour matrix (each a test):** 202 + record shows preparing on happy path; 400 unarmed target naming armed set (armed==false is ALSO a 400 for moves — "this deployment has no armed regions"); 400 to==current region; 400 non-exec kind (static/service/addr) and fleet placement; 409 move in flight; cut: 409 unless ready; abort: 200 clears preparing/ready, 409 cutting; every successful write bumps control/serial; 404 matrix byte-identical to the instance GET's. |
| 52 | |
| 53 | - [ ] Failing tests → implement → test → commit. |
| 54 | |
| 55 | ### Task 4: Prefix mirror (internal/lease/mirror.go) |
| 56 | |
| 57 | **Files:** Create `internal/lease/mirror.go`, `internal/lease/mirror_test.go`. |
| 58 | |
| 59 | **Produces:** |
| 60 | - `type MirrorStore interface { List(ctx, bucket, prefix string) (map[string]string, error) /* key→etag */; Copy(ctx, srcBucket, srcKey, dstBucket, dstKey string) error; Delete(ctx, bucket, key string) error }` |
| 61 | - `func NewS3Mirror(region, dstRegion string) MirrorStore` — aws-sdk-go-v2 client(s); CopyObject with URL-encoded CopySource; dst client in the sibling bucket's region. |
| 62 | - `func MirrorPrefix(ctx context.Context, m MirrorStore, srcBucket, srcPrefix, dstBucket, dstPrefix string) (changed int, err error)` — list both, copy new/etag-mismatched, delete dst keys absent at src; returns changed==0 ⇒ caught up. |
| 63 | |
| 64 | - [ ] Failing tests against an in-memory fake MirrorStore: initial full copy; delta only; deletion mirrored; changed==0 when identical; error propagation → implement → test → commit. (NewS3Mirror itself is exercised live in Task 8's rehearsal, not unit-tested.) |
| 65 | |
| 66 | ### Task 5: The source-box move worker (internal/host/movespass.go) |
| 67 | |
| 68 | **Files:** Create `internal/host/movespass.go`, `internal/host/movespass_test.go`. |
| 69 | |
| 70 | **Consumes:** Task 1 transitions, Task 4 `MirrorPrefix`, `readEdgesFiles`-equivalent host-side parse of `control/edges/` for target coordinates (host has its own edges reader in steering_loop.go — `loadEdgesFileBodies` + a small parse), `registry.SetStatus` (verify exact setter name at build time; the registry has route mutators), activator quiesce via the keep-set (Task 6). |
| 71 | |
| 72 | **Produces:** `func movesPass(ctx context.Context, s release.Store, reg *registry.Registry, cfg moveCfg) error` called from the agent loop (Task 7); `moveCfg{Region, OwnBucket, OwnBucketRegion, Prefix string; Mirror func(dstRegion string) lease.MirrorStore; DataDir string; Quiesced func(host string) bool}`. |
| 73 | |
| 74 | **Semantics (each pinned by a table-driven test with DirStore + fake mirror):** |
| 75 | - Scans records in this box's scope with `Move != nil`. |
| 76 | - preparing: resolve target coords from edges files (placement match; missing/ambiguous → log, skip — NOT a state change); one `MirrorPrefix` round `instances/<host>/ → moves/<host>/`; if changed==0 promote to ready (CAS). Abort seen (Move cleared by console) ⇒ nothing to do — rounds simply stop. |
| 77 | - ready: keep mirroring each pass (writes after ready still flow). |
| 78 | - cutting, in order: set route Status "moving" (edge stops serving; RetireExcept stops the child); if child still up (Quiesced false) → return, next pass retries; final MirrorPrefix round; delete local `<DataDir>/<host>.db{,-wal,-shm}` + litestream meta dirs (reuse the exact list from `ExecBackend.Restore`'s cleanup — extract a shared helper in Task 6); CAS flip Region=To + clear Move; route retirement left to the existing reconcile pass. |
| 79 | - Resume idempotence: every cutting step re-runnable; a flip that lost the CAS because it already happened is terminal success. |
| 80 | - The pass NEVER acts on a record whose Region ≠ cfg.Region (post-flip: source stops seeing it; target never runs movespass for it). |
| 81 | |
| 82 | - [ ] Failing tests (state table: prepare-not-caught-up no-op, prepare-caught-up promotes, ready keeps mirroring, cut full sequence ordering asserted via fake-mirror/op log, cut waits for quiesce, resume-from-cutting, missing target coords skip) → implement → test → commit. |
| 83 | |
| 84 | ### Task 6: Activator: staging-fallback restore + shared purge + quiesce visibility |
| 85 | |
| 86 | **Files:** Modify `internal/activator/backend_exec.go`, `internal/activator/activator.go`; test in `internal/activator/backend_exec_test.go`. |
| 87 | |
| 88 | **Produces:** |
| 89 | - `func PurgeLocalDB(dataDir, host, dbPath string)` — the one list of local artifacts (db, -wal, -shm, both litestream meta dirs), used by Restore's pre-restore cleanup AND Task 5's cut. |
| 90 | - `ExecBackend.Restore` fallback: own `instances/<host>/` has no replica AND `moves/<host>/` exists in own bucket → server-side copy `moves/→instances/` (via a `Mirror lease.MirrorStore` field on ExecBackend, nil = no fallback), then restore as normal; delete `moves/<host>/` only after `CheckDB` passes; corrupt staged replica quarantines exactly like a corrupt own one and leaves staging in place (evidence). |
| 91 | - `Service.Idle(host) bool` (or reuse an existing accessor if one exists — check `entries` map) for Task 5's `Quiesced`. |
| 92 | |
| 93 | - [ ] Failing tests (fallback copy+restore+cleanup; own-replica-beats-staging precedence; corrupt staging quarantines, staging kept; nil Mirror = old behaviour) → implement → test → commit. |
| 94 | |
| 95 | ### Task 7: Agent wiring |
| 96 | |
| 97 | **Files:** Modify `internal/host/agent.go` (arm movesPass on the same trigger as the reconcile pass, gated like every duty: only when deployment bucket + CARLOS_REGION set — a box with no placement code can still be a move SOURCE? NO: a record with `region:""` lives on the home box which has no code; moves FROM the home box need cfg.Region==record.Region — "" == "" matches, so arm whenever the deployment bucket exists and pass cfg.Region verbatim, empty included); `ExecBackend.Mirror = lease.NewS3Mirror(...)` when hibernate bucket configured. |
| 98 | |
| 99 | - [ ] Wire, `go vet ./...`, full `TMPDIR=/tmp go test ./...`, commit. |
| 100 | |
| 101 | ### Task 8: IAM module + rollout collateral |
| 102 | |
| 103 | **Files:** Modify `infra/modules/region-host/variables.tf` + `main.tf` (`variable "sibling_staging" { type = list(object({bucket_arn=string})) , default = [] }` → one policy statement: PutObject+DeleteObject on `${arn}/moves/*`, ListBucket on `${arn}` with `s3:prefix` condition `moves/*`); same for `region-remote-host`. README env/di contract notes (CARLOS_REGION on the home box optional, movesPass duty, staging prefix). |
| 104 | |
| 105 | - [ ] `tofu fmt` + `tofu validate` in the module dir (no apply — applies are per-deployment, tito's are tito-side per the 2026-08-28 ruling); commit. |
| 106 | |
| 107 | ### Task 9: Gate + PR |
| 108 | |
| 109 | - [ ] Full suite + vet; self-review diff; PR referencing #331 with the amendment called out; live-rehearsal plan (flagship: create a canary instance in `us`, move it `us→uk`, prove serve + data survival, move back) listed in the PR as the deploy-time verification — NOT run until the fleet carries the binary and the IAM grant is applied. |
| 110 | |
amadan