Instance Moves Implementation Plan

> For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Two-phase, data-following instance moves between regions (issue #331): prepare mirrors the instance's replica prefix into the target region's staging area; cut stops, final-syncs, purges, and flips the record's region.

Architecture: The record carries move:{to,state}; the console writes requests (+ serial bump), the SOURCE box's move worker does everything else in delta rounds; the target box needs only a staging fallback in the activator's Restore. Never two writers: the region flip is the source's last act.

Tech Stack: Go; S3 server-side CopyObject via aws-sdk-go-v2 (already a dependency through litestream/lss3); the repo's DirStore + fake patterns for tests.

Spec: docs/superpowers/specs/2026-08-28-instance-move-design.md (as amended: prepare copies the REPLICA prefix, never tails the DB twice).

Global Constraints

• Wire-shape rule: every field additive + omitempty; pre-change records/edges files parse unchanged.
• Move verbs accept ONLY exec-backed kind:"instance" (empty kind = instance) routes; fleet placements (fleet/<name>) refused.
• Target region must be in the armed-placement set (armedPlacements, #329) — no dormant path for moves.
• Sibling-bucket IAM stays moves/*-scoped: Put/Delete on objects, List with prefix condition. Never instances/*, never Get.
• Cut ordering is sacred: route Status "moving" → child down → final mirror round → local purge → CAS flip+clear. Idempotent on resume.
• TMPDIR=/tmp go test ./... green before any commit claims done.

---

Task 1: Record move shape + CAS transitions (internal/provision)

Files: Modify internal/provision/provision.go; test internal/provision/provision_test.go.

Produces:

• type MoveState struct { To string json:"to"; State string json:"state"; RequestedAt time.Time json:"requestedat"; ReadyAt *time.Time json:"readyat,omitempty" }
• Record.Move *MoveState json:"move,omitempty"``
• const MovePreparing/MoveReady/MoveCutting = "preparing"/"ready"/"cutting"
• func SetRecordMove(ctx, s, acct, app, host string, mut func(*Record) error) error — one narrow read-CAS-write with one retry, SetRecordRegion's exact shape, generalized: every transition (request, ready-promotion, abort-clear, cut-mark, flip+clear) is a mut that validates the current state and returns a typed error on an illegal transition (ErrMoveConflict).

Steps:

• [ ] Failing tests: request-on-clean sets preparing; request-on-inflight → ErrMoveConflict; ready only from preparing; cut only from ready; abort clears preparing/ready, refuses cutting; flip sets Region=To and clears Move in ONE write; lost-CAS retried once; pre-move JSON (no move key) round-trips byte-identical.
• [ ] Implement; TMPDIR=/tmp go test ./internal/provision.
• [ ] Commit.

Task 2: Edges files carry data-bucket coordinates (internal/host + console reader)

Files: Modify internal/host/edgesfile.go (fields DataBucket string json:"databucket,omitempty"`, DataBucketRegion string `json:"databucketregion,omitempty"` on edgesFile, sourced in edgesFileLoop from CARLOSHIBERNATEBUCKET/CARLOSHIBERNATEREGION, two new edgesFilePass params like placement); internal/console/instancesplacement.go (same fields on edgesFilePlacementView). Tests: internal/host/edgesfiletest.go` raw-wire body with/without.

• [ ] Failing raw-wire test (set → present between placement and generation; unset → absent) → implement → test → commit.

Task 3: Console move verbs

Files: Create internal/console/apiinstancemove.go; modify internal/console/server.go (routes POST .../instances/{host}/move, POST .../instances/{host}/move/cut, DELETE .../instances/{host}/move); test internal/console/apiinstancemove_test.go.

Consumes: Task 1 helpers; armedPlacements (#329); bumpSerial (domainattach.go's helper — reuse, don't copy); instanceGate.

Behaviour matrix (each a test): 202 + record shows preparing on happy path; 400 unarmed target naming armed set (armed==false is ALSO a 400 for moves — "this deployment has no armed regions"); 400 to==current region; 400 non-exec kind (static/service/addr) and fleet placement; 409 move in flight; cut: 409 unless ready; abort: 200 clears preparing/ready, 409 cutting; every successful write bumps control/serial; 404 matrix byte-identical to the instance GET's.

• [ ] Failing tests → implement → test → commit.

Task 4: Prefix mirror (internal/lease/mirror.go)

Files: Create internal/lease/mirror.go, internal/lease/mirror_test.go.

Produces:

• type MirrorStore interface { List(ctx, bucket, prefix string) (map[string]string, error) /* key→etag */; Copy(ctx, srcBucket, srcKey, dstBucket, dstKey string) error; Delete(ctx, bucket, key string) error }
• func NewS3Mirror(region, dstRegion string) MirrorStore — aws-sdk-go-v2 client(s); CopyObject with URL-encoded CopySource; dst client in the sibling bucket's region.
• func MirrorPrefix(ctx context.Context, m MirrorStore, srcBucket, srcPrefix, dstBucket, dstPrefix string) (changed int, err error) — list both, copy new/etag-mismatched, delete dst keys absent at src; returns changed==0 ⇒ caught up.

• [ ] Failing tests against an in-memory fake MirrorStore: initial full copy; delta only; deletion mirrored; changed==0 when identical; error propagation → implement → test → commit. (NewS3Mirror itself is exercised live in Task 8's rehearsal, not unit-tested.)

Task 5: The source-box move worker (internal/host/movespass.go)

Files: Create internal/host/movespass.go, internal/host/movespass_test.go.

Consumes: Task 1 transitions, Task 4 MirrorPrefix, readEdgesFiles-equivalent host-side parse of control/edges/ for target coordinates (host has its own edges reader in steering_loop.go — loadEdgesFileBodies + a small parse), registry.SetStatus (verify exact setter name at build time; the registry has route mutators), activator quiesce via the keep-set (Task 6).

Produces: func movesPass(ctx context.Context, s release.Store, reg *registry.Registry, cfg moveCfg) error called from the agent loop (Task 7); moveCfg{Region, OwnBucket, OwnBucketRegion, Prefix string; Mirror func(dstRegion string) lease.MirrorStore; DataDir string; Quiesced func(host string) bool}.

Semantics (each pinned by a table-driven test with DirStore + fake mirror):

• Scans records in this box's scope with Move != nil.
• preparing: resolve target coords from edges files (placement match; missing/ambiguous → log, skip — NOT a state change); one MirrorPrefix round instances/<host>/ → moves/<host>/; if changed==0 promote to ready (CAS). Abort seen (Move cleared by console) ⇒ nothing to do — rounds simply stop.
• ready: keep mirroring each pass (writes after ready still flow).
• cutting, in order: set route Status "moving" (edge stops serving; RetireExcept stops the child); if child still up (Quiesced false) → return, next pass retries; final MirrorPrefix round; delete local <DataDir>/<host>.db{,-wal,-shm} + litestream meta dirs (reuse the exact list from ExecBackend.Restore's cleanup — extract a shared helper in Task 6); CAS flip Region=To + clear Move; route retirement left to the existing reconcile pass.
• Resume idempotence: every cutting step re-runnable; a flip that lost the CAS because it already happened is terminal success.
• The pass NEVER acts on a record whose Region ≠ cfg.Region (post-flip: source stops seeing it; target never runs movespass for it).

• [ ] Failing tests (state table: prepare-not-caught-up no-op, prepare-caught-up promotes, ready keeps mirroring, cut full sequence ordering asserted via fake-mirror/op log, cut waits for quiesce, resume-from-cutting, missing target coords skip) → implement → test → commit.

Task 6: Activator: staging-fallback restore + shared purge + quiesce visibility

Files: Modify internal/activator/backend_exec.go, internal/activator/activator.go; test in internal/activator/backend_exec_test.go.

Produces:

• func PurgeLocalDB(dataDir, host, dbPath string) — the one list of local artifacts (db, -wal, -shm, both litestream meta dirs), used by Restore's pre-restore cleanup AND Task 5's cut.
• ExecBackend.Restore fallback: own instances/<host>/ has no replica AND moves/<host>/ exists in own bucket → server-side copy moves/→instances/ (via a Mirror lease.MirrorStore field on ExecBackend, nil = no fallback), then restore as normal; delete moves/<host>/ only after CheckDB passes; corrupt staged replica quarantines exactly like a corrupt own one and leaves staging in place (evidence).
• Service.Idle(host) bool (or reuse an existing accessor if one exists — check entries map) for Task 5's Quiesced.

• [ ] Failing tests (fallback copy+restore+cleanup; own-replica-beats-staging precedence; corrupt staging quarantines, staging kept; nil Mirror = old behaviour) → implement → test → commit.

Task 7: Agent wiring

Files: Modify internal/host/agent.go (arm movesPass on the same trigger as the reconcile pass, gated like every duty: only when deployment bucket + CARLOS_REGION set — a box with no placement code can still be a move SOURCE? NO: a record with region:"" lives on the home box which has no code; moves FROM the home box need cfg.Region==record.Region — "" == "" matches, so arm whenever the deployment bucket exists and pass cfg.Region verbatim, empty included); ExecBackend.Mirror = lease.NewS3Mirror(...) when hibernate bucket configured.

• [ ] Wire, go vet ./..., full TMPDIR=/tmp go test ./..., commit.

Task 8: IAM module + rollout collateral

Files: Modify infra/modules/region-host/variables.tf + main.tf (variable "sibling_staging" { type = list(object({bucket_arn=string})) , default = [] } → one policy statement: PutObject+DeleteObject on ${arn}/moves/*, ListBucket on ${arn} with s3:prefix condition moves/*); same for region-remote-host. README env/di contract notes (CARLOS_REGION on the home box optional, movesPass duty, staging prefix).

• [ ] tofu fmt + tofu validate in the module dir (no apply — applies are per-deployment, tito's are tito-side per the 2026-08-28 ruling); commit.

Task 9: Gate + PR

• [ ] Full suite + vet; self-review diff; PR referencing #331 with the amendment called out; live-rehearsal plan (flagship: create a canary instance in us, move it us→uk, prove serve + data survival, move back) listed in the PR as the deploy-time verification — NOT run until the fleet carries the binary and the IAM grant is applied.