a201dd1a1 · carlos/platform

diff --git a/cmd/console/main.go b/cmd/console/main.go

index e3dc2757e..bda5c2259 100644

--- a/cmd/console/main.go

+++ b/cmd/console/main.go

@@ -270,6 +270,7 @@ func serve() {

// from it. Empty on a CARLOS_DEPLOYMENT_DIR deployment, which

// correctly draws no bucket node.

Bucket: os.Getenv("CARLOS_DEPLOYMENT_BUCKET"),

+ HomeRegion: strings.TrimSpace(os.Getenv("CONSOLE_HOME_REGION")),

ReleasePub: pub,

Key: key,

EdgeIPs: edgeIPs(),

diff --git a/docs/site/cli.md b/docs/site/cli.md

index 6749533fe..2e8ee9ec5 100644

--- a/docs/site/cli.md

+++ b/docs/site/cli.md

@@ -321,6 +321,14 @@ has armed its Route53 steering converger. In v1 only instances on the shared

pool can be steered; a pool-scoped route is refused, and the refusal names

that as the reason.

+A **static site on a platform hostname is steered from the moment it is

+created** — you do not have to ask. Static is the one kind every edge can

+serve on its own, straight from the channel pointer, so there is no reason

+for one edge to answer for all of them. `off` turns it back off if you want

+a single answer, and nothing re-applies the default to a host that already

+exists. Custom domains are not steered for you, because their DNS is yours:

+point them at the steering hostname the app's Settings page names.

+

### carlos routes

The app's routes on this deployment: where each one sends traffic, which

diff --git a/docs/superpowers/plans/2026-08-28-instance-moves.md b/docs/superpowers/plans/2026-08-28-instance-moves.md

new file mode 100644

index 000000000..3da28981a

--- /dev/null

+++ b/docs/superpowers/plans/2026-08-28-instance-moves.md

@@ -0,0 +1,109 @@

+# Instance Moves Implementation Plan

+

+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

+

+**Goal:** Two-phase, data-following instance moves between regions (issue #331): prepare mirrors the instance's replica prefix into the target region's staging area; cut stops, final-syncs, purges, and flips the record's region.

+

+**Architecture:** The record carries `move:{to,state}`; the console writes requests (+ serial bump), the SOURCE box's move worker does everything else in delta rounds; the target box needs only a staging fallback in the activator's Restore. Never two writers: the region flip is the source's last act.

+

+**Tech Stack:** Go; S3 server-side CopyObject via aws-sdk-go-v2 (already a dependency through litestream/lss3); the repo's DirStore + fake patterns for tests.

+

+**Spec:** `docs/superpowers/specs/2026-08-28-instance-move-design.md` (as amended: prepare copies the REPLICA prefix, never tails the DB twice).

+

+## Global Constraints

+

+- Wire-shape rule: every field additive + omitempty; pre-change records/edges files parse unchanged.

+- Move verbs accept ONLY exec-backed `kind:"instance"` (empty kind = instance) routes; fleet placements (`fleet/<name>`) refused.

+- Target region must be in the armed-placement set (`armedPlacements`, #329) — no dormant path for moves.

+- Sibling-bucket IAM stays `moves/*`-scoped: Put/Delete on objects, List with prefix condition. Never `instances/*`, never Get.

+- Cut ordering is sacred: route Status "moving" → child down → final mirror round → local purge → CAS flip+clear. Idempotent on resume.

+- `TMPDIR=/tmp go test ./...` green before any commit claims done.

+

+---

+

+### Task 1: Record move shape + CAS transitions (internal/provision)

+

+**Files:** Modify `internal/provision/provision.go`; test `internal/provision/provision_test.go`.

+

+**Produces:**

+- `type MoveState struct { To string `json:"to"`; State string `json:"state"`; RequestedAt time.Time `json:"requested_at"`; ReadyAt *time.Time `json:"ready_at,omitempty"` }`

+- `Record.Move *MoveState `json:"move,omitempty"``

+- `const MovePreparing/MoveReady/MoveCutting = "preparing"/"ready"/"cutting"`

+- `func SetRecordMove(ctx, s, acct, app, host string, mut func(*Record) error) error` — one narrow read-CAS-write with one retry, `SetRecordRegion`'s exact shape, generalized: every transition (request, ready-promotion, abort-clear, cut-mark, flip+clear) is a `mut` that validates the current state and returns a typed error on an illegal transition (`ErrMoveConflict`).

+

+**Steps:**

+- [ ] Failing tests: request-on-clean sets preparing; request-on-inflight → ErrMoveConflict; ready only from preparing; cut only from ready; abort clears preparing/ready, refuses cutting; flip sets Region=To and clears Move in ONE write; lost-CAS retried once; pre-move JSON (no `move` key) round-trips byte-identical.

+- [ ] Implement; `TMPDIR=/tmp go test ./internal/provision`.

+- [ ] Commit.

+

+### Task 2: Edges files carry data-bucket coordinates (internal/host + console reader)

+

+**Files:** Modify `internal/host/edgesfile.go` (fields `DataBucket string `json:"data_bucket,omitempty"``, `DataBucketRegion string `json:"data_bucket_region,omitempty"`` on `edgesFile`, sourced in `edgesFileLoop` from `CARLOS_HIBERNATE_BUCKET`/`CARLOS_HIBERNATE_REGION`, two new `edgesFilePass` params like `placement`); `internal/console/instancesplacement.go` (same fields on `edgesFilePlacementView`). Tests: `internal/host/edgesfile_test.go` raw-wire body with/without.

+

+- [ ] Failing raw-wire test (set → present between placement and generation; unset → absent) → implement → test → commit.

+

+### Task 3: Console move verbs

+

+**Files:** Create `internal/console/apiinstancemove.go`; modify `internal/console/server.go` (routes `POST .../instances/{host}/move`, `POST .../instances/{host}/move/cut`, `DELETE .../instances/{host}/move`); test `internal/console/apiinstancemove_test.go`.

+

+**Consumes:** Task 1 helpers; `armedPlacements` (#329); `bumpSerial` (domainattach.go's helper — reuse, don't copy); `instanceGate`.

+

+**Behaviour matrix (each a test):** 202 + record shows preparing on happy path; 400 unarmed target naming armed set (armed==false is ALSO a 400 for moves — "this deployment has no armed regions"); 400 to==current region; 400 non-exec kind (static/service/addr) and fleet placement; 409 move in flight; cut: 409 unless ready; abort: 200 clears preparing/ready, 409 cutting; every successful write bumps control/serial; 404 matrix byte-identical to the instance GET's.

+

+- [ ] Failing tests → implement → test → commit.

+

+### Task 4: Prefix mirror (internal/lease/mirror.go)

+

+**Files:** Create `internal/lease/mirror.go`, `internal/lease/mirror_test.go`.

+

+**Produces:**

+- `type MirrorStore interface { List(ctx, bucket, prefix string) (map[string]string, error) /* key→etag */; Copy(ctx, srcBucket, srcKey, dstBucket, dstKey string) error; Delete(ctx, bucket, key string) error }`

+- `func NewS3Mirror(region, dstRegion string) MirrorStore` — aws-sdk-go-v2 client(s); CopyObject with URL-encoded CopySource; dst client in the sibling bucket's region.

+- `func MirrorPrefix(ctx context.Context, m MirrorStore, srcBucket, srcPrefix, dstBucket, dstPrefix string) (changed int, err error)` — list both, copy new/etag-mismatched, delete dst keys absent at src; returns changed==0 ⇒ caught up.

+

+- [ ] Failing tests against an in-memory fake MirrorStore: initial full copy; delta only; deletion mirrored; changed==0 when identical; error propagation → implement → test → commit. (NewS3Mirror itself is exercised live in Task 8's rehearsal, not unit-tested.)

+

+### Task 5: The source-box move worker (internal/host/movespass.go)

+

+**Files:** Create `internal/host/movespass.go`, `internal/host/movespass_test.go`.

+

+**Consumes:** Task 1 transitions, Task 4 `MirrorPrefix`, `readEdgesFiles`-equivalent host-side parse of `control/edges/` for target coordinates (host has its own edges reader in steering_loop.go — `loadEdgesFileBodies` + a small parse), `registry.SetStatus` (verify exact setter name at build time; the registry has route mutators), activator quiesce via the keep-set (Task 6).

+

+**Produces:** `func movesPass(ctx context.Context, s release.Store, reg *registry.Registry, cfg moveCfg) error` called from the agent loop (Task 7); `moveCfg{Region, OwnBucket, OwnBucketRegion, Prefix string; Mirror func(dstRegion string) lease.MirrorStore; DataDir string; Quiesced func(host string) bool}`.

+

+**Semantics (each pinned by a table-driven test with DirStore + fake mirror):**

+- Scans records in this box's scope with `Move != nil`.

+- preparing: resolve target coords from edges files (placement match; missing/ambiguous → log, skip — NOT a state change); one `MirrorPrefix` round `instances/<host>/ → moves/<host>/`; if changed==0 promote to ready (CAS). Abort seen (Move cleared by console) ⇒ nothing to do — rounds simply stop.

+- ready: keep mirroring each pass (writes after ready still flow).

+- cutting, in order: set route Status "moving" (edge stops serving; RetireExcept stops the child); if child still up (Quiesced false) → return, next pass retries; final MirrorPrefix round; delete local `<DataDir>/<host>.db{,-wal,-shm}` + litestream meta dirs (reuse the exact list from `ExecBackend.Restore`'s cleanup — extract a shared helper in Task 6); CAS flip Region=To + clear Move; route retirement left to the existing reconcile pass.

+- Resume idempotence: every cutting step re-runnable; a flip that lost the CAS because it already happened is terminal success.

+- The pass NEVER acts on a record whose Region ≠ cfg.Region (post-flip: source stops seeing it; target never runs movespass for it).

+

+- [ ] Failing tests (state table: prepare-not-caught-up no-op, prepare-caught-up promotes, ready keeps mirroring, cut full sequence ordering asserted via fake-mirror/op log, cut waits for quiesce, resume-from-cutting, missing target coords skip) → implement → test → commit.

+

+### Task 6: Activator: staging-fallback restore + shared purge + quiesce visibility

+

+**Files:** Modify `internal/activator/backend_exec.go`, `internal/activator/activator.go`; test in `internal/activator/backend_exec_test.go`.

+

+**Produces:**

+- `func PurgeLocalDB(dataDir, host, dbPath string)` — the one list of local artifacts (db, -wal, -shm, both litestream meta dirs), used by Restore's pre-restore cleanup AND Task 5's cut.

+- `ExecBackend.Restore` fallback: own `instances/<host>/` has no replica AND `moves/<host>/` exists in own bucket → server-side copy `moves/→instances/` (via a `Mirror lease.MirrorStore` field on ExecBackend, nil = no fallback), then restore as normal; delete `moves/<host>/` only after `CheckDB` passes; corrupt staged replica quarantines exactly like a corrupt own one and leaves staging in place (evidence).

+- `Service.Idle(host) bool` (or reuse an existing accessor if one exists — check `entries` map) for Task 5's `Quiesced`.

+

+- [ ] Failing tests (fallback copy+restore+cleanup; own-replica-beats-staging precedence; corrupt staging quarantines, staging kept; nil Mirror = old behaviour) → implement → test → commit.

+

+### Task 7: Agent wiring

+

+**Files:** Modify `internal/host/agent.go` (arm movesPass on the same trigger as the reconcile pass, gated like every duty: only when deployment bucket + CARLOS_REGION set — a box with no placement code can still be a move SOURCE? NO: a record with `region:""` lives on the home box which has no code; moves FROM the home box need cfg.Region==record.Region — "" == "" matches, so arm whenever the deployment bucket exists and pass cfg.Region verbatim, empty included); `ExecBackend.Mirror = lease.NewS3Mirror(...)` when hibernate bucket configured.

+

+- [ ] Wire, `go vet ./...`, full `TMPDIR=/tmp go test ./...`, commit.

+

+### Task 8: IAM module + rollout collateral

+

+**Files:** Modify `infra/modules/region-host/variables.tf` + `main.tf` (`variable "sibling_staging" { type = list(object({bucket_arn=string})) , default = [] }` → one policy statement: PutObject+DeleteObject on `${arn}/moves/*`, ListBucket on `${arn}` with `s3:prefix` condition `moves/*`); same for `region-remote-host`. README env/di contract notes (CARLOS_REGION on the home box optional, movesPass duty, staging prefix).

+

+- [ ] `tofu fmt` + `tofu validate` in the module dir (no apply — applies are per-deployment, tito's are tito-side per the 2026-08-28 ruling); commit.

+

+### Task 9: Gate + PR

+

+- [ ] Full suite + vet; self-review diff; PR referencing #331 with the amendment called out; live-rehearsal plan (flagship: create a canary instance in `us`, move it `us→uk`, prove serve + data survival, move back) listed in the PR as the deploy-time verification — NOT run until the fleet carries the binary and the IAM grant is applied.

diff --git a/docs/superpowers/specs/2026-08-28-instance-move-design.md b/docs/superpowers/specs/2026-08-28-instance-move-design.md

index f283f8a97..571c75eaf 100644

--- a/docs/superpowers/specs/2026-08-28-instance-move-design.md

+++ b/docs/superpowers/specs/2026-08-28-instance-move-design.md

@@ -36,7 +36,7 @@ move is in flight:

```json

"move": {

"to": "uk",

- "state": "preparing" | "ready" | "cutting",

+ "state": "preparing" | "ready" | "cutting" | "aborting",

"requested_at": "<RFC3339>",

"ready_at": "<RFC3339, when state >= ready>"

}

@@ -67,48 +67,66 @@ instance itself):

calls this only after freezing writes; the platform does not verify

the freeze (it cannot — the freeze is app-level), it documents it.

- `DELETE .../instances/{host}/move` — abort, legal in `preparing` or

- `ready` (clears `move`; the source drops its second replica; staged

- bytes in the target bucket are garbage that the next prepare

- overwrites and the target's post-restore cleanup deletes). Refused

- 409 in `cutting`: past that point the move completes forward.

+ `ready`. TWO-PHASE (review round 1): the door marks `aborting`; the

+ SOURCE box deletes the staged copy from the target bucket and then

+ clears `move` — a bare clear would leave staging behind to win a

+ later landing on the target (staging-wins below). Refused 409 in

+ `cutting` (completes forward) and in `aborting` (already under way).

- State is read, not pushed: the existing single-record GET already

returns the record verbatim, so `move.state` is pollable today with

zero new read surface.

+- Every move write bumps `control/serial` (the config fast-convergence

+ nudge the env door already uses), so the source box notices in

+ seconds, not on the hourly floor. Between states the source's own

+ move worker self-paces its delta rounds.

## What the source box does

The reconcile/activator pass on the box whose `CARLOS_REGION` matches

`region` (the source — the record still points at it):

-1. **preparing**: alongside its normal own-bucket replication, the

- activator opens a SECOND litestream replica of the instance DB to

- `s3://<target-data-bucket>/moves/<host>/` — the lease package's

- `ReplicateWithLease` pointed at the sibling bucket's staging prefix.

+1. **preparing** (AMENDED during planning, 2026-08-28: the draft said

+ "a second litestream replica of the live DB" — two litestream

+ processes cannot tail one WAL, both want to own checkpointing. The

+ fallback IS the design, and it is better): the move copies the

+ REPLICA, never the database. The activator already keeps

+ `instances/<host>/` in the source's own bucket continuously synced

+ (running child) or final-checkpointed (parked child) — the move

+ worker mirrors that prefix into

+ `s3://<target-data-bucket>/moves/<host>/` with server-side

+ CopyObject (its IAM: read own bucket, push sibling staging), in

+ delta rounds every ~30s — List both sides, copy new/changed keys,

+ delete keys the source pruned (a true mirror, so litestream

+ retention holds in staging too). Uniform for running and parked

+ instances; no wake, no litestream internals touched.

Target bucket name and AWS region are derived, not configured

per-move: each box publishes its data bucket coordinates as two new

additive fields in its edges file (`data_bucket`, `data_bucket_region`

— same additive rule as `placement`), so the source reads the

target's coordinates from the same file set everything else reads.

- A wake is forced if the instance is asleep? NO — a parked instance's

- state is already fully in the source bucket; for a parked instance

- the "second replica" is a bucket-to-staging copy performed by the

- source box (its IAM can read its own bucket and push to the sibling's

- staging), and prepare completes without ever waking the child.

-2. **ready promotion**: once the staging replica reports caught up

- (litestream generation synced; for the parked case, copy complete),

- the source CASes `move.state: "preparing" → "ready"`, stamping

- `ready_at`. The second replica keeps streaming — ready is a level,

- not an edge; writes after ready keep flowing to staging.

-3. **cutting**: on seeing `state:"cutting"`, the source (in order):

- final `SyncAndWait` on the staging replica → stop the child (normal

- stopping path, which also checkpoints to the source's OWN bucket —

- the abort-recovery copy) → close both replicators → **delete the

- local DB, WAL/SHM, and litestream meta dirs** (the warm-cache purge;

- without it a later move back would resurrect a stale local file — the

- activator's warm-cache trust is exactly right for crashes and exactly

- wrong here) → one CAS that sets `region: <to>` and clears `move` →

- remove its own route row on its next pass (the record has left its

- scope, the existing retirement path does this unprompted).

+2. **ready promotion**: once a delta round completes with zero

+ differences against a fresh listing, the source CASes

+ `move.state: "preparing" → "ready"`, stamping `ready_at`. Rounds

+ keep running — ready is a level, not an edge; writes after ready

+ keep flowing into staging on the next round.

+3. **cutting** (order amended, review round 1): on seeing

+ `state:"cutting"`, the source (in order): set the route's Status to

+ "moving" (a non-ready status: the edge

+ stops serving it, the edges file drops it, and the activator's

+ RetireExcept sweep stops the child through the ordinary stop path —

+ which final-checkpoints to the source's OWN bucket — but SWALLOWS a

+ checkpoint failure, safe only while the warm cache stood behind it)

+ → wait for the child to be down → **re-checkpoint the local DB with

+ the lease and wait** (proving the own replica carries the tail —

+ review round 1, the failed-final-checkpoint data-loss hole) → one

+ final delta round into staging (now byte-identical to the own replica) →

+ **delete the local DB, WAL/SHM, and litestream meta dirs** (the

+ warm-cache purge; without it a later move back would resurrect a

+ stale local file — the activator's warm-cache trust is exactly right

+ for crashes and exactly wrong here) → one CAS that sets

+ `region: <to>` and clears `move` → remove its own route row on its

+ next pass (the record has left its scope, the existing retirement

+ path does this unprompted).

If the box dies mid-cut, the record still says `cutting` with the old

region: the source resumes the sequence on restart (every step is

idempotent — sync again, stop a stopped child, delete absent files,

@@ -136,12 +154,11 @@ recovery door that already exists).

List, no Delete, never `instances/*`. Residency-at-rest holds

(cross-region reads remain impossible at IAM); the write blast radius

is confined to staging keys no box ever trusts as live state without

-its own restore-time `CheckDB`. Litestream against a push-only prefix:

-the lease/generation dance needs List on its own prefix — checked

-during build; if litestream cannot run push-only, prepare falls back to

-periodic own-checkpoint + copy-into-staging (same convergence, chunkier

-cadence), and the spec's contract (`ready` = staged copy caught up)

-does not change.

+its own restore-time `CheckDB`. The mirror needs three actions on the

+sibling, all `moves/*`-scoped: `s3:PutObject` and `s3:DeleteObject` on

+the object ARNs, and `s3:ListBucket` on the bucket ARN with an

+`s3:prefix` condition of `moves/*` — still no reads of sibling data,

+still nothing outside staging.

## Instance env

@@ -159,6 +176,15 @@ titogo's boot-clears-freeze rule needs exactly this.

(400 naming the kind) — the only class whose state the platform owns.

- Fleet placements (`fleet/<name>`): refused; fleets keep D6

state-does-not-follow semantics on purpose.

+- Pooled records (`pool != ""`): refused in v1 — their data lives on

+ the pool's box, not the platform region box that would drive the

+ move. Non-hibernating instances (`hibernate:false`): refused — they

+ run outside the activator, so nothing manages (or could quiesce)

+ their data.

+- The worker acts only on routes its OWN registry serves: region

+ equality alone would let a standby or second same-region box drive a

+ cut whose child runs on its sibling (per-box registries make its

+ Quiesced trivially true).

- Automatic freeze verification; region-aware unit economics; the

member-facing residence door (titogo builds on these verbs).

diff --git a/infra/modules/region-host/main.tf b/infra/modules/region-host/main.tf

index ae5afb52f..deb71e96f 100644

--- a/infra/modules/region-host/main.tf

+++ b/infra/modules/region-host/main.tf

@@ -78,7 +78,12 @@ resource "aws_iam_role_policy" "host_buckets" {

role = aws_iam_role.host.id

policy = jsonencode({

Version = "2012-10-17"

- Statement = [

+ # The two sibling-staging statement groups (instance-move spec, issue

+ # #331) are appended only when sibling_staging_arns is non-empty:

+ # push-only into moves/* — a box can stage a residence move TOWARD a

+ # sibling region but can never read that region's data or touch its

+ # live instances/* replica lineage.

+ Statement = concat([

{

Effect = "Allow"

Action = ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"]

@@ -131,7 +136,19 @@ resource "aws_iam_role_policy" "host_buckets" {

]

Resource = "*"

},

- ]

+ ],

+ [for arn in var.sibling_staging_arns : {

+ Effect = "Allow"

+ Action = ["s3:PutObject", "s3:DeleteObject"]

+ Resource = "${arn}/moves/*"

+ }],

+ [for arn in var.sibling_staging_arns : {

+ Effect = "Allow"

+ Action = ["s3:ListBucket"]

+ Resource = arn

+ Condition = { StringLike = { "s3:prefix" = "moves/*" } }

+ }]

+ )

})

}

diff --git a/infra/modules/region-host/variables.tf b/infra/modules/region-host/variables.tf

index 3f0a898c0..b508af535 100644

--- a/infra/modules/region-host/variables.tf

+++ b/infra/modules/region-host/variables.tf

@@ -114,3 +114,9 @@ variable "alerts_email" {

type = string

default = ""

}

+

+variable "sibling_staging_arns" {

+ description = "Sibling regions' data-bucket ARNs this host may STAGE residence-moves into (instance-move spec, issue #331). The grant is moves/*-scoped push-only: Put/Delete on the staging objects and a prefix-conditioned ListBucket — never instances/*, never GetObject, so residency-at-rest holds while a move's delta rounds run. Empty (the default): no cross-bucket statement at all."

+ type = list(string)

+ default = []

+}

diff --git a/infra/modules/region-remote-host/main.tf b/infra/modules/region-remote-host/main.tf

index ce7d0ecf5..d18648e76 100644

--- a/infra/modules/region-remote-host/main.tf

+++ b/infra/modules/region-remote-host/main.tf

@@ -74,7 +74,11 @@ resource "aws_iam_user_policy" "host" {

user = aws_iam_user.host.name

policy = jsonencode({

Version = "2012-10-17"

- Statement = [

+ # Sibling staging (instance-move spec, issue #331) — same push-only

+ # moves/* shape as region-host's host_buckets policy; keep the two in

+ # step (this module's README already carries that rule for the S3

+ # action list).

+ Statement = concat([

{

Sid = "OwnRegionDataObjects"

Effect = "Allow"

@@ -87,7 +91,19 @@ resource "aws_iam_user_policy" "host" {

Action = ["s3:ListBucket", "s3:GetBucketLocation"]

Resource = [aws_s3_bucket.data.arn]

},

- ]

+ ],

+ [for arn in var.sibling_staging_arns : {

+ Effect = "Allow"

+ Action = ["s3:PutObject", "s3:DeleteObject"]

+ Resource = "${arn}/moves/*"

+ }],

+ [for arn in var.sibling_staging_arns : {

+ Effect = "Allow"

+ Action = ["s3:ListBucket"]

+ Resource = arn

+ Condition = { StringLike = { "s3:prefix" = "moves/*" } }

+ }]

+ )

})

}

diff --git a/infra/modules/region-remote-host/variables.tf b/infra/modules/region-remote-host/variables.tf

index 2096ca518..9fe68ecd6 100644

--- a/infra/modules/region-remote-host/variables.tf

+++ b/infra/modules/region-remote-host/variables.tf

@@ -13,3 +13,9 @@ variable "deployment_bucket_arn" {

type = string

default = ""

}

+

+variable "sibling_staging_arns" {

+ description = "Sibling regions' data-bucket ARNs this host may STAGE residence-moves into (instance-move spec, issue #331). moves/*-scoped push-only, exactly region-host's statement shape: Put/Delete on staging objects + prefix-conditioned ListBucket; never instances/*, never GetObject. Empty (the default): no cross-bucket statement."

+ type = list(string)

+ default = []

+}

diff --git a/internal/activator/activator.go b/internal/activator/activator.go

index 2ec94c275..609ccf743 100644

--- a/internal/activator/activator.go

+++ b/internal/activator/activator.go

@@ -1080,6 +1080,18 @@ func (a *Service) RetireExcept(keep map[string]bool) {

}

+// Quiesced answers whether a host currently has NO live activator entry —

+// no child running, none waking, none mid-stop. The residence-move cut

+// (internal/host's move worker) gates its final mirror round on this: the

+// route's Status change retires the entry through RetireExcept, and only a

+// quiet host's replica prefix is byte-final.

+func (a *Service) Quiesced(host string) bool {

+ a.mu.Lock()

+ defer a.mu.Unlock()

+ _, live := a.entries[host]

+ return !live

+}

+

// Shutdown checkpoints and stops every child — the agent going down takes its

// instances with it (they're its children; leaving them orphaned would strand

// the leases until TTL expiry and leave the sockets bound against the next

diff --git a/internal/activator/backend_exec.go b/internal/activator/backend_exec.go

index a8fac9f93..3e0d41be6 100644

--- a/internal/activator/backend_exec.go

+++ b/internal/activator/backend_exec.go

@@ -60,6 +60,14 @@ type ExecBackend struct {

// but a carlos app is a separate binary from the release store — there is

// nothing sensible to invent here, and the adopt pass owns the symlinks.

Materialize func(ctx context.Context, host string) error

+ // Mirror, when non-nil, arms Restore's staging fallback (issue #331):

+ // a host with no replica under the ordinary instances/ prefix may have

+ // one under moves/<host>/ — a residence move's staged copy, pushed by

+ // the SOURCE box. Restore then copies it into place (own bucket, own

+ // IAM), restores as normal, and deletes the staging copy once CheckDB

+ // vouches for what it restored. Nil = the fallback does not exist,

+ // byte-identical to the pre-move behaviour.

+ Mirror lease.MirrorStore

// Region is the box's CARLOS placement code (CARLOS_REGION), handed to

// every exec child as its own CARLOS_REGION (issue #329, titogo's

// residence design: the app reads where it lives — e.g. to clear a move

@@ -226,11 +234,27 @@ func (b *ExecBackend) Restore(ctx context.Context, rt registry.Route) (bool, err

// its MetaDirSuffix — spelled out because this package never imports

// litestream) as any box-wide backup replicator would use, and the lease

// package's activator-scoped ".activator-<db>-litestream".

- for _, p := range []string{dbPath + "-wal", dbPath + "-shm"} {

- _ = os.Remove(p)

+ PurgeLocalDBArtifacts(dbPath, false)

+

+ // The staging fallback (issue #331): when the ordinary prefix is empty

+ // and a move staged a replica here, promote it — server-side, own

+ // bucket — and restore from the promoted copy. Precedence is the

+ // ordinary prefix ALWAYS: a host that re-lands here after moving away

+ // and back must restore its own lineage, never regress to an old

+ // staged copy (the source's cut deleted the own prefix it left behind,

+ // so a non-empty own prefix is by construction the newer story).

+ var promoted map[string]string

+ if b.Mirror != nil {

+ var err error

+ if promoted, err = b.promoteStagedReplica(ctx, host); err != nil {

+ // HARD failure, same rule as restoreReplica's own error paths

+ // (review round 1): "continuing without it" here would fall

+ // through to a FRESH boot + seed, and once that empty database

+ // replicates, the staged data is shadowed forever. A failed

+ // wake retries; a seeded ghost does not.

+ return false, fmt.Errorf("staging fallback for %s: %w", host, err)

+ }

}

- _ = os.RemoveAll(dbPath + "-litestream")

- _ = os.RemoveAll(filepath.Join(filepath.Dir(dbPath), ".activator-"+filepath.Base(dbPath)+"-litestream"))

restored, err := restoreReplica(ctx, dbPath, b.Cfg.Bucket, b.Cfg.hibernatePrefix(host), b.Cfg.Region)

if err != nil {

@@ -270,6 +294,14 @@ func (b *ExecBackend) Restore(ctx context.Context, rt registry.Route) (bool, err

return false, fmt.Errorf("restored replica for %s is corrupt: %w", host, err)

}

log.Printf("activator: restored %s from s3://%s/%s", host, b.Cfg.Bucket, b.Cfg.hibernatePrefix(host))

+ // A clean restore of a copy promoted THIS call is the moment the

+ // staging has served its purpose; a corrupt one (above)

+ // deliberately LEFT it in place as evidence, and staging that was

+ // never consulted (an abort's leftovers under a live own lineage)

+ // is not this restore's to touch.

+ if len(promoted) > 0 {

+ b.dropStagedReplica(ctx, promoted)

+ }

} else {

// Never replicated: a brand-new instance boots fresh and creates its

// own DB — the register-only create path (provisioning as a metadata

@@ -999,3 +1031,67 @@ func (p *childProc) Stop(ctx context.Context) error {

return fmt.Errorf("instance did not exit on SIGTERM; killed")

}

+

+// movesPrefix is where a residence move stages a replica in the TARGET's

+// data bucket (instance-move spec): moves/<host>/ beside the ordinary

+// instances/ prefix. A free function so the host's move worker and this

+// package spell it identically.

+func MovesPrefix(host string) string { return "moves/" + host + "/" }

+

+// PurgeLocalDBArtifacts removes the local litestream footprint next to a

+// DB — WAL, SHM and both meta dirs — and, when full is true, the DB file

+// itself. Restore's pre-restore cleanup uses the partial form (the DB's

+// absence is what got us here); a residence move's cut uses the full form:

+// the activator's warm-cache trust is exactly right for crashes and exactly

+// wrong after a cut, where a stale local file would resurrect a moved-away

+// instance's history.

+func PurgeLocalDBArtifacts(dbPath string, full bool) {

+ for _, p := range []string{dbPath + "-wal", dbPath + "-shm"} {

+ _ = os.Remove(p)

+ }

+ _ = os.RemoveAll(dbPath + "-litestream")

+ _ = os.RemoveAll(filepath.Join(filepath.Dir(dbPath), ".activator-"+filepath.Base(dbPath)+"-litestream"))

+ if full {

+ _ = os.Remove(dbPath)

+ }

+}

+

+// promoteStagedReplica copies moves/<host>/ over instances/<host>/ in this

+// box's OWN bucket when — and only when — the ordinary prefix is empty. A

+// no-op (nil error) when there is nothing staged.

+func (b *ExecBackend) promoteStagedReplica(ctx context.Context, host string) (map[string]string, error) {

+ // STAGING WINS when present (review round 1). A staged replica exists

+ // only while an inbound move is landing or has just landed: it was

+ // mirrored from the then-live data at cut, so it is by construction

+ // newer than any lineage this box kept from a PREVIOUS residence — the

+ // boomerang case (ie→uk→ie) that "own lineage never loses" restored

+ // months-stale data from. An abandoned move's staging can't linger to

+ // poison this: abort is two-phase, and the source deletes staging

+ // before the move clears (MoveAborting).

+ //

+ // The trailing slash on the guard listing matters: hibernatePrefix has

+ // none (path.Join), and a slashless List would let app.example.com's

+ // keys answer for app.example.co.

+ staged, err := b.Mirror.List(ctx, b.Cfg.Bucket, MovesPrefix(host))

+ if err != nil {

+ return nil, err

+ }

+ if len(staged) == 0 {

+ return nil, nil

+ }

+ log.Printf("activator: %s: promoting %d staged object(s) from %s (residence move landing)", host, len(staged), MovesPrefix(host))

+ if _, err := lease.MirrorPrefix(ctx, b.Mirror, b.Cfg.Bucket, MovesPrefix(host), b.Cfg.Bucket, b.Cfg.hibernatePrefix(host)); err != nil {

+ return nil, err

+ }

+ return staged, nil

+}

+

+// dropStagedReplica deletes the staged keys promote just returned — no

+// third listing of a prefix this restore has already walked twice.

+// Best-effort: a leftover staging copy is storage, not a correctness

+// problem.

+func (b *ExecBackend) dropStagedReplica(ctx context.Context, staged map[string]string) {

+ for k := range staged {

+ _ = b.Mirror.Delete(ctx, b.Cfg.Bucket, k)

+ }

+}

diff --git a/internal/activator/backend_exec_test.go b/internal/activator/backend_exec_test.go

index 7e3fb3a8a..84f291c12 100644

--- a/internal/activator/backend_exec_test.go

+++ b/internal/activator/backend_exec_test.go

@@ -2,6 +2,7 @@ package activator

import (

"context"

+ "database/sql"

"os"

"os/exec"

"path/filepath"

@@ -9,6 +10,8 @@ import (

"testing"

"time"

+ _ "modernc.org/sqlite"

+

"github.com/carlosframework/platform/internal/cgroup"

"github.com/carlosframework/platform/internal/registry"

)

@@ -103,3 +106,142 @@ func TestExecBackendStartChildRegion(t *testing.T) {

})

}

+

+// The staging fallback (issue #331): a host whose ordinary prefix is empty

+// restores from a move's staged copy — promoted server-side, dropped after

+// a clean restore — and the ordinary prefix ALWAYS wins when it exists.

+func TestRestoreStagingFallback(t *testing.T) {

+ ctx := context.Background()

+ b := testExecBackend(t)

+ f := newFakeMirrorStore()

+ b.Mirror = f

+

+ host := "moved.example.com"

+ rt := registry.Route{Host: host, Kind: "instance", Socket: filepath.Join(t.TempDir(), "i.sock")}

+

+ // restoreReplica is stubbed for the WHOLE test (the real one dials S3):

+ // it restores — materializing a genuinely valid SQLite file, so CheckDB

+ // passes — exactly when the fake bucket holds objects under the prefix

+ // it was pointed at, which is the promotion's observable effect.

+ var restoredFrom []string

+ oldRestore := restoreReplica

+ restoreReplica = func(_ context.Context, dbPath, bucket, prefix, region string) (bool, error) {

+ objs, _ := f.List(context.Background(), bucket, prefix)

+ if len(objs) == 0 {

+ return false, nil

+ }

+ restoredFrom = append(restoredFrom, bucket+"/"+prefix)

+ return true, writeMinimalSQLite(dbPath)

+ }

+ defer func() { restoreReplica = oldRestore }()

+

+ // Nothing staged, nothing own: plain fresh boot (restored=false→fresh=true).

+ fresh, err := b.Restore(ctx, rt)

+ if err != nil || !fresh {

+ t.Fatalf("empty everything: fresh=%v err=%v, want fresh boot", fresh, err)

+ }

+

+ // Staged copy present, own prefix empty: promoted then restored.

+ f.put(b.Cfg.Bucket, "moves/"+host+"/generations/a/snapshot", "e1")

+ _ = os.Remove(filepath.Join(b.DataDir, host+".db"))

+ fresh, err = b.Restore(ctx, rt)

+ if err != nil {

+ t.Fatalf("staged restore: %v", err)

+ }

+ if fresh {

+ t.Fatal("staged restore reported a fresh boot")

+ }

+ if len(restoredFrom) != 1 || restoredFrom[0] != b.Cfg.Bucket+"/"+b.Cfg.hibernatePrefix(host) {

+ t.Fatalf("restored from %v, want the promoted ordinary prefix %s", restoredFrom, b.Cfg.hibernatePrefix(host))

+ }

+ own, _ := f.List(ctx, b.Cfg.Bucket, b.Cfg.hibernatePrefix(host))

+ if len(own) != 1 {

+ t.Fatalf("promotion left own prefix = %v", own)

+ }

+ staged, _ := f.List(ctx, b.Cfg.Bucket, "moves/"+host+"/")

+ if len(staged) != 0 {

+ t.Fatalf("staging not dropped after a clean restore: %v", staged)

+ }

+

+ // STAGING WINS (review round 1): a fresh staged copy beside a stale

+ // own lineage is the boomerang landing — the staged bytes replace the

+ // own prefix and are what gets restored.

+ f.put(b.Cfg.Bucket, "moves/"+host+"/generations/b/newer-snapshot", "e9")

+ _ = os.Remove(filepath.Join(b.DataDir, host+".db"))

+ restoredFrom = nil

+ if _, err := b.Restore(ctx, rt); err != nil {

+ t.Fatal(err)

+ }

+ own, _ = f.List(ctx, b.Cfg.Bucket, b.Cfg.hibernatePrefix(host))

+ if len(own) != 1 {

+ t.Fatalf("own prefix after boomerang promote = %v, want exactly the staged copy", own)

+ }

+ for k := range own {

+ if !strings.Contains(k, "newer-snapshot") {

+ t.Fatalf("own prefix kept the stale lineage: %v", own)

+ }

+ staged, _ = f.List(ctx, b.Cfg.Bucket, "moves/"+host+"/")

+ if len(staged) != 0 {

+ t.Fatalf("staging not dropped after the boomerang restore: %v", staged)

+ }

+

+ // A staging-fallback ERROR fails the wake — never a fresh boot + seed

+ // that would shadow the staged data forever.

+ f.put(b.Cfg.Bucket, "moves/"+host+"/generations/c/snapshot", "eA")

+ f.failList = true

+ _ = os.Remove(filepath.Join(b.DataDir, host+".db"))

+ if _, err := b.Restore(ctx, rt); err == nil {

+ t.Fatal("List failure in the staging fallback did not fail the restore")

+ }

+ f.failList = false

+}

+

+// fakeMirrorStore is the test double for lease.MirrorStore, bucket→key→etag.

+type fakeMirrorStore struct {

+ buckets map[string]map[string]string

+ failList bool

+}

+

+func newFakeMirrorStore() *fakeMirrorStore {

+ return &fakeMirrorStore{buckets: map[string]map[string]string{}}

+}

+func (f *fakeMirrorStore) put(bucket, key, etag string) {

+ if f.buckets[bucket] == nil {

+ f.buckets[bucket] = map[string]string{}

+ }

+ f.buckets[bucket][key] = etag

+}

+func (f *fakeMirrorStore) List(_ context.Context, bucket, prefix string) (map[string]string, error) {

+ if f.failList {

+ return nil, context.DeadlineExceeded

+ }

+ out := map[string]string{}

+ for k, v := range f.buckets[bucket] {

+ if strings.HasPrefix(k, prefix) {

+ out[k] = v

+ }

+ return out, nil

+}

+func (f *fakeMirrorStore) Copy(_ context.Context, srcBucket, srcKey, dstBucket, dstKey string) error {

+ f.put(dstBucket, dstKey, f.buckets[srcBucket][srcKey])

+ return nil

+}

+func (f *fakeMirrorStore) Delete(_ context.Context, bucket, key string) error {

+ delete(f.buckets[bucket], key)

+ return nil

+}

+

+// writeMinimalSQLite creates a genuinely valid empty SQLite database so

+// lease.CheckDB passes on the stubbed restore's output.

+func writeMinimalSQLite(dbPath string) error {

+ db, err := sql.Open("sqlite", dbPath)

+ if err != nil {

+ return err

+ }

+ if _, err := db.Exec("create table t(x)"); err != nil {

+ return err

+ }

+ return db.Close()

+}

diff --git a/internal/console/apiinstancemove.go b/internal/console/apiinstancemove.go

new file mode 100644

index 000000000..23e674b7c

--- /dev/null

+++ b/internal/console/apiinstancemove.go

@@ -0,0 +1,217 @@

+package console

+

+import (

+ "errors"

+ "log"

+ "net/http"

+ "os"

+ "slices"

+ "strings"

+ "time"

+

+ "github.com/carlosframework/platform/internal/provision"

+ "github.com/carlosframework/platform/internal/registry"

+)

+

+// The residence-move doors (issue #331, instance-move spec): the console

+// writes REQUESTS onto the record and bumps the serial; everything that

+// touches data happens on the source box's move worker. Three verbs:

+//

+// POST .../instances/{host}/move {"region":"uk"} → preparing

+// POST .../instances/{host}/move/cut → cutting

+// DELETE .../instances/{host}/move → aborted

+//

+// State is read through the ordinary instance GET — the record carries

+// `move` verbatim while one is in flight.

+

+// moveGate is the shared front half: the instance family's gate, the

+// byte-identical 404 for a host that names no record, and the class rail —

+// only an exec-backed instance route's state is the platform's to move

+// (spec §Out of scope), and fleet placements keep their deliberate

+// state-does-not-follow semantics.

+func (s *Server) moveGate(w http.ResponseWriter, r *http.Request) (acct *Account, app, host string, rec *provision.Record, ok bool) {

+ _, acct, app, gok := s.instanceGate(w, r)

+ if !gok {

+ return nil, "", "", nil, false

+ }

+ host = r.PathValue("host")

+ if !provision.ValidHostShape(host) {

+ http.NotFound(w, r)

+ return nil, "", "", nil, false

+ }

+ ctx := r.Context()

+ rec, err := provision.GetRecord(ctx, s.cfg.Releases, acct.Sqid, app, host)

+ if errors.Is(err, os.ErrNotExist) {

+ http.NotFound(w, r)

+ return nil, "", "", nil, false

+ }

+ if err != nil {

+ log.Printf("console: api instance move %s/%s/%s: %v", acct.Sqid, app, host, err)

+ jsonError(w, http.StatusBadGateway, "could not read the instance")

+ return nil, "", "", nil, false

+ }

+ if k := rec.Kind; k != "" && k != "instance" {

+ jsonError(w, http.StatusBadRequest, "only exec-backed instances move — this route is kind "+k)

+ return nil, "", "", nil, false

+ }

+ if rec.Addr != "" || len(rec.Addrs) > 0 {

+ jsonError(w, http.StatusBadRequest, "an addr-backed route's process is not platform-managed — nothing to move")

+ return nil, "", "", nil, false

+ }

+ if _, fleet := registry.FleetPlacement(rec.Region); fleet {

+ jsonError(w, http.StatusBadRequest, "fleet-placed instances keep state-does-not-follow semantics — move is for platform regions")

+ return nil, "", "", nil, false

+ }

+ if rec.Pool != "" {

+ jsonError(w, http.StatusBadRequest, "pooled instances do not move in v1 — their data lives on the pool's own box")

+ return nil, "", "", nil, false

+ }

+ if !rec.HibernateOrDefault() {

+ jsonError(w, http.StatusBadRequest, "a non-hibernating instance runs outside the activator — nothing manages its data, nothing can move it")

+ return nil, "", "", nil, false

+ }

+ return acct, app, host, rec, true

+}

+

+// moveView is the flat answer every move door speaks — titogo's consumer

+// contract (teamtito/titogo#1848, region-doors design): state, target, one

+// freshness stamp. "none" is a real state, not an error: a host with no

+// move in flight answers 200 {"state":"none"} so a poller needs no 404

+// branch. Target carries the same NAME the caller used — the home code is

+// un-normalized on the way out (see homeRegionOut).

+type moveView struct {

+ State string `json:"state"`

+ Target string `json:"target,omitempty"`

+ UpdatedAt int64 `json:"updated_at,omitempty"`

+}

+

+func (s *Server) moveViewOf(rec *provision.Record) moveView {

+ if rec == nil || rec.Move == nil {

+ return moveView{State: "none"}

+ }

+ return moveView{

+ State: rec.Move.State,

+ Target: s.homeRegionOut(rec.Move.To),

+ UpdatedAt: rec.Move.UpdatedAt.Unix(),

+ }

+}

+

+// armedCodes renders the armed half of an armedPlacements answer as the

+// sorted comma list every refusal names — one spelling for the create door

+// and the move door alike.

+func armedCodes(set map[string]bool) string {

+ codes := make([]string, 0, len(set))

+ for c, armed := range set {

+ if armed {

+ codes = append(codes, c)

+ }

+ slices.Sort(codes)

+ return strings.Join(codes, ", ")

+}

+

+// homeRegionIn/Out translate between the home region's NAME (Config's

+// CONSOLE_HOME_REGION, what pickers and callers speak — issue #346) and the

+// record's "" (the unconstrained value the fleet's reconcile has always

+// scoped by). The record NEVER stores the code; the API never shows "".

+func (s *Server) homeRegionIn(region string) string {

+ if s.cfg.HomeRegion != "" && region == s.cfg.HomeRegion {

+ return ""

+ }

+ return region

+}

+

+func (s *Server) homeRegionOut(region string) string {

+ if region == "" {

+ return s.cfg.HomeRegion

+ }

+ return region

+}

+

+// moveTransition applies one guarded transition and maps its outcomes:

+// ErrMoveConflict → 409 with the transition's own sentence, success → the

+// flat move view + a serial bump so the source box notices in seconds.

+func (s *Server) moveTransition(w http.ResponseWriter, r *http.Request, acct *Account, app, host string, mut func(*provision.Record) error, okStatus int) {

+ ctx := r.Context()

+ rec, err := provision.SetRecordMove(ctx, s.cfg.Releases, acct.Sqid, app, host, mut)

+ if errors.Is(err, provision.ErrMoveConflict) {

+ jsonError(w, http.StatusConflict, err.Error())

+ return

+ }

+ if err != nil {

+ log.Printf("console: api instance move %s/%s/%s: %v", acct.Sqid, app, host, err)

+ jsonError(w, http.StatusBadGateway, "could not write the move — try again")

+ return

+ }

+ s.bumpSerial(ctx)

+ writeJSON(w, okStatus, s.moveViewOf(rec))

+}

+

+// handleAPIInstanceMoveStatus answers the poll: the flat view, "none" when

+// quiet — the tenant's freeze loop lives on this door.

+func (s *Server) handleAPIInstanceMoveStatus(w http.ResponseWriter, r *http.Request) {

+ _, _, _, rec, ok := s.moveGate(w, r)

+ if !ok {

+ return

+ }

+ writeJSON(w, http.StatusOK, s.moveViewOf(rec))

+}

+

+func (s *Server) handleAPIInstanceMoveRequest(w http.ResponseWriter, r *http.Request) {

+ acct, app, host, _, ok := s.moveGate(w, r)

+ if !ok {

+ return

+ }

+ var req struct {

+ Region string `json:"region"`

+ }

+ if !readJSON(w, r, &req) {

+ return

+ }

+ region := strings.TrimSpace(req.Region)

+ to := s.homeRegionIn(region)

+ if region == "" || !provision.ValidRegion(to) {

+ jsonError(w, http.StatusBadRequest, "invalid region")

+ return

+ }

+ // A move to the HOME region ("" after normalization) has no armed-edges

+ // row to validate against and no staging lane wired for the home's

+ // bucket yet — refused plainly rather than wedging in preparing.

+ if to == "" {

+ jsonError(w, http.StatusBadRequest, "moves to the home region are not supported yet — the home publishes no staging lane")

+ return

+ }

+ // The armed gate is MANDATORY for moves — unlike create there is no

+ // dormant path, because a move needs a live target box to restore on;

+ // a deployment with no armed placements cannot be a move target at all.

+ set, armed := armedPlacements(r.Context(), s.cfg.Releases)

+ if !armed {

+ jsonError(w, http.StatusBadRequest, "this deployment publishes no armed regions — nothing to move to")

+ return

+ }

+ if !set[to] {

+ jsonError(w, http.StatusBadRequest, "region "+to+" is not armed in this deployment — armed: "+armedCodes(set))

+ return

+ }

+ s.moveTransition(w, r, acct, app, host, provision.MoveRequest(to, time.Now()), http.StatusAccepted)

+}

+

+func (s *Server) handleAPIInstanceMoveCut(w http.ResponseWriter, r *http.Request) {

+ acct, app, host, _, ok := s.moveGate(w, r)

+ if !ok {

+ return

+ }

+ s.moveTransition(w, r, acct, app, host, provision.MoveMarkCutting(), http.StatusAccepted)

+}

+

+// Abort is two-phase (review round 1): this door marks MoveAborting and the

+// SOURCE box deletes the staged copy before clearing the field — a bare

+// clear would leave staging behind to win a later landing on the target

+// (staging-wins). 202, like its siblings: the caller polls the record.

+func (s *Server) handleAPIInstanceMoveAbort(w http.ResponseWriter, r *http.Request) {

+ acct, app, host, _, ok := s.moveGate(w, r)

+ if !ok {

+ return

+ }

+ s.moveTransition(w, r, acct, app, host, provision.MoveAbort(), http.StatusAccepted)

+}

diff --git a/internal/console/apiinstancemove_test.go b/internal/console/apiinstancemove_test.go

new file mode 100644

index 000000000..6b4a8a136

--- /dev/null

+++ b/internal/console/apiinstancemove_test.go

@@ -0,0 +1,302 @@

+package console

+

+import (

+ "context"

+ "encoding/json"

+ "fmt"

+ "net/http"

+ "strings"

+ "testing"

+

+ "time"

+

+ "github.com/carlosframework/platform/internal/provision"

+)

+

+// The residence-move doors (issue #331): the console writes guarded

+// transitions and serial bumps; the state machine's 400/409 matrix is the

+// tenant's whole interface, so every cell is pinned.

+func TestAPIInstanceMoveVerbs(t *testing.T) {

+ fx, token := instanceFixture(t, "code-inst-move")

+ m := titoManifest()

+ m.Regions = nil

+ m.Pool = "" // pooled records are refused at the move door — separate rail, pinned below

+ enableInstances(t, fx, "titogo", m)

+ u := instancesURL(fx, "bab", "titogo")

+ put := func(key, body string) {

+ t.Helper()

+ if _, err := fx.rel.Put(context.Background(), key, []byte(body), ""); err != nil {

+ t.Fatal(err)

+ }

+ // Create BEFORE arming: once uk-box publishes a placement, the #329

+ // gate would refuse region ie (not armed) — the record predates arming,

+ // exactly like every real record on a fleet that grows regions later.

+ if code, _, out := rawReq(t, http.MethodPost, u, token, `{"host":"acme.go.tito.io","region":"ie"}`); code != http.StatusCreated {

+ t.Fatalf("create = %d %s", code, out)

+ }

+ put("control/edges/uk-box.json", `{"v":1,"box":"uk-box","eip":"1.2.3.4","placement":"uk","data_bucket":"data-uk","data_bucket_region":"eu-west-2","hosts":{}}`)

+ moveURL := u + "/acme.go.tito.io/move"

+

+ // Unarmed target → 400 naming the set.

+ if code, _, out := rawReq(t, http.MethodPost, moveURL, token, `{"region":"se"}`); code != http.StatusBadRequest || !strings.Contains(out, "armed: uk") {

+ t.Fatalf("move to unarmed = %d %s, want 400 naming the set", code, out)

+ }

+ // Same region → 409 (the transition's own conflict).

+ if code, _, out := rawReq(t, http.MethodPost, u+"/acme.go.tito.io/move", token, `{"region":"ie"}`); code != http.StatusBadRequest && code != http.StatusConflict {

+ t.Fatalf("move to current region = %d %s, want a refusal", code, out)

+ } else if code == http.StatusBadRequest && !strings.Contains(out, "armed") {

+ // "ie" is not armed either — refused at the armed gate, also fine.

+ _ = out

+ }

+ // Cut before any move → 409.

+ if code, _, out := rawReq(t, http.MethodPost, moveURL+"/cut", token, ""); code != http.StatusConflict {

+ t.Fatalf("cut with no move = %d %s, want 409", code, out)

+ }

+ // The poll door answers "none" before any move — 200, never a 404.

+ code, _, out := rawReq(t, http.MethodGet, moveURL, token, "")

+ if code != http.StatusOK || !strings.Contains(out, `"state":"none"`) {

+ t.Fatalf("move status with no move = %d %s, want 200 none", code, out)

+ }

+ // Happy path: request → the FLAT view (titogo's consumer contract):

+ // state, target, updated_at — never the record.

+ code, _, out = rawReq(t, http.MethodPost, moveURL, token, `{"region":"uk"}`)

+ if code != http.StatusAccepted {

+ t.Fatalf("move request = %d %s, want 202", code, out)

+ }

+ var mv struct {

+ State string `json:"state"`

+ Target string `json:"target"`

+ UpdatedAt int64 `json:"updated_at"`

+ }

+ if err := json.Unmarshal([]byte(out), &mv); err != nil {

+ t.Fatal(err)

+ }

+ if mv.State != provision.MovePreparing || mv.Target != "uk" || mv.UpdatedAt == 0 {

+ t.Fatalf("after request: %+v", mv)

+ }

+ // The record itself still carries the nested truth (and region ie).

+ if code, _, out := rawReq(t, http.MethodGet, u+"/acme.go.tito.io", token, ""); code != http.StatusOK || !strings.Contains(out, `"region":"ie"`) || !strings.Contains(out, provision.MovePreparing) {

+ t.Fatalf("record after request: %d %s", code, out)

+ }

+ // The poll door mirrors it.

+ if code, _, out := rawReq(t, http.MethodGet, moveURL, token, ""); code != http.StatusOK || !strings.Contains(out, `"target":"uk"`) {

+ t.Fatalf("move status mid-move = %d %s", code, out)

+ }

+ // Second request while in flight → 409.

+ if code, _, out := rawReq(t, http.MethodPost, moveURL, token, `{"region":"uk"}`); code != http.StatusConflict {

+ t.Fatalf("second move = %d %s, want 409", code, out)

+ }

+ // Cut from preparing → 409 (only ready cuts).

+ if code, _, out := rawReq(t, http.MethodPost, moveURL+"/cut", token, ""); code != http.StatusConflict {

+ t.Fatalf("cut from preparing = %d %s, want 409", code, out)

+ }

+ // Abort from preparing → 202 aborting (two-phase: the SOURCE box

+ // deletes staging, then clears the field).

+ if code, _, out := rawReq(t, http.MethodDelete, moveURL, token, ""); code != http.StatusAccepted || !strings.Contains(out, provision.MoveAborting) {

+ t.Fatalf("abort = %d %s, want 202 aborting", code, out)

+ }

+ // A second abort while aborting → 409; a new request while aborting →

+ // 409 (still in flight).

+ if code, _, _ := rawReq(t, http.MethodDelete, moveURL, token, ""); code != http.StatusConflict {

+ t.Fatalf("double abort = %d, want 409", code)

+ }

+ if code, _, _ := rawReq(t, http.MethodPost, moveURL, token, `{"region":"uk"}`); code != http.StatusConflict {

+ t.Fatalf("request while aborting = %d, want 409", code)

+ }

+ // The box's half clears it; then a fresh request proceeds.

+ if _, err := provision.SetRecordMove(context.Background(), fx.rel, "bab", "titogo", "acme.go.tito.io", provision.MoveAbortComplete()); err != nil {

+ t.Fatal(err)

+ }

+ code, _, out = rawReq(t, http.MethodGet, u+"/acme.go.tito.io", token, "")

+ if code != http.StatusOK || strings.Contains(out, `"move"`) {

+ t.Fatalf("after abort-complete: %d %s, want no move key", code, out)

+ }

+ if code, _, _ := rawReq(t, http.MethodPost, moveURL, token, `{"region":"uk"}`); code != http.StatusAccepted {

+ t.Fatal("re-request")

+ }

+ if _, err := provision.SetRecordMove(context.Background(), fx.rel, "bab", "titogo", "acme.go.tito.io", provision.MovePromoteReady(time.Now())); err != nil {

+ t.Fatal(err)

+ }

+ if code, _, out := rawReq(t, http.MethodPost, moveURL+"/cut", token, ""); code != http.StatusAccepted {

+ t.Fatalf("cut from ready = %d %s, want 202", code, out)

+ }

+ if code, _, out := rawReq(t, http.MethodDelete, moveURL, token, ""); code != http.StatusConflict {

+ t.Fatalf("abort from cutting = %d %s, want 409 (completes forward)", code, out)

+ }

+ // The serial was bumped by the successful writes.

+ if _, _, err := fx.rel.Get(context.Background(), "control/serial"); err != nil {

+ t.Fatalf("control/serial not written: %v", err)

+ }

+ // 404 discipline: unknown host is the byte-identical not-found.

+ if code, _, _ := rawReq(t, http.MethodPost, u+"/ghost.go.tito.io/move", token, `{"region":"uk"}`); code != http.StatusNotFound {

+ t.Fatalf("move on unknown host = %d, want 404", code)

+ }

+}

+

+// The class rails: only an exec-backed instance route moves.

+func TestAPIInstanceMoveClassRails(t *testing.T) {

+ fx, token := instanceFixture(t, "code-inst-move-rails")

+ m := titoManifest()

+ m.Regions = nil

+ m.Pool = ""

+ enableInstances(t, fx, "titogo", m)

+ u := instancesURL(fx, "bab", "titogo")

+ if _, err := fx.rel.Put(context.Background(), "control/edges/uk-box.json",

+ []byte(`{"v":1,"box":"uk-box","eip":"1.2.3.4","placement":"uk","hosts":{}}`), ""); err != nil {

+ t.Fatal(err)

+ }

+ mk := func(body string) string {

+ t.Helper()

+ code, _, out := rawReq(t, http.MethodPost, u, token, body)

+ if code != http.StatusCreated {

+ t.Fatalf("create %s = %d %s", body, code, out)

+ }

+ var rec provision.Record

+ if err := json.Unmarshal([]byte(out), &rec); err != nil {

+ t.Fatal(err)

+ }

+ return rec.Host

+ }

+ for _, tc := range []struct {

+ name, body, wantIn string

+ }{

+ {"static", `{"host":"st.go.tito.io","kind":"static"}`, "kind static"},

+ {"addr-backed", `{"host":"ad.go.tito.io","addrs":["10.0.0.1:80"],"kind":"service"}`, "kind service"},

+ } {

+ t.Run(tc.name, func(t *testing.T) {

+ host := mk(tc.body)

+ code, _, out := rawReq(t, http.MethodPost, u+"/"+host+"/move", token, `{"region":"uk"}`)

+ if code != http.StatusBadRequest || !strings.Contains(out, tc.wantIn) {

+ t.Fatalf("move %s = %d %s, want 400 mentioning %q", tc.name, code, out, tc.wantIn)

+ }

+ })

+ }

+ // Fleet-placed: region carries fleet/<name>.

+ host := mk(`{"host":"fl.go.tito.io"}`)

+ if err := provision.SetRecordRegion(context.Background(), fx.rel, "bab", "titogo", host, "fleet/pi"); err != nil {

+ t.Fatal(err)

+ }

+ code, _, out := rawReq(t, http.MethodPost, u+"/"+host+"/move", token, `{"region":"uk"}`)

+ if code != http.StatusBadRequest || !strings.Contains(out, "fleet") {

+ t.Fatalf("move fleet-placed = %d %s, want 400 mentioning fleet", code, out)

+ }

+ _ = fmt.Sprint()

+ _ = provisionPooledMoveRefusal(t, fx, token, u)

+}

+

+// provisionPooledMoveRefusal pins the pool rail: a pooled record's data

+// lives on the pool's box, and v1 refuses the class at the door.

+func provisionPooledMoveRefusal(t *testing.T, fx *consoleFixture, token, u string) bool {

+ t.Helper()

+ rec := provision.Record{Schema: 1, Host: "pooled.go.tito.io", Slug: "pooled", App: "titogo",

+ Account: "bab", Pool: "dedicated-x", Channel: "stable", CreatedAt: time.Now().UTC()}

+ if err := provision.CreateRecord(context.Background(), fx.rel, rec); err != nil {

+ t.Fatal(err)

+ }

+ code, _, out := rawReq(t, http.MethodPost, u+"/pooled.go.tito.io/move", token, `{"region":"uk"}`)

+ if code != http.StatusBadRequest || !strings.Contains(out, "pool") {

+ t.Fatalf("move pooled = %d %s, want 400 mentioning the pool rail", code, out)

+ }

+ return true

+}

+

+// The armed-regions endpoint (issue #346): codes + armed flags from the

+// edges files (the create door's own predicate), the home region under its

+// NAME, and the deployment default — the picker contract.

+func TestAPIRegions(t *testing.T) {

+ fx, token := instanceFixture(t, "code-inst-regions")

+ m := titoManifest()

+ m.Regions = nil

+ m.Pool = ""

+ enableInstances(t, fx, "titogo", m)

+ base := fx.ts.URL + "/api/cli/apps/bab/titogo/regions"

+ put := func(key, body string) {

+ t.Helper()

+ if _, err := fx.rel.Put(context.Background(), key, []byte(body), ""); err != nil {

+ t.Fatal(err)

+ }

+

+ // No named home, no placement-publishing boxes: empty list, empty

+ // default — consumers hide their pickers.

+ code, _, out := rawReq(t, http.MethodGet, base, token, "")

+ if code != http.StatusOK {

+ t.Fatalf("regions = %d %s", code, out)

+ }

+ var rl struct {

+ Regions []struct {

+ Code string `json:"code"`

+ Armed bool `json:"armed"`

+ } `json:"regions"`

+ Default string `json:"default"`

+ }

+ if err := json.Unmarshal([]byte(out), &rl); err != nil {

+ t.Fatal(err)

+ }

+ if len(rl.Regions) != 0 || rl.Default != "" {

+ t.Fatalf("empty deployment answered %+v", rl)

+ }

+

+ // A named home + one armed box + one EIP-less placement.

+ fx.srv.cfg.HomeRegion = "ie"

+ put("control/edges/uk-box.json", `{"v":1,"box":"uk-box","eip":"1.2.3.4","placement":"uk","hosts":{}}`)

+ put("control/edges/se-box.json", `{"v":1,"box":"se-box","eip":"","placement":"se","hosts":{}}`)

+ code, _, out = rawReq(t, http.MethodGet, base, token, "")

+ if code != http.StatusOK {

+ t.Fatalf("regions = %d %s", code, out)

+ }

+ if err := json.Unmarshal([]byte(out), &rl); err != nil {

+ t.Fatal(err)

+ }

+ want := "[{ie true} {se false} {uk true}]"

+ got := fmt.Sprint(func() (r []struct {

+ Code string

+ Armed bool

+ }) {

+ for _, x := range rl.Regions {

+ r = append(r, struct {

+ Code string

+ Armed bool

+ }{x.Code, x.Armed})

+ }

+ return

+ }())

+ if got != want || rl.Default != "ie" {

+ t.Fatalf("regions = %s default=%q, want %s / ie", got, rl.Default, want)

+ }

+}

+

+// The home region's NAME normalizes to the record's "" at the create door

+// (issue #346): a picker offering "ie" creates exactly what an unconstrained

+// create always made, and the fleet's scope rules never see the code.

+func TestAPIInstanceCreateNormalizesHomeRegion(t *testing.T) {

+ fx, token := instanceFixture(t, "code-inst-homenorm")

+ m := titoManifest()

+ m.Regions = nil

+ m.Pool = ""

+ enableInstances(t, fx, "titogo", m)

+ fx.srv.cfg.HomeRegion = "ie"

+ if _, err := fx.rel.Put(context.Background(), "control/edges/uk-box.json",

+ []byte(`{"v":1,"box":"uk-box","eip":"1.2.3.4","placement":"uk","hosts":{}}`), ""); err != nil {

+ t.Fatal(err)

+ }

+ u := instancesURL(fx, "bab", "titogo")

+ // "ie" (the home's name, NOT in the armed edges set) is accepted and

+ // stored as "".

+ code, _, out := rawReq(t, http.MethodPost, u, token, `{"host":"home.go.tito.io","region":"ie"}`)

+ if code != http.StatusCreated {

+ t.Fatalf("create region ie (home name) = %d %s, want 201", code, out)

+ }

+ rec, err := provision.GetRecord(context.Background(), fx.rel, "bab", "titogo", "home.go.tito.io")

+ if err != nil {

+ t.Fatal(err)

+ }

+ if rec.Region != "" {

+ t.Fatalf("record region = %q, want the normalized unconstrained value", rec.Region)

+ }

+ // A move targeting the home name is refused plainly (no staging lane).

+ if code, _, out := rawReq(t, http.MethodPost, u+"/home.go.tito.io/move", token, `{"region":"ie"}`); code != http.StatusBadRequest || !strings.Contains(out, "home region") {

+ t.Fatalf("move to home = %d %s, want the plain refusal", code, out)

+ }

+}

diff --git a/internal/console/apiinstances.go b/internal/console/apiinstances.go

index 954ca0e79..1eeadc26e 100644

--- a/internal/console/apiinstances.go

+++ b/internal/console/apiinstances.go

@@ -558,7 +558,13 @@ func (s *Server) handleAPIInstanceCreate(w http.ResponseWriter, r *http.Request)

// membership: a label naming another claimed app in the account belongs

// to that app. Custom-domain manifests never enter apex space (ValidHost

// pinned the suffix above), so this check cannot touch them.

- if s.cfg.AppsDomain != "" && m.Domain == acct.Sqid+"."+s.cfg.AppsDomain {

+ //

+ // The same question answers a second one further down (the steering

+ // default), so it is asked once, here, and named: is this manifest

+ // pinned to the account's platform subdomain, or to a domain of the

+ // app's own?

+ platformNamespace := s.cfg.AppsDomain != "" && m.Domain == acct.Sqid+"."+s.cfg.AppsDomain

+ if platformNamespace {

if label := strings.TrimSuffix(host, "."+m.Domain); label != app {

claimed, err := s.store.HasAppIn(acct.ID, label)

if err != nil {

@@ -593,7 +599,10 @@ func (s *Server) handleAPIInstanceCreate(w http.ResponseWriter, r *http.Request)

jsonError(w, http.StatusBadGateway, "could not check this app's placement")

return

}

- region := strings.TrimSpace(req.Region)

+ // The home region's NAME normalizes to the record's "" before any

+ // validation (issue #346): pickers speak the code, records keep the

+ // unconstrained value reconcile has always scoped by.

+ region := s.homeRegionIn(strings.TrimSpace(req.Region))

if placed != "" {

region = placed

} else if !provision.ValidRegion(region) || !instanceRegionAllowed(m.Regions, region) {

@@ -613,14 +622,9 @@ func (s *Server) handleAPIInstanceCreate(w http.ResponseWriter, r *http.Request)

// dormant until some box publishes a placement code, so a console rolled

// ahead of its fleet changes nothing.

if placed == "" && region != "" {

- if set, armed := armedPlacements(ctx, s.cfg.Releases); armed && !set[region] {

- codes := make([]string, 0, len(set))

- for c := range set {

- codes = append(codes, c)

- }

- slices.Sort(codes)

+ if set, published := armedPlacements(ctx, s.cfg.Releases); published && !set[region] {

jsonError(w, http.StatusBadRequest,

- "region "+region+" is not armed in this deployment — armed: "+strings.Join(codes, ", "))

+ "region "+region+" is not armed in this deployment — armed: "+armedCodes(set))

return

}

@@ -654,6 +658,43 @@ func (s *Server) handleAPIInstanceCreate(w http.ResponseWriter, r *http.Request)

Addrs: req.Addrs,

Health: req.Health,

}

+ // A static platform host is HA by default. Static is the one serving

+ // class every platform box materializes unconditionally

+ // (provision.everywhereClass): no unit, no process, content read from

+ // the bucket channel pointer, so a far edge serves it with no home box

+ // and no backhaul. The content half of that is already true — every

+ // armed edge holds the site and its account wildcard cert. What was

+ // missing is DNS: with no opt-in the host resolves to one edge, so

+ // several healthy boxes hold the site and every visitor is sent to one

+ // of them.

+ //

+ // Scoped to the platform namespace deliberately. internal/steering's

+ // steeredHosts only ever builds latency records for hosts under the

+ // apps domain, so defaulting a custom domain's record to "latency"

+ // would write an opt-in no pass will ever honour (#306's shape); a

+ // custom domain reaches the edges by its owner's own DNS, and the

+ // console's attach guidance already tells them to CNAME the steering

+ // hostname.

+ //

+ // Scoped to static deliberately too: a home-boxed route reaches a far

+ // edge by backhaul, a different trade with a different cost, not one to

+ // make silently for every app on the deployment.

+ //

+ // And it honours v1's pool rail, the same one the PATCH door enforces

+ // with a 409 (steering needs the shared pool, because a pool's own

+ // edges are not the deployment's). A default that quietly wrote what

+ // the update door refuses would be the worse kind of inconsistency: the

+ // record would carry an opt-in nobody could have asked for through the

+ // front door, and the converger would skip it anyway.

+ //

+ // This is a value at MINT, never a converge: `carlos steering --app x

+ // --host y off` is the escape hatch, and nothing re-stamps a record

+ // that already exists. The opt-in is inert on a deployment that has not

+ // set CARLOS_STEERING_ZONE_ID, so a single-box deployment states the

+ // intent harmlessly and becomes HA the day it arms a second edge.

+ if rec.Kind == "static" && platformNamespace && rec.Pool == "" {

+ rec.Steering = "latency"

+ }

// The shape rails, at the door — kind vocabulary, one upstream shape per

// record, addr syntax, and the combinations that would wedge a host (a

// static route with an upstream, a hibernating addr-backed one). The

diff --git a/internal/console/apiinstances_test.go b/internal/console/apiinstances_test.go

index b9a5bd7ff..3edc08607 100644

--- a/internal/console/apiinstances_test.go

+++ b/internal/console/apiinstances_test.go

@@ -2184,3 +2184,128 @@ func TestAPIInstanceCreateArmedPlacementGate(t *testing.T) {

t.Fatalf("create region se (no EIP) = %d %s, want 400", code, out)

}

+

+// A static site is HA by default. Static is the one serving class every

+// platform box materializes unconditionally (provision.everywhereClass):

+// no unit, no process, content read from the bucket channel pointer, so a

+// far edge serves it with no home box and no backhaul. The content half of

+// that was already true and measured — every flagship edge served every

+// static host with a valid cert. What was missing was DNS: without an

+// opt-in the host resolves to one edge, so four healthy boxes held the site

+// and every visitor went to one of them.

+//

+// The opt-in is inert until a deployment sets CARLOS_STEERING_ZONE_ID, so a

+// single-box deployment states the intent harmlessly and becomes HA the day

+// it arms a second edge. `carlos steering --app x --host y off` is the

+// escape hatch, and nothing here ever converges an existing record.

+func TestAPIInstanceCreateStaticSteersByDefault(t *testing.T) {

+ fx, member := instanceFixture(t, "code-inst-static-steer")

+ fx.srv.cfg.AppsDomain = "oncarlos.test"

+ if status, _, out := rawReq(t, http.MethodPut, enableURL(fx, "bab", "titogo"), member, `{}`); status != http.StatusOK {

+ t.Fatalf("enable = %d %s", status, out)

+ }

+ host := "titogo." + fx.sqid + ".oncarlos.test"

+

+ status, _, out := rawReq(t, http.MethodPost, instancesURL(fx, "bab", "titogo"), member,

+ `{"host":"`+host+`","kind":"static","channel":"edge"}`)

+ if status != http.StatusCreated {

+ t.Fatalf("static apex create = %d %s, want 201", status, out)

+ }

+ stored, err := provision.GetRecord(context.Background(), fx.rel, fx.sqid, "titogo", host)

+ if err != nil {

+ t.Fatal(err)

+ }

+ if stored.Steering != "latency" {

+ t.Fatalf("stored Steering = %q, want latency — a static platform host is HA by default", stored.Steering)

+ }

+ // Whatever the door writes has to survive ValidateShape, or the record

+ // it just stored is one no box will ever materialize.

+ if err := provision.ValidateShape(*stored); err != nil {

+ t.Fatalf("the defaulted record does not validate: %v", err)

+ }

+}

+

+// The default is scoped to the platform namespace, and deliberately so. The

+// steering converger only ever writes latency records for hosts under the

+// apps domain (internal/steering's steeredHosts), so defaulting a custom

+// domain's record to "latency" would write an opt-in that does nothing —

+// a claim in the record that no pass will honour, which is #306's shape.

+// A custom domain reaches the edges by its owner's own DNS, and the console

+// already tells them to CNAME the steering hostname.

+func TestAPIInstanceCreateStaticCustomDomainDoesNotSteer(t *testing.T) {

+ fx, member := instanceFixture(t, "code-inst-static-custom")

+ fx.srv.cfg.AppsDomain = "oncarlos.test"

+ // The operator-written manifest, pinned to the app's OWN domain rather

+ // than the account's platform subdomain — the custom-domain shape.

+ enableInstances(t, fx, "titogo", titoManifest())

+ host := "site.go.tito.io"

+

+ status, _, out := rawReq(t, http.MethodPost, instancesURL(fx, "bab", "titogo"), member,

+ `{"host":"`+host+`","region":"ie","kind":"static","channel":"edge"}`)

+ if status != http.StatusCreated {

+ t.Fatalf("custom-domain static create = %d %s, want 201", status, out)

+ }

+ stored, err := provision.GetRecord(context.Background(), fx.rel, fx.sqid, "titogo", host)

+ if err != nil {

+ t.Fatal(err)

+ }

+ if stored.Steering != "" {

+ t.Fatalf("stored Steering = %q, want empty — the converger never steers a host outside the apps domain", stored.Steering)

+ }

+}

+

+// Only static. A home-boxed instance route reaches a far edge by backhaul,

+// which is a different trade with a different cost, and widening this

+// default to every kind would make that trade silently for every app on the

+// deployment. `carlos steering` remains the door for those, per host.

+func TestAPIInstanceCreateNonStaticDoesNotSteer(t *testing.T) {

+ fx, member := instanceFixture(t, "code-inst-nonstatic-steer")

+ fx.srv.cfg.AppsDomain = "oncarlos.test"

+ if status, _, out := rawReq(t, http.MethodPut, enableURL(fx, "bab", "titogo"), member, `{}`); status != http.StatusOK {

+ t.Fatalf("enable = %d %s", status, out)

+ }

+ host := "titogo." + fx.sqid + ".oncarlos.test"

+

+ status, _, out := rawReq(t, http.MethodPost, instancesURL(fx, "bab", "titogo"), member,

+ `{"host":"`+host+`","channel":"edge"}`) // no kind: the instance default

+ if status != http.StatusCreated {

+ t.Fatalf("instance create = %d %s, want 201", status, out)

+ }

+ stored, err := provision.GetRecord(context.Background(), fx.rel, fx.sqid, "titogo", host)

+ if err != nil {

+ t.Fatal(err)

+ }

+ if stored.Steering != "" {

+ t.Fatalf("stored Steering = %q, want empty — only static is steered by default", stored.Steering)

+ }

+}

+

+// v1's pool rail applies to the default exactly as it applies to the PATCH

+// door, which answers 409 for the same request. A default that wrote what

+// the update door refuses would put an opt-in in the record that nobody

+// could have asked for through the front door — and that the converger

+// skips regardless, since a pool's own edges are not the deployment's.

+func TestAPIInstanceCreateStaticInAPoolDoesNotSteer(t *testing.T) {

+ fx, member := instanceFixture(t, "code-inst-static-pooled")

+ fx.srv.cfg.AppsDomain = "oncarlos.test"

+ // An operator-written manifest can pin BOTH the account's platform

+ // subdomain and a pool; the member-facing enable door cannot.

+ enableInstances(t, fx, "titogo", provision.Manifest{

+ Schema: 1, App: "titogo", Domain: fx.sqid + ".oncarlos.test",

+ Pool: "tito", Channel: "stable", Seed: []string{"seed"},

+ })

+ host := "titogo." + fx.sqid + ".oncarlos.test"

+

+ status, _, out := rawReq(t, http.MethodPost, instancesURL(fx, "bab", "titogo"), member,

+ `{"host":"`+host+`","kind":"static","channel":"edge"}`)

+ if status != http.StatusCreated {

+ t.Fatalf("pooled static create = %d %s, want 201", status, out)

+ }

+ stored, err := provision.GetRecord(context.Background(), fx.rel, fx.sqid, "titogo", host)

+ if err != nil {

+ t.Fatal(err)

+ }

+ if stored.Steering != "" {

+ t.Fatalf("stored Steering = %q, want empty — steering requires the shared pool", stored.Steering)

+ }

+}

diff --git a/internal/console/apiregions.go b/internal/console/apiregions.go

new file mode 100644

index 000000000..59365e080

--- /dev/null

+++ b/internal/console/apiregions.go

@@ -0,0 +1,53 @@

+package console

+

+import (

+ "net/http"

+ "sort"

+)

+

+// The armed-regions endpoint (issue #346): one authoritative answer to

+// "which regions can serve right now", for consumer pickers — titogo's

+// signup picker and move doors consume it through a 60s cache and hide

+// themselves when it is unreachable, so it is never in a request path.

+//

+// GET /api/cli/apps/{account}/{app}/regions

+// {"regions":[{"code":"ie","armed":true},{"code":"uk","armed":true}],

+// "default":"ie"}

+//

+// Codes are CARLOS placement codes, never AWS region strings. armed is

+// EXACTLY the create door's predicate (#329's armedPlacements — a picker

+// built on this can never offer what create would refuse): a box publishes

+// the code, and has a dial address. The home region appears under its NAME

+// (Config.HomeRegion) with armed true — create accepts it by normalizing

+// the name to the record's "" — and is also the default. A deployment with

+// no named home and no placement-publishing boxes answers empty lists, and

+// consumers hide their pickers.

+//

+// The instance family's gate, deliberately: the answer is deployment

+// topology (which regions exist), the same information the create door's

+// refusal message already hands the same bearer.

+func (s *Server) handleAPIRegions(w http.ResponseWriter, r *http.Request) {

+ _, _, _, ok := s.instanceGate(w, r)

+ if !ok {

+ return

+ }

+ type region struct {

+ Code string `json:"code"`

+ Armed bool `json:"armed"`

+ }

+ set, _ := armedPlacements(r.Context(), s.cfg.Releases)

+ out := make([]region, 0, len(set)+1)

+ if s.cfg.HomeRegion != "" {

+ out = append(out, region{Code: s.cfg.HomeRegion, Armed: true})

+ // A box that publishes the home's code explicitly (a future coded

+ // home) must not appear twice.

+ delete(set, s.cfg.HomeRegion)

+ }

+ for code, armed := range set {

+ out = append(out, region{Code: code, Armed: armed})

+ }

+ sort.Slice(out, func(i, j int) bool { return out[i].Code < out[j].Code })

+ writeJSON(w, http.StatusOK, map[string]any{

+ "regions": out, "default": s.cfg.HomeRegion,

+ })

+}

diff --git a/internal/console/authcli.go b/internal/console/authcli.go

index 14eab60b2..cd8e3f795 100644

--- a/internal/console/authcli.go

+++ b/internal/console/authcli.go

@@ -567,6 +567,9 @@ func (s *Server) registerCLIRoutes() {

// door, the ANSWER a credential gets here moves from 401 to 403.

s.cliRoute("PUT /api/cli/apps/{account}/{app}/instances-manifest", svcrole.Set{}, s.handleAPIInstanceEnable)

s.cliRoute("GET /api/cli/apps/{account}/{app}/instances", svcrole.Allow(svcrole.Instance), s.handleAPIInstanceList)

+ // The armed-regions answer for consumer pickers (issue #346) — the

+ // same gate and role as the list beside it; see handleAPIRegions.

+ s.cliRoute("GET /api/cli/apps/{account}/{app}/regions", svcrole.Allow(svcrole.Instance), s.handleAPIRegions)

// Moving a route between channels (channelmove.go). The zero Set, for

// the manifest door's reason: this decides what a route SERVES, and a

// credential the app itself holds must not be able to walk its own route

@@ -583,6 +586,15 @@ func (s *Server) registerCLIRoutes() {

// keeps serving. Same {host} wildcard, same gate, same byte-identical

// 404 as its GET and DELETE siblings.

s.cliRoute("PATCH /api/cli/apps/{account}/{app}/instances/{host}", svcrole.Allow(svcrole.Instance), s.handleAPIInstanceUpdate)

+ // The residence-move doors (issue #331): requested by the tenant's

+ // ROUTER with the app-held instance token (titogo's residence design —

+ // never by an instance about itself... which the role system cannot yet

+ // express; the Instance role is the same authority the create and

+ // delete doors already grant this credential).

+ s.cliRoute("GET /api/cli/apps/{account}/{app}/instances/{host}/move", svcrole.Allow(svcrole.Instance), s.handleAPIInstanceMoveStatus)

+ s.cliRoute("POST /api/cli/apps/{account}/{app}/instances/{host}/move", svcrole.Allow(svcrole.Instance), s.handleAPIInstanceMoveRequest)

+ s.cliRoute("POST /api/cli/apps/{account}/{app}/instances/{host}/move/cut", svcrole.Allow(svcrole.Instance), s.handleAPIInstanceMoveCut)

+ s.cliRoute("DELETE /api/cli/apps/{account}/{app}/instances/{host}/move", svcrole.Allow(svcrole.Instance), s.handleAPIInstanceMoveAbort)

s.cliRoute("DELETE /api/cli/apps/{account}/{app}/instances/{host}", svcrole.Allow(svcrole.Instance), s.handleAPIInstanceDelete)

// The zero Set, deliberately: a credential may never mint a credential,

// including itself. This is the escalation cut (spec §3). It takes no

diff --git a/internal/console/instancesplacement.go b/internal/console/instancesplacement.go

index 649325f8d..41969cf15 100644

--- a/internal/console/instancesplacement.go

+++ b/internal/console/instancesplacement.go

@@ -96,23 +96,24 @@ func readEdgesFiles(ctx context.Context, s release.Store) []edgesFilePlacementVi

return out

}

-// armedPlacements folds the fleet's edges files into the set of placement

-// codes an instance may be created into (issue #329): every code published

-// by a box that also has a dial address (an EIP — a box nothing can reach

-// is not a landing target). armed reports whether ANY file carried a

-// placement code at all: false means the fleet predates the field (or the

-// read failed) and the gate stays DORMANT — the door then accepts what it

-// always accepted, so rolling boxes and console in either order is safe.

-func armedPlacements(ctx context.Context, s release.Store) (set map[string]bool, armed bool) {

+// armedPlacements folds the fleet's edges files into every published

+// placement code and whether it can take an instance today (issue #329):

+// armed means some box publishing the code also has a dial address (an

+// EIP — a box nothing can reach is not a landing target), so set[code]

+// false is a region that EXISTS but cannot land one, which the regions

+// endpoint (#346) shows and the create door refuses alike. The second

+// return reports whether ANY file carried a placement code at all: false

+// means the fleet predates the field (or the read failed) and the create

+// gate stays DORMANT — the door then accepts what it always accepted, so

+// rolling boxes and console in either order is safe.

+func armedPlacements(ctx context.Context, s release.Store) (set map[string]bool, published bool) {

set = map[string]bool{}

for _, f := range readEdgesFiles(ctx, s) {

if f.Placement == "" {

continue

}

- armed = true

- if f.EIP != "" {

- set[f.Placement] = true

- }

+ published = true

+ set[f.Placement] = set[f.Placement] || f.EIP != ""

}

- return set, armed

+ return set, published

}

diff --git a/internal/console/server.go b/internal/console/server.go

index 5328ba156..5ec66f115 100644

--- a/internal/console/server.go

+++ b/internal/console/server.go

@@ -454,7 +454,15 @@ type Config struct {

// handler reaching for os.Getenv would hide it from every reader of this

// struct. "" is legal — a local CARLOS_DEPLOYMENT_DIR deployment has no

// bucket name to draw — and simply draws no bucket node.

- Bucket string

+ Bucket string

+ // HomeRegion NAMES the deployment's home placement (CONSOLE_HOME_REGION,

+ // e.g. "ie") for consumer pickers (issue #346). It is a NAME, not a

+ // scope: records for the home keep region "" (the unconstrained value

+ // reconcile has always matched) and the create door normalizes this

+ // code to "" on the way in — so the fleet's boxes never see the code

+ // and nothing about placement changes when it is set. "" is legal: the

+ // regions endpoint then reports no home row and an empty default.

+ HomeRegion string

ReleasePub ed25519.PublicKey // pinned channel-pointer verification key

// Key signs channel pointers for promote/rollback (CARLOS_RELEASE_KEY).

// nil is a legal deployment: the console runs read-only, the promotion

diff --git a/internal/console/svcroutes_test.go b/internal/console/svcroutes_test.go

index c7fd77ac8..f5fac7a98 100644

--- a/internal/console/svcroutes_test.go

+++ b/internal/console/svcroutes_test.go

@@ -222,10 +222,23 @@ func TestNoRoleReachesAMintFleetOrPolicyDoor(t *testing.T) {

// the domains list answer that role has always reached, so this discloses

// nothing the list does not. The WRITE beside it is admin-only, and

// TestAPIFeaturesRoles pins that an instance credential never reaches it.

+// The three move doors (#331) are the fourth, fifth and sixth, together:

+// they did not exist before service principals, so listing them in `want`

+// would make the legacy map lie. The instance role reaches them

+// DELIBERATELY — the tenant's ROUTER drives a move with the app-held

+// CARLOS_INSTANCE_TOKEN (the residence design's stated caller), and a

+// credential that may already CREATE and DELETE instances (both in `want`)

+// is not gaining authority by relocating one. TestAPIInstanceMoveVerbs

+// pins what the doors themselves refuse.

var exemptFromLegacyPin = map[string]bool{

"GET /api/cli/apps": true,

- "GET /api/cli/apps/{account}/{app}/domains/{host}": true,

- "GET /api/cli/apps/{account}/{app}/features": true,

+ "GET /api/cli/apps/{account}/{app}/domains/{host}": true,

+ "GET /api/cli/apps/{account}/{app}/features": true,

+ "GET /api/cli/apps/{account}/{app}/regions": true,

+ "GET /api/cli/apps/{account}/{app}/instances/{host}/move": true,

+ "POST /api/cli/apps/{account}/{app}/instances/{host}/move": true,

+ "POST /api/cli/apps/{account}/{app}/instances/{host}/move/cut": true,

+ "DELETE /api/cli/apps/{account}/{app}/instances/{host}/move": true,

}

// TestInstanceRoleMatchesTodaysAppTokenReach is the compatibility pin of spec

diff --git a/internal/host/agent.go b/internal/host/agent.go

index 13bb1580c..0eab657ac 100644

--- a/internal/host/agent.go

+++ b/internal/host/agent.go

@@ -307,6 +307,18 @@ func RunAgent(args []string) {

if logSink != nil {

svc.Logs = logSink

}

+ // The residence-move machinery (issue #331), armed with the

+ // activator: the staging-fallback restore needs a same-region

+ // mirror client, and the move worker needs the service's quiesce

+ // answer. Both are best-effort — a mirror client that cannot

+ // build logs and leaves the fallback dark, exactly like every

+ // other optional duty here.

+ if m, err := lease.NewS3Mirror(context.Background(), cfg.Region, cfg.Region); err != nil {

+ log.Printf("agent: staging-fallback mirror client: %v (restore fallback disarmed)", err)

+ } else {

+ eb.Mirror = m

+ }

+ armMovesPass(*cfg, dd, reg, svc)

svc.Start()

act = svc

log.Printf("agent: hibernation activator armed (bucket %s, idle %s)", cfg.Bucket, cfg.Idle)

@@ -1451,6 +1463,7 @@ func fastConvergeLoop(reg *registry.Registry, svc *activator.Service, sd *schedu

if err := reconcileSerialized(ctx, store, reg, provisionTickConfig(os.Getenv)); err != nil {

log.Printf("agent: fast converge reconcile: %v", err)

}

+ runMovesPass(ctx, store)

// Folded once per pass and handed in, never re-read inside the

// loop: this leg runs on every serial bump (a seconds cadence),

// and the fold is a LoadBoxes plus a List plus one Get per edges

@@ -1667,12 +1680,33 @@ func provisionTickConfig(getenv func(string) string) provision.ReconcileConfig {

if dd == "" {

dd = "/data"

}

- return provision.ReconcileConfig{

+ cfg := provision.ReconcileConfig{

Region: region,

Pools: pools,

DataDir: dd,

Now: time.Now,

}

+ // The root release public key, same source and same parser as adoptLoop's

+ // and domainsTickConfig's. It guards ONE decision in this pass: whether a

+ // trash marker may retire a production-flagged app's routes (#340).

+ //

+ // Absence is tolerated and leaves ReleasePub nil, which retires trashed

+ // routes UNGUARDED — deliberately the same end state domains.Config

+ // documents and accepts for the same decision (§1c), because two passes

+ // reaching opposite conclusions about one app is worse than either

+ // answer. A key SET but unparseable lands in the same place, so it is

+ // logged here rather than swallowed: it is a persistent misconfiguration,

+ // and the operator who typo'd it must hear about the typo rather than

+ // silently losing the guard.

+ if raw := strings.TrimSpace(getenv("CARLOS_RELEASE_PUBKEY")); raw != "" {

+ pub, err := release.ParsePublicKey(raw, "CARLOS_RELEASE_PUBKEY")

+ if err != nil {

+ log.Printf("agent: instances pass: %v — a trash marker will retire routes without checking the production flag", err)

+ } else {

+ cfg.ReleasePub = pub

+ }

+ return cfg

}

// resolveHost is the production DNS seam: the default resolver with a hard

@@ -1916,6 +1950,7 @@ func instancesLoop(reg *registry.Registry) {

if err := reconcileSerialized(ctx, store, reg, cfg); err != nil {

log.Printf("agent: instances pass: %v", err)

}

+ runMovesPass(ctx, store)

deploymentRoutes := edgesfile.DeploymentRoutes(ctx, store, time.Now())

if err := domains.ReconcileAliases(ctx, store, reg, time.Now, deploymentRoutes); err != nil {

log.Printf("agent: instances alias pass: %v", err)

diff --git a/internal/host/edgesfile.go b/internal/host/edgesfile.go

index d57bbd1aa..9c4e86bdb 100644

--- a/internal/host/edgesfile.go

+++ b/internal/host/edgesfile.go

@@ -85,9 +85,17 @@ type edgesFile struct {

// console's create door folds every file's Placement into the armed set

// a requested instance region is validated against — dormant until at

// least one box publishes the field, so roll order does not matter.

- Placement string `json:"placement,omitempty"`

- Generation string `json:"generation"`

- Hosts map[string]edgeHost `json:"hosts"`

+ Placement string `json:"placement,omitempty"`

+ // DataBucket/DataBucketRegion are this box's hibernate/data bucket

+ // coordinates (CARLOS_HIBERNATE_BUCKET/_REGION verbatim) — published so

+ // a move SOURCE can aim its staging mirror at the target box's bucket

+ // from the same file set everything else reads (instance-move spec).

+ // Additive + omitempty like Placement; a box with no hibernate bucket

+ // (activator disabled) publishes neither key.

+ DataBucket string `json:"data_bucket,omitempty"`

+ DataBucketRegion string `json:"data_bucket_region,omitempty"`

+ Generation string `json:"generation"`

+ Hosts map[string]edgeHost `json:"hosts"`

}

// edgesFilePass builds this box's edges file from the routes currently in

@@ -123,7 +131,12 @@ type edgesFile struct {

// once per alias per process (cleared when the alias publishes).

var withheldAliasSaid sync.Map

-func edgesFilePass(ctx context.Context, s release.Store, reg *registry.Registry, box, eip, region, placement string, last *[]byte) error {

+// edgesFilePass publishes one file. ident is this box's IDENTITY half of

+// the file — Box, Region, Placement, DataBucket, DataBucketRegion — built

+// once by the loop; eip is passed per pass because it is re-probed every

+// tick and held across blanks (below). Hosts are derived here.

+func edgesFilePass(ctx context.Context, s release.Store, reg *registry.Registry, ident edgesFile, eip string, last *[]byte) error {

+ box := ident.Box

routes, err := reg.List()

if err != nil {

return fmt.Errorf("edges: list: %w", err)

@@ -257,7 +270,8 @@ func edgesFilePass(ctx context.Context, s release.Store, reg *registry.Registry,

}

- f := edgesFile{V: 1, Box: box, EIP: eip, Region: region, Placement: placement, Hosts: hosts}

+ f := ident

+ f.V, f.EIP, f.Hosts, f.Generation = 1, eip, hosts, ""

gate, err := json.Marshal(f)

if err != nil {

return fmt.Errorf("edges: marshal: %w", err)

@@ -371,7 +385,13 @@ func edgesFileLoop(reg *registry.Registry) {

// The placement code alongside it: CARLOS_REGION, the same env the

// reconcile pass scopes by — the two vocabularies travel in one file so

// a reader can join them without a second source.

- placement := strings.TrimSpace(os.Getenv("CARLOS_REGION"))

+ ident := edgesFile{

+ Box: box,

+ Region: region,

+ Placement: strings.TrimSpace(os.Getenv("CARLOS_REGION")),

+ DataBucket: strings.TrimSpace(os.Getenv("CARLOS_HIBERNATE_BUCKET")),

+ DataBucketRegion: strings.TrimSpace(os.Getenv("CARLOS_HIBERNATE_REGION")),

+ }

log.Printf("agent: edges file loop armed (5min + jitter, %s)", edgesFileKey(box))

last := seedEdgesFileGate(ctx, store, box)

delay := time.Minute

@@ -379,7 +399,7 @@ func edgesFileLoop(reg *registry.Registry) {

time.Sleep(delay)

delay = 5*time.Minute + rand.N(time.Minute)

eip := readPublicIP()

- if err := edgesFilePass(ctx, store, reg, box, eip, region, placement, &last); err != nil {

+ if err := edgesFilePass(ctx, store, reg, ident, eip, &last); err != nil {

log.Printf("agent: edges file pass: %v", err)

}

diff --git a/internal/host/edgesfile_test.go b/internal/host/edgesfile_test.go

index 2c219db67..96fd900ff 100644

--- a/internal/host/edgesfile_test.go

+++ b/internal/host/edgesfile_test.go

@@ -55,7 +55,7 @@ func TestEdgesFilePublishesHomedHosts(t *testing.T) {

s, reg := release.NewDirStore(t.TempDir()), edgesTestRegistry(t)

seedRoute(t, reg, "app.example.com")

var lastBody []byte

- if err := edgesFilePass(context.Background(), s, reg, "annie", "1.2.3.4", "", "", &lastBody); err != nil {

+ if err := edgesFilePass(context.Background(), s, reg, edgesFile{Box: "annie", Region: "", Placement: "", DataBucket: "", DataBucketRegion: ""}, "1.2.3.4", &lastBody); err != nil {

t.Fatal(err)

}

got, _, err := s.Get(context.Background(), "control/edges/annie.json")

@@ -86,7 +86,7 @@ func TestEdgesFilePublishesWithEmptyEIP(t *testing.T) {

s, reg := release.NewDirStore(t.TempDir()), edgesTestRegistry(t)

seedRoute(t, reg, "app.example.com")

var lastBody []byte

- if err := edgesFilePass(context.Background(), s, reg, "annie", "", "", "", &lastBody); err != nil {

+ if err := edgesFilePass(context.Background(), s, reg, edgesFile{Box: "annie", Region: "", Placement: "", DataBucket: "", DataBucketRegion: ""}, "", &lastBody); err != nil {

t.Fatal(err)

}

got, _, err := s.Get(context.Background(), "control/edges/annie.json")

@@ -112,7 +112,7 @@ func TestEdgesFileSkipsWriteWhenUnchanged(t *testing.T) {

seedRoute(t, reg, "app.example.com")

var lastBody []byte

- if err := edgesFilePass(context.Background(), s, reg, "annie", "1.2.3.4", "", "", &lastBody); err != nil {

+ if err := edgesFilePass(context.Background(), s, reg, edgesFile{Box: "annie", Region: "", Placement: "", DataBucket: "", DataBucketRegion: ""}, "1.2.3.4", &lastBody); err != nil {

t.Fatal(err)

}

if s.puts != 1 {

@@ -122,7 +122,7 @@ func TestEdgesFileSkipsWriteWhenUnchanged(t *testing.T) {

// Second pass, identical registry state: a fresh generation timestamp

// would differ from the first pass's, but the change-gate excludes

// Generation, so this must not write.

- if err := edgesFilePass(context.Background(), s, reg, "annie", "1.2.3.4", "", "", &lastBody); err != nil {

+ if err := edgesFilePass(context.Background(), s, reg, edgesFile{Box: "annie", Region: "", Placement: "", DataBucket: "", DataBucketRegion: ""}, "1.2.3.4", &lastBody); err != nil {

t.Fatal(err)

}

if s.puts != 1 {

@@ -131,7 +131,7 @@ func TestEdgesFileSkipsWriteWhenUnchanged(t *testing.T) {

// A registry change makes the next pass write again.

seedRoute(t, reg, "second.example.com")

- if err := edgesFilePass(context.Background(), s, reg, "annie", "1.2.3.4", "", "", &lastBody); err != nil {

+ if err := edgesFilePass(context.Background(), s, reg, edgesFile{Box: "annie", Region: "", Placement: "", DataBucket: "", DataBucketRegion: ""}, "1.2.3.4", &lastBody); err != nil {

t.Fatal(err)

}

if s.puts != 2 {

@@ -151,12 +151,12 @@ func TestEdgesFileHoldsLastKnownGoodEIPOnTransientBlank(t *testing.T) {

seedRoute(t, reg, "app.example.com")

var last []byte

- if err := edgesFilePass(context.Background(), s, reg, "annie", "1.2.3.4", "", "", &last); err != nil {

+ if err := edgesFilePass(context.Background(), s, reg, edgesFile{Box: "annie", Region: "", Placement: "", DataBucket: "", DataBucketRegion: ""}, "1.2.3.4", &last); err != nil {

t.Fatal(err)

}

// Simulate a transient IMDS blip: this tick's probe returns "".

- if err := edgesFilePass(context.Background(), s, reg, "annie", "", "", "", &last); err != nil {

+ if err := edgesFilePass(context.Background(), s, reg, edgesFile{Box: "annie", Region: "", Placement: "", DataBucket: "", DataBucketRegion: ""}, "", &last); err != nil {

t.Fatal(err)

}

@@ -186,7 +186,7 @@ func TestEdgesFileExcludesNonReadyRows(t *testing.T) {

}

var last []byte

- if err := edgesFilePass(context.Background(), s, reg, "annie", "1.2.3.4", "", "", &last); err != nil {

+ if err := edgesFilePass(context.Background(), s, reg, edgesFile{Box: "annie", Region: "", Placement: "", DataBucket: "", DataBucketRegion: ""}, "1.2.3.4", &last); err != nil {

t.Fatal(err)

}

got, _, err := s.Get(context.Background(), "control/edges/annie.json")

@@ -219,7 +219,7 @@ func TestEdgesFileWildcardRowPublishesVerbatim(t *testing.T) {

}

var last []byte

- if err := edgesFilePass(context.Background(), s, reg, "annie", "1.2.3.4", "", "", &last); err != nil {

+ if err := edgesFilePass(context.Background(), s, reg, edgesFile{Box: "annie", Region: "", Placement: "", DataBucket: "", DataBucketRegion: ""}, "1.2.3.4", &last); err != nil {

t.Fatal(err)

}

got, _, err := s.Get(context.Background(), "control/edges/annie.json")

@@ -250,7 +250,7 @@ func TestEdgesFileRawWireBodyMatchesSchema(t *testing.T) {

seedRoute(t, reg, "app.example.com")

var last []byte

- if err := edgesFilePass(context.Background(), s, reg, "quiet-harbour", "203.0.113.10", "", "", &last); err != nil {

+ if err := edgesFilePass(context.Background(), s, reg, edgesFile{Box: "quiet-harbour", Region: "", Placement: "", DataBucket: "", DataBucketRegion: ""}, "203.0.113.10", &last); err != nil {

t.Fatal(err)

}

got, _, err := s.Get(context.Background(), "control/edges/quiet-harbour.json")

@@ -302,7 +302,7 @@ func TestEdgesFileRawWireBodyRegionField(t *testing.T) {

t.Run("unset region is omitted entirely", func(t *testing.T) {

var last []byte

- if err := edgesFilePass(context.Background(), s, reg, "quiet-harbour", "203.0.113.10", "", "", &last); err != nil {

+ if err := edgesFilePass(context.Background(), s, reg, edgesFile{Box: "quiet-harbour", Region: "", Placement: "", DataBucket: "", DataBucketRegion: ""}, "203.0.113.10", &last); err != nil {

t.Fatal(err)

}

got, _, err := s.Get(context.Background(), "control/edges/quiet-harbour.json")

@@ -321,7 +321,7 @@ func TestEdgesFileRawWireBodyRegionField(t *testing.T) {

s2, reg2 := release.NewDirStore(t.TempDir()), edgesTestRegistry(t)

seedRoute(t, reg2, "app.example.com")

var last []byte

- if err := edgesFilePass(context.Background(), s2, reg2, "quiet-harbour", "203.0.113.10", "eu-west-1", "", &last); err != nil {

+ if err := edgesFilePass(context.Background(), s2, reg2, edgesFile{Box: "quiet-harbour", Region: "eu-west-1", Placement: "", DataBucket: "", DataBucketRegion: ""}, "203.0.113.10", &last); err != nil {

t.Fatal(err)

}

got, _, err := s2.Get(context.Background(), "control/edges/quiet-harbour.json")

@@ -423,7 +423,7 @@ func TestEdgesFileSeedGateSkipsWriteOnUnchangedBucketState(t *testing.T) {

// A prior process (or the prior pass of this one) already published.

var priming []byte

- if err := edgesFilePass(context.Background(), real, reg, "annie", "1.2.3.4", "", "", &priming); err != nil {

+ if err := edgesFilePass(context.Background(), real, reg, edgesFile{Box: "annie", Region: "", Placement: "", DataBucket: "", DataBucketRegion: ""}, "1.2.3.4", &priming); err != nil {

t.Fatal(err)

}

@@ -435,7 +435,7 @@ func TestEdgesFileSeedGateSkipsWriteOnUnchangedBucketState(t *testing.T) {

}

counting := &putCountingStore{Store: real}

- if err := edgesFilePass(context.Background(), counting, reg, "annie", "1.2.3.4", "", "", &last); err != nil {

+ if err := edgesFilePass(context.Background(), counting, reg, edgesFile{Box: "annie", Region: "", Placement: "", DataBucket: "", DataBucketRegion: ""}, "1.2.3.4", &last); err != nil {

t.Fatal(err)

}

if counting.puts != 0 {

@@ -454,7 +454,7 @@ func TestEdgesFileSeedHoldsEIPThroughRestartBlip(t *testing.T) {

seedRoute(t, reg, "app.example.com")

var priming []byte

- if err := edgesFilePass(context.Background(), s, reg, "annie", "1.2.3.4", "", "", &priming); err != nil {

+ if err := edgesFilePass(context.Background(), s, reg, edgesFile{Box: "annie", Region: "", Placement: "", DataBucket: "", DataBucketRegion: ""}, "1.2.3.4", &priming); err != nil {

t.Fatal(err)

}

@@ -466,7 +466,7 @@ func TestEdgesFileSeedHoldsEIPThroughRestartBlip(t *testing.T) {

// This tick both hits a transient IMDS blip AND sees a real registry

// change, so the pass must write.

seedRoute(t, reg, "second.example.com")

- if err := edgesFilePass(context.Background(), s, reg, "annie", "", "", "", &last); err != nil {

+ if err := edgesFilePass(context.Background(), s, reg, edgesFile{Box: "annie", Region: "", Placement: "", DataBucket: "", DataBucketRegion: ""}, "", &last); err != nil {

t.Fatal(err)

}

@@ -495,13 +495,13 @@ func TestEdgesFileUnseededLastPublishesBlankEIPOnFirstTickBlip(t *testing.T) {

seedRoute(t, reg, "app.example.com")

var priming []byte

- if err := edgesFilePass(context.Background(), s, reg, "annie", "1.2.3.4", "", "", &priming); err != nil {

+ if err := edgesFilePass(context.Background(), s, reg, edgesFile{Box: "annie", Region: "", Placement: "", DataBucket: "", DataBucketRegion: ""}, "1.2.3.4", &priming); err != nil {

t.Fatal(err)

}

// The pre-fix shape: a fresh process's *last starts nil (no seed read).

var unseeded []byte

- if err := edgesFilePass(context.Background(), s, reg, "annie", "", "", "", &unseeded); err != nil {

+ if err := edgesFilePass(context.Background(), s, reg, edgesFile{Box: "annie", Region: "", Placement: "", DataBucket: "", DataBucketRegion: ""}, "", &unseeded); err != nil {

t.Fatal(err)

}

@@ -545,7 +545,7 @@ func TestEdgesFileSeedPreRegionBucketWritesOnceWhenRegionArrives(t *testing.T) {

// This build now has CARLOS_REGION set: the first pass must write, once.

counting := &putCountingStore{Store: real}

- if err := edgesFilePass(context.Background(), counting, reg, "annie", "1.2.3.4", "eu-west-1", "", &last); err != nil {

+ if err := edgesFilePass(context.Background(), counting, reg, edgesFile{Box: "annie", Region: "eu-west-1", Placement: "", DataBucket: "", DataBucketRegion: ""}, "1.2.3.4", &last); err != nil {

t.Fatal(err)

}

if counting.puts != 1 {

@@ -565,7 +565,7 @@ func TestEdgesFileSeedPreRegionBucketWritesOnceWhenRegionArrives(t *testing.T) {

// A second, identical-region pass must not write again — the gate now

// includes region on both sides.

- if err := edgesFilePass(context.Background(), counting, reg, "annie", "1.2.3.4", "eu-west-1", "", &last); err != nil {

+ if err := edgesFilePass(context.Background(), counting, reg, edgesFile{Box: "annie", Region: "eu-west-1", Placement: "", DataBucket: "", DataBucketRegion: ""}, "1.2.3.4", &last); err != nil {

t.Fatal(err)

}

if counting.puts != 1 {

@@ -588,7 +588,7 @@ func TestEdgesFileMarksPlatformRows(t *testing.T) {

}

var lastBody []byte

- if err := edgesFilePass(context.Background(), s, reg, "annie", "1.2.3.4", "", "", &lastBody); err != nil {

+ if err := edgesFilePass(context.Background(), s, reg, edgesFile{Box: "annie", Region: "", Placement: "", DataBucket: "", DataBucketRegion: ""}, "1.2.3.4", &lastBody); err != nil {

t.Fatal(err)

}

body, _, err := s.Get(context.Background(), "control/edges/annie.json")

@@ -654,7 +654,7 @@ func TestEdgesFilePublishesPointingAliasesOfServedPrimaries(t *testing.T) {

publish := func() map[string]hostView {

t.Helper()

var lastBody []byte

- if err := edgesFilePass(context.Background(), s, reg, "annie", "1.2.3.4", "", "", &lastBody); err != nil {

+ if err := edgesFilePass(context.Background(), s, reg, edgesFile{Box: "annie", Region: "", Placement: "", DataBucket: "", DataBucketRegion: ""}, "1.2.3.4", &lastBody); err != nil {

t.Fatal(err)

}

body, _, err := s.Get(context.Background(), "control/edges/annie.json")

@@ -734,7 +734,7 @@ func TestEdgesFilePublishesDelegatedWildcardParents(t *testing.T) {

}

var lastBody []byte

- if err := edgesFilePass(context.Background(), s, reg, "annie", "1.2.3.4", "", "", &lastBody); err != nil {

+ if err := edgesFilePass(context.Background(), s, reg, edgesFile{Box: "annie", Region: "", Placement: "", DataBucket: "", DataBucketRegion: ""}, "1.2.3.4", &lastBody); err != nil {

t.Fatal(err)

}

body, _, err := s.Get(context.Background(), "control/edges/annie.json")

@@ -797,7 +797,7 @@ func TestEdgesFileBoxNameJoinsTheStatusSnapshotLabel(t *testing.T) {

s, reg := release.NewDirStore(t.TempDir()), edgesTestRegistry(t)

seedRoute(t, reg, "app.example.com")

var last []byte

- if err := edgesFilePass(ctx, s, reg, box, "203.0.113.7", "eu-west-1", "", &last); err != nil {

+ if err := edgesFilePass(ctx, s, reg, edgesFile{Box: box, Region: "eu-west-1", Placement: "", DataBucket: "", DataBucketRegion: ""}, "203.0.113.7", &last); err != nil {

t.Fatal(err)

}

if err := status.WriteBox(ctx, s, snapshot); err != nil {

@@ -817,7 +817,7 @@ func TestEdgesFileRawWireBodyPlacementField(t *testing.T) {

s, reg := release.NewDirStore(t.TempDir()), edgesTestRegistry(t)

seedRoute(t, reg, "app.example.com")

var last []byte

- if err := edgesFilePass(context.Background(), s, reg, "quiet-harbour", "203.0.113.10", "eu-west-2", "uk", &last); err != nil {

+ if err := edgesFilePass(context.Background(), s, reg, edgesFile{Box: "quiet-harbour", Region: "eu-west-2", Placement: "uk", DataBucket: "", DataBucketRegion: ""}, "203.0.113.10", &last); err != nil {

t.Fatal(err)

}

got, _, err := s.Get(context.Background(), "control/edges/quiet-harbour.json")

@@ -841,7 +841,7 @@ func TestEdgesFileRawWireBodyPlacementField(t *testing.T) {

s2, reg2 := release.NewDirStore(t.TempDir()), edgesTestRegistry(t)

seedRoute(t, reg2, "app.example.com")

var last2 []byte

- if err := edgesFilePass(context.Background(), s2, reg2, "quiet-harbour", "203.0.113.10", "", "", &last2); err != nil {

+ if err := edgesFilePass(context.Background(), s2, reg2, edgesFile{Box: "quiet-harbour", Region: "", Placement: "", DataBucket: "", DataBucketRegion: ""}, "203.0.113.10", &last2); err != nil {

t.Fatal(err)

}

got2, _, err := s2.Get(context.Background(), "control/edges/quiet-harbour.json")

@@ -852,3 +852,42 @@ func TestEdgesFileRawWireBodyPlacementField(t *testing.T) {

t.Fatalf("raw wire body with no CARLOS_REGION carries a placement key: %s", got2)

}

+

+// Data-bucket coordinates (instance-move spec): published verbatim from the

+// hibernate env pair, absent when unset — the same additive rule as

+// placement.

+func TestEdgesFileRawWireBodyDataBucketFields(t *testing.T) {

+ s, reg := release.NewDirStore(t.TempDir()), edgesTestRegistry(t)

+ seedRoute(t, reg, "app.example.com")

+ var last []byte

+ if err := edgesFilePass(context.Background(), s, reg, edgesFile{Box: "quiet-harbour", Region: "eu-west-2", Placement: "uk", DataBucket: "carlos-tito-uk-1", DataBucketRegion: "eu-west-2"}, "203.0.113.10", &last); err != nil {

+ t.Fatal(err)

+ }

+ got, _, err := s.Get(context.Background(), "control/edges/quiet-harbour.json")

+ if err != nil {

+ t.Fatal(err)

+ }

+ var probe struct {

+ DataBucket string `json:"data_bucket"`

+ DataBucketRegion string `json:"data_bucket_region"`

+ }

+ if err := json.Unmarshal(got, &probe); err != nil {

+ t.Fatal(err)

+ }

+ if probe.DataBucket != "carlos-tito-uk-1" || probe.DataBucketRegion != "eu-west-2" {

+ t.Fatalf("data bucket fields = %+v", probe)

+ }

+ s2, reg2 := release.NewDirStore(t.TempDir()), edgesTestRegistry(t)

+ seedRoute(t, reg2, "app.example.com")

+ var last2 []byte

+ if err := edgesFilePass(context.Background(), s2, reg2, edgesFile{Box: "quiet-harbour", Region: "", Placement: "", DataBucket: "", DataBucketRegion: ""}, "203.0.113.10", &last2); err != nil {

+ t.Fatal(err)

+ }

+ got2, _, err := s2.Get(context.Background(), "control/edges/quiet-harbour.json")

+ if err != nil {

+ t.Fatal(err)

+ }

+ if strings.Contains(string(got2), "data_bucket") {

+ t.Fatalf("unset hibernate env still published a data_bucket key: %s", got2)

+ }

+}

diff --git a/internal/host/host_test.go b/internal/host/host_test.go

index 2dad9eb30..77a958504 100644

--- a/internal/host/host_test.go

+++ b/internal/host/host_test.go

@@ -1,10 +1,12 @@

package host

import (

+ "bytes"

"context"

"crypto/ed25519"

"encoding/base64"

"encoding/json"

+ "log"

"net"

"os"

"path/filepath"

@@ -808,6 +810,34 @@ func TestProvisionTickConfig(t *testing.T) {

if got.Region != "ie" || got.Pools != nil {

t.Errorf("Region = %q Pools = %v, want ie / nil", got.Region, got.Pools)

}

+

+ // The release pubkey (#340): present and parseable arms the production

+ // guard on trash-driven route retirement.

+ if got.ReleasePub != nil {

+ t.Error("ReleasePub set with CARLOS_RELEASE_PUBKEY unset")

+ }

+ _, pubB64, err := release.GenerateKey()

+ if err != nil {

+ t.Fatal(err)

+ }

+ got = provisionTickConfig(env(map[string]string{"CARLOS_RELEASE_PUBKEY": pubB64}))

+ if got.ReleasePub == nil {

+ t.Error("ReleasePub nil with a valid CARLOS_RELEASE_PUBKEY")

+ }

+

+ // A key SET but unparseable must not be mistaken for a key that parsed.

+ // It lands on nil — the same unguarded end state domains.Config accepts

+ // for the same decision — and says so in the log rather than silently.

+ var logged bytes.Buffer

+ log.SetOutput(&logged)

+ t.Cleanup(func() { log.SetOutput(os.Stderr) })

+ got = provisionTickConfig(env(map[string]string{"CARLOS_RELEASE_PUBKEY": "not-a-key"}))

+ if got.ReleasePub != nil {

+ t.Error("ReleasePub set from an unparseable CARLOS_RELEASE_PUBKEY")

+ }

+ if !strings.Contains(logged.String(), "CARLOS_RELEASE_PUBKEY") {

+ t.Errorf("an unparseable key was swallowed; log was:\n%s", logged.String())

+ }

}

func TestBootstrapIdempotent(t *testing.T) {

diff --git a/internal/host/movespass.go b/internal/host/movespass.go

new file mode 100644

index 000000000..c3d195093

--- /dev/null

+++ b/internal/host/movespass.go

@@ -0,0 +1,345 @@

+package host

+

+import (

+ "context"

+ "encoding/json"

+ "errors"

+ "fmt"

+ "log"

+ "os"

+ "path/filepath"

+ "strings"

+ "sync"

+ "sync/atomic"

+ "time"

+

+ "github.com/carlosframework/platform/internal/activator"

+ "github.com/carlosframework/platform/internal/lease"

+ "github.com/carlosframework/platform/internal/provision"

+ "github.com/carlosframework/platform/internal/registry"

+ "github.com/carlosframework/platform/internal/release"

+)

+

+// The residence-move worker (issue #331, instance-move spec): everything a

+// move does to DATA happens here, on the SOURCE box — the box whose

+// CARLOS_REGION matches the record's region for as long as the move is in

+// flight. The console only writes requests; the target box only restores.

+//

+// One pass = one delta round per in-flight move. The caller runs it on the

+// same trigger as the reconcile pass (serial bumps land in seconds via

+// carlos-adopt.path), and prepare's convergence is the round-over-round

+// shrinking of the delta, not a resident goroutine — restart-safe because

+// every step derives from the record and the buckets.

+//

+// The cut ordering is the never-two-writers argument, mechanically:

+//

+// route Status "moving" (edge stops serving; RetireExcept stops the child)

+// → child quiesced (Quiesced(host) — retries next pass until true)

+// → final mirror round (own replica is byte-final; staging matches it)

+// → local purge (the warm-cache trap: a stale local file must

+// never resurrect a moved-away history)

+// → CAS flip+clear (Region=To, Move=nil, ONE write — the last act)

+//

+// Every step is idempotent; a box that dies mid-cut resumes at the top and

+// falls through the already-done steps.

+

+// movesWorker is the one home for the worker's cross-pass state: the wired

+// config, the memoized per-region mirror clients (built once per target

+// region, not per pass — a client build is a credential-chain resolution),

+// the once-per-host status nudge, and the per-host trouble throttle.

+// RunAgent writes the pointer once; both converge loops read it.

+type movesWorker struct {

+ cfg moveCfg

+ mu sync.Mutex

+ mirrors map[string]lease.MirrorStore // dstRegion → client pair

+ nudged sync.Map // host → true: status→quiesce hop nudged

+ trouble sync.Map // host → time.Time of last logged error

+}

+

+var armedMovesWorker atomic.Pointer[movesWorker]

+

+func (w *movesWorker) mirror(ctx context.Context, dstRegion string) (lease.MirrorStore, error) {

+ w.mu.Lock()

+ m := w.mirrors[dstRegion]

+ w.mu.Unlock()

+ if m != nil {

+ return m, nil

+ }

+ m, err := w.cfg.Mirror(ctx, dstRegion)

+ if err != nil {

+ return nil, err

+ }

+ w.mu.Lock()

+ w.mirrors[dstRegion] = m

+ w.mu.Unlock()

+ return m, nil

+}

+

+// armMovesPass wires the worker with what only RunAgent has in hand: the

+// activator's hibernate coordinates (the own-bucket side of every mirror),

+// the data dir the cut purges, the registry the cut quiesces through, and

+// the service that answers whether a child is down. Called once, before

+// svc.Start.

+func armMovesPass(hib activator.Config, dataDir string, reg *registry.Registry, svc *activator.Service) {

+

+ cfg := moveCfg{

+ Region: strings.TrimSpace(os.Getenv("CARLOS_REGION")),

+ OwnBucket: hib.Bucket,

+ OwnBucketRegion: hib.Region,

+ OwnPrefix: hib.Prefix,

+ DataDir: dataDir,

+ Mirror: func(ctx context.Context, dstRegion string) (lease.MirrorStore, error) {

+ return lease.NewS3Mirror(ctx, hib.Region, dstRegion)

+ },

+ Quiesced: svc.Quiesced,

+ SetStatus: reg.SetStatus,

+ OwnsRoute: func(host string) bool {

+ rt, err := reg.Get(host)

+ return err == nil && rt != nil && rt.Provisioned

+ },

+ Checkpoint: func(ctx context.Context, dbPath string) error {

+ host := strings.TrimSuffix(filepath.Base(dbPath), ".db")

+ rep, err := lease.ReplicateWithLease(ctx, dbPath, hib.Bucket, activator.HibernatePrefix(hib.Prefix, host), hib.Region)

+ if err != nil {

+ return err

+ }

+ defer rep.Close(ctx)

+ return rep.SyncAndWait(ctx)

+ },

+ Now: time.Now,

+ }

+ armedMovesWorker.Store(&movesWorker{cfg: cfg, mirrors: map[string]lease.MirrorStore{}})

+}

+

+// runMovesPass is the loops' call: inert until armed (no activator = no

+// exec instances = nothing this box could ever move).

+func runMovesPass(ctx context.Context, s release.Store) {

+ w := armedMovesWorker.Load()

+ if w == nil || s == nil {

+ return

+ }

+ if err := movesPass(ctx, s, w); err != nil {

+ log.Printf("agent: moves pass: %v", err)

+ }

+}

+

+// moveCfg carries what the worker needs, injectable for tests.

+type moveCfg struct {

+ Region string // this box's placement code, "" included (the home)

+ OwnBucket string // CARLOS_HIBERNATE_BUCKET

+ OwnBucketRegion string // CARLOS_HIBERNATE_REGION

+ OwnPrefix string // CARLOS_HIBERNATE_PREFIX ("instances" in production)

+ DataDir string // /data — where the local DB lives

+ Mirror func(ctx context.Context, dstRegion string) (lease.MirrorStore, error)

+ Quiesced func(host string) bool

+ SetStatus func(host, status string) error

+ // OwnsRoute answers whether THIS box's registry serves the host

+ // (review round 1): region equality alone would let a standby or

+ // second same-region box drive a cut whose child runs on its sibling —

+ // each box has its OWN registry, so the sibling's Quiesced is

+ // trivially, wrongly true. The box that owns the route owns the move.

+ OwnsRoute func(host string) bool

+ // Checkpoint re-replicates the LOCAL DB to the own bucket and waits

+ // (review round 1): the activator's stop logs — and swallows — a failed

+ // final checkpoint, because pre-move the local warm cache made that

+ // safe. The cut deletes the warm cache, so it must first prove the own

+ // replica really carries the tail. Nil skips (tests without S3).

+ Checkpoint func(ctx context.Context, dbPath string) error

+ Now func() time.Time

+}

+

+// movesPass runs one round over every in-flight move this box owns.

+func movesPass(ctx context.Context, s release.Store, w *movesWorker) error {

+ cfg := w.cfg

+ recs, err := provision.ListAllRecords(ctx, s)

+ if err != nil {

+ return fmt.Errorf("moves: list records: %w", err)

+ }

+ var targets map[string]edgesFile // lazy: most passes carry no move

+ for _, rec := range recs {

+ if rec.Move == nil || rec.Region != cfg.Region {

+ continue

+ }

+ // The rails the console's moveGate also holds (and this side must

+ // re-hold — records have other writers): only an activator-managed

+ // exec instance's state is this worker's to ship. A pooled record's

+ // data lives on the pool's box, not this one (v1 refuses the whole

+ // class), and a non-hibernating instance runs OUTSIDE the activator

+ // — Quiesced would answer true over a live, writing process.

+ if rec.Pool != "" || !rec.HibernateOrDefault() || (rec.Kind != "" && rec.Kind != "instance") {

+ continue

+ }

+ if cfg.OwnsRoute != nil && !cfg.OwnsRoute(rec.Host) {

+ continue // another box's child; its owner drives the move

+ }

+ if targets == nil {

+ if targets, err = moveTargets(ctx, s); err != nil {

+ log.Printf("moves: %v — every in-flight move waits for the next pass", err)

+ return nil

+ }

+ if err := runMoveRound(ctx, s, w, rec, targets); err != nil {

+ // One move's trouble must not starve its neighbours — and a

+ // PERSISTENT one (an unapplied staging grant's AccessDenied,

+ // review round 1) must not flood the journal at the fast

+ // converge loop's cadence: once a minute per host is plenty.

+ w.logTrouble(rec.Host, err)

+ continue

+ }

+ w.trouble.Delete(rec.Host)

+ }

+ return nil

+}

+

+// moveTargets folds the fleet's edges files into placement → coordinates.

+// A duplicate placement code keeps the lexically-first box (deterministic

+// everywhere, same rule as foldHomes) — and a target with no data bucket is

+// simply absent, which runMoveRound reports per-move.

+func moveTargets(ctx context.Context, s release.Store) (map[string]edgesFile, error) {

+ bodies, err := loadEdgesFileBodies(ctx, s)

+ if err != nil {

+ return nil, err

+ }

+ out := map[string]edgesFile{}

+ for _, body := range bodies {

+ var t edgesFile // the writer's own shape — no parallel decl to drift

+ if err := json.Unmarshal(body, &t); err != nil || t.V < 1 || t.Placement == "" || t.DataBucket == "" {

+ continue

+ }

+ if prev, dup := out[t.Placement]; dup && prev.Box <= t.Box {

+ continue

+ }

+ out[t.Placement] = t

+ }

+ return out, nil

+}

+

+func runMoveRound(ctx context.Context, s release.Store, w *movesWorker, rec provision.Record, targets map[string]edgesFile) error {

+ cfg := w.cfg

+ tgt, ok := targets[rec.Move.To]

+ if !ok {

+ return fmt.Errorf("target region %s publishes no data bucket yet — waiting", rec.Move.To)

+ }

+ // The mirror pair is built lazily and memoized per target region: the

+ // cut's not-yet-quiesced early return must not pay a credential-chain

+ // resolution it never uses.

+ mirror := func() (lease.MirrorStore, error) { return w.mirror(ctx, tgt.DataBucketRegion) }

+ srcPrefix := activator.HibernatePrefix(cfg.OwnPrefix, rec.Host)

+ dstPrefix := activator.MovesPrefix(rec.Host)

+

+ switch rec.Move.State {

+ case provision.MovePreparing, provision.MoveReady:

+ m, err := mirror()

+ if err != nil {

+ return err

+ }

+ changed, err := lease.MirrorPrefix(ctx, m, cfg.OwnBucket, srcPrefix, tgt.DataBucket, dstPrefix)

+ if err != nil {

+ return err

+ }

+ if rec.Move.State == provision.MovePreparing && changed == 0 {

+ if _, err := provision.SetRecordMove(ctx, s, rec.Account, rec.App, rec.Host, provision.MovePromoteReady(cfg.Now())); err != nil && !errors.Is(err, provision.ErrMoveConflict) {

+ return err

+ }

+ nudgeSerial(ctx, s)

+ log.Printf("moves: %s: staged replica caught up in %s/%s — ready to cut", rec.Host, tgt.DataBucket, dstPrefix)

+ }

+ return nil

+

+ case provision.MoveAborting:

+ // Abort's box half: delete the staged copy, THEN clear the move —

+ // a lingering staging would win a later landing on the target

+ // (staging-wins, activator.promoteStagedReplica).

+ m, err := mirror()

+ if err != nil {

+ return err

+ }

+ if err := lease.DeletePrefix(ctx, m, tgt.DataBucket, dstPrefix); err != nil {

+ return err

+ }

+ if _, err := provision.SetRecordMove(ctx, s, rec.Account, rec.App, rec.Host, provision.MoveAbortComplete()); err != nil && !errors.Is(err, provision.ErrMoveConflict) {

+ return err

+ }

+ nudgeSerial(ctx, s)

+ log.Printf("moves: %s: abort complete — staged copy deleted, instance stays in %q", rec.Host, rec.Region)

+ return nil

+

+ case provision.MoveCutting:

+ // Step 1+2: quiesce. SetStatus is idempotent; a route already

+ // "moving" (a resumed cut) just re-writes the word.

+ if err := cfg.SetStatus(rec.Host, "moving"); err != nil {

+ return fmt.Errorf("set route status: %w", err)

+ }

+ if !cfg.Quiesced(rec.Host) {

+ if _, done := w.nudged.LoadOrStore(rec.Host, true); !done {

+ nudgeSerial(ctx, s)

+ }

+ return nil // the retire is in flight; the nudged pass re-enters here

+ }

+ // Step 3: prove the own replica carries the tail. The activator's

+ // stop SWALLOWS a failed final checkpoint (safe pre-move: the warm

+ // cache stood behind it; fatal here: step 5 deletes that cache).

+ // Re-checkpointing an already-synced DB is cheap and idempotent; a

+ // held lease (the child not fully released) or an S3 blip returns

+ // an error and the pass retries.

+ dbPath := filepath.Join(cfg.DataDir, rec.Host+".db")

+ if cfg.Checkpoint != nil {

+ if _, statErr := os.Stat(dbPath); statErr == nil {

+ if err := cfg.Checkpoint(ctx, dbPath); err != nil {

+ return fmt.Errorf("final checkpoint: %w", err)

+ }

+ // Step 4: the byte-final mirror round.

+ m, err := mirror()

+ if err != nil {

+ return err

+ }

+ if _, err := lease.MirrorPrefix(ctx, m, cfg.OwnBucket, srcPrefix, tgt.DataBucket, dstPrefix); err != nil {

+ return err

+ }

+ // Step 5: the warm-cache purge.

+ activator.PurgeLocalDBArtifacts(dbPath, true)

+ // Step 6: the last act — Region flips, Move clears, one write.

+ if _, err := provision.SetRecordMove(ctx, s, rec.Account, rec.App, rec.Host, provision.MoveComplete()); err != nil && !errors.Is(err, provision.ErrMoveConflict) {

+ return err

+ }

+ w.nudged.Delete(rec.Host)

+ // The flip's nudge is what turns the TARGET's mint from a

+ // 5-minute wait into a seconds-cadence converge.

+ nudgeSerial(ctx, s)

+ log.Printf("moves: %s: cut complete — now homed in %s (route retires on the next reconcile pass)", rec.Host, rec.Move.To)

+ return nil

+ }

+ return nil

+}

+

+// nudgeSerial bumps control/serial after a box-side move write, so the

+// NEXT actor — this box's own cut continuation, the target's mint, the

+// tenant's poll — reacts on the fast-converge path instead of the 5-minute

+// floor. The live rehearsal measured that floor three times over in one

+// cut (status→quiesce→flip 331s, target mint+wake 700s); with the nudge,

+// each hop is a serial-triggered pass. Best-effort like the console's own

+// bumpSerial: a failed bump costs cadence, never correctness.

+func nudgeSerial(ctx context.Context, s release.Store) {

+ if err := release.BumpSerial(ctx, s, time.Now()); err != nil {

+ log.Printf("moves: serial nudge: %v (falling back to the timer floor)", err)

+ }

+}

+

+// movingStatusNudged remembers which hosts this process already nudged for

+// the status→quiesce hop, so a child that refuses to die does not turn the

+// fast-converge loop into a 2s self-triggering cycle: ONE nudge per host

+// per process, then the 5-minute floor stands under a wedged stop.

+var movingStatusNudged sync.Map

+

+// logTrouble rate-limits per-host move errors to one line a minute; the

+// pass clears a host's entry on its next clean round.

+func (w *movesWorker) logTrouble(host string, err error) {

+ now := time.Now()

+ if last, ok := w.trouble.Load(host); ok && now.Sub(last.(time.Time)) < time.Minute {

+ return

+ }

+ w.trouble.Store(host, now)

+ log.Printf("moves: %s: %v", host, err)

+}

diff --git a/internal/host/movespass_test.go b/internal/host/movespass_test.go

new file mode 100644

index 000000000..4cd60ecd7

--- /dev/null

+++ b/internal/host/movespass_test.go

@@ -0,0 +1,301 @@

+package host

+

+import (

+ "context"

+ "os"

+ "path/filepath"

+ "strings"

+ "testing"

+ "time"

+

+ "github.com/carlosframework/platform/internal/lease"

+ "github.com/carlosframework/platform/internal/provision"

+ "github.com/carlosframework/platform/internal/release"

+)

+

+// fakeMoveMirror: bucket→key→etag plus an op log for ordering assertions.

+type fakeMoveMirror struct {

+ buckets map[string]map[string]string

+ ops []string

+}

+

+func newFakeMoveMirror() *fakeMoveMirror {

+ return &fakeMoveMirror{buckets: map[string]map[string]string{}}

+}

+func (f *fakeMoveMirror) put(bucket, key, etag string) {

+ if f.buckets[bucket] == nil {

+ f.buckets[bucket] = map[string]string{}

+ }

+ f.buckets[bucket][key] = etag

+}

+func (f *fakeMoveMirror) List(_ context.Context, bucket, prefix string) (map[string]string, error) {

+ out := map[string]string{}

+ for k, v := range f.buckets[bucket] {

+ if strings.HasPrefix(k, prefix) {

+ out[k] = v

+ }

+ return out, nil

+}

+func (f *fakeMoveMirror) Copy(_ context.Context, srcBucket, srcKey, dstBucket, dstKey string) error {

+ f.ops = append(f.ops, "copy "+dstBucket+"/"+dstKey)

+ f.put(dstBucket, dstKey, f.buckets[srcBucket][srcKey])

+ return nil

+}

+func (f *fakeMoveMirror) Delete(_ context.Context, bucket, key string) error {

+ f.ops = append(f.ops, "delete "+bucket+"/"+key)

+ delete(f.buckets[bucket], key)

+ return nil

+}

+

+type moveHarness struct {

+ s release.Store

+ f *fakeMoveMirror

+ cfg moveCfg

+ statuses []string

+ quiet bool

+ owns bool

+}

+

+func newMoveHarness(t *testing.T) *moveHarness {

+ t.Helper()

+ h := &moveHarness{s: release.NewDirStore(t.TempDir()), f: newFakeMoveMirror(), quiet: true, owns: true}

+ h.cfg = moveCfg{

+ Region: "ie",

+ OwnBucket: "data-ie",

+ OwnBucketRegion: "eu-west-1",

+ OwnPrefix: "instances",

+ DataDir: t.TempDir(),

+ Mirror: func(_ context.Context, dstRegion string) (lease.MirrorStore, error) {

+ return h.f, nil

+ },

+ Quiesced: func(string) bool { return h.quiet },

+ SetStatus: func(host, status string) error { h.statuses = append(h.statuses, host+"="+status); return nil },

+ OwnsRoute: func(host string) bool { return h.owns },

+ Checkpoint: func(_ context.Context, dbPath string) error {

+ h.f.ops = append(h.f.ops, "checkpoint "+filepath.Base(dbPath))

+ return nil

+ },

+ Now: time.Now,

+ }

+ // The target's edges file: uk publishes a data bucket.

+ if _, err := h.s.Put(context.Background(), "control/edges/uk-box.json",

+ []byte(`{"v":1,"box":"uk-box","eip":"1.2.3.4","placement":"uk","data_bucket":"data-uk","data_bucket_region":"eu-west-2","hosts":{}}`), ""); err != nil {

+ t.Fatal(err)

+ }

+ return h

+}

+

+// worker wraps the harness cfg the way armMovesPass does in production —

+// a fresh worker per call, so memoized mirrors/nudges never leak between

+// the table's cases.

+func (h *moveHarness) worker() *movesWorker {

+ return &movesWorker{cfg: h.cfg, mirrors: map[string]lease.MirrorStore{}}

+}

+

+func (h *moveHarness) record(t *testing.T, host string, mut ...func(*provision.Record) error) provision.Record {

+ t.Helper()

+ rec := provision.Record{Schema: 1, Host: host, Slug: "s", App: "app", Account: "bab",

+ Region: "ie", Channel: "stable", CreatedAt: time.Now().UTC()}

+ if err := provision.CreateRecord(context.Background(), h.s, rec); err != nil {

+ t.Fatal(err)

+ }

+ for _, m := range mut {

+ if _, err := provision.SetRecordMove(context.Background(), h.s, "bab", "app", host, m); err != nil {

+ t.Fatal(err)

+ }

+ got, err := provision.GetRecord(context.Background(), h.s, "bab", "app", host)

+ if err != nil {

+ t.Fatal(err)

+ }

+ return *got

+}

+

+func (h *moveHarness) read(t *testing.T, host string) provision.Record {

+ t.Helper()

+ got, err := provision.GetRecord(context.Background(), h.s, "bab", "app", host)

+ if err != nil {

+ t.Fatal(err)

+ }

+ return *got

+}

+

+func TestMovesPassPrepareAndPromote(t *testing.T) {

+ h := newMoveHarness(t)

+ host := "a.go.example.com"

+ h.record(t, host, provision.MoveRequest("uk", time.Now()))

+ h.f.put("data-ie", "instances/"+host+"/generations/g/snapshot", "e1")

+ h.f.put("data-ie", "instances/"+host+"/generations/g/wal/000", "e2")

+

+ // Round 1: copies happen; the SAME pass does not promote (the round

+ // that copied cannot also claim caught-up — a write may have landed

+ // mid-copy; only a round that finds NOTHING to do promotes).

+ if err := movesPass(context.Background(), h.s, h.worker()); err != nil {

+ t.Fatal(err)

+ }

+ if got := h.read(t, host); got.Move.State != provision.MovePreparing {

+ t.Fatalf("after the copying round: %s, want still preparing", got.Move.State)

+ }

+ staged, _ := h.f.List(context.Background(), "data-uk", "moves/"+host+"/")

+ if len(staged) != 2 {

+ t.Fatalf("staging = %v", staged)

+ }

+ // Round 2: nothing to do → ready.

+ if err := movesPass(context.Background(), h.s, h.worker()); err != nil {

+ t.Fatal(err)

+ }

+ got := h.read(t, host)

+ if got.Move.State != provision.MoveReady || got.Move.ReadyAt == nil {

+ t.Fatalf("after the quiet round: %+v, want ready", got.Move)

+ }

+ // Ready keeps mirroring: a late write flows on the next round.

+ h.f.put("data-ie", "instances/"+host+"/generations/g/wal/001", "e3")

+ if err := movesPass(context.Background(), h.s, h.worker()); err != nil {

+ t.Fatal(err)

+ }

+ staged, _ = h.f.List(context.Background(), "data-uk", "moves/"+host+"/")

+ if len(staged) != 3 || h.read(t, host).Move.State != provision.MoveReady {

+ t.Fatalf("late write did not flow: %v", staged)

+ }

+}

+

+func TestMovesPassCutSequence(t *testing.T) {

+ h := newMoveHarness(t)

+ host := "a.go.example.com"

+ h.record(t, host, provision.MoveRequest("uk", time.Now()), provision.MovePromoteReady(time.Now()), provision.MoveMarkCutting())

+ h.f.put("data-ie", "instances/"+host+"/generations/g/snapshot", "e1")

+ // A local warm cache that must be purged.

+ db := filepath.Join(h.cfg.DataDir, host+".db")

+ if err := writeFileForTest(db, "stale"); err != nil {

+ t.Fatal(err)

+ }

+

+ // Not quiesced yet: the pass sets the status and stops there.

+ h.quiet = false

+ if err := movesPass(context.Background(), h.s, h.worker()); err != nil {

+ t.Fatal(err)

+ }

+ if len(h.statuses) != 1 || h.statuses[0] != host+"=moving" {

+ t.Fatalf("statuses = %v", h.statuses)

+ }

+ if got := h.read(t, host); got.Region != "ie" || got.Move.State != provision.MoveCutting {

+ t.Fatalf("cut advanced past an un-quiesced child: %+v", got)

+ }

+ if len(h.f.ops) != 0 {

+ t.Fatalf("mirror ran before quiesce: %v", h.f.ops)

+ }

+

+ // Quiesced: the whole tail runs, in order — the final own-bucket

+ // checkpoint strictly BEFORE the mirror round (the mirror of a stale

+ // replica is faithful and useless).

+ h.quiet = true

+ if err := movesPass(context.Background(), h.s, h.worker()); err != nil {

+ t.Fatal(err)

+ }

+ ckIdx, cpIdx := -1, -1

+ for i, op := range h.f.ops {

+ if strings.HasPrefix(op, "checkpoint ") && ckIdx == -1 {

+ ckIdx = i

+ }

+ if strings.HasPrefix(op, "copy ") && cpIdx == -1 {

+ cpIdx = i

+ }

+ if ckIdx == -1 || cpIdx == -1 || ckIdx > cpIdx {

+ t.Fatalf("cut op order = %v, want checkpoint before the mirror copy", h.f.ops)

+ }

+ got := h.read(t, host)

+ if got.Region != "uk" || got.Move != nil {

+ t.Fatalf("after cut: region=%q move=%+v", got.Region, got.Move)

+ }

+ if fileExistsForTest(db) {

+ t.Fatal("local DB survived the cut — the warm-cache trap")

+ }

+ staged, _ := h.f.List(context.Background(), "data-uk", "moves/"+host+"/")

+ if len(staged) != 1 {

+ t.Fatalf("staging after final round = %v", staged)

+ }

+ // The pass is now inert for this record (region uk ≠ cfg ie).

+ before := len(h.f.ops)

+ if err := movesPass(context.Background(), h.s, h.worker()); err != nil {

+ t.Fatal(err)

+ }

+ if len(h.f.ops) != before {

+ t.Fatal("a completed move kept mirroring")

+ }

+}

+

+func TestMovesPassSkipsWithoutTargetCoords(t *testing.T) {

+ h := newMoveHarness(t)

+ host := "a.go.example.com"

+ h.record(t, host, provision.MoveRequest("se", time.Now())) // se publishes nothing

+ h.f.put("data-ie", "instances/"+host+"/generations/g/snapshot", "e1")

+ if err := movesPass(context.Background(), h.s, h.worker()); err != nil {

+ t.Fatal(err)

+ }

+ if got := h.read(t, host); got.Move.State != provision.MovePreparing {

+ t.Fatalf("state advanced with no target coords: %+v", got.Move)

+ }

+ if len(h.f.ops) != 0 {

+ t.Fatalf("mirrored toward an unknown target: %v", h.f.ops)

+ }

+}

+

+func writeFileForTest(path, content string) error { return os.WriteFile(path, []byte(content), 0o600) }

+func fileExistsForTest(path string) bool { _, err := os.Stat(path); return err == nil }

+

+// The rails and the ownership gate (review round 1): pooled, non-hibernate,

+// non-instance, and not-my-route records are skipped untouched.

+func TestMovesPassRailsAndOwnership(t *testing.T) {

+ h := newMoveHarness(t)

+ host := "a.go.example.com"

+ h.record(t, host, provision.MoveRequest("uk", time.Now()))

+ h.f.put("data-ie", "instances/"+host+"/generations/g/snapshot", "e1")

+

+ // Not this box's route (a standby sharing the region): untouched.

+ h.owns = false

+ if err := movesPass(context.Background(), h.s, h.worker()); err != nil {

+ t.Fatal(err)

+ }

+ if len(h.f.ops) != 0 || h.read(t, host).Move.State != provision.MovePreparing {

+ t.Fatalf("a non-owning box acted: ops=%v", h.f.ops)

+ }

+ h.owns = true

+

+ // A pooled sibling with a move field: skipped (the platform box holds

+ // none of its data — a round here would promote ready over nothing).

+ pooled := "p.go.example.com"

+ rec := provision.Record{Schema: 1, Host: pooled, Slug: "p", App: "app", Account: "bab",

+ Region: "ie", Pool: "dedicated-x", Channel: "stable", CreatedAt: time.Now().UTC()}

+ if err := provision.CreateRecord(context.Background(), h.s, rec); err != nil {

+ t.Fatal(err)

+ }

+ if _, err := provision.SetRecordMove(context.Background(), h.s, "bab", "app", pooled, provision.MoveRequest("uk", time.Now())); err != nil {

+ t.Fatal(err)

+ }

+ if err := movesPass(context.Background(), h.s, h.worker()); err != nil {

+ t.Fatal(err)

+ }

+ if h.read(t, pooled).Move.State != provision.MovePreparing {

+ t.Fatal("a pooled record's move advanced on the platform box")

+ }

+}

+

+// Abort's box half: staging deleted, then the field cleared.

+func TestMovesPassAbortPhase(t *testing.T) {

+ h := newMoveHarness(t)

+ host := "a.go.example.com"

+ h.record(t, host, provision.MoveRequest("uk", time.Now()), provision.MoveAbort())

+ h.f.put("data-uk", "moves/"+host+"/generations/g/snapshot", "e1")

+ if err := movesPass(context.Background(), h.s, h.worker()); err != nil {

+ t.Fatal(err)

+ }

+ staged, _ := h.f.List(context.Background(), "data-uk", "moves/"+host+"/")

+ if len(staged) != 0 {

+ t.Fatalf("staging survived the abort: %v", staged)

+ }

+ if got := h.read(t, host); got.Move != nil || got.Region != "ie" {

+ t.Fatalf("after abort phase: %+v region=%q", got.Move, got.Region)

+ }

+}

diff --git a/internal/lease/mirror.go b/internal/lease/mirror.go

new file mode 100644

index 000000000..2061a4c4e

--- /dev/null

+++ b/internal/lease/mirror.go

@@ -0,0 +1,213 @@

+package lease

+

+import (

+ "context"

+ "fmt"

+ "net/url"

+ "sort"

+ "strings"

+ "sync"

+

+ awss3 "github.com/aws/aws-sdk-go-v2/service/s3"

+)

+

+// The staging mirror (issue #331, instance-move spec): a residence move

+// copies the instance's REPLICA prefix — the litestream generations the

+// activator already maintains — never the database itself. MirrorPrefix is

+// the one delta round; the move worker calls it repeatedly until a round

+// reports nothing changed, and once more after the child is down.

+//

+// The interface exists for the tests (an in-memory fake) and because the

+// destination bucket lives in ANOTHER region: the implementation holds one

+// client per side.

+

+// MirrorStore is the three S3 actions a delta round needs. List answers

+// key→ETag under a prefix (keys RELATIVE to the bucket root, not the

+// prefix); Copy is a server-side CopyObject; Delete removes one key.

+type MirrorStore interface {

+ List(ctx context.Context, bucket, prefix string) (map[string]string, error)

+ Copy(ctx context.Context, srcBucket, srcKey, dstBucket, dstKey string) error

+ Delete(ctx context.Context, bucket, key string) error

+}

+

+// MirrorPrefix makes dstBucket/dstPrefix an exact mirror of

+// srcBucket/srcPrefix: copy keys that are new or whose ETag differs, delete

+// keys the source no longer has (litestream retention must hold in staging

+// too — a restore that replays pruned WAL segments is a restore nobody

+// tested). Returns how many objects changed; 0 means the sides were

+// identical when listed — the move worker's "caught up".

+//

+// ETag equality is a valid sameness test here even for multipart uploads:

+// both sides of a server-side COPY carry the copy's own single-part ETag,

+// and litestream's own objects are single PUTs.

+func MirrorPrefix(ctx context.Context, m MirrorStore, srcBucket, srcPrefix, dstBucket, dstPrefix string) (int, error) {

+ // Both prefixes are DIRECTORY boundaries, normalized here to end in

+ // "/": activator.HibernatePrefix carries none (path.Join), MovesPrefix

+ // carries one, and relative-key math over a slashless prefix would both

+ // mint fused keys (instances/hgenerations/…) and let a listing bleed

+ // into a sibling host whose name merely extends this one's.

+ if !strings.HasSuffix(srcPrefix, "/") {

+ srcPrefix += "/"

+ }

+ if !strings.HasSuffix(dstPrefix, "/") {

+ dstPrefix += "/"

+ }

+ src, err := m.List(ctx, srcBucket, srcPrefix)

+ if err != nil {

+ return 0, fmt.Errorf("mirror: list source %s/%s: %w", srcBucket, srcPrefix, err)

+ }

+ dst, err := m.List(ctx, dstBucket, dstPrefix)

+ if err != nil {

+ return 0, fmt.Errorf("mirror: list staging %s/%s: %w", dstBucket, dstPrefix, err)

+ }

+ // Deterministic order: a failed round resumes the same way everywhere.

+ rel := make([]string, 0, len(src))

+ for k := range src {

+ rel = append(rel, k[len(srcPrefix):])

+ }

+ sort.Strings(rel)

+ changed := 0

+ for _, r := range rel {

+ // The lease's lock.json is COORDINATION state, not data (live

+ // rehearsal catch, 2026-08-29): the renewer rewrites it on every

+ // TTL renewal, so mirroring it makes "changed==0" unreachable for

+ // any awake instance — and promoting it would hand the target a

+ // stale foreign lease. Skipped on both sides: never copied, and

+ // the destination's own copy (if any) is never deleted.

+ if r == leaseLockName {

+ continue

+ }

+ sk, dk := srcPrefix+r, dstPrefix+r

+ if et, ok := dst[dk]; ok && et == src[sk] {

+ continue

+ }

+ if err := m.Copy(ctx, srcBucket, sk, dstBucket, dk); err != nil {

+ return changed, fmt.Errorf("mirror: copy %s: %w", sk, err)

+ }

+ changed++

+ }

+ for dk := range dst {

+ r := dk[len(dstPrefix):]

+ if r == leaseLockName {

+ continue

+ }

+ if _, ok := src[srcPrefix+r]; !ok {

+ if err := m.Delete(ctx, dstBucket, dk); err != nil {

+ return changed, fmt.Errorf("mirror: delete %s: %w", dk, err)

+ }

+ changed++

+ }

+ return changed, nil

+}

+

+// leaseLockName is the coordination file this package's own leaser keeps

+// beside every replica. MirrorPrefix's exclusion of it is lease POLICY, not

+// generic mirroring — which is why the mirror lives in this package.

+const leaseLockName = "lock.json"

+

+// DeletePrefix removes every object under prefix (the lease lock included:

+// a deleted staging area holds nothing back). The two callers are abort's

+// box half (errors surface — the abort retries) and the activator's

+// post-restore cleanup (best-effort at the call site).

+func DeletePrefix(ctx context.Context, m MirrorStore, bucket, prefix string) error {

+ objs, err := m.List(ctx, bucket, prefix)

+ if err != nil {

+ return err

+ }

+ for k := range objs {

+ if err := m.Delete(ctx, bucket, k); err != nil {

+ return err

+ }

+ return nil

+}

+

+// s3Mirror implements MirrorStore over two regional clients: reads and the

+// copy call go to the DESTINATION region's client (CopyObject executes in

+// the destination bucket's region and pulls from the source), source

+// listings to the source's.

+type s3Mirror struct {

+ src, dst *awss3.Client

+ // byBucket memoizes which client answered for a bucket, so only the

+ // FIRST List of each bucket pays the wrong-region probe — and a later

+ // transient error on the right client is not retried against a region

+ // that can only answer with a redirect.

+ mu sync.Mutex

+ byBucket map[string]*awss3.Client

+}

+

+// NewS3Mirror builds the two-region pair with the same credential chain and

+// endpoint compatibility every other S3 consumer here uses (endpoint.go's

+// newLeaserClient).

+func NewS3Mirror(ctx context.Context, srcRegion, dstRegion string) (MirrorStore, error) {

+ src, _, err := newLeaserClient(ctx, srcRegion)

+ if err != nil {

+ return nil, err

+ }

+ dst, _, err := newLeaserClient(ctx, dstRegion)

+ if err != nil {

+ return nil, err

+ }

+ return &s3Mirror{src: src, dst: dst, byBucket: map[string]*awss3.Client{}}, nil

+}

+

+func (m *s3Mirror) list(ctx context.Context, c *awss3.Client, bucket, prefix string) (map[string]string, error) {

+ out := map[string]string{}

+ pager := awss3.NewListObjectsV2Paginator(c, &awss3.ListObjectsV2Input{Bucket: &bucket, Prefix: &prefix})

+ for pager.HasMorePages() {

+ resp, err := pager.NextPage(ctx)

+ if err != nil {

+ return nil, err

+ }

+ for _, o := range resp.Contents {

+ if o.Key == nil {

+ continue

+ }

+ et := ""

+ if o.ETag != nil {

+ et = *o.ETag

+ }

+ out[*o.Key] = et

+ }

+ return out, nil

+}

+

+// List routes by memoized bucket→client binding: the interface's caller

+// never says which region a bucket lives in, so the FIRST touch of a bucket

+// probes source-then-destination (a wrong-region List is an error, so the

+// probe is unambiguous) and the winner is remembered.

+func (m *s3Mirror) List(ctx context.Context, bucket, prefix string) (map[string]string, error) {

+ m.mu.Lock()

+ c := m.byBucket[bucket]

+ m.mu.Unlock()

+ if c != nil {

+ return m.list(ctx, c, bucket, prefix)

+ }

+ out, err := m.list(ctx, m.src, bucket, prefix)

+ c = m.src

+ if err != nil {

+ if out, err = m.list(ctx, m.dst, bucket, prefix); err != nil {

+ return nil, err

+ }

+ c = m.dst

+ }

+ m.mu.Lock()

+ m.byBucket[bucket] = c

+ m.mu.Unlock()

+ return out, nil

+}

+

+func (m *s3Mirror) Copy(ctx context.Context, srcBucket, srcKey, dstBucket, dstKey string) error {

+ source := url.PathEscape(srcBucket + "/" + srcKey)

+ _, err := m.dst.CopyObject(ctx, &awss3.CopyObjectInput{

+ Bucket: &dstBucket, Key: &dstKey, CopySource: &source,

+ })

+ return err

+}

+

+func (m *s3Mirror) Delete(ctx context.Context, bucket, key string) error {

+ _, err := m.dst.DeleteObject(ctx, &awss3.DeleteObjectInput{Bucket: &bucket, Key: &key})

+ return err

+}

diff --git a/internal/lease/mirror_test.go b/internal/lease/mirror_test.go

new file mode 100644

index 000000000..124b3b896

--- /dev/null

+++ b/internal/lease/mirror_test.go

@@ -0,0 +1,133 @@

+package lease

+

+import (

+ "context"

+ "fmt"

+ "strings"

+ "testing"

+)

+

+// fakeMirror is an in-memory MirrorStore: buckets of key→etag, plus an op

+// log so ordering tests can assert the sequence.

+type fakeMirror struct {

+ buckets map[string]map[string]string

+ ops []string

+ failOn string

+}

+

+func newFakeMirror() *fakeMirror { return &fakeMirror{buckets: map[string]map[string]string{}} }

+

+func (f *fakeMirror) put(bucket, key, etag string) {

+ if f.buckets[bucket] == nil {

+ f.buckets[bucket] = map[string]string{}

+ }

+ f.buckets[bucket][key] = etag

+}

+

+func (f *fakeMirror) List(_ context.Context, bucket, prefix string) (map[string]string, error) {

+ out := map[string]string{}

+ for k, v := range f.buckets[bucket] {

+ if strings.HasPrefix(k, prefix) {

+ out[k] = v

+ }

+ return out, nil

+}

+

+func (f *fakeMirror) Copy(_ context.Context, srcBucket, srcKey, dstBucket, dstKey string) error {

+ if f.failOn == srcKey {

+ return fmt.Errorf("injected copy failure on %s", srcKey)

+ }

+ f.ops = append(f.ops, "copy "+srcKey+" -> "+dstBucket+"/"+dstKey)

+ f.put(dstBucket, dstKey, f.buckets[srcBucket][srcKey])

+ return nil

+}

+

+func (f *fakeMirror) Delete(_ context.Context, bucket, key string) error {

+ f.ops = append(f.ops, "delete "+bucket+"/"+key)

+ delete(f.buckets[bucket], key)

+ return nil

+}

+

+func TestMirrorPrefix(t *testing.T) {

+ ctx := context.Background()

+ f := newFakeMirror()

+ f.put("src", "instances/h/generations/a/snapshot", "e1")

+ f.put("src", "instances/h/generations/a/wal/000", "e2")

+

+ // Initial round: full copy.

+ n, err := MirrorPrefix(ctx, f, "src", "instances/h/", "dst", "moves/h/")

+ if err != nil || n != 2 {

+ t.Fatalf("initial round = %d, %v; want 2 copies", n, err)

+ }

+ // Identical: zero.

+ if n, err = MirrorPrefix(ctx, f, "src", "instances/h/", "dst", "moves/h/"); err != nil || n != 0 {

+ t.Fatalf("caught-up round = %d, %v; want 0", n, err)

+ }

+ // Delta: one changed etag, one new, one pruned at source.

+ f.put("src", "instances/h/generations/a/wal/000", "e2b")

+ f.put("src", "instances/h/generations/a/wal/001", "e3")

+ delete(f.buckets["src"], "instances/h/generations/a/snapshot")

+ n, err = MirrorPrefix(ctx, f, "src", "instances/h/", "dst", "moves/h/")

+ if err != nil || n != 3 {

+ t.Fatalf("delta round = %d, %v; want 2 copies + 1 delete", n, err)

+ }

+ got, _ := f.List(ctx, "dst", "moves/h/")

+ want := map[string]string{

+ "moves/h/generations/a/wal/000": "e2b",

+ "moves/h/generations/a/wal/001": "e3",

+ }

+ if len(got) != len(want) {

+ t.Fatalf("staging = %v, want %v", got, want)

+ }

+ for k, v := range want {

+ if got[k] != v {

+ t.Fatalf("staging[%s] = %q, want %q", k, got[k], v)

+ }

+ // A copy failure surfaces with the partial count.

+ f.put("src", "instances/h/generations/a/wal/002", "e4")

+ f.failOn = "instances/h/generations/a/wal/002"

+ if _, err := MirrorPrefix(ctx, f, "src", "instances/h/", "dst", "moves/h/"); err == nil {

+ t.Fatal("injected failure did not surface")

+ }

+}

+

+// Prefix boundaries: a slashless prefix (activator.HibernatePrefix's shape)

+// is normalized to a directory boundary — no fused keys, and a sibling host

+// whose name extends this one's is never swept in.

+func TestMirrorPrefixNormalizesBoundaries(t *testing.T) {

+ ctx := context.Background()

+ f := newFakeMirror()

+ f.put("src", "instances/h/generations/a/snapshot", "e1")

+ f.put("src", "instances/h2/generations/a/snapshot", "eX") // the sibling

+ n, err := MirrorPrefix(ctx, f, "src", "instances/h", "dst", "moves/h")

+ if err != nil || n != 1 {

+ t.Fatalf("round = %d, %v; want exactly the one h object", n, err)

+ }

+ got, _ := f.List(ctx, "dst", "")

+ if _, ok := got["moves/h/generations/a/snapshot"]; !ok || len(got) != 1 {

+ t.Fatalf("staging keys = %v, want the one properly-joined key", got)

+ }

+}

+

+// lock.json is coordination state: never copied, never deleted, and its

+// churn (the renewer rewrites it every TTL) must not keep changed above 0.

+func TestMirrorPrefixSkipsTheLease(t *testing.T) {

+ ctx := context.Background()

+ f := newFakeMirror()

+ f.put("src", "instances/h/lock.json", "renewal-1")

+ f.put("src", "instances/h/generations/a/snapshot", "e1")

+ f.put("dst", "moves/h/lock.json", "old-foreign-lease")

+ if n, err := MirrorPrefix(ctx, f, "src", "instances/h/", "dst", "moves/h/"); err != nil || n != 1 {

+ t.Fatalf("round = %d, %v; want only the snapshot to move", n, err)

+ }

+ f.put("src", "instances/h/lock.json", "renewal-2") // the churn

+ if n, err := MirrorPrefix(ctx, f, "src", "instances/h/", "dst", "moves/h/"); err != nil || n != 0 {

+ t.Fatalf("churn round = %d, %v; want caught-up despite the renewal", n, err)

+ }

+ got, _ := f.List(ctx, "dst", "moves/h/")

+ if got["moves/h/lock.json"] != "old-foreign-lease" || len(got) != 2 {

+ t.Fatalf("staging = %v; the dst lease must be untouched", got)

+ }

+}

diff --git a/internal/provision/provision.go b/internal/provision/provision.go

index 2108074b1..cd5f59187 100644

--- a/internal/provision/provision.go

+++ b/internal/provision/provision.go

@@ -42,6 +42,12 @@ type Record struct {

CreatedAt time.Time `json:"created_at"`

CreatedBy string `json:"created_by,omitempty"`

+ // Move is an in-flight residence move (issue #331, instance-move spec):

+ // present only between move-prepare and the cut's terminal flip, absent

+ // on every record at rest. Additive + omitempty — pre-move records parse

+ // unchanged. Written ONLY through SetRecordMove's guarded transitions.

+ Move *MoveState `json:"move,omitempty"`

+

// --- shape fields (2026-08-10 route-records design §3) --------------

// Every one is omitempty with a zero value that reproduces the mint

// reconcile hardcoded before they existed: a record written by an older

@@ -645,30 +651,192 @@ func CreateRecord(ctx context.Context, s release.Store, rec Record) error {

// the record is gone, not a manufactured success for a write that touched

// nothing.

func SetRecordRegion(ctx context.Context, s release.Store, acct, app, host, region string) error {

+ _, err := updateRecord(ctx, s, acct, app, host, func(rec *Record) error {

+ rec.Region = region

+ return nil

+ })

+ return err

+}

+

+// updateRecord is the one read-CAS-put-one-retry scaffold under both

+// in-place record writers (SetRecordRegion, SetRecordMove): re-reading and

+// re-applying a narrow mutation on a lost race changes nothing about what

+// it meant. Returns the record as written.

+func updateRecord(ctx context.Context, s release.Store, acct, app, host string, mut func(*Record) error) (*Record, error) {

key := RecordKey(acct, app, host)

- update := func() error {

+ update := func() (*Record, error) {

raw, etag, err := s.Get(ctx, key)

if err != nil {

- return err

+ return nil, err

}

var rec Record

if err := json.Unmarshal(raw, &rec); err != nil {

- return fmt.Errorf("provision: record %s/%s/%s: %w", acct, app, host, err)

+ return nil, fmt.Errorf("provision: record %s/%s/%s: %w", acct, app, host, err)

+ }

+ if err := mut(&rec); err != nil {

+ return nil, err

}

- rec.Region = region

body, err := json.Marshal(rec)

if err != nil {

- return err

+ return nil, err

}

- _, err = s.Put(ctx, key, body, etag)

- return err

+ if _, err := s.Put(ctx, key, body, etag); err != nil {

+ return nil, err

+ }

+ return &rec, nil

}

- if err := update(); errors.Is(err, release.ErrCASConflict) {

+ rec, err := update()

+ if errors.Is(err, release.ErrCASConflict) {

return update() // one retry on a lost race; a second loss surfaces

- } else if err != nil {

- return err

}

- return nil

+ return rec, err

+}

+

+// MoveState is the record's in-flight move (issue #331). State is one of

+// the Move* constants; To is the target placement code. ReadyAt is stamped

+// by the source box's preparing→ready promotion, so the tenant can see how

+// stale its freeze-window budget is.

+type MoveState struct {

+ To string `json:"to"`

+ State string `json:"state"`

+ RequestedAt time.Time `json:"requested_at"`

+ ReadyAt *time.Time `json:"ready_at,omitempty"`

+ // UpdatedAt is when the state last changed — stamped by SetRecordMove

+ // itself on every transition that leaves a move in place, so the

+ // console's flat move view can carry one freshness stamp without each

+ // transition re-implementing it.

+ UpdatedAt time.Time `json:"updated_at"`

+}

+

+// The move states, in order. There is no "done" state: the terminal

+// transition flips Region and CLEARS Move in one write — a record at rest

+// never says anything about moves.

+const (

+ MovePreparing = "preparing"

+ MoveReady = "ready"

+ MoveCutting = "cutting"

+ // MoveAborting is abort's in-flight half (review round 1): the console

+ // marks it, the SOURCE box deletes the staged copy from the target

+ // bucket and then clears Move — because a staged replica left behind by

+ // a bare clear would win a LATER landing on that box (staging-wins,

+ // backend_exec.go's promoteStagedReplica) with data from an abandoned

+ // move.

+ MoveAborting = "aborting"

+)

+

+// ErrMoveConflict is a transition refused because the record's current move

+// state does not admit it — a second prepare while one is in flight, a cut

+// before ready, an abort after the point of no return. The console maps it

+// to 409; the box treats it as "someone else already advanced this" and

+// re-reads.

+var ErrMoveConflict = errors.New("provision: move state conflict")

+

+// SetRecordMove applies one guarded move transition: the same narrow

+// read-CAS-write with one retry that SetRecordRegion uses (and for the same

+// reason — re-reading and re-applying a transition on a lost race changes

+// nothing about what it meant; an ILLEGAL transition after the re-read is

+// ErrMoveConflict, which is the answer, not a race). mut sees the freshly

+// read record and either mutates it or returns the error that names why it

+// can't.

+func SetRecordMove(ctx context.Context, s release.Store, acct, app, host string, mut func(*Record) error) (*Record, error) {

+ return updateRecord(ctx, s, acct, app, host, func(rec *Record) error {

+ if err := mut(rec); err != nil {

+ return err

+ }

+ if rec.Move != nil {

+ rec.Move.UpdatedAt = time.Now().UTC()

+ }

+ return nil

+ })

+}

+

+// The five transitions, each a mut for SetRecordMove. They are the ONLY

+// writers of Record.Move.

+

+// MoveRequest starts a move to region `to`. Refused when one is already in

+// flight — the caller aborts first or waits.

+func MoveRequest(to string, now time.Time) func(*Record) error {

+ return func(rec *Record) error {

+ if rec.Move != nil {

+ return fmt.Errorf("%w: a move to %s is already %s", ErrMoveConflict, rec.Move.To, rec.Move.State)

+ }

+ if rec.Region == to {

+ return fmt.Errorf("%w: the instance is already in %s", ErrMoveConflict, to)

+ }

+ rec.Move = &MoveState{To: to, State: MovePreparing, RequestedAt: now.UTC()}

+ return nil

+ }

+}

+

+// MovePromoteReady is the source box's preparing→ready promotion.

+func MovePromoteReady(now time.Time) func(*Record) error {

+ return func(rec *Record) error {

+ if rec.Move == nil || rec.Move.State != MovePreparing {

+ return fmt.Errorf("%w: ready is only reached from preparing", ErrMoveConflict)

+ }

+ t := now.UTC()

+ rec.Move.State = MoveReady

+ rec.Move.ReadyAt = &t

+ return nil

+ }

+}

+

+// MoveMarkCutting is the console's cut verb: legal only from ready — the

+// tenant freezes writes first, and a cut from preparing would cut over a

+// replica that was never caught up.

+func MoveMarkCutting() func(*Record) error {

+ return func(rec *Record) error {

+ if rec.Move == nil || rec.Move.State != MoveReady {

+ return fmt.Errorf("%w: cut is only legal from ready", ErrMoveConflict)

+ }

+ rec.Move.State = MoveCutting

+ return nil

+ }

+}

+

+// MoveAbort marks a move for abort — legal before the point of no return.

+// The clear itself is the source box's (MoveAbortComplete), after it has

+// deleted the staged copy.

+func MoveAbort() func(*Record) error {

+ return func(rec *Record) error {

+ if rec.Move == nil {

+ return fmt.Errorf("%w: no move in flight", ErrMoveConflict)

+ }

+ switch rec.Move.State {

+ case MoveCutting:

+ return fmt.Errorf("%w: past the point of no return — the cut completes forward", ErrMoveConflict)

+ case MoveAborting:

+ return fmt.Errorf("%w: already aborting", ErrMoveConflict)

+ }

+ rec.Move.State = MoveAborting

+ return nil

+ }

+}

+

+// MoveAbortComplete is the source box's half of an abort: staging is gone,

+// the move field clears.

+func MoveAbortComplete() func(*Record) error {

+ return func(rec *Record) error {

+ if rec.Move == nil || rec.Move.State != MoveAborting {

+ return fmt.Errorf("%w: abort-complete is only legal from aborting", ErrMoveConflict)

+ }

+ rec.Move = nil

+ return nil

+ }

+}

+

+// MoveComplete is the source's terminal act: Region flips to the target and

+// Move clears, in ONE write — every reader that knows nothing about moves

+// sees the old region right up until it sees the new one.

+func MoveComplete() func(*Record) error {

+ return func(rec *Record) error {

+ if rec.Move == nil || rec.Move.State != MoveCutting {

+ return fmt.Errorf("%w: complete is only legal from cutting", ErrMoveConflict)

+ }

+ rec.Region = rec.Move.To

+ rec.Move = nil

+ return nil

+ }

}

// DeleteRecord retires an instance's route authorization. Object deletion

diff --git a/internal/provision/provision_test.go b/internal/provision/provision_test.go

index c92266f39..37bf85390 100644

--- a/internal/provision/provision_test.go

+++ b/internal/provision/provision_test.go

@@ -886,3 +886,112 @@ func TestHealthWireTagIsProbeNotHealth(t *testing.T) {

t.Fatalf("marshal used the collided health key: %s", b)

}

+

+// The move transitions (issue #331): SetRecordMove is the one writer of

+// Record.Move, and each transition validates the state it leaves from. The

+// terminal MoveComplete flips Region and clears Move in ONE write.

+func TestSetRecordMoveTransitions(t *testing.T) {

+ ctx := context.Background()

+ s := release.NewDirStore(t.TempDir())

+ rec := Record{Schema: 1, Host: "a.go.example.com", Slug: "a", App: "app", Account: "bab",

+ Region: "ie", Channel: "stable", CreatedAt: time.Now().UTC()}

+ if err := CreateRecord(ctx, s, rec); err != nil {

+ t.Fatal(err)

+ }

+ mv := func(mut func(*Record) error) error {

+ _, err := SetRecordMove(ctx, s, "bab", "app", "a.go.example.com", mut)

+ return err

+ }

+ read := func() Record {

+ t.Helper()

+ got, err := GetRecord(ctx, s, "bab", "app", "a.go.example.com")

+ if err != nil {

+ t.Fatal(err)

+ }

+ return *got

+ }

+ now := time.Now()

+

+ if err := mv(MoveRequest("ie", now)); !errors.Is(err, ErrMoveConflict) {

+ t.Fatalf("move to the current region = %v, want ErrMoveConflict", err)

+ }

+ if err := mv(MoveMarkCutting()); !errors.Is(err, ErrMoveConflict) {

+ t.Fatalf("cut with no move in flight = %v, want ErrMoveConflict", err)

+ }

+ if err := mv(MoveRequest("uk", now)); err != nil {

+ t.Fatal(err)

+ }

+ if got := read(); got.Move == nil || got.Move.State != MovePreparing || got.Move.To != "uk" || got.Region != "ie" {

+ t.Fatalf("after request: %+v", got.Move)

+ }

+ if err := mv(MoveRequest("se", now)); !errors.Is(err, ErrMoveConflict) {

+ t.Fatalf("second request while in flight = %v, want ErrMoveConflict", err)

+ }

+ if err := mv(MoveMarkCutting()); !errors.Is(err, ErrMoveConflict) {

+ t.Fatalf("cut from preparing = %v, want ErrMoveConflict", err)

+ }

+ if err := mv(MovePromoteReady(now)); err != nil {

+ t.Fatal(err)

+ }

+ if got := read(); got.Move.State != MoveReady || got.Move.ReadyAt == nil {

+ t.Fatalf("after promote: %+v", got.Move)

+ }

+ if err := mv(MoveMarkCutting()); err != nil {

+ t.Fatal(err)

+ }

+ if err := mv(MoveAbort()); !errors.Is(err, ErrMoveConflict) {

+ t.Fatalf("abort from cutting = %v, want ErrMoveConflict (completes forward)", err)

+ }

+ if err := mv(MoveComplete()); err != nil {

+ t.Fatal(err)

+ }

+ if got := read(); got.Region != "uk" || got.Move != nil {

+ t.Fatalf("after complete: region=%q move=%+v, want uk/nil in one write", got.Region, got.Move)

+ }

+ // Abort is two-phase: ready → aborting (cut and promote both refuse

+ // there) → cleared only by MoveAbortComplete.

+ if err := mv(MoveRequest("ie", now)); err != nil {

+ t.Fatal(err)

+ }

+ if err := mv(MovePromoteReady(now)); err != nil {

+ t.Fatal(err)

+ }

+ if err := mv(MoveAbort()); err != nil {

+ t.Fatal(err)

+ }

+ if got := read(); got.Move == nil || got.Move.State != MoveAborting {

+ t.Fatalf("after abort: %+v, want aborting", got.Move)

+ }

+ if err := mv(MoveMarkCutting()); !errors.Is(err, ErrMoveConflict) {

+ t.Fatalf("cut from aborting = %v, want ErrMoveConflict", err)

+ }

+ if err := mv(MoveAbort()); !errors.Is(err, ErrMoveConflict) {

+ t.Fatalf("double abort = %v, want ErrMoveConflict", err)

+ }

+ if err := mv(MoveAbortComplete()); err != nil {

+ t.Fatal(err)

+ }

+ if got := read(); got.Move != nil || got.Region != "uk" {

+ t.Fatalf("after abort-complete: %+v region=%q", got.Move, got.Region)

+ }

+}

+

+// A record written before the move field existed round-trips byte-identical

+// through the CAS helper's read-marshal cycle — the wire-shape rule.

+func TestMoveFieldAbsentOnPreMoveRecords(t *testing.T) {

+ raw := []byte(`{"schema":1,"host":"a.go.example.com","slug":"a","app":"app","account":"bab","region":"ie","channel":"stable","created_at":"2026-08-01T00:00:00Z"}`)

+ var rec Record

+ if err := json.Unmarshal(raw, &rec); err != nil {

+ t.Fatal(err)

+ }

+ if rec.Move != nil {

+ t.Fatalf("pre-move record decoded a Move: %+v", rec.Move)

+ }

+ out, err := json.Marshal(rec)

+ if err != nil {

+ t.Fatal(err)

+ }

+ if strings.Contains(string(out), "move") {

+ t.Fatalf("re-marshal grew a move key: %s", out)

+ }

+}

diff --git a/internal/provision/reconcile.go b/internal/provision/reconcile.go

index f7706bb18..7dcaa45a8 100644

--- a/internal/provision/reconcile.go

+++ b/internal/provision/reconcile.go

@@ -12,6 +12,7 @@ package provision

import (

"context"

+ "crypto/ed25519"

"encoding/json"

"errors"

"fmt"

@@ -48,6 +49,17 @@ type ReconcileConfig struct {

// it would mean, because adoption is what grants the removal phase

// authority over that row.

DryRun bool

+ // ReleasePub verifies the production flag before a trash marker is

+ // allowed to retire an app's routes (#340), the same guard and the same

+ // key domains.Config carries for the same decision. nil leaves the

+ // retirement UNGUARDED rather than disabling it — domains §1c's posture,

+ // carried over deliberately so the two passes cannot reach opposite

+ // conclusions about the same app: a keyless deployment has no way to

+ // make the guard pass, and holding every trash forever there would break

+ // hiding entirely. Nothing destructive to the bucket happens on this

+ // path; the auto-purge that does still lives in domains, still refuses

+ // without a key.

+ ReleasePub ed25519.PublicKey

}

// acctApp identifies one app within one account — the unit both the record

@@ -55,6 +67,43 @@ type ReconcileConfig struct {

// only within an account, never globally.

type acctApp struct{ acct, app string }

+// trashState is what one app's trash marker means to this pass. Two fields

+// and not one bool, because the two questions have different safe answers:

+// see the trash cache in Reconcile for which is which.

+type trashState struct {

+ declineMint bool

+ removable bool

+}

+

+// productionClear reports whether a trashed app's routes may be retired —

+// i.e. whether we can positively see that it is NOT flagged production.

+// Dropping every route of a production app is an outage, which is precisely

+// what the flag exists to prevent, so an unreadable flag refuses in the same

+// direction as an ON one: the state an attacker with delete-but-not-sign can

+// manufacture is exactly the unreadable one.

+//

+// A nil key is the keyless box, which acts on the marker — domains §1c's

+// posture, and the comment on ReconcileConfig.ReleasePub says why.

+//

+// Both refusals log EVERY pass, deliberately: an operator who wrote that

+// marker on purpose has to be told why nothing happened, or they will assume

+// deletion is broken.

+func productionClear(ctx context.Context, s release.Store, pub ed25519.PublicKey, acct, app string) bool {

+ if pub == nil {

+ return true

+ }

+ flag, err := release.GetProduction(ctx, s, pub, acct, app)

+ switch {

+ case err != nil:

+ log.Printf("provision: cannot read the production flag for %s/%s — refusing to act on the trash marker: %v", acct, app, err)

+ return false

+ case flag != nil && flag.Production:

+ log.Printf("provision: trash marker on production app %s/%s — refusing to act on it; turn production off first", acct, app)

+ return false

+ }

+ return true

+}

+

// ---- serving classes -------------------------------------------------------

//

// Which boxes materialize a record. The multi-region ingress design's Part 3

@@ -667,19 +716,32 @@ func Reconcile(ctx context.Context, s release.Store, reg *registry.Registry, cfg

}

// --- trash cache, per (acct, app), lazily ---------------------------------

- // An app in the trash provisions nothing. domains.Reconcile drops a

- // trashed app's routes every pass, so minting (or seed-converging) here

- // would re-raise them next tick and the two loops would fight forever.

- // The record stays in recs, so the removal phase below never retires the

- // route either — dropping a trashed app's routes is domains.Reconcile's

- // call alone, with its production-flag guard; this pass only declines to

- // create. An unreadable or malformed marker declines minting too: not

- // minting is the reversible direction, and a bucket writer who could

- // plant a marker to stall minting can already retire the instance

- // outright by deleting its record.

- trashCache := map[acctApp]bool{}

+ // An app in the trash provisions nothing, and its existing routes are

+ // RETIRED — both halves are this pass's job (#340).

+ //

+ // Retirement used to be domains.Reconcile's alone, on the grounds that it

+ // held the production-flag guard. That deferral was a live bug: the

+ // domains pass runs only on the box holding the housekeeper lease

+ // (agent.go's domainsLoop wraps it in runHousekeeperDuties), and its

+ // registry write is local, so on a multi-box deployment every OTHER box

+ // kept a trashed app's route forever and went on serving it from its own

+ // edge. Single-box deployments were correct by accident. This pass runs

+ // on every box and already reads the marker, so the trigger belongs here

+ // — and the guard comes with it (cfg.ReleasePub). Both passes now remove

+ // and neither mints, so they agree rather than fight.

+ //

+ // One marker, two DIFFERENT answers, and conflating them is the trap:

+ //

+ // declineMint — true also when the marker is UNREADABLE. Not minting

+ // is the reversible direction, and a bucket writer who could plant

+ // garbage here to stall minting can already retire the instance

+ // outright by deleting its record.

+ // removable — true ONLY for a marker positively read and present,

+ // whose app is not flagged production. Removal is not reversible, so

+ // an unreadable marker (or an unreadable flag) must never reach it.

+ trashCache := map[acctApp]trashState{}

trashSeen := map[acctApp]bool{}

- trashedFor := func(acct, app string) bool {

+ trashFor := func(acct, app string) trashState {

k := acctApp{acct, app}

if trashSeen[k] {

return trashCache[k]

@@ -689,13 +751,14 @@ func Reconcile(ctx context.Context, s release.Store, reg *registry.Registry, cfg

switch {

case err != nil:

log.Printf("provision: trash marker for %s/%s unreadable: %v — not minting for this app", acct, app, err)

- trashCache[k] = true

+ trashCache[k] = trashState{declineMint: true}

case tomb != nil:

log.Printf("provision: %s/%s is in the trash — not minting instance routes for it", acct, app)

- trashCache[k] = true

+ trashCache[k] = trashState{declineMint: true, removable: productionClear(ctx, s, cfg.ReleasePub, acct, app)}

}

return trashCache[k]

}

+ trashedFor := func(acct, app string) bool { return trashFor(acct, app).declineMint }

// --- mint / converge / conflict -------------------------------------------

routes, err := reg.List()

@@ -835,13 +898,16 @@ func Reconcile(ctx context.Context, s release.Store, reg *registry.Registry, cfg

// Three conditions, each load-bearing: Provisioned (the flag — not the

// app, not the channel — grants removal authority; a hand-added route

// never has it), scope (this box's own region/pool jurisdiction — a

- // foreign box's provisioned route is not this pass's business), and

- // absence from the record set this pass just read (the bucket's own

- // decision to retire the instance, not a guess this pass made).

+ // foreign box's provisioned route is not this pass's business), and a

+ // retirement the BUCKET decided rather than one this pass guessed —

+ // either the record is gone, or the app carries a trash marker (#340,

+ // and see the trash cache above for why the marker's two answers are

+ // not the same answer).

//

// Scope is asked through the SAME inScope as the record scan above, with

// the class read off the row (everywhereRoute) because the record whose

- // class it was is exactly what is gone. Both halves must widen together:

+ // class it was may be exactly what is gone. Both halves must widen

+ // together:

// a far box that mints a static route under the widened mint gate and then

// judges it out of scope here would serve a retired site forever.

//

@@ -858,22 +924,36 @@ func Reconcile(ctx context.Context, s release.Store, reg *registry.Registry, cfg

if !rt.Provisioned || !cfg.inScope(rt.Region, rt.Pool, everywhereRoute(rt)) {

continue

}

+ // Two triggers, asked in this order because the first is the

+ // cheap one: the bucket retired the instance (its record is

+ // gone), or the bucket retired the whole APP (a trash marker,

+ // while the record stays put). Both name the bucket's own

+ // decision; neither is a guess this pass made.

+ //

+ // The trash lookup needs both halves of the app's identity off

+ // the row. A Provisioned row always carries them — routeFor

+ // copies them from the record — so a row missing either is a

+ // shape this pass did not mint, and not one it will retire.

+ why := "record gone"

if _, ok := recs[rt.Host]; ok {

- continue

+ if rt.Account == "" || rt.App == "" || !trashFor(rt.Account, rt.App).removable {

+ continue

+ }

+ why = "app in the trash"

}

if protect[rt.Host] {

log.Printf("provision: not removing %s: its record exists but could not be read cleanly this pass", rt.Host)

continue

}

if cfg.DryRun {

- log.Printf("provision: dry-run: would remove retired %s route %s (record gone)", rt.Kind, rt.Host)

+ log.Printf("provision: dry-run: would remove retired %s route %s (%s)", rt.Kind, rt.Host, why)

continue

}

if err := reg.Delete(rt.Host); err != nil {

log.Printf("provision: remove retired %s route %s: %v", rt.Kind, rt.Host, err)

continue

}

- log.Printf("provision: removed retired %s route %s (record gone)", rt.Kind, rt.Host)

+ log.Printf("provision: removed retired %s route %s (%s)", rt.Kind, rt.Host, why)

}

return nil

diff --git a/internal/provision/reconcile_test.go b/internal/provision/reconcile_test.go

index dc73f2e16..b23f39392 100644

--- a/internal/provision/reconcile_test.go

+++ b/internal/provision/reconcile_test.go

@@ -1,11 +1,17 @@

package provision

import (

+ "bytes"

"context"

+ "crypto/ed25519"

"encoding/json"

+ "errors"

"fmt"

+ "log"

+ "os"

"path/filepath"

"reflect"

+ "strings"

"testing"

"time"

@@ -85,6 +91,27 @@ func putReplace(t *testing.T, s release.Store, key string, body []byte) {

}

+// releaseKeys mints a throwaway release signing pair — the same helper

+// internal/domains/reconcile_test.go carries, for the same reason: the

+// production flag is signed, so a test that exercises the guard needs both

+// halves of a real key.

+func releaseKeys(t *testing.T) (ed25519.PrivateKey, ed25519.PublicKey) {

+ t.Helper()

+ seedB64, pubB64, err := release.GenerateKey()

+ if err != nil {

+ t.Fatal(err)

+ }

+ priv, err := release.ParseSigningKey(seedB64, "test")

+ if err != nil {

+ t.Fatal(err)

+ }

+ pub, err := release.ParsePublicKey(pubB64, "test")

+ if err != nil {

+ t.Fatal(err)

+ }

+ return priv, pub

+}

+

func mustJSON(t *testing.T, v any) []byte {

t.Helper()

b, err := json.Marshal(v)

@@ -1266,12 +1293,20 @@ func TestReconcileDeclinesTrashedApp(t *testing.T) {

mustGet(t, reg, "acme.go.tito.io")

}

-// Trash: declining to MINT is the whole refusal — a route that already

-// exists when the app is trashed is not retired by this pass. Its record is

-// still in the bucket (present in recs), so the removal phase's "record

-// gone" trigger never fires; dropping a trashed app's routes is

-// domains.Reconcile's call alone, behind its production-flag guard.

-func TestReconcileLeavesTrashedAppsExistingRoute(t *testing.T) {

+// Trash: a route whose app is in the trash is RETIRED, on every box that

+// serves it (#340). This is provision's job and not domains.Reconcile's,

+// even though domains removes the same routes for the same reason: the

+// domains pass runs only on the box holding the housekeeper lease, so on a

+// multi-box deployment every other box kept a trashed app's route forever

+// and went on serving it — publicly, from its own edge, with the app

+// deleted. Single-box deployments were correct by accident.

+//

+// The record stays in the bucket while an app is trashed, so the removal

+// phase's "record gone" trigger never fires for it. The trash marker is the

+// second trigger, asked under exactly the same three gates: Provisioned,

+// in scope, and nothing failed to read this pass.

+func TestReconcileRemovesTrashedAppsRoute(t *testing.T) {

+ _, pub := releaseKeys(t)

s, sqid := setup(t)

if err := CreateRecord(ctx, s, acmeRecord(sqid)); err != nil {

t.Fatal(err)

@@ -1285,13 +1320,219 @@ func TestReconcileLeavesTrashedAppsExistingRoute(t *testing.T) {

if err := release.TrashApp(ctx, s, sqid, "tito", "paul@keymail.dev", time.Now()); err != nil {

t.Fatal(err)

}

- cfg := ReconcileConfig{Region: "ie", Pools: []string{"tito"}, DataDir: "/data"}

+ cfg := ReconcileConfig{Region: "ie", Pools: []string{"tito"}, DataDir: "/data", ReleasePub: pub}

+ if err := Reconcile(ctx, s, reg, cfg); err != nil {

+ t.Fatal(err)

+ }

+ if rt, err := reg.Get("acme.go.tito.io"); err != nil || rt != nil {

+ t.Fatalf("a trashed app's route survived: %+v %v", rt, err)

+ }

+}

+

+// #340's own shape: the FAR box, the everywhere class, a static site. This

+// is the case that was live on flagship — three edges publishing a trashed

+// app's host in their edges files while the lease holder had already

+// dropped it, so even the lease holder served it again through the

+// deployment-wide host map.

+func TestReconcileRemovesTrashedEverywhereRouteOnAFarBox(t *testing.T) {

+ _, pub := releaseKeys(t)

+ s, sqid := setup(t)

+ rec := titoRecord(sqid, "site")

+ rec.Kind = "static"

+ if err := CreateRecord(ctx, s, rec); err != nil {

+ t.Fatal(err)

+ }

+ reg := regWith(t)

+ cfg := farBoxCfg()

+ cfg.ReleasePub = pub

+ if err := Reconcile(ctx, s, reg, cfg); err != nil {

+ t.Fatal(err)

+ }

+ mustGet(t, reg, "site.go.tito.io")

+

+ if err := release.TrashApp(ctx, s, sqid, "tito", "paul@keymail.dev", time.Now()); err != nil {

+ t.Fatal(err)

+ }

+ if err := Reconcile(ctx, s, reg, cfg); err != nil {

+ t.Fatal(err)

+ }

+ if rt, err := reg.Get("site.go.tito.io"); err != nil || rt != nil {

+ t.Errorf("a far box kept serving a trashed app's static route: %+v %v", rt, err)

+ }

+}

+

+// The two trash questions are NOT the same question. Declining to mint is

+// reversible and answers "true" to an unreadable marker on purpose;

+// removal is not reversible and must answer only to a marker positively

+// read and present. A pass that conflated them would let a corrupt object

+// — or a bucket writer with delete-but-not-sign — retire live routes.

+func TestReconcileKeepsTrashedAppsRouteWhenTheMarkerIsUnreadable(t *testing.T) {

+ _, pub := releaseKeys(t)

+ s, sqid := setup(t)

+ if err := CreateRecord(ctx, s, acmeRecord(sqid)); err != nil {

+ t.Fatal(err)

+ }

+ reg := regWith(t, registry.Route{

+ Host: "acme.go.tito.io", Slug: "acme", Kind: "instance",

+ Socket: "/run/carlos/acme.go.tito.io.sock", DBPath: "/data/acme.go.tito.io.db",

+ App: "tito", Account: sqid, Pool: "tito", Region: "ie", Channel: "stable",

+ Hibernate: true, Provisioned: true, Seed: `["seed"]`, Status: "ready",

+ })

+ if err := release.TrashApp(ctx, s, sqid, "tito", "paul@keymail.dev", time.Now()); err != nil {

+ t.Fatal(err)

+ }

+ putReplace(t, s, release.AppTombstoneKey(sqid, "tito"), []byte("{not json"))

+ cfg := ReconcileConfig{Region: "ie", Pools: []string{"tito"}, DataDir: "/data", ReleasePub: pub}

if err := Reconcile(ctx, s, reg, cfg); err != nil {

t.Fatal(err)

}

mustGet(t, reg, "acme.go.tito.io")

}

+// The production flag is why this removal lived in domains.Reconcile at

+// all: dropping every route of a production app is an outage, which is

+// exactly what the flag exists to prevent. Moving the trigger without the

+// guard would have been a regression, not a fix. Same refusal, same

+// wording, same pass-after-pass loudness as domains §1c.

+func TestReconcileNeverRemovesRoutesOfAProductionFlaggedTrashedApp(t *testing.T) {

+ priv, pub := releaseKeys(t)

+ s, sqid := setup(t)

+ if err := CreateRecord(ctx, s, acmeRecord(sqid)); err != nil {

+ t.Fatal(err)

+ }

+ reg := regWith(t, registry.Route{

+ Host: "acme.go.tito.io", Slug: "acme", Kind: "instance",

+ Socket: "/run/carlos/acme.go.tito.io.sock", DBPath: "/data/acme.go.tito.io.db",

+ App: "tito", Account: sqid, Pool: "tito", Region: "ie", Channel: "stable",

+ Hibernate: true, Provisioned: true, Seed: `["seed"]`, Status: "ready",

+ })

+ // The flag goes on FIRST: SetProduction refuses a trashed app.

+ if err := release.SetProduction(ctx, s, priv, sqid, "tito", true, "paul@keymail.dev"); err != nil {

+ t.Fatal(err)

+ }

+ if err := release.TrashApp(ctx, s, sqid, "tito", "eve@evil.example", time.Now()); err != nil {

+ t.Fatal(err)

+ }

+

+ var logged bytes.Buffer

+ log.SetOutput(&logged)

+ t.Cleanup(func() { log.SetOutput(os.Stderr) })

+

+ cfg := ReconcileConfig{Region: "ie", Pools: []string{"tito"}, DataDir: "/data", ReleasePub: pub}

+ for pass := 1; pass <= 2; pass++ {

+ if err := Reconcile(ctx, s, reg, cfg); err != nil {

+ t.Fatalf("pass %d: %v", pass, err)

+ }

+ mustGet(t, reg, "acme.go.tito.io")

+ }

+ if !strings.Contains(logged.String(), "trash marker on production app") {

+ t.Errorf("no refusal logged; log was:\n%s", logged.String())

+ }

+}

+

+// flagUnreadableStore breaks exactly one read — the app's production flag —

+// the way an attacker with delete-but-not-sign, or a half-broken bucket,

+// would. Everything else reads normally, so the trash marker is perfectly

+// valid and only the flag is in doubt.

+type flagUnreadableStore struct {

+ release.Store

+ key string

+}

+

+func (s flagUnreadableStore) Get(ctx context.Context, key string) ([]byte, string, error) {

+ if key == s.key {

+ return nil, "", errors.New("bucket refused the read")

+ }

+ return s.Store.Get(ctx, key)

+}

+

+// An UNREADABLE flag refuses in the same direction as an ON one, here as

+// everywhere else in this platform: the state an attacker can manufacture

+// is exactly the unreadable one.

+func TestReconcileDoesNotRemoveTrashedRoutesWhenTheProductionFlagIsUnreadable(t *testing.T) {

+ _, pub := releaseKeys(t)

+ dir, sqid := setup(t)

+ if err := CreateRecord(ctx, dir, acmeRecord(sqid)); err != nil {

+ t.Fatal(err)

+ }

+ reg := regWith(t, registry.Route{

+ Host: "acme.go.tito.io", Slug: "acme", Kind: "instance",

+ Socket: "/run/carlos/acme.go.tito.io.sock", DBPath: "/data/acme.go.tito.io.db",

+ App: "tito", Account: sqid, Pool: "tito", Region: "ie", Channel: "stable",

+ Hibernate: true, Provisioned: true, Seed: `["seed"]`, Status: "ready",

+ })

+ if err := release.TrashApp(ctx, dir, sqid, "tito", "paul@keymail.dev", time.Now()); err != nil {

+ t.Fatal(err)

+ }

+ s := flagUnreadableStore{Store: dir, key: release.ProductionKey(sqid, "tito")}

+

+ var logged bytes.Buffer

+ log.SetOutput(&logged)

+ t.Cleanup(func() { log.SetOutput(os.Stderr) })

+

+ cfg := ReconcileConfig{Region: "ie", Pools: []string{"tito"}, DataDir: "/data", ReleasePub: pub}

+ if err := Reconcile(ctx, s, reg, cfg); err != nil {

+ t.Fatal(err)

+ }

+ mustGet(t, reg, "acme.go.tito.io")

+ if !strings.Contains(logged.String(), "cannot read the production flag") {

+ t.Errorf("no refusal logged; log was:\n%s", logged.String())

+ }

+}

+

+// A KEYLESS box acts on the marker, deliberately — domains §1c's posture,

+// carried over verbatim so the two passes cannot disagree about the same

+// app. It cannot verify a flag, and the alternative — hold every trash

+// forever wherever there is no pubkey — would break hiding entirely on

+// legitimately keyless deployments, whose owners have no way to make the

+// guard pass.

+func TestReconcileRemovesTrashedAppsRouteOnAKeylessBox(t *testing.T) {

+ s, sqid := setup(t)

+ if err := CreateRecord(ctx, s, acmeRecord(sqid)); err != nil {

+ t.Fatal(err)

+ }

+ reg := regWith(t, registry.Route{

+ Host: "acme.go.tito.io", Slug: "acme", Kind: "instance",

+ Socket: "/run/carlos/acme.go.tito.io.sock", DBPath: "/data/acme.go.tito.io.db",

+ App: "tito", Account: sqid, Pool: "tito", Region: "ie", Channel: "stable",

+ Hibernate: true, Provisioned: true, Seed: `["seed"]`, Status: "ready",

+ })

+ if err := release.TrashApp(ctx, s, sqid, "tito", "paul@keymail.dev", time.Now()); err != nil {

+ t.Fatal(err)

+ }

+ cfg := ReconcileConfig{Region: "ie", Pools: []string{"tito"}, DataDir: "/data"} // no ReleasePub

+ if err := Reconcile(ctx, s, reg, cfg); err != nil {

+ t.Fatal(err)

+ }

+ if rt, err := reg.Get("acme.go.tito.io"); err != nil || rt != nil {

+ t.Fatalf("a keyless box kept a trashed app's route: %+v %v", rt, err)

+ }

+}

+

+// Provisioned is the flag that grants removal authority, and the trash

+// trigger does not widen it: a hand-added route this pass never adopted is

+// not this pass's to retire, whatever the app's marker says.

+func TestReconcileNeverRemovesAHandAddedRouteOfATrashedApp(t *testing.T) {

+ _, pub := releaseKeys(t)

+ s, sqid := setup(t)

+ if err := CreateRecord(ctx, s, acmeRecord(sqid)); err != nil {

+ t.Fatal(err)

+ }

+ reg := regWith(t, registry.Route{

+ Host: "hand.go.tito.io", Slug: "hand", Kind: "instance",

+ App: "tito", Account: sqid, Pool: "tito", Region: "ie", Channel: "stable",

+ Status: "ready", // no Provisioned

+ })

+ if err := release.TrashApp(ctx, s, sqid, "tito", "paul@keymail.dev", time.Now()); err != nil {

+ t.Fatal(err)

+ }

+ cfg := ReconcileConfig{Region: "ie", Pools: []string{"tito"}, DataDir: "/data", ReleasePub: pub}

+ if err := Reconcile(ctx, s, reg, cfg); err != nil {

+ t.Fatal(err)

+ }

+ mustGet(t, reg, "hand.go.tito.io")

+}

+

// Trash: an unreadable or malformed marker also declines minting — not

// minting is the reversible direction, and a writer who could plant garbage

// here to stall minting can already retire the instance outright by

diff --git a/internal/registry/registry.go b/internal/registry/registry.go

index 0c9a5fafa..130626a37 100644

--- a/internal/registry/registry.go

+++ b/internal/registry/registry.go

@@ -832,6 +832,27 @@ func (r *Registry) SetChannel(host, channel string) error {

return nil

}

+// SetStatus writes a route's lifecycle status word. Same contract as

+// SetChannel: a narrow UPDATE that cannot clobber concurrent column writes,

+// and an error when no row matched. The residence-move cut (internal/host's

+// move worker) is its first caller — status "moving" takes a route out of

+// the edge's served set and the activator's managed set in one write, which

+// is what quiesces the child ahead of the final mirror round.

+func (r *Registry) SetStatus(host, status string) error {

+ res, err := r.sql.Exec(`UPDATE routes SET status = ? WHERE host = ?`, status, host)

+ if err != nil {

+ return err

+ }

+ n, err := res.RowsAffected()

+ if err != nil {

+ return err

+ }

+ if n == 0 {

+ return fmt.Errorf("no route for host %q", host)

+ }

+ return nil

+}

+

// SetCaps writes a route's operator-granted capability tokens — the ONLY

// writer of that column, and deliberately the only one: `carlos route

// -grant/-revoke` is the operator's door to a transfer of box authority, and

carlos / platform Public

Merge cert-warm-outcomes (which carries origin/main) into domains-settings-state