amadan
docs: instance moves for every app — design record
The platform runs the whole move; an app declares freeze/thaw/verify hooks and which stores follow. Widens #331 to every app, closes the wake-before- freeze gap behind Tito's failed move, and names adoption, the edge-local door for app routers, retention and stores as slices.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+605
−0
docs/superpowers/specs/2026-09-24-instance-moves-for-every-app-design.md+605 −0
diff --git a/docs/superpowers/specs/2026-09-24-instance-moves-for-every-app-design.md b/docs/superpowers/specs/2026-09-24-instance-moves-for-every-app-design.md| new file mode 100644 |
| index 000000000..86cab28e3 |
| --- /dev/null |
| +++ b/docs/superpowers/specs/2026-09-24-instance-moves-for-every-app-design.md |
| @@ -0,0 +1,605 @@ |
| +# Instance moves for every app: the platform runs the move, the app answers three calls |
| + |
| +**Date:** 2026-09-24 |
| +**Status:** DRAFT for review. Filed as a request on amadan `carlos/platform` |
| +for a CARLOS session to own and build; nothing here is built. |
| +**Builds on:** `2026-08-28-instance-move-design.md` (#331, the two-phase, |
| +data-following move: shipped, and every word of its never-two-writers |
| +argument stands). This record widens who can use it and what follows the |
| +instance; it does not reopen the cut ordering. |
| +**First adopter:** Tito Go. Its half is |
| +`teamtito/titogo` `docs/superpowers/specs/2026-09-24-moves-on-the-platform-design.md` |
| +(branch `docs-controld-onto-platform-design`). |
| + |
| +## 1. The problem, as found |
| + |
| +Paul, 2026-09-24: *"This is something that anyone using the platform should |
| +be able to offer with their apps."* |
| + |
| +#331 shipped the data half of a move for one class of route and left the rest |
| +to the tenant: the tenant freezes writes over its own channel, copies its own |
| +uploads, and tells its own router where the instance went (#331 spec, |
| +"Decisions taken in conversation"). Only one tenant has built that half. Its |
| +two production attempts are the whole record, and both failed |
| +(Tito's router registry, `/data/titogo/registry.db`, table `moves`, read |
| +2026-09-24): |
| + |
| +| row | host | to | when (UTC) | outcome | |
| +|---|---|---|---|---| |
| +| 1 | `move-canary.go.tito.io` | se | 2026-08-30 16:19 | aborted: `could not freeze: Post "http://titogo/__internal/move/freeze": dial unix /run/carlos/move-canary.go.tito.io.sock: connect: no such file or directory` | |
| +| 2 | `techarena.go.tito.io` | se | 2026-08-31 11:14 | failed: `instanceapi: POST 404: 404 page not found` | |
| + |
| +**Row 1 is a platform gap wearing a tenant's error.** move-canary was a |
| +platform instance, asleep. The tenant's freeze is a raw dial of the instance |
| +socket (titogo `internal/carlos/move_order.go:30-36`), and a sleeping |
| +instance has no socket: the activator removes it on hibernation. The only |
| +things that wake an instance are an edge request and a scheduled tick |
| +(`internal/activator/activator.go:2777-2779`, `ServeScheduled`). The move |
| +API gives the tenant neither, so a tenant that wants to freeze a sleeping |
| +instance has to send it a public HTTP request first and hope. Tito's router |
| +already does exactly that for ordinary traffic (titogo |
| +`internal/carlos/wake.go:10-19`, a GET to `https://<host>/` "rides the same |
| +path a buyer's browser does"). That is a workaround for a missing verb. |
| + |
| +**Row 2 is the class rail doing its job.** techarena ran under Tito's own |
| +supervisor (`titogo@.service`), fronted by the platform as an addr route to |
| +Tito's router. There is no platform record for it, so the move door's |
| +byte-identical 404 answered (`internal/console/apiinstancemove.go:43-46`). |
| +That is correct: the platform cannot move state it does not run. But no door |
| +lets an app bring such an instance under a record, so on carlos-tito-ie |
| +**71 of the app's hosts are addr rows pointing at `127.0.0.1:8444` |
| +(`provisioned=0`) and 13 are platform socket routes** (box registry, |
| +2026-09-24). Most of the app's customers cannot move at all. |
| + |
| +And where a tenant does get past both, three more things it has to build |
| +itself: |
| + |
| +1. **Its router.** An app that fronts many instances behind its own router |
| + (Tito's root domain, `go.tito.io/<slug>/…`, is served by Tito's router, |
| + which dials each instance socket directly) finds the socket gone after a |
| + cut. The platform's edge follows the move. The app's router does not. |
| + Tito carries a `routes.home` column for this and nothing writes it |
| + (titogo `internal/carlos/registry.go:539-541`, "a future narrow setter |
| + for home"; `move_sweep.go:648-664`, `finishMove` records `done` and |
| + logs `served_by`, and never re-homes the row). |
| +2. **Its files.** The platform moves "the instance DB and record only" |
| + (#331 spec). Anything in a store stays where it was. |
| +3. **Knowing it finished.** The record's `move` clears in the flip |
| + (`internal/provision/provision.go:1036-1044`), so a finished move and a |
| + never-started one look the same on the move GET (`{"state":"none"}`, |
| + `apiinstancemove.go:88-91`). Tito infers "done" by re-reading the |
| + record's region (titogo `move_sweep.go:369-445`). |
| + |
| +Every app that wants to offer moves would rebuild those, and the one that |
| +did never completed a move. This record moves them into the platform. |
| + |
| +## 2. The shape, in one paragraph |
| + |
| +A move is a platform verb on one instance: `carlos instances move <host> --to |
| +us`, or the same three console doors #331 already serves. The app declares |
| +once, on its instances manifest, which HTTP paths on the instance answer |
| +**freeze**, **thaw** and **verify**, and which of its stores follow the |
| +instance. From then on the platform drives the whole move: it copies the |
| +replica and the declared stores in delta rounds while the instance keeps |
| +serving, **wakes the instance and calls freeze** when the copy has caught |
| +up, cuts (stop, final checkpoint, final delta, purge, flip), lands the |
| +instance on the target box, wakes it there and calls verify, holds the host |
| +at `503 + Retry-After` at every edge for the whole window, and keeps a move |
| +log with a terminal state. The source region's copy is retained for a fence |
| +and then deleted. An app with its own router dials instances through a |
| +box-local edge door and stops caring which box an instance lives on. |
| + |
| +## 3. Rulings asked for |
| + |
| +These are the decisions this record needs before a plan is written. Each is |
| +stated as the proposal; §14 lists the ones still genuinely open. |
| + |
| +- **(proposed) The platform owns the freeze.** An app that declares hooks |
| + never sends its own freeze. An app that declares none gets today's #331 |
| + behaviour exactly: the cut stops the child without warning, and the edge |
| + holds the host at 503 until the target serves (§6.4). No hook is ever |
| + required to move safely: consistency comes from stop-then-checkpoint, not |
| + from the app. The freeze buys a graceful window, not correctness. |
| +- **(proposed) Hooks ride the tick's trust path.** Delivered by the agent |
| + over the instance socket through a waking entry point, authenticated with |
| + the instance admin token the scheduled-tick runner already mints and sends |
| + (`internal/host/schedule_wiring.go:147-182`). No new credential, no new |
| + trust path. |
| +- **(proposed) Stores follow only when declared,** and only stores the |
| + platform provisioned. A bucket the app brought itself is the app's |
| + (§7.3). |
| +- **(proposed) Rollback after the cut is a move back,** never an automatic |
| + undo. Which means moves to and from the home region must work (§6.6). |
| +- **(proposed) An app's own router stops routing by location.** The platform |
| + gives it one box-local door that reaches any instance wherever it lives |
| + (§8). A tenant-side `home` column is the thing this retires. |
| + |
| +## 4. Non-goals |
| + |
| +- Fleet placements (`fleet/<name>`), pooled records and non-hibernating |
| + instances stay refused (`apiinstancemove.go:61-72`, the #331 rails). |
| +- Moving addr-backed or unit-backed routes as such. An addr-backed instance |
| + first becomes a platform instance (§9); then it moves like any other. |
| +- `replica_regions` and the replica role (#330, discussion #162): a move |
| + moves the serving home. What happens to replicas is §13. |
| +- Automatic scheduling ("move at 03:00"). The door takes a request now; the |
| + app schedules if it wants to. Tito already does. |
| +- Region-aware billing and unit economics. |
| + |
| +## 5. What an app declares |
| + |
| +One additive, omitempty object on the instances manifest |
| +(`provision.Manifest`, `internal/provision/provision.go:226-241`), written by |
| +`carlos instances enable --app a --moves …`, a signed-in person's act, gated |
| +exactly as `--private-storage` is (the 2026-09-22 private-storage spec, |
| +§3). An `instance`-role credential can request a move (it already can, |
| +`internal/console/authcli.go:625-628`) but can never change what a move |
| +does. |
| + |
| +```json |
| +"moves": { |
| + "hooks": { |
| + "freeze": "/.well-known/carlos/move/freeze", |
| + "thaw": "/.well-known/carlos/move/thaw", |
| + "verify": "/.well-known/carlos/move/verify" |
| + }, |
| + "freeze_budget_seconds": 600, |
| + "stores": ["private"], |
| + "store_prefixes": { "uploads": "{host}/" } |
| +} |
| +``` |
| + |
| +- **`hooks`**: any subset. Paths are validated with the schedule |
| + declaration's path rules (`internal/schedule/declaration.go:38-60`: must |
| + start with `/`, no scheme or host, no `..`, no whitespace, ≤256 bytes). |
| + Absent = no hook of that kind. |
| +- **`freeze_budget_seconds`**: how long the app will hold writes. The |
| + platform passes it as the freeze deadline and never extends a freeze past |
| + it (§6.3). Default 600, cap 1800. |
| +- **`stores`**: which of the instance's bound stores follow it. `"private"` |
| + names the per-instance environment store (2026-09-22 spec §5). |
| +- **`store_prefixes`**: for an app-level store shared by every instance, the |
| + key prefix that belongs to one instance, as a template over `{host}` and |
| + `{slug}`. Only those keys follow. §7.2. |
| + |
| +A manifest without `moves` gets #331 exactly as it is today. |
| + |
| +## 6. The lifecycle |
| + |
| +### 6.1 States |
| + |
| +The record keeps #331's in-flight object (`provision.MoveState`, |
| +`provision.go:903-940`) and its narrow CAS (`SetRecordMove`, |
| +`provision.go:949`). States grow from four to seven; every new one is |
| +additive, and a record written by today's console still parses. |
| + |
| +``` |
| +preparing ──► ready ──► freezing ──► cutting ──► landing ──► (record at rest, log: done) |
| + │ │ │ |
| + └───────────┴───────────┴──► aborting ──► (record at rest, log: aborted) |
| +``` |
| + |
| +| state | who writes it | what is happening | legal exits | |
| +|---|---|---|---| |
| +| `preparing` | console (request) | source mirrors replica and declared stores into target staging, delta rounds | `ready`, `aborting` | |
| +| `ready` | source box | a round found zero differences; rounds keep running | `freezing` (cut requested), `aborting` | |
| +| `freezing` | console (cut) | source wakes the instance and calls freeze; tenant-held apps skip straight through | `cutting`, `aborting` | |
| +| `cutting` | source box | #331's cut, unchanged in order, plus the store's final delta | `landing` (the flip) | |
| +| `landing` | source box (in the flip write) | record's `region` is now the target; target mints, restores, wakes, verifies | record at rest | |
| +| `aborting` | console | source deletes staging, calls thaw, clears | record at rest | |
| + |
| +`freezing` is the new point before which abort is clean. `cutting` stays |
| +the point of no return (`provision.go:1013-1015`). |
| + |
| +**`landing` needs one change to #331's flip.** Today the flip sets `region` |
| +and clears `move` in one write (`provision.go:1036-1044`). Here the flip |
| +sets `region` and writes `move.state = "landing"` in that same single |
| +write, and the TARGET box clears `move` once verify has answered (or its |
| +budget has run out, §6.5). Region still flips in exactly one write, and the |
| +source still has nothing to do after it. What changes is that the record |
| +stays visibly in flight until the new home has proved it serves. |
| + |
| +### 6.2 The move log |
| + |
| +Every transition appends to |
| +`moves/<account>/<app>/<host>/<move-id>.json` in the deployment bucket: an |
| +object per move with every state, its timestamp, the box that wrote it, |
| +bytes and object counts copied, the freeze and cut durations, and the |
| +terminal outcome (`done`, `aborted`, `failed`) with its reason. The record |
| +says what is in flight; the log says what happened. `GET …/move` answers the |
| +in-flight state when there is one and otherwise the newest log's terminal |
| +state for 30 days: |
| + |
| +```json |
| +{"state":"done","target":"us","from":"ie","move_id":"…","updated_at":1790000000, |
| + "timings":{"prepare_s":95,"freeze_s":41,"cut_s":38,"land_s":22}} |
| +``` |
| + |
| +`"none"` then means what it says: no move this month. The flat shape |
| +titogo's client already parses (`apiinstancemove.go:76-86`) survives; the |
| +new fields are additive. |
| + |
| +### 6.3 Freeze: wake first, then ask |
| + |
| +When the console writes `freezing`, the source box's move worker: |
| + |
| +1. **Wakes the instance** through a new activator entry point, |
| + `Service.ServeHook`, a sibling of `ServeScheduled` (`activator.go:2777`) |
| + with the same shape: it wakes a sleeping instance, holds the in-flight |
| + count for the call, and does not stamp `lastActive`. This is the fix for |
| + row 1: the platform, which is the only thing that can wake, wakes. |
| +2. **POSTs the freeze hook** over the instance socket, with: |
| + - `Authorization: Bearer <instance admin token>`, minted and read exactly |
| + as the tick runner does (`schedule_wiring.go:161-182`, |
| + `EnsureAdminToken` then read); |
| + - `X-Carlos-Move-Id`, `X-Carlos-Move-To` (placement code), |
| + `X-Carlos-Move-Deadline` (RFC 3339, now + `freeze_budget_seconds`); |
| + - an empty body. |
| +3. **Reads the answer.** `200` = frozen: the app now refuses writes and will |
| + hold until the deadline. `409` = the app refuses to move now (a sale is |
| + closing, a job is mid-flight), with a sentence in the body; the move goes |
| + to `aborting` with that sentence as the reason. Anything else, or no |
| + answer within 30 s, is retried twice and then aborts. |
| +4. Writes `cutting`. |
| + |
| +The app's contract for a freeze is small and is the whole of what it has to |
| +build: |
| + |
| +- refuse mutating requests with `503 + Retry-After` until thawed or the |
| + deadline passes; |
| +- **clear the freeze itself at the deadline**, whatever else happens. That is |
| + the app's safety valve and the platform relies on it: a dead agent can |
| + never leave an app frozen past its own budget; |
| +- on boot, if `CARLOS_REGION` equals the frozen target, clear the freeze. |
| + The process woke in the target, so the move happened |
| + (`internal/activator/backend_exec.go:989-993` already sets `CARLOS_REGION` |
| + per child). |
| + |
| +Tito's instance already implements exactly this contract |
| +(titogo `internal/instance/move_freeze.go:119-132` boot-clear, |
| +`:154-181` freeze with a clamped 10-minute deadline). It changes its path |
| +and its authentication, nothing else. |
| + |
| +**No freeze hook declared:** `freezing` is written and immediately advanced |
| +to `cutting` by the same pass. The instance learns about the move when it is |
| +stopped. |
| + |
| +### 6.4 The window nobody serves, closed at the edge |
| + |
| +Today the cut sets the route's status to `moving` (`movespass.go:278`), the |
| +edge skips every route that is not `ready` (`internal/edge/edge.go:576-578`), |
| +the source's edges file stops naming the host, and until the target mints, |
| +**every edge answers `404 unknown host`** (`edge.go:1383-1387` then |
| +`:1999`). A buyer mid-checkout gets a 404. A webhook sender gets a 404, and |
| +some senders treat 4xx as permanent. |
| + |
| +The change: a route in status `moving` stays in the source edge's table as a |
| +**holding route** that answers every request `503 + Retry-After: 5`, marked |
| +as the edge's own answer (`edge.MarkAnswered`, as the sidecar refusal does, |
| +`activator.go:2820-2825`). The source's edges file keeps publishing the host |
| +with a new role, `moving`. Every other edge backhauls to the source and gets |
| +the same 503. The fold ranks `home` above `moving` (`homemap.go` |
| +`foldHomes`), so the moment the target's file names the host as `home`, |
| +every edge dials the target, and the source retires its holding route on |
| +its next pass. Nothing ever sees a 404 for a host that exists. |
| + |
| +### 6.5 Landing and verify |
| + |
| +The target box mints the route when the record's region names it (#331, |
| +unchanged), restores from staging through `promoteStagedReplica` |
| +(`backend_exec.go:1569-1600`), and then its move worker (new: the target has |
| +never run one) sees `move.state == "landing"` on a route it owns and: |
| + |
| +1. waits for the stores' env redelivery to have reached the environment |
| + bundle (§7.1), so the instance wakes into the right coordinates; |
| +2. wakes the instance through `ServeHook` and POSTs `verify` with the same |
| + headers as freeze. The app checks whatever it cares about: its region, |
| + its schema, that its files are reachable. It answers `200` or `500` with |
| + a sentence; |
| +3. clears `move` (CAS, `landing` → at rest) and writes `done`, or `done` |
| + with `verify_failed` and the sentence, to the log. |
| + |
| +**A failed verify does not roll anything back.** The data is already in the |
| +target and the source copy is intact (§6.7). It pages (§11), shows on the |
| +console and in `carlos instances move --status`, and a person decides: |
| +leave it, fix forward, or move back. A landing that has not verified within |
| +15 minutes is logged the same way. The target never blocks traffic on |
| +verify: the edge serves the new home as soon as it is `ready`. |
| + |
| +**No verify hook declared:** landing clears on the target's first |
| +successful wake. |
| + |
| +### 6.6 Abort, rollback, and the home region |
| + |
| +- **Abort** is legal in `preparing`, `ready` and `freezing`. It is #331's |
| + two-phase abort (`provision.go:1005-1034`, `movespass.go:257-273`) with |
| + one addition: if the move got as far as `freezing`, the source calls |
| + **thaw** before clearing. A thaw that fails is logged, not retried |
| + forever: the app's own deadline clears the freeze. |
| +- **After `cutting`, the move completes forward.** Rollback is a second move |
| + in the other direction, started by a person. |
| +- **That means a move back to the home region has to work.** It does not |
| + today: `apiinstancemove.go:196-202` refuses a move to home because the home |
| + box publishes no staging lane (its edges file carries no `data_bucket`; |
| + `moveTargets` drops it, `movespass.go:206-223`). The home box publishes |
| + its data bucket like every other box, and the region-host module grants |
| + every sibling push-only `moves/*` on it. For Tito, `region-ie.tf` already |
| + lists the other five as staging targets; the five siblings each need `ie` |
| + added. Until this ships, **a move out of the home region is one-way**, and |
| + the door should say so in its 202. |
| + |
| +### 6.7 What the source keeps, and for how long |
| + |
| +Nothing deletes the source's replica prefix after a move today: `movespass.go` |
| +deletes only staging (`:265`), and `lease.DeletePrefix` has no other caller |
| +(`internal/lease/mirror.go:113`). So a moved instance's full history stays |
| +in the region it left, indefinitely. That breaks the promise a move is |
| +bought for. |
| + |
| +The source box keeps `instances/<host>/` for a **retention fence**, default |
| +7 days from `done`. It is the rollback evidence and the only copy if the |
| +target's bucket fails. After the fence it deletes the prefix and the |
| +declared stores' source copies, and logs the deletion with a count of |
| +objects removed. Self-healing, never silent: if the deletion fails it |
| +retries on the next pass and logs once a minute |
| +(`movespass.go:367-376`'s throttle). A second move of the same host before |
| +the fence resets it. `carlos instances move --status` shows "source copy in |
| +ie until 2026-10-01". |
| + |
| +## 7. Stores follow the instance |
| + |
| +### 7.1 A private store |
| + |
| +A store today is deployment-wide in one region: `CARLOS_STORE_REGION` is |
| +read once by the console (`2026-09-02-object-storage-self-serve-design.md` |
| +§1, `cmd/console/main.go:492-524`). So the first piece is **a store backend |
| +per placement**: each armed region publishes its store region and bucket |
| +prefix (the edges file again, beside `data_bucket`), and a new store is |
| +created in its instance's region. |
| + |
| +When `stores` names `"private"`, the move: |
| + |
| +1. **preparing**: provisions a twin of the instance's environment store in |
| + the target region (same entitlement, same walk, a new bucket and |
| + credentials, `internal/objstore/provisioner.go`), and mirrors objects |
| + into it in the same delta rounds as the replica. `ready` requires both |
| + mirrors to be caught up. |
| +2. **cutting**: after the child is down, one final store delta. The child |
| + was the only writer, and a stopped child cannot write, so this delta is |
| + final. |
| +3. **the flip**: in the same pass, rewrites the environment's store |
| + coordinates in its config bundle to the twin's |
| + (`appconfig.ApplyEnvironment`, per the 2026-09-19 spec §4), sealed as |
| + today. The target wakes the instance only after that delivery has landed |
| + (§6.5 step 1). |
| +4. **after the fence**: deletes the source store (§6.7). |
| + |
| +### 7.2 A shared app store with a per-instance prefix |
| + |
| +When `store_prefixes` names an app-level store, only the keys under the |
| +instance's prefix follow, into the same app's store twin in the target |
| +region. The twin is provisioned on first use and shared by every instance of |
| +the app homed there. The env the instance receives changes bucket and |
| +region, not key layout. The app must read its store coordinates from env on |
| +every boot, not cache them. |
| + |
| +This only works for an app whose keys are prefix-shaped. An app that keys a |
| +shared bucket by random token and records ownership only in each instance's |
| +database (Tito: `assets/<token>`, titogo `internal/instance/assets.go:132`; |
| +the live bucket lists exactly that) has no prefix to declare. It moves to a |
| +private store first (§7.1), and that is a one-time copy the app does, not |
| +the platform. |
| + |
| +### 7.3 A bucket the app brought itself |
| + |
| +Not the platform's to copy: it holds no credentials for it and cannot know |
| +which keys are whose. The move door's 202 body lists |
| +`"not_moved": ["app-managed buckets are not moved"]` whenever the app's |
| +manifest has no `stores` declaration, so nobody reads a finished move as a |
| +complete one. Tito's uploads bucket (`titogo-uploads-665106592142`, |
| +eu-west-1, set through `TITO_S3_*`) is this case today; the Tito record |
| +covers how it leaves it. |
| + |
| +## 8. Apps with their own router |
| + |
| +Some apps front many instances behind a router of their own. Tito's root |
| +domain, `go.tito.io/<slug>/…`, is served by Tito's router, because that is |
| +where Tito's admin credentials live and where it signs identity headers for |
| +the instance. For those requests the router dials each instance socket |
| +itself: `/run/carlos/<host>.sock` for platform instances, which it reads off |
| +the platform's registry row. Two things break that on a move: the socket |
| +exists only on one box, and only while the instance is awake. |
| + |
| +The platform already solves both for its own traffic. The edge wakes, and |
| +the edge backhauls to whichever box homes a host (`homemap.go:611`, |
| +`Home`; `edge.go:1379-1387`). So the proposal is to **let an app's router use |
| +the edge instead of the socket**: |
| + |
| +- **The door.** A box-local unix socket, `/run/carlos/edge-local.sock`, owned |
| + by the edge, mode 0660, group = a per-app group the app's router unit |
| + joins (`SupplementaryGroups=`). A request on it carries `Host: <instance |
| + host>` and is handled exactly as if it had arrived from the internet at |
| + this edge: local socket if this box homes the host (with the wake), signed |
| + backhaul if another box does, holding 503 if it is moving. It is refused |
| + for any host not owned by the dialling app (the edge checks the host's |
| + record's account and app against the group's). |
| +- **What it is not.** It is not a bypass. The password gate, the |
| + hook-path refusal (§10) and the edge's own XFF rules apply as on the |
| + public door. Headers the app sets (Tito's signed `X-Tito-Auth-*`) pass |
| + through untouched, as on any request. The one difference: the edge treats |
| + the router's XFF as a chain to carry, the way it treats a verified hop |
| + (`homemap.go:44-55`), because the caller is a process on the box that |
| + proved group membership, not a stranger stating its own address. |
| +- **What it retires.** The app no longer needs a `home` column, a hairpin of |
| + its own, a wake nudge through the public internet, or a per-region |
| + router to reach a moved instance. Tito's hairpin (titogo PR #1906, "the hairpin: a remote home is an HTTPS hop, not a wall") was |
| + built to cross regions over public HTTPS. It becomes one dial. |
| + |
| +The alternative, for apps that would rather keep their own routing: the |
| +instance GET already carries `served_by` (`internal/console/apiinstances.go:174-181`) |
| +and, with §6.2, a move's terminal state. An app can poll that and re-point |
| +its routes itself. That is the #331 contract with better signals, and it |
| +stays available. It is not what we recommend, because every such app would |
| +rebuild the same state machine. |
| + |
| +## 9. Bringing an instance under a record (adoption) |
| + |
| +An app that started by running instances itself (Tito's controld, 71 hosts) |
| +cannot move any of them until each one has a platform record. #126 blocked |
| +that for good reason: exec children shared one uid and one flat `/data`. |
| +The fix is merged and rolled: the isolated-exec series |
| +(`2026-09-11-isolated-exec-design.md`, merge `c1d40eeb`, 2026-09-12; |
| +the final fix round `34beb1b3` is in `v0.20.0` and `v0.21.0`). carlos-tito-ie |
| +runs `carlos v0.21.0`. **But the layout has not been migrated on that box**: |
| +`/data/layout/` does not exist and `/data/instances/` holds only `_probe` |
| +(read 2026-09-24), so its 13 platform instances still run flat, as one uid. |
| +Step 4 of that design's rollout (§8.6, "carlos-tito-ie: Tito's operator |
| +rolls it … then migrates") is the gate. It is an operator act on a built |
| +feature, not new code. |
| + |
| +After that, adoption needs one new door: |
| + |
| +- `POST …/instances` with `"import": {"sha256": "…", "bytes": N}` answers |
| + a one-time presigned PUT URL for `imports/<host>/snapshot.db` in the |
| + target region's data bucket (15-minute expiry, exact length, exact |
| + checksum). The app uploads a consistent snapshot (`VACUUM INTO`, taken |
| + while its own supervisor has the instance frozen). |
| +- The record is written with `import: pending`. The activator's `Restore` |
| + gains one more fallback beside staging (`backend_exec.go:290`): no own |
| + replica, no staging, an `imports/<host>/snapshot.db` → verify its checksum, |
| + `CheckDB`, place it at the row's DB path, start replicating from |
| + generation one, delete the import. |
| +- **The host already has a route**, the app's own addr row. Reconcile |
| + today adopts an unprovisioned row only when its shape matches the record, |
| + and otherwise logs "exists but is not the route this record describes" |
| + and leaves it alone (`internal/provision/reconcile.go:1022-1034`). That is |
| + the right default and adoption must not weaken it. Instead, an |
| + `import: pending` record carries the addr it is replacing, and reconcile |
| + swaps that one row, and only if its addr matches, in one registry |
| + transaction: the addr row becomes the exec row, on this box. The edge picks up the new row on its next |
| + reload. Requests during that reload go to whichever row the edge still |
| + holds, and the app's supervisor must keep its own process frozen until |
| + the platform instance serves. `verify` (§6.5) is how the platform knows it |
| + does. |
| + |
| +Adoption and move are separate verbs on purpose. An adopted instance lands |
| +in the region its data is already in, and then moves like any other. |
| + |
| +## 10. Security |
| + |
| +- **Staging stays push-only.** The `moves/*` grant (`infra/modules/region-host`, |
| + `sibling_staging_arns`) is unchanged: Put, Delete and prefix-conditioned |
| + List on siblings' `moves/*`, never Get, never `instances/*`. Tito's |
| + deployment has it applied on all six boxes (IAM read, 2026-09-24). Store |
| + twins widen it by one statement: Put/Delete/List on |
| + `arn:aws:s3:::<store-bucket-prefix>-*/moves/*` in sibling regions, and the |
| + target's own promotion copies `moves/` into place, exactly as the replica |
| + does. No box ever reads another region's data. |
| +- **Hooks never come from outside.** The edge refuses (404, byte-identical to |
| + unknown host) any public request whose path matches a declared hook path |
| + for that host's app. The app must not have to tell a real freeze from a |
| + forged one by headers. The only way a request reaches a hook path is the |
| + agent's `ServeHook` over the socket, with the admin token. |
| +- **The admin token's reach does not grow.** It already arrives on every |
| + tick. A hook is one more path it is presented on, by the same sender, over |
| + the same socket. |
| +- **The edge door is scoped per app, by group and by record.** A process |
| + in app A's group cannot reach app B's instances through it. It is on the |
| + box, so it is no more exposed than the sockets it replaces. Unlike those |
| + sockets, it works under the isolated layout, where an app's router can no |
| + longer see `/run/carlos` at all. |
| +- **Imports are one-shot and pinned.** The presigned PUT is bound to one key, |
| + one length and one checksum, expires in 15 minutes, and the activator |
| + verifies the checksum again before trusting the file. |
| +- **Record authority.** A move is a record write, and discussion #171 |
| + ("A region box's IAM role can write any account's records") means any |
| + compromised box can forge one today. This design adds no new writer class, |
| + but a forged `move.to` is a data-exfiltration primitive. #171's fix bounds |
| + it, and this record depends on it before moves are offered self-serve to |
| + apps outside the deployment's own operator. |
| + |
| +## 11. Observability |
| + |
| +- `carlos instances move <host>`: `--to <region>` requests, `--cut`, |
| + `--abort`, `--status`, `--wait` (polls to a terminal state, printing each |
| + transition with its timing). Designed for a user with no box access (the |
| + 2026-08-08 rule): every fact it prints comes from the console. |
| +- The console's instance page shows the in-flight state and the last |
| + 30 days of move logs. |
| +- Box logs keep the `moves: <host>: …` shape (`movespass.go`) and add one |
| + line per transition with its duration. |
| +- **Stuck states page.** Each state has a time fence, and passing it is |
| + visible and alerting, never an automatic action on the far side of the cut: |
| + |
| +| state | fence | what happens past it | |
| +|---|---|---| |
| +| `preparing` | 60 min without a caught-up round | logged `stalled` with the last delta size; stays abortable | |
| +| `freezing` | 60 s | abort (the app never froze; clean) | |
| +| `cutting` | 15 min | pages; completes forward only | |
| +| `landing` | 15 min | logged `done, verify_timeout`; pages | |
| +| source retention | 7 days | source copy deleted, logged | |
| + |
| +## 12. Gap list against today's code |
| + |
| +| # | gap | where | |
| +|---|---|---| |
| +| G1 | No wake before a tenant's freeze; nothing lets a tenant wake at all | `activator.go:2777-2779` (ticks only); titogo `move_order.go:30-36` | |
| +| G2 | No hook contract; freeze is tenant-held by design | #331 spec "The two verbs"; `apiinstancemove.go:232-238` | |
| +| G3 | Edge answers 404 for a moving host | `edge.go:576-578`, `:1383-1387`, `:1999` | |
| +| G4 | Moves to the home region refused, so rollback out of home is impossible | `apiinstancemove.go:196-202`; `movespass.go:214` | |
| +| G5 | Terminal state invisible; `"none"` after done | `provision.go:1036-1044`; `apiinstancemove.go:88-91` | |
| +| G6 | Source replica never deleted after a move | `movespass.go:265` (staging only); `lease/mirror.go:113` | |
| +| G7 | Stores are one region per deployment, and nothing copies one | `CARLOS_STORE_REGION`, `cmd/console/main.go:492-524` | |
| +| G8 | No CLI verb; console API only | `instances_cli.go` (verbs: enable, create, list, delete, set-upstreams, set-channel, set-environment) | |
| +| G9 | No adoption/import door for an externally-run instance, and reconcile will not replace a differently-shaped row | `apiinstancemove.go:43-46`; `reconcile.go:1022-1034`; `Manifest.Seed` is a first-boot argv, not data (`provision.go:233`) | |
| +| G10 | Isolated layout built but not migrated on carlos-tito-ie | isolated-exec spec §8.6 step 4; box read 2026-09-24 | |
| +| G11 | An app's own router cannot reach a moved instance | titogo `registry.go:539-541`, `router.go:580-582`, `move_sweep.go:648-664` | |
| +| G12 | **Verify before building:** the tick runner fires a route without checking its status, and `ServeScheduled` does not check it either, so a tick due during a cut could wake the source after the purge and restore it from its own replica: a second writer the target never hears about | `internal/schedule/runner.go:143-152`; `activator.go:2781-2795` (checks fences, kind, hibernate; not status). The due-wake index does skip non-ready routes (`activator.go:2547`), which is why this needs a test, not an assumption | |
| +| G13 | Record writes are box-forgeable, so a forged `move.to` exfiltrates | discussion #171 | |
| + |
| +## 13. Slices |
| + |
| +Each slice is its own PR, lands dormant, and is useful alone. The order puts |
| +Tito's first real move (`elatromme` → `us`, a 5 MB database, not an import |
| +mirror) as early as it can safely be. |
| + |
| +| slice | what | depends on | unblocks | |
| +|---|---|---|---| |
| +| **P0** | Run `carlos system layout migrate` on carlos-tito-ie (operator act, built) and fix G12 with a test | nothing new | P6, and any adoption | |
| +| **P1** | `ServeHook`, the manifest `moves.hooks` declaration, `freezing` state, freeze/thaw calls with the tick's token; hook-path refusal at the edge | P0 for G12 | row 1's failure mode, for every app | |
| +| **P2** | Holding route: `moving` edges role, 503 instead of 404 | — | a cut nobody sees as an error | |
| +| **P3** | Home region publishes a staging lane; moves to and from home | infra: add `ie` to each sibling's `sibling_staging_arns` | rollback | |
| +| **P4** | Move log, terminal state on the GET, `landing` + verify, `carlos instances move` | P1 | apps stop inferring done | |
| +| **P5** | Source retention fence and deletion | P4 (needs `done` time) | the residency promise | |
| +| **P6** | Adoption: import door, `imports/` restore fallback, addr-row swap | P0 | every controld-style instance | |
| +| **P7** | Edge-local door for app routers | P0 (isolated layout is why the sockets stop being reachable) | apps with their own router, Tito's root domain | |
| +| **P8** | Per-placement store backend; private store and prefix stores follow | P4, P5 | uploads residency | |
| + |
| +**For elatromme → us, the minimum is P0, P6, P1, P3, and P7 or its polling |
| +alternative** (§8), plus Tito's own slices in the linked record. P2 is small, |
| +and a customer's first move should not show buyers a 404, so it belongs in |
| +that set too. P8 is the only way uploads move, and whether the first move |
| +waits for it is a product call (§14). |
| + |
| +## 14. Open questions |
| + |
| +1. **Freeze budget ownership.** The app declares the budget, and the |
| + platform never extends a freeze past it. If `cutting` itself runs past |
| + the budget (a slow final checkpoint), the app unfreezes while its process |
| + is already stopped, which is harmless. Is that acceptable for every app, |
| + or should the platform refuse to cut when the measured cut time (#331's |
| + rehearsal: status→quiesce→flip 331 s before the serial nudge) exceeds the |
| + declared budget? |
| +2. **Replicas on a move.** A home with `replicas` (discussion #162): does a |
| + move of the home carry the replica set's placements unchanged, refuse |
| + while replicas exist, or re-point them? |
| +3. **The edge door's identity.** Group membership per app on the box is the |
| + simplest proof. Is a per-app group acceptable under the isolated layout's |
| + uid model, or should the door take the app's scoped key instead? |
| +4. **What the control plane keeps in the home region.** The record, the |
| + config bundle, the secrets and the move log all live in the deployment |
| + bucket, in the home region. A move relocates data, not configuration. |
| + Is that the promise we make to apps, stated in `docs/site`, or does the |
| + config bundle need a regional copy too? |
| +5. **Self-serve for other accounts.** Until #171 is fixed, should the move |
| + doors be operator-only for accounts other than the deployment's own? |