carlos / platform Public

Docs PR for the design record: https://github.com/carlosframework/platform/pull/419 ("docs: instance moves for every app — design record"). Branch docs-instance-moves-design is on amadan too.

Claimed. I'm starting with P0 on branch instance-moves-p0: the tick-wake cut fence (G12), the MoveAbort allowlist (G16), and the replica rail on the move request (G14). It's built and tests pass, and an Opus review is running. The branch description has the details.

One design note for later slices. P0 reuses the activator's maintenance fence (FenceAndStop), which has lived in memory with no way to lift it and no production caller. P0 adds LiftFence(host, generation). P1's abort-until-the-purge should lift through the same call. The worker already lifts any fence whose record is at rest on this box again.

P0 review (Opus) is in. The two medium findings are fixed on instance-moves-p0 (84526d9d): the moves pass ran from two goroutines with no shared lock, so a stale pass could lift a live cut's fence. That concurrency predated P0; two passes could already drive one cut at once.

Carried forward for later slices, not fixed in P0:

• Fence ownership (latent). Move fences and maintenance-worker fences share one slot per host in the activator. The maintenance side has no production caller yet, but the day it does, a cut would overwrite its fence and the lift would reopen it. Key fences by owner before both are live.
• P1 must widen the abort allowlist to freezing and update TestMoveAbortRefusesStatesItDoesNotKnow, which pins it refused today.
• P1: nothing writes a moving row back to ready, and reconcile does not converge status. Abort-in-cutting has to do it itself; the at-rest lift only reopens the activator.
• A cut whose fence keeps failing sits in cutting with no abort until P1's abort-until-the-purge lands.
• Ticks refused during a cut (503, retryable) can exhaust a timer's retry budget on a long cut. That is the same as G15's once-timer loss; P4 carries timers.

P0 merged to main as 408db5e (squash) and GitHub is fast-forwarded. It is not rolled. Rollout order is consoles before boxes (design §6.1), because the MoveAbort allowlist has to be live on every console before any box writes landing. I'm keeping the claim on #175 for the remaining slices. P1 onward waits on Paul's rulings on the §3 proposals and the §14 open questions.

Paul ruled all sixteen decisions on 2026-09-25, and each followed Astra's recommendation. The table is in §0 of the design record on docs-instance-moves-design (a6ca7a1 or later). Astra's recommendations are committed unedited beside it as CODEX-RULINGS-INSTANCE-MOVES.md.

Changes from the record as written:

• Q3: the edge door authenticates an app-scoped credential; a Unix group is not enough.
• Q6: the move-scoped IAM user carries an enforced date condition, plus a reaper.
• Two new requirements: M1 is a platform write fence on moved stores before the final delta; M2 is an API/CLI recovery path for a stuck landing.
• P4 joins the minimum path. elatromme → us is now P0, P7, P9, P6, P1, P2, P3, P4.
• Q5: moves for other accounts are operator-only until #171 is fixed.
• Q7: direct-mode carlos add must name its account.

Next, per the ruling order: P7 (edge door) and the Q7 direct-add fix. Both can start now.

Correction: the rulings commit is a2692fe0, not a6ca7a1.

Progress, 2026-09-28.

Q7 (carlos add names its account) is built on carlos-add-account, with its copy approved and docs in. The merge is held. It breaks any caller that relied on the ops default, and the next build off main could roll to carlos-tito-ie before Tito's route sync passes an account (tito/titogo #17). The platform-infrastructure scripts are fixed in GitHub PR #37.

P7 (the edge-local door): the plan is on instance-moves-p7 at docs/superpowers/plans/2026-09-28-instance-moves-p7-edge-door.md, rev 4, after three Astra rounds. All findings are committed unedited beside it.

• Round 3 disproved my claim that a believed visitor address costs only attribution. It drives the password gate's guess limit and any app that authorises by IP.
• Paul ruled on 2026-09-28 that the door believes no caller-supplied address. Door requests reach the app with X-Forwarded-For: unknown on any box, and an app that needs the buyer's IP carries it in its own signed header. This is recorded as P7-trust in §0 of the design record.
• For Tito: the router has to send the buyer's IP in a Tito-signed header, next to X-Tito-Auth-*. The instance has to read it from there, not from X-Forwarded-For.
• The door key still identifies the calling app and confines it to the app's hosts.

Astra round 4 is running. The build starts when it comes back clean.

P7 is merged to main as 88abfae (squash), and GitHub is fast-forwarded. It is not rolled, and it's dormant: no box opens the door until it sets CARLOS_EDGE_DOOR.

What it gives apps: /run/carlos/edge-local.sock, dialled with Host: <instance> and X-Carlos-Door-Key. Mint keys with carlos instances door-key create --app <app>. A door request is served as a request from the box itself, with the box's public address and https. No caller-supplied visitor address is believed (Paul, 2026-09-28). It wakes a sleeping instance, backhauls to the box that holds a moved one, and applies the gate. Remote hosts are reachable only when their box publishes a single owner.

The plan, six Astra rounds and the as-built notes from Fable's final review are all in docs/superpowers/plans/ on main.

Rollout, when approved:

1. Consoles first.
2. Boxes next.
3. On carlos-tito-ie only, set CARLOS_EDGE_DOOR=/run/carlos/edge-local.sock. This needs scoped approval.

Tito's side (router, relay, import-sync):

• mint a key;
• enrol each process's user with carlos bootstrap --socket-consumer;
• switch from computed socket paths to the door;
• send the buyer's IP in a Tito-signed header.

P9 must not run on carlos-tito-ie before that switch.

Still held: carlos-add-account (Q7), which waits on tito/titogo #17. P0 is merged but not rolled.

Bug found in production, 2026-10-01: a busy instance can never reach ready.

runMoveRound promotes preparing to ready only when a round's lease.MirrorPrefix reports changed == 0 (internal/host/movespass.go:329). The file comment at line 31 says convergence is "the round-over-round shrinking of the delta", but the code waits for zero. An instance that stays awake keeps producing L0 files every few seconds, and litestream compaction keeps rewriting level files (138 MB each for a 293 MB database). No round ever copies nothing.

What happened: bdn/ips kpmg.bdn.oncarlos.com sat in preparing (IE to uk) for 80 minutes, re-staging about 400 MB per round. It was aborted cleanly (staging deleted, instance untouched). An idle throwaway bdf/hello instance moved IE to uk on v0.22.0 in about 6 minutes, with about 50 s of 503 at the cut.

Fix in progress on branch move-ready-delta-threshold: ready when a round copies under a byte threshold (default 16 MiB, overridable by environment variable). This is safe because the cut already quiesces the instance, takes a final checkpoint and runs a byte-exact final MirrorPrefix (lines ~400-415), so the threshold only bounds the cut's remaining work.

Correction to the diagnosis above (Astra's review of move-ready-delta-threshold, confirmed in production). The main cause is multipart ETags, not only busy writes.

Litestream uploads large LTX files in multipart pieces. All six large objects under instances/kpmg.bdn.oncarlos.com/ carry "<md5>-27" ETags (138 MB each). S3 CopyObject gives the destination a single-part ETag, so MirrorPrefix's ETag-equality skip never matches. Every round re-copied about 830 MB, even with no new writes. Any instance with a database over roughly 5 MiB per level file is affected; small idle instances (the bdf/hello rehearsal) are not.

Paul approved the fix: copy receipts under the staging prefix, recording source ETag and size plus the resulting destination ETag. An object is skipped only on an exact match. Copies are conditional on the listed source ETag. Being built on the same branch, together with the 16 MiB ready threshold.

P1, production 2026-10-02: a completed move left kpmg.bdn.oncarlos.com unable to start for 1h54m (06:37 to 08:31 UTC). Do not move anything else until this is fixed.

With v0.22.2 the move reached ready within minutes and the cut completed (region: uk). On the UK box every wake then failed with "instance not healthy within 30s". There were two causes, and neither is about the database:

1. The per-host data directory does not move. The move carries only the litestream replica. /data/<host>/ holds the app's instance.key and report.token. On the target, the app minted a new key, and data in the DB sealed under the original key can't be read with it. The original key stayed on the source box. Had the source cleanup removed /data/<host>/, that data would have been unrecoverable.
2. Config doesn't land on the target. The UK box had <host>.env but no <host>.config.env. The app logged build app: File storage is unavailable. Ask a deployment operator to provision the object store. In Ireland, kpmg.bdn.oncarlos.com.config.env (1265 bytes) is regenerated as soon as the route is back. The store binding (carlos-store-bdn-ips, eu-west-1) and any SSM-backed values are regional.

Recovery (Paul approved it): the record's region was hand-edited back to home with a conditional put, and the control serial was bumped. Ireland restored from its own bucket, which still held the replica as of the cut, and served at 08:31:07 on its original key. The UK copy never served, so no data was lost. UK leftovers (local DB, minted key, parked data in carlos-data-uk) are still to be cleaned up.

What has to be true before the next move:

• The cut carries /data/<host>/ (or the platform owns the key material).
• Config and store bindings are rendered on the target and checked before the flip.
• A post-cut health check rolls back automatically, which needs P3.
• The bdf/hello rehearsal passed only because hello has no key file and no store. A rehearsal has to use an app with the same shape as the one being moved.

For the Tito side (Paul ruled M18, 2026-10-02): the move-any-app spec (branch design-moves-any-region, Phase 1) makes every move mutation operator-only: request, cut, abort and resolve. Instance-role service tokens keep only the status read, until a follow-on owner-surface spec designs owner and service authorization properly. Tito Go's router can still read move status, but can't start or cut a move once Phase 1 ships. Nothing changes in production until then. The reason is that Astra found a member-minted instance token could otherwise bypass the owner gate.

Related: both the move fencing and the #199 delete-time park now depend on one shared host-lifecycle claim (control/hosts/<host>.json, CAS, with operation, op id and generation). It's being drafted as a small standalone spec by the #199 session and cited by both.