carlos / platform Public

What Tito needs before it can place an instance outside the home region at signup. Tito Go has switched region choice and moves off until these hold (titogo TITO_REMOTE_PLACEMENT, default off). Please verify against this list when the placement work in #175 ("Instance moves for every app") is done. Paul asked for this on 2026-10-02.

What happened

A real customer signed up on 2026-10-01 at 23:28 UTC and picked Germany. Tito's signup calls POST …/instances with region: de, and the console accepted it, because de is in GET …/regions as armed. It was the first titogo instance ever placed outside Ireland. It stayed unusable for 20 hours:

1. No certificate on the backhaul. The ie and uk edges backhauled goodbot.go.tito.io to linus-pottery (de), and de refused every handshake with acme: not available here (errNotIssuer, internal/edge/certsync.go:103). de's /etc/carlos/host.env had none of CARLOS_WILDCARD_DOMAINS, PLATFORM_APPS_DOMAIN, PLATFORM_EDGE_IPS, CARLOS_DNS_PROVIDER=route53 or CARLOS_ACME_DELEGATION_ZONE. uk and se have them; de, us and in did not. Tito added them by hand on 2026-10-02 (backups at /etc/carlos/host.env.bak-20261002T17*), and the de edge then served *.go.tito.io from the mirrored wildcard. Infra or bootstrap should own this, so a new box can't come up half-configured.
2. No app config on the box. Every region box logs agent: config sync loop disabled: CARLOS_SECRETS_KEY not set. Only ie holds the key. goodbot's env on de was just CARLOS_ADMIN_TOKEN: no TITO_AUTH_PUBKEY, TITO_ROOT_DOMAIN, SMTP or store credentials. It booted mail: log-only and could not verify the app router's signed identity. This is the same gap #175's 2026-10-02 P1 note found from the move side.
3. Nothing reached it from the root domain. Tito's router dials /run/carlos/<host>.sock on ie, which never exists for a de instance. The P7 door would fix this, but it isn't rolled (boxes are on v0.21.0).
4. Nobody was told. The console showed the instance as Setting up for 18 hours. No alert fired, and the instances API gave Tito no way to tell "placed but cannot serve" from "still provisioning".

Resolution: the account held nothing but first-wake seed data, so Tito deleted the de record and re-created the host in ie with the same name (2026-10-02 19:46 UTC). The de route is retired, and its parked data in carlos-tito-de was left in place.

Requirements (please verify each)

R1. Armed means servable, per app. A region in GET …/apps/{sqid}/{app}/regions is armed: true only when an instance of this app created there would serve end to end. That means certificates for the app's domains, the app's config deliverable, and reachability from the app's own processes. Box existence is not enough. If that can't be computed, add a per-app flag the operator sets, and leave regions unarmed by default.

R2. A placed instance gets its full config on its box before its first wake. That is <host>.config.env with every key in the app's bundle and its secrets. Whether that's by distributing CARLOS_SECRETS_KEY (internal/appconfig/crypto.go:11 says it must reach every decrypting host) or by per-app or per-region keys is the platform's call. Tito's preference is per-app keys, because one key today decrypts every app's bundle on every box that holds it.

R3. Region-scoped config values. Some values have to differ by region: store bucket and region, any service endpoint, and a payments relay address if Tito runs one per region. Today a bundle is region-blind (TITO_S3_BUCKET/TITO_S3_REGION point at eu-west-1, TITO_RELAY_SOCKET at an ie-only socket). An app needs a way to say "this key, in region X". Without it, Tito's promise that files are stored in the chosen region is false.

R4. Box config is complete by construction. Every region box carries what the home box carries for certificates (the five host.env keys in item 1), from bootstrap or infra, with a check that fails loudly when one is missing.

R5. The app's processes can reach a remote instance. That's the P7 door, rolled and enabled on the home box: wake, backhaul to the placing box, a holding response while not ready. Tito still has to switch its router, relay and import-sync to the door (Tito-side), and has to send the buyer's IP in a signed header.

R6. A servability signal the app can poll. The instances API (GET …/instances/{host}) answers with something that separates "provisioning" from "serving" from "failed: <reason>". Signup's preparing page waits on it today by probing the public host. The console should page somebody when an instance stays in Setting up longer than a bound.

R7. Moves. The items in #175's P1 note: the per-host data dir or key material, config rendered on the target before the flip, automatic rollback, and a rehearsal on an app shaped like the one being moved. Tito re-opens its move doors only after those hold.

How to verify

For titogo (account bae), in each of uk, se, de, us and in:

• carlos instances create --app titogo --host <canary>.go.tito.io --region <r>.
• Within a stated bound, the instance serves GET /api/version on its own host from a visitor outside that region.
• Its <host>.config.env on the placing box holds every key in the bundle.
• A request through the ie door with Host: <canary>.go.tito.io reaches it awake and asleep.
• GET …/regions stays unarmed for any region where one of those fails.

Tito will run its own end-to-end pass (sign up, root-domain admin, mail, uploads) before setting TITO_REMOTE_PLACEMENT=on.

Tito-side follow-ups (not asks): payments egress from a remote region (titogo moves design §5.1), and stopping controld's fleet watch from starting phantom titogo@ units for platform-managed hosts.