carlos / platform Public

Fixes discussion #110.

The answer to the issue's question

A region box today is edge-only — not by design, but because nothing tells it otherwise and nothing checks.

An activator arms on CARLOS_HIBERNATE_BUCKET in the box's /etc/carlos/host.env. Nothing in this repo puts it there: carlos bootstrap deliberately doesn't, and infra/modules/region-host grants the host role its region's data bucket and then never mentions it again. A region box brought up the documented way serves TLS, converges routes and stages binaries perfectly — and cannot run a single instance.

Why the 502 proves it

internal/activator's serve() answers 503 for every wake failure it has (lease held, wake timeout, spawn error) and declines the request only when the route is not a hibernating instance. So a bare 502 is not a failed wake — it is the edge proxying straight to the socket path the route derives, which only an armed activator would ever create, into the stdlib ReverseProxy's default handler.

The empty data bucket says the same thing independently: every wake claims an S3 lease before anything else, so a bucket with no lock object means no wake was ever attempted.

What changed

armed now means "can host". It was a box publishing a placement code and having an EIP — an edge-membership fact — while the regions endpoint, the create door and every consumer picker read the set as "regions that can take an instance right now". armedPlacements now also requires the box's published data_bucket, which is CARLOS_HIBERNATE_BUCKET verbatim from the same env, read in the same process, that decides whether the activator arms. Already on the wire since the instance-move work; nothing read it.

Self-healing: the moment a region box's host.env grows the variable, its next edges-file pass publishes the key and the region arms itself, with no console change. Until then create refuses it by name and lists the regions that can really take one.

Dormant until some box publishes a data bucket — the roll-order rule the placement field itself got. A fleet whose boxes all predate the field would otherwise go from every region armed to none the moment this console rolled, refusing creates into regions that host perfectly well.

The box's own half: edge Reload now names the condition — which host, which socket nobody will bind, which variable fixes it — throttled, since it is a standing condition. Previously silent: a 13-second dial retry and a bodiless 502 that reads as "your backend is broken". Not a refusal to serve; carlos edge without the agent has no activator by design.

data_bucket_arn's description now says that granting the bucket is half the job and the box must also be told about it.

Tests

TestArmedPlacementsRequiresHosting covers dormancy, the reachability half surviving, a reachable-but-not-hosting box going unarmed, and a region arming itself on republish. Verified to fail against the old predicate with exactly the right message.

Three edge tests cover the new log line and, importantly, its silence on an armed box and on an unflagged instance.

Not fixed here — both still open, both hit in this investigation

• Per-instance DNS for a region-placed host is recorded but not built (internal/console/apiinstances.go). Such a host still resolves to the home edge and arrives over the backhaul, whose dial failures are also bare 502s. If a region box turns out to be armed and still 502s, this is where to look next.
• The console's health join reads its OWN local registry (instanceHealth), so any route homed on another box reads "Setting up" forever, regardless of region. That is what the issue saw; it is structural, not region-specific. The fleet snapshot carries only route_versions, not the columns healthWord needs.

Effect on tito

us de in se uk go unarmed and carlos instances create --region us starts refusing with a real message. That is the intended behaviour change — the issue's own option 1 — and it is reversible per region by adding CARLOS_HIBERNATE_BUCKET/_REGION to that box's host.env and restarting carlos-edge, which is also the pilot the issue asks for.