carlos / platform Public

regions: armed must mean "can host", not "is an edge" (#110)

Five of tito's six armed regions accepted an instance and then served
502 forever. The record was accepted, the box placed it, minted its own
route, staged the binary and repointed the live symlink — and nothing
ever ran it. No unit, no socket, nothing under /data, and a data bucket
that was still completely empty fifteen minutes later.

The empty bucket is the tell. Every wake claims an S3 lease before it
does anything else, so a bucket with no lock object means no wake was
ever attempted. The status code says the same thing from the other end:
internal/activator's serve() answers 503 for every wake failure it has —
lease held, wake timeout, spawn error — and declines the request only
when the route is not a hibernating instance. A bare 502 is therefore
not a failed wake. It is the edge proxying straight to the socket path
the route derives, which only an armed activator would ever create, and
getting the stdlib ReverseProxy's default handler.

What arms an activator is CARLOS_HIBERNATE_BUCKET in the box's
/etc/carlos/host.env, and nothing in this repo puts it there — not
`carlos bootstrap`, not the region-host module, which grants the host
role its region's data bucket and then never mentions it again. A region
box brought up the documented way is an edge that cannot host. So the
answer to the issue's actual question is: a region box today is
edge-only, not by design but because nothing tells it otherwise and
nothing checks.

`armed` was the half that should have caught it. It was computed from a
box publishing a placement code and having an EIP — an EDGE-MEMBERSHIP
fact — while the regions endpoint, the create door and every consumer
picker read the set as "regions that can take an instance right now".
For five regions those two diverged completely, and from the consumer
side there was no signal at all: create succeeded, placement was
correct, the record looked healthy-ish, and the only symptom was a dead
host.

The hosting signal was already on the wire. Each box publishes
data_bucket in its edges file, written from CARLOS_HIBERNATE_BUCKET
verbatim and omitted when that env is unset — the SAME env, read in the
same process, that decides whether the activator arms at all. So a
published data bucket means that box's activator is armed, which is
exactly what a region picker is asking. armedPlacements now requires it
alongside the EIP.

It heals with no console change: the moment a region box's host.env
grows the variable, its next edges-file pass publishes the key and the
region arms itself. Until then create refuses it by name and lists the
regions that can really take one, instead of accepting an instance that
will never run.

DORMANT until some box publishes a data bucket, the roll-order rule the
placement field itself got. A fleet whose boxes all predate the field
would otherwise go from every region armed to none the moment this
console rolled — refusing creates into regions that host perfectly well,
which is worse than the bug being fixed.

The box's own half: edge Reload now names the condition. A
hibernate-flagged instance route on a box with no activator was
unservable in silence — every request falling into a 13-second dial
retry and coming back as a bodiless 502 that reads as "your backend is
broken", for a backend that was never going to exist. It now says which
host, which socket nobody will bind, and which variable fixes it,
throttled because it is a standing condition. Not a refusal to serve:
`carlos edge` without the agent has no activator by design, and dropping
the route would only trade a 502 for a 404.

Not fixed here, and both still open: per-instance DNS for a region-placed
host is recorded but not built (apiinstances.go), so such a host still
resolves to the home edge and arrives over the backhaul; and the
console's health join reads its OWN local registry, so any route homed
on another box reads "Setting up" forever regardless of region.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ggesa34CVBQ97guKaEi9jw