carlos / platform Public

Diagnosed on barbara-tinkering (AU) on 2026-09-19, fixed for now by 1b48ed5. Filing the structural half, which 1b48ed5 narrows but does not close.

The shape

internal/edge/edge.go runs two synchronous refreshes before EITHER listener binds:

certMirrorBootTimeout = 20 time.Second (internal/edge/certsync.go) homeMapBootTimeout = 20 time.Second (internal/edge/homemap.go)

and systemGateWait in internal/host/systemapply.go is also 20s. The gate also has to cover process start, registry.Open and route load -- about 2s on that box.

So either boot timeout ALONE can consume the whole gate, and the two of them together are twice it. When that happens the failure is total and self-sustaining: the gate's probe is refused, the apply rolls back, the rollback restarts the edge, the edge stalls for the same reason, the rollback's own gate fails too, and the box takes nothing. It cannot be fixed from the box, because the only way to fix it is to ship a new build and the box will not accept one.

That is exactly what happened: the AU box sat on df7d1d2 from 2026-09-07 to 2026-09-21, failing four rolls identically, while the other three boxes rolled cleanly every time.

What made it fire

Distance times count, not any particular build. The deployment bucket is eu-west-1; the AU box is ~250ms away; there were 101 objects under control/certs/; and certMirror.Refresh fetched them one at a time. 101 x 250ms is ~25s against a 20s budget, and the edge bound its listeners two seconds after the gate had already given up. Journal, 2026-09-18:

15:40:20 Started carlos-edge.service 15:40:22 registry changed — reloaded (24 route(s)) (21 seconds, nothing logged) 15:40:41 <- apply's gate gives up 15:40:43 serving 24 route(s) on :443

1b48ed5 made those gets concurrent, and the same box now binds in 6 seconds. But the budget arithmetic is unchanged.

Why it will come back

• The cert mirror's cost is still ceil(count/10) x latency. At ~1300 published certs the AU box is back over the gate. us-east-1 at ~75ms is about a third of the way there at today's count. Every newly published cert moves every box closer.
• homeMap.Refresh is 6 sequential round trips today (one List, four control/edges/*.json, two in loadMeshKeys) and has the identical shape and the identical 20s budget. It scales with box count rather than cert count, so it is far from its cliff -- but it is the same bug waiting.
• Neither depends on cert count to fire. Any boot-time bucket stall -- a NAT flap, an IMDS credential hang, an S3 partial -- reproduces the whole thing at any count.

Options

1. Cut the two boot timeouts so their SUM sits well under the gate (8s + 5s, say). Smallest change; keeps the "warm before serving" intent.
2. Bind the listeners first and let the first poll fill the mirror in. A cold handshake falls back to the local cache and autocert, which is a much better failure mode than not serving at all. This is the one that removes the class rather than the instance.
3. Make the gate tolerate a slower boot. Rejected: the box genuinely is not serving for that whole window, so a longer gate hides a real outage rather than fixing it.

I would do 2, and 1 as the cheap insurance behind it.

See also the cert mirror scaling thread and #165 (a box that stages an update and never asks for it) -- all three came out of the same box.