amadan
Hotfix in review (platform-1e), on Paul's ruling 2026-10-13. Branch hotfix-startup-sweep-281 from v0.27.0, now at 330dfa54. The edge's own startup is unchanged from v0.27.0, so the gate is unchanged. activatorStart.Run (sweep, recovery, Start, key pass) runs in the background. Instance admission waits on a startup latch for up to 30 s, then returns the wake-timeout 503. Opus r2 says SAFE TO RELEASE, and Astra r4 and CI are pending. Ships as v0.27.1. Asks 1 and 3 are covered. Ask 2, a large-box startup test without stubs, stays open as a follow-up.
Agent startup is not bounded under the 20s update gate: sweep and recovery still run before the edge listens Issue open
This came out of the v0.26.0 roll incident. annie-welding (127 instances) failed the update gate and rolled back, after about 35 seconds of edge errors.
v0.26.1 fixes that cause: the instance-key backfill now runs in the background after Start (hotfix-instance-key-gate). Astra's review of the hotfix found the remaining risk, which existed before it.
The remaining risk.
activatorStart.Run(internal/host/layoutleg.go:396) still waits synchronously for the instance-unit sweep (allowed 60s) and for layout recovery. Recovery handles journals one at a time with no overall deadline. Hold restoration handles hosts one at a time and lists every route for each host (internal/layout/pass.go:525, :1028).- The startup intent read is allowed 30s.
- Any of these, on a big or slow box, can exceed the 20-second update gate. When it does, the box rolls back after tens of seconds of edge errors. It is the same shape as the Sydney cert-mirror failure (au-box-gate-failure) and the v0.26.0 incident.
Asks.
- Bound everything on the path before listening well inside the gate. Defer what can wait, or make it run in parallel with a hard cap.
- Add an end-to-end startup test on a large box that doesn't stub sweep and recovery.
- Make the background key pass take the agent's lifetime context, and join it with a bound on shutdown. Make its one-minute budget strict: the slot wait and local I/O don't observe cancellation yet.
The large-box startup test is merged to main as d2f58fd. It is test-only and sits on top of the hotfix 6dfbff1.
What it covers:
- 300 routes and a cold real instancectl.
- Real sweep, recovery and hold restoration over 8 interrupted moves and 4 held hosts.
- The gate probe is answered within 3 s; it was measured at 20–120 ms.
- A worker, a sidecar request and a scheduled delivery each wait at the latch.
- Holds are restored before Start.
- A wake that overruns the 30 s limit gets a 503.
- A clean shutdown join.
- An AST guard that pins runAgent's execution order to the order the test copies.
Mutation checks caught all of these: a missing latch, hold restoration moved after Start, async recovery, a wait before the edge, and a retry that hid a missed deadline. Astra reviewed it over two rounds.
Still open on this issue: on SIGTERM, a wake held at the latch makes shutdown wait the edge's full 10 s drain, and startup keeps running during it. platform-1e has recorded the fix: cancel startup and refuse held wakes before the drain. The follow-up of sharing the orchestration between runAgent and the test is also still open.