Edge restart: the stopping agent recycles an instance and strands its lease for 15 minutes closed
Claimed by claude~paul@keymail.dev1 week ago
claude agent, operated by paul@keymail.devdriven by paul@keymail.devsession 5b793570 · claude-code/2.1.288 · DOCS-T1
Seen on flagship annie-welding (v0.27.2) at 2026-10-14 13:18Z, during the zero-downtime Gate F retry. That step was one systemctl restart carlos-edge, there to set CARLOS_SWAP=overlap. Nothing here is specific to the flag. Any edge restart can do this, including every carlos system update and each Gate G arming.
Sequence (old agent pid 2680076, new agent pid 322617; full log in the Gate F retry report on #272):
13:18:31 systemd stops carlos-instance@bunkhouse.bdd… as part of the edge stop. It deactivates cleanly.
13:18:33 the OLD agent logs instance bunkhouse… exited unexpectedly — recycling, then hibernated.
13:18:47–48 the old agent is still alive and provisioning. It starts the unit again, which fails at once: Failed to set up standard output: Connection refused (status 209/STDOUT), because the log socket was already gone.
13:18:52 the NEW agent logs LEASE HELD for bunkhouse… by "ip-172-31-13-235:2680076" — a previous incarnation hasn't released; refusing to wake (retry after 14m41s).
bunkhouse refused every wake until 13:33:35, then woke healthy.
Likely cause. During shutdown, the activator treats a stop the shutdown itself caused as an unexpected exit and recycles the instance. That recycle (or the wake attempt) takes or refreshes the replica lease under the old incarnation, and nothing releases it before the process exits. The next agent then honours the lease for its whole 15-minute TTL.
Expected.
Once shutdown has begun, the agent recycles and wakes nothing.
Every lease the stopping agent holds is released, or handed to its successor, before it exits.
Impact. On a box with many instances, any instance caught in that window is down for up to 15 minutes after a routine restart. The window is the 17 s restart that annie's #281 sweep produces. This should be fixed before Gate G restarts adi-knitting, barbara-tinkering and iris-clockmaking, or those restarts should be accepted as a known risk.
claude agent, operated by paul@keymail.devdriven by paul@keymail.devsession 5b793570 · claude-code/2.1.288 · DOCS-T1
Fixed. Merged to main as e15ad6b (fix-shutdown-lease-292), after Astra rounds 1–4, the last of which was ACCEPT. Census ruling census-ruling-292 is from platform-1e.
Root cause. carlos-instance@'s BindsTo=/After=carlos-edge makes systemd stop every instance about 10 s before the old agent gets SIGTERM. In that gap the old agent recycled bunkhouse, woke it again for a queued request (taking the lease), and blocked in a systemd start queued behind the edge restart. With swaps off, Shutdown neither cancelled nor joined that wake, so the agent exited holding the lease.
Fix.
Shutdown cancels and joins waking entries whatever CARLOS_SWAP is set to. Once shutdown begins, nothing recycles, wakes or starts.
A cancelled legacy start is stopped through instance-control and proven gone (inactive unit, empty cgroup) before its lease is released. If that can't be proven, the lease is kept and expires on its TTL, which is better than two writers.
instance-control (root) serialises legacy start and stop per host. A /stop fences at the accept watermark, then cancels and joins in-flight starts (10 s bound) before its systemd stop. Keep-alive is off, so each request gets its own connection.
A failed lease release is retried as a conditional delete against the same ETag, and retired at the record's own expiry.
Roll note. The release that carries this must restart instance-control. Updaters from v0.26.0 onward do that on their own.
Known trade-off. A new agent's start that overlaps the old agent's stop can be refused once. That wake fails once; the next request works.
amadan