🤖 Awake-instance convergence: diagnosis + fix

Status: IMPLEMENTED (sweep-driven). This documents a production gap — awake hibernate-managed instances kept serving their old binary after a release ships — and the fix that closed it. The fix landed as a fourth staleness case in the activator's existing idle sweep (activator.go, sweepOnce); it needed no new router control channel, no updater changes, and no change to the deploy path. The sections below keep the original diagnosis, then describe what was built and why it differs from the first proposal.

The symptom

After release-server.sh ship (or the hourly lchat update run) repoints an instance's live/<host> symlink to a new binary:

• lchat instances / lchat update status show the new version.
• But /api/version over the instance's socket returns the old build.

Hibernating instances converge on their next wake; awake ones (anything with an open WebSocket or recent traffic — notably every bot-sidecar-bearing space, which never sleeps) serve the stale binary indefinitely. The reporting surfaces all read the symlink target, so the fleet looks converged while it isn't. (Confirmed live 2026-07-13: after promoting a release, family, test, and team each served a different older build than the symlink named.)

Why — the exact mechanism

An awake hibernate-flagged instance is a child process of the router, not a systemd unit (activator.go: activatorSvc.entries[host].proc, an *exec.Cmd via childProc). It's spawned once by s3Backend.start, which execs liveLink(host) = /opt/lchat/live/<host> → versions/<ver>/lchat. A running child never re-reads the symlink — that's the root of the gap.

The update path (updater.go updatePass) has two branches:

• Unit-managed hosts repoint the symlink and systemctl restart the unit (deployHostVersion in control.go) — so they converge immediately.
• Hibernating hosts (updater.go phase 2) do only pointLive — a pure symlink repoint, no signal to the running process. The code comment is explicit: "a currently-AWAKE child keeps its old binary until it next sleeps (a running process doesn't follow the symlink)."

update status (runUpdateStatus) and lchat instances both build their "current" column from liveVersion(host) (os.Readlink), never from the child's /api/version — hence the false "converged" reading.

What already exists (and what doesn't)

• #778 added the whole hibernating-repoint phase, hibernatingPrefs (parked-DB read), WAL-ownership repair, and the narrow setVersions.
• #774 added parked-DB pref reads to update status.
• The "compare symlinks, never restart awake children" behaviour is updater.go's symlink comparison + deploy's hibernating branch — not a separate function.

What does NOT exist anywhere: a primitive that cycles an awake activator child onto the new symlink. lchat deploy <host> --version also only repoints for hibernating hosts (control.go, the if rt.Hibernate branch is a pure pointLive). The only graceful cycle today is the heavyweight manual hibernateOff → update → hibernateOn flip documented in docs/server-releases.md.

Why awake children aren't restarted (the design tension)

This is deliberate, for two real reasons:

1. Dropped connections. An instance is kept awake by in-flight work, including open WebSockets (idle = zero in-flight AND elapsed idle window, activator.go). Cycling an awake child cuts live WS/requests.
2. The single-writer S3 lease. Each wake acquires a per-host Litestream lease; a naive external pkill races the lease and can 502 within the router's dial-retry window (the sweep only reaps every Idle/4, ≤30s).

So "never converges" is the current cost of "never drop a live connection."

The safe primitive that already exists

The activator's own stop is graceful: childProc.stop sends SIGTERM (the instance's 10s drain), SIGKILL only on overstay; beginStopLocked → stop takes the final S3 checkpoint, releases the lease, stops the sidecar first, and deletes the entry — leaving the host asleep. The next request or due-wake re-execs liveLink(host), which now resolves to the new binary. So a respawn-onto-new-symlink path exists; it's just lazy (next request), not triggered by a ship.

The fix as built (smallest safe change)

The convergence check is a fourth staleness case in the sweep the activator already runs (sweepOnce, every Idle/4, capped 30s). The sweep already reaps awake children for three reasons — lost lease, dead child, idle — each via the graceful beginStopLocked. Convergence is the fourth:

1. Record the spawned build. The backend's start reads liveVersion(host) immediately before cmd.Start() and returns it, so spawnVersion names the build the kernel actually resolved the symlink to — not a value re-read later in wake(), which a ship landing during the up-to-30s health wait could make disagree with the running binary. A read error leaves it empty, which disables the check for that entry (fail-safe: never cycle on a version we couldn't read).
2. Notice the drift, then wait out a grace. Each sweep compares the live symlink against spawnVersion (the pure convergeDecision helper holds this state machine, so it's unit-tested directly). On first divergence it stamps staleSince; only once the drift has persisted convergeGrace (90s) does it cycle. The grace means a symlink that moved then rolled straight back (a bad promote reverted) never drops a connection, and a briefly-busy space gets a chance to idle-hibernate — which converges for free — first. A transient symlink-read error is treated as "unknown" — it neither cycles nor resets the accrued grace.
3. Cycle gracefully, one per tick. Past the grace it calls the same beginStopLocked the other three cases use: sidecar stopped first, SIGTERM drain, final S3 checkpoint, lease released, entry deleted → asleep. A subsequent member request — or the due-time index for a space with a scheduled charter — then wakes it, re-execing liveLink(host) = the new binary. (The sidecar itself is SIGKILLed during the stop and can't self-wake; a bot space with no members present and no scheduled work simply stays asleep until someone arrives, which is correct hibernation.) At most maxConvergeCyclesPerSweep (1) per tick, so a fleet-wide ship staggers across sweeps instead of blipping every space at once.

Unlike idle-hibernation (which only fires at inflight == 0), this cycle can fire with a request in flight — that's deliberate, since the always-awake spaces it targets never reach inflight == 0. The graceful stop drains in-flight HTTP just as systemctl restart does, so the only thing cut is a request still running past the instance's ~10s drain (a large upload) — identical to a unit restart landing mid-request, not a new failure mode.

No repoint-site call, no control channel: the updater/deploy still just pointLive, and the router's own sweep picks up the drift within ≤30s. This reuses the graceful stop and the lazy wake-re-execs-the-symlink path — it extends the existing convergence machinery rather than adding a new one.

Exclusions (all automatic — no special-casing)

• Pinned canaries. The updater never repoints a pinned host's symlink, so spawnVersion keeps matching and the check never fires. Honouring the pin needs no code here — it falls out of "cycle only when the symlink moved."
• Unit-managed (non-hibernating) hosts. Not activator children — they have no entries[host], so the sweep never sees them; they converge via systemctl restart as before.
• Bot-sidecar spaces are the PRIMARY target. They never idle, so the idle case never converges them; this cycle does, and the sidecar is stopped-first and re-spawned on wake, so it's sidecar-safe. This automates exactly what the manual hibernate --off/--on flip did by hand.

Why this shape, not the first proposal

The original proposal (kept in git history) added a per-host cycle verb to a new router control socket and gated on inflight == 0. Both were dropped:

• No control channel. A push from the repoint site would have added the first per-host command into the live router — new attack surface on the process fronting every space. The sweep already wakes hosts on a timer, so a ≤30s lazy pull needs none of that. The convergence latency (seconds to a couple of minutes) is irrelevant given the 72h supply-chain bake window.
• Grace window, not inflight == 0. An inflight == 0 gate would leave the primary target — always-awake bot-sidecar spaces, whose persistent WebSocket keeps inflight > 0 forever — stale indefinitely, defeating the point. Cycling them requires a brief connection drop, which the graceful SIGTERM drain + the router's actStopping connection-shed makes clean: clients reconnect onto the new binary, exactly as they do across a normal unit systemctl restart. The grace window (plus one-per-tick) is what keeps it gentle — a space that goes quiet converges via idle-hibernate first, and only a persistently-awake space is force-cycled, once, per ship.

This is a latency/observability fix, not a correctness emergency: the bake window holds releases 72h and the lazy next-wake path already converged sleeping spaces. The gap was the always-awake spaces and the dishonest status column — both now closed (see below).

Companion: honest status (already shipped)

lchat update status --probe reads each instance's /api/version over its socket and flags an awake-stale instance (⚠) where the old symlink-only column read "current". That shipped first as a small, safe, independent step; this sweep fix removes the underlying staleness the probe surfaces. The manual lchat hibernate <host> --off → update → --on flip still works for forcing an immediate converge, but is no longer needed for routine ships.