| 1 | # 🤖 Awake-instance convergence: diagnosis + fix |
| 2 | |
| 3 | **Status: IMPLEMENTED (sweep-driven).** This documents a production gap — awake |
| 4 | hibernate-managed instances kept serving their old binary after a release ships |
| 5 | — and the fix that closed it. The fix landed as a fourth staleness case in the |
| 6 | activator's existing idle sweep (`activator.go`, `sweepOnce`); it needed **no |
| 7 | new router control channel**, no updater changes, and no change to the deploy |
| 8 | path. The sections below keep the original diagnosis, then describe what was |
| 9 | built and why it differs from the first proposal. |
| 10 | |
| 11 | ## The symptom |
| 12 | |
| 13 | After `release-server.sh ship` (or the hourly `lchat update run`) repoints an |
| 14 | instance's `live/<host>` symlink to a new binary: |
| 15 | |
| 16 | - `lchat instances` / `lchat update status` show the **new** version. |
| 17 | - But `/api/version` over the instance's socket returns the **old** build. |
| 18 | |
| 19 | Hibernating instances converge on their next wake; **awake** ones (anything |
| 20 | with an open WebSocket or recent traffic — notably every bot-sidecar-bearing |
| 21 | space, which never sleeps) serve the stale binary indefinitely. The reporting |
| 22 | surfaces all read the symlink target, so the fleet *looks* converged while it |
| 23 | isn't. (Confirmed live 2026-07-13: after promoting a release, `family`, |
| 24 | `test`, and `team` each served a different older build than the symlink named.) |
| 25 | |
| 26 | ## Why — the exact mechanism |
| 27 | |
| 28 | An awake hibernate-flagged instance is a **child process of the router**, not a |
| 29 | `systemd` unit (`activator.go`: `activatorSvc.entries[host].proc`, an |
| 30 | `*exec.Cmd` via `childProc`). It's spawned once by `s3Backend.start`, which |
| 31 | execs `liveLink(host)` = `/opt/lchat/live/<host>` → `versions/<ver>/lchat`. **A |
| 32 | running child never re-reads the symlink** — that's the root of the gap. |
| 33 | |
| 34 | The update path (`updater.go updatePass`) has two branches: |
| 35 | |
| 36 | - **Unit-managed hosts** repoint the symlink *and* `systemctl restart` the unit |
| 37 | (`deployHostVersion` in `control.go`) — so they converge immediately. |
| 38 | - **Hibernating hosts** (`updater.go` phase 2) do **only** `pointLive` — a pure |
| 39 | symlink repoint, no signal to the running process. The code comment is |
| 40 | explicit: *"a currently-AWAKE child keeps its old binary until it next |
| 41 | sleeps (a running process doesn't follow the symlink)."* |
| 42 | |
| 43 | `update status` (`runUpdateStatus`) and `lchat instances` both build their |
| 44 | "current" column from `liveVersion(host)` (`os.Readlink`), never from the |
| 45 | child's `/api/version` — hence the false "converged" reading. |
| 46 | |
| 47 | ### What already exists (and what doesn't) |
| 48 | |
| 49 | - **#778** added the whole hibernating-repoint phase, `hibernatingPrefs` |
| 50 | (parked-DB read), WAL-ownership repair, and the narrow `setVersions`. |
| 51 | - **#774** added parked-DB pref reads to `update status`. |
| 52 | - The "compare symlinks, never restart awake children" behaviour is |
| 53 | `updater.go`'s symlink comparison + `deploy`'s hibernating branch — not a |
| 54 | separate function. |
| 55 | |
| 56 | **What does NOT exist anywhere:** a primitive that cycles an *awake* activator |
| 57 | child onto the new symlink. `lchat deploy <host> --version` also only repoints |
| 58 | for hibernating hosts (`control.go`, the `if rt.Hibernate` branch is a pure |
| 59 | `pointLive`). The only graceful cycle today is the heavyweight manual |
| 60 | `hibernateOff` → update → `hibernateOn` flip documented in |
| 61 | `docs/server-releases.md`. |
| 62 | |
| 63 | ## Why awake children aren't restarted (the design tension) |
| 64 | |
| 65 | This is deliberate, for two real reasons: |
| 66 | |
| 67 | 1. **Dropped connections.** An instance is kept awake *by* in-flight work, |
| 68 | including open WebSockets (idle = zero in-flight AND elapsed idle window, |
| 69 | `activator.go`). Cycling an awake child cuts live WS/requests. |
| 70 | 2. **The single-writer S3 lease.** Each wake acquires a per-host Litestream |
| 71 | lease; a naive external `pkill` races the lease and can 502 within the |
| 72 | router's dial-retry window (the sweep only reaps every `Idle/4`, ≤30s). |
| 73 | |
| 74 | So "never converges" is the current cost of "never drop a live connection." |
| 75 | |
| 76 | ## The safe primitive that already exists |
| 77 | |
| 78 | The activator's own stop **is** graceful: `childProc.stop` sends `SIGTERM` |
| 79 | (the instance's 10s drain), `SIGKILL` only on overstay; `beginStopLocked` → |
| 80 | `stop` takes the final S3 checkpoint, releases the lease, stops the sidecar |
| 81 | first, and deletes the entry — leaving the host **asleep**. The next request or |
| 82 | due-wake re-execs `liveLink(host)`, which now resolves to the **new** binary. |
| 83 | So a respawn-onto-new-symlink path exists; it's just **lazy (next request), not |
| 84 | triggered by a ship.** |
| 85 | |
| 86 | ## The fix as built (smallest safe change) |
| 87 | |
| 88 | The convergence check is a **fourth staleness case in the sweep** the activator |
| 89 | already runs (`sweepOnce`, every `Idle/4`, capped 30s). The sweep already reaps |
| 90 | awake children for three reasons — lost lease, dead child, idle — each via the |
| 91 | graceful `beginStopLocked`. Convergence is the fourth: |
| 92 | |
| 93 | 1. **Record the spawned build.** The backend's `start` reads `liveVersion(host)` |
| 94 | **immediately before `cmd.Start()`** and returns it, so `spawnVersion` names |
| 95 | the build the kernel actually resolved the symlink to — not a value re-read |
| 96 | later in `wake()`, which a ship landing during the up-to-30s health wait |
| 97 | could make disagree with the running binary. A read error leaves it empty, |
| 98 | which disables the check for that entry (fail-safe: never cycle on a version |
| 99 | we couldn't read). |
| 100 | 2. **Notice the drift, then wait out a grace.** Each sweep compares the live |
| 101 | symlink against `spawnVersion` (the pure `convergeDecision` helper holds this |
| 102 | state machine, so it's unit-tested directly). On first divergence it stamps |
| 103 | `staleSince`; only once the drift has persisted `convergeGrace` (90s) does it |
| 104 | cycle. The grace means a symlink that moved then rolled straight back (a bad |
| 105 | promote reverted) never drops a connection, and a briefly-busy space gets a |
| 106 | chance to idle-hibernate — which converges for free — first. A transient |
| 107 | symlink-read error is treated as "unknown" — it neither cycles nor resets the |
| 108 | accrued grace. |
| 109 | 3. **Cycle gracefully, one per tick.** Past the grace it calls the same |
| 110 | `beginStopLocked` the other three cases use: sidecar stopped first, SIGTERM |
| 111 | drain, final S3 checkpoint, lease released, entry deleted → **asleep**. A |
| 112 | subsequent **member request** — or the **due-time index** for a space with a |
| 113 | scheduled charter — then wakes it, re-execing `liveLink(host)` = the new |
| 114 | binary. (The sidecar itself is SIGKILLed during the stop and can't self-wake; |
| 115 | a bot space with no members present and no scheduled work simply stays asleep |
| 116 | until someone arrives, which is correct hibernation.) At most |
| 117 | `maxConvergeCyclesPerSweep` (1) per tick, so a fleet-wide ship staggers across |
| 118 | sweeps instead of blipping every space at once. |
| 119 | |
| 120 | Unlike idle-hibernation (which only fires at `inflight == 0`), this cycle can |
| 121 | fire with a request in flight — that's deliberate, since the always-awake spaces |
| 122 | it targets never reach `inflight == 0`. The graceful stop drains in-flight HTTP |
| 123 | just as `systemctl restart` does, so the only thing cut is a request still |
| 124 | running past the instance's ~10s drain (a large upload) — identical to a unit |
| 125 | restart landing mid-request, not a new failure mode. |
| 126 | |
| 127 | No repoint-site call, no control channel: the updater/deploy still just |
| 128 | `pointLive`, and the router's own sweep picks up the drift within ≤30s. This |
| 129 | **reuses** the graceful stop and the lazy wake-re-execs-the-symlink path — it |
| 130 | extends the existing convergence machinery rather than adding a new one. |
| 131 | |
| 132 | ### Exclusions (all automatic — no special-casing) |
| 133 | |
| 134 | - **Pinned canaries.** The updater never repoints a pinned host's symlink, so |
| 135 | `spawnVersion` keeps matching and the check never fires. Honouring the pin |
| 136 | needs no code here — it falls out of "cycle only when the symlink moved." |
| 137 | - **Unit-managed (non-hibernating) hosts.** Not activator children — they have |
| 138 | no `entries[host]`, so the sweep never sees them; they converge via |
| 139 | `systemctl restart` as before. |
| 140 | - **Bot-sidecar spaces are the PRIMARY target.** They never idle, so the idle |
| 141 | case never converges them; this cycle does, and the sidecar is stopped-first |
| 142 | and re-spawned on wake, so it's sidecar-safe. This automates exactly what the |
| 143 | manual `hibernate --off`/`--on` flip did by hand. |
| 144 | |
| 145 | ## Why this shape, not the first proposal |
| 146 | |
| 147 | The original proposal (kept in git history) added a per-host `cycle` verb to a |
| 148 | new router control socket and gated on `inflight == 0`. Both were dropped: |
| 149 | |
| 150 | - **No control channel.** A push from the repoint site would have added the |
| 151 | first per-host *command* into the live router — new attack surface on the |
| 152 | process fronting every space. The sweep already wakes hosts on a timer, so a |
| 153 | ≤30s lazy pull needs none of that. The convergence latency (seconds to a |
| 154 | couple of minutes) is irrelevant given the 72h supply-chain bake window. |
| 155 | - **Grace window, not `inflight == 0`.** An `inflight == 0` gate would leave the |
| 156 | *primary* target — always-awake bot-sidecar spaces, whose persistent |
| 157 | WebSocket keeps `inflight > 0` forever — stale indefinitely, defeating the |
| 158 | point. Cycling them requires a brief connection drop, which the graceful |
| 159 | SIGTERM drain + the router's `actStopping` connection-shed makes clean: |
| 160 | clients reconnect onto the new binary, exactly as they do across a normal |
| 161 | unit `systemctl restart`. The grace window (plus one-per-tick) is what keeps |
| 162 | it gentle — a space that goes quiet converges via idle-hibernate first, and |
| 163 | only a persistently-awake space is force-cycled, once, per ship. |
| 164 | |
| 165 | This is a *latency/observability* fix, not a correctness emergency: the bake |
| 166 | window holds releases 72h and the lazy next-wake path already converged |
| 167 | sleeping spaces. The gap was the always-awake spaces and the dishonest `status` |
| 168 | column — both now closed (see below). |
| 169 | |
| 170 | ## Companion: honest `status` (already shipped) |
| 171 | |
| 172 | `lchat update status --probe` reads each instance's `/api/version` over its |
| 173 | socket and flags an awake-stale instance (⚠) where the old symlink-only column |
| 174 | read "current". That shipped first as a small, safe, independent step; this |
| 175 | sweep fix removes the underlying staleness the probe surfaces. The manual |
| 176 | `lchat hibernate <host> --off` → update → `--on` flip still works for forcing an |
| 177 | immediate converge, but is no longer needed for routine ships. |
| 178 | |
amadan