amadan

eleven / ocho Public

updater: control-binary self-update gets a health-check + auto-rollback (#1037)

* server: health-check + auto-rollback the control binary's self-update

controlSelfUpdate swapped /opt/lchat/lchat in place and restarted
lchat-control/lchat-cloud with zero verification — a boot-broken control
build would brick the box's whole control plane, and with root SSH gone no
operator could reach in to fix it by hand.

Now the swap is proven before it's committed:
- controld serves an unauthenticated, state-free GET /healthz on the
  root-only control socket, reporting {"build": buildVersion}.
- waitControlHealthy polls it (mirrors waitHealthyVersion), tolerant of the
  connection-refused flapping a Restart=always crash-looper produces.
- After the swap + restart, controlSelfUpdate confirms the new build; if it
  never comes up it restores .prev, restarts again, and confirms recovery —
  screaming a distinct CONTROL PLANE BRICKED error (no loop) if even the
  rollback is unhealthy. The router is never restarted (unchanged posture).
- The verified build is also staged into the version store so gcVersions
  protects it and rollback/re-deploy can reach it; .prev only ever advances
  to a health-confirmed build.

controlSelfUpdate now returns an error (updatePass logs it, never fails the
pass on it). New tests cover healthy-commit, rollback-on-unhealthy, and the
brick case; the existing swap test keeps pinning "router never restarts".

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* server: harden control self-update health gate (F2–F5 review fixes)

Adversarial review of the health-check + rollback found four gaps:

- F2: the gate rolled back a GOOD binary when lchat-control wasn't running —
  try-restart is a no-op on an inactive unit, so /healthz could never answer
  and every pass would roll back + scream CONTROL PLANE BRICKED. Now: if the
  probe fails, consult `isActive(lchat-control)`. Active → probe authoritative,
  roll back. Inactive → fall back to a controld-independent `<self> version`
  integrity check (controlBinaryReports); it passing commits, failing rolls
  back. Same strong/weak gate on the rollback recovery check.
- F3: a bad head was retried hourly. Persist a quarantine file under optDir
  (readFailedControlVersion/writeFailedControlVersion/clearFailedControlVersion,
  mirroring the manifest stamp). Written on rollback, short-circuits the next
  pass, cleared on a healthy commit or when the channel moves on. The rollback
  log now honestly says "quarantined".
- F4: thread the in-pass `poisoned` set into controlSelfUpdate — a build that
  failed a health check on an instance this pass is skipped for the control
  binary too, making the long-standing comment true.
- F5: after a healthy commit, warn (log only, no rollback) if lchat-cloud
  didn't come back active.

F1 (residual risk: a crash inside the swap→health window leaves the unproven
new binary with no auto-rollback until a human renames .prev; future fix is a
boot-time ExecStartPre recovery) is documented in the controlSelfUpdate comment
only, deliberately not implemented (it touches the ingress start path).

isActive + controlBinaryReports are vars so tests fake unit state and the
version exec. New tests: InactiveControldCommits (F2), QuarantinesFailedHead
(F3), SkipsPoisonedHead (F4). Router-never-restarted stays pinned throughout.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Paul Campbell d218ac03d89be8b4e5ba6e48ffdb2f4c17c6a923 parent eb4110f
3 files changed, +540 −38
  • control_selfupdate_test.go +253 −11
  • controld.go +24 −1
  • updater.go +263 −26
diff --git a/control_selfupdate_test.go b/control_selfupdate_test.go
index e62ab2d..6159ecc 100644
--- a/control_selfupdate_test.go
+++ b/control_selfupdate_test.go
@@ -3,25 +3,36 @@ package main
import (
"crypto/sha256"
"encoding/hex"
+ "fmt"
+ "net"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
+ "sync"
"testing"
"time"
)
// selfUpdateEnv points every seam at temp doubles: a fake control binary on
-// the same temp "filesystem" the download lands on, a fresh bake state, and a
-// recording systemctl.
+// the same temp "filesystem" the download lands on, a fresh bake state, a
+// recording systemctl, and the health probe aimed at a temp socket path (which
+// a test binds via fakeControlSocket, or leaves unbound to model a refused
+// dial). deployHealthTimeout is shrunk so an unhealthy path fails in ~a beat.
+// By default isActive reports controld UP (so the live /healthz probe is the
+// authoritative gate, the pre-F2 behaviour); tests exercising the
+// inactive-controld fallback override isActive + controlBinaryReports.
func selfUpdateEnv(t *testing.T, newBinary []byte) (self string, entry releaseEntry, calls *[]string) {
t.Helper()
oldOpt, oldSelf, oldSysctl, oldChannel := optDir, selfPath, systemctl, os.Getenv("LCHAT_CONTROL_CHANNEL")
- oldBuild := buildVersion
+ oldBuild, oldSock, oldTimeout := buildVersion, controlSocketPath, deployHealthTimeout
+ oldActive, oldBinReports := isActive, controlBinaryReports
buildVersion = "testold1" // the dev-build guard must not fire in these tests
t.Cleanup(func() {
optDir, selfPath, systemctl, buildVersion = oldOpt, oldSelf, oldSysctl, oldBuild
+ controlSocketPath, deployHealthTimeout = oldSock, oldTimeout
+ isActive, controlBinaryReports = oldActive, oldBinReports
os.Setenv("LCHAT_CONTROL_CHANNEL", oldChannel)
})
optDir = t.TempDir()
@@ -38,6 +49,24 @@ func selfUpdateEnv(t *testing.T, newBinary []byte) (self string, entry releaseEn
}
os.Setenv("LCHAT_CONTROL_CHANNEL", "stable")
+ // Default seams: controld is up (the /healthz probe is the gate), and the
+ // `version` integrity check is a poisoned stub — a test that expects the
+ // inactive-controld fallback must stub both, or this loud error trips.
+ isActive = func(string) bool { return true }
+ controlBinaryReports = func(string, string) error {
+ return fmt.Errorf("controlBinaryReports not stubbed for this test")
+ }
+
+ // The health probe dials a unix socket, whose sun_path is short-limited — a
+ // dedicated short temp dir, not optDir (which can be a long macOS path).
+ sockDir, err := os.MkdirTemp("", "lch")
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() { os.RemoveAll(sockDir) })
+ controlSocketPath = filepath.Join(sockDir, "c.sock")
+ deployHealthTimeout = 400 * time.Millisecond
+
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Write(newBinary)
}))
@@ -48,22 +77,58 @@ func selfUpdateEnv(t *testing.T, newBinary []byte) (self string, entry releaseEn
return self, entry, calls
}
-// The swap end to end: verified download, atomic install over the running
-// path, a .prev rollback beside it, and the backend services bounced —
-// router deliberately untouched.
+// fakeControlSocket binds a unix socket at controlSocketPath that answers
+// GET /healthz with a build — the same shape controld's real /healthz serves —
+// so waitControlHealthy sees a healthy new build or a stale old one. It returns
+// a setter so a test can flip the reported build mid-run (e.g. old→new after a
+// channel moves). A test that wants a REFUSED dial (a crash-looping binary)
+// simply doesn't call this, leaving the path unbound.
+func fakeControlSocket(t *testing.T, build string) (set func(string)) {
+ t.Helper()
+ var mu sync.Mutex
+ cur := build
+ set = func(b string) { mu.Lock(); cur = b; mu.Unlock() }
+ mux := http.NewServeMux()
+ mux.HandleFunc("GET /healthz", func(w http.ResponseWriter, _ *http.Request) {
+ mu.Lock()
+ b := cur
+ mu.Unlock()
+ writeJSON(w, http.StatusOK, map[string]any{"build": b})
+ })
+ ln, err := net.Listen("unix", controlSocketPath)
+ if err != nil {
+ t.Fatal(err)
+ }
+ srv := &http.Server{Handler: mux}
+ go srv.Serve(ln)
+ t.Cleanup(func() { srv.Close() })
+ return set
+}
+
+// The swap end to end: verified download, staged into the version store, atomic
+// install over the running path, a .prev rollback beside it, and the backend
+// services bounced — router deliberately untouched. The control socket reports
+// the new build, so the health probe passes and the swap commits.
func TestControlSelfUpdateSwaps(t *testing.T) {
newBin := []byte("NEW-CONTROL-BINARY")
self, entry, calls := selfUpdateEnv(t, newBin)
+ fakeControlSocket(t, entry.Version) // healthy on the new build
m := &releaseManifest{Schema: 1, GeneratedAt: time.Now(), Channels: map[string]releaseEntry{"stable": entry}}
// ignoreBake=true is the deliberate override path; the bake-hold path is
// pinned separately below.
- controlSelfUpdate(m, loadFirstSeen(), false, true)
+ if err := controlSelfUpdate(m, loadFirstSeen(), nil, false, true); err != nil {
+ t.Fatalf("healthy swap returned error: %v", err)
+ }
got, err := os.ReadFile(self)
if err != nil || string(got) != string(newBin) {
t.Fatalf("control binary not swapped: %q %v", got, err)
}
+ // Staged into the store too, so gcVersions protects it and rollback can reach it.
+ if staged, err := os.ReadFile(versionBinPath(entry.Version)); err != nil || string(staged) != string(newBin) {
+ t.Fatalf("new build not staged into the version store: %q %v", staged, err)
+ }
prev, err := os.ReadFile(self + ".prev-" + buildVersion)
if err != nil || string(prev) != "OLD-CONTROL-BINARY" {
t.Fatalf("rollback copy missing/wrong: %q %v", prev, err)
@@ -77,6 +142,183 @@ func TestControlSelfUpdateSwaps(t *testing.T) {
}
}
+// The happy path in isolation: a healthy new build commits, does NOT roll back,
+// bounces the backends exactly once (two calls, not four), and returns nil.
+func TestControlSelfUpdateHealthyCommits(t *testing.T) {
+ newBin := []byte("NEW-CONTROL-BINARY")
+ self, entry, calls := selfUpdateEnv(t, newBin)
+ fakeControlSocket(t, entry.Version)
+ m := &releaseManifest{Schema: 1, GeneratedAt: time.Now(), Channels: map[string]releaseEntry{"stable": entry}}
+
+ if err := controlSelfUpdate(m, loadFirstSeen(), nil, false, true); err != nil {
+ t.Fatalf("healthy commit returned error: %v", err)
+ }
+ if got, _ := os.ReadFile(self); string(got) != string(newBin) {
+ t.Fatalf("healthy build should stay swapped in: %q", got)
+ }
+ // One restart of each backend, never a second (rollback) round.
+ if n := strings.Count(strings.Join(*calls, ";"), "try-restart"); n != 2 {
+ t.Fatalf("expected exactly 2 restarts (no rollback), got %d: %v", n, *calls)
+ }
+}
+
+// The whole point: a swapped-in build that never reports its version is rolled
+// back. The control socket reports the OLD build (a restart that didn't take),
+// so the new-build probe times out; the binary on disk is restored to .prev's
+// content, the backends are restarted a SECOND time, and the function errors.
+func TestControlSelfUpdateRollsBackOnUnhealthy(t *testing.T) {
+ newBin := []byte("NEW-CONTROL-BINARY")
+ self, entry, calls := selfUpdateEnv(t, newBin)
+ fakeControlSocket(t, buildVersion) // stale: reports the OLD build after the swap
+ m := &releaseManifest{Schema: 1, GeneratedAt: time.Now(), Channels: map[string]releaseEntry{"stable": entry}}
+
+ err := controlSelfUpdate(m, loadFirstSeen(), nil, false, true)
+ if err == nil {
+ t.Fatal("an unhealthy swap must return an error")
+ }
+ if got, _ := os.ReadFile(self); string(got) != "OLD-CONTROL-BINARY" {
+ t.Fatalf("binary not restored to the last-known-good: %q", got)
+ }
+ joined := strings.Join(*calls, ";")
+ if n := strings.Count(joined, "try-restart lchat-control.service"); n != 2 {
+ t.Fatalf("expected control restarted twice (swap + rollback), got %d: %v", n, *calls)
+ }
+ if strings.Contains(joined, "lchat-router") {
+ t.Fatalf("the router must never restart, even on rollback: %v", *calls)
+ }
+}
+
+// The genuine brick: the new build is unhealthy AND the rolled-back old build is
+// unhealthy too (nothing ever answers /healthz). The swap is still physically
+// reverted on disk, but a distinct, screaming error is surfaced — and it does
+// not loop.
+func TestControlSelfUpdateRollbackAlsoFailsScreams(t *testing.T) {
+ newBin := []byte("NEW-CONTROL-BINARY")
+ self, entry, _ := selfUpdateEnv(t, newBin)
+ // No fakeControlSocket at all → every /healthz dial is refused → both the
+ // post-swap and the post-rollback probes time out.
+ m := &releaseManifest{Schema: 1, GeneratedAt: time.Now(), Channels: map[string]releaseEntry{"stable": entry}}
+
+ err := controlSelfUpdate(m, loadFirstSeen(), nil, false, true)
+ if err == nil || !strings.Contains(err.Error(), "CONTROL PLANE BRICKED") {
+ t.Fatalf("expected a distinct BRICKED error, got: %v", err)
+ }
+ // The rollback rename is a pure file op, so it still restores the good binary
+ // even when the box won't come back up on it.
+ if got, _ := os.ReadFile(self); string(got) != "OLD-CONTROL-BINARY" {
+ t.Fatalf("rollback rename should still restore the old binary: %q", got)
+ }
+}
+
+// F2: when lchat-control isn't running, a failed /healthz probe is NOT
+// authoritative — try-restart was a no-op and nothing could answer. A
+// genuinely good binary (it execs and self-reports the target build via the
+// `version` integrity check) must COMMIT, not be falsely rolled back, and the
+// BRICKED path must not be taken. No socket is bound (the probe refuses).
+func TestControlSelfUpdateInactiveControldCommits(t *testing.T) {
+ newBin := []byte("NEW-CONTROL-BINARY")
+ self, entry, calls := selfUpdateEnv(t, newBin)
+ isActive = func(string) bool { return false } // controld not running
+ controlBinaryReports = func(_, want string) error {
+ if want == entry.Version {
+ return nil // the new binary loads + self-reports the target build
+ }
+ return fmt.Errorf("unexpected integrity target %q", want)
+ }
+ m := &releaseManifest{Schema: 1, GeneratedAt: time.Now(), Channels: map[string]releaseEntry{"stable": entry}}
+
+ if err := controlSelfUpdate(m, loadFirstSeen(), nil, false, true); err != nil {
+ t.Fatalf("inactive controld + good binary must commit, got error: %v", err)
+ }
+ if got, _ := os.ReadFile(self); string(got) != string(newBin) {
+ t.Fatalf("good binary must stay committed, not rolled back: %q", got)
+ }
+ joined := strings.Join(*calls, ";")
+ if n := strings.Count(joined, "try-restart"); n != 2 {
+ t.Fatalf("expected exactly 2 restarts (commit, no rollback), got %d: %v", n, *calls)
+ }
+ if strings.Contains(joined, "lchat-router") {
+ t.Fatalf("router must never restart: %v", *calls)
+ }
+ if got := readFailedControlVersion(); got != "" {
+ t.Fatalf("a committed build must not be quarantined: %q", got)
+ }
+}
+
+// F3: a head that fails its health check is quarantined (persisted under
+// optDir), so the next hourly pass short-circuits it instead of reinstalling +
+// rolling it back forever — and the quarantine lifts when the channel moves to
+// a good build.
+func TestControlSelfUpdateQuarantinesFailedHead(t *testing.T) {
+ newBin := []byte("NEW-CONTROL-BINARY")
+ self, entry, calls := selfUpdateEnv(t, newBin)
+ setBuild := fakeControlSocket(t, buildVersion) // stale: reports OLD after the swap
+ m := &releaseManifest{Schema: 1, GeneratedAt: time.Now(), Channels: map[string]releaseEntry{"stable": entry}}
+
+ // First pass: the head fails its health check, is rolled back and quarantined.
+ if err := controlSelfUpdate(m, loadFirstSeen(), nil, false, true); err == nil {
+ t.Fatal("an unhealthy head must return an error")
+ }
+ if got := readFailedControlVersion(); got != entry.Version {
+ t.Fatalf("failed head not quarantined: %q", got)
+ }
+ if got, _ := os.ReadFile(self); string(got) != "OLD-CONTROL-BINARY" {
+ t.Fatalf("binary not rolled back: %q", got)
+ }
+
+ // Second pass on the SAME head: the quarantine short-circuits it — no
+ // download, no swap, no restarts.
+ before := len(*calls)
+ if err := controlSelfUpdate(m, loadFirstSeen(), nil, false, true); err != nil {
+ t.Fatalf("quarantined head should be a silent no-op, got: %v", err)
+ }
+ if len(*calls) != before {
+ t.Fatalf("quarantined head must not restart anything: %v", (*calls)[before:])
+ }
+ if got, _ := os.ReadFile(self); string(got) != "OLD-CONTROL-BINARY" {
+ t.Fatalf("quarantined head must not swap: %q", got)
+ }
+
+ // The channel moves to a DIFFERENT, healthy head: the quarantine clears and
+ // the new build commits.
+ e2 := entry
+ e2.Version = "def5678" // same download bytes + sha, new version id
+ setBuild(e2.Version) // socket now reports the new build → healthy
+ m2 := &releaseManifest{Schema: 1, GeneratedAt: time.Now(), Channels: map[string]releaseEntry{"stable": e2}}
+ if err := controlSelfUpdate(m2, loadFirstSeen(), nil, false, true); err != nil {
+ t.Fatalf("healthy new head should commit, got: %v", err)
+ }
+ if got := readFailedControlVersion(); got != "" {
+ t.Fatalf("quarantine should be cleared after the channel moved on, still %q", got)
+ }
+ if got, _ := os.ReadFile(self); string(got) != string(newBin) {
+ t.Fatalf("new head not committed: %q", got)
+ }
+}
+
+// F4: a head that failed a health check on an INSTANCE this pass (in the
+// poisoned set) must never then become the control plane — no download, no
+// swap, no quarantine write.
+func TestControlSelfUpdateSkipsPoisonedHead(t *testing.T) {
+ newBin := []byte("NEW-CONTROL-BINARY")
+ self, entry, calls := selfUpdateEnv(t, newBin)
+ m := &releaseManifest{Schema: 1, GeneratedAt: time.Now(), Channels: map[string]releaseEntry{"stable": entry}}
+
+ poisoned := map[string]bool{entry.Version: true}
+ if err := controlSelfUpdate(m, loadFirstSeen(), poisoned, false, true); err != nil {
+ t.Fatalf("a poisoned head should be a silent no-op, got: %v", err)
+ }
+ if got, _ := os.ReadFile(self); string(got) != "OLD-CONTROL-BINARY" {
+ t.Fatalf("poisoned head must not swap: %q", got)
+ }
+ if len(*calls) != 0 {
+ t.Fatalf("poisoned head must not restart anything: %v", *calls)
+ }
+ if got := readFailedControlVersion(); got != "" {
+ t.Fatalf("skipping a poisoned head must not write a quarantine: %q", got)
+ }
+}
+
// A head still inside its bake window is held — the trust plane is the LAST
// thing on a box that should take a new build.
func TestControlSelfUpdateHonorsBake(t *testing.T) {
@@ -85,7 +327,7 @@ func TestControlSelfUpdateHonorsBake(t *testing.T) {
bake := loadFirstSeen()
bake.observe(m, time.Now()) // first sighting is now → inside the window
- controlSelfUpdate(m, bake, false, false)
+ controlSelfUpdate(m, bake, nil, false, false)
if got, _ := os.ReadFile(self); string(got) != "OLD-CONTROL-BINARY" {
t.Fatalf("baking head must not install: %q", got)
@@ -102,7 +344,7 @@ func TestControlSelfUpdateNoOpsAndBadSha(t *testing.T) {
self, entry, _ := selfUpdateEnv(t, []byte("NEW"))
os.Setenv("LCHAT_CONTROL_CHANNEL", "off")
- controlSelfUpdate(&releaseManifest{Channels: map[string]releaseEntry{"stable": entry}}, loadFirstSeen(), false, true)
+ controlSelfUpdate(&releaseManifest{Channels: map[string]releaseEntry{"stable": entry}}, loadFirstSeen(), nil, false, true)
if got, _ := os.ReadFile(self); string(got) != "OLD-CONTROL-BINARY" {
t.Fatal("off must be a no-op")
}
@@ -110,7 +352,7 @@ func TestControlSelfUpdateNoOpsAndBadSha(t *testing.T) {
same := entry
same.Version = buildVersion
- controlSelfUpdate(&releaseManifest{Channels: map[string]releaseEntry{"stable": same}}, loadFirstSeen(), false, true)
+ controlSelfUpdate(&releaseManifest{Channels: map[string]releaseEntry{"stable": same}}, loadFirstSeen(), nil, false, true)
if got, _ := os.ReadFile(self); string(got) != "OLD-CONTROL-BINARY" {
t.Fatal("at-head must be a no-op")
}
@@ -118,7 +360,7 @@ func TestControlSelfUpdateNoOpsAndBadSha(t *testing.T) {
bad := entry
bad.SHA256 = strings.Repeat("00", 32)
bad.ARM64SHA256 = strings.Repeat("00", 32) // whichever arch runs the test
- controlSelfUpdate(&releaseManifest{Channels: map[string]releaseEntry{"stable": bad}}, loadFirstSeen(), false, true)
+ controlSelfUpdate(&releaseManifest{Channels: map[string]releaseEntry{"stable": bad}}, loadFirstSeen(), nil, false, true)
if got, _ := os.ReadFile(self); string(got) != "OLD-CONTROL-BINARY" {
t.Fatal("a sha mismatch must never install")
}
diff --git a/controld.go b/controld.go
index 9dc71df..06b7412 100644
--- a/controld.go
+++ b/controld.go
@@ -38,6 +38,12 @@ import (
// the cloud unit by systemd, but never by anything running as the lchat user.
const controlTokenPath = "/etc/lchat/control.token"
+// controlSocketPath is the control daemon's unix socket: where controld listens
+// (runControld's -socket default below) AND where the control self-update dials
+// /healthz to confirm a freshly-swapped control binary came up. One source of
+// truth for both, and a var so tests can point the health probe at a temp socket.
+var controlSocketPath = envOr("LCHAT_CONTROL_SOCKET", "/run/lchat/control.sock")
+
// freeDiskBytes reports the free space where instance DBs live. A var so tests
// can simulate a full disk without filling one.
var freeDiskBytes = func(path string) (uint64, error) {
@@ -102,6 +108,15 @@ func writeControlErr(w http.ResponseWriter, status int, code, msg string) {
func (c *controld) handler() http.Handler {
mux := http.NewServeMux()
+ // Unauthenticated, state-free liveness+build probe. The box's own control
+ // self-update (controlSelfUpdate in updater.go) swaps THIS binary in place and
+ // then dials /healthz to confirm the new build actually came up before it
+ // commits — rolling the swap back if it can't. It has to answer the instant the
+ // HTTP listener binds, so it deliberately touches no DB/registry state a
+ // half-started process might still lack; it reveals only the compiled-in build
+ // string, and only over the root-only control socket. No token: the swap probe
+ // runs in the same already-root updater and there's nothing here worth gating.
+ mux.HandleFunc("GET /healthz", c.healthzHandler)
mux.HandleFunc("POST /instances", c.auth(c.createHandler))
mux.HandleFunc("DELETE /instances/{host}", c.auth(c.deleteHandler))
mux.HandleFunc("GET /instances/{host}/stats", c.auth(c.statsHandler))
@@ -113,6 +128,14 @@ func (c *controld) handler() http.Handler {
return mux
}
+// healthzHandler answers the control self-update's health probe with the running
+// control binary's build id. It touches nothing on c (no registry, no disk) so a
+// process that has only just bound its listener still answers — see the route
+// registration in handler() for the full why, including why it's unauthenticated.
+func (c *controld) healthzHandler(w http.ResponseWriter, _ *http.Request) {
+ writeJSON(w, http.StatusOK, map[string]any{"build": buildVersion})
+}
+
// replicationHandler records one instance's replication class on its registry
// route (the durability ladder — registry.go) and reloads the router so the
// next wake reads it. The class is wake-time config, so nothing is restarted
@@ -592,7 +615,7 @@ func bearerTokenHeader(r *http.Request) string {
func runControld(args []string) {
fs := flag.NewFlagSet("controld", flag.ExitOnError)
registryPath := fs.String("registry", envOr("LCHAT_REGISTRY", "/data/registry.db"), "router registry SQLite path")
- socketPath := fs.String("socket", envOr("LCHAT_CONTROL_SOCKET", "/run/lchat/control.sock"), "unix socket to serve the control API on")
+ socketPath := fs.String("socket", controlSocketPath, "unix socket to serve the control API on")
tokenFile := fs.String("token-file", envOr("LCHAT_CONTROL_TOKEN_FILE", controlTokenPath), "file holding the control bearer token")
maxInstances := fs.Int("max-instances", envInt("LCHAT_MAX_INSTANCES", 40), "refuse creation beyond this many instances on the box (materialized only under -hibernate-creates)")
minFreeGB := fs.Int("min-free-gb", envInt("LCHAT_MIN_FREE_GB", 5), "refuse creation when the data disk has less than this free (GB)")
diff --git a/updater.go b/updater.go
index 104df14..c66e977 100644
--- a/updater.go
+++ b/updater.go
@@ -553,7 +553,15 @@ func updatePass(reg *Registry, unitTmpl string, dryRun, ignoreBake bool) error {
// Last, the box's own trust plane: the control binary rides the same
// signed feed (see controlSelfUpdate) — after instances, so a build that
// poisons a health check this pass never also becomes the control plane.
- controlSelfUpdate(m, bake, dryRun, ignoreBake)
+ // A self-update failure is already logged loudly inside (it rolls itself
+ // back); surface the summary here but never fail the whole pass on it —
+ // the instances above already converged. Passing `poisoned` makes the
+ // "a build that poisons a health check this pass never also becomes the
+ // control plane" promise real: a head that just failed on an instance is
+ // skipped for the control binary too.
+ if err := controlSelfUpdate(m, bake, poisoned, dryRun, ignoreBake); err != nil {
+ fmt.Printf("update: control self-update: %v\n", err)
+ }
return nil
}
@@ -571,49 +579,100 @@ func updatePass(reg *Registry, unitTmpl string, dryRun, ignoreBake bool) error {
// box to take a new build — and "off" disables self-update entirely. The
// swap is an atomic same-filesystem rename over the existing path (running
// services keep their old inode until restarted; sudo never sees a missing
-// path), with the outgoing build kept beside it as .prev-<build> — the
-// rollback is one root rename, or a re-pin of the channel. lchat-control and
-// lchat-cloud are bounced to converge immediately (backend sockets, invisible
-// to members); the ROUTER is deliberately NOT restarted here — its front door
-// should never blip on a timer's schedule, so it runs the old build until an
-// operator restarts it (`lchat-svc restart lchat-router`), loudly noted below.
-func controlSelfUpdate(m *releaseManifest, bake *firstSeenState, dryRun, ignoreBake bool) {
+// path), with the outgoing build kept beside it as .prev-<build>.
+//
+// The swap is NOT committed blind: after bouncing lchat-control/lchat-cloud
+// onto the new binary we confirm the control plane reports the new build. The
+// strong gate is the control daemon's live /healthz (waitControlHealthy); when
+// lchat-control isn't running to answer it, we fall back to the weaker but
+// controld-independent `<self> version` integrity check (controlBinaryReports)
+// rather than roll back a binary the probe simply couldn't reach. If neither
+// gate passes we restore .prev and confirm the box recovered — a boot-broken
+// control plane rolling itself back is the whole point of this function, since
+// with root SSH gone an operator can't reach in to fix a bricked box by hand.
+// lchat-control and lchat-cloud are the only units bounced (backend sockets,
+// invisible to members); the ROUTER is deliberately NOT restarted here — its
+// front door should never blip on a timer's schedule, and it has no /healthz to
+// probe, so it runs the old build until an operator restarts it, noted below.
+//
+// A rolled-back head is QUARANTINED (writeFailedControlVersion, persisted under
+// optDir) so the hourly pass doesn't reinstall + roll it back forever; the
+// quarantine clears when a good build commits or the channel moves past it. A
+// head that poisoned an instance's health check THIS pass is skipped up front.
+//
+// Returns an error only when something actually went wrong (a pre-swap failure,
+// or a swap that was rolled back); every no-op path returns nil. updatePass
+// logs the summary but never fails the pass on it — the instances already
+// converged before this runs.
+//
+// RESIDUAL RISK (deliberately not addressed here): the swap → health-check
+// window is not crash-atomic. A reboot or kill in that window can leave the
+// loose /opt/lchat/lchat as the unproven NEW build with no process left to run
+// the rollback — so nothing auto-recovers until a human renames .prev back.
+// Closing it means a boot-time recovery (an ExecStartPre on lchat-router /
+// lchat-control that verifies the loose binary and reverts to .prev on doubt);
+// that's deferred because it touches the ingress start path, which we don't
+// want a timer-adjacent change to destabilise.
+func controlSelfUpdate(m *releaseManifest, bake *firstSeenState, poisoned map[string]bool, dryRun, ignoreBake bool) error {
if buildVersion == "dev" {
// An unstamped binary (local build, test binary) has no place in the
// channel ordering — self-updating it would clobber whatever the
// operator is hand-running. Only release-stamped builds converge.
- return
+ return nil
}
channel := envOr("LCHAT_CONTROL_CHANNEL", "stable")
if channel == "off" {
- return
+ return nil
}
e, ok := m.Channels[channel]
if !ok || e.Version == buildVersion {
- return
+ return nil
+ }
+ // The channel moved past a previously-quarantined head — forget the
+ // quarantine and let the new head be tried.
+ if failed := readFailedControlVersion(); failed != "" && failed != e.Version {
+ clearFailedControlVersion()
+ }
+ // A head we already swapped in, health-checked, and rolled back stays
+ // quarantined until a human pins/patches the channel or it moves on — never
+ // reinstalled + rolled back every hour.
+ if e.Version == readFailedControlVersion() {
+ fmt.Printf("update: control build %s is quarantined (failed its health check; pin or patch the channel to move)\n", e.Version)
+ return nil
+ }
+ // A build that failed a health check on an instance THIS pass must never
+ // then become the control plane (updatePass runs us after the instances).
+ if poisoned[e.Version] {
+ fmt.Printf("update: skipping control update to %s — it failed a health check on an instance this pass\n", e.Version)
+ return nil
}
if hold := bakeHold(bake, channel, e.Version, ignoreBake, time.Now()); hold != "" {
fmt.Printf("update: control binary %s -> %s: %s\n", buildVersion, e.Version, hold)
- return
+ return nil
}
if dryRun {
fmt.Printf("update: control binary would update %s -> %s (%s; dry run)\n", buildVersion, e.Version, channel)
- return
+ return nil
}
self, err := selfPath()
if err != nil {
- fmt.Printf("update: control self-update: %v\n", err)
- return
+ return err
}
tmp, err := downloadReleaseBinary(e)
if err != nil {
- fmt.Printf("update: control self-update: %v\n", err)
- return
+ return err
}
defer os.Remove(tmp)
if err := os.Chmod(tmp, 0o755); err != nil {
- fmt.Printf("update: control self-update: chmod: %v\n", err)
- return
+ return fmt.Errorf("chmod: %v", err)
+ }
+ // Stage the verified build into the version store BEFORE swapping it live.
+ // gcVersions keeps every channel head, so this protects the good target from
+ // the pruner and makes it reachable to `lchat rollback`/re-deploy — the swap
+ // alone would leave the new build only as the loose /opt/lchat/lchat, a file
+ // the version store has never heard of.
+ if err := stageVersion(tmp, e.Version); err != nil {
+ return fmt.Errorf("stage %s: %v", e.Version, err)
}
// One .prev at a time: the previous previous build is already two behind
// and reachable from the version store anyway.
@@ -622,24 +681,202 @@ func controlSelfUpdate(m *releaseManifest, bake *firstSeenState, dryRun, ignoreB
os.Remove(p)
}
}
+ // .prev is written HERE, from the still-running (known-good) binary, and only
+ // ever just before a swap — so it always points at the last build that passed
+ // its health check. A rollback below restores it and returns immediately,
+ // never advancing .prev past an unproven build; the invariant is that the
+ // loose /opt/lchat/lchat is always either health-confirmed or being actively
+ // rolled back to one.
prev := self + ".prev-" + buildVersion
if err := copyFileAtomic(self, prev, 0o755); err != nil {
- fmt.Printf("update: control self-update: keep rollback copy: %v\n", err)
- return
+ return fmt.Errorf("keep rollback copy: %v", err)
}
// downloadReleaseBinary lands in incomingDir() — same filesystem as the
// control binary, so this rename atomically replaces the path.
if err := os.Rename(tmp, self); err != nil {
- fmt.Printf("update: control self-update: install: %v\n", err)
- return
+ return fmt.Errorf("install: %v", err)
+ }
+ // The binary is swapped; prove it comes up before walking away. Bounce the
+ // backend units onto it, then confirm the control plane reports the new build.
+ restartControlUnits()
+ if herr := waitControlHealthy(e.Version, deployHealthTimeout); herr != nil {
+ // The live probe failed — but it's only AUTHORITATIVE when lchat-control
+ // is actually running to answer it. try-restart is a no-op on a box where
+ // the unit is inactive (never enabled, or intentionally stopped), so a
+ // failed probe there means "nobody answered", not "the binary is bad".
+ if isActive(controlUnit) {
+ return rollbackControl(self, prev, e.Version, herr) // authoritative failure
+ }
+ // controld isn't up to probe. Fall back to the weaker, controld-independent
+ // gate: run `<self> version` and require it names the target build. That
+ // proves the new binary loads and self-reports the right build/arch (a
+ // corrupt or wrong-arch binary fails), NOT that the service boots — hence
+ // "weaker", used only because the strong gate is unavailable. controld
+ // starts fresh from this binary when next needed.
+ if verr := controlBinaryReports(self, e.Version); verr != nil {
+ return rollbackControl(self, prev, e.Version, fmt.Errorf("lchat-control inactive and the new binary failed its integrity check: %v", verr))
+ }
+ log.Printf("update: lchat-control inactive — committing control update to %s on the `version` integrity check (weaker gate; live probe unavailable)", e.Version)
+ // fall through to commit
+ }
+ // Committed. Any stale quarantine no longer applies.
+ clearFailedControlVersion()
+ // F5: cloud rode the same swap+restart but isn't the health gate (the router
+ // and chat are unaffected by it) — so don't roll back on it, just flag it.
+ if !isActive(cloudUnit) {
+ log.Printf("update: WARN lchat-cloud not active after control update to %s — check the provisioner", e.Version)
+ }
+ fmt.Printf("update: control binary %s -> %s (%s; healthy; rollback at %s)\n", buildVersion, e.Version, channel, prev)
+ fmt.Printf("update: lchat-router still runs %s — restart it when convenient to converge\n", buildVersion)
+ return nil
+}
+
+// rollbackControl restores the last-known-good control binary (.prev) after a
+// swapped-in build failed its health check, confirms the box came back, and
+// quarantines the bad build so the hourly pass doesn't reinstall + roll it back
+// forever. It returns a non-nil error describing the rollback — or a screaming,
+// non-looping CONTROL PLANE BRICKED error if recovery itself fails.
+func rollbackControl(self, prev, newVer string, healthErr error) error {
+ log.Printf("update: control binary %s -> %s is UNHEALTHY (%v) — rolling back to %s", buildVersion, newVer, healthErr, buildVersion)
+ if rberr := os.Rename(prev, self); rberr != nil {
+ return fmt.Errorf("CONTROL PLANE BRICKED: %s failed its health check AND restoring %s failed: %v (health error: %v)", newVer, prev, rberr, healthErr)
+ }
+ restartControlUnits()
+ // Confirm the restored binary with the same strong/weak gate: the live probe
+ // when controld runs, else the `version` integrity check.
+ var recerr error
+ if isActive(controlUnit) {
+ recerr = waitControlHealthy(buildVersion, deployHealthTimeout)
+ } else {
+ recerr = controlBinaryReports(self, buildVersion)
+ }
+ if recerr != nil {
+ // Genuine brick: neither the new build nor the restored old one is
+ // healthy. Scream, but do NOT loop — a human has to look now.
+ log.Printf("update: CONTROL PLANE BRICKED — rolled back to %s but IT is unhealthy too (%v); box needs a human NOW", buildVersion, recerr)
+ return fmt.Errorf("CONTROL PLANE BRICKED: %s unhealthy, rolled back to %s but it is unhealthy too (%v) — box needs a human NOW (original health error: %v)", newVer, buildVersion, recerr, healthErr)
+ }
+ // Persist the quarantine so the next hourly pass short-circuits this head
+ // instead of reinstalling + rolling it back again.
+ if err := writeFailedControlVersion(newVer); err != nil {
+ log.Printf("update: warning: couldn't persist control quarantine for %s: %v", newVer, err)
}
- fmt.Printf("update: control binary %s -> %s (%s; rollback at %s)\n", buildVersion, e.Version, channel, prev)
- for _, unit := range []string{"lchat-control.service", "lchat-cloud.service"} {
+ log.Printf("update: rolled back to %s and it is healthy again — %s is quarantined until a good build ships or you pin/patch the channel", buildVersion, newVer)
+ return fmt.Errorf("control binary %s failed its health check and was rolled back to %s: %v", newVer, buildVersion, healthErr)
+}
+
+// controlUnit / cloudUnit are the systemd units that exec the control binary.
+const (
+ controlUnit = "lchat-control.service"
+ cloudUnit = "lchat-cloud.service"
+)
+
+// restartControlUnits bounces the backend units that exec the control binary —
+// lchat-control and lchat-cloud — onto whatever /opt/lchat/lchat now is. The
+// ROUTER is deliberately excluded: it's the fleet's only ingress, has no
+// /healthz to probe, and must never blip on the updater's schedule (it keeps
+// its old inode until an operator restarts it). Restart failures only log — the
+// health probe that follows is the real gate on whether the new build is up.
+func restartControlUnits() {
+ for _, unit := range []string{controlUnit, cloudUnit} {
if err := systemctl("try-restart", unit); err != nil {
fmt.Printf("update: %s restart: %v (new build applies on its next start)\n", unit, err)
}
}
- fmt.Printf("update: lchat-router still runs %s — restart it when convenient to converge\n", buildVersion)
+}
+
+// isActive reports whether a systemd unit is currently active (`systemctl
+// is-active --quiet` exits 0 only when active). A var so tests can fake the
+// box's unit state — it decides whether the control /healthz probe is even
+// answerable, so a failed probe on a box that simply isn't running controld
+// doesn't roll back a perfectly good binary.
+var isActive = func(unit string) bool {
+ return systemctlQuery("is-active", "--quiet", unit) == nil
+}
+
+// controlBinaryReports is the controld-independent integrity gate: run
+// `<path> version` and require its output names wantBuild. It proves the binary
+// execs and self-reports the expected build/arch, which is weaker than a live
+// /healthz (it does NOT prove the service boots) but is all we have when
+// lchat-control isn't running to answer a probe. A var so tests can fake the
+// exec without a real binary on disk.
+var controlBinaryReports = func(path, wantBuild string) error {
+ got, err := readBinaryVersion(path)
+ if err != nil {
+ return fmt.Errorf("run %s version: %v", path, err)
+ }
+ if !strings.Contains(got, wantBuild) {
+ return fmt.Errorf("%s reported build %q, want %q", path, got, wantBuild)
+ }
+ return nil
+}
+
+// controlFailedVersionPath persists the last control head that failed its health
+// check and was rolled back (mirrors manifestStampPath). It lives under optDir
+// so it survives the updater process and is read on the next hourly pass.
+func controlFailedVersionPath() string { return filepath.Join(optDir, ".control-failed-version") }
+
+// readFailedControlVersion returns the quarantined control build, or "" if none.
+func readFailedControlVersion() string {
+ raw, err := os.ReadFile(controlFailedVersionPath())
+ if err != nil {
+ return ""
+ }
+ return strings.TrimSpace(string(raw))
+}
+
+// writeFailedControlVersion records ver as quarantined.
+func writeFailedControlVersion(ver string) error {
+ return os.WriteFile(controlFailedVersionPath(), []byte(ver+"\n"), 0o644)
+}
+
+// clearFailedControlVersion lifts the quarantine (a good build committed, or the
+// channel moved on). Absent file is fine — clearing "nothing" is a no-op.
+func clearFailedControlVersion() {
+ if err := os.Remove(controlFailedVersionPath()); err != nil && !os.IsNotExist(err) {
+ log.Printf("update: warning: couldn't clear control quarantine file: %v", err)
+ }
+}
+
+// waitControlHealthy polls the control daemon's /healthz until it reports
+// wantBuild, or timeout elapses — the gate on the box's self-update. After the
+// control binary is swapped and lchat-control restarted, this confirms the NEW
+// build came up (wantBuild = the new version); on the rollback path it confirms
+// the OLD build came back (wantBuild = buildVersion, this updater process's own
+// still-running id).
+//
+// lchat-control runs Restart=always (RestartSec=1), so a crash-looping build
+// flaps between activating and active: the socket comes and goes and dials fail
+// with "connection refused" in between. A transient refusal is therefore
+// neither success nor failure — keep retrying across it until either the build
+// matches (healthy, return early) or the whole timeout expires (unhealthy). It
+// mirrors waitHealthyVersion, but against the control socket and /healthz.
+func waitControlHealthy(wantBuild string, timeout time.Duration) error {
+ deadline := time.Now().Add(timeout)
+ client := unixHTTPClient(controlSocketPath)
+ var last error
+ for time.Now().Before(deadline) {
+ resp, err := client.Get("http://lchat/healthz")
+ if err == nil {
+ var body struct {
+ Build string `json:"build"`
+ }
+ derr := json.NewDecoder(io.LimitReader(resp.Body, 1<<16)).Decode(&body)
+ resp.Body.Close()
+ if derr == nil && body.Build == wantBuild {
+ return nil
+ }
+ if derr != nil {
+ last = fmt.Errorf("read /healthz: %v", derr)
+ } else {
+ last = fmt.Errorf("control build %q != expected %q", body.Build, wantBuild)
+ }
+ } else {
+ last = err // often "connection refused" mid-restart — retried, not fatal
+ }
+ time.Sleep(150 * time.Millisecond)
+ }
+ return fmt.Errorf("control daemon didn't report build %q within %s: %v", wantBuild, timeout, last)
}
// gcVersions prunes version-store dirs nothing references anymore: kept are