amadan
server: bot-sidecar co-spawn for hibernating instances (#550 Part 4 v2) (#670)
The activator now manages a hibernating instance's bot sidecar as a second child: spawned right after the instance turns healthy on wake, SIGTERMed first at sleep — the sidecar exists exactly while its instance is awake, so it can neither hold a sleeping host awake nor wake it back on reconnect. This is what gates hibernation for bot-bearing instances. - Detection is the filesystem convention the systemd units use: a host has a sidecar iff /data/<host>-bot.env exists (botEnvPath/botSessionDir, shared with add-bot). No registry column. - Co-spawn mirrors lchat-bot@.service: exec live/<host> `bot run` under the sidecar env file, LCHAT_*-clean like the instance spawn. Spawn failure is loud but never fails the wake (members > bots); the sweeper retries with doubling, capped backoff (Restart=always semantics, no supervisor), reset after a healthy run. - Co-stop: the sleep path stops the sidecar before the next-due probe and the instance stop, so it never outlives its instance — a dead child, a lost lease, a retire, and router shutdown all ride the same path. - The activity question: the sidecar dials its instance's public URL through the router (loopback /etc/hosts pin), so every request it makes — the WS dial and all REST — now carries X-Lchat-Sidecar (internal/bot/ sidecarmark.go). The activator serves marked requests while awake without stamping activity, refuses them (503, no wake) while asleep, and the proxy strips the header before the backend. Spoofing is harmless by construction: the marker only downgrades the sender's own requests — it can't suppress other clients' stamps or wake anything. - lchat hibernate --on/--off hands the systemd sidecar unit over and back (disable before the reload; re-enable after the instance is healthy), and add-bot on a hibernating host stages without starting the unit — the activator co-spawns on the next wake — so systemd and the activator never double-run one sidecar. Honest gap, documented: member-bot charters run in the sidecar, so a sleeping instance's charter schedules don't fire until something else wakes it — the due-time index covers server-side schedules only.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
12 files changed,
+917
−54
activator.go+290 −34activator_test.go+368 −2bot.go+24 −8bot_run_test.go+43 −0control.go+38 −5control_test.go+65 −2docs/bots.md+10 −0docs/instance-router.md+29 −0internal/bot/announcer.go+3 −1internal/bot/runtime.go+9 −2internal/bot/sidecarmark.go+32 −0router.go+6 −0
diff --git a/activator.go b/activator.go| index 79e23df..2072cdf 100644 |
| --- a/activator.go |
| +++ b/activator.go |
| @@ -14,9 +14,21 @@ package main |
| // |
| // Scope (this phase): ONE box. No gossip/HRW placement, no peer proxying — |
| // membership of one, hash picks self, leases uncontested except against a |
| -// previous incarnation of ourselves. Explicit non-goals held for follow-ups: |
| -// - No bot sidecar co-spawn: a hibernating instance's sidecar isn't managed |
| -// (a running sidecar's WebSocket also holds the instance awake forever). |
| +// previous incarnation of ourselves. |
| +// |
| +// Bot sidecar co-spawn (a former non-goal, now in): a host with a sidecar |
| +// staged (/data/<host>-bot.env — the same file lchat-bot@.service execs |
| +// against) gets its `bot run` child spawned right after the instance turns |
| +// healthy, and SIGTERMed first at sleep — the sidecar exists exactly while its |
| +// instance is awake, so its lifecycle solves itself. Its own connections flow |
| +// back through this router (the box pins its hosted names to loopback), so the |
| +// sidecar marks them (bot.SidecarHeader) and the activator excludes them from |
| +// the idle clock — otherwise the pair could never sleep, and an orphaned |
| +// sidecar's reconnect loop would wake the instance right back (see serveHTTP). |
| +// One honest gap: member-bot charters run IN the sidecar (docs/member-bots.md), |
| +// so a sleeping instance's charter schedules don't fire until something else |
| +// wakes it — the due-time index below covers server-side schedules only. The |
| +// hardened dedicated-sidecar tier (bot keys off-box) stays a non-goal here. |
| // |
| // Opt-in is PER HOST (routes.hibernate, registry.go): with the bucket env set, |
| // only flagged instances are activator-managed; the rest stay on their systemd |
| @@ -57,6 +69,7 @@ import ( |
| "github.com/benbjohnson/litestream" |
| "lchat/internal/activator" |
| + "lchat/internal/bot" |
| ) |
| // hibernateConfig is the router-process opt-in, read from env. Nil (bucket |
| @@ -115,6 +128,12 @@ type actBackend interface { |
| // delivery (unix seconds; 0 = nothing scheduled). Queried at sleep time, |
| // while the socket still answers, to feed the due-time index. |
| nextDue(ctx context.Context, rt Route) (int64, error) |
| + // hasSidecar reports whether host has a bot sidecar staged on this box. |
| + hasSidecar(host string) bool |
| + // startSidecar spawns the host's bot sidecar. Fire-and-forget: a sidecar |
| + // is a pure outbound client with no socket, so there's nothing to |
| + // health-check — its own connect/backoff loop recovers from a slow start. |
| + startSidecar(ctx context.Context, rt Route) (actProc, error) |
| } |
| // actDueStore persists the due-time index — the registry in production |
| @@ -184,13 +203,39 @@ type actEntry struct { |
| // AND the idle window has passed since the last request finished. In-flight |
| // counting (not just timestamps) is what keeps an instance with one open |
| // WebSocket awake — the proxy's ServeHTTP blocks for the tunnel's life. |
| + // (Sidecar-marked requests register in neither — see beginSidecarRequest.) |
| inflight int |
| lastActive time.Time |
| rep actRep |
| proc actProc |
| + |
| + // Bot sidecar co-spawn: the sidecar exists exactly while its instance is |
| + // awake. All sidecar fields are written under a.mu in the awake state only |
| + // (wake's attach, the sweeper's restart); a stop flips the state under the |
| + // same lock first, so stop() owns them uncontended from then on. |
| + sidecar actProc // the co-spawned `bot run` child; nil = none running |
| + sidecarWanted bool // the host had a sidecar staged at wake time |
| + sidecarStarted time.Time // when the current sidecar spawned (backoff reset) |
| + sidecarBackoff time.Duration // last restart delay (doubles, capped) |
| + sidecarRetryAt time.Time // earliest next restart attempt |
| } |
| +// Sidecar restart pacing — Restart=always semantics without a supervisor: the |
| +// sweeper redoes a dead sidecar with exponential backoff from the systemd |
| +// unit's RestartSec, capped, and a run that survived sidecarHealthyRun earns a |
| +// fresh backoff (so one crash a day never escalates to the cap). |
| +const ( |
| + sidecarRestartBase = 2 * time.Second |
| + sidecarRestartMax = time.Minute |
| + sidecarHealthyRun = time.Minute |
| +) |
| + |
| +// sidecarStopTimeout bounds the sidecar's stop at sleep time. `bot run` has no |
| +// drain handler, so the SIGTERM is effectively immediate; the SIGKILL fallback |
| +// behind this timeout is insurance. |
| +const sidecarStopTimeout = 5 * time.Second |
| + |
| // wakeWaitBound caps how long a request queues on someone else's wake or stop |
| // before giving up with 503 — a wedged wake must not pile up connections |
| // forever. A var so tests can shrink it. |
| @@ -307,6 +352,75 @@ func (a *activatorSvc) beginRequest(ctx context.Context, rt Route) (release func |
| } |
| } |
| +// errSidecarAsleep: a sidecar-marked request arrived for a host with no awake |
| +// instance. Sidecar traffic never wakes anything — the activator itself spawns |
| +// the sidecar as part of a wake, so a marked request for a sleeping host is |
| +// either a stray reconnect racing a stop or an outsider spoofing the marker; |
| +// both get a 503 and change nothing. |
| +var errSidecarAsleep = errors.New("instance asleep; sidecar requests do not wake it") |
| + |
| +// beginSidecarRequest is beginRequest for sidecar-marked traffic: pass it |
| +// through to an awake instance WITHOUT registering activity — no in-flight |
| +// count, no lastActive stamp (the sidecar's own long-lived WebSocket must not |
| +// hold its instance awake; that non-count is the whole reason the pair can |
| +// sleep) — and refuse, rather than wake, when the instance is asleep or |
| +// stopping. |
| +func (a *activatorSvc) beginSidecarRequest(ctx context.Context, rt Route) error { |
| + deadline := time.NewTimer(wakeWaitBound) |
| + defer deadline.Stop() |
| + for { |
| + a.mu.Lock() |
| + e := a.entries[rt.Host] |
| + if e == nil { |
| + a.mu.Unlock() |
| + return errSidecarAsleep |
| + } |
| + switch e.state { |
| + case actAwake: |
| + // Same dead-child / lost-lease detection as a member request, but |
| + // the recovery stops at the stop: the loop then finds a stopping |
| + // (or absent) entry and turns this request away instead of |
| + // chaining into a re-wake. |
| + if repErr := e.rep.Err(); repErr != nil || e.proc.exited() { |
| + if repErr != nil { |
| + log.Printf("activator: LEASE LOST for %s (%v) — standing down", e.host, repErr) |
| + } else { |
| + log.Printf("activator: instance %s exited unexpectedly — recycling", e.host) |
| + } |
| + a.beginStopLocked(e) |
| + a.mu.Unlock() |
| + continue |
| + } |
| + a.mu.Unlock() |
| + return nil |
| + |
| + case actWaking: |
| + // Joining a wake someone else started is fine — the co-spawned |
| + // sidecar boots moments before its entry flips awake, so its first |
| + // dial can land in that sliver; joining can't initiate anything. |
| + // A failed wake just reads as asleep (the member path already |
| + // logged it). |
| + ready := e.ready |
| + a.mu.Unlock() |
| + select { |
| + case <-ready: |
| + if e.err != nil { |
| + return errSidecarAsleep |
| + } |
| + continue |
| + case <-ctx.Done(): |
| + return ctx.Err() |
| + case <-deadline.C: |
| + return errWakeTimeout |
| + } |
| + |
| + default: // actStopping — the stop exists to shed exactly this connection |
| + a.mu.Unlock() |
| + return errSidecarAsleep |
| + } |
| + } |
| +} |
| + |
| func (a *activatorSvc) releaseFunc(e *actEntry) func() { |
| var once sync.Once |
| return func() { |
| @@ -319,14 +433,15 @@ func (a *activatorSvc) releaseFunc(e *actEntry) func() { |
| } |
| } |
| -// wake drives asleep→awake: lease, restore-if-missing, spawn, replicate. It |
| -// runs on the first requester's goroutine but under its own generous timeout, |
| -// detached from that request's context — queued requests still need the wake |
| -// to finish even if the requester who started it hangs up. |
| +// wake drives asleep→awake: lease, restore-if-missing, spawn, replicate, |
| +// co-spawn the sidecar. It runs on the first requester's goroutine but under |
| +// its own generous timeout, detached from that request's context — queued |
| +// requests still need the wake to finish even if the requester who started it |
| +// hangs up. |
| func (a *activatorSvc) wake(e *actEntry) { |
| ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute) |
| defer cancel() |
| - rep, proc, err := a.doWake(ctx, e.route) |
| + w, err := a.doWake(ctx, e.route) |
| a.mu.Lock() |
| if err != nil { |
| @@ -334,14 +449,29 @@ func (a *activatorSvc) wake(e *actEntry) { |
| delete(a.entries, e.host) // next request retries from scratch |
| } else { |
| e.state = actAwake |
| - e.rep, e.proc = rep, proc |
| + e.rep, e.proc = w.rep, w.proc |
| + e.sidecar, e.sidecarWanted = w.sidecar, w.sidecarWanted |
| + e.sidecarStarted = a.now() |
| + e.sidecarBackoff = 0 |
| + e.sidecarRetryAt = time.Time{} // a wake-time spawn failure retries on the next sweep |
| e.lastActive = a.now() |
| } |
| a.mu.Unlock() |
| close(e.ready) |
| } |
| -func (a *activatorSvc) doWake(ctx context.Context, rt Route) (actRep, actProc, error) { |
| +// wokenInstance is a successful wake's yield. sidecar is nil when the host has |
| +// none staged — or when the spawn failed, which is deliberately not an error: |
| +// members > bots, so a space whose bots won't start still serves (the sweeper |
| +// retries via sidecarWanted). |
| +type wokenInstance struct { |
| + rep actRep |
| + proc actProc |
| + sidecar actProc |
| + sidecarWanted bool |
| +} |
| + |
| +func (a *activatorSvc) doWake(ctx context.Context, rt Route) (*wokenInstance, error) { |
| host := rt.Host |
| dbPath := rt.DBPath |
| if dbPath == "" { |
| @@ -359,7 +489,7 @@ func (a *activatorSvc) doWake(ctx context.Context, rt Route) (actRep, actProc, e |
| log.Printf("activator: LEASE HELD for %s by %q — a previous incarnation hasn't released; refusing to wake (retry after %s)", |
| host, held.holder, held.retryAfter) |
| } |
| - return nil, nil, err |
| + return nil, err |
| } |
| // 2. Restore the DB from S3 when the local copy is gone. When it's still |
| @@ -367,7 +497,7 @@ func (a *activatorSvc) doWake(ctx context.Context, rt Route) (actRep, actProc, e |
| // copy can only be what we ourselves checkpointed at the last sleep. |
| if err := a.backend.restore(ctx, host, dbPath); err != nil { |
| _ = lease.release(ctx) |
| - return nil, nil, fmt.Errorf("restore %s: %w", host, err) |
| + return nil, fmt.Errorf("restore %s: %w", host, err) |
| } |
| // 3. Spawn before replication starts: a fresh instance creates its own DB |
| @@ -377,7 +507,7 @@ func (a *activatorSvc) doWake(ctx context.Context, rt Route) (actRep, actProc, e |
| proc, err := a.backend.start(ctx, rt) |
| if err != nil { |
| _ = lease.release(ctx) |
| - return nil, nil, fmt.Errorf("start %s: %w", host, err) |
| + return nil, fmt.Errorf("start %s: %w", host, err) |
| } |
| // 4. Trade the held lease for running replication. |
| @@ -387,9 +517,24 @@ func (a *activatorSvc) doWake(ctx context.Context, rt Route) (actRep, actProc, e |
| defer cancel() |
| _ = proc.stop(stopCtx) |
| _ = lease.release(ctx) |
| - return nil, nil, fmt.Errorf("replicate %s: %w", host, err) |
| + return nil, fmt.Errorf("replicate %s: %w", host, err) |
| + } |
| + |
| + // 5. Co-spawn the bot sidecar, last — only once the instance is healthy |
| + // and replicating, so the sidecar always boots against a live instance |
| + // (its marked requests pass through the router while we're awake). A |
| + // spawn failure is loud but never fails the wake: members come before |
| + // bots, and the sweeper retries with backoff. |
| + w := &wokenInstance{rep: rep, proc: proc} |
| + if a.backend.hasSidecar(host) { |
| + w.sidecarWanted = true |
| + if sc, err := a.backend.startSidecar(ctx, rt); err != nil { |
| + log.Printf("activator: SIDECAR for %s failed to start: %v — serving without bots (the sweep retries)", host, err) |
| + } else { |
| + w.sidecar = sc |
| + } |
| } |
| - return rep, proc, nil |
| + return w, nil |
| } |
| // actStopTimeout bounds a graceful instance stop (the instance's own SIGTERM |
| @@ -411,6 +556,20 @@ func (a *activatorSvc) beginStopLocked(e *actEntry) { |
| // the entry. The local DB stays in place — it's the warm cache for the next |
| // wake; S3 is the durable truth under the lease. |
| func (a *activatorSvc) stop(e *actEntry) { |
| + // Co-stop, sidecar FIRST — before the next-due probe and the instance's |
| + // own SIGTERM. Two reasons: the sidecar must never outlive its instance |
| + // (an orphan's reconnect loop would be a standing wake source — the |
| + // ping-pong this feature exists to prevent — and its member-bot charters |
| + // would keep firing into a host we're about to stop), and with the bots |
| + // gone first, the schedule the next-due probe reads can't shift under it. |
| + if e.sidecar != nil { |
| + scCtx, scCancel := context.WithTimeout(context.Background(), sidecarStopTimeout) |
| + if err := e.sidecar.stop(scCtx); err != nil { |
| + log.Printf("activator: stop sidecar %s: %v", e.host, err) |
| + } |
| + scCancel() |
| + } |
| + |
| // The due-time index: ask the instance for its earliest pending scheduled |
| // delivery BEFORE stopping it — the socket dies with the process. (When the |
| // stop was triggered by a dead child, the query just fails fast.) Any |
| @@ -516,6 +675,37 @@ func (a *activatorSvc) sweepOnce() { |
| if e.inflight == 0 && now.Sub(e.lastActive) >= a.cfg.Idle { |
| log.Printf("activator: %s idle %s — hibernating", e.host, now.Sub(e.lastActive).Truncate(time.Second)) |
| a.beginStopLocked(e) |
| + continue |
| + } |
| + // Sidecar supervision — Restart=always, sweep-driven, no supervisor |
| + // goroutine: a sidecar that died (or failed to spawn at wake) while |
| + // its instance stays awake is respawned with doubling, capped backoff; |
| + // a run past sidecarHealthyRun resets it. Spawned under a.mu |
| + // deliberately: an exec is quick, and the lock is what makes a restart |
| + // impossible to race with a stop (sidecar fields only change in the |
| + // awake state, and every stop flips the state under this same lock |
| + // first). Only what the wake meant to spawn is managed — a bot env |
| + // file appearing mid-flight (add-bot on an awake host) stays systemd's |
| + // until the next wake, so the two never double-run one sidecar. |
| + if e.sidecarWanted && (e.sidecar == nil || e.sidecar.exited()) && !now.Before(e.sidecarRetryAt) { |
| + if e.sidecar != nil && now.Sub(e.sidecarStarted) >= sidecarHealthyRun { |
| + e.sidecarBackoff = 0 |
| + } |
| + if e.sidecarBackoff == 0 { |
| + e.sidecarBackoff = sidecarRestartBase |
| + } else { |
| + e.sidecarBackoff = min(e.sidecarBackoff*2, sidecarRestartMax) |
| + } |
| + e.sidecarRetryAt = now.Add(e.sidecarBackoff) |
| + sc, err := a.backend.startSidecar(context.Background(), e.route) |
| + if err != nil { |
| + log.Printf("activator: SIDECAR restart for %s failed: %v (next attempt in %s)", e.host, err, e.sidecarBackoff) |
| + e.sidecar = nil |
| + continue |
| + } |
| + log.Printf("activator: sidecar for %s was down — restarted (backoff now %s)", e.host, e.sidecarBackoff) |
| + e.sidecar = sc |
| + e.sidecarStarted = now |
| } |
| } |
| // Due-time wakes: a request-less wake through the same singleflight state |
| @@ -625,6 +815,30 @@ func (a *activatorSvc) shutdown(ctx context.Context) { |
| // join a wake), proxy, stamp activity. Lease contention and wake timeouts are |
| // 503s — "sleeping spaces are slow to wake", never a 5xx cascade of retries. |
| func (a *activatorSvc) serveHTTP(w http.ResponseWriter, r *http.Request, rt Route, proxy http.Handler) { |
| + // Sidecar-marked traffic (bot.SidecarHeader — the Go sidecar stamps every |
| + // REST call and its WebSocket dial) is activity-invisible: proxied while |
| + // the instance is awake, refused while it isn't, never a wake. The |
| + // co-spawned sidecar dials its instance's public URL through this very |
| + // router (the box pins its hosted names to loopback), so without this its |
| + // keepalives would read as member activity — the instance could never |
| + // idle — and after a sleep its reconnect loop would be the wake source: |
| + // ping-pong forever. Spoofing the marker is harmless by construction: it |
| + // only DOWNGRADES the sender's own requests (no activity credit; 503 while |
| + // asleep). It cannot suppress other clients' activity stamps and cannot |
| + // wake a sleeping host, so an outsider sending it gains nothing. |
| + if r.Header.Get(bot.SidecarHeader) != "" { |
| + if err := a.beginSidecarRequest(r.Context(), rt); err != nil { |
| + if errors.Is(err, context.Canceled) { |
| + return // the sidecar hung up while queued on a wake |
| + } |
| + w.Header().Set("Retry-After", "5") |
| + http.Error(w, "instance is asleep; sidecar requests do not wake it", http.StatusServiceUnavailable) |
| + return |
| + } |
| + proxy.ServeHTTP(w, r) |
| + return |
| + } |
| + |
| release, err := a.beginRequest(r.Context(), rt) |
| if err != nil { |
| var held *errLeaseHeld |
| @@ -772,29 +986,14 @@ func (b *s3Backend) start(ctx context.Context, rt Route) (actProc, error) { |
| } |
| } |
| - // Environment: the unit gives an instance a clean env plus its |
| - // EnvironmentFile. Approximate that from inside the router: inherit the |
| - // process env but strip every LCHAT_* var first, so the router's own |
| - // config (hibernation, registry, wildcard tokens) can never leak into a |
| - // child; then layer the instance's env file (admin token, per-instance |
| - // config) on top. The env file is optional, like the unit's "-" prefix. |
| - env := make([]string, 0, 32) |
| - for _, kv := range os.Environ() { |
| - if strings.HasPrefix(kv, "LCHAT_") { |
| - continue |
| - } |
| - env = append(env, kv) |
| - } |
| - if fileEnv, err := readEnvFile(filepath.Join(instanceDataDir, host+".env")); err == nil { |
| - for k, v := range fileEnv { |
| - env = append(env, k+"="+v) |
| - } |
| - } else if !os.IsNotExist(err) { |
| + // The env file is optional, like the unit's "-" prefix. |
| + fileEnv, err := readEnvFile(filepath.Join(instanceDataDir, host+".env")) |
| + if err != nil && !os.IsNotExist(err) { |
| return nil, fmt.Errorf("read env file: %w", err) |
| } |
| cmd := exec.Command(bin, "--socket", rt.Socket, "--db", dbPath) |
| - cmd.Env = env |
| + cmd.Env = childEnv(fileEnv) |
| // The child logs through the router's own stderr/stdout (journald under |
| // systemd); there's no per-instance unit to own a log stream anymore. |
| cmd.Stdout = os.Stdout |
| @@ -818,6 +1017,63 @@ func (b *s3Backend) start(ctx context.Context, rt Route) (actProc, error) { |
| return p, nil |
| } |
| +// childEnv approximates a systemd unit's environment from inside the router: |
| +// inherit the process env but strip every LCHAT_* var first, so the router's |
| +// own config (hibernation, registry, wildcard tokens) can never leak into a |
| +// child; then layer the child's EnvironmentFile on top. A nil fileEnv (no env |
| +// file, like the unit's "-" prefix) yields just the cleaned inherit. |
| +func childEnv(fileEnv map[string]string) []string { |
| + env := make([]string, 0, 32) |
| + for _, kv := range os.Environ() { |
| + if strings.HasPrefix(kv, "LCHAT_") { |
| + continue |
| + } |
| + env = append(env, kv) |
| + } |
| + for k, v := range fileEnv { |
| + env = append(env, k+"="+v) |
| + } |
| + return env |
| +} |
| + |
| +// hasSidecar: a host has a bot sidecar iff its sidecar EnvironmentFile exists |
| +// — the same /data/<host>-bot.env convention add-bot writes and |
| +// lchat-bot@.service reads. Derived from the filesystem on purpose, the way |
| +// the units do it: no registry column to drift from what's actually staged. |
| +func (b *s3Backend) hasSidecar(host string) bool { |
| + _, err := os.Stat(botEnvPath(host)) |
| + return err == nil |
| +} |
| + |
| +// startSidecar mirrors lchat-bot@.service: exec the host's live symlink with |
| +// `bot run` under the sidecar's EnvironmentFile (LCHAT_BASE, session paths, |
| +// invites, optional ANTHROPIC_API_KEY), LCHAT_*-clean like the instance |
| +// spawn. No health wait — the sidecar is a pure outbound client with no |
| +// socket to probe; its own connect/backoff loop absorbs a slow instance. The |
| +// live symlink exists by now: the instance spawn staged it. The ctx is unused |
| +// (an exec doesn't block); it's here for the backend seam's symmetry. |
| +func (b *s3Backend) startSidecar(_ context.Context, rt Route) (actProc, error) { |
| + fileEnv, err := readEnvFile(botEnvPath(rt.Host)) |
| + if err != nil { |
| + return nil, fmt.Errorf("read bot env: %w", err) |
| + } |
| + cmd := exec.Command(liveLink(rt.Host), "bot", "run") |
| + cmd.Env = childEnv(fileEnv) |
| + // Like the instance child: logs through the router's own streams |
| + // (journald under systemd) — no per-host unit owns a log stream anymore. |
| + cmd.Stdout = os.Stdout |
| + cmd.Stderr = os.Stderr |
| + if err := cmd.Start(); err != nil { |
| + return nil, err |
| + } |
| + p := &childProc{cmd: cmd, done: make(chan struct{})} |
| + go func() { |
| + p.err = cmd.Wait() |
| + close(p.done) |
| + }() |
| + return p, nil |
| +} |
| + |
| // waitHealthyCtx is waitHealthy bounded by ctx as well as its own timeout, so |
| // an aborted wake doesn't keep probing a socket nobody wants anymore. |
| func waitHealthyCtx(ctx context.Context, socket string, timeout time.Duration) error { |
diff --git a/activator_test.go b/activator_test.go| index 8735d89..a24796b 100644 |
| --- a/activator_test.go |
| +++ b/activator_test.go |
| @@ -18,6 +18,8 @@ import ( |
| "sync/atomic" |
| "testing" |
| "time" |
| + |
| + "lchat/internal/bot" |
| ) |
| // --- fakes ------------------------------------------------------------------- |
| @@ -57,6 +59,10 @@ type fakeProc struct { |
| // stopGate, when non-nil, blocks stop() until closed — holds an entry in |
| // the stopping state so a test can poke at it. |
| stopGate chan struct{} |
| + // label + rec feed the backend's order log ("instance" / "sidecar"), so |
| + // the co-spawn/co-stop tests can assert interleavings. |
| + label string |
| + rec *fakeBackend |
| } |
| func (p *fakeProc) stop(context.Context) error { |
| @@ -67,8 +73,11 @@ func (p *fakeProc) stop(context.Context) error { |
| <-gate |
| } |
| p.mu.Lock() |
| - defer p.mu.Unlock() |
| p.stopped = true |
| + p.mu.Unlock() |
| + if p.rec != nil { |
| + p.rec.record("stop:" + p.label) |
| + } |
| return nil |
| } |
| func (p *fakeProc) exited() bool { |
| @@ -98,6 +107,30 @@ type fakeBackend struct { |
| procs []*fakeProc |
| dueVal int64 // what the instance reports at the sleep-time next-due probe |
| dueErr error // when non-nil, the probe fails (dead socket, missing token, …) |
| + |
| + // Bot sidecar co-spawn. sidecarStaged is hasSidecar's answer (false by |
| + // default, so every pre-co-spawn test doubles as the "host without a |
| + // sidecar spawns none" case); sidecarErr injects spawn failures. |
| + sidecarStaged bool |
| + sidecarErr error |
| + sidecarStarts int |
| + sidecars []*fakeProc |
| + // order interleaves backend calls and proc stops (start:instance, |
| + // start:sidecar, nextDue, stop:sidecar, stop:instance) for the ordering |
| + // assertions. |
| + order []string |
| +} |
| + |
| +func (b *fakeBackend) record(ev string) { |
| + b.mu.Lock() |
| + b.order = append(b.order, ev) |
| + b.mu.Unlock() |
| +} |
| + |
| +func (b *fakeBackend) events() []string { |
| + b.mu.Lock() |
| + defer b.mu.Unlock() |
| + return append([]string(nil), b.order...) |
| } |
| func (b *fakeBackend) acquire(context.Context, string) (actLease, error) { |
| @@ -123,8 +156,9 @@ func (b *fakeBackend) restore(context.Context, string, string) error { |
| func (b *fakeBackend) start(context.Context, Route) (actProc, error) { |
| b.mu.Lock() |
| b.starts++ |
| + b.order = append(b.order, "start:instance") |
| gate := b.startGate |
| - proc := &fakeProc{} |
| + proc := &fakeProc{label: "instance", rec: b} |
| b.procs = append(b.procs, proc) |
| b.mu.Unlock() |
| if gate != nil { |
| @@ -136,12 +170,50 @@ func (b *fakeBackend) start(context.Context, Route) (actProc, error) { |
| func (b *fakeBackend) nextDue(context.Context, Route) (int64, error) { |
| b.mu.Lock() |
| defer b.mu.Unlock() |
| + b.order = append(b.order, "nextDue") |
| if b.dueErr != nil { |
| return 0, b.dueErr |
| } |
| return b.dueVal, nil |
| } |
| +func (b *fakeBackend) hasSidecar(string) bool { |
| + b.mu.Lock() |
| + defer b.mu.Unlock() |
| + return b.sidecarStaged |
| +} |
| + |
| +func (b *fakeBackend) startSidecar(context.Context, Route) (actProc, error) { |
| + b.mu.Lock() |
| + defer b.mu.Unlock() |
| + b.sidecarStarts++ |
| + b.order = append(b.order, "start:sidecar") |
| + if b.sidecarErr != nil { |
| + return nil, b.sidecarErr |
| + } |
| + p := &fakeProc{label: "sidecar", rec: b} |
| + b.sidecars = append(b.sidecars, p) |
| + return p, nil |
| +} |
| + |
| +func (b *fakeBackend) sidecarStartCount() int { |
| + b.mu.Lock() |
| + defer b.mu.Unlock() |
| + return b.sidecarStarts |
| +} |
| + |
| +func (b *fakeBackend) lastSidecar() *fakeProc { |
| + b.mu.Lock() |
| + defer b.mu.Unlock() |
| + return b.sidecars[len(b.sidecars)-1] |
| +} |
| + |
| +func (b *fakeBackend) setSidecarErr(err error) { |
| + b.mu.Lock() |
| + defer b.mu.Unlock() |
| + b.sidecarErr = err |
| +} |
| + |
| func (b *fakeBackend) setDue(v int64) { |
| b.mu.Lock() |
| defer b.mu.Unlock() |
| @@ -851,6 +923,300 @@ func TestActivatorDueSurvivesRestart(t *testing.T) { |
| } |
| } |
| +// --- bot sidecar co-spawn ---------------------------------------------------- |
| + |
| +// indexOf: position of ev in the backend's order log (-1 = absent). |
| +func indexOf(events []string, ev string) int { |
| + for i, e := range events { |
| + if e == ev { |
| + return i |
| + } |
| + } |
| + return -1 |
| +} |
| + |
| +// A host with a sidecar staged gets it co-spawned on wake, strictly AFTER the |
| +// instance process is up — the sidecar dials the instance, so spawning it |
| +// first would just burn its reconnect backoff. (Every other test in this file |
| +// runs with sidecarStaged=false and asserts nothing sidecar-ish happens, which |
| +// is the "host without a sidecar behaves exactly as today" half.) |
| +func TestActivatorCoSpawnsSidecarAfterInstance(t *testing.T) { |
| + a, b, _, _ := testActivator(t, time.Minute) |
| + b.sidecarStaged = true |
| + |
| + release, err := a.beginRequest(context.Background(), testRoute("bots.example.com")) |
| + if err != nil { |
| + t.Fatalf("beginRequest: %v", err) |
| + } |
| + release() |
| + |
| + if got := b.sidecarStartCount(); got != 1 { |
| + t.Fatalf("sidecar starts = %d, want 1", got) |
| + } |
| + ev := b.events() |
| + if i, s := indexOf(ev, "start:instance"), indexOf(ev, "start:sidecar"); i == -1 || s == -1 || s < i { |
| + t.Fatalf("sidecar did not start strictly after the instance: %v", ev) |
| + } |
| +} |
| + |
| +// A sidecar that fails to spawn must not fail the wake — members > bots: the |
| +// space serves without its bots, and the sweeper keeps retrying with backoff |
| +// until a spawn sticks. |
| +func TestActivatorSidecarFailureDoesNotFailWake(t *testing.T) { |
| + a, b, clk, _ := testActivator(t, time.Hour) |
| + b.sidecarStaged = true |
| + b.setSidecarErr(errors.New("exec: live symlink gone")) |
| + route := testRoute("botless.example.com") |
| + |
| + release, err := a.beginRequest(context.Background(), route) |
| + if err != nil { |
| + t.Fatalf("beginRequest with a failing sidecar: %v", err) |
| + } |
| + release() |
| + if !a.entryAwake(route.Host) { |
| + t.Fatal("instance not awake after a sidecar-only failure") |
| + } |
| + if got := b.sidecarStartCount(); got != 1 { |
| + t.Fatalf("sidecar attempts = %d, want 1 (the wake's)", got) |
| + } |
| + |
| + // The sweeper retries: attempt 2 immediately (first retry), then gated by |
| + // the doubling backoff; once the fault clears, a retry lands the sidecar. |
| + a.sweepOnce() |
| + if got := b.sidecarStartCount(); got != 2 { |
| + t.Fatalf("sidecar attempts after first sweep = %d, want 2", got) |
| + } |
| + a.sweepOnce() // still inside the 2s gate — no attempt |
| + if got := b.sidecarStartCount(); got != 2 { |
| + t.Fatalf("backoff gate ignored: attempts = %d, want 2", got) |
| + } |
| + b.setSidecarErr(nil) |
| + clk.advance(3 * time.Second) |
| + a.sweepOnce() |
| + if got := b.sidecarStartCount(); got != 3 { |
| + t.Fatalf("sidecar attempts after fault cleared = %d, want 3", got) |
| + } |
| + if b.lastSidecar().isStopped() || b.lastSidecar().exited() { |
| + t.Fatal("recovered sidecar isn't running") |
| + } |
| +} |
| + |
| +// Co-stop ordering at sleep: the sidecar is stopped FIRST — before the |
| +// next-due probe (so bots can't shift the schedule being recorded) and before |
| +// the instance's own stop (so the sidecar never outlives its instance). |
| +func TestActivatorCoStopOrdering(t *testing.T) { |
| + a, b, clk, _ := testActivator(t, 10*time.Second) |
| + b.sidecarStaged = true |
| + |
| + release, err := a.beginRequest(context.Background(), testRoute("bots.example.com")) |
| + if err != nil { |
| + t.Fatalf("beginRequest: %v", err) |
| + } |
| + release() |
| + |
| + clk.advance(11 * time.Second) |
| + a.sweepOnce() |
| + waitFor(t, "hibernation", func() bool { return a.entryCount() == 0 }) |
| + |
| + if !b.lastSidecar().isStopped() { |
| + t.Fatal("sidecar not stopped at sleep") |
| + } |
| + ev := b.events() |
| + sc, due, inst := indexOf(ev, "stop:sidecar"), indexOf(ev, "nextDue"), indexOf(ev, "stop:instance") |
| + if sc == -1 || due == -1 || inst == -1 || !(sc < due && due < inst) { |
| + t.Fatalf("co-stop order wrong: %v (want stop:sidecar < nextDue < stop:instance)", ev) |
| + } |
| +} |
| + |
| +// Orphan prevention: whatever takes the instance down — here a crashed child, |
| +// but a lost lease and a retire ride the identical stop path — takes the |
| +// sidecar with it. An orphaned sidecar would reconnect-loop against the router |
| +// forever (503s at best; a wake source if the marker rule ever broke). |
| +func TestActivatorInstanceDeathStopsSidecar(t *testing.T) { |
| + a, b, _, _ := testActivator(t, time.Hour) |
| + b.sidecarStaged = true |
| + |
| + release, err := a.beginRequest(context.Background(), testRoute("bots.example.com")) |
| + if err != nil { |
| + t.Fatalf("beginRequest: %v", err) |
| + } |
| + release() |
| + |
| + p := b.lastProc() |
| + p.mu.Lock() |
| + p.dead = true |
| + p.mu.Unlock() |
| + a.sweepOnce() |
| + waitFor(t, "stand-down", func() bool { return a.entryCount() == 0 }) |
| + if !b.lastSidecar().isStopped() { |
| + t.Fatal("instance died but its sidecar was left running (orphan)") |
| + } |
| +} |
| + |
| +// Sidecar death while the instance stays awake: restarted with doubling, |
| +// capped backoff — Restart=always semantics, sweep-driven — and a run that |
| +// held for sidecarHealthyRun earns a fresh backoff. |
| +func TestActivatorSidecarRestartBackoff(t *testing.T) { |
| + a, b, clk, _ := testActivator(t, time.Hour) |
| + b.sidecarStaged = true |
| + |
| + release, err := a.beginRequest(context.Background(), testRoute("bots.example.com")) |
| + if err != nil { |
| + t.Fatalf("beginRequest: %v", err) |
| + } |
| + release() |
| + kill := func() { |
| + sc := b.lastSidecar() |
| + sc.mu.Lock() |
| + sc.dead = true |
| + sc.mu.Unlock() |
| + } |
| + |
| + // T0: first crash — restarted on the next sweep (attempt 2), gate 2s. |
| + kill() |
| + a.sweepOnce() |
| + if got := b.sidecarStartCount(); got != 2 { |
| + t.Fatalf("first restart: attempts = %d, want 2", got) |
| + } |
| + // Crash-loop: still inside the 2s gate — no restart. |
| + kill() |
| + a.sweepOnce() |
| + if got := b.sidecarStartCount(); got != 2 { |
| + t.Fatalf("gate ignored: attempts = %d, want 2", got) |
| + } |
| + // T3: past the 2s gate — attempt 3, gate doubles to 4s. |
| + clk.advance(3 * time.Second) |
| + a.sweepOnce() |
| + if got := b.sidecarStartCount(); got != 3 { |
| + t.Fatalf("second restart: attempts = %d, want 3", got) |
| + } |
| + // T6: 3s later is inside the 4s gate — the doubling is real. |
| + kill() |
| + clk.advance(3 * time.Second) |
| + a.sweepOnce() |
| + if got := b.sidecarStartCount(); got != 3 { |
| + t.Fatalf("doubled gate ignored: attempts = %d, want 3", got) |
| + } |
| + // T8: past it — attempt 4 (gate now 8s). |
| + clk.advance(2 * time.Second) |
| + a.sweepOnce() |
| + if got := b.sidecarStartCount(); got != 4 { |
| + t.Fatalf("third restart: attempts = %d, want 4", got) |
| + } |
| + // A healthy run (≥ sidecarHealthyRun) resets the ladder: the next crash |
| + // restarts immediately and its gate is back to the 2s base… |
| + clk.advance(2 * time.Minute) |
| + kill() |
| + a.sweepOnce() |
| + if got := b.sidecarStartCount(); got != 5 { |
| + t.Fatalf("post-healthy restart: attempts = %d, want 5", got) |
| + } |
| + // …which a 3s advance clears (16s — the unreset ladder — would not). |
| + kill() |
| + clk.advance(3 * time.Second) |
| + a.sweepOnce() |
| + if got := b.sidecarStartCount(); got != 6 { |
| + t.Fatalf("backoff not reset after a healthy run: attempts = %d, want 6", got) |
| + } |
| +} |
| + |
| +// The activity question (#550 Part 4 v2): the co-spawned sidecar dials its |
| +// instance's public URL through this very router (the box pins its hosted |
| +// names to loopback), so without the marker its keepalives would be member |
| +// activity and the pair could never sleep. Full front-door check: marked |
| +// requests are served while awake — with the marker stripped before the |
| +// backend — but stamp nothing (the host hibernates on schedule beneath them), |
| +// and once asleep they get 503 and wake nothing. An outsider spoofing the |
| +// marker gets exactly that same treatment: it can't wake the host and can't |
| +// suppress anyone else's stamps (stamps are per-request), so the marker is |
| +// not a DoS lever in either direction. |
| +func TestActivatorSidecarRequestsInvisibleToActivity(t *testing.T) { |
| + a, b, clk, reg := testActivator(t, 10*time.Second) |
| + |
| + // A live backend listener stands in for the child a wake would have |
| + // spawned (the fake backend's start() spawns nothing), recording whether |
| + // the sidecar marker ever leaks through the proxy. |
| + sock := tmpSock(t) |
| + ln, err := net.Listen("unix", sock) |
| + if err != nil { |
| + t.Fatalf("listen unix: %v", err) |
| + } |
| + var leaked atomic.Bool |
| + srv := &http.Server{Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { |
| + if r.Header.Get(bot.SidecarHeader) != "" { |
| + leaked.Store(true) |
| + } |
| + io.WriteString(w, "ok") |
| + })} |
| + go srv.Serve(ln) |
| + t.Cleanup(func() { srv.Close() }) |
| + |
| + const host = "bots.example.com" |
| + if err := reg.upsert(Route{Host: host, Kind: "instance", Socket: sock, Hibernate: true}); err != nil { |
| + t.Fatal(err) |
| + } |
| + rt, err := newRouter(reg, a) |
| + if err != nil { |
| + t.Fatalf("new router: %v", err) |
| + } |
| + front := httptest.NewServer(rt) |
| + t.Cleanup(front.Close) |
| + |
| + markedGet := func() int { |
| + req, _ := http.NewRequest("GET", front.URL+"/", nil) |
| + req.Host = host |
| + req.Header.Set(bot.SidecarHeader, "1") |
| + resp, err := http.DefaultClient.Do(req) |
| + if err != nil { |
| + t.Fatalf("marked request: %v", err) |
| + } |
| + defer resp.Body.Close() |
| + io.Copy(io.Discard, resp.Body) |
| + return resp.StatusCode |
| + } |
| + |
| + // A member request wakes the host. |
| + if body, code := proxyGet(t, front.URL, host); code != 200 || body != "ok" { |
| + t.Fatalf("member wake: got %d %q", code, body) |
| + } |
| + // Marked requests are served while awake… |
| + if code := markedGet(); code != 200 { |
| + t.Fatalf("marked request while awake: %d, want 200", code) |
| + } |
| + // …but keep stamping nothing: sidecar traffic riding through the whole |
| + // idle window must not delay the sleep. |
| + clk.advance(6 * time.Second) |
| + if code := markedGet(); code != 200 { |
| + t.Fatalf("marked request mid-window: %d, want 200", code) |
| + } |
| + clk.advance(5 * time.Second) // 11s since the member request |
| + a.sweepOnce() |
| + waitFor(t, "hibernation beneath sidecar traffic", func() bool { return a.entryCount() == 0 }) |
| + |
| + // Asleep: marked requests are refused and wake NOTHING — this is also the |
| + // outsider-spoof case, and it's harmless. |
| + if code := markedGet(); code != 503 { |
| + t.Fatalf("marked request while asleep: %d, want 503", code) |
| + } |
| + if a.entryCount() != 0 { |
| + t.Fatal("a marked request created an entry (a wake) for a sleeping host") |
| + } |
| + if _, _, starts := b.counts(); starts != 1 { |
| + t.Fatalf("starts = %d after marked requests, want 1 (no sidecar-triggered wake)", starts) |
| + } |
| + // An unmarked request still wakes it, and the marker never reached the |
| + // backend on any request. |
| + if _, code := proxyGet(t, front.URL, host); code != 200 { |
| + t.Fatalf("member re-wake: %d, want 200", code) |
| + } |
| + if _, _, starts := b.counts(); starts != 2 { |
| + t.Fatalf("starts = %d after member re-wake, want 2", starts) |
| + } |
| + if leaked.Load() { |
| + t.Fatal("the sidecar marker leaked through the proxy to the backend") |
| + } |
| +} |
| + |
| func TestHibernateConfigFromEnv(t *testing.T) { |
| t.Setenv("LCHAT_HIBERNATE_BUCKET", "") |
| if cfg, err := hibernateConfigFromEnv(); err != nil || cfg != nil { |
diff --git a/bot.go b/bot.go| index 269ea5f..ee39878 100644 |
| --- a/bot.go |
| +++ b/bot.go |
| @@ -9,6 +9,15 @@ import ( |
| "strings" |
| ) |
| +// botEnvPath / botSessionDir: the per-instance sidecar convention — the |
| +// EnvironmentFile add-bot writes (which lchat-bot@.service and the activator's |
| +// co-spawn both exec against) and the session dir the sidecar persists its |
| +// identities in. "Does this host have a sidecar?" is defined as botEnvPath |
| +// existing — derived from the filesystem everywhere, the way the systemd units |
| +// do it, never a registry column that could drift from what's staged. |
| +func botEnvPath(host string) string { return filepath.Join(instanceDataDir, host+"-bot.env") } |
| +func botSessionDir(host string) string { return filepath.Join(instanceDataDir, host+"-bot") } |
| + |
| // runAddBot: `lchat add-bot <host>` — provision the Node sidecar (Claude + |
| // Eleven) for an instance. It mints an agent invite per identity via the |
| // instance's admin API (through agent/mint-invite.mjs, which reuses |
| @@ -57,7 +66,7 @@ func runAddBot(args []string) { |
| base := "https://" + host |
| mintScript := filepath.Join(*agentDir, "mint-invite.mjs") |
| - sessionDir := filepath.Join(instanceDataDir, host+"-bot") |
| + sessionDir := botSessionDir(host) |
| // One agent invite per identity. Both are catalog-available so Eleven's |
| // roster baseline sees the joined members (docs/bots.md). (The Announcer is |
| @@ -91,8 +100,8 @@ func runAddBot(args []string) { |
| if err := os.MkdirAll(sessionDir, 0o750); err != nil { |
| die("add-bot: mkdir %s: %v", sessionDir, err) |
| } |
| - botEnvPath := filepath.Join(instanceDataDir, host+"-bot.env") |
| - if err := writeEnvFile(botEnvPath, botEnv); err != nil { |
| + envPath := botEnvPath(host) |
| + if err := writeEnvFile(envPath, botEnv); err != nil { |
| die("add-bot: write bot env: %v", err) |
| } |
| // add-bot runs as root, so MkdirAll/writeEnvFile leave these root-owned. But the |
| @@ -100,16 +109,23 @@ func runAddBot(args []string) { |
| // into sessionDir — otherwise the sidecar can't persist, and the first boot |
| // crash-loops after burning its one-time invites. Hand both to the service user. |
| chownToService(sessionDir) |
| - chownToService(botEnvPath) |
| + chownToService(envPath) |
| unit := *botUnit + host |
| - if err := systemctl("enable", "--now", unit); err != nil { |
| - die("add-bot: start %s: %v (invites minted; env written to %s)", unit, err, botEnvPath) |
| + if rt.Hibernate { |
| + // An activator-managed host co-spawns its sidecar with each wake |
| + // (activator.go) — starting the always-on unit here would double-run |
| + // it. Staging the env file is enough: the sidecar joins on the host's |
| + // next wake and provisions itself from the invites then. |
| + fmt.Printf("added bot sidecar for %s (hibernating host: the activator co-spawns it on the next wake — no %s unit)\n", host, unit) |
| + } else if err := systemctl("enable", "--now", unit); err != nil { |
| + die("add-bot: start %s: %v (invites minted; env written to %s)", unit, err, envPath) |
| + } else { |
| + fmt.Printf("added bot sidecar for %s (%s)\n", host, unit) |
| } |
| - fmt.Printf("added bot sidecar for %s (%s)\n", host, unit) |
| if apiKey == "" { |
| fmt.Printf(" note: no ANTHROPIC_API_KEY in env — bots are deterministic-only (Eleven welcomes +\n"+ |
| - " slash commands); add ANTHROPIC_API_KEY to %s and restart %s for AI replies\n", botEnvPath, unit) |
| + " slash commands); add ANTHROPIC_API_KEY to %s and restart %s for AI replies\n", envPath, unit) |
| } |
| fmt.Printf(" logs: journalctl -u %s -f\n", unit) |
| } |
diff --git a/bot_run_test.go b/bot_run_test.go| index 6b59c5a..62f52f0 100644 |
| --- a/bot_run_test.go |
| +++ b/bot_run_test.go |
| @@ -9,6 +9,7 @@ import ( |
| "os" |
| "strings" |
| "sync" |
| + "sync/atomic" |
| "testing" |
| "time" |
| @@ -490,3 +491,45 @@ func TestBotEngineChangelogBadge(t *testing.T) { |
| t.Fatalf("announcer with a key present should badge bot, got %q", kind) |
| } |
| } |
| + |
| +// The live sidecar marks EVERY request it makes to its instance — the |
| +// WebSocket dial and all REST calls — with bot.SidecarHeader, so a hibernating |
| +// instance's router can exclude bot traffic from its idle clock and refuse to |
| +// be woken by a sidecar reconnect loop (activator.go). Pinned at the wire: a |
| +// recording front door in the engine's path fails the test on any unmarked |
| +// request. (Provisioning happens against the unwrapped server on purpose — |
| +// claiming an invite is an operator action and stays unmarked.) |
| +func TestBotRuntimeMarksAllRequests(t *testing.T) { |
| + ts, s := newTestServer(t) |
| + handler := s.handler() |
| + var seen atomic.Int32 |
| + front := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { |
| + seen.Add(1) |
| + if r.Header.Get(bot.SidecarHeader) == "" { |
| + t.Errorf("unmarked sidecar request: %s %s", r.Method, r.URL.Path) |
| + } |
| + handler.ServeHTTP(w, r) |
| + })) |
| + t.Cleanup(front.Close) |
| + |
| + eleven := provisionNamed(t, ts.URL, "Eleven", true) |
| + sessionPath := t.TempDir() + "/eleven.session.json" |
| + if err := eleven.Save(sessionPath); err != nil { |
| + t.Fatal(err) |
| + } |
| + |
| + ready := make(chan struct{}) |
| + ctx, cancel := context.WithCancel(context.Background()) |
| + defer cancel() |
| + eng := &bot.Engine{Base: front.URL, Session: eleven, SessionPath: sessionPath, |
| + Recipe: bot.DefaultRecipe(), Ready: func() { close(ready) }, Logf: t.Logf} |
| + go eng.Run(ctx) |
| + awaitReady(t, ready) |
| + |
| + // Ready fires after the connect sequence — config, roster, threads, |
| + // declare-kind, catch-up, and the WS upgrade itself — all through the |
| + // front door; make sure it actually saw them. |
| + if seen.Load() < 5 { |
| + t.Fatalf("front door saw only %d engine requests", seen.Load()) |
| + } |
| +} |
diff --git a/control.go b/control.go| index c923357..843fe14 100644 |
| --- a/control.go |
| +++ b/control.go |
| @@ -588,6 +588,7 @@ func runHibernate(args []string) { |
| fs := flag.NewFlagSet("hibernate", flag.ExitOnError) |
| registryPath := fs.String("registry", envOr("LCHAT_REGISTRY", "/data/registry.db"), "router registry SQLite path") |
| unitTmpl := fs.String("unit", envOr("LCHAT_UNIT", "lchat@"), "systemd instance unit template prefix") |
| + botUnit := fs.String("bot-unit", envOr("LCHAT_BOT_UNIT", "lchat-bot@"), "systemd bot unit template prefix") |
| routerUnit := fs.String("router-unit", envOr("LCHAT_ROUTER_UNIT", "lchat-router"), "router systemd unit to reload") |
| on := fs.Bool("on", false, "hand the instance to the activator: stop+disable its unit; the router wakes it per request") |
| off := fs.Bool("off", false, "hand the instance back to systemd: the activator releases it, then its unit is enabled+started") |
| @@ -622,18 +623,20 @@ func runHibernate(args []string) { |
| fmt.Printf("%s is already hibernating\n", host) |
| return |
| } |
| - if err := hibernateOn(reg, rt, *unitTmpl, *routerUnit); err != nil { |
| + if err := hibernateOn(reg, rt, *unitTmpl, *botUnit, *routerUnit); err != nil { |
| die("hibernate %s --on: %v", host, err) |
| } |
| fmt.Printf("%s is now activator-managed: unit stopped+disabled; the router wakes it per request and sleeps it when idle\n", host) |
| - fmt.Printf(" note: a bot sidecar (lchat-bot@%s) keeps a WebSocket open and would hold it awake forever — stop the sidecar if this host should actually sleep\n", host) |
| + if _, err := os.Stat(botEnvPath(host)); err == nil { |
| + fmt.Printf(" bot sidecar: co-managed by the activator — spawned on each wake, stopped at each sleep (%s%s disabled; --off hands it back to systemd)\n", *botUnit, host) |
| + } |
| return |
| } |
| if !rt.Hibernate { |
| fmt.Printf("%s is not hibernating\n", host) |
| return |
| } |
| - if err := hibernateOff(reg, rt, *unitTmpl, *routerUnit); err != nil { |
| + if err := hibernateOff(reg, rt, *unitTmpl, *botUnit, *routerUnit); err != nil { |
| die("hibernate %s --off: %v", host, err) |
| } |
| fmt.Printf("%s is back under systemd: unit enabled and healthy; the activator has released it\n", host) |
| @@ -653,6 +656,8 @@ func runHibernate(args []string) { |
| // is dead until the wake in step 4 — and requests in it ride the router's |
| // 13s dial-retry window, the same shape as a deploy restart. Disable (not |
| // just stop) so a reboot can't resurrect the unit to fight the activator. |
| +// The bot sidecar unit (if the host has one staged) goes down here too — |
| +// the activator co-spawns the sidecar with each wake from now on. |
| // 3. Reload the router: the route becomes activator-managed. |
| // 4. A warm request through the public path — best-effort, but it both |
| // closes the gap immediately (the wake binds the socket, so any retrying |
| @@ -662,7 +667,7 @@ func runHibernate(args []string) { |
| // Nothing is checkpointed here and no S3 is touched: the DB is already local, |
| // so the first wake's restore sees it as the warm cache and skips; replication |
| // under the fresh lease is what backfills S3. |
| -func hibernateOn(reg *Registry, rt *Route, unitTmpl, routerUnit string) error { |
| +func hibernateOn(reg *Registry, rt *Route, unitTmpl, botUnit, routerUnit string) error { |
| // Existing instances' env files were written for the systemd path (root |
| // reads them); the router's activator reads them as the service user, so |
| // hand the file over first — a flip that can't wake is worse than no flip. |
| @@ -675,6 +680,22 @@ func hibernateOn(reg *Registry, rt *Route, unitTmpl, routerUnit string) error { |
| if err := systemctl("disable", "--now", unitTmpl+rt.Host); err != nil { |
| return fmt.Errorf("stop unit: %v (flag is set but the unit still runs — the host serves as before; re-run, or undo with `lchat hibernate %s --off`)", err, rt.Host) |
| } |
| + if _, err := os.Stat(botEnvPath(rt.Host)); err == nil { |
| + // The bot sidecar hands over too: from here the activator co-spawns it |
| + // with each wake and stops it at each sleep, so the always-on unit |
| + // must go — two sidecars would double every bot reply. Before the |
| + // reload, like the instance unit, so a systemd sidecar and a |
| + // co-spawned one never overlap. Best-effort: a sidecar that won't |
| + // stop cleanly must not strand the flip (members > bots). |
| + _ = systemctl("disable", "--now", botUnit+rt.Host) |
| + // Same ownership handover as the instance env above, for the pieces |
| + // the co-spawned sidecar (a router child, running as the service |
| + // user) reads and writes. Non-recursive on purpose: the session files |
| + // inside were already written by the unit's service user, and add-bot |
| + // chowns fresh ones at creation. |
| + _ = matchOwner(botEnvPath(rt.Host), instanceDataDir) |
| + _ = matchOwner(botSessionDir(rt.Host), instanceDataDir) |
| + } |
| if err := systemctl("reload", routerUnit); err != nil { |
| return fmt.Errorf("reload %s: %v (unit is stopped — reload the router by hand to bring the host back)", routerUnit, err) |
| } |
| @@ -701,10 +722,12 @@ func hibernateOn(reg *Registry, rt *Route, unitTmpl, routerUnit string) error { |
| // design once the flag is off, and a later flip back on wakes from the |
| // local DB (the warm cache, which is the newer truth). |
| // 4. Enable + start the unit and health-check it, exactly like add-instance. |
| +// A staged bot sidecar then goes back on its always-on unit (instance |
| +// first, then sidecar — the same bootstrap order add-bot documents). |
| // |
| // Requests between the child's stop and the unit binding ride the router's 13s |
| // dial-retry window, same as a deploy restart. |
| -func hibernateOff(reg *Registry, rt *Route, unitTmpl, routerUnit string) error { |
| +func hibernateOff(reg *Registry, rt *Route, unitTmpl, botUnit, routerUnit string) error { |
| if err := reg.setHibernate(rt.Host, false); err != nil { |
| return fmt.Errorf("flag: %v", err) |
| } |
| @@ -721,6 +744,16 @@ func hibernateOff(reg *Registry, rt *Route, unitTmpl, routerUnit string) error { |
| if err := waitHealthy(rt.Socket, 20*time.Second); err != nil { |
| return fmt.Errorf("%v (unit started but unhealthy — check journalctl -u %s)", err, unit) |
| } |
| + // Hand the sidecar back too: the co-spawned one died with the activator's |
| + // child (the retire's stop takes the sidecar down first), so a host with |
| + // one staged returns to its always-on unit — after the instance is |
| + // healthy, matching add-bot's "instance first" bootstrap order. A failure |
| + // here doesn't fail the flip (the instance is fine); tell the operator. |
| + if _, err := os.Stat(botEnvPath(rt.Host)); err == nil { |
| + if err := systemctl("enable", "--now", botUnit+rt.Host); err != nil { |
| + fmt.Printf(" warning: sidecar unit %s%s failed to start: %v — start it by hand\n", botUnit, rt.Host, err) |
| + } |
| + } |
| return nil |
| } |
diff --git a/control_test.go b/control_test.go| index 2fa6732..38d48a5 100644 |
| --- a/control_test.go |
| +++ b/control_test.go |
| @@ -553,7 +553,7 @@ func TestHibernateOnOrdering(t *testing.T) { |
| confirmPublicReady = func(h, _ string) { calls = append(calls, "confirm "+h) } |
| rt, _ := reg.get(host) |
| - if err := hibernateOn(reg, rt, "lchat@", "lchat-router"); err != nil { |
| + if err := hibernateOn(reg, rt, "lchat@", "lchat-bot@", "lchat-router"); err != nil { |
| t.Fatalf("hibernateOn: %v", err) |
| } |
| @@ -628,7 +628,7 @@ func TestHibernateOffOrdering(t *testing.T) { |
| } |
| rt, _ := reg.get(host) |
| - if err := hibernateOff(reg, rt, "lchat@", "lchat-router"); err != nil { |
| + if err := hibernateOff(reg, rt, "lchat@", "lchat-bot@", "lchat-router"); err != nil { |
| t.Fatalf("hibernateOff: %v", err) |
| } |
| want := []string{"reload lchat-router", "enable --now lchat@" + host} |
| @@ -640,6 +640,69 @@ func TestHibernateOffOrdering(t *testing.T) { |
| } |
| } |
| +// A host with a bot sidecar staged (its -bot.env exists) hands the sidecar |
| +// over at the hibernate flips: --on disables the always-on unit BEFORE the |
| +// router reload (the activator co-spawns from then on; a systemd sidecar and |
| +// a co-spawned one would double every bot reply), and --off re-enables it |
| +// LAST — only after the instance unit is healthy, add-bot's "instance first" |
| +// bootstrap order. A host without one (the two ordering tests above) never |
| +// touches the bot unit. |
| +func TestHibernateFlipHandsSidecarUnitOver(t *testing.T) { |
| + reg := controlTestEnv(t) |
| + const host = "flip-bots.example.com" |
| + sock := filepath.Join(instanceSocketDir, host+".sock") |
| + if err := reg.upsert(Route{Host: host, Kind: "instance", Socket: sock, Status: "ready"}); err != nil { |
| + t.Fatal(err) |
| + } |
| + // The instance env (hibernateOn chowns it) and the sidecar staging — |
| + // exactly what add-instance + add-bot leave on disk. |
| + if err := os.WriteFile(filepath.Join(instanceDataDir, host+".env"), []byte("LCHAT_ADMIN_TOKEN=t\n"), 0o600); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := os.WriteFile(botEnvPath(host), []byte("LCHAT_BASE=https://"+host+"\n"), 0o600); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := os.MkdirAll(botSessionDir(host), 0o750); err != nil { |
| + t.Fatal(err) |
| + } |
| + |
| + var calls []string |
| + systemctl = func(args ...string) error { |
| + call := strings.Join(args, " ") |
| + calls = append(calls, call) |
| + if call == "enable --now lchat@"+host { |
| + // Bind like a started unit so hibernateOff's health check passes. |
| + ln, err := net.Listen("unix", sock) |
| + if err != nil { |
| + return err |
| + } |
| + srv := &http.Server{Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(200) })} |
| + go srv.Serve(ln) |
| + t.Cleanup(func() { srv.Close() }) |
| + } |
| + return nil |
| + } |
| + |
| + rt, _ := reg.get(host) |
| + if err := hibernateOn(reg, rt, "lchat@", "lchat-bot@", "lchat-router"); err != nil { |
| + t.Fatalf("hibernateOn: %v", err) |
| + } |
| + wantOn := []string{"disable --now lchat@" + host, "disable --now lchat-bot@" + host, "reload lchat-router"} |
| + if !reflect.DeepEqual(calls, wantOn) { |
| + t.Errorf("flip-on sequence = %v, want %v", calls, wantOn) |
| + } |
| + |
| + calls = nil |
| + rt, _ = reg.get(host) |
| + if err := hibernateOff(reg, rt, "lchat@", "lchat-bot@", "lchat-router"); err != nil { |
| + t.Fatalf("hibernateOff: %v", err) |
| + } |
| + wantOff := []string{"reload lchat-router", "enable --now lchat@" + host, "enable --now lchat-bot@" + host} |
| + if !reflect.DeepEqual(calls, wantOff) { |
| + t.Errorf("flip-off sequence = %v, want %v", calls, wantOff) |
| + } |
| +} |
| + |
| // waitSocketQuiet is the flip-off's release observation: quiet means no |
| // listener (dial failure), whether or not the socket file lingers. |
| func TestWaitSocketQuiet(t *testing.T) { |
diff --git a/docs/bots.md b/docs/bots.md| index eb9beac..5e52af5 100644 |
| --- a/docs/bots.md |
| +++ b/docs/bots.md |
| @@ -77,6 +77,16 @@ is a server**, and adding one to a thread extends that thread's E2E envelope to |
| the machine the operator chose. Where that machine lives is the single biggest |
| security decision the operator makes about bots. |
| +One wrinkle of being a persistent outbound client: on a **hibernating** |
| +instance (the activator, `activator.go`), an always-on sidecar would either |
| +hold the instance awake forever or wake it right back on every reconnect. So |
| +there the sidecar is **co-spawned** — started by the activator when its |
| +instance wakes, stopped first when it sleeps — and every request the Go |
| +sidecar makes carries a marker header the router excludes from the instance's |
| +idle clock (`internal/bot/sidecarmark.go`). None of this changes the trust |
| +model: the sidecar is still its own process with its own keys, still outbound- |
| +only. See `docs/instance-router.md § Sidecars on hibernating instances`. |
| + |
| ## Direction: a Go sidecar (decided 2026-07-06, shipped 2026-07-07) |
| The sidecar was ported to **Go** — a subcommand of the single binary (`lchat |
diff --git a/docs/instance-router.md b/docs/instance-router.md| index d8f2755..2b1d9fa 100644 |
| --- a/docs/instance-router.md |
| +++ b/docs/instance-router.md |
| @@ -249,6 +249,35 @@ regenerate `/etc/hosts` on boot, disable that so the pins persist.) |
| Its secrets (an API key, the persisted agent sessions) live in its own env + |
| disk, the same way the messenger's admin token lives in the instance DB. |
| +### Sidecars on hibernating instances: co-spawn |
| + |
| +A **hibernating** instance (activator-managed, `lchat hibernate <host> --on`) |
| +doesn't keep its sidecar on the always-on unit — a persistent WebSocket client |
| +would either hold the sleeping instance awake forever or turn its reconnect |
| +loop into a wake loop. Instead the activator **co-spawns** the sidecar: when a |
| +wake brings the instance up (socket healthy, replication running), it also |
| +execs `live/<host> bot run` under `/data/<host>-bot.env` — the same file the |
| +unit reads; "has a sidecar" *is* that file existing — and at sleep it SIGTERMs |
| +the sidecar *first*, then stops the instance. The sidecar exists exactly while |
| +its instance is awake. If it dies in between, the activator restarts it with |
| +capped exponential backoff (`Restart=always`, cheaply); if it won't start at |
| +all, the instance still serves — members before bots. |
| + |
| +Because the sidecar dials the instance's *public URL through the router* (the |
| +loopback pin above), its traffic marks itself with a header |
| +(`X-Lchat-Sidecar`, `internal/bot/sidecarmark.go`) that the activator excludes |
| +from the idle clock and refuses to be woken by — otherwise the pair could |
| +never sleep. The marker is spoofable-but-harmless: it only downgrades the |
| +sender's own requests (no activity credit; 503 while asleep) and the router |
| +strips it before proxying. The Node reference sidecar (`agent/`) does **not** |
| +send the marker, so it isn't suitable for a hibernating instance — use the Go |
| +sidecar there. The flip commands hand the unit over automatically: |
| +`hibernate --on` disables `lchat-bot@<host>` (co-spawn takes over), |
| +`hibernate --off` re-enables it once the instance unit is healthy. One honest |
| +gap: member-bot charters run in the sidecar, so a sleeping instance's charter |
| +schedules don't fire until something else wakes it — the due-time index covers |
| +server-side schedules (future messages, reminders) only. |
| + |
| ### Migrating a Node sidecar to Go |
| The session files are **byte-compatible** (`internal/bot/session.go` matches the |
diff --git a/internal/bot/announcer.go b/internal/bot/announcer.go| index 1669645..99f26d5 100644 |
| --- a/internal/bot/announcer.go |
| +++ b/internal/bot/announcer.go |
| @@ -54,7 +54,9 @@ type announcerState struct { |
| // announcer announces what ships *after* it, not history. |
| func (a *Announcer) Tick() error { |
| if a.client == nil { |
| - a.client = &http.Client{Timeout: 30 * time.Second} |
| + // Marked like all sidecar-to-instance traffic (sidecarmark.go), so a |
| + // recipe's announce poller can't hold a hibernating instance awake. |
| + a.client = &http.Client{Timeout: 30 * time.Second, Transport: markTransport{}} |
| } |
| a.declareKind() |
diff --git a/internal/bot/runtime.go b/internal/bot/runtime.go| index ce07576..95207d7 100644 |
| --- a/internal/bot/runtime.go |
| +++ b/internal/bot/runtime.go |
| @@ -197,7 +197,10 @@ type threadEntry struct { |
| // Run connects and serves until ctx is cancelled, reconnecting with backoff on |
| // any drop — the same loop shape as the Node runtime's ws.on("close"). |
| func (r *Runtime) Run(ctx context.Context) error { |
| - r.client = &http.Client{Timeout: 30 * time.Second} |
| + // markTransport stamps every REST call with SidecarHeader (sidecarmark.go) |
| + // so a hibernating instance's router can exclude bot traffic from its idle |
| + // clock; a non-hibernating instance ignores it. |
| + r.client = &http.Client{Timeout: 30 * time.Second, Transport: markTransport{}} |
| r.keys = map[string]threadEntry{} |
| r.cursors = map[string]int64{} |
| r.agents = map[string]bool{} |
| @@ -248,7 +251,11 @@ func (r *Runtime) Run(ctx context.Context) error { |
| func (r *Runtime) connectOnce(ctx context.Context) error { |
| wsBase := strings.Replace(r.Base, "http", "ws", 1) |
| - conn, _, err := websocket.Dial(ctx, wsBase+"/ws?token="+url.QueryEscape(r.Session.Token), nil) |
| + // The WebSocket carries the sidecar marker too — it's the long-lived |
| + // connection the activity exclusion exists for (sidecarmark.go). |
| + conn, _, err := websocket.Dial(ctx, wsBase+"/ws?token="+url.QueryEscape(r.Session.Token), &websocket.DialOptions{ |
| + HTTPHeader: http.Header{SidecarHeader: []string{"1"}}, |
| + }) |
| if err != nil { |
| return err |
| } |
diff --git a/internal/bot/sidecarmark.go b/internal/bot/sidecarmark.go| new file mode 100644 |
| index 0000000..f40f4a3 |
| --- /dev/null |
| +++ b/internal/bot/sidecarmark.go |
| @@ -0,0 +1,32 @@ |
| +package bot |
| + |
| +import "net/http" |
| + |
| +// SidecarHeader marks every request the live sidecar makes to its instance — |
| +// the WebSocket dial and all REST calls — so a hibernation-aware router |
| +// (activator.go) can tell sidecar traffic from member traffic. A co-spawned |
| +// sidecar's connections flow through its own instance's router (the box pins |
| +// its hosted names to loopback), and without the marker its keepalives would |
| +// read as member activity: the instance could never idle, and after a sleep |
| +// the reconnect loop would be a standing wake source. Marked requests are |
| +// served while the instance is awake but stamp no activity, and never wake a |
| +// sleeping one. |
| +// |
| +// The marker is deliberately spoofable-but-harmless: it only DOWNGRADES the |
| +// sender's own requests (no activity credit; 503 while the instance sleeps). |
| +// It cannot suppress other clients' activity stamps and cannot wake anything, |
| +// so an outsider sending it gains nothing. Provisioning (provision.go) stays |
| +// unmarked on purpose — claiming an invite is an operator action and should |
| +// behave like a person's request. |
| +const SidecarHeader = "X-Lchat-Sidecar" |
| + |
| +// markTransport stamps SidecarHeader on every request — the Runtime's HTTP |
| +// client uses it so no REST call site can forget the marker. It clones the |
| +// request first, per the RoundTripper contract. |
| +type markTransport struct{} |
| + |
| +func (markTransport) RoundTrip(req *http.Request) (*http.Response, error) { |
| + req = req.Clone(req.Context()) |
| + req.Header.Set(SidecarHeader, "1") |
| + return http.DefaultTransport.RoundTrip(req) |
| +} |
diff --git a/router.go b/router.go| index 6482817..56503ee 100644 |
| --- a/router.go |
| +++ b/router.go |
| @@ -17,6 +17,8 @@ import ( |
| "time" |
| "golang.org/x/crypto/acme/autocert" |
| + |
| + "lchat/internal/bot" |
| ) |
| // router is the multi-instance front door: it terminates TLS (autocert) and |
| @@ -158,6 +160,10 @@ func unixProxy(socketPath string) *httputil.ReverseProxy { |
| pr.Out.URL.Scheme = "http" |
| pr.Out.URL.Host = "lchat" // placeholder; the unix dialer ignores it |
| pr.Out.Host = pr.In.Host // backend sees the real hostname |
| + // The sidecar marker is a router-edge signal only (activity |
| + // exclusion, activator.go); strip it so nothing downstream can |
| + // ever grow a meaning an outsider could spoof. |
| + pr.Out.Header.Del(bot.SidecarHeader) |
| }, |
| Transport: &http.Transport{ |
| DialContext: func(ctx context.Context, _, _ string) (net.Conn, error) { |