🤖 The instance router — a single-machine bootstrap

Status: shipped — this page is the design rationale, not a procedure. The machinery described here is live (lchat bootstrap/router/add-instance…), and day-to-day operation of it goes through ./deploy/ops — if something here reads like steps to run and no ops verb covers it (e.g. hardening a brand new box), the move is to add the verb, not to follow the prose.

The problem

The messenger is a single static, no-cgo binary with SQLite compiled in, and it runs as one fully-isolated instance per chat — each with its own disk, admin token, and subdomain. The obvious way to run several of those on one machine is to reach for an orchestrator (Kamal, a PaaS, Kubernetes): it gives you TLS, subdomain routing, a per-app disk, env injection, and process supervision.

But that's a lot of machinery around a single file you could scp. This doc describes a lighter alternative for the common case — many instances on one machine — that keeps the two properties process-per-instance gives you and that are worth keeping:

• Per-instance versioning / canary. Deploy one instance on a new build while the others stay on the old one. A single shared process can't do that.
• Isolation. A crash or a resource hog in one instance must not touch the others.

…and adds zero-downtime deploys. Those reinforce each other: zero-downtime means "run the new version alongside the old and hand over," so per-instance versioning falls out of doing it right.

No container platform required — just the binary, a small router, and systemd.

One binary, many ways to deploy

The messenger makes no assumptions about how it's run. Put it behind nginx or Caddy, on a PaaS, in Kubernetes, under Kamal, or one-container-per-instance on any platform — all of those work, and nothing here precludes them. The instance binary stands alone.

What this doc describes is a batteries-included path for the common case: bootstrapping a single machine from zero to a running, auto-TLS'd, multi-instance box with zero-downtime deploys and no external orchestrator. It's the simplest way, not the only way — the router below is optional convenience.

The shape

              :443 / :80
  internet ──▶ lchat router ──(reverse proxy over unix sockets)──▶ lchat instance (chat-a)  ── chat-a.db
               (autocert HTTP-01,                                  lchat instance (chat-b)  ── chat-b.db
                HostPolicy = route table,                          lchat instance (chat-c)  ── chat-c.db
                atomic upstream map)                               …
                    │                                              home server (home/)      ── home.db   ┐ service
                    │                                              push relay  (push/)       ── push.db   ┘ routes
                    └── control API on a localhost-only socket (add / deploy / list)

  systemd supervises everything: router + lchat@<host> (instance) + lchat-bot@<host> (sidecar) + home/push units

One binary, three roles (subcommands):

• lchat router — the front door (TLS + routing + control plane).
• lchat instance — one messenger instance (today's main() minus the TCP listener and minus the process-global env reads).
• control subcommands — add-instance, deploy, instances, admin-link.

The router

• :443 — TLS via golang.org/x/crypto/acme/autocert (stdlib-adjacent, no DNS-API credentials). HostPolicy returns nil only for hosts in the route table, so the route list is the cert allowlist. Cache: autocert.DirCache on the persistent disk. HTTP-01 (well-known) is the challenge; a brand-new host is issued on first hit — no wildcard, no DNS-provider step.
• :80 — m.HTTPHandler(nil): serves /.well-known/acme-challenge/…, 301-redirects everything else to https.
• Routing — the route table maps host → backend (a unix socket at a version); httputil.ReverseProxy dials it. The upstream map is an atomic.Pointer so reloads and deploys swap it without dropping a connection. WebSocket upgrades pass through (Go ≥1.12). A backend is usually one instance — see Backends, below.
• Unknown hosts — 404 by default; with --fallback-url (env LCHAT_FALLBACK_URL, e.g. the apex marketing site) they 302 there instead, so under a wildcard DNS record a typo'd or retired subdomain lands somewhere useful. Scope caveat: on :80 this always works, but over HTTPS it only helps hosts that still have a cert (e.g. a removed instance) — a never-served host fails the TLS handshake before any redirect, because HostPolicy refuses to mint certs for unknown hosts (anything else would let strangers burn the ACME rate limit). Full coverage needs the wildcard-cert option below.
• Control API — shipped as lchat controld (lchat-control.service): a local-only unix socket (/run/lchat/control.sock), gated by a root-only bearer token, never exposed on a served hostname. This is the god surface (it creates instances), so it stays off the public interface; its one consumer is the cloud provisioner (lchat cloud), the public, unprivileged policy layer for self-serve space creation. Full design: cloud-provisioner.md.

Threat model is unchanged: TLS terminates at the router, but message bodies are already E2E ciphertext — the router sees exactly what the app sees, never plaintext.

🤖 Wildcard TLS (DNS-01) — implemented

HTTP-01 issues one cert per host on first hit — simple, needs no DNS credentials, but rides the Let's Encrypt per-registered-domain rate limit (~50 new certs/week, a hard ceiling on space creation) and can't cover a host before it's requested. So the router can also hold a wildcard (*.example.com) obtained over ACME DNS-01 (wildcard.go): set LCHAT_WILDCARD_DOMAINS=example.com (comma list) and a DNS-01 provider (below) in /data/router.env and restart. The two coexist — the wildcard serves any host exactly one label under a configured domain; everything else (other domains, the apex, deeper levels) takes the per-host HTTP-01 path, which also keeps covering the wildcard domains until the first issuance lands, so a DNS-provider/ACME outage degrades to today's behaviour rather than a dead router.

DNS-01 provider (pluggable; LCHAT_DNS_PROVIDER). The solver that publishes the challenge TXT is a seam (dnsProvider), so the wildcard works against more than one DNS API without changing anything else:

• DNSimple — the default. Leave LCHAT_DNS_PROVIDER unset (or set it to dnsimple) and provide LCHAT_DNSIMPLE_TOKEN, a DNSimple account API token that can manage the zone. This is the fully-supported path for self-hosters and production; nothing about it changed when the seam landed. A missing token leaves the wildcard off (per-host issuance continues).
• Route53 — the additive off-ramp (docs/multi-edge.md). Set LCHAT_DNS_PROVIDER=route53; the solver authenticates with the standard AWS credential chain (env vars or an EC2/container IAM role), so there is no DNS token on disk. The role is scoped to just the wildcard domain's hosted zone — this is how the dev boxes obtain a shared *.dev.11ch.at cert from their instance role. Route53 is global, so no region is required (the solver defaults the signing region to us-east-1; an explicit AWS_REGION wins).

Mechanics: a renewal loop checks at startup and every 12h (hourly after a failure), re-issuing when the cert is missing or <30 days from expiry. Issuance publishes a _acme-challenge.<domain> TXT via the selected provider (TTL 60, always cleaned up) — DNSimple via its records API, Route53 via ChangeResourceRecordSets (UPSERT, then a wait for the change to go INSYNC) — waits briefly for propagation, and finalizes with a fresh ECDSA key. The cert+key persist as wildcard-<domain>.pem (0600) in the acme cache dir — same backup story as the autocert cache — and load synchronously on restart; the ACME account key is wildcard-account.key alongside. HostPolicy still gates which hosts are actually served a backend; the wildcard only changes how the handshake gets a cert. Deliberate side effect: an unknown subdomain of a wildcard domain now completes TLS and gets the router's fallback/404 instead of a handshake error.

The instance process

lchat instance --db /data/<host>.db --socket /run/lchat/<host>.sock --config <file>

It's today's server with two changes: it listens on a unix socket instead of a TCP port, and it takes per-instance config from flags/a file the router writes rather than from process-global env (see below). Everything else is unchanged — its own SQLite (WAL), its own admin token (ensureAdminToken, in the DB), its own VAPID (ensureVAPID, in the DB). handler() is untouched, so newTestServer and the existing handler tests keep working as-is.

Backends: one per instance (default), one per version (option)

The router maps host → backend (a socket at a version) and doesn't care how many hosts a backend serves. That leaves two ways to slice processes:

One backend per instance (the default). Each instance is its own process, socket, and DB — full crash and resource isolation, and per-version deploys come for free: each unit execs a per-instance live symlink into a version store, so moving one instance to a new build is one lchat deploy <host> --version <ver> while the rest stay put (a trunk instance next to stable ones), no coordination. Canary is just "which binary does this unit run." See Versioned, side-by-side deploys below.

One backend per version (an option, later). Alternatively a single process serves many hosts at one version (an in-process host map, one DB per host), and the router sends each host to the process running its assigned version:

  stable-host  ─┐
  other-host   ─┼─▶ backend @ stable  (one process, many DBs)
  another-host ─┘
  trunk-host    ──▶ backend @ trunk   (one process)

This trades isolation for fewer processes — fewer Go runtimes (less RAM), fewer deploy targets — and the costs are real: a crash or OOM takes down every host on that version, there's no per-host resource cap, and deploying a version restarts all its hosts together (one big reconnect, not a staggered roll). Moving a host between versions also needs a clean SQLite hand-off (two processes must never write one DB file), so it suits hosts that live permanently on a channel (a dedicated trunk instance) more than ad-hoc per-instance canaries.

Recommendation: default to one-backend-per-instance — isolation is cheap to keep, per-version is already free, and a dozen processes cost a few hundred MB (RAM isn't the binding resource; disk is). Keep the route table as host → backend so consolidating onto per-version backends stays a config change, not a rewrite, and reach for it only if process count ever actually bites. It needs the in-process host map, so it's the second thing to build, not the first.

Beyond messenger instances: sibling services

A deployment usually runs sibling services next to the messenger that want the same TLS + host routing — for example the other two binaries in this repo:

• the home server (home/) — the passkey-backup store, its own binary + DB. Its host is the WebAuthn RP origin, so it must keep the same hostname across any move (only the DNS record changes) or it orphans every passkey.
• the push relay (push/) — a shared APNs relay that lets instances (yours and third parties') send iOS push without holding Apple's signing key (see push-relay.md).

To the router these are just another host → upstream mapping: a service-kind route to a local unix socket, TLS via autocert, no lifecycle management (they deploy on their own cadence). That's the only difference from a messenger instance, which the router also create/deploy/version-manages.

Because a route is nothing but host → upstream, anyone running their own home server or push relay is first-class — point a host at their backend the same way. This is deliberately the simple version (a flat mapping + a health check); it can be hardened — readiness gating, retries, per-service resource isolation — later, if it earns it.

Bots (sidecars)

An instance can have a sidecar hosting the default agents. It connects to its instance as a WebSocket client over the instance's public host — authenticating with a one-time agent invite and persisting its session to disk. It serves no inbound traffic, so it never needs a socket or a cert, and the router never routes to it.

The sidecar is the same lchat binary — lchat bot run (botcmd.go, internal/bot/). No separate runtime: the Go port mirrors crypto.js's wire format byte-for-byte (internal/e2ee, pinned by cross-language vectors generated from crypto.js itself), so its ciphertext can't drift from what clients expect, and the box that runs the server can run its bots with no extra install. The Node reference (agent/) stays in the repo as the functional example for self-hosters who prefer it, but the default deployment is Go.

Why still a separate process? Because a bot is a full E2E client — it holds its own keypair, and that private key must never live in the server process (the core rule). So a bot is a member that runs on a server, not a server feature — run wherever the operator wants the keys to live (same box, off-box, a Raspberry Pi), since it only ever dials out.

One process, the whole pair. lchat bot run -base https://<host> <claude>.session.json <eleven>.session.json runs both identities as goroutines in one process, each its own connection. The standard pair is zero-config: with no recipe file an identity gets the conversational default (command floor + model; badges AI when ANTHROPIC_API_KEY is set). A custom bot drops a <session>.recipe.json alongside its session (docs/bot-recipes.md) — behaviour as data, no code. (A legacy session named Announcer is skipped with a log line — that identity is retired; the server announces its own changelog now. See docs/bots.md.)

Supervise it as one lchat-bot@<host>.service per instance. The unit execs the same per-instance live symlink as the instance itself (ExecStart=/opt/lchat/live/%i bot run), so the bot runs the same version as its app by construction — and deploy/update restart the bot right after they repoint that symlink and health-check the instance (deployHostVersion → restartBotSidecar), so the sidecar's code never lags the server. bot run with no positional args reads LCHAT_BASE + the LCHAT_*_SESSION paths from the unit's EnvironmentFile, so the template is fully generic — a full trio or an Eleven-only box both work with no per-host args. Because the sidecar backfills on reconnect, that restart is a plain reconnect — no socket-swap, no drain. One wrinkle: it reaches its instance at the public URL through the router, so the box must resolve that public hostname to itself. Pin every hostname the box hosts to loopback in /etc/hosts (127.0.0.1 chat.example.com …) so on-box clients — sidecars, and an instance's own calls to a sibling service like the push relay — reach the local router directly (still over TLS+SNI, so the right cert is served) instead of hair-pinning out to the public IP. This also sidesteps a migration trap: after you move onto this box and flip DNS, an external resolver may still cache the old box's IP for these hosts, so an on-box client that resolves them externally can silently talk to the old box — or fail once it's gone. (On cloud images that regenerate /etc/hosts on boot, disable that so the pins persist.)

Its secrets (an API key, the persisted agent sessions) live in its own env + disk, the same way the messenger's admin token lives in the instance DB.

Sidecars on hibernating instances: co-spawn

A hibernating instance (activator-managed, lchat hibernate <host> --on) doesn't keep its sidecar on the always-on unit — a persistent WebSocket client would either hold the sleeping instance awake forever or turn its reconnect loop into a wake loop. Instead the activator co-spawns the sidecar: when a wake brings the instance up (socket healthy, replication running), it also execs live/<host> bot run under /data/<host>-bot.env — the same file the unit reads; "has a sidecar" is that file existing — and at sleep it SIGTERMs the sidecar first, then stops the instance. The sidecar exists exactly while its instance is awake. If it dies in between, the activator restarts it with capped exponential backoff (Restart=always, cheaply); if it won't start at all, the instance still serves — members before bots.

Because the sidecar dials the instance's public URL through the router (the loopback pin above), its traffic marks itself with a header (X-Lchat-Sidecar, internal/bot/sidecarmark.go) that the activator excludes from the idle clock and refuses to be woken by — otherwise the pair could never sleep. The marker is spoofable-but-harmless: it only downgrades the sender's own requests (no activity credit; 503 while asleep) and the router strips it before proxying. The Node reference sidecar (agent/) does not send the marker, so it isn't suitable for a hibernating instance — use the Go sidecar there. The flip commands hand the unit over automatically: hibernate --on disables lchat-bot@<host> (co-spawn takes over), hibernate --off re-enables it once the instance unit is healthy. One honest gap: member-bot charters run in the sidecar, so a sleeping instance's charter schedules don't fire until something else wakes it — the due-time index covers server-side schedules (future messages, reminders) only.

Migrating a Node sidecar to Go

The session files are byte-compatible (internal/bot/session.go matches the Node session.json shape), so a running Node sidecar swaps to Go in place with no re-provisioning and no lost identity:

1. The shipped lchat-bot@.service template already execs the versioned Go sidecar — ExecStart=/opt/lchat/live/%i bot run — sourcing LCHAT_BASE and the LCHAT_*_SESSION paths (and ANTHROPIC_API_KEY, and LCHAT_CONNECT_BROKER_URL/_TOKEN for connected-services) from /data/%i-bot.env. Nothing per-host to write; add-bot just enables it.
2. systemctl daemon-reload && systemctl restart lchat-bot@<host>. The bot reconnects, re-declares its kind, and backfills — members see no gap.
3. The Go sidecar keeps the Node sidecar's .state.json (announce memory) for any custom changelog-recipe bot, so nothing replays.

Once no unit on the box still ExecStarts Node, the Node runtime and the agent/node_modules install can be removed entirely — the box becomes a pure Go deployment. (add-bot itself no longer needs Node either: since #856 it mints agent invites in Go via internal/e2ee, not agent/mint-invite.mjs.)

Upgrading an existing box to version-tracked bots

Older boxes may carry a hand-written drop-in pinning the bot to a fixed binary — /etc/systemd/system/lchat-bot@<host>.service.d/override.conf with ExecStart=/opt/lchat/lchat bot run … (or an even older canary path). A drop-in ExecStart wins over the template, so those bots keep running one frozen build and never track their instance. Migrate each once:

# drop the pin so the versioned template (live/%i) takes effect, and reinstall
# the current template in case the box predates it
rm -f /etc/systemd/system/lchat-bot@<host>.service.d/override.conf
scp deploy/systemd/lchat-bot@.service root@<box>:/etc/systemd/system/   # or re-run `lchat bootstrap`
systemctl daemon-reload
systemctl restart lchat-bot@<host>

After this the bot execs /opt/lchat/live/<host> and every subsequent deploy/update keeps it in lock-step with the app automatically.

The registry (a small control DB)

The router owns one SQLite table. Status lives here (not a flat file) so an add-in-flight survives a router restart, and it's where per-instance config that used to be env now lives:

CREATE TABLE routes (
  host             TEXT PRIMARY KEY,
  kind             TEXT NOT NULL,   -- 'instance' (messenger, lifecycle-managed) | 'service' (home, push, …)
                                    -- | 'hooks' (an instance's in.<host> webhook-ingest sidecar, add-hooks)
  socket           TEXT NOT NULL,   -- unix socket the router proxies to
  status           TEXT NOT NULL,   -- pending|creating|starting|issuing|ready|failed
  last_error       TEXT,
  -- instance-only fields (null for registered services):
  db_path          TEXT,
  version          TEXT,            -- running binary version (per-instance skew is normal)
  app_store_url    TEXT,            -- was LCHAT_APP_STORE_URL
  home_origin      TEXT,            -- was LCHAT_HOME_ORIGIN
  push_relay_url   TEXT,            -- was LCHAT_PUSH_RELAY_URL
  push_relay_token TEXT,            -- was LCHAT_PUSH_RELAY_TOKEN (per-instance bearer)
  created_at       INTEGER
);

🤖 The hooks kind is how incoming webhooks reach an instance (issue #610, docs/webhooks.md): lchat add-hooks <host> registers in.<host> → /run/lchat/<host>-hooks.sock, where lchat-hooks@<host> (the webhook-ingest sidecar) listens. No router code special-cases it — it's an ordinary route. TLS rides the per-host autocert path (the *.domain wildcard cert covers one label only; wildcard DNS resolves two labels down with no zone change), so an 11ch.at space needs no DNS work; a custom-domain space's owner adds an in.<host> record. lchat remove <host> deletes the in.<host> row (and, with --stop, the hooks unit) along with the instance's.

Per-instance config: a per-instance EnvironmentFile

Config stays env-shaped — the instance reads the same LCHAT_* vars it always has — but instead of process-global env it comes from a per-instance systemd EnvironmentFile, /data/<host>.env (mode 0600), written by add-instance. No instance code changes: EnvironmentFile=-/data/%i.env in the unit, and the existing os.Getenv calls just work.

| Var | Source | |---|---| | LCHAT_ADMIN_TOKEN | /data/<host>.env — the DB stores only its hash, so this file is the one recoverable copy (admin-link reads it) | | VAPID keypair | the instance DB (ensureVAPID) — self-generated, no config needed | | LCHAT_APP_STORE_URL, LCHAT_HOME_ORIGIN, LCHAT_PUSH_RELAY_URL/_TOKEN | /data/<host>.env (from add-instance flags) | | LCHAT_DB / socket path | derived from <host> by the unit | | APNs .p8 | not on instances at all — the push relay holds it (see push-relay.md) |

The routes table tracks routing + status; the EnvironmentFile is the source of per-instance config and the admin token. The push-relay work already did the hardest part — the one secret that can't sit safely in a per-host file (the Apple signer) is out of instances entirely, so each carries only a revocable bearer.

Provisioning the box

Because the server is a static, no-cgo binary, a fresh box needs no build toolchain and no language runtime for the messenger, the router, or the home/push services — provisioning is just "copy binaries + unit files + make directories":

1. scp the lchat binary (plus home/push and litestream) to the box — or run deploy/install.sh, which downloads + verifies the latest stable release and does step 2 for you.
2. lchat bootstrap — writes the systemd units (router + @ templates), creates /data, /run/lchat, and the autocert cache dir, then enables and starts the router and the auto-update timer.
3. Open inbound :80 / :443.

That's the whole dependency story for the Go side — nothing to apt install, no version to keep patched, no runtime to manage. A bare box with no Go on it is not a concern; it's the point.

🤖 The public status collectors ride the binary too. bootstrap also lays down the SSH-access-ledger and fleet-hardening collector scripts, their four systemd units, and the static viewer pages (web/status/*.html + status.css, into /opt/lchat/status-pages/), and enables their 5-minute timers — all embedded, source-of-truth deploy/. This closes the gap the fleet hardening opened: with operator SSH on WireGuard-only and /opt/lchat + /etc/systemd/system root-owned, there's no root shell to scp collectors onto a box, so the binary must place them. To refresh just the collectors + pages on a live box — no root shell, no router restart, no member-facing blip — run sudo -n /opt/lchat/lchat bootstrap --collectors-only (allowlisted by deploy/sudoers-operators's /opt/lchat/lchat *). Only the .html + .css viewer assets are embedded; the collector-generated JSON stays mutable on the box. Details: docs/status.md.

There is no runtime dependency — not even for bots. The sidecar is now the same lchat binary (lchat bot run; see Bots), so a box that runs the server can run its bots with nothing else installed. The Node reference (agent/) remains for self-hosters who want it, but it's optional; a stock box is pure static Go. (A bot can still run off-box — it's just a WebSocket client to the public URL — if you'd rather keep its keys on a different machine.)

add-instance

lchat add-instance <host> runs the steps an operator would otherwise do by hand:

mint token → write /data/<host>.env (0600, the unit's EnvironmentFile)
           → register route (status: creating)
           → systemctl enable --now lchat@<host>  → wait for the socket
           → status: ready → systemctl reload lchat-router → print the admin link

The route's status (creating/ready/failed + last_error) is recorded in the registry and shown by lchat instances. The cert issues lazily on the host's first HTTPS hit (proven fine in practice); pre-warming it (m.GetCertificate(&tls.ClientHelloInfo{ServerName: host})) and a fully async, retrying state machine are refinements the schema already allows. lchat admin-link <host> recovers the admin link from the EnvironmentFile (the DB holds only the token's hash).

Versioned, side-by-side deploys

Per-version deploys are the point of this box, so they're first-class rather than "point a unit at a different file by hand." Two pieces make one instance's version independent of every other's:

• A content store: built binaries live at /opt/lchat/versions/<ver>/lchat, keyed by build id (the git short sha, stamped in at build via -ldflags "-X main.buildVersion=<sha>" — see deploy/build-linux.sh).
• A per-instance pointer: each instance's unit execs /opt/lchat/live/<host>, a symlink into the store. The registry records each host's current + previous version (lchat instances shows it).

  /opt/lchat/live/test.example.com   ─▶ versions/<canary>/lchat   (one instance moved)
  /opt/lchat/live/family.example.com ─▶ versions/<stable>/lchat
  /opt/lchat/live/paulca…          ─▶ versions/<stable>/lchat

Flow (canary → promote):

./deploy/build-linux.sh                       # build linux/amd64, stamped with the git sha
scp lchat-linux-amd64 root@box:/opt/lchat/incoming
ssh box lchat stage --file /opt/lchat/incoming      # register it → prints <ver>
ssh box lchat deploy test.example.com --version <ver> # canary ONE instance
# …verify test…
ssh box lchat deploy --all --version <ver>          # promote the rest
ssh box lchat rollback test.example.com               # or step back to its previous version

stage copies a binary into the store (reading its own build id via lchat version). deploy --version <ver> atomically repoints live/<host>, restarts, and health-checks — see below. deploy <host> with no --version just restarts on the current version (the binary's already in place). rollback repoints to the recorded previous version and restarts. init-versions is the one-time adoption step (stage the running binary, create a live/<host> link for every existing instance) when moving a box onto this scheme.

The restart is a graceful drain + a retrying router (rather than dual-socket blue/green — simpler, and enough since clients auto-reconnect):

1. systemctl restart lchat@<host>. On SIGTERM the instance drains in-flight HTTP (http.Server.Shutdown) and closes WebSockets (so browsers reconnect), then exits; systemd starts the new process, which rebinds the socket.
2. Across the brief gap (old exited, new not yet bound → dials fail ENOENT/ECONNREFUSED) the router retries the dial for a short window, so requests queue a beat instead of failing.
3. For a --version deploy, deploy polls /api/version until the instance reports the expected build (not merely "answers 200") — so a stale socket or a restart that didn't take fails loudly. On failure it auto-rolls-back: repoint the symlink to the previous version, restart, and error out. --all stops on the first failure (already-promoted hosts stay on the new version).

Blast radius is one instance: its WebSocket clients reconnect once; HTTP requests see at most a little added latency. Canary = deploy one instance alone. Verified in practice: 120 requests against an instance during a live deploy all returned 200.

The router itself changes rarely; when it does, its own front listener gets zero-downtime via SO_REUSEPORT / systemd socket activation.

🤖 Auto-updating (release channels)

Instances can keep themselves current without an operator driving deploy — built on the versioned-deploy machinery above plus the signed release feed described in docs/server-releases.md:

• lchat update run (fired hourly by lchat-update.timer, installed + enabled by bootstrap; root, like lchat-control) fetches and Ed25519-verifies the channel manifest, asks each ready instance what its admin chose (GET /api/update-prefs over its unix socket — a pre-feature binary 404s, which reads as the defaults: auto-update on, stable channel), then per instance: downloads + sha256-checks + stages the channel's binary and applies it with the exact deploy --version dance (repoint → restart → health-check → auto-rollback → restart the host's bot sidecar so it tracks the new version too). A version that fails its health check on one host is not tried on the rest that pass ("poisoned" for the run); other per-host failures log and continue, so one unreachable instance never strands the fleet.
• Prefs are per-instance admin settings (update_auto, update_download, update_channel in the instance's own config table, set from the admin consoles). Download-only means "staged, apply manually with lchat deploy <host> --version <ver>".
• Pinning (the canary story). In addition to a channel, an instance can be pinned to one exact version (update_pin) — the updater skips a pinned instance entirely and its build survives GC, so an hourly pass never stomps a deliberately-deployed canary back to the channel head. Pin beats channel. The canary flow is one command: lchat deploy test.example.com --version <ver> --pin (or lchat update pin <host> [<ver>], defaulting to what it runs now). The pin is instance state — written through the instance's own admin API (token from its EnvironmentFile), reported by /api/update-prefs, and shown in its admin panel as a "Pinned to <ver>" entry in the channel picker; the admin resumes updates by picking a channel (one write that sets the channel and clears the pin), or the operator runs lchat update unpin <host>.
• A deploy flock (/opt/lchat/.deploy.lock) serialises deploy, rollback, and update run across processes; the loser errors immediately ("another deploy/rollback/update is running") instead of interleaving the swap dance. A timer pass that loses simply retries next tick.
• Anti-replay: the newest accepted generated_at persists in /opt/lchat/manifest-stamp; older (validly signed but stale) manifests are refused, so the feed can be pulled backward only by re-signing — a deliberate act with the key.
• The bake window (supply-chain hold): beta/stable heads aren't installed until they've sat at the channel head for 24 h/72 h (+ per-box jitter) by the box's own clock (/opt/lchat/channel-first-seen.json) — a compromised release caught and pulled back inside the window never reaches subscribers, and a pull-back to a previously-seen version applies immediately. Local override: lchat update run --ignore-bake (root); design + numbers in docs/server-releases.md § The bake window.
• Major version pinning: lchat update major <host> <n> (or the update_major admin setting) keeps an instance on one major — minors and patches keep flowing, a head whose semver crosses the pin is held (halt-and-notify; a semver-less release also holds, fail closed). Clear with lchat update major <host> clear. See docs/server-releases.md § Major version pinning.
• GC: after a pass, version-store dirs nothing references (no route's current/previous, no channel head, no live symlink, not the control binary's own build) are pruned, and crashed dl-* downloads in incoming/ are swept — hourly edge updates stay disk-bounded.
• lchat update status prints HOST / CHANNEL / AUTO / CURRENT / AVAILABLE across the fleet; lchat update run --dry-run shows what a pass would do.

Supervision (systemd)

• lchat-router.service — Restart=always, socket-activated (its own zero-downtime).
• lchat@.service — a template unit, one per instance. MemoryMax= + CPUWeight= give resource isolation; Restart=always gives crash isolation. Bump LimitNOFILE (instances × WS connections × SQLite files).
• lchat-bot@.service — the optional per-instance sidecar (lchat bot run; see Bots). Restart=always; execs the same live/%i binary as the instance, so a deploy/update restarts it onto the matching version (it backfills on reconnect).

Hardening

Everything on one box means the box is the blast radius, so treat it as the security boundary. The shipped units and bootstrap already apply the service-level items below; the OS-level ones are a few one-time commands.

Run unprivileged. bootstrap creates a locked system user (lchat) and owns /data + /run/lchat to it; every unit runs as that user. Only the router touches privileged ports, so it alone keeps CAP_NET_BIND_SERVICE (AmbientCapabilities + a matching CapabilityBoundingSet) to bind :80/:443 as non-root; everything else binds a unix socket and needs no capabilities (CapabilityBoundingSet=).

Sandbox each unit. The units set the usual systemd confinement: NoNewPrivileges, ProtectSystem=strict with ReadWritePaths=/data /run/lchat (the whole FS is read-only otherwise), ProtectHome, PrivateTmp, PrivateDevices, ProtectKernel*, ProtectControlGroups, ProtectProc=invisible, RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX, RestrictNamespaces, SystemCallFilter=@system-service, and MemoryMax. systemd-analyze security <unit> should score each in the low single digits.

> MemoryDenyWriteExecute note. Every unit — including the Go sidecar — > can set MemoryDenyWriteExecute=yes, since the static Go binary never maps > W+X memory. (The legacy Node sidecar was the one exception — V8's JIT maps > writable+executable pages, so it crashed on boot with that directive; a Go > sidecar has no such caveat and gets the same hardening as the router and > instances.)

🤖 OS level (one-time):

• Firewall — allow only 80, 443, and the WireGuard port (51820/udp) inbound; default-deny the rest. SSH gets no public allow — it is reachable only over the WireGuard interface (next bullet):

``sh ufw allow 80/tcp && ufw allow 443/tcp && ufw allow 51820/udp ufw allow in on wg0 to any port 22 proto tcp ufw enable # add the allows *before* enabling so you don't lock yourself out ``

Instances and services talk over unix sockets, so nothing else needs a public port. sshd itself is left unchanged (still listening on 22, key-only: PasswordAuthentication no) — the firewall does the gating, so there's no boot-order coupling between sshd and the tunnel.

• SSH over WireGuard — operator SSH rides a WireGuard tunnel instead of a public port. WireGuard is silent to unauthenticated packets, so to a scanner the port doesn't exist: the constant background brute-forcing every public 22 attracts drops to zero, and the tunnel key becomes the hard boundary for box access. The box runs one wg0 interface with a peer per operator; each operator's config points at the box and keeps the tunnel warm:

```ini # box: /etc/wireguard/wg0.conf — pick any private subnet for the tunnel [Interface] Address = <wg-subnet>.1/24 ListenPort = 51820 PrivateKey = <box key>

[Peer] # one block per operator PublicKey = <operator pubkey> AllowedIPs = <wg-subnet>.2/32 # that operator's fixed tunnel address ```

```ini # operator machine: the client side of the same tunnel [Interface] Address = <wg-subnet>.2/32 PrivateKey = <operator key>

[Peer] PublicKey = <box pubkey> Endpoint = <box public IP>:51820 AllowedIPs = <wg-subnet>.0/24 # route only the tunnel subnet, not all traffic PersistentKeepalive = 25 # keep NAT mappings alive so ssh works instantly ```

Operator machines run the tunnel persistently (launchd/systemd) and add an ~/.ssh/config alias whose HostName is the box's WireGuard address, so every existing ssh <box> command keeps working unchanged.

• Root SSH is bootstrap-only. A box's first root session runs lchat bootstrap, then the operator-setup stage: create a named account per operator (groups lchat-operators + systemd-journal, the latter so operators can read journals), install the committed sudo allowlist deploy/sudoers-operators at /etc/sudoers.d/lchat-operators (mode 0440) and the env-placement helper deploy/lchat-env-install at /usr/local/sbin/lchat-env-install (mode 0755), add the WireGuard peers — and then disable root logins: lchat bootstrap --ssh-noroot writes the drop-in /etc/ssh/sshd_config.d/10-lchat-noroot.conf (PermitRootLogin no), validates with sshd -t, reloads sshd, and then confirms the effective config (sshd -T) actually reports no — failing loudly when it doesn't. That last check exists because sshdconfig is first-match-wins and a box hardened before this verb existed can carry a drop-in that sorts before 10- (kepler's 00-hardening.conf, found 2026-07-23, silently kept prohibit-password); the fix is renaming the offender to sort after 10- (keeping its other directives, e.g. PasswordAuthentication no, effective) and re-running the verb. The verb refuses until the lchat-operators group has members (a fresh box can't shut root out before anyone else can get in), and every full bootstrap run converges the same drop-in once operators exist — so a box that missed the step picks it up on the next provision pass, via the allowlisted sudo -n /opt/lchat/lchat, with no root shell needed. Thereafter operators are named users over the tunnel; root capability exists only through the allowlist, and anything new that needs root gets a line added via a reviewed PR + redeploy, never a root shell or a hand-edit on the box. Operators who join after the lockdown are added the same no-root-shell way: an existing operator runs ops provision operator <name> <box> (--github <handle> | --keys-file <path>), which drives the allowlisted bootstrap --add-operator <name> with the keys piped on stdin — account + groups + `authorizedkeys (replaced, so re-running is also key rotation). The account is only half of access: the joiner's WireGuard peer must be on the box's wg0 too, or nothing routes. (Encoded after kepler was found carrying an operator's peer but no account, 2026-07-29.) Two small helpers back the allowlist, because a couple of routine privileged ops don't fit a fixed command line — install both alongside the sudoers file (/usr/local/sbin/…, mode 0755): - **deploy/lchat-env-install** places a secret: sops -d file.env | ssh <box> sudo lchat-env-install <name>. - **deploy/lchat-svc** runs systemctl scoped to lchat` units. This indirection is required on Ubuntu 26.04, whose default sudo is sudo-rs (the Rust rewrite): sudo-rs rejects argument wildcards, so a sudoers line like `systemctl restart lchat- is a syntax error — only a whole-argument is allowed. The helper takes the wildcard (lchat-svc ) and enforces the lchat unit prefix itself. Operators run sudo lchat-svc restart lchat-bot@<host>, sudo lchat-svc daemon-reload, etc. (Keep every other command spec in the sudoers file either exact-args or <path> ` for the same reason.) Every operator SSH session is published to the public status ledger. Be honest about what this buys: the allowlist includes deploying binaries, so it is accident-proofing, attribution, and a reviewable ops surface — not a hard security boundary; the hard boundary is the WireGuard key.
• Break-glass — with no public SSH, a lost tunnel key or a broken wg0 is recovered through the hosting provider's out-of-band path: the web console, or rescue mode (boot the rescue system, mount the disk, repair the config). Rare true-root operations (OS surgery, filesystem repair) go the same way — they're what the console is for, not a reason to keep a root login open.
• The deadman pattern — when flipping any remotely-reachable config (firewall rules, sshd drop-ins, the WireGuard interface), schedule the automatic revert first, verify from outside over a fresh connection, and only then cancel the revert — so a mistake un-does itself instead of locking you out:

``sh systemd-run --on-active=10min --unit=sshflip-revert ufw disable # the safety net # …apply the change, verify from a NEW connection from outside… systemctl stop sshflip-revert.timer # all good — disarm ``

• Automatic security updates — enable unattended-upgrades.
• If the box hosts its own services, pin their hostnames to loopback in /etc/hosts (see Bots).

Backups

Collapsing onto one box means every instance's data is on one disk — a single failure loses everything, and there's no platform to restore from.

• Litestream per instance DB → object storage, point-in-time restore.
• Back up the autocert cache dir too — it holds issued certs and their private keys. Treat as key material (0600); losing it forces re-issue.
• Encrypted attachments are stored as blobs inside SQLite (the blobs table), so the DB backup covers them — but they're also what drives disk growth (see sizing).

Sizing

Server-side memory is light by design: crypto and image downscaling happen in the browser; the server only moves ciphertext. A modest box goes a long way.

• RAM: each instance is ~20–40 MB idle (Go runtime + modernc SQLite + embedded assets); the router ~20 MB. A dozen instances is a few hundred MB. Set MemoryMax per instance so one can't OOM the box.
• CPU: no server-side crypto or image work — just HTTP, SQLite, WebSocket fan-out, and TLS termination. Two vCPUs handle a small fleet comfortably.
• Disk is the binding resource, not RAM. Attachments are blobs in SQLite, so DBs grow with media. Watch disk, and let Litestream offload history. If you outgrow anything first, it'll be disk.

Migrating an existing deployment onto the box

Moving an instance from wherever it runs now onto this setup is data movement plus a DNS flip:

1. Snapshot the DB cleanly. VACUUM INTO a fresh copy (folds the WAL into one consistent file); briefly freeze writes for a clean cut.
2. Copy it over to the box as /data/<host>.db.
3. Register it (add-instance <host> against the copied DB), which warms the cert.
4. Flip DNS for the host to the box's IP; verify; retire the old deployment.

Two things make this safe:

• Keep the hostname the same and there's no key work. Browser keys (and passkeys, for the home server) are per-origin, so an unchanged origin leaves members' keys and history untouched — only the DNS record moves.
• A host on a domain not covered by a wildcard cert simply exercises the on-demand HTTP-01 path — no special handling.

Rehearse the whole flow with a throwaway instance first — under a temp host, run the router, add-instance, cert issuance, a socket-swap deploy, a crash-restart, and a Litestream restore, all at zero real-user risk before touching anything real.

What you're trading away (honest costs)

1. The router is a new SPOF for all instances. But it's tiny, holds no per-request state, is Restart=always (sub-second recovery), and is smaller and more stable than a single-process design's whole-app SPOF. Its own deploys are socket-activated.
2. You own backups + supervision now. Litestream + systemd cover what a platform did, but it's your operational surface — it has to actually be set up and monitored.
3. One box. Multi-node failover is exactly where an orchestrator earns its keep; this is deliberately single-box. Revisit if that changes.
4. Migration is a real one-time project — snapshot DBs, flip DNS, retire the old deployment, per instance.

Testing

• Router: host→socket routing, unknown-host → 404, HostPolicy allowlist, atomic upstream swap under concurrent load, WebSocket pass-through.
• add-instance state machine: creating / issuing / failed+retry transitions.
• Instance: unchanged — handler() tests via newTestServer still exercise the real router.
• Deploy: a socket swap drains without dropping an in-flight request (integration).

Deploy tooling (replaces deploy/miren.sh)

• lchat bootstrap — one-time box setup: write systemd units, make dirs, start the router, and install the status collectors + viewer pages + their timers (no toolchain needed — just the copied binary). --collectors-only refreshes just the collectors/pages on a live box (no root shell, no router restart).
• lchat add-instance <host> — create + start + warm cert (async; instances shows status). Stages the running binary + points the host's live link first.
• lchat stage --file <binary> — register a built binary in the version store.
• lchat deploy <host> [--version <ver>] — zero-downtime restart; with --version, repoint the host at that staged version and health-check the build (auto-rollback on mismatch).
• lchat deploy --all [--version <ver>] — roll the fleet, one instance at a time. A --version sweep skips pinned instances (a pin is someone's live canary; the pin used to gate only the auto-updater, and a manual promote could stomp it) — --include-pinned overrides deliberately, and naming a host explicitly always deploys it.
• lchat rollback <host> [--all] — step back to the recorded previous version.
• lchat init-versions — one-time adoption: seed the store + create live links for existing instances.
• lchat version — print the binary's build id (also at /api/version).
• lchat admin-link <host> — recover the admin link from the instance DB (no platform env lookup needed).
• lchat instances — the status table (now with a VERSION column).
• lchat controld / lchat cloud — the local control daemon + the public self-serve provisioner built on it (see cloud-provisioner.md).

Operator specifics — box addresses, DNS/record ids, the concrete migration runbook — live in CLAUDE.local.md, not here.