| 1 | # 🤖 The instance router — a single-machine bootstrap |
| 2 | |
| 3 | Status: **shipped — this page is the design rationale, not a procedure.** The |
| 4 | machinery described here is live (`lchat bootstrap`/`router`/`add-instance`…), |
| 5 | and day-to-day operation of it goes through **`./deploy/ops`** — if something |
| 6 | here reads like steps to run and no ops verb covers it (e.g. hardening a brand |
| 7 | new box), the move is to add the verb, not to follow the prose. |
| 8 | |
| 9 | ## The problem |
| 10 | |
| 11 | The messenger is a single static, no-cgo binary with SQLite compiled in, and it |
| 12 | runs as **one fully-isolated instance per chat** — each with its own disk, admin |
| 13 | token, and subdomain. The obvious way to run several of those on one machine is |
| 14 | to reach for an orchestrator (Kamal, a PaaS, Kubernetes): it gives you TLS, |
| 15 | subdomain routing, a per-app disk, env injection, and process supervision. |
| 16 | |
| 17 | But that's a lot of machinery around a single file you could `scp`. This doc |
| 18 | describes a lighter alternative for the common case — many instances on one |
| 19 | machine — that keeps the two properties process-per-instance gives you and that |
| 20 | are worth keeping: |
| 21 | |
| 22 | - **Per-instance versioning / canary.** Deploy one instance on a new build while |
| 23 | the others stay on the old one. A single shared process can't do that. |
| 24 | - **Isolation.** A crash or a resource hog in one instance must not touch the |
| 25 | others. |
| 26 | |
| 27 | …and adds **zero-downtime deploys**. Those reinforce each other: zero-downtime |
| 28 | *means* "run the new version alongside the old and hand over," so per-instance |
| 29 | versioning falls out of doing it right. |
| 30 | |
| 31 | No container platform required — just the binary, a small router, and systemd. |
| 32 | |
| 33 | ## One binary, many ways to deploy |
| 34 | |
| 35 | The messenger makes no assumptions about how it's run. Put it behind nginx or |
| 36 | Caddy, on a PaaS, in Kubernetes, under Kamal, or one-container-per-instance on |
| 37 | any platform — all of those work, and nothing here precludes them. The instance |
| 38 | binary stands alone. |
| 39 | |
| 40 | **What this doc describes is a batteries-included path for the common case:** |
| 41 | bootstrapping a single machine from zero to a running, auto-TLS'd, multi-instance |
| 42 | box with zero-downtime deploys and no external orchestrator. It's the *simplest* |
| 43 | way, not the *only* way — the router below is optional convenience. |
| 44 | |
| 45 | ## The shape |
| 46 | |
| 47 | ``` |
| 48 | :443 / :80 |
| 49 | internet ──▶ lchat router ──(reverse proxy over unix sockets)──▶ lchat instance (chat-a) ── chat-a.db |
| 50 | (autocert HTTP-01, lchat instance (chat-b) ── chat-b.db |
| 51 | HostPolicy = route table, lchat instance (chat-c) ── chat-c.db |
| 52 | atomic upstream map) … |
| 53 | │ home server (home/) ── home.db ┐ service |
| 54 | │ push relay (push/) ── push.db ┘ routes |
| 55 | └── control API on a localhost-only socket (add / deploy / list) |
| 56 | |
| 57 | systemd supervises everything: router + lchat@<host> (instance) + lchat-bot@<host> (sidecar) + home/push units |
| 58 | ``` |
| 59 | |
| 60 | One binary, three roles (subcommands): |
| 61 | |
| 62 | - `lchat router` — the front door (TLS + routing + control plane). |
| 63 | - `lchat instance` — one messenger instance (today's `main()` minus the TCP |
| 64 | listener and minus the process-global env reads). |
| 65 | - control subcommands — `add-instance`, `deploy`, `instances`, `admin-link`. |
| 66 | |
| 67 | ## The router |
| 68 | |
| 69 | - **`:443`** — TLS via `golang.org/x/crypto/acme/autocert` (stdlib-adjacent, no |
| 70 | DNS-API credentials). `HostPolicy` returns nil only for hosts in the route |
| 71 | table, so the route list *is* the cert allowlist. `Cache: autocert.DirCache` on |
| 72 | the persistent disk. HTTP-01 (well-known) is the challenge; a brand-new host is |
| 73 | issued on first hit — no wildcard, no DNS-provider step. |
| 74 | - **`:80`** — `m.HTTPHandler(nil)`: serves `/.well-known/acme-challenge/…`, |
| 75 | 301-redirects everything else to https. |
| 76 | - **Routing** — the route table maps **host → backend** (a unix socket at a |
| 77 | version); `httputil.ReverseProxy` dials it. The upstream map is an |
| 78 | `atomic.Pointer` so reloads and deploys swap it without dropping a connection. |
| 79 | WebSocket upgrades pass through (Go ≥1.12). A backend is usually one instance — |
| 80 | see *Backends*, below. |
| 81 | - **Unknown hosts** — 404 by default; with `--fallback-url` (env |
| 82 | `LCHAT_FALLBACK_URL`, e.g. the apex marketing site) they 302 there instead, so |
| 83 | under a wildcard DNS record a typo'd or retired subdomain lands somewhere |
| 84 | useful. Scope caveat: on `:80` this always works, but over HTTPS it only helps |
| 85 | hosts that still *have* a cert (e.g. a removed instance) — a never-served host |
| 86 | fails the TLS handshake before any redirect, because `HostPolicy` refuses to |
| 87 | mint certs for unknown hosts (anything else would let strangers burn the ACME |
| 88 | rate limit). Full coverage needs the wildcard-cert option below. |
| 89 | - **Control API** — shipped as `lchat controld` (`lchat-control.service`): a |
| 90 | local-only unix socket (`/run/lchat/control.sock`), gated by a root-only |
| 91 | bearer token, never exposed on a served hostname. This is the god surface (it |
| 92 | creates instances), so it stays off the public interface; its one consumer is |
| 93 | the cloud provisioner (`lchat cloud`), the public, unprivileged policy layer |
| 94 | for self-serve space creation. Full design: `cloud-provisioner.md`. |
| 95 | |
| 96 | Threat model is unchanged: TLS terminates at the router, but message bodies are |
| 97 | already E2E ciphertext — the router sees exactly what the app sees, never |
| 98 | plaintext. |
| 99 | |
| 100 | ### 🤖 Wildcard TLS (DNS-01) — implemented |
| 101 | |
| 102 | HTTP-01 issues one cert per host on first hit — simple, needs no DNS credentials, |
| 103 | but rides the Let's Encrypt per-registered-domain rate limit (~50 new certs/week, |
| 104 | a hard ceiling on space creation) and can't cover a host before it's requested. |
| 105 | So the router can also hold a **wildcard** (`*.example.com`) obtained over ACME |
| 106 | **DNS-01** (`wildcard.go`): set `LCHAT_WILDCARD_DOMAINS=example.com` (comma list) |
| 107 | and a DNS-01 provider (below) in `/data/router.env` and restart. The two coexist |
| 108 | — the wildcard serves any host exactly one label under a configured domain; |
| 109 | everything else (other domains, the apex, deeper levels) takes the per-host |
| 110 | HTTP-01 path, which also keeps covering the wildcard domains until the first |
| 111 | issuance lands, so a DNS-provider/ACME outage degrades to today's behaviour |
| 112 | rather than a dead router. |
| 113 | |
| 114 | **DNS-01 provider (pluggable; `LCHAT_DNS_PROVIDER`).** The solver that publishes |
| 115 | the challenge TXT is a seam (`dnsProvider`), so the wildcard works against more |
| 116 | than one DNS API without changing anything else: |
| 117 | |
| 118 | - **DNSimple — the default.** Leave `LCHAT_DNS_PROVIDER` unset (or set it to |
| 119 | `dnsimple`) and provide `LCHAT_DNSIMPLE_TOKEN`, a DNSimple **account** API |
| 120 | token that can manage the zone. This is the fully-supported path for |
| 121 | self-hosters and production; nothing about it changed when the seam landed. A |
| 122 | missing token leaves the wildcard off (per-host issuance continues). |
| 123 | - **Route53 — the additive off-ramp** (`docs/multi-edge.md`). Set |
| 124 | `LCHAT_DNS_PROVIDER=route53`; the solver authenticates with the **standard AWS |
| 125 | credential chain** (env vars or an EC2/container **IAM role**), so there is *no |
| 126 | DNS token on disk*. The role is scoped to just the wildcard domain's hosted |
| 127 | zone — this is how the dev boxes obtain a shared `*.dev.11ch.at` cert from |
| 128 | their instance role. Route53 is global, so no region is required (the solver |
| 129 | defaults the signing region to `us-east-1`; an explicit `AWS_REGION` wins). |
| 130 | |
| 131 | Mechanics: a renewal loop checks at startup and every 12h (hourly after a |
| 132 | failure), re-issuing when the cert is missing or <30 days from expiry. Issuance |
| 133 | publishes a `_acme-challenge.<domain>` TXT via the selected provider (TTL 60, |
| 134 | always cleaned up) — DNSimple via its records API, Route53 via |
| 135 | `ChangeResourceRecordSets` (UPSERT, then a wait for the change to go INSYNC) — |
| 136 | waits briefly for propagation, and finalizes with a fresh ECDSA key. The cert+key persist as `wildcard-<domain>.pem` (0600) in the acme |
| 137 | cache dir — same backup story as the autocert cache — and load synchronously on |
| 138 | restart; the ACME account key is `wildcard-account.key` alongside. `HostPolicy` |
| 139 | still gates which hosts are actually *served* a backend; the wildcard only |
| 140 | changes how the handshake gets a cert. Deliberate side effect: an unknown |
| 141 | subdomain of a wildcard domain now completes TLS and gets the router's |
| 142 | fallback/404 instead of a handshake error. |
| 143 | |
| 144 | ## The instance process |
| 145 | |
| 146 | `lchat instance --db /data/<host>.db --socket /run/lchat/<host>.sock --config <file>` |
| 147 | |
| 148 | It's today's server with two changes: it listens on a **unix socket** instead of |
| 149 | a TCP port, and it takes per-instance config from flags/a file the router writes |
| 150 | rather than from process-global env (see below). Everything else is unchanged — |
| 151 | its own SQLite (WAL), its own admin token (`ensureAdminToken`, in the DB), its own |
| 152 | VAPID (`ensureVAPID`, in the DB). `handler()` is untouched, so `newTestServer` |
| 153 | and the existing handler tests keep working as-is. |
| 154 | |
| 155 | ## Backends: one per instance (default), one per version (option) |
| 156 | |
| 157 | The router maps **host → backend** (a socket at a version) and doesn't care how |
| 158 | many hosts a backend serves. That leaves two ways to slice processes: |
| 159 | |
| 160 | **One backend per instance (the default).** Each instance is its own process, |
| 161 | socket, and DB — full crash and resource isolation, and per-version deploys come |
| 162 | *for free*: each unit execs a per-instance `live` symlink into a version store, so |
| 163 | moving one instance to a new build is one `lchat deploy <host> --version <ver>` |
| 164 | while the rest stay put (a `trunk` instance next to `stable` ones), no coordination. |
| 165 | Canary is just "which binary does this unit run." See *Versioned, side-by-side |
| 166 | deploys* below. |
| 167 | |
| 168 | **One backend per version (an option, later).** Alternatively a single process |
| 169 | serves *many* hosts at one version (an in-process host map, one DB per host), and |
| 170 | the router sends each host to the process running its assigned version: |
| 171 | |
| 172 | ``` |
| 173 | stable-host ─┐ |
| 174 | other-host ─┼─▶ backend @ stable (one process, many DBs) |
| 175 | another-host ─┘ |
| 176 | trunk-host ──▶ backend @ trunk (one process) |
| 177 | ``` |
| 178 | |
| 179 | This trades isolation for fewer processes — fewer Go runtimes (less RAM), fewer |
| 180 | deploy targets — and the costs are real: a crash or OOM takes down *every* host on |
| 181 | that version, there's no per-host resource cap, and deploying a version restarts |
| 182 | all its hosts together (one big reconnect, not a staggered roll). Moving a host |
| 183 | between versions also needs a clean SQLite hand-off (two processes must never write |
| 184 | one DB file), so it suits hosts that live permanently on a channel (a dedicated |
| 185 | trunk instance) more than ad-hoc per-instance canaries. |
| 186 | |
| 187 | **Recommendation:** default to one-backend-per-instance — isolation is cheap to |
| 188 | keep, per-version is already free, and a dozen processes cost a few hundred MB |
| 189 | (RAM isn't the binding resource; disk is). Keep the route table as host → backend |
| 190 | so consolidating onto per-version backends stays a config change, not a rewrite, |
| 191 | and reach for it only if process count ever actually bites. It needs the |
| 192 | in-process host map, so it's the *second* thing to build, not the first. |
| 193 | |
| 194 | ## Beyond messenger instances: sibling services |
| 195 | |
| 196 | A deployment usually runs sibling services next to the messenger that want the |
| 197 | same TLS + host routing — for example the other two binaries in this repo: |
| 198 | |
| 199 | - **the home server (`home/`)** — the passkey-backup store, its own binary + DB. |
| 200 | Its host *is* the WebAuthn RP origin, so it must keep the same hostname across |
| 201 | any move (only the DNS record changes) or it orphans every passkey. |
| 202 | - **the push relay (`push/`)** — a shared APNs relay that lets instances (yours |
| 203 | *and* third parties') send iOS push without holding Apple's signing key (see |
| 204 | `push-relay.md`). |
| 205 | |
| 206 | To the router these are just another **host → upstream** mapping: a `service`-kind |
| 207 | route to a local unix socket, TLS via autocert, no lifecycle management (they |
| 208 | deploy on their own cadence). That's the only difference from a messenger |
| 209 | instance, which the router *also* create/deploy/version-manages. |
| 210 | |
| 211 | Because a route is nothing but host → upstream, **anyone running their own home |
| 212 | server or push relay is first-class** — point a host at their backend the same |
| 213 | way. This is deliberately the simple version (a flat mapping + a health check); it |
| 214 | can be hardened — readiness gating, retries, per-service resource isolation — |
| 215 | later, if it earns it. |
| 216 | |
| 217 | ## Bots (sidecars) |
| 218 | |
| 219 | An instance can have a **sidecar** hosting the default agents. It connects to its |
| 220 | instance as a **WebSocket client** over the instance's public host — |
| 221 | authenticating with a one-time agent invite and persisting its session to disk. |
| 222 | It serves no inbound traffic, so it never needs a socket or a cert, and the |
| 223 | router never routes *to* it. |
| 224 | |
| 225 | **The sidecar is the same `lchat` binary** — `lchat bot run` (`botcmd.go`, |
| 226 | `internal/bot/`). No separate runtime: the Go port mirrors `crypto.js`'s wire |
| 227 | format byte-for-byte (`internal/e2ee`, pinned by cross-language vectors generated |
| 228 | from `crypto.js` itself), so its ciphertext can't drift from what clients expect, |
| 229 | and the box that runs the server can run its bots with **no extra install**. The |
| 230 | Node reference (`agent/`) stays in the repo as the functional example for |
| 231 | self-hosters who prefer it, but the default deployment is Go. |
| 232 | |
| 233 | **Why still a separate process?** Because a bot is a full E2E client — it holds |
| 234 | its own keypair, and that private key must never live in the server process (the |
| 235 | core rule). So a bot is a *member that runs on a server*, not a server feature — |
| 236 | run wherever the operator wants the keys to live (same box, off-box, a Raspberry |
| 237 | Pi), since it only ever dials out. |
| 238 | |
| 239 | **One process, the whole pair.** `lchat bot run -base https://<host> |
| 240 | <claude>.session.json <eleven>.session.json` runs both |
| 241 | identities as goroutines in one process, each its own connection. The standard |
| 242 | pair is **zero-config**: with no recipe file an identity gets the |
| 243 | conversational default (command floor + model; badges AI when |
| 244 | `ANTHROPIC_API_KEY` is set). A custom bot drops a `<session>.recipe.json` |
| 245 | alongside its session (`docs/bot-recipes.md`) — behaviour as data, no code. (A |
| 246 | legacy session named `Announcer` is skipped with a log line — that identity is |
| 247 | retired; the server announces its own changelog now. See docs/bots.md.) |
| 248 | |
| 249 | Supervise it as **one `lchat-bot@<host>.service` per instance**. The unit execs |
| 250 | the *same* per-instance live symlink as the instance itself |
| 251 | (`ExecStart=/opt/lchat/live/%i bot run`), so the bot runs the **same version as its |
| 252 | app by construction** — and `deploy`/`update` restart the bot right after they |
| 253 | repoint that symlink and health-check the instance (`deployHostVersion → |
| 254 | restartBotSidecar`), so the sidecar's code never lags the server. `bot run` with no |
| 255 | positional args reads `LCHAT_BASE` + the `LCHAT_*_SESSION` paths from the unit's |
| 256 | EnvironmentFile, so the template is fully generic — a full trio or an Eleven-only |
| 257 | box both work with no per-host args. Because the sidecar backfills on reconnect, |
| 258 | that restart is a plain reconnect — no socket-swap, no drain. One wrinkle: it reaches its instance at the |
| 259 | public URL through the router, so the box must resolve that public hostname **to |
| 260 | itself**. Pin every hostname the box hosts to loopback in `/etc/hosts` |
| 261 | (`127.0.0.1 chat.example.com …`) so on-box clients — sidecars, and an instance's |
| 262 | own calls to a sibling service like the push relay — reach the local router |
| 263 | directly (still over TLS+SNI, so the right cert is served) instead of |
| 264 | hair-pinning out to the public IP. This also sidesteps a migration trap: after |
| 265 | you move onto this box and flip DNS, an external resolver may still cache the |
| 266 | *old* box's IP for these hosts, so an on-box client that resolves them externally |
| 267 | can silently talk to the old box — or fail once it's gone. (On cloud images that |
| 268 | regenerate `/etc/hosts` on boot, disable that so the pins persist.) |
| 269 | |
| 270 | Its secrets (an API key, the persisted agent sessions) live in its own env + |
| 271 | disk, the same way the messenger's admin token lives in the instance DB. |
| 272 | |
| 273 | ### Sidecars on hibernating instances: co-spawn |
| 274 | |
| 275 | A **hibernating** instance (activator-managed, `lchat hibernate <host> --on`) |
| 276 | doesn't keep its sidecar on the always-on unit — a persistent WebSocket client |
| 277 | would either hold the sleeping instance awake forever or turn its reconnect |
| 278 | loop into a wake loop. Instead the activator **co-spawns** the sidecar: when a |
| 279 | wake brings the instance up (socket healthy, replication running), it also |
| 280 | execs `live/<host> bot run` under `/data/<host>-bot.env` — the same file the |
| 281 | unit reads; "has a sidecar" *is* that file existing — and at sleep it SIGTERMs |
| 282 | the sidecar *first*, then stops the instance. The sidecar exists exactly while |
| 283 | its instance is awake. If it dies in between, the activator restarts it with |
| 284 | capped exponential backoff (`Restart=always`, cheaply); if it won't start at |
| 285 | all, the instance still serves — members before bots. |
| 286 | |
| 287 | Because the sidecar dials the instance's *public URL through the router* (the |
| 288 | loopback pin above), its traffic marks itself with a header |
| 289 | (`X-Lchat-Sidecar`, `internal/bot/sidecarmark.go`) that the activator excludes |
| 290 | from the idle clock and refuses to be woken by — otherwise the pair could |
| 291 | never sleep. The marker is spoofable-but-harmless: it only downgrades the |
| 292 | sender's own requests (no activity credit; 503 while asleep) and the router |
| 293 | strips it before proxying. The Node reference sidecar (`agent/`) does **not** |
| 294 | send the marker, so it isn't suitable for a hibernating instance — use the Go |
| 295 | sidecar there. The flip commands hand the unit over automatically: |
| 296 | `hibernate --on` disables `lchat-bot@<host>` (co-spawn takes over), |
| 297 | `hibernate --off` re-enables it once the instance unit is healthy. One honest |
| 298 | gap: member-bot charters run in the sidecar, so a sleeping instance's charter |
| 299 | schedules don't fire until something else wakes it — the due-time index covers |
| 300 | server-side schedules (future messages, reminders) only. |
| 301 | |
| 302 | ### Migrating a Node sidecar to Go |
| 303 | |
| 304 | The session files are **byte-compatible** (`internal/bot/session.go` matches the |
| 305 | Node `session.json` shape), so a running Node sidecar swaps to Go in place with |
| 306 | no re-provisioning and no lost identity: |
| 307 | |
| 308 | 1. The shipped `lchat-bot@.service` template already execs the versioned Go |
| 309 | sidecar — `ExecStart=/opt/lchat/live/%i bot run` — sourcing `LCHAT_BASE` and |
| 310 | the `LCHAT_*_SESSION` paths (and `ANTHROPIC_API_KEY`, and |
| 311 | `LCHAT_CONNECT_BROKER_URL`/`_TOKEN` for connected-services) from |
| 312 | `/data/%i-bot.env`. Nothing per-host to write; `add-bot` just enables it. |
| 313 | 2. `systemctl daemon-reload && systemctl restart lchat-bot@<host>`. The bot |
| 314 | reconnects, re-declares its kind, and backfills — members see no gap. |
| 315 | 3. The Go sidecar keeps the Node sidecar's `.state.json` (announce memory) for |
| 316 | any custom changelog-recipe bot, so nothing replays. |
| 317 | |
| 318 | Once **no** unit on the box still `ExecStart`s Node, the Node runtime and the |
| 319 | `agent/node_modules` install can be removed entirely — the box becomes a pure |
| 320 | Go deployment. (`add-bot` itself no longer needs Node either: since #856 it |
| 321 | mints agent invites in Go via `internal/e2ee`, not `agent/mint-invite.mjs`.) |
| 322 | |
| 323 | ### Upgrading an existing box to version-tracked bots |
| 324 | |
| 325 | Older boxes may carry a **hand-written drop-in** pinning the bot to a fixed |
| 326 | binary — `/etc/systemd/system/lchat-bot@<host>.service.d/override.conf` with |
| 327 | `ExecStart=/opt/lchat/lchat bot run …` (or an even older canary path). A drop-in |
| 328 | `ExecStart` **wins over the template**, so those bots keep running one frozen |
| 329 | build and never track their instance. Migrate each once: |
| 330 | |
| 331 | ```sh |
| 332 | # drop the pin so the versioned template (live/%i) takes effect, and reinstall |
| 333 | # the current template in case the box predates it |
| 334 | rm -f /etc/systemd/system/lchat-bot@<host>.service.d/override.conf |
| 335 | scp deploy/systemd/lchat-bot@.service root@<box>:/etc/systemd/system/ # or re-run `lchat bootstrap` |
| 336 | systemctl daemon-reload |
| 337 | systemctl restart lchat-bot@<host> |
| 338 | ``` |
| 339 | |
| 340 | After this the bot execs `/opt/lchat/live/<host>` and every subsequent |
| 341 | `deploy`/`update` keeps it in lock-step with the app automatically. |
| 342 | |
| 343 | ## The registry (a small control DB) |
| 344 | |
| 345 | The router owns one SQLite table. Status lives here (not a flat file) so an |
| 346 | add-in-flight survives a router restart, and it's where per-instance config that |
| 347 | used to be env now lives: |
| 348 | |
| 349 | ```sql |
| 350 | CREATE TABLE routes ( |
| 351 | host TEXT PRIMARY KEY, |
| 352 | kind TEXT NOT NULL, -- 'instance' (messenger, lifecycle-managed) | 'service' (home, push, …) |
| 353 | -- | 'hooks' (an instance's in.<host> webhook-ingest sidecar, add-hooks) |
| 354 | socket TEXT NOT NULL, -- unix socket the router proxies to |
| 355 | status TEXT NOT NULL, -- pending|creating|starting|issuing|ready|failed |
| 356 | last_error TEXT, |
| 357 | -- instance-only fields (null for registered services): |
| 358 | db_path TEXT, |
| 359 | version TEXT, -- running binary version (per-instance skew is normal) |
| 360 | app_store_url TEXT, -- was LCHAT_APP_STORE_URL |
| 361 | home_origin TEXT, -- was LCHAT_HOME_ORIGIN |
| 362 | push_relay_url TEXT, -- was LCHAT_PUSH_RELAY_URL |
| 363 | push_relay_token TEXT, -- was LCHAT_PUSH_RELAY_TOKEN (per-instance bearer) |
| 364 | created_at INTEGER |
| 365 | ); |
| 366 | ``` |
| 367 | |
| 368 | 🤖 The `hooks` kind is how incoming webhooks reach an instance (issue #610, |
| 369 | `docs/webhooks.md`): `lchat add-hooks <host>` registers `in.<host>` → |
| 370 | `/run/lchat/<host>-hooks.sock`, where `lchat-hooks@<host>` (the webhook-ingest |
| 371 | sidecar) listens. No router code special-cases it — it's an ordinary route. |
| 372 | TLS rides the per-host autocert path (the `*.domain` wildcard cert covers one |
| 373 | label only; wildcard DNS resolves two labels down with no zone change), so an |
| 374 | 11ch.at space needs no DNS work; a custom-domain space's owner adds an |
| 375 | `in.<host>` record. `lchat remove <host>` deletes the `in.<host>` row (and, |
| 376 | with `--stop`, the hooks unit) along with the instance's. |
| 377 | |
| 378 | ### Per-instance config: a per-instance EnvironmentFile |
| 379 | |
| 380 | Config stays env-shaped — the instance reads the same `LCHAT_*` vars it always |
| 381 | has — but instead of process-global env it comes from a per-instance systemd |
| 382 | **EnvironmentFile**, `/data/<host>.env` (mode `0600`), written by `add-instance`. |
| 383 | No instance code changes: `EnvironmentFile=-/data/%i.env` in the unit, and the |
| 384 | existing `os.Getenv` calls just work. |
| 385 | |
| 386 | | Var | Source | |
| 387 | |---|---| |
| 388 | | `LCHAT_ADMIN_TOKEN` | `/data/<host>.env` — the DB stores only its *hash*, so this file is the one recoverable copy (`admin-link` reads it) | |
| 389 | | VAPID keypair | the instance DB (`ensureVAPID`) — self-generated, no config needed | |
| 390 | | `LCHAT_APP_STORE_URL`, `LCHAT_HOME_ORIGIN`, `LCHAT_PUSH_RELAY_URL`/`_TOKEN` | `/data/<host>.env` (from `add-instance` flags) | |
| 391 | | `LCHAT_DB` / socket path | derived from `<host>` by the unit | |
| 392 | | APNs `.p8` | **not on instances at all** — the push relay holds it (see `push-relay.md`) | |
| 393 | |
| 394 | The `routes` table tracks routing + status; the EnvironmentFile is the source of |
| 395 | per-instance config and the admin token. The push-relay work already did the |
| 396 | hardest part — the one secret that can't sit safely in a per-host file (the Apple |
| 397 | signer) is out of instances entirely, so each carries only a revocable bearer. |
| 398 | |
| 399 | ## Provisioning the box |
| 400 | |
| 401 | Because the server is a static, no-cgo binary, a fresh box needs **no build |
| 402 | toolchain and no language runtime** for the messenger, the router, or the |
| 403 | home/push services — provisioning is just "copy binaries + unit files + make |
| 404 | directories": |
| 405 | |
| 406 | 1. `scp` the `lchat` binary (plus `home`/`push` and `litestream`) to the box — |
| 407 | or run `deploy/install.sh`, which downloads + verifies the latest stable |
| 408 | release and does step 2 for you. |
| 409 | 2. `lchat bootstrap` — writes the systemd units (router + `@` templates), creates |
| 410 | `/data`, `/run/lchat`, and the autocert cache dir, then enables and starts |
| 411 | the router and the auto-update timer. |
| 412 | 3. Open inbound `:80` / `:443`. |
| 413 | |
| 414 | That's the whole dependency story for the Go side — nothing to `apt install`, no |
| 415 | version to keep patched, no runtime to manage. A bare box with *no* Go on it is |
| 416 | not a concern; it's the point. |
| 417 | |
| 418 | 🤖 **The public status collectors ride the binary too.** `bootstrap` also lays |
| 419 | down the SSH-access-ledger and fleet-hardening collector scripts, their four |
| 420 | systemd units, and the static viewer pages (`web/status/*.html` + `status.css`, |
| 421 | into `/opt/lchat/status-pages/`), and enables their 5-minute timers — all |
| 422 | embedded, source-of-truth `deploy/`. This closes the gap the fleet hardening |
| 423 | opened: with operator SSH on WireGuard-only and `/opt/lchat` + |
| 424 | `/etc/systemd/system` root-owned, there's no root shell to scp collectors onto a |
| 425 | box, so the binary must place them. To refresh just the collectors + pages on a |
| 426 | live box — no root shell, no router restart, no member-facing blip — run |
| 427 | `sudo -n /opt/lchat/lchat bootstrap --collectors-only` (allowlisted by |
| 428 | `deploy/sudoers-operators`'s `/opt/lchat/lchat *`). Only the `.html` + `.css` |
| 429 | viewer assets are embedded; the collector-generated JSON stays mutable on the |
| 430 | box. Details: `docs/status.md`. |
| 431 | |
| 432 | **There is no runtime dependency — not even for bots.** The sidecar is now the |
| 433 | same `lchat` binary (`lchat bot run`; see *Bots*), so a box that runs the server |
| 434 | can run its bots with nothing else installed. The Node reference (`agent/`) |
| 435 | remains for self-hosters who want it, but it's optional; a stock box is pure |
| 436 | static Go. (A bot can still run **off-box** — it's just a WebSocket client to the |
| 437 | public URL — if you'd rather keep its keys on a different machine.) |
| 438 | |
| 439 | ## add-instance |
| 440 | |
| 441 | `lchat add-instance <host>` runs the steps an operator would otherwise do by hand: |
| 442 | |
| 443 | ``` |
| 444 | mint token → write /data/<host>.env (0600, the unit's EnvironmentFile) |
| 445 | → register route (status: creating) |
| 446 | → systemctl enable --now lchat@<host> → wait for the socket |
| 447 | → status: ready → systemctl reload lchat-router → print the admin link |
| 448 | ``` |
| 449 | |
| 450 | The route's `status` (`creating`/`ready`/`failed` + `last_error`) is recorded in |
| 451 | the registry and shown by `lchat instances`. The cert issues lazily on the host's |
| 452 | first HTTPS hit (proven fine in practice); pre-warming it |
| 453 | (`m.GetCertificate(&tls.ClientHelloInfo{ServerName: host})`) and a fully async, |
| 454 | retrying state machine are refinements the schema already allows. |
| 455 | `lchat admin-link <host>` recovers the admin link from the EnvironmentFile (the DB |
| 456 | holds only the token's hash). |
| 457 | |
| 458 | ## Versioned, side-by-side deploys |
| 459 | |
| 460 | Per-version deploys are the point of this box, so they're first-class rather than |
| 461 | "point a unit at a different file by hand." Two pieces make one instance's version |
| 462 | independent of every other's: |
| 463 | |
| 464 | - **A content store**: built binaries live at `/opt/lchat/versions/<ver>/lchat`, |
| 465 | keyed by build id (the git short sha, stamped in at build via |
| 466 | `-ldflags "-X main.buildVersion=<sha>"` — see `deploy/build-linux.sh`). |
| 467 | - **A per-instance pointer**: each instance's unit execs `/opt/lchat/live/<host>`, |
| 468 | a symlink into the store. The registry records each host's current + previous |
| 469 | version (`lchat instances` shows it). |
| 470 | |
| 471 | ``` |
| 472 | /opt/lchat/live/test.example.com ─▶ versions/<canary>/lchat (one instance moved) |
| 473 | /opt/lchat/live/family.example.com ─▶ versions/<stable>/lchat |
| 474 | /opt/lchat/live/paulca… ─▶ versions/<stable>/lchat |
| 475 | ``` |
| 476 | |
| 477 | **Flow (canary → promote):** |
| 478 | |
| 479 | ``` |
| 480 | ./deploy/build-linux.sh # build linux/amd64, stamped with the git sha |
| 481 | scp lchat-linux-amd64 root@box:/opt/lchat/incoming |
| 482 | ssh box lchat stage --file /opt/lchat/incoming # register it → prints <ver> |
| 483 | ssh box lchat deploy test.example.com --version <ver> # canary ONE instance |
| 484 | # …verify test… |
| 485 | ssh box lchat deploy --all --version <ver> # promote the rest |
| 486 | ssh box lchat rollback test.example.com # or step back to its previous version |
| 487 | ``` |
| 488 | |
| 489 | `stage` copies a binary into the store (reading its own build id via `lchat |
| 490 | version`). `deploy --version <ver>` atomically repoints `live/<host>`, restarts, |
| 491 | and health-checks — see below. `deploy <host>` with **no** `--version` just |
| 492 | restarts on the current version (the binary's already in place). `rollback` |
| 493 | repoints to the recorded previous version and restarts. `init-versions` is the |
| 494 | one-time adoption step (stage the running binary, create a `live/<host>` link for |
| 495 | every existing instance) when moving a box onto this scheme. |
| 496 | |
| 497 | **The restart is a graceful drain + a retrying router** (rather than dual-socket |
| 498 | blue/green — simpler, and enough since clients auto-reconnect): |
| 499 | |
| 500 | 1. `systemctl restart lchat@<host>`. On **SIGTERM** the instance drains in-flight |
| 501 | HTTP (`http.Server.Shutdown`) and closes WebSockets (so browsers reconnect), |
| 502 | then exits; systemd starts the new process, which rebinds the socket. |
| 503 | 2. Across the brief gap (old exited, new not yet bound → dials fail |
| 504 | `ENOENT`/`ECONNREFUSED`) the **router retries** the dial for a short window, so |
| 505 | requests queue a beat instead of failing. |
| 506 | 3. For a `--version` deploy, `deploy` polls `/api/version` until the instance |
| 507 | reports the **expected `build`** (not merely "answers 200") — so a stale socket |
| 508 | or a restart that didn't take fails loudly. On failure it **auto-rolls-back**: |
| 509 | repoint the symlink to the previous version, restart, and error out. `--all` |
| 510 | stops on the first failure (already-promoted hosts stay on the new version). |
| 511 | |
| 512 | Blast radius is **one instance**: its WebSocket clients reconnect once; HTTP |
| 513 | requests see at most a little added latency. Canary = deploy one instance alone. |
| 514 | Verified in practice: 120 requests against an instance during a live deploy all |
| 515 | returned 200. |
| 516 | |
| 517 | The router itself changes rarely; when it does, its own front listener gets |
| 518 | zero-downtime via `SO_REUSEPORT` / systemd socket activation. |
| 519 | |
| 520 | ## 🤖 Auto-updating (release channels) |
| 521 | |
| 522 | Instances can keep themselves current without an operator driving `deploy` — |
| 523 | built on the versioned-deploy machinery above plus the signed release feed |
| 524 | described in `docs/server-releases.md`: |
| 525 | |
| 526 | - **`lchat update run`** (fired hourly by `lchat-update.timer`, installed + |
| 527 | enabled by `bootstrap`; root, like `lchat-control`) fetches and |
| 528 | Ed25519-verifies the channel manifest, asks each *ready* instance what its |
| 529 | admin chose (`GET /api/update-prefs` over its unix socket — a pre-feature |
| 530 | binary 404s, which reads as the defaults: auto-update on, stable channel), |
| 531 | then per instance: downloads + sha256-checks + stages the channel's binary |
| 532 | and applies it with the exact `deploy --version` dance (repoint → restart → |
| 533 | health-check → auto-rollback → **restart the host's bot sidecar** so it tracks |
| 534 | the new version too). A version that fails its health check on one |
| 535 | host is not tried on the rest that pass ("poisoned" for the run); other |
| 536 | per-host failures log and continue, so one unreachable instance never |
| 537 | strands the fleet. |
| 538 | - **Prefs are per-instance admin settings** (`update_auto`, `update_download`, |
| 539 | `update_channel` in the instance's own config table, set from the admin |
| 540 | consoles). Download-only means "staged, apply manually with |
| 541 | `lchat deploy <host> --version <ver>`". |
| 542 | - **Pinning (the canary story).** In addition to a channel, an instance can be |
| 543 | pinned to one exact version (`update_pin`) — the updater skips a pinned |
| 544 | instance entirely and its build survives GC, so an hourly pass never stomps |
| 545 | a deliberately-deployed canary back to the channel head. Pin beats channel. |
| 546 | The canary flow is one command: `lchat deploy test.example.com --version |
| 547 | <ver> --pin` (or `lchat update pin <host> [<ver>]`, defaulting to what it |
| 548 | runs now). The pin is *instance state* — written through the instance's own |
| 549 | admin API (token from its EnvironmentFile), reported by `/api/update-prefs`, |
| 550 | and shown in its admin panel as a "Pinned to <ver>" entry in the channel |
| 551 | picker; the admin resumes updates by picking a channel (one write that sets |
| 552 | the channel and clears the pin), or the operator runs `lchat update unpin |
| 553 | <host>`. |
| 554 | - **A deploy flock** (`/opt/lchat/.deploy.lock`) serialises `deploy`, |
| 555 | `rollback`, and `update run` across processes; the loser errors immediately |
| 556 | ("another deploy/rollback/update is running") instead of interleaving the |
| 557 | swap dance. A timer pass that loses simply retries next tick. |
| 558 | - **Anti-replay**: the newest accepted `generated_at` persists in |
| 559 | `/opt/lchat/manifest-stamp`; older (validly signed but stale) manifests are |
| 560 | refused, so the feed can be pulled backward only by *re-signing* — a |
| 561 | deliberate act with the key. |
| 562 | - **The bake window (supply-chain hold)**: beta/stable heads aren't installed |
| 563 | until they've sat at the channel head for 24 h/72 h (+ per-box jitter) by |
| 564 | the box's *own* clock (`/opt/lchat/channel-first-seen.json`) — a compromised |
| 565 | release caught and pulled back inside the window never reaches subscribers, |
| 566 | and a pull-back to a previously-seen version applies immediately. Local |
| 567 | override: `lchat update run --ignore-bake` (root); design + numbers in |
| 568 | `docs/server-releases.md § The bake window`. |
| 569 | - **Major version pinning**: `lchat update major <host> <n>` (or the |
| 570 | `update_major` admin setting) keeps an instance on one major — minors and |
| 571 | patches keep flowing, a head whose `semver` crosses the pin is held |
| 572 | (halt-and-notify; a semver-less release also holds, fail closed). Clear |
| 573 | with `lchat update major <host> clear`. See |
| 574 | `docs/server-releases.md § Major version pinning`. |
| 575 | - **GC**: after a pass, version-store dirs nothing references (no route's |
| 576 | current/previous, no channel head, no live symlink, not the control binary's |
| 577 | own build) are pruned, and crashed `dl-*` downloads in `incoming/` are |
| 578 | swept — hourly edge updates stay disk-bounded. |
| 579 | - **`lchat update status`** prints HOST / CHANNEL / AUTO / CURRENT / AVAILABLE |
| 580 | across the fleet; `lchat update run --dry-run` shows what a pass would do. |
| 581 | |
| 582 | ## Supervision (systemd) |
| 583 | |
| 584 | - `lchat-router.service` — `Restart=always`, socket-activated (its own |
| 585 | zero-downtime). |
| 586 | - `lchat@.service` — a template unit, one per instance. `MemoryMax=` + `CPUWeight=` |
| 587 | give resource isolation; `Restart=always` gives crash isolation. Bump |
| 588 | `LimitNOFILE` (instances × WS connections × SQLite files). |
| 589 | - `lchat-bot@.service` — the optional per-instance sidecar (`lchat bot run`; see |
| 590 | *Bots*). `Restart=always`; execs the same `live/%i` binary as the instance, so a |
| 591 | deploy/update restarts it onto the matching version (it backfills on reconnect). |
| 592 | |
| 593 | ## Hardening |
| 594 | |
| 595 | Everything on one box means the box is the blast radius, so treat it as the |
| 596 | security boundary. The shipped units and `bootstrap` already apply the service-level |
| 597 | items below; the OS-level ones are a few one-time commands. |
| 598 | |
| 599 | **Run unprivileged.** `bootstrap` creates a locked system user (`lchat`) and owns |
| 600 | `/data` + `/run/lchat` to it; every unit runs as that user. Only the router touches |
| 601 | privileged ports, so it alone keeps `CAP_NET_BIND_SERVICE` (`AmbientCapabilities` + |
| 602 | a matching `CapabilityBoundingSet`) to bind `:80`/`:443` as non-root; everything |
| 603 | else binds a unix socket and needs no capabilities (`CapabilityBoundingSet=`). |
| 604 | |
| 605 | **Sandbox each unit.** The units set the usual systemd confinement: |
| 606 | `NoNewPrivileges`, `ProtectSystem=strict` with `ReadWritePaths=/data /run/lchat` |
| 607 | (the whole FS is read-only otherwise), `ProtectHome`, `PrivateTmp`, |
| 608 | `PrivateDevices`, `ProtectKernel*`, `ProtectControlGroups`, `ProtectProc=invisible`, |
| 609 | `RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX`, `RestrictNamespaces`, |
| 610 | `SystemCallFilter=@system-service`, and `MemoryMax`. `systemd-analyze security |
| 611 | <unit>` should score each in the low single digits. |
| 612 | |
| 613 | > **`MemoryDenyWriteExecute` note.** Every unit — including the Go sidecar — |
| 614 | > can set `MemoryDenyWriteExecute=yes`, since the static Go binary never maps |
| 615 | > W+X memory. (The legacy Node sidecar was the one exception — V8's JIT maps |
| 616 | > writable+executable pages, so it crashed on boot with that directive; a Go |
| 617 | > sidecar has no such caveat and gets the same hardening as the router and |
| 618 | > instances.) |
| 619 | |
| 620 | 🤖 **OS level** (one-time): |
| 621 | |
| 622 | - **Firewall** — allow only `80`, `443`, and the WireGuard port (`51820/udp`) |
| 623 | inbound; default-deny the rest. SSH gets **no public allow** — it is reachable |
| 624 | only over the WireGuard interface (next bullet): |
| 625 | |
| 626 | ```sh |
| 627 | ufw allow 80/tcp && ufw allow 443/tcp && ufw allow 51820/udp |
| 628 | ufw allow in on wg0 to any port 22 proto tcp |
| 629 | ufw enable # add the allows *before* enabling so you don't lock yourself out |
| 630 | ``` |
| 631 | |
| 632 | Instances and services talk over unix sockets, so nothing else needs a public |
| 633 | port. `sshd` itself is left unchanged (still listening on 22, key-only: |
| 634 | `PasswordAuthentication no`) — the *firewall* does the gating, so there's no |
| 635 | boot-order coupling between sshd and the tunnel. |
| 636 | - **SSH over WireGuard** — operator SSH rides a WireGuard tunnel instead of a |
| 637 | public port. WireGuard is silent to unauthenticated packets, so to a scanner |
| 638 | the port doesn't exist: the constant background brute-forcing every public 22 |
| 639 | attracts drops to zero, and the tunnel key becomes the hard boundary for box |
| 640 | access. The box runs one `wg0` interface with a peer per operator; each |
| 641 | operator's config points at the box and keeps the tunnel warm: |
| 642 | |
| 643 | ```ini |
| 644 | # box: /etc/wireguard/wg0.conf — pick any private subnet for the tunnel |
| 645 | [Interface] |
| 646 | Address = <wg-subnet>.1/24 |
| 647 | ListenPort = 51820 |
| 648 | PrivateKey = <box key> |
| 649 | |
| 650 | [Peer] # one block per operator |
| 651 | PublicKey = <operator pubkey> |
| 652 | AllowedIPs = <wg-subnet>.2/32 # that operator's fixed tunnel address |
| 653 | ``` |
| 654 | |
| 655 | ```ini |
| 656 | # operator machine: the client side of the same tunnel |
| 657 | [Interface] |
| 658 | Address = <wg-subnet>.2/32 |
| 659 | PrivateKey = <operator key> |
| 660 | |
| 661 | [Peer] |
| 662 | PublicKey = <box pubkey> |
| 663 | Endpoint = <box public IP>:51820 |
| 664 | AllowedIPs = <wg-subnet>.0/24 # route only the tunnel subnet, not all traffic |
| 665 | PersistentKeepalive = 25 # keep NAT mappings alive so ssh works instantly |
| 666 | ``` |
| 667 | |
| 668 | Operator machines run the tunnel persistently (launchd/systemd) and add an |
| 669 | `~/.ssh/config` alias whose `HostName` is the box's WireGuard address, so |
| 670 | every existing `ssh <box>` command keeps working unchanged. |
| 671 | - **Root SSH is bootstrap-only.** A box's *first* root session runs |
| 672 | `lchat bootstrap`, then the operator-setup stage: create a named account per |
| 673 | operator (groups `lchat-operators` + `systemd-journal`, the latter so |
| 674 | operators can read journals), install the committed sudo allowlist |
| 675 | `deploy/sudoers-operators` at `/etc/sudoers.d/lchat-operators` (mode `0440`) |
| 676 | and the env-placement helper `deploy/lchat-env-install` at |
| 677 | `/usr/local/sbin/lchat-env-install` (mode `0755`), add the WireGuard peers — |
| 678 | and then disable root logins: `lchat bootstrap --ssh-noroot` writes the |
| 679 | drop-in `/etc/ssh/sshd_config.d/10-lchat-noroot.conf` (`PermitRootLogin |
| 680 | no`), validates with `sshd -t`, reloads sshd, and then **confirms the |
| 681 | effective config** (`sshd -T`) actually reports `no` — failing loudly when |
| 682 | it doesn't. That last check exists because sshd_config is first-match-wins |
| 683 | and a box hardened before this verb existed can carry a drop-in that sorts |
| 684 | *before* `10-` (kepler's `00-hardening.conf`, found 2026-07-23, silently |
| 685 | kept `prohibit-password`); the fix is renaming the offender to sort after |
| 686 | `10-` (keeping its other directives, e.g. `PasswordAuthentication no`, |
| 687 | effective) and re-running the verb. |
| 688 | The verb refuses until the `lchat-operators` group has members (a fresh box |
| 689 | can't shut root out before anyone else can get in), and every full |
| 690 | `bootstrap` run converges the same drop-in once operators exist — so a box |
| 691 | that missed the step picks it up on the next provision pass, via the |
| 692 | allowlisted `sudo -n /opt/lchat/lchat`, with no root shell needed. |
| 693 | Thereafter operators are named users over the tunnel; root capability exists |
| 694 | only through the allowlist, and anything new that needs root gets a line |
| 695 | added **via a reviewed PR + redeploy**, never a root shell or a hand-edit on |
| 696 | the box. |
| 697 | **Operators who join after the lockdown** are added the same no-root-shell |
| 698 | way: an existing operator runs `ops provision operator <name> <box> |
| 699 | (--github <handle> | --keys-file <path>)`, which drives the allowlisted |
| 700 | `bootstrap --add-operator <name>` with the keys piped on stdin — account + |
| 701 | groups + `authorized_keys` (replaced, so re-running is also key rotation). |
| 702 | The account is only half of access: the joiner's WireGuard peer must be on |
| 703 | the box's `wg0` too, or nothing routes. (Encoded after kepler was found |
| 704 | carrying an operator's peer but no account, 2026-07-29.) Two small helpers back the allowlist, because a couple of routine |
| 705 | privileged ops don't fit a fixed command line — install both alongside the |
| 706 | sudoers file (`/usr/local/sbin/…`, mode `0755`): |
| 707 | - **`deploy/lchat-env-install`** places a secret: |
| 708 | `sops -d file.env | ssh <box> sudo lchat-env-install <name>`. |
| 709 | - **`deploy/lchat-svc`** runs `systemctl` scoped to `lchat*` units. This |
| 710 | indirection is *required* on Ubuntu 26.04, whose default sudo is |
| 711 | **sudo-rs** (the Rust rewrite): sudo-rs rejects argument wildcards, so a |
| 712 | sudoers line like `systemctl restart lchat-*` is a syntax error — only a |
| 713 | whole-argument `*` is allowed. The helper takes the wildcard (`lchat-svc |
| 714 | *`) and enforces the `lchat*` unit prefix itself. Operators run |
| 715 | `sudo lchat-svc restart lchat-bot@<host>`, `sudo lchat-svc daemon-reload`, |
| 716 | etc. (Keep every other command spec in the sudoers file either |
| 717 | exact-args or `<path> *` for the same reason.) |
| 718 | Every operator SSH session is published to the public status ledger. Be honest about what |
| 719 | this buys: the allowlist includes deploying binaries, so it is |
| 720 | accident-proofing, attribution, and a reviewable ops surface — **not** a hard |
| 721 | security boundary; the hard boundary is the WireGuard key. |
| 722 | - **Break-glass** — with no public SSH, a lost tunnel key or a broken `wg0` is |
| 723 | recovered through the hosting provider's out-of-band path: the web console, |
| 724 | or rescue mode (boot the rescue system, mount the disk, repair the config). |
| 725 | Rare true-root operations (OS surgery, filesystem repair) go the same way — |
| 726 | they're what the console is for, not a reason to keep a root login open. |
| 727 | - **The deadman pattern** — when flipping any remotely-reachable config |
| 728 | (firewall rules, sshd drop-ins, the WireGuard interface), schedule the |
| 729 | automatic revert *first*, verify from outside over a *fresh* connection, and |
| 730 | only then cancel the revert — so a mistake un-does itself instead of locking |
| 731 | you out: |
| 732 | |
| 733 | ```sh |
| 734 | systemd-run --on-active=10min --unit=sshflip-revert ufw disable # the safety net |
| 735 | # …apply the change, verify from a NEW connection from outside… |
| 736 | systemctl stop sshflip-revert.timer # all good — disarm |
| 737 | ``` |
| 738 | - **Automatic security updates** — enable `unattended-upgrades`. |
| 739 | - If the box hosts its own services, pin their hostnames to loopback in |
| 740 | `/etc/hosts` (see *Bots*). |
| 741 | |
| 742 | ## Backups |
| 743 | |
| 744 | Collapsing onto one box means every instance's data is on one disk — a single |
| 745 | failure loses everything, and there's no platform to restore from. |
| 746 | |
| 747 | - **Litestream per instance DB** → object storage, point-in-time restore. |
| 748 | - **Back up the autocert cache dir** too — it holds issued certs *and their |
| 749 | private keys*. Treat as key material (`0600`); losing it forces re-issue. |
| 750 | - Encrypted attachments are stored as blobs **inside** SQLite (the `blobs` table), |
| 751 | so the DB backup covers them — but they're also what drives disk growth (see |
| 752 | sizing). |
| 753 | |
| 754 | ## Sizing |
| 755 | |
| 756 | Server-side memory is light **by design**: crypto and image downscaling happen in |
| 757 | the browser; the server only moves ciphertext. A modest box goes a long way. |
| 758 | |
| 759 | - **RAM:** each instance is ~20–40 MB idle (Go runtime + modernc SQLite + embedded |
| 760 | assets); the router ~20 MB. A dozen instances is a few hundred MB. Set |
| 761 | `MemoryMax` per instance so one can't OOM the box. |
| 762 | - **CPU:** no server-side crypto or image work — just HTTP, SQLite, WebSocket |
| 763 | fan-out, and TLS termination. Two vCPUs handle a small fleet comfortably. |
| 764 | - **Disk is the binding resource, not RAM.** Attachments are blobs in SQLite, so |
| 765 | DBs grow with media. Watch disk, and let Litestream offload history. If you |
| 766 | outgrow anything first, it'll be disk. |
| 767 | |
| 768 | ## Migrating an existing deployment onto the box |
| 769 | |
| 770 | Moving an instance from wherever it runs now onto this setup is data movement plus |
| 771 | a DNS flip: |
| 772 | |
| 773 | 1. **Snapshot the DB cleanly.** `VACUUM INTO` a fresh copy (folds the WAL into one |
| 774 | consistent file); briefly freeze writes for a clean cut. |
| 775 | 2. **Copy it over** to the box as `/data/<host>.db`. |
| 776 | 3. **Register it** (`add-instance <host>` against the copied DB), which warms the |
| 777 | cert. |
| 778 | 4. **Flip DNS** for the host to the box's IP; verify; retire the old deployment. |
| 779 | |
| 780 | Two things make this safe: |
| 781 | |
| 782 | - **Keep the hostname the same and there's no key work.** Browser keys (and |
| 783 | passkeys, for the home server) are per-origin, so an unchanged origin leaves |
| 784 | members' keys and history untouched — only the DNS record moves. |
| 785 | - **A host on a domain not covered by a wildcard cert** simply exercises the |
| 786 | on-demand HTTP-01 path — no special handling. |
| 787 | |
| 788 | **Rehearse the whole flow with a throwaway instance first** — under a temp host, |
| 789 | run the router, `add-instance`, cert issuance, a socket-swap deploy, a |
| 790 | crash-restart, and a Litestream restore, all at zero real-user risk before |
| 791 | touching anything real. |
| 792 | |
| 793 | ## What you're trading away (honest costs) |
| 794 | |
| 795 | 1. **The router is a new SPOF** for all instances. But it's tiny, holds no |
| 796 | per-request state, is `Restart=always` (sub-second recovery), and is smaller |
| 797 | and more stable than a single-process design's whole-app SPOF. Its own deploys |
| 798 | are socket-activated. |
| 799 | 2. **You own backups + supervision now.** Litestream + systemd cover what a |
| 800 | platform did, but it's your operational surface — it has to actually be set up |
| 801 | and monitored. |
| 802 | 3. **One box.** Multi-node failover is exactly where an orchestrator earns its |
| 803 | keep; this is deliberately single-box. Revisit if that changes. |
| 804 | 4. **Migration is a real one-time project** — snapshot DBs, flip DNS, retire the |
| 805 | old deployment, per instance. |
| 806 | |
| 807 | ## Testing |
| 808 | |
| 809 | - Router: host→socket routing, unknown-host → 404, `HostPolicy` allowlist, atomic |
| 810 | upstream swap under concurrent load, WebSocket pass-through. |
| 811 | - `add-instance` state machine: creating / issuing / failed+retry transitions. |
| 812 | - Instance: unchanged — `handler()` tests via `newTestServer` still exercise the |
| 813 | real router. |
| 814 | - Deploy: a socket swap drains without dropping an in-flight request (integration). |
| 815 | |
| 816 | ## Deploy tooling (replaces deploy/miren.sh) |
| 817 | |
| 818 | - `lchat bootstrap` — one-time box setup: write systemd units, make dirs, start |
| 819 | the router, and install the status collectors + viewer pages + their timers |
| 820 | (no toolchain needed — just the copied binary). `--collectors-only` refreshes |
| 821 | just the collectors/pages on a live box (no root shell, no router restart). |
| 822 | - `lchat add-instance <host>` — create + start + warm cert (async; `instances` |
| 823 | shows status). Stages the running binary + points the host's `live` link first. |
| 824 | - `lchat stage --file <binary>` — register a built binary in the version store. |
| 825 | - `lchat deploy <host> [--version <ver>]` — zero-downtime restart; with `--version`, |
| 826 | repoint the host at that staged version and health-check the `build` (auto-rollback |
| 827 | on mismatch). |
| 828 | - `lchat deploy --all [--version <ver>]` — roll the fleet, one instance at a |
| 829 | time. A `--version` sweep **skips pinned instances** (a pin is someone's live |
| 830 | canary; the pin used to gate only the auto-updater, and a manual promote |
| 831 | could stomp it) — `--include-pinned` overrides deliberately, and naming a |
| 832 | host explicitly always deploys it. |
| 833 | - `lchat rollback <host> [--all]` — step back to the recorded previous version. |
| 834 | - `lchat init-versions` — one-time adoption: seed the store + create `live` links |
| 835 | for existing instances. |
| 836 | - `lchat version` — print the binary's build id (also at `/api/version`). |
| 837 | - `lchat admin-link <host>` — recover the admin link from the instance DB (no |
| 838 | platform env lookup needed). |
| 839 | - `lchat instances` — the status table (now with a VERSION column). |
| 840 | - `lchat controld` / `lchat cloud` — the local control daemon + the public |
| 841 | self-serve provisioner built on it (see `cloud-provisioner.md`). |
| 842 | |
| 843 | Operator specifics — box addresses, DNS/record ids, the concrete migration |
| 844 | runbook — live in `CLAUDE.local.md`, not here. |
| 845 | |
amadan