amadan
docs: deploying-instances.md — the operator's guide to running instances on a router box (#275)
A comprehensive, environment-free guide for operators standing up their own instances on an lchat-router box: add-instance (all flags), bots, the versioned canary → promote → rollback deploy flow, throwaway canary instances pinned to branch builds, data-safe removal vs full teardown, sibling services, and troubleshooting. Commands verified against control.go/versions.go/bot.go.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
1 file changed,
+327
−0
docs/deploying-instances.md+327 −0
diff --git a/docs/deploying-instances.md b/docs/deploying-instances.md| new file mode 100644 |
| index 0000000..d2f4569 |
| --- /dev/null |
| +++ b/docs/deploying-instances.md |
| @@ -0,0 +1,327 @@ |
| +# 🤖 Deploying instances — the operator's guide |
| + |
| +How to run messenger instances on a box that uses the `lchat` router |
| +(`docs/instance-router.md` is the design; `deploy/README.md` is the one-page |
| +fresh-box bootstrap). This doc is the day-to-day operator reference: adding |
| +instances, giving them bots, shipping new builds safely (canary → promote → |
| +rollback), standing up throwaway canary instances for branch builds, and tearing |
| +things down. Everything here is generic — hosts are `chat.example.com`, the box |
| +is `<box>`; your own hostnames, IPs, and secrets live wherever you keep operator |
| +notes, not in this repo. |
| + |
| +## The mental model |
| + |
| +One static Go binary (`lchat`) plays three roles on the box: |
| + |
| +- **The router** (`lchat-router.service`) — binds `:80`/`:443`, terminates TLS |
| + (Let's Encrypt via autocert HTTP-01), and reverse-proxies each hostname to a |
| + unix socket. The route table lives in a small SQLite registry |
| + (`/data/registry.db`). |
| +- **Instances** (`lchat@<host>.service`, one per chat) — each is its own process, |
| + socket (`/run/lchat/<host>.sock`), and database (`/data/<host>.db`). Fully |
| + isolated: its own admin token, its own crash/memory blast radius, its own |
| + binary version. |
| +- **The control CLI** (`/opt/lchat/lchat add-instance | deploy | instances | …`) |
| + — run on the box over SSH; it edits the registry, writes env files, and drives |
| + systemd. |
| + |
| +Two directories carry the versioning scheme: |
| + |
| +``` |
| +/opt/lchat/versions/<ver>/lchat # the content store: every staged build, keyed by build id |
| +/opt/lchat/live/<host> # per-instance symlink into the store; the unit execs this |
| +``` |
| + |
| +Each instance's unit execs its *own* `live/<host>` symlink, so two instances can |
| +run two different builds at once. That's what makes canarying a one-liner. |
| + |
| +Per-instance config is an **EnvironmentFile** at `/data/<host>.env` (mode 0600), |
| +written by `add-instance`. It holds the one secret that matters — the plaintext |
| +`LCHAT_ADMIN_TOKEN` (the DB stores only its hash, so this file is the only |
| +recoverable copy) — plus optional per-instance vars (`LCHAT_PUSH_RELAY_URL`/ |
| +`_TOKEN`, `LCHAT_HOME_ORIGIN`, `LCHAT_APP_STORE_URL`). |
| + |
| +Nothing here weakens the E2E model: the router and the instance only ever see |
| +ciphertext, and the admin token authorises roster management, not decryption. |
| + |
| +## Prerequisites |
| + |
| +- **A bootstrapped box.** `scp` the binary to `/opt/lchat/lchat`, then |
| + `lchat bootstrap` (idempotent) — it creates the `lchat` service user, the |
| + dirs, and the systemd units, and starts the router. See `deploy/README.md`. |
| +- **DNS** for the new host pointing at the box — either a wildcard |
| + (`*.example.com A <box-ip>`) so new instances need no DNS step, or one A |
| + record per host. Low TTL (60s) makes future moves painless. |
| +- **Inbound `:80` and `:443` open** (both firewall on the box and any cloud |
| + firewall). Port 80 is required — autocert's HTTP-01 challenge answers there. |
| +- Certificates issue **lazily on the host's first HTTPS hit**, in a second or |
| + two. There's no pre-issuance step; if DNS resolves and :80 is reachable, the |
| + first `curl https://<host>/` just works. |
| + |
| +## Adding an instance |
| + |
| +One command, run on the box: |
| + |
| +```sh |
| +ssh root@<box> /opt/lchat/lchat add-instance chat.example.com |
| +``` |
| + |
| +It mints an admin token, writes `/data/chat.example.com.env`, registers the |
| +route, stages the current binary and points `live/<host>` at it, starts |
| +`lchat@chat.example.com`, waits for the socket to answer, reloads the router, |
| +and prints the admin link: |
| + |
| +``` |
| +added chat.example.com |
| + admin: https://chat.example.com/admin#token=… |
| + (recover later: lchat admin-link chat.example.com) |
| +``` |
| + |
| +Optional flags (all become vars in the instance's env file): |
| + |
| +| flag | what it sets | |
| +|---|---| |
| +| `--admin-token <tok>` | import an existing token instead of minting — for migrations, so saved admin links keep working (see `docs/instance-migration.md`) | |
| +| `--push-relay-url` / `--push-relay-token` | iOS push via a shared APNs relay (`docs/push-relay.md`); mint the token on the relay first | |
| +| `--home-origin` | the passkey-backup home server this instance offers members | |
| +| `--app-store-url` | where the welcome page's "Get the app" button points | |
| + |
| +All of these are optional — an instance without them is fully functional for |
| +web + Web Push (VAPID is self-generated into the instance DB; no config). |
| + |
| +Then, immediately: |
| + |
| +- **Store the admin token somewhere durable** (a password manager). It exists |
| + in exactly one recoverable place — `/data/<host>.env` on the box — and |
| + `lchat admin-link <host>` reads it from there, so losing the box means losing |
| + the link. |
| +- **Verify:** |
| + |
| +```sh |
| +ssh root@<box> /opt/lchat/lchat instances # HOST / KIND / STATUS / VERSION / SOCKET |
| +curl -s https://chat.example.com/api/version # {"build":"<ver>"} over a fresh LE cert |
| +``` |
| + |
| +Open the printed admin link in a browser to name the group and mint member |
| +invites. |
| + |
| +If `add-instance` fails partway, the route is left in status `failed` with a |
| +`last error` line shown by `lchat instances`; fix the cause and re-run — a |
| +failed/creating route is overwritten on retry. |
| + |
| +## Adding bots (optional) |
| + |
| +Each instance can have a **sidecar** hosting the three default agents (Claude / |
| +Eleven / Announcer). The sidecar is Node (it reuses `web/static/crypto.js`, so |
| +its E2E crypto can't drift from the browser's), so the box needs a one-time Node |
| +setup — see `deploy/README.md § Bots` for the `apt`/`rsync`/`npm ci` recipe. |
| +Then, per instance: |
| + |
| +```sh |
| +ssh root@<box> 'ANTHROPIC_API_KEY=sk-ant-… /opt/lchat/lchat add-bot chat.example.com' |
| +``` |
| + |
| +`add-bot` mints one agent invite per identity via the instance's admin API, |
| +writes `/data/<host>-bot.env` (0600 — invites, session paths, the API key), and |
| +starts `lchat-bot@<host>`. Notes: |
| + |
| +- **Instance first, then sidecar** — Eleven baselines "who already exists" from |
| + the roster on its first tick; booting the sidecar against a not-yet-ready |
| + instance can make it welcome-DM long-standing members later. |
| +- Omit `ANTHROPIC_API_KEY` for deterministic-only bots (Eleven still welcomes |
| + members and answers slash commands; no AI replies). Add the key to |
| + `/data/<host>-bot.env` and `systemctl restart lchat-bot@<host>` later to give |
| + it a brain. |
| +- The API key is read from the **environment**, deliberately not a flag — a |
| + flag would leak it into the process table and shell history. |
| +- The sidecar connects to its instance over the **public URL**. If the sidecar |
| + runs on the same box as the router, pin every hosted hostname to loopback in |
| + `/etc/hosts` (`127.0.0.1 chat.example.com …`) so on-box clients reach the |
| + local router instead of depending on external DNS — and disable cloud-init's |
| + `manage_etc_hosts` so the pins survive reboots. See |
| + `docs/instance-router.md § Bots`. |
| +- Sidecar logs: `journalctl -u lchat-bot@chat.example.com -f`. |
| +- Updating sidecar *code* is separate from binary deploys: re-`rsync` `agent/` + |
| + `web/static/` to `/opt/lchat/` and restart the `lchat-bot@` units (they |
| + backfill on reconnect, so a restart is safe). |
| + |
| +## Shipping a new build (canary → promote) |
| + |
| +Deploys are **data-safe by construction**: they never touch `/data` — DB |
| +migrations are additive (`CREATE TABLE IF NOT EXISTS`), so new code over old |
| +data is always fine. The flow is: build locally, stage on the box, prove the |
| +build on one instance, then promote the fleet. |
| + |
| +```sh |
| +# 1. Build linux/amd64, stamped with the git short sha as its build id |
| +./deploy/build-linux.sh # → lchat-linux-amd64, prints <ver> |
| + |
| +# 2. Copy it up and register it in the version store |
| +scp lchat-linux-amd64 root@<box>:/opt/lchat/incoming |
| +ssh root@<box> /opt/lchat/lchat stage --file /opt/lchat/incoming # → prints <ver> |
| + |
| +# 3. Canary ONE instance |
| +ssh root@<box> /opt/lchat/lchat deploy canary.example.com --version <ver> |
| +# …verify: open the app, send a message, check /api/version reports <ver>… |
| + |
| +# 4. Promote everyone (one at a time, health-checked, stops on first failure) |
| +ssh root@<box> /opt/lchat/lchat deploy --all --version <ver> |
| + |
| +# 5. Who runs what |
| +ssh root@<box> /opt/lchat/lchat instances # VERSION column shows per-instance skew |
| +``` |
| + |
| +What `deploy --version` actually does: atomically repoint |
| +`/opt/lchat/live/<host>` at the staged build, `systemctl restart` the instance |
| +(SIGTERM drains in-flight HTTP and closes WebSockets so clients reconnect; the |
| +router retries dials across the sub-second gap, so requests queue rather than |
| +fail), then poll `/api/version` until the instance reports the **expected build |
| +id** — not merely a 200. On mismatch it **auto-rolls-back** to the previous |
| +version and errors out. `deploy --all` walks the fleet one instance at a time |
| +and stops at the first failure, leaving already-promoted instances on the new |
| +build. |
| + |
| +Stepping back is explicit too: |
| + |
| +```sh |
| +ssh root@<box> /opt/lchat/lchat rollback chat.example.com # → its recorded previous version |
| +``` |
| + |
| +Other notes: |
| + |
| +- `deploy <host>` with **no** `--version` just gracefully restarts the instance |
| + on its current build (useful after editing its env file). |
| +- Build ids are the git short sha by default; `./deploy/build-linux.sh |
| + my-label` stamps a custom id (handy for branch builds, below). `lchat stage` |
| + reads the id out of the binary itself, so the store can't mislabel a build. |
| +- The build script targets linux/amd64; on an arm64 box set |
| + `GOARCH=arm64` (edit or inline-override the script). |
| +- **⚠️ `deploy --all` moves *every* instance**, including one you've pinned to |
| + a branch build. Before promoting a fleet, check the VERSION column of |
| + `lchat instances` — if a canary instance is meant to stay on its branch |
| + build, deploy the others individually instead of using `--all`. |
| +- To also refresh the **control/router** binary (rarely needed — instance |
| + builds don't require it): atomically replace `/opt/lchat/lchat` (`cp` to a |
| + temp name + `mv` over it) and `systemctl restart lchat-router`. |
| + |
| +### Verifying a canary |
| + |
| +"Verify" should be more than a 200. A reasonable checklist for a few minutes on |
| +the canary instance before promoting: |
| + |
| +- `curl -s https://<host>/api/version` reports the new build id. |
| +- Open the app in a browser: history decrypts, sending a message round-trips, |
| + and a WebSocket reconnect happened cleanly (no stuck "connecting"). |
| +- If the change touched uploads/push/bots, exercise that path specifically. |
| +- `journalctl -u lchat@<host> -e` shows no new errors. |
| + |
| +## Standing up a canary *instance* (branch builds) |
| + |
| +Sometimes the thing to prove isn't a build of `main` on a real instance, but an |
| +**unmerged branch** on a **throwaway instance** — real TLS, real devices, zero |
| +real users, and the fleet (and the control binary) untouched. Because versions |
| +are per-instance, this needs no special machinery: |
| + |
| +```sh |
| +# on your machine, on the feature branch: |
| +./deploy/build-linux.sh # id = the branch's git short sha |
| +scp lchat-linux-amd64 root@<box>:/opt/lchat/incoming |
| + |
| +# on the box: |
| +/opt/lchat/lchat stage --file /opt/lchat/incoming # → <branch-ver> |
| +/opt/lchat/lchat add-instance canary-feature.example.com # starts on the CURRENT fleet build |
| +/opt/lchat/lchat deploy canary-feature.example.com --version <branch-ver> # pin it to the branch |
| +``` |
| + |
| +The branch binary exists only in the version store; nothing else runs it. |
| +`lchat instances` shows the skew explicitly. When the feature merges and ships: |
| + |
| +```sh |
| +/opt/lchat/lchat deploy canary-feature.example.com --version <new-main-ver> # rejoin the fleet |
| +# …or tear the whole instance down (below) if it was a throwaway. |
| +``` |
| + |
| +Two cautions: |
| + |
| +- Data written by a branch build persists in `/data/<host>.db`. Migrations are |
| + additive so *moving forward* is safe, but a branch that created tables/rows a |
| + later design abandons leaves them behind. For feature trials, prefer a |
| + **throwaway** instance you'll delete over pinning a real one. |
| +- Remember the `deploy --all` warning above — a fleet promotion will stomp the |
| + pin. |
| + |
| +## Removing an instance |
| + |
| +`remove` is deliberately data-safe by default — it only stops *routing*: |
| + |
| +```sh |
| +/opt/lchat/lchat remove chat.example.com # route deleted; instance still running, DB/env kept |
| +/opt/lchat/lchat remove chat.example.com --stop # also stop+disable lchat@<host> and lchat-bot@<host> |
| +``` |
| + |
| +Even with `--stop`, the DB and env file stay on disk, so the instance can be |
| +resurrected: re-run `add-instance` passing `--admin-token` with the token still |
| +in `/data/<host>.env` (a plain `add-instance` would mint a fresh token and |
| +invalidate saved admin links). **Full teardown** — only for an instance you truly mean to destroy — is the |
| +manual part, on the box: |
| + |
| +```sh |
| +rm /data/chat.example.com.db /data/chat.example.com.db-wal /data/chat.example.com.db-shm # the DB |
| +rm /data/chat.example.com.env # the admin token |
| +rm -rf /data/chat.example.com-blobs # disk-backed attachments (if present) |
| +rm -rf /data/chat.example.com-bot /data/chat.example.com-bot.env # sidecar sessions + env (if it had bots) |
| +rm /opt/lchat/live/chat.example.com # the version pointer |
| +``` |
| + |
| +…then delete the host's DNS record (if it wasn't wildcard-covered) and the |
| +stored admin token. There is no undo after the DB is gone. |
| + |
| +## Sibling services (home server, push relay, third-party backends) |
| + |
| +The router can front processes it doesn't manage — anything already listening |
| +on a local unix socket: |
| + |
| +```sh |
| +/opt/lchat/lchat add-service home.example.com --socket /run/lchat/home.sock |
| +``` |
| + |
| +Unlike an instance, the router only *proxies* a service: no unit is created, no |
| +version is tracked (`instances` shows `-`), and the backend must already be up |
| +when you add it. This is how the home server (`home/`) and push relay (`push/`) |
| +run on the same box — each is its own binary + systemd unit |
| +(`lchat-home.service` / `lchat-push.service`, installed by `bootstrap`, inert |
| +until their `/data/{home,push}.env` exists), deployed by replacing its binary |
| +and restarting its unit. |
| + |
| +## Migrating an instance in from elsewhere |
| + |
| +Moving an existing instance (from another box, or another hosting setup) onto |
| +this one is a separate, validated procedure — `VACUUM INTO` snapshot, copy the |
| +DB **and its `<host>-blobs/` directory**, `add-instance --admin-token` to |
| +preserve the token, DNS flip with the old deployment kept as rollback. Follow |
| +`docs/instance-migration.md`; the key property is that the **hostname must not |
| +change** (browser keys are per-origin — a new origin means every member re-keys |
| +via add-device links). |
| + |
| +## Troubleshooting |
| + |
| +- **`lchat instances`** is the first stop: per-route status |
| + (`creating`/`ready`/`failed`), the running VERSION, and a `last error` line |
| + when something failed. |
| +- **Logs:** `journalctl -u lchat@<host> -f` (instance), |
| + `-u lchat-router -f` (TLS/routing), `-u lchat-bot@<host> -f` (sidecar). |
| +- **Cert won't issue:** check DNS actually resolves to the box, and `:80` is |
| + reachable from the internet (HTTP-01 answers there). The router only issues |
| + for hosts in its route table — an unknown host is refused by `HostPolicy` |
| + before ACME is attempted. |
| +- **On-box clients (sidecars) can't reach an instance / reach a stale server:** |
| + the `/etc/hosts` loopback pins are missing or were regenerated on boot — see |
| + the bots section above. |
| +- **Health check by hand:** `curl -s https://<host>/api/version` (build id), or |
| + on the box against the socket: |
| + `curl -s --unix-socket /run/lchat/<host>.sock http://localhost/api/version`. |
| +- **Recover an admin link:** `lchat admin-link <host>`. |
| +- **A deploy "failed" but the fleet is half-promoted:** that's by design — |
| + `deploy --all` stops at the first unhealthy instance and leaves earlier ones |
| + on the new build. Fix the failing instance (its own deploy auto-rolled it |
| + back), then re-run. |