| 1 | # 🤖 Deploying instances — the mental model |
| 2 | |
| 3 | > **Not a procedure.** Day-to-day operation goes through **`./deploy/ops`** |
| 4 | > (releases: `publish`/`promote`/`ship`; branch canaries: |
| 5 | > `canary create|deploy|destroy`; preflight: `doctor`). Where this page shows |
| 6 | > command sequences, they document what the verbs do — run the verb, and if |
| 7 | > no verb covers your task, **extend ops** rather than following prose here. |
| 8 | |
| 9 | What a box that uses the `lchat` router is made of, and why the pieces are |
| 10 | shaped this way (`docs/instance-router.md` is the design; `deploy/README.md` |
| 11 | is the one-page fresh-box bootstrap). Everything here is generic — hosts are |
| 12 | `chat.example.com`, the box is `<box>`. |
| 13 | |
| 14 | ## The mental model |
| 15 | |
| 16 | One static Go binary (`lchat`) plays three roles on the box: |
| 17 | |
| 18 | - **The router** (`lchat-router.service`) — binds `:80`/`:443`, terminates TLS |
| 19 | (Let's Encrypt via autocert HTTP-01), and reverse-proxies each hostname to a |
| 20 | unix socket. The route table lives in a small SQLite registry |
| 21 | (`/data/registry.db`). |
| 22 | - **Instances** (`lchat@<host>.service`, one per chat) — each is its own process, |
| 23 | socket (`/run/lchat/<host>.sock`), and database (`/data/<host>.db`). Fully |
| 24 | isolated: its own admin token, its own crash/memory blast radius, its own |
| 25 | binary version. |
| 26 | - **The control CLI** (`/opt/lchat/lchat add-instance | deploy | instances | …`) |
| 27 | — run on the box over SSH; it edits the registry, writes env files, and drives |
| 28 | systemd. |
| 29 | |
| 30 | Two directories carry the versioning scheme: |
| 31 | |
| 32 | ``` |
| 33 | /opt/lchat/versions/<ver>/lchat # the content store: every staged build, keyed by build id |
| 34 | /opt/lchat/live/<host> # per-instance symlink into the store; the unit execs this |
| 35 | ``` |
| 36 | |
| 37 | Each instance's unit execs its *own* `live/<host>` symlink, so two instances can |
| 38 | run two different builds at once. That's what makes canarying a one-liner. |
| 39 | |
| 40 | Per-instance config is an **EnvironmentFile** at `/data/<host>.env` (mode 0600), |
| 41 | written by `add-instance`. It holds the one secret that matters — the plaintext |
| 42 | `LCHAT_ADMIN_TOKEN` (the DB stores only its hash, so this file is the only |
| 43 | recoverable copy) — plus optional per-instance vars (`LCHAT_PUSH_RELAY_URL`/ |
| 44 | `_TOKEN`, `LCHAT_HOME_ORIGIN`, `LCHAT_APP_STORE_URL`). |
| 45 | |
| 46 | Nothing here weakens the E2E model: the router and the instance only ever see |
| 47 | ciphertext, and the admin token authorises roster management, not decryption. |
| 48 | |
| 49 | ## Prerequisites |
| 50 | |
| 51 | - **A bootstrapped box.** `scp` the binary to `/opt/lchat/lchat`, then |
| 52 | `lchat bootstrap` (idempotent) — it creates the `lchat` service user, the |
| 53 | dirs, and the systemd units, and starts the router. See `deploy/README.md`. |
| 54 | - **DNS** for the new host pointing at the box — either a wildcard |
| 55 | (`*.example.com A <box-ip>`) so new instances need no DNS step, or one A |
| 56 | record per host. Low TTL (60s) makes future moves painless. |
| 57 | - **Inbound `:80` and `:443` open** (both firewall on the box and any cloud |
| 58 | firewall). Port 80 is required — autocert's HTTP-01 challenge answers there. |
| 59 | - Certificates issue **lazily on the host's first HTTPS hit**, in a second or |
| 60 | two. There's no pre-issuance step; if DNS resolves and :80 is reachable, the |
| 61 | first `curl https://<host>/` just works. |
| 62 | |
| 63 | ## Adding an instance |
| 64 | |
| 65 | One command, run on the box: |
| 66 | |
| 67 | ```sh |
| 68 | ssh root@<box> /opt/lchat/lchat add-instance chat.example.com |
| 69 | ``` |
| 70 | |
| 71 | It mints an admin token, writes `/data/chat.example.com.env`, registers the |
| 72 | route, stages the current binary and points `live/<host>` at it, starts |
| 73 | `lchat@chat.example.com`, waits for the socket to answer, reloads the router, |
| 74 | and prints a one-time setup link (open it and you're the admin) plus the |
| 75 | admin-token link: |
| 76 | |
| 77 | ``` |
| 78 | added chat.example.com |
| 79 | get started (one-time link, makes you the admin): |
| 80 | https://chat.example.com/join/setup#… |
| 81 | admin token link: https://chat.example.com/admin#token=… |
| 82 | (recover later: lchat admin-link chat.example.com / lchat setup-invite chat.example.com) |
| 83 | ``` |
| 84 | |
| 85 | Optional flags (all become vars in the instance's env file): |
| 86 | |
| 87 | | flag | what it sets | |
| 88 | |---|---| |
| 89 | | `--admin-token <tok>` | import an existing token instead of minting — for migrations, so saved admin links keep working (see `docs/instance-migration.md`) | |
| 90 | | `--push-relay-url` / `--push-relay-token` | iOS push via a shared APNs relay (`docs/push-relay.md`); mint the token on the relay first | |
| 91 | | `--home-origin` | the passkey-backup home server this instance offers members | |
| 92 | | `--app-store-url` | where the welcome page's "Get the app" button points | |
| 93 | |
| 94 | All of these are optional — an instance without them is fully functional for |
| 95 | web + Web Push (VAPID is self-generated into the instance DB; no config). |
| 96 | |
| 97 | Then, immediately: |
| 98 | |
| 99 | - **Store the admin token somewhere durable** (a password manager). It exists |
| 100 | in exactly one recoverable place — `/data/<host>.env` on the box — and |
| 101 | `lchat admin-link <host>` reads it from there, so losing the box means losing |
| 102 | the link. |
| 103 | - **Verify:** |
| 104 | |
| 105 | ```sh |
| 106 | ssh root@<box> /opt/lchat/lchat instances # HOST / KIND / STATUS / VERSION / SOCKET |
| 107 | curl -s https://chat.example.com/api/version # {"build":"<ver>"} over a fresh LE cert |
| 108 | ``` |
| 109 | |
| 110 | Open the printed admin link in a browser to name the group and mint member |
| 111 | invites. |
| 112 | |
| 113 | If `add-instance` fails partway, the route is left in status `failed` with a |
| 114 | `last error` line shown by `lchat instances`; fix the cause and re-run — a |
| 115 | failed/creating route is overwritten on retry. |
| 116 | |
| 117 | ## Adding bots (optional) |
| 118 | |
| 119 | Each instance can have a **sidecar** hosting the default agents (Claude / |
| 120 | Eleven). The sidecar is Node (it reuses `web/static/crypto.js`, so |
| 121 | its E2E crypto can't drift from the browser's), so the box needs a one-time Node |
| 122 | setup — see `deploy/README.md § Bots` for the `apt`/`rsync`/`npm ci` recipe. |
| 123 | Then, per instance: |
| 124 | |
| 125 | ```sh |
| 126 | ssh root@<box> 'ANTHROPIC_API_KEY=sk-ant-… /opt/lchat/lchat add-bot chat.example.com' |
| 127 | ``` |
| 128 | |
| 129 | `add-bot` mints one agent invite per identity via the instance's admin API, |
| 130 | writes `/data/<host>-bot.env` (0600 — invites, session paths, the API key), and |
| 131 | starts `lchat-bot@<host>`. Notes: |
| 132 | |
| 133 | - **Instance first, then sidecar** — Eleven baselines "who already exists" from |
| 134 | the roster on its first tick; booting the sidecar against a not-yet-ready |
| 135 | instance can make it welcome-DM long-standing members later. |
| 136 | - Omit `ANTHROPIC_API_KEY` for deterministic-only bots (Eleven still welcomes |
| 137 | members and answers slash commands; no AI replies). Add the key to |
| 138 | `/data/<host>-bot.env` and `systemctl restart lchat-bot@<host>` later to give |
| 139 | it a brain. |
| 140 | - The API key is read from the **environment**, deliberately not a flag — a |
| 141 | flag would leak it into the process table and shell history. |
| 142 | - The sidecar connects to its instance over the **public URL**. If the sidecar |
| 143 | runs on the same box as the router, pin every hosted hostname to loopback in |
| 144 | `/etc/hosts` (`127.0.0.1 chat.example.com …`) so on-box clients reach the |
| 145 | local router instead of depending on external DNS — and disable cloud-init's |
| 146 | `manage_etc_hosts` so the pins survive reboots. See |
| 147 | `docs/instance-router.md § Bots`. |
| 148 | - Sidecar logs: `journalctl -u lchat-bot@chat.example.com -f`. |
| 149 | - Updating sidecar *code* is separate from binary deploys: re-`rsync` `agent/` + |
| 150 | `web/static/` to `/opt/lchat/` and restart the `lchat-bot@` units (they |
| 151 | backfill on reconnect, so a restart is safe). |
| 152 | |
| 153 | ## Shipping a new build (canary → promote) |
| 154 | |
| 155 | Deploys are **data-safe by construction**: they never touch `/data` — DB |
| 156 | migrations are additive (`CREATE TABLE IF NOT EXISTS`), so new code over old |
| 157 | data is always fine. The flow is: build locally, stage on the box, prove the |
| 158 | build on one instance, then promote the fleet. |
| 159 | |
| 160 | ```sh |
| 161 | # 1. Build linux/amd64, stamped with the git short sha as its build id |
| 162 | ./deploy/build-linux.sh # → lchat-linux-amd64, prints <ver> |
| 163 | |
| 164 | # 2. Copy it up and register it in the version store |
| 165 | scp lchat-linux-amd64 root@<box>:/opt/lchat/incoming |
| 166 | ssh root@<box> /opt/lchat/lchat stage --file /opt/lchat/incoming # → prints <ver> |
| 167 | |
| 168 | # 3. Canary ONE instance |
| 169 | ssh root@<box> /opt/lchat/lchat deploy canary.example.com --version <ver> |
| 170 | # …verify: open the app, send a message, check /api/version reports <ver>… |
| 171 | |
| 172 | # 4. Promote everyone (one at a time, health-checked, stops on first failure) |
| 173 | ssh root@<box> /opt/lchat/lchat deploy --all --version <ver> |
| 174 | |
| 175 | # 5. Who runs what |
| 176 | ssh root@<box> /opt/lchat/lchat instances # VERSION column shows per-instance skew |
| 177 | ``` |
| 178 | |
| 179 | What `deploy --version` actually does: atomically repoint |
| 180 | `/opt/lchat/live/<host>` at the staged build, `systemctl restart` the instance |
| 181 | (SIGTERM drains in-flight HTTP and closes WebSockets so clients reconnect; the |
| 182 | router retries dials across the sub-second gap, so requests queue rather than |
| 183 | fail), then poll `/api/version` until the instance reports the **expected build |
| 184 | id** — not merely a 200. On mismatch it **auto-rolls-back** to the previous |
| 185 | version and errors out. `deploy --all` walks the fleet one instance at a time |
| 186 | and stops at the first failure, leaving already-promoted instances on the new |
| 187 | build. |
| 188 | |
| 189 | Stepping back is explicit too: |
| 190 | |
| 191 | ```sh |
| 192 | ssh root@<box> /opt/lchat/lchat rollback chat.example.com # → its recorded previous version |
| 193 | ``` |
| 194 | |
| 195 | Other notes: |
| 196 | |
| 197 | - `deploy <host>` with **no** `--version` just gracefully restarts the instance |
| 198 | on its current build (useful after editing its env file). |
| 199 | - Build ids are the git short sha by default; `./deploy/build-linux.sh |
| 200 | my-label` stamps a custom id (handy for branch builds, below). `lchat stage` |
| 201 | reads the id out of the binary itself, so the store can't mislabel a build. |
| 202 | - The build script targets linux/amd64; on an arm64 box set |
| 203 | `GOARCH=arm64` (edit or inline-override the script). |
| 204 | - **⚠️ `deploy --all` moves *every* instance**, including one you've pinned to |
| 205 | a branch build. Before promoting a fleet, check the VERSION column of |
| 206 | `lchat instances` — if a canary instance is meant to stay on its branch |
| 207 | build, deploy the others individually instead of using `--all`. |
| 208 | - To also refresh the **control/router** binary (rarely needed — instance |
| 209 | builds don't require it): atomically replace `/opt/lchat/lchat` (`cp` to a |
| 210 | temp name + `mv` over it) and `systemctl restart lchat-router`. |
| 211 | |
| 212 | ### Verifying a canary |
| 213 | |
| 214 | "Verify" should be more than a 200. A reasonable checklist for a few minutes on |
| 215 | the canary instance before promoting: |
| 216 | |
| 217 | - `curl -s https://<host>/api/version` reports the new build id. |
| 218 | - Open the app in a browser: history decrypts, sending a message round-trips, |
| 219 | and a WebSocket reconnect happened cleanly (no stuck "connecting"). |
| 220 | - If the change touched uploads/push/bots, exercise that path specifically. |
| 221 | - `journalctl -u lchat@<host> -e` shows no new errors. |
| 222 | |
| 223 | ## Standing up a canary *instance* (branch builds) |
| 224 | |
| 225 | Sometimes the thing to prove isn't a build of `main` on a real instance, but an |
| 226 | **unmerged branch** on a **throwaway instance** — real TLS, real devices, zero |
| 227 | real users, and the fleet (and the control binary) untouched. Because versions |
| 228 | are per-instance, this needs no special machinery: |
| 229 | |
| 230 | ```sh |
| 231 | # on your machine, on the feature branch: |
| 232 | ./deploy/build-linux.sh # id = the branch's git short sha |
| 233 | scp lchat-linux-amd64 root@<box>:/opt/lchat/incoming |
| 234 | |
| 235 | # on the box: |
| 236 | /opt/lchat/lchat stage --file /opt/lchat/incoming # → <branch-ver> |
| 237 | /opt/lchat/lchat add-instance canary-feature.example.com # starts on the CURRENT fleet build |
| 238 | /opt/lchat/lchat deploy canary-feature.example.com --version <branch-ver> # pin it to the branch |
| 239 | ``` |
| 240 | |
| 241 | The branch binary exists only in the version store; nothing else runs it. |
| 242 | `lchat instances` shows the skew explicitly. When the feature merges and ships: |
| 243 | |
| 244 | ```sh |
| 245 | /opt/lchat/lchat deploy canary-feature.example.com --version <new-main-ver> # rejoin the fleet |
| 246 | # …or tear the whole instance down (below) if it was a throwaway. |
| 247 | ``` |
| 248 | |
| 249 | Two cautions: |
| 250 | |
| 251 | - Data written by a branch build persists in `/data/<host>.db`. Migrations are |
| 252 | additive so *moving forward* is safe, but a branch that created tables/rows a |
| 253 | later design abandons leaves them behind. For feature trials, prefer a |
| 254 | **throwaway** instance you'll delete over pinning a real one. |
| 255 | - Remember the `deploy --all` warning above — a fleet promotion will stomp the |
| 256 | pin. |
| 257 | |
| 258 | ## Removing an instance |
| 259 | |
| 260 | `remove` is deliberately data-safe by default — it only stops *routing*: |
| 261 | |
| 262 | ```sh |
| 263 | /opt/lchat/lchat remove chat.example.com # route deleted; instance still running, DB/env kept |
| 264 | /opt/lchat/lchat remove chat.example.com --stop # also stop+disable lchat@<host> and lchat-bot@<host> |
| 265 | ``` |
| 266 | |
| 267 | Even with `--stop`, the DB and env file stay on disk, so the instance can be |
| 268 | resurrected: re-run `add-instance` passing `--admin-token` with the token still |
| 269 | in `/data/<host>.env` (a plain `add-instance` would mint a fresh token and |
| 270 | invalidate saved admin links). **Full teardown** — only for an instance you truly mean to destroy — is the |
| 271 | manual part, on the box: |
| 272 | |
| 273 | ```sh |
| 274 | rm /data/chat.example.com.db /data/chat.example.com.db-wal /data/chat.example.com.db-shm # the DB |
| 275 | rm /data/chat.example.com.env # the admin token |
| 276 | rm -rf /data/chat.example.com-blobs # disk-backed attachments (if present) |
| 277 | rm -rf /data/chat.example.com-bot /data/chat.example.com-bot.env # sidecar sessions + env (if it had bots) |
| 278 | rm /opt/lchat/live/chat.example.com # the version pointer |
| 279 | ``` |
| 280 | |
| 281 | …then delete the host's DNS record (if it wasn't wildcard-covered) and the |
| 282 | stored admin token. There is no undo after the DB is gone. |
| 283 | |
| 284 | ## Sibling services (home server, push relay, third-party backends) |
| 285 | |
| 286 | The router can front processes it doesn't manage — anything already listening |
| 287 | on a local unix socket: |
| 288 | |
| 289 | ```sh |
| 290 | /opt/lchat/lchat add-service home.example.com --socket /run/lchat/home.sock |
| 291 | ``` |
| 292 | |
| 293 | Unlike an instance, the router only *proxies* a service: no unit is created, no |
| 294 | version is tracked (`instances` shows `-`), and the backend must already be up |
| 295 | when you add it. This is how the home server (`home/`) and push relay (`push/`) |
| 296 | run on the same box — each is its own binary + systemd unit |
| 297 | (`lchat-home.service` / `lchat-push.service`, installed by `bootstrap`, inert |
| 298 | until their `/data/{home,push}.env` exists), deployed by replacing its binary |
| 299 | and restarting its unit. |
| 300 | |
| 301 | ## Migrating an instance in from elsewhere |
| 302 | |
| 303 | Moving an existing instance (from another box, or another hosting setup) onto |
| 304 | this one is a separate, validated procedure — `VACUUM INTO` snapshot, copy the |
| 305 | DB **and its `<host>-blobs/` directory**, `add-instance --admin-token` to |
| 306 | preserve the token, DNS flip with the old deployment kept as rollback. Follow |
| 307 | `docs/instance-migration.md`; the key property is that the **hostname must not |
| 308 | change** (browser keys are per-origin — a new origin means every member re-keys |
| 309 | via add-device links). |
| 310 | |
| 311 | ## Troubleshooting |
| 312 | |
| 313 | - **`lchat instances`** is the first stop: per-route status |
| 314 | (`creating`/`ready`/`failed`), the running VERSION, and a `last error` line |
| 315 | when something failed. |
| 316 | - **Logs:** `journalctl -u lchat@<host> -f` (instance), |
| 317 | `-u lchat-router -f` (TLS/routing), `-u lchat-bot@<host> -f` (sidecar). |
| 318 | - **Cert won't issue:** check DNS actually resolves to the box, and `:80` is |
| 319 | reachable from the internet (HTTP-01 answers there). The router only issues |
| 320 | for hosts in its route table — an unknown host is refused by `HostPolicy` |
| 321 | before ACME is attempted. |
| 322 | - **On-box clients (sidecars) can't reach an instance / reach a stale server:** |
| 323 | the `/etc/hosts` loopback pins are missing or were regenerated on boot — see |
| 324 | the bots section above. |
| 325 | - **Health check by hand:** `curl -s https://<host>/api/version` (build id), or |
| 326 | on the box against the socket: |
| 327 | `curl -s --unix-socket /run/lchat/<host>.sock http://localhost/api/version`. |
| 328 | - **Recover an admin link:** `lchat admin-link <host>`. |
| 329 | - **A deploy "failed" but the fleet is half-promoted:** that's by design — |
| 330 | `deploy --all` stops at the first unhealthy instance and leaves earlier ones |
| 331 | on the new build. Fix the failing instance (its own deploy auto-rolled it |
| 332 | back), then re-run. |
| 333 | |
amadan