🤖 Deploying instances — the mental model

> Not a procedure. Day-to-day operation goes through ./deploy/ops > (releases: publish/promote/ship; branch canaries: > canary create|deploy|destroy; preflight: doctor). Where this page shows > command sequences, they document what the verbs do — run the verb, and if > no verb covers your task, extend ops rather than following prose here.

What a box that uses the lchat router is made of, and why the pieces are shaped this way (docs/instance-router.md is the design; deploy/README.md is the one-page fresh-box bootstrap). Everything here is generic — hosts are chat.example.com, the box is <box>.

The mental model

One static Go binary (lchat) plays three roles on the box:

• The router (lchat-router.service) — binds :80/:443, terminates TLS (Let's Encrypt via autocert HTTP-01), and reverse-proxies each hostname to a unix socket. The route table lives in a small SQLite registry (/data/registry.db).
• Instances (lchat@<host>.service, one per chat) — each is its own process, socket (/run/lchat/<host>.sock), and database (/data/<host>.db). Fully isolated: its own admin token, its own crash/memory blast radius, its own binary version.
• The control CLI (/opt/lchat/lchat add-instance | deploy | instances | …) — run on the box over SSH; it edits the registry, writes env files, and drives systemd.

Two directories carry the versioning scheme:

/opt/lchat/versions/<ver>/lchat     # the content store: every staged build, keyed by build id
/opt/lchat/live/<host>              # per-instance symlink into the store; the unit execs this

Each instance's unit execs its own live/<host> symlink, so two instances can run two different builds at once. That's what makes canarying a one-liner.

Per-instance config is an EnvironmentFile at /data/<host>.env (mode 0600), written by add-instance. It holds the one secret that matters — the plaintext LCHAT_ADMIN_TOKEN (the DB stores only its hash, so this file is the only recoverable copy) — plus optional per-instance vars (LCHAT_PUSH_RELAY_URL/ _TOKEN, LCHAT_HOME_ORIGIN, LCHAT_APP_STORE_URL).

Nothing here weakens the E2E model: the router and the instance only ever see ciphertext, and the admin token authorises roster management, not decryption.

Prerequisites

• A bootstrapped box. scp the binary to /opt/lchat/lchat, then lchat bootstrap (idempotent) — it creates the lchat service user, the dirs, and the systemd units, and starts the router. See deploy/README.md.
• DNS for the new host pointing at the box — either a wildcard (*.example.com A <box-ip>) so new instances need no DNS step, or one A record per host. Low TTL (60s) makes future moves painless.
• Inbound :80 and :443 open (both firewall on the box and any cloud firewall). Port 80 is required — autocert's HTTP-01 challenge answers there.
• Certificates issue lazily on the host's first HTTPS hit, in a second or two. There's no pre-issuance step; if DNS resolves and :80 is reachable, the first curl https://<host>/ just works.

Adding an instance

One command, run on the box:

ssh root@<box> /opt/lchat/lchat add-instance chat.example.com

It mints an admin token, writes /data/chat.example.com.env, registers the route, stages the current binary and points live/<host> at it, starts lchat@chat.example.com, waits for the socket to answer, reloads the router, and prints a one-time setup link (open it and you're the admin) plus the admin-token link:

added chat.example.com
  get started (one-time link, makes you the admin):
  https://chat.example.com/join/setup#…
  admin token link: https://chat.example.com/admin#token=…
  (recover later: lchat admin-link chat.example.com / lchat setup-invite chat.example.com)

Optional flags (all become vars in the instance's env file):

| flag | what it sets | |---|---| | --admin-token <tok> | import an existing token instead of minting — for migrations, so saved admin links keep working (see docs/instance-migration.md) | | --push-relay-url / --push-relay-token | iOS push via a shared APNs relay (docs/push-relay.md); mint the token on the relay first | | --home-origin | the passkey-backup home server this instance offers members | | --app-store-url | where the welcome page's "Get the app" button points |

All of these are optional — an instance without them is fully functional for web + Web Push (VAPID is self-generated into the instance DB; no config).

Then, immediately:

• Store the admin token somewhere durable (a password manager). It exists in exactly one recoverable place — /data/<host>.env on the box — and lchat admin-link <host> reads it from there, so losing the box means losing the link.
• Verify:

ssh root@<box> /opt/lchat/lchat instances     # HOST / KIND / STATUS / VERSION / SOCKET
curl -s https://chat.example.com/api/version  # {"build":"<ver>"} over a fresh LE cert

Open the printed admin link in a browser to name the group and mint member invites.

If add-instance fails partway, the route is left in status failed with a last error line shown by lchat instances; fix the cause and re-run — a failed/creating route is overwritten on retry.

Adding bots (optional)

Each instance can have a sidecar hosting the default agents (Claude / Eleven). The sidecar is Node (it reuses web/static/crypto.js, so its E2E crypto can't drift from the browser's), so the box needs a one-time Node setup — see deploy/README.md § Bots for the apt/rsync/npm ci recipe. Then, per instance:

ssh root@<box> 'ANTHROPIC_API_KEY=sk-ant-… /opt/lchat/lchat add-bot chat.example.com'

add-bot mints one agent invite per identity via the instance's admin API, writes /data/<host>-bot.env (0600 — invites, session paths, the API key), and starts lchat-bot@<host>. Notes:

• Instance first, then sidecar — Eleven baselines "who already exists" from the roster on its first tick; booting the sidecar against a not-yet-ready instance can make it welcome-DM long-standing members later.
• Omit ANTHROPIC_API_KEY for deterministic-only bots (Eleven still welcomes members and answers slash commands; no AI replies). Add the key to /data/<host>-bot.env and systemctl restart lchat-bot@<host> later to give it a brain.
• The API key is read from the environment, deliberately not a flag — a flag would leak it into the process table and shell history.
• The sidecar connects to its instance over the public URL. If the sidecar runs on the same box as the router, pin every hosted hostname to loopback in /etc/hosts (127.0.0.1 chat.example.com …) so on-box clients reach the local router instead of depending on external DNS — and disable cloud-init's manage_etc_hosts so the pins survive reboots. See docs/instance-router.md § Bots.
• Sidecar logs: journalctl -u lchat-bot@chat.example.com -f.
• Updating sidecar code is separate from binary deploys: re-rsync agent/ + web/static/ to /opt/lchat/ and restart the lchat-bot@ units (they backfill on reconnect, so a restart is safe).

Shipping a new build (canary → promote)

Deploys are data-safe by construction: they never touch /data — DB migrations are additive (CREATE TABLE IF NOT EXISTS), so new code over old data is always fine. The flow is: build locally, stage on the box, prove the build on one instance, then promote the fleet.

# 1. Build linux/amd64, stamped with the git short sha as its build id
./deploy/build-linux.sh                       # → lchat-linux-amd64, prints <ver>

# 2. Copy it up and register it in the version store
scp lchat-linux-amd64 root@<box>:/opt/lchat/incoming
ssh root@<box> /opt/lchat/lchat stage --file /opt/lchat/incoming   # → prints <ver>

# 3. Canary ONE instance
ssh root@<box> /opt/lchat/lchat deploy canary.example.com --version <ver>
# …verify: open the app, send a message, check /api/version reports <ver>…

# 4. Promote everyone (one at a time, health-checked, stops on first failure)
ssh root@<box> /opt/lchat/lchat deploy --all --version <ver>

# 5. Who runs what
ssh root@<box> /opt/lchat/lchat instances     # VERSION column shows per-instance skew

What deploy --version actually does: atomically repoint /opt/lchat/live/<host> at the staged build, systemctl restart the instance (SIGTERM drains in-flight HTTP and closes WebSockets so clients reconnect; the router retries dials across the sub-second gap, so requests queue rather than fail), then poll /api/version until the instance reports the expected build id — not merely a 200. On mismatch it auto-rolls-back to the previous version and errors out. deploy --all walks the fleet one instance at a time and stops at the first failure, leaving already-promoted instances on the new build.

Stepping back is explicit too:

ssh root@<box> /opt/lchat/lchat rollback chat.example.com   # → its recorded previous version

Other notes:

• deploy <host> with no --version just gracefully restarts the instance on its current build (useful after editing its env file).
• Build ids are the git short sha by default; ./deploy/build-linux.sh my-label stamps a custom id (handy for branch builds, below). lchat stage reads the id out of the binary itself, so the store can't mislabel a build.
• The build script targets linux/amd64; on an arm64 box set GOARCH=arm64 (edit or inline-override the script).
• *⚠️ deploy --all moves every instance*, including one you've pinned to a branch build. Before promoting a fleet, check the VERSION column of lchat instances — if a canary instance is meant to stay on its branch build, deploy the others individually instead of using --all.
• To also refresh the control/router binary (rarely needed — instance builds don't require it): atomically replace /opt/lchat/lchat (cp to a temp name + mv over it) and systemctl restart lchat-router.

Verifying a canary

"Verify" should be more than a 200. A reasonable checklist for a few minutes on the canary instance before promoting:

• curl -s https://<host>/api/version reports the new build id.
• Open the app in a browser: history decrypts, sending a message round-trips, and a WebSocket reconnect happened cleanly (no stuck "connecting").
• If the change touched uploads/push/bots, exercise that path specifically.
• journalctl -u lchat@<host> -e shows no new errors.

Standing up a canary instance (branch builds)

Sometimes the thing to prove isn't a build of main on a real instance, but an unmerged branch on a throwaway instance — real TLS, real devices, zero real users, and the fleet (and the control binary) untouched. Because versions are per-instance, this needs no special machinery:

# on your machine, on the feature branch:
./deploy/build-linux.sh                       # id = the branch's git short sha
scp lchat-linux-amd64 root@<box>:/opt/lchat/incoming

# on the box:
/opt/lchat/lchat stage --file /opt/lchat/incoming        # → <branch-ver>
/opt/lchat/lchat add-instance canary-feature.example.com # starts on the CURRENT fleet build
/opt/lchat/lchat deploy canary-feature.example.com --version <branch-ver>   # pin it to the branch

The branch binary exists only in the version store; nothing else runs it. lchat instances shows the skew explicitly. When the feature merges and ships:

/opt/lchat/lchat deploy canary-feature.example.com --version <new-main-ver>  # rejoin the fleet
# …or tear the whole instance down (below) if it was a throwaway.

Two cautions:

• Data written by a branch build persists in /data/<host>.db. Migrations are additive so moving forward is safe, but a branch that created tables/rows a later design abandons leaves them behind. For feature trials, prefer a throwaway instance you'll delete over pinning a real one.
• Remember the deploy --all warning above — a fleet promotion will stomp the pin.

Removing an instance

remove is deliberately data-safe by default — it only stops routing:

/opt/lchat/lchat remove chat.example.com          # route deleted; instance still running, DB/env kept
/opt/lchat/lchat remove chat.example.com --stop   # also stop+disable lchat@<host> and lchat-bot@<host>

Even with --stop, the DB and env file stay on disk, so the instance can be resurrected: re-run add-instance passing --admin-token with the token still in /data/<host>.env (a plain add-instance would mint a fresh token and invalidate saved admin links). Full teardown — only for an instance you truly mean to destroy — is the manual part, on the box:

rm /data/chat.example.com.db /data/chat.example.com.db-wal /data/chat.example.com.db-shm  # the DB
rm /data/chat.example.com.env                                # the admin token
rm -rf /data/chat.example.com-blobs                          # disk-backed attachments (if present)
rm -rf /data/chat.example.com-bot /data/chat.example.com-bot.env  # sidecar sessions + env (if it had bots)
rm /opt/lchat/live/chat.example.com                          # the version pointer

…then delete the host's DNS record (if it wasn't wildcard-covered) and the stored admin token. There is no undo after the DB is gone.

Sibling services (home server, push relay, third-party backends)

The router can front processes it doesn't manage — anything already listening on a local unix socket:

/opt/lchat/lchat add-service home.example.com --socket /run/lchat/home.sock

Unlike an instance, the router only proxies a service: no unit is created, no version is tracked (instances shows -), and the backend must already be up when you add it. This is how the home server (home/) and push relay (push/) run on the same box — each is its own binary + systemd unit (lchat-home.service / lchat-push.service, installed by bootstrap, inert until their /data/{home,push}.env exists), deployed by replacing its binary and restarting its unit.

Migrating an instance in from elsewhere

Moving an existing instance (from another box, or another hosting setup) onto this one is a separate, validated procedure — VACUUM INTO snapshot, copy the DB and its <host>-blobs/ directory, add-instance --admin-token to preserve the token, DNS flip with the old deployment kept as rollback. Follow docs/instance-migration.md; the key property is that the hostname must not change (browser keys are per-origin — a new origin means every member re-keys via add-device links).

Troubleshooting

• lchat instances is the first stop: per-route status (creating/ready/failed), the running VERSION, and a last error line when something failed.
• Logs: journalctl -u lchat@<host> -f (instance), -u lchat-router -f (TLS/routing), -u lchat-bot@<host> -f (sidecar).
• Cert won't issue: check DNS actually resolves to the box, and :80 is reachable from the internet (HTTP-01 answers there). The router only issues for hosts in its route table — an unknown host is refused by HostPolicy before ACME is attempted.
• On-box clients (sidecars) can't reach an instance / reach a stale server: the /etc/hosts loopback pins are missing or were regenerated on boot — see the bots section above.
• Health check by hand: curl -s https://<host>/api/version (build id), or on the box against the socket: curl -s --unix-socket /run/lchat/<host>.sock http://localhost/api/version.
• Recover an admin link: lchat admin-link <host>.
• A deploy "failed" but the fleet is half-promoted: that's by design — deploy --all stops at the first unhealthy instance and leaves earlier ones on the new build. Fix the failing instance (its own deploy auto-rolled it back), then re-run.