| 1 | # 🤖 The operator surface: `./deploy/ops` |
| 2 | |
| 3 | One command surface for deploying and managing the fleet, so operating doesn't |
| 4 | mean re-gathering knowledge from `CLAUDE.md`, `docs/server-releases.md`, |
| 5 | `docs/multi-edge-dev-cutover.md`, and the vault's `RUNBOOK.md` every time. |
| 6 | Bare `./deploy/ops` lists the verbs: |
| 7 | |
| 8 | ```sh |
| 9 | ./deploy/ops doctor # can I deploy right now? (read-only preflight) |
| 10 | ./deploy/ops validate # run every local check (go|web|shell|infra) |
| 11 | ./deploy/ops provision units # re-run bootstrap so NEW systemd units land on the fleet |
| 12 | ./deploy/ops provision apex 11ch.at # declare a box the short-link apex |
| 13 | ./deploy/ops provision status # read-only: which box runs the short-link redirector |
| 14 | ./deploy/ops publish "notes" # build main -> release -> point edge at it |
| 15 | ./deploy/ops promote <ver> beta # move a channel pointer (edge|beta|stable) |
| 16 | ./deploy/ops ship # every fleet box catches up NOW |
| 17 | ./deploy/ops canary create x # per-session test instance (test-x.11ch.at) |
| 18 | ./deploy/ops canary deploy x # build THIS branch onto it — always pinned |
| 19 | ./deploy/ops canary reset x # wipe back to first-run (iterate onboarding) |
| 20 | ./deploy/ops canary destroy x # reset + remove, data archived to trash |
| 21 | ./deploy/ops dev plan # dev-mesh tofu plan, AMI-replacement guarded |
| 22 | ./deploy/ops dev apply # tofu apply behind the same guard |
| 23 | ./deploy/ops dev deploy # dev mesh: build -> scp -> swap -> restart |
| 24 | ./deploy/ops dev provision # dev mesh: refresh systemd units + route short links |
| 25 | ./deploy/ops dev rebuild # destroy + recreate the Dublin box (new keys) |
| 26 | ./deploy/ops ledger "why..." # draft the public SSH-access why-entry |
| 27 | ./deploy/ops pages push # committed web/status data -> the status box |
| 28 | ``` |
| 29 | |
| 30 | Verbs that need credentials fetch them themselves (the `release-server.sh` |
| 31 | shape): `publish`/`promote` pull the signing key from Bitwarden, and `ops dev` |
| 32 | pulls `AWS_*` from your per-operator dev agent item and `DNSIMPLE_*` from the |
| 33 | DNSimple token item — unlocking interactively when the vault is locked, |
| 34 | degrading to instructions (never a hang) without a tty, and syncing bw's |
| 35 | local cache when a lookup looks stale. Values already in the environment |
| 36 | always win, and no secret is ever printed. Which items, and where each value |
| 37 | lives inside them, is the **vault contract** — `deploy/vault-items.sh`, the |
| 38 | one place scripts and vault agree: scripts read exactly what it says and fail |
| 39 | crisply when the vault doesn't match (never a fuzzy search), and renaming a |
| 40 | vault item is a reviewed PR there plus the vault edit, in step. |
| 41 | |
| 42 | Each verb is a readable standalone script beside the dispatcher |
| 43 | (`deploy/ops-*.sh`, `deploy/doctor.sh`) — the outlive-AI rule: no runtime, no |
| 44 | machinery only an LLM can drive. The verbs *encode the rules* that used to |
| 45 | live only in prose: publish refuses a HEAD that isn't landed main; there is no |
| 46 | `dev` channel; canary deploys always `--pin`; canary destroy wipes data the |
| 47 | way `lchat remove` deliberately doesn't (via the canary-gated `lchat reset` — |
| 48 | the sudoers allowlist has no `rm`); fleet sweeps leave `AUTO=pinned` rows |
| 49 | alone; a `tofu plan` that would replace the Dublin box refuses to look green. |
| 50 | When a verb's advice goes stale, **fix it in the script, not in prose** — a |
| 51 | script that runs drifts loudly, where a document drifts silently. |
| 52 | |
| 53 | **The standing rule — no cookbooks.** Operational procedure lives in these |
| 54 | scripts and nowhere else. When a task has no verb, or a verb falls short, |
| 55 | **extend ops** (a new verb, subcommand, or flag in a small PR) and then run |
| 56 | it — never hand-run a sequence reconstructed from `docs/`, `CLAUDE.md`, |
| 57 | `CLAUDE.local.md`, or an old runbook. The docs are design rationale and |
| 58 | history; each time one is caught carrying live procedure, encode the |
| 59 | procedure here and banner the doc. This is what keeps operations safe and |
| 60 | repeatable for humans and agents alike. |
| 61 | |
| 62 | Start with the preflight. `ops doctor` checks local tooling, the Bitwarden |
| 63 | signing key (unlocking to verify by default — `--no-unlock` prevents it; the |
| 64 | session token stays in that one process and is never printed), `gh` write |
| 65 | access to the releases repo, WireGuard + SSH + `lchat-operators` sudo on every |
| 66 | fleet box, and the dev mesh's key/profile/firewall/SSH — printing the fix |
| 67 | under each failure. It is read-only, never prints a secret, and exits non-zero |
| 68 | on hard failures only, so it works as a gate in other scripts. |
| 69 | |
| 70 | Two obligations the surface reminds you of, because they are easy to forget: |
| 71 | box-SSH verbs (`ship`, `canary`, `dev deploy`, doctor's `fleet` section — use |
| 72 | `--no-ssh` to avoid it) incur a **why-entry in the public access ledger** |
| 73 | (`web/status/ssh/`), which `ops ledger` drafts; and box aliases come from your |
| 74 | `~/.ssh/config` (defaults in the scripts, overrides via `LCHAT_BOXES` / |
| 75 | `LCHAT_CANARY_BOX` / `LCHAT_DEV_BOX_*`, environment specifics in |
| 76 | `CLAUDE.local.md`). |
| 77 | |
| 78 | # 🤖 Single-machine bootstrap |
| 79 | |
| 80 | Running the messenger behind the `lchat` router on one box (see |
| 81 | `docs/instance-router.md`). The Go side is a single static binary — no runtime. |
| 82 | |
| 83 | ```sh |
| 84 | # 1. copy the binary, then let the binary provision the box |
| 85 | scp lchat root@<box>:/opt/lchat/lchat |
| 86 | ssh root@<box> '/opt/lchat/lchat bootstrap' # writes units + dirs, starts the router |
| 87 | |
| 88 | # 2. add an instance (mints token + env file, starts it, registers + reloads) |
| 89 | ssh root@<box> '/opt/lchat/lchat add-instance chat.example.com' |
| 90 | |
| 91 | # 3. operator setup — the last thing the root session does before retiring itself: |
| 92 | # create a named account per operator (groups lchat-operators + systemd-journal), |
| 93 | # install deploy/sudoers-operators -> /etc/sudoers.d/lchat-operators (0440) |
| 94 | # and deploy/lchat-env-install + deploy/lchat-svc -> /usr/local/sbin/ (0755), |
| 95 | # add each operator's WireGuard peer, close public 22, and run |
| 96 | # `lchat bootstrap --ssh-noroot` — writes the PermitRootLogin-no drop-in |
| 97 | # (/etc/ssh/sshd_config.d/10-lchat-noroot.conf), validates, reloads sshd, |
| 98 | # and confirms the EFFECTIVE config reports no (sshd is first-match-wins; |
| 99 | # a pre-existing drop-in sorting before 10- silently wins otherwise). |
| 100 | # It refuses until lchat-operators has members, and every later full |
| 101 | # bootstrap converges the same drop-in — a box that missed the step picks |
| 102 | # it up on the next provision pass, root shell not required. |
| 103 | ``` |
| 104 | |
| 105 | `lchat bootstrap` embeds the unit files (source `deploy/systemd/`) and writes |
| 106 | them, creates `/data` + `/run/lchat`, and starts the router. It also installs |
| 107 | the **public status collectors** — the SSH access ledger + fleet hardening |
| 108 | posture scripts (`deploy/{ssh-ledger-collect,hardening-posture}.sh`), their four |
| 109 | systemd units, and the static viewer pages (`web/status/*.html` + `status.css`, |
| 110 | into `/opt/lchat/status-pages/`) — and enables their 5-minute timers. These ride |
| 111 | the binary because a hardened box has no root shell to scp them onto (operator |
| 112 | SSH is WireGuard-only; `/opt/lchat` + `/etc/systemd/system` are root-owned). To |
| 113 | refresh just the collectors + pages on a live box without touching the router, |
| 114 | run `sudo -n /opt/lchat/lchat bootstrap --collectors-only` (allowlisted by |
| 115 | `deploy/sudoers-operators`) — no member-facing blip. The collector-generated |
| 116 | JSON (fleet/ssh/hardening/… data) stays mutable on the box; only the `.html` + |
| 117 | `.css` viewer assets are embedded. To install by hand instead: `scp |
| 118 | deploy/systemd/*.service` to `/etc/systemd/system/`, drop a |
| 119 | `/etc/tmpfiles.d/lchat.conf` (`d /run/lchat 0710 root root -`), `daemon-reload`, |
| 120 | `systemctl enable --now lchat-router`. |
| 121 | |
| 122 | **Root SSH is for first bootstrap only.** After step 3, operators are named |
| 123 | users over the WireGuard tunnel, and root capability exists only through the |
| 124 | committed sudoers allowlist (`deploy/sudoers-operators`) — anything new that |
| 125 | needs root gets an allowlist line via a reviewed PR, never a root shell. The |
| 126 | full recipe (firewall, WireGuard peer configs, break-glass via the provider's |
| 127 | rescue console, the deadman revert pattern): `docs/instance-router.md |
| 128 | § Hardening`. |
| 129 | |
| 130 | - `lchat-router.service` — the front door (binds :80/:443, autocert HTTP-01). |
| 131 | - `lchat@.service` — an instance template; runs one instance on |
| 132 | `/run/lchat/<host>.sock`, reading its token + config from `/data/<host>.env`. |
| 133 | - `lchat-bot@.service` — the optional per-instance Node sidecar (see Bots). |
| 134 | - Runs as root here so it can bind :80/:443; for a non-root user add |
| 135 | `AmbientCapabilities=CAP_NET_BIND_SERVICE` to the router unit and a dedicated |
| 136 | `User=`/`Group=` with matching ownership on `/run/lchat` + `/data`. |
| 137 | |
| 138 | ## Bots (sidecars) |
| 139 | |
| 140 | The sidecar is the **same Go binary** as the instance — `lchat bot run` — so a box |
| 141 | that runs the server runs its bots with **no extra install** (no Node runtime). The |
| 142 | `lchat-bot@.service` template execs the instance's own versioned live symlink |
| 143 | (`ExecStart=/opt/lchat/live/%i bot run`), so a bot always runs the **same version |
| 144 | as its app**, and `deploy`/`update` restart it automatically when they move the |
| 145 | instance (`docs/instance-router.md § Bots`, § Auto-updating). |
| 146 | |
| 147 | ```sh |
| 148 | # per instance: mint the three agent invites + start the sidecar |
| 149 | # (API key via env, not a flag, so it doesn't leak into ps/shell history) |
| 150 | ssh root@<box> 'ANTHROPIC_API_KEY=sk-ant-... /opt/lchat/lchat add-bot chat.example.com' |
| 151 | ``` |
| 152 | |
| 153 | `add-bot` mints an agent invite per identity (Claude / Eleven) via the |
| 154 | instance's admin API, writes `/data/<host>-bot.env` (invites + `LCHAT_BASE` + |
| 155 | `LCHAT_*_SESSION` paths + optional `ANTHROPIC_API_KEY`), and enables |
| 156 | `lchat-bot@<host>` — which reads all of that from the env file, so there's nothing |
| 157 | per-host to configure in the unit. Omit `ANTHROPIC_API_KEY` for a deterministic-only |
| 158 | bot (Eleven still welcomes new members + answers slash commands). Start the |
| 159 | **instance before** the sidecar so Eleven baselines the roster correctly. |
| 160 | |
| 161 | > The Node reference sidecar (`agent/`) is kept on purpose as a runnable example; |
| 162 | > self-hosters who prefer it still need a Node runtime + `npm ci` in `/opt/lchat/agent` |
| 163 | > and point the unit's `ExecStart` at `node /opt/lchat/agent/agent.mjs`. The default |
| 164 | > deployment is Go. |
| 165 | |
amadan