🤖 The operator surface: ./deploy/ops

One command surface for deploying and managing the fleet, so operating doesn't mean re-gathering knowledge from CLAUDE.md, docs/server-releases.md, docs/multi-edge-dev-cutover.md, and the vault's RUNBOOK.md every time. Bare ./deploy/ops lists the verbs:

./deploy/ops doctor              # can I deploy right now? (read-only preflight)
./deploy/ops validate            # run every local check (go|web|shell|infra)
./deploy/ops provision units     # re-run bootstrap so NEW systemd units land on the fleet
./deploy/ops provision apex 11ch.at   # declare a box the short-link apex
./deploy/ops provision status    # read-only: which box runs the short-link redirector
./deploy/ops publish "notes"     # build main -> release -> point edge at it
./deploy/ops promote <ver> beta  # move a channel pointer (edge|beta|stable)
./deploy/ops ship                # every fleet box catches up NOW
./deploy/ops canary create x     # per-session test instance (test-x.11ch.at)
./deploy/ops canary deploy x     # build THIS branch onto it — always pinned
./deploy/ops canary reset x      # wipe back to first-run (iterate onboarding)
./deploy/ops canary destroy x    # reset + remove, data archived to trash
./deploy/ops dev plan            # dev-mesh tofu plan, AMI-replacement guarded
./deploy/ops dev apply           # tofu apply behind the same guard
./deploy/ops dev deploy          # dev mesh: build -> scp -> swap -> restart
./deploy/ops dev provision       # dev mesh: refresh systemd units + route short links
./deploy/ops dev rebuild         # destroy + recreate the Dublin box (new keys)
./deploy/ops ledger "why..."     # draft the public SSH-access why-entry
./deploy/ops pages push          # committed web/status data -> the status box

Verbs that need credentials fetch them themselves (the release-server.sh shape): publish/promote pull the signing key from Bitwarden, and ops dev pulls AWS_* from your per-operator dev agent item and DNSIMPLE_* from the DNSimple token item — unlocking interactively when the vault is locked, degrading to instructions (never a hang) without a tty, and syncing bw's local cache when a lookup looks stale. Values already in the environment always win, and no secret is ever printed. Which items, and where each value lives inside them, is the vault contract — deploy/vault-items.sh, the one place scripts and vault agree: scripts read exactly what it says and fail crisply when the vault doesn't match (never a fuzzy search), and renaming a vault item is a reviewed PR there plus the vault edit, in step.

Each verb is a readable standalone script beside the dispatcher (deploy/ops-*.sh, deploy/doctor.sh) — the outlive-AI rule: no runtime, no machinery only an LLM can drive. The verbs encode the rules that used to live only in prose: publish refuses a HEAD that isn't landed main; there is no dev channel; canary deploys always --pin; canary destroy wipes data the way lchat remove deliberately doesn't (via the canary-gated lchat reset — the sudoers allowlist has no rm); fleet sweeps leave AUTO=pinned rows alone; a tofu plan that would replace the Dublin box refuses to look green. When a verb's advice goes stale, fix it in the script, not in prose — a script that runs drifts loudly, where a document drifts silently.

The standing rule — no cookbooks. Operational procedure lives in these scripts and nowhere else. When a task has no verb, or a verb falls short, extend ops (a new verb, subcommand, or flag in a small PR) and then run it — never hand-run a sequence reconstructed from docs/, CLAUDE.md, CLAUDE.local.md, or an old runbook. The docs are design rationale and history; each time one is caught carrying live procedure, encode the procedure here and banner the doc. This is what keeps operations safe and repeatable for humans and agents alike.

Start with the preflight. ops doctor checks local tooling, the Bitwarden signing key (unlocking to verify by default — --no-unlock prevents it; the session token stays in that one process and is never printed), gh write access to the releases repo, WireGuard + SSH + lchat-operators sudo on every fleet box, and the dev mesh's key/profile/firewall/SSH — printing the fix under each failure. It is read-only, never prints a secret, and exits non-zero on hard failures only, so it works as a gate in other scripts.

Two obligations the surface reminds you of, because they are easy to forget: box-SSH verbs (ship, canary, dev deploy, doctor's fleet section — use --no-ssh to avoid it) incur a why-entry in the public access ledger (web/status/ssh/), which ops ledger drafts; and box aliases come from your ~/.ssh/config (defaults in the scripts, overrides via LCHAT_BOXES / LCHAT_CANARY_BOX / LCHAT_DEV_BOX_*, environment specifics in CLAUDE.local.md).

🤖 Single-machine bootstrap

Running the messenger behind the lchat router on one box (see docs/instance-router.md). The Go side is a single static binary — no runtime.

# 1. copy the binary, then let the binary provision the box
scp lchat root@<box>:/opt/lchat/lchat
ssh root@<box> '/opt/lchat/lchat bootstrap'   # writes units + dirs, starts the router

# 2. add an instance (mints token + env file, starts it, registers + reloads)
ssh root@<box> '/opt/lchat/lchat add-instance chat.example.com'

# 3. operator setup — the last thing the root session does before retiring itself:
#    create a named account per operator (groups lchat-operators + systemd-journal),
#    install deploy/sudoers-operators -> /etc/sudoers.d/lchat-operators (0440)
#    and deploy/lchat-env-install + deploy/lchat-svc -> /usr/local/sbin/ (0755),
#    add each operator's WireGuard peer, close public 22, and run
#    `lchat bootstrap --ssh-noroot` — writes the PermitRootLogin-no drop-in
#    (/etc/ssh/sshd_config.d/10-lchat-noroot.conf), validates, reloads sshd,
#    and confirms the EFFECTIVE config reports no (sshd is first-match-wins;
#    a pre-existing drop-in sorting before 10- silently wins otherwise).
#    It refuses until lchat-operators has members, and every later full
#    bootstrap converges the same drop-in — a box that missed the step picks
#    it up on the next provision pass, root shell not required.

lchat bootstrap embeds the unit files (source deploy/systemd/) and writes them, creates /data + /run/lchat, and starts the router. It also installs the public status collectors — the SSH access ledger + fleet hardening posture scripts (deploy/{ssh-ledger-collect,hardening-posture}.sh), their four systemd units, and the static viewer pages (web/status/*.html + status.css, into /opt/lchat/status-pages/) — and enables their 5-minute timers. These ride the binary because a hardened box has no root shell to scp them onto (operator SSH is WireGuard-only; /opt/lchat + /etc/systemd/system are root-owned). To refresh just the collectors + pages on a live box without touching the router, run sudo -n /opt/lchat/lchat bootstrap --collectors-only (allowlisted by deploy/sudoers-operators) — no member-facing blip. The collector-generated JSON (fleet/ssh/hardening/… data) stays mutable on the box; only the .html + .css viewer assets are embedded. To install by hand instead: scp deploy/systemd/*.service to /etc/systemd/system/, drop a /etc/tmpfiles.d/lchat.conf (d /run/lchat 0710 root root -), daemon-reload, systemctl enable --now lchat-router.

Root SSH is for first bootstrap only. After step 3, operators are named users over the WireGuard tunnel, and root capability exists only through the committed sudoers allowlist (deploy/sudoers-operators) — anything new that needs root gets an allowlist line via a reviewed PR, never a root shell. The full recipe (firewall, WireGuard peer configs, break-glass via the provider's rescue console, the deadman revert pattern): docs/instance-router.md § Hardening.

• lchat-router.service — the front door (binds :80/:443, autocert HTTP-01).
• lchat@.service — an instance template; runs one instance on /run/lchat/<host>.sock, reading its token + config from /data/<host>.env.
• lchat-bot@.service — the optional per-instance Node sidecar (see Bots).
• Runs as root here so it can bind :80/:443; for a non-root user add AmbientCapabilities=CAP_NET_BIND_SERVICE to the router unit and a dedicated User=/Group= with matching ownership on /run/lchat + /data.

Bots (sidecars)

The sidecar is the same Go binary as the instance — lchat bot run — so a box that runs the server runs its bots with no extra install (no Node runtime). The lchat-bot@.service template execs the instance's own versioned live symlink (ExecStart=/opt/lchat/live/%i bot run), so a bot always runs the same version as its app, and deploy/update restart it automatically when they move the instance (docs/instance-router.md § Bots, § Auto-updating).

# per instance: mint the three agent invites + start the sidecar
# (API key via env, not a flag, so it doesn't leak into ps/shell history)
ssh root@<box> 'ANTHROPIC_API_KEY=sk-ant-... /opt/lchat/lchat add-bot chat.example.com'

add-bot mints an agent invite per identity (Claude / Eleven) via the instance's admin API, writes /data/<host>-bot.env (invites + LCHAT_BASE + LCHAT_*_SESSION paths + optional ANTHROPIC_API_KEY), and enables lchat-bot@<host> — which reads all of that from the env file, so there's nothing per-host to configure in the unit. Omit ANTHROPIC_API_KEY for a deterministic-only bot (Eleven still welcomes new members + answers slash commands). Start the instance before the sidecar so Eleven baselines the roster correctly.

> The Node reference sidecar (agent/) is kept on purpose as a runnable example; > self-hosters who prefer it still need a Node runtime + npm ci in /opt/lchat/agent > and point the unit's ExecStart at node /opt/lchat/agent/agent.mjs. The default > deployment is Go.