amadan
deploy: `./deploy/ops` — the operator command surface (supersedes #1190) (#1217)
* deploy: doctor.sh — the read-only operator preflight (ported from #1190)
Ported verbatim from the worktree-ops-doctor branch, then grown per its
handoff plan: the dev section now actually opens SSH to both dev boxes
(one-attempt classification, same as the fleet section — a timeout names
the single-/32 security group rather than reading as a dead box), and the
header frames doctor as verb one of the ./deploy/ops surface.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VZSkXqyaYHXEPaKzvUCchi
* deploy: ./deploy/ops — one command surface that encodes the operating knowledge
The dispatcher routes to per-verb standalone scripts (the outlive-AI rule:
each is readable in a sitting, no runtime). Bare `ops` lists the verbs.
What the verbs encode that previously lived only in prose:
- publish refuses a HEAD origin/main doesn't contain (edge tracks main; a
branch build must never become the fleet's edge by accident) — branch
builds belong on a pinned canary instead
- promote knows there are exactly three channels and says so when someone
reaches for the nonexistent 'dev' one
- ship converges EVERY fleet box, not whichever one you remembered, and
keeps sweeping past a CDN-lagged box before reporting
- canary create always flags --canary (the deliberate signal lchat reset
gates on), deploy always --pin (or the hourly auto-updater moves the
instance out from under the session), destroy wipes data the way the
hardened sudoers allowlist actually permits (canary-gated reset archives
to /data/trash; remove --stop drops route + sidecar units) and the
hostnames are structurally test-<slug> only
- dev plan refuses to look green when tofu would REPLACE the Dublin box
(compute.tf's most_recent AMI with no lifecycle guard); dev deploy
builds the right arch per box, stages beside the live binary, swaps
once, parks a .bak, and health-checks the running stamp
- ledger drafts the public SSH-access why-entry every box session owes
deploy/release.sh is deleted: since #1192 release-server.sh pulls and
unlocks the Bitwarden key itself, and release.sh's own extraction looked
for a 'credential' field the item doesn't have — redundant and drifted,
which is exactly the sprawl this surface exists to end.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VZSkXqyaYHXEPaKzvUCchi
* docs: point the operating docs at ./deploy/ops
CLAUDE.md's deploy principles open with the surface (and the per-session
canary section now teaches the encoded path — including correcting the
pre-hardening teardown advice: there is no allowlisted rm; destroy is the
canary-gated reset + remove --stop). deploy/README.md grows § The operator
surface; server-releases.md and the dev cutover runbook name their ops
verbs; the fleet guide's manual-deploy walkthrough notes the verbs that
now run it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VZSkXqyaYHXEPaKzvUCchi
* deploy: doctor classifies 'Host key verification failed' as its own case
Found live on a fresh machine: BatchMode can't accept a first-seen host
key, so the box reads as a bare raw-error FAIL with no fix hint — when in
fact it was reachable and the fix is a one-off interactive ssh. Both the
fleet and dev sections gain the case, distinct from unreachable/refused.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VZSkXqyaYHXEPaKzvUCchi
* deploy: ops dev sorts its own credentials out, and gains rebuild
ops dev plan/rebuild now pull AWS_* + DNSIMPLE_* from the operator's
per-operator dev agent item in Bitwarden (found by search — each operator
has their own aws-dev-agent-<name> identity; LCHAT_DEV_CREDS_ITEM pins one).
The release-server.sh shape throughout: environment values always win, the
unlock is interactive-only (no tty degrades to instructions, never a hang,
stderr stays attached — the doctor lesson), exports are allowlisted to the
five known names so a stray line in the item can never reach the
environment, and no value is ever printed. Doctor's dev section now treats
a standing aws profile as a convenience, not a prerequisite.
ops dev rebuild is the deliberate key rotation: EC2 injects the keypair
only at first boot, so replacing the Dublin box is how you take over a box
whose key you don't hold. It generates the per-machine key if absent,
refuses until compute.tf's committed public key IS your key and the /32
admits your IP (tofu applies what the file says — a mismatch would build a
box that trusts someone else, or one you can't reach), requires --yes plus
tofu's own plan approval, drops the stale known_hosts entry (same EIP, new
host key), waits for SSH, and provisions binary + `lchat bootstrap`. The
env files + mesh-token pairing stay by hand — secrets don't ride scripts.
Also: build-linux.sh compiles the package at the CURRENT directory, so the
verbs now cd to the repo root before building — rebuild would otherwise
have compiled infra/dev-mesh, and canary/dev deploys built from whatever
directory ops was invoked in.
The retired sops vault's incantation is gone from the runbook, the
dev-mesh README, and main.tf's comment — all now point at the Bitwarden
path the script encodes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VZSkXqyaYHXEPaKzvUCchi
* deploy: ops dev survives bw's stale cache and the split DNSimple item
Two findings from the first live run (james):
- bw reads a LOCAL CACHE of the vault, so an item deleted or deduplicated
in another client is invisible until `bw sync` — the run kept reporting
a duplicate the operator had already removed. Item lookup now syncs and
retries once on any surprising result (none, or more than one) before
believing it, and two items genuinely SHARING a name is called out as
its own error (bw can't fetch by an ambiguous name; the fix is a
vault-side rename/delete, which the retry then picks up).
- The Bitwarden migration split the DNSimple token out of the sops-era
aws-dev-agent bundle into its own item ('lchat DNSimple DNS-01 token
…'). When the agent item doesn't carry DNSIMPLE_* lines, the token is
now fetched from that item (field named DNSIMPLE_TOKEN, any *token*
field, login password, or a single-line note — first hit wins,
LCHAT_DNSIMPLE_ITEM pins the item), and DNSIMPLE_ACCOUNT defaults to
the public 176940 — applied even when a hand-exported token skips
Bitwarden entirely.
The live run also finally exercised the interactive unlock prompt — the
branch #1190 shipped unverified — and it behaves.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VZSkXqyaYHXEPaKzvUCchi
* deploy: the vault contract — exact Bitwarden item names/slots, no searching
Item lookup had drifted heuristic: the DNSimple fetch searched by substring
and cascaded through four possible value locations, and the signing-key item
name was spelled out in two scripts. Heuristics fail quietly wrong — a
renamed item or an extra match picks a plausible value instead of erroring —
where this surface's whole point is failing loudly and precisely.
deploy/vault-items.sh is now the contract: the ONE place scripts and vault
agree on item names and where each value lives (names only — nothing secret).
Scripts read exactly what it says and fail crisply when the vault doesn't
match; renaming a vault item is a reviewed PR there plus the vault edit, in
step (the sudoers-operators rule, applied to the vault).
- The DNSimple token: exact item name, the login PASSWORD slot, nothing
else (confirmed against the real vault item). Cascade deleted. A failed
fetch gets one cache sync + retry, then a contract error.
- The dev agent item stays the one convention entry — per-operator names
are inherent (revocable per-operator IAM identities) — resolved as
'exactly one item with the contract prefix', pinnable.
- release-server.sh and doctor.sh read BW_ITEM_SIGNING_KEY from the same
file. Its >40-char field heuristic is documented in the contract as the
remaining migration (naming the field wants a coordinated vault edit).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VZSkXqyaYHXEPaKzvUCchi
* deploy+infra: per-operator SSH allowlist on the dev mesh; ops dev apply
ssh_ingress_cidr becomes a list(string) — one /32 per operator (paul,
james) — so admitting a second operator no longer means evicting the
first. A security-group rule change applies in place: no instance touch,
no downtime. doctor's IP check and rebuild's preflight now read the whole
allowlist (every a.b.c.d/nn in the variable block; keep literal IPs out
of its description text).
ops dev apply closes the last hand-run seam: same self-serve credential
fetch as plan, same refuse-to-replace guard (shared plan_guard), tofu's
interactive approve still the final gate. Extra args pass through, so
`ops dev apply -target=aws_security_group.ie_node` lands an allowlist
change even while an AMI replacement is pending — and the guard plans
exactly the graph the apply will touch.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VZSkXqyaYHXEPaKzvUCchi
* deploy: name the Dublin key for what it is — the shared box key
'Per-machine key' read as operator-specific and confused the first
operator to hit it. The evidence says otherwise: the key's comment is the
box's name (eleven-dev-box, set at creation), the AWS key pair is
box-scoped (eleven-dev-mesh-ie), and the runbook has every operator use
the same keyfile path — one shared box credential, created at
provisioning, never vaulted, custody with whoever provisioned. Doctor's
hint and ops-dev's comments now say so, and rebuild's key generation
notes the fresh key must be shared onward for the same reason.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VZSkXqyaYHXEPaKzvUCchi
* deploy+docs: no more cookbooks — ops is the procedure, docs are history
The standing rule, now stated where every session reads it (CLAUDE.md's
deploy principles, deploy/README.md, and bare `ops` itself): operational
procedure lives in the ops scripts and nowhere else. A task with no verb
gets the verb ADDED (a small PR) and then run — never a hand-run sequence
reconstructed from docs/, CLAUDE.md, CLAUDE.local.md, or memory. Docs are
design rationale and history; where one is caught carrying live procedure,
encode it and banner the doc. CLAUDE.local.md is re-scoped to personal
preferences only — box addressing rides ~/.ssh/config + the LCHAT_* env
the scripts read, secrets ride the vault contract.
Docs demoted accordingly: the multi-edge cutover runbook and the sops
retirement plan are marked historical records; deploying-instances.md is
re-headed as the mental model behind the verbs; instance-router.md's stale
"proposal, no code yet" status now says shipped-design-rationale;
instance-migration.md is marked not-yet-encoded (build `ops migrate` from
it when next needed, then mark it historical); server-releases.md's
publishing section now shows the ops verbs instead of the hand bw export.
And the one procedure a script still leaned on is encoded: `ops dev
rebuild` now finishes the job instead of pointing at the runbook — writes
/data/router.env + cloud.env from derived facts (bucket via tofu output),
reads the mesh token from the surviving Helsinki box over SSH (never
printed) so the pair matches, mints a fresh cloud admin token, restarts
router/cloud/control, and has Helsinki `mesh-seed` the fresh registry
(meshreplicate.go — built for exactly this catch-up), env sourced
explicitly since sudo strips it. What remains unencoded (dev home server
env, cloud demo/pool extras) is declared IN the verb's output as
extend-me-here, not in a doc.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VZSkXqyaYHXEPaKzvUCchi
---------
Co-authored-by: Claude <noreply@anthropic.com>
21 files changed,
+1440
−104
CLAUDE.md+42 −20deploy/README.md+70 −0deploy/doctor.sh+339 −0deploy/ops+60 −0deploy/ops-canary.sh+141 −0deploy/ops-dev.sh+514 −0deploy/ops-ledger.sh+68 −0deploy/ops-release.sh+101 −0deploy/release-server.sh+6 −2deploy/release.sh+0 −34deploy/vault-items.sh+28 −0docs/deploying-instances.md+12 −10docs/instance-migration.md+7 −1docs/instance-router.md+5 −2docs/multi-edge-dev-cutover.md+12 −5docs/operator-vault-migration.md+4 −0docs/planet-scale-guide/deploys.html+1 −1docs/server-releases.md+12 −17infra/dev-mesh/README.md+7 −5infra/dev-mesh/compute.tf+9 −5infra/dev-mesh/main.tf+2 −2
diff --git a/CLAUDE.md b/CLAUDE.md| index 1468a9b..134ef75 100644 |
| --- a/CLAUDE.md |
| +++ b/CLAUDE.md |
| @@ -574,17 +574,32 @@ are tests, not testing). |
| Deployed as a single static binary per chat. Each chat is a **fully-isolated |
| instance** — its own process, DB, admin token, and subdomain — supervised by |
| systemd behind the instance router (`docs/instance-router.md`); shipping is |
| -via the signed release channels (`deploy/release-server.sh`, |
| -`docs/server-releases.md`). The `Dockerfile` exists for self-hosters on |
| -container platforms (README § Docker); nothing in our production runs in |
| -Docker (the original Miren container path was retired 2026-07-15). |
| - |
| -> Environment specifics — server, platform/cluster name, the instance↔subdomain |
| -> map, DNS, and secret locations — live in **`CLAUDE.local.md`** (gitignored), |
| -> not here. |
| +via the signed release channels (`docs/server-releases.md`). The operator |
| +entry point for all of it is **`./deploy/ops`** (`deploy/README.md § The |
| +operator surface`). The `Dockerfile` exists for self-hosters on container |
| +platforms (README § Docker); nothing in our production runs in Docker (the |
| +original Miren container path was retired 2026-07-15). |
| + |
| +> Box addressing rides `~/.ssh/config` plus the `LCHAT_*` env overrides the |
| +> ops scripts read (defaults live in the scripts themselves); secrets live in |
| +> Bitwarden under the vault contract (`deploy/vault-items.sh`). |
| +> **`CLAUDE.local.md` holds personal preferences** (e.g. the manual-testing |
| +> choice) — never operational procedure. A local file or doc that contradicts |
| +> an ops verb is stale by definition: fix the verb, not the prose. |
| Principles that hold regardless of environment: |
| +- 🤖 **Operations run through `./deploy/ops` — never reconstructed from |
| + prose. NO LLM COOKBOOKS.** Bare `ops` lists the verbs; `ops doctor` is the |
| + read-only preflight that checks every credential and access path an |
| + operator needs and prints the fix under each failure, never a secret. If a |
| + task has no verb, or a verb falls short, **extend ops in a small PR and |
| + then run it** — do not hand-run command sequences gleaned from `docs/`, |
| + this file, `CLAUDE.local.md`, or memory; that path is how snowballing |
| + runbooks happen, and it is closed. The documents under `docs/` are design |
| + rationale and history: where one appears to give operational steps, the |
| + script is the truth and the doc is stale (`deploy/README.md § The operator |
| + surface`). |
| - **Deploys are data-safe.** `lchat deploy` repoints the instance's `live` |
| symlink and restarts — it never touches `/data/<host>.db*`. **Never delete |
| an instance's data to update it**; `lchat remove` deliberately leaves |
| @@ -614,20 +629,27 @@ Instances are disposable — wildcard DNS **and** a wildcard TLS cert cover |
| rate limit. So instead of sharing one `test` instance across parallel |
| sessions (they trample each other): |
| +The encoded path is `./deploy/ops canary create|deploy|destroy <slug>` — |
| +each verb enforces the rules below, so prefer it over hand-run `lchat` |
| +commands. The rules themselves, and why: |
| + |
| - **Every Claude session doing server work creates its own canary**: |
| - `lchat add-instance test-<slug>.11ch.at` on the router box (box specifics |
| - in `CLAUDE.local.md`), slug = your branch/feature. The `test-` prefix keeps |
| - canaries visually distinct from real member spaces (short sqid subdomains). |
| -- **A canary never auto-updates — pin it.** Deploy branch builds with |
| - `lchat deploy test-<slug>.11ch.at --version <ver> --pin`; the hourly |
| - auto-updater leaves pinned instances alone, and `lchat update status` shows |
| - them as `AUTO=pinned`. The session owns its canary's version for its whole |
| - life (`lchat rollback` / re-deploy as you iterate). |
| + `ops canary create <slug>` (box override: `LCHAT_CANARY_BOX`), slug = |
| + your branch/feature. The `test-` prefix keeps canaries visually distinct |
| + from real member spaces (short sqid subdomains); creation also flags the |
| + route `--canary` — the deliberate signal (never inferred from the |
| + hostname) that lets `lchat reset` wipe it at teardown. |
| +- **A canary never auto-updates — pin it.** `ops canary deploy <slug>` |
| + builds the current branch and deploys `--pin`; the hourly auto-updater |
| + leaves pinned instances alone, and `lchat update status` shows them as |
| + `AUTO=pinned`. The session owns its canary's version for its whole life |
| + (`lchat rollback` / re-deploy as you iterate). |
| - **The session tears its canary down** when the work merges: |
| - `lchat remove test-<slug>.11ch.at --stop`, then delete |
| - `/data/test-<slug>.11ch.at.*` (removal alone never deletes data — that's |
| - deliberate for real instances; a canary's data is yours to destroy). If you |
| - added a bot sidecar, disable `lchat-bot@test-<slug>.11ch.at` too. |
| + `ops canary destroy <slug>` — `lchat reset` (archives the DB to |
| + `/data/trash`; the canary flag is its gate) then `lchat remove --stop` |
| + (which also disables bot/hooks sidecar units). Removal alone never deletes |
| + data — that's deliberate for real instances; the canary-gated reset is how |
| + a hardened box (no allowlisted `rm`) destroys a canary's data anyway. |
| - Canaries hold **no real member data** and their admin tokens are **not** |
| stored in 1Password — everything about them dies with the session. |
| - Corollary for fleet-wide operations (channel changes, unpin sweeps, |
diff --git a/deploy/README.md b/deploy/README.md| index f912f52..77ebf22 100644 |
| --- a/deploy/README.md |
| +++ b/deploy/README.md |
| @@ -1,3 +1,73 @@ |
| +# 🤖 The operator surface: `./deploy/ops` |
| + |
| +One command surface for deploying and managing the fleet, so operating doesn't |
| +mean re-gathering knowledge from `CLAUDE.md`, `docs/server-releases.md`, |
| +`docs/multi-edge-dev-cutover.md`, and the vault's `RUNBOOK.md` every time. |
| +Bare `./deploy/ops` lists the verbs: |
| + |
| +```sh |
| +./deploy/ops doctor # can I deploy right now? (read-only preflight) |
| +./deploy/ops publish "notes" # build main -> release -> point edge at it |
| +./deploy/ops promote <ver> beta # move a channel pointer (edge|beta|stable) |
| +./deploy/ops ship # every fleet box catches up NOW |
| +./deploy/ops canary create x # per-session test instance (test-x.11ch.at) |
| +./deploy/ops canary deploy x # build THIS branch onto it — always pinned |
| +./deploy/ops canary destroy x # reset + remove, data archived to trash |
| +./deploy/ops dev plan # dev-mesh tofu plan, AMI-replacement guarded |
| +./deploy/ops dev apply # tofu apply behind the same guard |
| +./deploy/ops dev deploy # dev mesh: build -> scp -> swap -> restart |
| +./deploy/ops dev rebuild # destroy + recreate the Dublin box (new keys) |
| +./deploy/ops ledger "why..." # draft the public SSH-access why-entry |
| +``` |
| + |
| +Verbs that need credentials fetch them themselves (the `release-server.sh` |
| +shape): `publish`/`promote` pull the signing key from Bitwarden, and `ops dev` |
| +pulls `AWS_*` from your per-operator dev agent item and `DNSIMPLE_*` from the |
| +DNSimple token item — unlocking interactively when the vault is locked, |
| +degrading to instructions (never a hang) without a tty, and syncing bw's |
| +local cache when a lookup looks stale. Values already in the environment |
| +always win, and no secret is ever printed. Which items, and where each value |
| +lives inside them, is the **vault contract** — `deploy/vault-items.sh`, the |
| +one place scripts and vault agree: scripts read exactly what it says and fail |
| +crisply when the vault doesn't match (never a fuzzy search), and renaming a |
| +vault item is a reviewed PR there plus the vault edit, in step. |
| + |
| +Each verb is a readable standalone script beside the dispatcher |
| +(`deploy/ops-*.sh`, `deploy/doctor.sh`) — the outlive-AI rule: no runtime, no |
| +machinery only an LLM can drive. The verbs *encode the rules* that used to |
| +live only in prose: publish refuses a HEAD that isn't landed main; there is no |
| +`dev` channel; canary deploys always `--pin`; canary destroy wipes data the |
| +way `lchat remove` deliberately doesn't (via the canary-gated `lchat reset` — |
| +the sudoers allowlist has no `rm`); fleet sweeps leave `AUTO=pinned` rows |
| +alone; a `tofu plan` that would replace the Dublin box refuses to look green. |
| +When a verb's advice goes stale, **fix it in the script, not in prose** — a |
| +script that runs drifts loudly, where a document drifts silently. |
| + |
| +**The standing rule — no cookbooks.** Operational procedure lives in these |
| +scripts and nowhere else. When a task has no verb, or a verb falls short, |
| +**extend ops** (a new verb, subcommand, or flag in a small PR) and then run |
| +it — never hand-run a sequence reconstructed from `docs/`, `CLAUDE.md`, |
| +`CLAUDE.local.md`, or an old runbook. The docs are design rationale and |
| +history; each time one is caught carrying live procedure, encode the |
| +procedure here and banner the doc. This is what keeps operations safe and |
| +repeatable for humans and agents alike. |
| + |
| +Start with the preflight. `ops doctor` checks local tooling, the Bitwarden |
| +signing key (unlocking to verify by default — `--no-unlock` prevents it; the |
| +session token stays in that one process and is never printed), `gh` write |
| +access to the releases repo, WireGuard + SSH + `lchat-operators` sudo on every |
| +fleet box, and the dev mesh's key/profile/firewall/SSH — printing the fix |
| +under each failure. It is read-only, never prints a secret, and exits non-zero |
| +on hard failures only, so it works as a gate in other scripts. |
| + |
| +Two obligations the surface reminds you of, because they are easy to forget: |
| +box-SSH verbs (`ship`, `canary`, `dev deploy`, doctor's `fleet` section — use |
| +`--no-ssh` to avoid it) incur a **why-entry in the public access ledger** |
| +(`web/status/ssh/`), which `ops ledger` drafts; and box aliases come from your |
| +`~/.ssh/config` (defaults in the scripts, overrides via `LCHAT_BOXES` / |
| +`LCHAT_CANARY_BOX` / `LCHAT_DEV_BOX_*`, environment specifics in |
| +`CLAUDE.local.md`). |
| + |
| # 🤖 Single-machine bootstrap |
| Running the messenger behind the `lchat` router on one box (see |
diff --git a/deploy/doctor.sh b/deploy/doctor.sh| new file mode 100755 |
| index 0000000..caf7c87 |
| --- /dev/null |
| +++ b/deploy/doctor.sh |
| @@ -0,0 +1,339 @@ |
| +#!/usr/bin/env sh |
| +# 🤖 doctor — one read-only preflight for every credential and access path an |
| +# operator needs. Answers "can I actually deploy right now?" in one command, |
| +# instead of six documents. Verb one of the ops surface (`./deploy/ops`) — as |
| +# verbs land there, their prerequisites get a check here, so the preflight |
| +# stays honest. |
| +# |
| +# ./deploy/ops doctor # everything (doctor.sh runs standalone too) |
| +# ./deploy/ops doctor release # one section: tools|release|fleet|dev |
| +# ./deploy/ops doctor --no-ssh # skip the box checks (see the note below) |
| +# ./deploy/ops doctor --no-unlock # never prompt (it prompts to unlock by |
| +# # default, so the key check is real) |
| +# |
| +# Why this exists: the knowledge for this was spread across CLAUDE.md, |
| +# CLAUDE.local.md, docs/server-releases.md, docs/multi-edge-dev-cutover.md, the |
| +# vault's RUNBOOK.md, and raw tofu source — and drifted from reality in four |
| +# places at once. Prose drifts silently; a script that runs drifts loudly. Every |
| +# check below is a fact about *this machine right now*, not a claim in a doc. |
| +# |
| +# Two rules this script holds to, because it touches real credentials: |
| +# 1. It NEVER prints a secret. Checks assert reachability/permission and |
| +# discard output — a fetched credential is tested for existence, never |
| +# echoed. The one interactive moment (the Bitwarden unlock) captures the |
| +# session token into a variable and never displays it. |
| +# 2. It NEVER mutates anything — no deploys, no writes, no unlocks. Every |
| +# remote command is a read (`lchat instances`, `id`, `groups`). |
| +# |
| +# NOTE: the `fleet` section opens real operator SSH sessions, which incur a |
| +# why-entry in the public access ledger (web/status/ssh/, RUNBOOK.md § Box |
| +# access). Use --no-ssh when you only want the local picture. |
| +set -u |
| + |
| +section="${1:-all}" |
| +no_ssh=no |
| +unlock=yes |
| +for a in "$@"; do |
| + [ "$a" = "--no-ssh" ] && no_ssh=yes |
| + [ "$a" = "--no-unlock" ] && unlock=no |
| +done |
| +case "$section" in --*) section=all ;; esac |
| + |
| +root="$(CDPATH= cd -- "$(dirname -- "$0")/.." && pwd)" |
| +# The Bitwarden item names checked below — the vault contract. |
| +# shellcheck source=deploy/vault-items.sh |
| +. "$root/deploy/vault-items.sh" |
| +# Only used to point at gen-fleet-conf.sh when a tunnel is down. |
| +VAULT="${LCHAT_VAULT_DIR:-$HOME/elevenmessenger/secrets-vault}" |
| +# Box aliases from ~/.ssh/config; `ssh -G` resolves user+hostname, so the ssh |
| +# config stays the single source of truth for how to reach a box. |
| +BOXES="${LCHAT_BOXES:-elevenbox kepler}" |
| +RELEASES_REPO="${LCHAT_RELEASES_REPO:-elevenmessenger/releases}" |
| + |
| +fails=0 |
| +warns=0 |
| + |
| +# --- output helpers ------------------------------------------------------- |
| +# Two columns: a fixed-width label, then a verdict. Fix hints are indented |
| +# under the failing line so the next action is always adjacent to the problem. |
| +ok() { printf ' \033[32mok\033[0m %s\n' "$1"; } |
| +bad() { printf ' \033[31mFAIL\033[0m %s\n' "$1"; fails=$((fails + 1)); } |
| +warn() { printf ' \033[33mwarn\033[0m %s\n' "$1"; warns=$((warns + 1)); } |
| +fix() { printf ' → %s\n' "$1"; } |
| +head_() { printf '\n\033[1m%s\033[0m\n' "$1"; } |
| +have() { command -v "$1" >/dev/null 2>&1; } |
| + |
| +want() { [ "$section" = all ] || [ "$section" = "$1" ]; } |
| + |
| +# --- tools ---------------------------------------------------------------- |
| +check_tools() { |
| + head_ "Local tooling" |
| + # Each tool is annotated with what breaks without it, so a missing one is |
| + # self-explanatory rather than a bare name. |
| + for pair in \ |
| + "bw:release signing key (Bitwarden)" \ |
| + "gh:publishing to the releases repo" \ |
| + "jq:parsing manifests + bw/gh output" \ |
| + "go:building the release binaries" \ |
| + "tofu:the dev-mesh infrastructure" \ |
| + "aws:the dev-mesh + release S3 mirror" \ |
| + "ssh:reaching every box" |
| + do |
| + t="${pair%%:*}"; why="${pair#*:}" |
| + if have "$t"; then ok "$t"; else |
| + bad "$t — needed for $why" |
| + fix "brew install $t" |
| + fi |
| + done |
| +} |
| + |
| +# --- release publishing --------------------------------------------------- |
| +# Confirm the item is reachable WITHOUT printing it: we only look at whether |
| +# the fetch succeeded, never at the value. |
| +check_signing_item() { |
| + if bw get item "$BW_ITEM_SIGNING_KEY" \ |
| + >/dev/null 2>&1; then |
| + ok "signing key item is present" |
| + else |
| + bad "signing key item not found in this vault" |
| + fix "check it is shared to your org collection (BITWARDEN-MIGRATION.md)" |
| + fi |
| +} |
| + |
| +check_release() { |
| + head_ "Publishing a release" |
| + |
| + if ! have bw; then |
| + bad "bw not installed — cannot reach the signing key" |
| + fix "brew install bitwarden-cli" |
| + else |
| + st="$(bw status 2>/dev/null | tr ',' '\n' | sed -n 's/.*"status":"\([a-z]*\)".*/\1/p')" |
| + case "$st" in |
| + unlocked) |
| + ok "bitwarden unlocked" |
| + check_signing_item ;; |
| + locked) |
| + # Unlock by default: "locked, so I can't tell you" is a non-answer, and |
| + # the whole point is to find out whether the key is actually reachable. |
| + # The session is this process's only — a child cannot export BW_SESSION |
| + # back to the calling shell — so this deepens the check and nothing |
| + # more. Without a tty there is nobody to prompt, so degrade instead of |
| + # hanging; --no-unlock forces that path deliberately. |
| + if [ "$unlock" = yes ] && [ -t 0 ]; then |
| + printf ' unlocking bitwarden to verify the signing key\n' |
| + # Do NOT redirect stderr here: `bw unlock` writes its master-password |
| + # prompt to stderr and the session token to stdout. Swallowing stderr |
| + # hides the prompt, and the script looks hung while it silently waits |
| + # for input. Only stdout is captured. |
| + BW_SESSION="$(bw unlock --raw)" |
| + export BW_SESSION |
| + if [ -n "${BW_SESSION:-}" ]; then |
| + ok "bitwarden unlocked (this run only)" |
| + check_signing_item |
| + else |
| + bad "unlock failed — signing key not verified" |
| + fix "wrong master password, or run 'bw unlock' by hand to see why" |
| + fi |
| + else |
| + warn "bitwarden is locked — signing key not verified" |
| + fix 'export BW_SESSION=$(bw unlock --raw) # then re-run' |
| + fi ;; |
| + *) |
| + bad "bitwarden not logged in" |
| + fix "bw login # EU vault: https://vault.bitwarden.eu" ;; |
| + esac |
| + fi |
| + |
| + if have gh && gh auth status >/dev/null 2>&1; then |
| + ok "gh authenticated" |
| + push="$(gh api "repos/$RELEASES_REPO" --jq .permissions.push 2>/dev/null)" |
| + if [ "$push" = "true" ]; then ok "write access to $RELEASES_REPO"; else |
| + bad "no write access to $RELEASES_REPO" |
| + fix "ask for push rights on the releases repo" |
| + fi |
| + else |
| + bad "gh not authenticated" |
| + fix "gh auth login" |
| + fi |
| + |
| + # release-server.sh refuses to build from a dirty tree; surface that here |
| + # rather than 20 minutes into a release. |
| + if [ -n "$(git -C "$root" status --porcelain 2>/dev/null)" ]; then |
| + warn "working tree is dirty — release-server.sh will refuse to publish" |
| + fix "commit or stash, and publish from a clean main" |
| + else |
| + ok "working tree is clean" |
| + fi |
| +} |
| + |
| +# --- fleet boxes ---------------------------------------------------------- |
| +check_fleet() { |
| + head_ "Fleet boxes (WireGuard + SSH + sudo)" |
| + |
| + if [ "$no_ssh" = yes ]; then |
| + warn "skipped (--no-ssh)" |
| + return |
| + fi |
| + |
| + for box in $BOXES; do |
| + # ssh -G resolves the config the way ssh itself will, so we test what will |
| + # actually happen rather than re-parsing ~/.ssh/config by hand. |
| + hostname="$(ssh -G "$box" 2>/dev/null | sed -n 's/^hostname //p' | head -1)" |
| + user="$(ssh -G "$box" 2>/dev/null | sed -n 's/^user //p' | head -1)" |
| + # One connection attempt tells us both things. A tunnel that is down and a |
| + # key that is refused look identical from a bare exit code, so classify the |
| + # error text instead — and skip ping, whose -W flag means milliseconds on |
| + # macOS and seconds on Linux. |
| + err="$(ssh -o ConnectTimeout=8 -o BatchMode=yes "$box" true 2>&1)" |
| + if [ -z "$err" ]; then |
| + ok "$box — tunnel up, ssh as '$user' ($hostname)" |
| + else |
| + case "$err" in |
| + *"Permission denied"*) |
| + bad "$box — reachable, but ssh as '$user' was refused" |
| + fix "confirm which key the box authorises; a key not at a default" |
| + fix "path needs IdentityFile + IdentitiesOnly in ~/.ssh/config" ;; |
| + *"Host key verification failed"*) |
| + # The box answered — BatchMode just can't accept a first-seen host |
| + # key. Distinct from unreachable/refused, and self-serve fixable. |
| + bad "$box — reachable, but its host key isn't in known_hosts yet" |
| + fix "one-off interactive: ssh $box # accept the host key, then re-run" ;; |
| + *"Could not resolve hostname"*) |
| + # ssh -G echoes an unknown alias back rather than failing, so an |
| + # absent Host block surfaces here as a resolve error. |
| + bad "$box — no Host block in ~/.ssh/config (or DNS failure)" |
| + fix "add a Host block for $box, or set LCHAT_BOXES" ;; |
| + *"timed out"*|*"No route"*|*"Network is unreachable"*) |
| + bad "$box — $hostname unreachable (WireGuard down?)" |
| + fix "bring the tunnel up, then re-run; regenerate it with" |
| + fix "cd $VAULT && ./gen-fleet-conf.sh > ~/Downloads/fleet.conf" ;; |
| + *) |
| + bad "$box — $(printf '%s' "$err" | head -1)" ;; |
| + esac |
| + continue |
| + fi |
| + |
| + # lchat-operators is what the sudoers allowlist keys off; without it the |
| + # ssh login succeeds but every privileged command fails. |
| + if ssh -o BatchMode=yes "$box" 'id -nG' 2>/dev/null \ |
| + | tr ' ' '\n' | grep -qx lchat-operators; then |
| + ok "$box — in lchat-operators" |
| + elif [ "$user" = root ]; then |
| + ok "$box — root (allowlist not applicable)" |
| + else |
| + bad "$box — not in lchat-operators" |
| + fix "ask an operator to add you to the group" |
| + fi |
| + |
| + if ssh -o BatchMode=yes "$box" \ |
| + 'sudo -n /opt/lchat/lchat instances' >/dev/null 2>&1; then |
| + ok "$box — sudo allowlist works (can deploy)" |
| + else |
| + bad "$box — 'sudo -n lchat' denied" |
| + fix "check /etc/sudoers.d/lchat-operators (deploy/sudoers-operators)" |
| + fi |
| + done |
| +} |
| + |
| +# --- dev mesh ------------------------------------------------------------- |
| +check_dev() { |
| + head_ "Dev mesh (Dublin + Helsinki)" |
| + |
| + # The Dublin box authorises exactly one key, committed in compute.tf: the |
| + # SHARED box key (its comment is the box's name, not a person's) — created |
| + # at provisioning, never vaulted, so copies exist only on operator machines. |
| + keyfile="$HOME/.ssh/eleven-dev-box" |
| + if [ -f "$keyfile" ]; then |
| + ok "eleven-dev-box key present" |
| + else |
| + bad "no $keyfile" |
| + fix "the shared dev-box key — not in any vault; ask an operator who holds it" |
| + fi |
| + |
| + # What `ops dev deploy` actually needs: a working SSH session to each box. |
| + # Same one-attempt classification as the fleet section — and here a timeout |
| + # is USUALLY the single-/32 security group (named by the CIDR check below), |
| + # not a dead box. |
| + if [ "$no_ssh" = yes ]; then |
| + warn "dev box ssh checks skipped (--no-ssh)" |
| + else |
| + for spec in "ie:${LCHAT_DEV_BOX_IE:-ubuntu@54.154.95.205}" \ |
| + "fi:${LCHAT_DEV_BOX_FI:-root@157.180.123.183}"; do |
| + node="${spec%%:*}"; dest="${spec#*:}" |
| + if [ "$node" = ie ] && [ -f "$keyfile" ]; then |
| + err="$(ssh -i "$keyfile" -o ConnectTimeout=8 -o BatchMode=yes "$dest" true 2>&1)" |
| + else |
| + err="$(ssh -o ConnectTimeout=8 -o BatchMode=yes "$dest" true 2>&1)" |
| + fi |
| + if [ -z "$err" ]; then |
| + ok "dev $node — ssh works ($dest)" |
| + else |
| + case "$err" in |
| + *"Permission denied"*) |
| + bad "dev $node — reachable, but ssh as '$dest' was refused" |
| + fix "ie wants ~/.ssh/eleven-dev-box; fi wants the 1Password hetzner key" ;; |
| + *"Host key verification failed"*) |
| + bad "dev $node — reachable, but its host key isn't in known_hosts yet" |
| + fix "one-off interactive: ssh $dest # accept the host key, then re-run" ;; |
| + *"timed out"*|*"No route"*|*"Network is unreachable"*) |
| + bad "dev $node — unreachable; if the IP check below fails, that is why" |
| + fix "your IP must be in the security group's allowlist (ssh_ingress_cidr)" ;; |
| + *) |
| + bad "dev $node — $(printf '%s' "$err" | head -1)" ;; |
| + esac |
| + fi |
| + done |
| + fi |
| + |
| + # ops dev pulls AWS_*/DNSIMPLE_* from the per-operator dev agent item in |
| + # Bitwarden itself, so a standing profile is a convenience, not a |
| + # prerequisite — only "no route to any credential" is a failure. |
| + if [ -n "${AWS_ACCESS_KEY_ID:-}" ]; then |
| + ok "AWS credentials in the environment" |
| + elif aws configure list-profiles 2>/dev/null | grep -qx eleven-dev; then |
| + ok "aws profile 'eleven-dev' configured" |
| + elif have bw; then |
| + warn "no standing AWS auth — ops dev will pull the dev agent creds from Bitwarden per run" |
| + else |
| + bad "no AWS auth at all (no env creds, no 'eleven-dev' profile, no bw)" |
| + fix "install bw, or seed the profile from your dev agent item (docs/secrets.md § AWS)" |
| + fi |
| + |
| + # SSH ingress is a short per-operator /32 allowlist (compute.tf, |
| + # ssh_ingress_cidr), so being outside it looks exactly like the box being |
| + # down. Say which it is. Anchored to the variable block — every a.b.c.d/nn |
| + # inside it counts, so keep literal IPs out of its description text. |
| + cidrs="$(awk '/variable "ssh_ingress_cidr"/,/^}/' \ |
| + "$root/infra/dev-mesh/compute.tf" 2>/dev/null \ |
| + | grep -oE '[0-9]+(\.[0-9]+){3}/[0-9]+')" |
| + if [ -n "$cidrs" ]; then |
| + mine="$(curl -s -m5 https://api.ipify.org 2>/dev/null)" |
| + if [ -n "$mine" ] && printf '%s\n' "$cidrs" | sed 's|/.*||' | grep -qxF "$mine"; then |
| + ok "your IP ($mine) is in the dev security group's allowlist" |
| + else |
| + bad "your IP (${mine:-unknown}) is not in [$(printf '%s' "$cidrs" | tr '\n' ' ')] — SSH will hang, not refuse" |
| + fix "add your /32 to var.ssh_ingress_cidr (a PR), then: ops dev apply" |
| + fix "removing someone else's /32 revokes their access — only with their say-so" |
| + fi |
| + fi |
| +} |
| + |
| +# --- run ------------------------------------------------------------------ |
| +printf '\033[1mlchat operator preflight\033[0m — read-only; no secrets printed\n' |
| + |
| +want tools && check_tools |
| +want release && check_release |
| +want fleet && check_fleet |
| +want dev && check_dev |
| + |
| +printf '\n' |
| +if [ "$fails" -eq 0 ] && [ "$warns" -eq 0 ]; then |
| + printf '\033[32mAll checks passed.\033[0m\n' |
| +elif [ "$fails" -eq 0 ]; then |
| + printf '\033[33m%s warning(s), no failures.\033[0m\n' "$warns" |
| +else |
| + printf '\033[31m%s failure(s), %s warning(s).\033[0m\n' "$fails" "$warns" |
| +fi |
| +# Non-zero on hard failures only, so this is usable as a gate in other scripts |
| +# while warnings (a locked vault, a dirty tree) stay informational. |
| +[ "$fails" -eq 0 ] |
diff --git a/deploy/ops b/deploy/ops| new file mode 100755 |
| index 0000000..22e1e95 |
| --- /dev/null |
| +++ b/deploy/ops |
| @@ -0,0 +1,60 @@ |
| +#!/usr/bin/env sh |
| +# 🤖 ops — the one operator command surface for deploying and managing the |
| +# fleet. Every verb encodes knowledge that used to live only in prose spread |
| +# across CLAUDE.md, docs/server-releases.md, docs/multi-edge-dev-cutover.md and |
| +# the operator runbook — prose drifts silently; a script that runs drifts |
| +# loudly. When a verb's advice goes stale, fix it IN THE SCRIPT, not in a doc. |
| +# |
| +# This file is deliberately nothing but a signpost: each verb stays a readable |
| +# standalone script beside it (the outlive-AI rule — a human can open any one |
| +# of them and understand it in a sitting), and this dispatcher only routes. |
| +# Add a verb = add a script + one case arm + one usage line. |
| +set -eu |
| + |
| +here="$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)" |
| + |
| +usage() { |
| + cat <<'EOF' |
| +ops — the operator command surface |
| + |
| + ./deploy/ops doctor [section] can I deploy right now? (read-only preflight) |
| + ./deploy/ops publish "notes" build main -> release -> point edge at it |
| + ./deploy/ops promote <ver> <chan> move beta/stable to a published version |
| + ./deploy/ops ship [box...] make every box's instances catch up NOW |
| + ./deploy/ops canary <create|deploy|destroy|list> [slug] |
| + per-session test instances (test-<slug>.11ch.at) |
| + ./deploy/ops dev <plan|apply|deploy|rebuild> |
| + the two-node dev mesh (tofu / scp+stage+swap) |
| + ./deploy/ops ledger "why..." draft the public SSH-access why-entry |
| + |
| +Start with `ops doctor` — it checks every credential and access path the other |
| +verbs need, and prints the fix under each failure. Verbs that open box SSH |
| +sessions (ship, canary, dev deploy) incur a why-entry in the public access |
| +ledger; `ops ledger` drafts it. |
| + |
| +Env overrides (defaults live in the verb scripts): |
| + LCHAT_BOXES fleet box ssh aliases (doctor, ship) |
| + LCHAT_CANARY_BOX the router box canaries live on |
| + LCHAT_RELEASES_REPO the public releases repo |
| + |
| +No verb for your task? EXTEND ops — a new verb or flag in a small PR |
| +(deploy/README.md § The operator surface) — never a hand-run sequence |
| +reconstructed from docs or runbooks. |
| +EOF |
| +} |
| + |
| +verb="${1:-}" |
| +[ $# -gt 0 ] && shift |
| +case "$verb" in |
| + doctor) exec "$here/doctor.sh" "$@" ;; |
| + publish|promote|ship) |
| + exec "$here/ops-release.sh" "$verb" "$@" ;; |
| + canary) exec "$here/ops-canary.sh" "$@" ;; |
| + dev) exec "$here/ops-dev.sh" "$@" ;; |
| + ledger) exec "$here/ops-ledger.sh" "$@" ;; |
| + ""|-h|--help|help) usage ;; |
| + *) |
| + printf 'ops: unknown verb %s\n\n' "$verb" >&2 |
| + usage >&2 |
| + exit 2 ;; |
| +esac |
diff --git a/deploy/ops-canary.sh b/deploy/ops-canary.sh| new file mode 100755 |
| index 0000000..1ebbfe6 |
| --- /dev/null |
| +++ b/deploy/ops-canary.sh |
| @@ -0,0 +1,141 @@ |
| +#!/usr/bin/env sh |
| +# 🤖 ops canary — per-session test instances (CLAUDE.md § Per-session canaries), |
| +# with the rules that used to live only in prose encoded as behaviour: |
| +# |
| +# ./deploy/ops canary create <slug> # test-<slug>.11ch.at, flagged --canary |
| +# ./deploy/ops canary deploy <slug> # build THIS branch -> stage -> deploy --pin |
| +# ./deploy/ops canary destroy <slug> # reset (archive DB) + remove --stop |
| +# ./deploy/ops canary list # every canary + pinned row on the box |
| +# |
| +# The rules, and where each is enforced: |
| +# - Canaries are test-<slug> on the wildcard domain — no DNS step, no cert |
| +# wait, visually distinct from real member spaces (this script only ever |
| +# forms test-<slug>.<domain> hostnames). |
| +# - Creation passes --canary: the registry flag is a DELIBERATE signal, never |
| +# inferred from the hostname — it is what lets `lchat reset` wipe the DB at |
| +# destroy time while refusing to touch any real instance. |
| +# - Deploys always --pin. An unpinned canary is moved onto the channel build |
| +# by the hourly auto-updater — out from under the session, silently. |
| +# - Destroy wipes the data. `lchat remove` deliberately leaves /data behind |
| +# (right for real instances, wrong for a canary), and the hardened sudoers |
| +# allowlist has no `rm` — so the encoded path is `lchat reset` (archives |
| +# the DB to /data/trash, canary-gated) then `remove --stop` (which also |
| +# disables any bot/hooks sidecar units). |
| +# - Admin tokens for canaries are stored NOWHERE (not 1Password) on purpose: |
| +# everything about a canary dies with its session. |
| +# - Fleet-wide operations skip AUTO=pinned rows — each is someone's live |
| +# canary. `list` shows them so a sweep can check first. |
| +set -eu |
| + |
| +here="$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)" |
| +root="$(CDPATH= cd -- "$here/.." && pwd)" |
| +# The router box canaries live on — an ssh alias; user/hostname come from |
| +# ~/.ssh/config over WireGuard (box specifics in CLAUDE.local.md). |
| +BOX="${LCHAT_CANARY_BOX:-${LCHAT_BOX:-elevenbox}}" |
| +DOMAIN="${LCHAT_CANARY_DOMAIN:-11ch.at}" |
| +LCHAT=/opt/lchat/lchat |
| + |
| +usage() { |
| + echo "usage: ops canary create|deploy|destroy <slug> | ops canary list" >&2 |
| + echo " slug: lowercase letters/digits/hyphens — your branch or feature name" >&2 |
| + exit 2 |
| +} |
| + |
| +cmd="${1:-}"; slug="${2:-}" |
| + |
| +host_for() { |
| + # Everything below derives the hostname from here, so this script is |
| + # structurally incapable of touching a non-test- host. |
| + case "$1" in |
| + *[!a-z0-9-]*|-*|"") echo "ops canary: bad slug '$1' (lowercase letters/digits/hyphens)" >&2; exit 2 ;; |
| + esac |
| + echo "test-$1.$DOMAIN" |
| +} |
| + |
| +case "$cmd" in |
| + |
| +create) |
| + host="$(host_for "$slug")" |
| + echo "ops canary: creating $host on $BOX (flagged --canary)" |
| + ssh "$BOX" "sudo -n $LCHAT add-instance --canary '$host'" |
| + cat <<EOF |
| + |
| +Canary up. The admin/setup link above is stored nowhere else — canary tokens |
| +deliberately die with the session (don't vault them). |
| + next: ./deploy/ops canary deploy $slug # put THIS branch's build on it |
| + tear down: ./deploy/ops canary destroy $slug # when the work merges |
| +EOF |
| + ;; |
| + |
| +deploy) |
| + host="$(host_for "$slug")" |
| + ver="$(git -C "$root" rev-parse --short HEAD)" |
| + if ! git -C "$root" diff --quiet || ! git -C "$root" diff --cached --quiet; then |
| + ver="${ver}-dirty" |
| + echo "ops canary: working tree has uncommitted changes — building anyway ($ver)" >&2 |
| + fi |
| + |
| + echo "ops canary: building $ver for linux/amd64" |
| + out="$(mktemp -t lchat-canary.XXXXXX)" |
| + trap 'rm -f "$out"' EXIT |
| + cd "$root" # build-linux.sh compiles the package at the current directory |
| + OUT="$out" "$here/build-linux.sh" "$ver" |
| + |
| + # A per-slug upload name so parallel sessions' uploads never trample each |
| + # other in the shared incoming dir (a fixed name is a race). |
| + up="/opt/lchat/incoming/lchat-upload-$slug" |
| + ssh "$BOX" 'mkdir -p /opt/lchat/incoming' |
| + scp -q "$out" "$BOX:$up" |
| + staged="$(ssh "$BOX" "sudo -n $LCHAT stage -file '$up' && rm -f '$up'")" |
| + echo " $staged" |
| + staged_ver="$(echo "$staged" | awk '{print $2}')" |
| + [ "$staged_ver" = "$ver" ] || { |
| + echo "ops canary: staged '$staged_ver' != built '$ver' — aborting" >&2; exit 1; } |
| + |
| + # --pin is not optional: without it the hourly auto-updater moves the canary |
| + # back onto its channel build, out from under the session. deploy itself |
| + # health-checks /api/version and auto-rolls back a bad build. |
| + ssh "$BOX" "sudo -n $LCHAT deploy '$host' --version '$ver' --pin" |
| + cat <<EOF |
| + |
| +$host is running $ver (pinned — the auto-updater will leave it alone). |
| + iterate: commit, then ./deploy/ops canary deploy $slug |
| + step back: ssh $BOX sudo -n $LCHAT rollback $host |
| +EOF |
| + ;; |
| + |
| +destroy) |
| + host="$(host_for "$slug")" |
| + echo "ops canary: destroying $host on $BOX" |
| + # reset first (it needs the registry row remove deletes): archives the DB + |
| + # blobs to /data/trash — canary-gated, so a host that was never flagged |
| + # --canary refuses here rather than wiping. That refusal is the safety |
| + # model working; flag it by hand only if it is genuinely a throwaway. |
| + ssh "$BOX" "sudo -n $LCHAT reset '$host' --yes" || { |
| + echo "ops canary: reset refused — if $host is genuinely a throwaway, flag it:" >&2 |
| + echo " ssh $BOX sudo -n $LCHAT canary $host --on # then re-run destroy" >&2 |
| + exit 1 |
| + } |
| + # remove --stop: drops the route (and in.<host>), stops + disables the |
| + # instance unit and any bot/hooks sidecar units. |
| + ssh "$BOX" "sudo -n $LCHAT remove '$host' --stop" |
| + cat <<EOF |
| + |
| +$host destroyed. Its DB is archived in /data/trash (the reaper's trash purge |
| +deletes it for good); the leftover env file is harmless and is overwritten if |
| +the slug is ever recreated. |
| +Remember the session's why-entry if you haven't yet: ./deploy/ops ledger "..." |
| +EOF |
| + ;; |
| + |
| +list) |
| + echo "canaries + pinned rows on $BOX (a pinned row is someone's live canary —" |
| + echo "fleet-wide operations must leave it alone):" |
| + echo |
| + ssh "$BOX" "sudo -n $LCHAT instances" | awk 'NR==1 || /test-|canary/' |
| + echo |
| + ssh "$BOX" "sudo -n $LCHAT update status" | awk 'NR==1 || /pinned/' |
| + ;; |
| + |
| +*) usage ;; |
| +esac |
diff --git a/deploy/ops-dev.sh b/deploy/ops-dev.sh| new file mode 100755 |
| index 0000000..0171e0d |
| --- /dev/null |
| +++ b/deploy/ops-dev.sh |
| @@ -0,0 +1,514 @@ |
| +#!/usr/bin/env sh |
| +# 🤖 ops dev — the two-node dev mesh (Dublin + Helsinki; docs/multi-edge.md, |
| +# runbook docs/multi-edge-dev-cutover.md). |
| +# |
| +# ./deploy/ops dev plan # tofu plan, with the AMI-replacement guard |
| +# ./deploy/ops dev deploy [ie|fi] # build -> scp -> swap -> restart (both by default) |
| +# ./deploy/ops dev deploy --stage-only # stop after scp (pair the swap with env edits) |
| +# ./deploy/ops dev rebuild --yes # destroy + recreate the Dublin box (new keys) |
| +# |
| +# What this encodes that prose used to carry: |
| +# - The dev mesh is NOT a release channel. Nothing here touches the signed |
| +# feed; deploying is scp + stage + swap, by hand, on purpose (the mesh runs |
| +# unmerged code the fleet updater must never see). |
| +# - The verb sorts its own credentials out (the release-server.sh shape): |
| +# values already in the environment win; otherwise Bitwarden fills the |
| +# gaps — AWS_* from the per-operator dev agent item (docs/secrets.md |
| +# § AWS), and DNSIMPLE_* from that item too when it carries them, else |
| +# from the standalone DNSimple token item the migration split out. |
| +# Unlocks interactively when the vault is locked; no tty -> instructions, |
| +# never a hang; values are exported for this process only, never printed. |
| +# - Dublin is arm64 (t4g), Helsinki amd64 — build both, ship the right one. |
| +# - The binary is staged BESIDE the live one and swapped in a separate step, |
| +# so router+cloud restart exactly once, and the previous binary is parked |
| +# as a .bak for one-command rollback. |
| +# - Dublin runs router + cloud + control (restart all three); Helsinki runs |
| +# only the router. |
| +# - compute.tf's aws_ami is `most_recent = true` with no lifecycle guard: the |
| +# day Canonical publishes a newer image, a plain `tofu apply` DESTROYS AND |
| +# RECREATES the Dublin box (its /data dies with it; only the EIP survives). |
| +# `plan` refuses to look green when that is pending; `rebuild` is the |
| +# deliberate version of the same event. |
| +# - The dev security group admits SSH only from the per-operator /32 |
| +# allowlist (compute.tf ssh_ingress_cidr), so being outside it looks |
| +# exactly like a dead box (`ops doctor dev` names which it is). Adding |
| +# yourself is a PR + `ops dev apply` (an in-place SG update); removing an |
| +# entry revokes that operator — only with their say-so. |
| +# - EC2 injects the SSH keypair ONLY at first boot — which is why `rebuild` |
| +# exists: swapping the key pair in compute.tf does nothing to a running |
| +# box, and replacing the box is the honest way to rotate a key you don't |
| +# hold. Helsinki is hand-provisioned Hetzner and cannot be rebuilt from |
| +# here (only its scoped S3 IAM user lives in the tofu). |
| +set -eu |
| + |
| +here="$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)" |
| +root="$(CDPATH= cd -- "$here/.." && pwd)" |
| +# The Bitwarden item names + value locations this script reads — a contract, |
| +# not a search (deploy/vault-items.sh). |
| +# shellcheck source=deploy/vault-items.sh |
| +. "$here/vault-items.sh" |
| + |
| +# The boxes, as committed in docs/multi-edge-dev-cutover.md. The Dublin key is |
| +# the SHARED box key (comment `eleven-dev-box` — box-named, not personal), |
| +# never vaulted: copies live only on operator machines. ssh falls back to |
| +# ~/.ssh/config if it's absent. |
| +BOX_IE="${LCHAT_DEV_BOX_IE:-ubuntu@54.154.95.205}" |
| +BOX_FI="${LCHAT_DEV_BOX_FI:-root@157.180.123.183}" |
| +KEY_IE="$HOME/.ssh/eleven-dev-box" |
| +AWS_PROFILE="${LCHAT_DEV_AWS_PROFILE:-eleven-dev}" |
| + |
| +ssh_ie() { if [ -f "$KEY_IE" ]; then ssh -i "$KEY_IE" -o ConnectTimeout=10 "$@"; else ssh -o ConnectTimeout=10 "$@"; fi; } |
| +scp_ie() { if [ -f "$KEY_IE" ]; then scp -i "$KEY_IE" "$@"; else scp "$@"; fi; } |
| + |
| +usage() { |
| + echo "usage: ops dev plan [tofu args] | ops dev apply [tofu args] |" >&2 |
| + echo " ops dev deploy [ie|fi] [--stage-only] | ops dev rebuild [--yes]" >&2 |
| + exit 2 |
| +} |
| + |
| +# plan_guard [tofu args...]: run a plan, show it, and hard-stop when it would |
| +# REPLACE a resource. Shared by `plan` (whose whole job this is) and `apply` |
| +# (so an interactive "yes" can't sleepwalk into the AMI rebuild). Extra args |
| +# pass through, so a targeted apply guards exactly the graph it will touch. |
| +plan_guard() { |
| + planout="$(mktemp -t lchat-dev-plan.XXXXXX)" |
| + # -detailed-exitcode: 0 no changes, 2 changes present, 1 error. Changes are |
| + # fine; what we refuse to shrug at is a pending instance REPLACEMENT. |
| + # (Captured to a file, not piped through tee — a pipeline would report tee's |
| + # exit status and silently lose tofu's.) |
| + rc=0 |
| + run_tofu plan -input=false -detailed-exitcode "$@" >"$planout" 2>&1 || rc=$? |
| + cat "$planout" |
| + if [ "$rc" -eq 1 ]; then rm -f "$planout"; exit 1; fi |
| + if grep -q "must be replaced" "$planout"; then |
| + rm -f "$planout" |
| + cat >&2 <<'EOF' |
| + |
| + ██ STOP — this plan REPLACES a resource (see "must be replaced" above). |
| + If it is aws_instance.ie_node, the cause is usually a newer Canonical AMI |
| + (compute.tf pins nothing: most_recent = true, no lifecycle ignore_changes). |
| + A plain apply would DESTROY the Dublin box — its /data dies with it; only |
| + the EIP (and DNS) survive. Your options: |
| + - apply just the safe resource: ops dev apply -target=<addr> |
| + - make the rebuild deliberate: ops dev rebuild (new keys, guided) |
| + - or add lifecycle { ignore_changes = [ami] } to the instance and re-plan |
| +EOF |
| + exit 1 |
| + fi |
| + rm -f "$planout" |
| +} |
| + |
| +# --- credentials ---------------------------------------------------------- |
| +# Only these five ever leave the Bitwarden item for the environment — an |
| +# allowlist, so a stray line in the item can never export something else. |
| +DEV_CRED_VARS='AWS_ACCESS_KEY_ID|AWS_SECRET_ACCESS_KEY|AWS_SESSION_TOKEN|DNSIMPLE_TOKEN|DNSIMPLE_ACCOUNT' |
| + |
| +have_aws_auth() { |
| + [ -n "${AWS_ACCESS_KEY_ID:-}" ] && return 0 |
| + aws configure list-profiles 2>/dev/null | grep -qx "$AWS_PROFILE" |
| +} |
| + |
| +# Run tofu with whichever auth we ended up with. Explicit env credentials win; |
| +# AWS_PROFILE is only set when that profile really exists (setting it to an |
| +# absent profile would itself be the error). |
| +run_tofu() { |
| + if [ -n "${AWS_ACCESS_KEY_ID:-}" ]; then tofu "$@"; else AWS_PROFILE="$AWS_PROFILE" tofu "$@"; fi |
| +} |
| + |
| +# find_item <search> <pinned-name> <hint> — resolve exactly one vault item, |
| +# echoing its name. bw reads a LOCAL CACHE of the vault: an item added, |
| +# removed, or deduplicated in another client is invisible here until |
| +# `bw sync` — so a surprising result (none, or more than one) gets one sync |
| +# and retry before it is believed. Two items SHARING a name is its own error: |
| +# `bw get item <name>` can't disambiguate, so the vault needs a rename/delete. |
| +find_item() { |
| + search="$1"; pinned="$2"; hint="$3" |
| + if [ -n "$pinned" ]; then printf '%s\n' "$pinned"; return 0; fi |
| + synced=no |
| + while :; do |
| + names="$(bw list items --search "$search" 2>/dev/null | jq -r '.[].name' || true)" |
| + n="$(printf '%s\n' "$names" | grep -c . || true)" |
| + if [ "$n" = 1 ]; then printf '%s\n' "$names"; return 0; fi |
| + if [ "$synced" = no ]; then |
| + echo "ops dev: $n items match '$search' — syncing bw's local cache and retrying" >&2 |
| + bw sync >/dev/null 2>&1 || true |
| + synced=yes |
| + continue |
| + fi |
| + if [ "$n" = 0 ]; then |
| + echo "ops dev: no item matching '$search' in your vault ($hint)" >&2 |
| + elif [ "$(printf '%s\n' "$names" | sort -u | grep -c .)" -lt "$n" ]; then |
| + echo "ops dev: DUPLICATE items named alike match '$search' — bw can't fetch by an" >&2 |
| + echo " ambiguous name; delete or rename one in the vault, then re-run:" >&2 |
| + printf '%s\n' "$names" | sed 's/^/ /' >&2 |
| + else |
| + echo "ops dev: several items match '$search' — pin one ($hint):" >&2 |
| + printf '%s\n' "$names" | sed 's/^/ /' >&2 |
| + fi |
| + return 1 |
| + done |
| +} |
| + |
| +load_dev_creds() { |
| + command -v bw >/dev/null 2>&1 || { |
| + echo "ops dev: bw is not installed, and the credentials aren't in the environment" >&2 |
| + return 1; } |
| + command -v jq >/dev/null 2>&1 || { echo "ops dev: jq is required" >&2; return 1; } |
| + st="$(bw status 2>/dev/null | jq -r '.status' 2>/dev/null || true)" |
| + case "$st" in |
| + unlocked) ;; |
| + locked) |
| + if [ -t 0 ]; then |
| + echo "ops dev: unlocking bitwarden to read the dev agent creds" >&2 |
| + # Do NOT redirect stderr: `bw unlock` writes its master-password prompt |
| + # to stderr and the session token to stdout — swallowing stderr hides |
| + # the prompt and the script looks hung (the doctor lesson). Only |
| + # stdout is captured, and the token is never displayed. |
| + BW_SESSION="$(bw unlock --raw)" || true |
| + [ -n "${BW_SESSION:-}" ] || { echo "ops dev: unlock failed" >&2; return 1; } |
| + export BW_SESSION |
| + else |
| + echo "ops dev: bitwarden is locked and there is no tty to prompt on" >&2 |
| + echo ' export BW_SESSION=$(bw unlock --raw) # then re-run' >&2 |
| + return 1 |
| + fi ;; |
| + *) |
| + echo "ops dev: bw is not logged in (bw login — EU vault: https://vault.bitwarden.eu)" >&2 |
| + return 1 ;; |
| + esac |
| + |
| + # The dev agent item is the one contract entry that is a naming CONVENTION |
| + # rather than an exact name — it is per-operator (each has a revocable |
| + # aws-dev-agent-<name> IAM identity, docs/secrets.md § AWS), so resolve |
| + # "exactly one item with the contract prefix"; LCHAT_DEV_CREDS_ITEM pins one. |
| + item="$(find_item "$BW_ITEM_DEV_AGENT_PREFIX" "${LCHAT_DEV_CREDS_ITEM:-}" \ |
| + "each operator gets their own dev agent creds, docs/secrets.md § AWS; LCHAT_DEV_CREDS_ITEM pins one")" \ |
| + || return 1 |
| + |
| + # The item is the sealed env file: KEY=VALUE lines in its secure note (or |
| + # custom fields). Pull only the allowlisted names; never echo a value. |
| + lines="$(bw get item "$item" 2>/dev/null | jq -r --arg re "^($DEV_CRED_VARS)=" ' |
| + ((.notes // "" | split("\n")) + [.fields[]? | "\(.name)=\(.value)"])[] |
| + | select(test($re))' || true)" |
| + if [ -n "$lines" ]; then |
| + while IFS= read -r line; do |
| + k="${line%%=*}" |
| + # Environment always wins — the item only fills gaps. ($k is allowlisted |
| + # by the select() above, so the eval can't be steered by item content.) |
| + eval "cur=\${$k:-}" |
| + # shellcheck disable=SC2163 # the K=V-operand form of export is intended |
| + [ -n "$cur" ] || export "$line" |
| + done <<EOF |
| +$lines |
| +EOF |
| + echo "ops dev: credentials loaded from bitwarden item '$item'" >&2 |
| + else |
| + echo "ops dev: note: '$item' carried no AWS_*/DNSIMPLE_* lines in its note or fields" >&2 |
| + fi |
| + |
| + # The Bitwarden migration split the DNSimple token out of the sops-era |
| + # bundle into its own item, so when the agent item didn't carry it, fetch it |
| + # from there — by the contract's EXACT name and slot (the login password), |
| + # no searching, no guessing. A failed fetch gets one cache sync + retry (a |
| + # freshly shared item is invisible to bw's local cache until then), then a |
| + # crisp contract error. The account id is public (committed in main.tf) — |
| + # only the token is a secret. |
| + if [ -z "${DNSIMPLE_TOKEN:-}" ]; then |
| + ditem="${LCHAT_DNSIMPLE_ITEM:-$BW_ITEM_DNSIMPLE}" |
| + json="$(bw get item "$ditem" 2>/dev/null || true)" |
| + if [ -z "$json" ]; then |
| + echo "ops dev: item '$ditem' not in bw's local cache — syncing and retrying" >&2 |
| + bw sync >/dev/null 2>&1 || true |
| + json="$(bw get item "$ditem" 2>/dev/null || true)" |
| + fi |
| + if [ -z "$json" ]; then |
| + echo "ops dev: no bitwarden item named exactly '$ditem'" >&2 |
| + echo " — the vault contract (deploy/vault-items.sh); is it shared to your collection," >&2 |
| + echo " or was it renamed without a matching PR?" >&2 |
| + return 1 |
| + fi |
| + tok="$(printf '%s' "$json" | jq -r '.login.password // empty' 2>/dev/null || true)" |
| + [ -n "$tok" ] || { |
| + echo "ops dev: the vault contract expects the DNSimple token in the PASSWORD slot" >&2 |
| + echo " of '$ditem' (deploy/vault-items.sh) — fix the item, or export DNSIMPLE_TOKEN" >&2 |
| + return 1; } |
| + export DNSIMPLE_TOKEN="$tok" |
| + echo "ops dev: DNSIMPLE_TOKEN loaded from bitwarden item '$ditem'" >&2 |
| + fi |
| +} |
| + |
| +# ensure_creds: make DNSIMPLE_TOKEN + AWS auth real, from env or Bitwarden. |
| +ensure_creds() { |
| + # The account id is public (176940, committed in main.tf) — default it here |
| + # rather than in the bw path, so a hand-exported DNSIMPLE_TOKEN (which skips |
| + # Bitwarden entirely) still gets a working provider config. |
| + [ -n "${DNSIMPLE_ACCOUNT:-}" ] || export DNSIMPLE_ACCOUNT=176940 |
| + if [ -z "${DNSIMPLE_TOKEN:-}" ] || ! have_aws_auth; then |
| + load_dev_creds || { |
| + echo "ops dev: still missing credentials — export DNSIMPLE_TOKEN/DNSIMPLE_ACCOUNT and" >&2 |
| + echo " AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY (or configure the '$AWS_PROFILE' aws" >&2 |
| + echo " profile), or make your dev agent item reachable in bw and re-run" >&2 |
| + exit 1 |
| + } |
| + fi |
| + [ -n "${DNSIMPLE_TOKEN:-}" ] || { |
| + echo "ops dev: DNSIMPLE_TOKEN still unset (the item didn't carry it?)" >&2; exit 1; } |
| + have_aws_auth || { |
| + echo "ops dev: no AWS credentials (env or '$AWS_PROFILE' profile)" >&2; exit 1; } |
| +} |
| + |
| +# swap <ie|fi> <ver>: back up the live binary, move the staged one in, restart |
| +# the units that box runs, and confirm the running stamp matches the build. |
| +swap_box() { |
| + which_box="$1"; ver="$2" |
| + if [ "$which_box" = ie ]; then box="$BOX_IE"; units="lchat-router lchat-cloud lchat-control"; run() { ssh_ie "$box" "$@"; } |
| + else box="$BOX_FI"; units="lchat-router"; run() { ssh -o ConnectTimeout=10 "$box" "$@"; } |
| + fi |
| + run "sudo cp /opt/lchat/lchat /opt/lchat/lchat.bak-pre-$ver \ |
| + && sudo mv /opt/lchat/incoming/lchat-new /opt/lchat/lchat \ |
| + && sudo chmod +x /opt/lchat/lchat \ |
| + && sudo systemctl restart $units" |
| + running="$(run '/opt/lchat/lchat version' 2>/dev/null || true)" |
| + if [ "$running" = "$ver" ] && run 'systemctl is-active --quiet lchat-router'; then |
| + echo " $which_box: running $running" |
| + else |
| + echo "ops dev: $which_box is NOT healthy after the swap (version '$running', wanted '$ver')" >&2 |
| + echo " inspect: ssh $box journalctl -u lchat-router -n 60 --no-pager" >&2 |
| + echo " rollback: ssh $box 'sudo mv /opt/lchat/lchat.bak-pre-$ver /opt/lchat/lchat && sudo systemctl restart $units'" >&2 |
| + exit 1 |
| + fi |
| +} |
| + |
| +cmd="${1:-}" |
| +[ $# -gt 0 ] && shift |
| +case "$cmd" in |
| + |
| +plan) |
| + cd "$root/infra/dev-mesh" |
| + ensure_creds |
| + [ -d .terraform ] || run_tofu init -input=false |
| + plan_guard "$@" |
| + ;; |
| + |
| +apply) |
| + # The self-serve apply: same credential fetch, same replacement guard — |
| + # tofu's own interactive approve remains the final gate. Extra args pass |
| + # through (e.g. `ops dev apply -target=aws_security_group.ie_node` to land |
| + # a security-group change while an AMI replacement is pending). |
| + cd "$root/infra/dev-mesh" |
| + ensure_creds |
| + [ -d .terraform ] || run_tofu init -input=false |
| + plan_guard "$@" |
| + run_tofu apply "$@" |
| + ;; |
| + |
| +deploy) |
| + targets="ie fi"; stage_only=no |
| + for a in "$@"; do |
| + case "$a" in |
| + ie|fi) targets="$a" ;; |
| + --stage-only) stage_only=yes ;; |
| + *) usage ;; |
| + esac |
| + done |
| + |
| + ver="$(git -C "$root" rev-parse --short HEAD)" |
| + if ! git -C "$root" diff --quiet || ! git -C "$root" diff --cached --quiet; then |
| + ver="${ver}-dirty" |
| + echo "ops dev: working tree has uncommitted changes — building anyway ($ver)" >&2 |
| + fi |
| + |
| + tmp="$(mktemp -d -t lchat-dev.XXXXXX)" |
| + trap 'rm -rf "$tmp"' EXIT |
| + cd "$root" # build-linux.sh compiles the package at the current directory |
| + for t in $targets; do |
| + case "$t" in |
| + ie) arch=arm64 ;; # Dublin is a Graviton box |
| + fi) arch=amd64 ;; |
| + esac |
| + echo "ops dev: building $ver for linux/$arch ($t)" |
| + OUT="$tmp/lchat-$t" ARCH="$arch" "$here/build-linux.sh" "$ver" |
| + done |
| + |
| + for t in $targets; do |
| + if [ "$t" = ie ]; then box="$BOX_IE"; else box="$BOX_FI"; fi |
| + echo "ops dev: staging on $t ($box)" |
| + # Stage beside the live binary, never over it — the swap is its own step. |
| + # An SSH hang here usually means you are outside the security group's |
| + # /32 allowlist, not that the box is down: run `ops doctor dev`. |
| + if [ "$t" = ie ]; then |
| + ssh_ie "$box" 'sudo mkdir -p /opt/lchat/incoming' |
| + scp_ie -q "$tmp/lchat-$t" "$box:/opt/lchat/incoming/lchat-new" |
| + else |
| + ssh -o ConnectTimeout=10 "$box" 'sudo mkdir -p /opt/lchat/incoming' |
| + scp -q "$tmp/lchat-$t" "$box:/opt/lchat/incoming/lchat-new" |
| + fi |
| + done |
| + |
| + if [ "$stage_only" = yes ]; then |
| + echo "ops dev: staged at /opt/lchat/incoming/lchat-new on: $targets (not swapped)" |
| + echo " swap by hand when ready (pairs with env edits — see the cutover runbook)" |
| + exit 0 |
| + fi |
| + |
| + for t in $targets; do |
| + echo "ops dev: swapping + restarting $t" |
| + swap_box "$t" "$ver" |
| + done |
| + echo |
| + echo "note: this opened dev-box SSH sessions — if this session hasn't written its" |
| + echo " access-ledger why-entry yet: ./deploy/ops ledger \"...\"" |
| + ;; |
| + |
| +rebuild) |
| + # Destroy + recreate the Dublin box — the deliberate key rotation. EC2 |
| + # injects the keypair only at first boot, so this is how you take over a box |
| + # whose key you don't hold: commit YOUR public key + /32 to compute.tf, |
| + # replace the instance, and it boots trusting only you. The EIP, S3 bucket |
| + # (hibernated spaces, mesh membership, the id ledger), IAM, and the Route53 |
| + # zone are separate resources and survive; the box's /data does not. |
| + confirm=no |
| + for a in "$@"; do case "$a" in --yes) confirm=yes ;; *) usage ;; esac; done |
| + ct="$root/infra/dev-mesh/compute.tf" |
| + |
| + # A fresh box trusts exactly one key — make sure it will be one you hold. |
| + # Generating is safe (refuses to overwrite); committing it is the PR's job. |
| + if [ ! -f "$KEY_IE" ]; then |
| + echo "ops dev: no $KEY_IE — generating a fresh box key (never vaulted; the" |
| + echo " rebuilt box trusts it alone, so share it with the other operators)" |
| + ssh-keygen -q -t ed25519 -N '' -C eleven-dev-box -f "$KEY_IE" |
| + fi |
| + mypub="$(ssh-keygen -y -f "$KEY_IE" | awk '{print $1" "$2}')" |
| + tfpub="$(awk '/resource "aws_key_pair" "ie_node"/,/^}/' "$ct" \ |
| + | sed -n 's/.*public_key *= *"\([^"]*\)".*/\1/p' | awk '{print $1" "$2}')" |
| + if [ "$mypub" != "$tfpub" ]; then |
| + echo "ops dev: compute.tf's key pair is NOT your key — tofu applies what the file says," >&2 |
| + echo " so the rebuilt box would trust someone else. Put your public key in" >&2 |
| + echo " aws_key_pair.ie_node (a reviewed PR — the public key is committed on purpose):" >&2 |
| + echo " public_key = \"$(ssh-keygen -y -f "$KEY_IE")\"" >&2 |
| + exit 1 |
| + fi |
| + |
| + # Same rule for the SSH allowlist: outside it, the post-rebuild SSH wait |
| + # would hang forever and read as a dead box. TF_VAR_ssh_ingress_cidr (a |
| + # JSON list) overrides the committed default, same as for tofu itself; |
| + # either way, every a.b.c.d/nn token counts. |
| + cidrs="$(printf '%s' "${TF_VAR_ssh_ingress_cidr:-}" \ |
| + | grep -oE '[0-9]+(\.[0-9]+){3}/[0-9]+' || true)" |
| + [ -n "$cidrs" ] || cidrs="$(awk '/variable "ssh_ingress_cidr"/,/^}/' "$ct" \ |
| + | grep -oE '[0-9]+(\.[0-9]+){3}/[0-9]+' || true)" |
| + mine="$(curl -s -m5 https://api.ipify.org 2>/dev/null || true)" |
| + if [ -z "$mine" ] || ! printf '%s\n' "$cidrs" | sed 's|/.*||' | grep -qxF "$mine"; then |
| + echo "ops dev: the security group would admit [$(printf '%s' "$cidrs" | tr '\n' ' ')] but your IP" >&2 |
| + echo " is ${mine:-unknown} — the rebuilt box would be unreachable from here. Add your /32" >&2 |
| + echo " to ssh_ingress_cidr in compute.tf (removing an entry revokes that operator)." >&2 |
| + exit 1 |
| + fi |
| + |
| + if [ "$confirm" != yes ]; then |
| + cat <<EOF |
| +ops dev rebuild — this DESTROYS and recreates the Dublin box. |
| + dies: everything on its disk — /data (non-hibernating dev spaces, the |
| + router registry, router/cloud/home env files and their tokens) |
| + survives: the EIP (DNS keeps pointing right), the S3 bucket (hibernated |
| + spaces, mesh membership, the id ledger), IAM, the Route53 zone |
| + needs: LCHAT_MESH_TOKEN re-paired with Helsinki afterwards, and the env |
| + files rebuilt (the runbook's Step 3 blocks) |
| +Coordinate first — this tears down the live half of the dev mesh until |
| +re-provisioned. Re-run with --yes to proceed (tofu will still show its plan |
| +and ask for approval). |
| +EOF |
| + exit 1 |
| + fi |
| + |
| + cd "$root/infra/dev-mesh" |
| + ensure_creds |
| + [ -d .terraform ] || run_tofu init -input=false |
| + # tofu's own interactive approve is the second gate — read the plan: it |
| + # should replace aws_instance.ie_node (and the key pair), nothing else |
| + # destructive. |
| + run_tofu apply -replace=aws_instance.ie_node |
| + |
| + ip="$(run_tofu output -raw ie_node_ip)" |
| + bucket="$(run_tofu output -raw mesh_bucket)" |
| + # Same EIP, new host key — the stale known_hosts entry would make every ssh |
| + # fail with "REMOTE HOST IDENTIFICATION HAS CHANGED", so drop it first. |
| + ssh-keygen -R "$ip" >/dev/null 2>&1 || true |
| + echo "ops dev: waiting for the new box to accept ssh ($ip)" |
| + i=0 |
| + until ssh -i "$KEY_IE" -o ConnectTimeout=5 -o StrictHostKeyChecking=accept-new \ |
| + "ubuntu@$ip" true 2>/dev/null; do |
| + i=$((i + 1)) |
| + [ "$i" -lt 60 ] || { echo "ops dev: box not reachable after 5m — check the AWS console" >&2; exit 1; } |
| + sleep 5 |
| + done |
| + |
| + # First provision: fresh arm64 binary + `lchat bootstrap` (writes the |
| + # router/control/cloud units and starts the router). After this, routine |
| + # updates go through `ops dev deploy ie`. |
| + ver="$(git -C "$root" rev-parse --short HEAD)" |
| + tmp="$(mktemp -d -t lchat-rebuild.XXXXXX)" |
| + trap 'rm -rf "$tmp"' EXIT |
| + cd "$root" # we were in infra/dev-mesh; build-linux.sh compiles the cwd package |
| + echo "ops dev: building $ver for linux/arm64 and bootstrapping" |
| + OUT="$tmp/lchat" ARCH=arm64 "$here/build-linux.sh" "$ver" |
| + scp_ie -q "$tmp/lchat" "ubuntu@$ip:/tmp/lchat" |
| + ssh_ie "ubuntu@$ip" 'sudo mkdir -p /opt/lchat && sudo mv /tmp/lchat /opt/lchat/lchat \ |
| + && sudo chmod +x /opt/lchat/lchat && sudo /opt/lchat/lchat bootstrap' |
| + |
| + # The env files are derived facts plus one secret — encoded here so no |
| + # runbook is needed. The mesh token is READ from the surviving Helsinki box |
| + # (the pair must match) and never printed; the cloud admin token is freshly |
| + # minted, because a rebuild is a new cloud. |
| + echo "ops dev: writing /data/router.env + /data/cloud.env" |
| + mesh_token="$(ssh -o ConnectTimeout=10 "$BOX_FI" \ |
| + "sudo grep '^LCHAT_MESH_TOKEN=' /data/router.env | cut -d= -f2-" 2>/dev/null || true)" |
| + [ -n "$mesh_token" ] || echo "ops dev: WARNING: couldn't read Helsinki's mesh token —" \ |
| + "router.env is written without LCHAT_MESH_TOKEN; the mesh stays unpaired until it is set on both boxes" >&2 |
| + admin_token="$(head -c 24 /dev/urandom | od -An -tx1 | tr -d ' \n')" |
| + { |
| + printf 'LCHAT_MESH_DOMAINS=dev.11ch.at\n' |
| + printf 'LCHAT_DNS_PROVIDER=route53\n' |
| + printf 'LCHAT_WILDCARD_DOMAINS=dev.11ch.at\n' |
| + printf 'AWS_REGION=eu-west-1\n' |
| + printf 'LCHAT_HIBERNATE_BUCKET=%s\n' "$bucket" |
| + printf 'LCHAT_MESH_REGION=ie\n' |
| + printf 'LCHAT_MESH_SELF=https://one.ie.dev.11ch.at\n' |
| + [ -z "$mesh_token" ] || printf 'LCHAT_MESH_TOKEN=%s\n' "$mesh_token" |
| + } | ssh_ie "ubuntu@$ip" 'sudo tee /data/router.env >/dev/null && sudo chmod 600 /data/router.env' |
| + { |
| + printf 'LCHAT_CLOUD_BASE_DOMAIN=dev.11ch.at\n' |
| + printf 'LCHAT_CLOUD_ID_LEDGER=1\n' |
| + printf 'LCHAT_HIBERNATE_BUCKET=%s\n' "$bucket" |
| + printf 'LCHAT_HIBERNATE_REGION=eu-west-1\n' |
| + printf 'LCHAT_CLOUD_REGION=ie\n' |
| + printf 'LCHAT_ADMIN_TOKEN=%s\n' "$admin_token" |
| + } | ssh_ie "ubuntu@$ip" 'sudo tee /data/cloud.env >/dev/null && sudo chmod 600 /data/cloud.env' |
| + ssh_ie "ubuntu@$ip" 'sudo systemctl restart lchat-router lchat-cloud lchat-control' |
| + |
| + # A fresh registry knows no routes: the surviving peer bulk-seeds every |
| + # in-scope route into it (meshreplicate.go — mesh-seed exists exactly for a |
| + # freshly joined box). sudo strips env, so the peer's mesh config is |
| + # sourced explicitly (the old runbook's hardest-won gotcha). |
| + if [ -n "$mesh_token" ]; then |
| + echo "ops dev: seeding routes from Helsinki" |
| + ssh -o ConnectTimeout=10 "$BOX_FI" \ |
| + "sudo sh -c 'set -a; . /data/router.env; set +a; /opt/lchat/lchat mesh-seed https://one.ie.dev.11ch.at'" \ |
| + || echo "ops dev: WARNING: mesh-seed failed — rerun it on Helsinki when the router is up" >&2 |
| + fi |
| + |
| + cat <<EOF |
| + |
| +Dublin rebuilt on $ip (same EIP — DNS untouched), running $ver, trusting only |
| +your key: env files written, units restarted, routes seeded from Helsinki. |
| +Not yet encoded in this verb — EXTEND IT here when next needed, don't reach |
| +for a runbook: the dev home server (/data/home.env + its unit) and the |
| +cloud.env demo/pool extras. |
| + verify: ssh ubuntu@$ip journalctl -u lchat-router -n 40 --no-pager | grep -i wildcard |
| + then: land the compute.tf PR if it changed for this rebuild, and |
| + ./deploy/ops ledger "rebuilt the dev-mesh Dublin box" |
| +EOF |
| + ;; |
| + |
| +*) usage ;; |
| +esac |
diff --git a/deploy/ops-ledger.sh b/deploy/ops-ledger.sh| new file mode 100755 |
| index 0000000..117a8a5 |
| --- /dev/null |
| +++ b/deploy/ops-ledger.sh |
| @@ -0,0 +1,68 @@ |
| +#!/usr/bin/env sh |
| +# 🤖 ops ledger — draft the public SSH-access why-entry for this session. |
| +# |
| +# ./deploy/ops ledger "canary session for the search dropdown" \ |
| +# --ref https://github.com/elevenmessenger/messenger/pull/1193 |
| +# |
| +# Every session that opens an operator SSH session into a box owes a why-entry |
| +# in web/status/ssh/ (RUNBOOK.md § Box access) — the rule is per-session, not |
| +# per-mutation, so read-only sessions count too. The box-side collector |
| +# publishes the who/when metadata automatically; the entry here is the human |
| +# "why" half, PR-reviewed (review is the redaction gate), merged like any |
| +# other change. This verb only DRAFTS the file — writing prose about why you |
| +# were on a box is a judgement call, so a human reads it before it ships. |
| +# |
| +# --slug <s> filename slug (default: derived from the why text) |
| +# --window "..." e.g. "09:00–11:30 UTC" (default: now, marked ongoing) |
| +# --ref <url> a PR/issue this session was for (repeatable) |
| +set -eu |
| + |
| +root="$(CDPATH= cd -- "$(dirname -- "$0")/.." && pwd)" |
| +command -v jq >/dev/null 2>&1 || { |
| + echo "ops ledger: jq is required (it builds the JSON safely)" >&2; exit 1; } |
| + |
| +why=""; slug=""; window=""; refs="" |
| +while [ $# -gt 0 ]; do |
| + case "$1" in |
| + --slug) slug="${2:?}"; shift 2 ;; |
| + --window) window="${2:?}"; shift 2 ;; |
| + --ref) refs="$refs $2"; shift 2 ;; |
| + -*) echo "ops ledger: unknown flag $1" >&2; exit 2 ;; |
| + *) why="$1"; shift ;; |
| + esac |
| +done |
| +[ -n "$why" ] || { |
| + echo 'usage: ops ledger "why this session was on the boxes" [--slug s] [--window "..."] [--ref url]...' >&2 |
| + exit 2; } |
| + |
| +# The operator name is public — take it from git identity, which is what the |
| +# existing entries use, and never guess from the machine's username. |
| +operator="$(git -C "$root" config user.name || true)" |
| +[ -n "$operator" ] || { |
| + echo "ops ledger: git config user.name is empty — set it; the entry names the operator publicly" >&2 |
| + exit 1; } |
| + |
| +date="$(date -u +%Y-%m-%d)" |
| +[ -n "$window" ] || window="$(date -u +%H:%M)– UTC (ongoing — fill in the end before the PR)" |
| +if [ -z "$slug" ]; then |
| + # First few words of the why, lowercased, non-alphanumerics folded to hyphens. |
| + slug="$(printf '%s' "$why" | tr '[:upper:]' '[:lower:]' | tr -cs 'a-z0-9' '-' \ |
| + | cut -c1-40 | sed 's/^-//; s/-$//')" |
| +fi |
| + |
| +out="$root/web/status/ssh/$date-$slug.json" |
| +[ ! -e "$out" ] || { echo "ops ledger: $out already exists — pick another --slug" >&2; exit 1; } |
| + |
| +# jq does the quoting, so the why text can contain anything. |
| +jq -n --arg date "$date" --arg operator "$operator" --arg window "$window" \ |
| + --arg why "$why" --arg refs "$refs" \ |
| + '{date: $date, operator: $operator, window: $window, why: $why, |
| + refs: ($refs | split(" ") | map(select(. != "")))}' > "$out" |
| + |
| +cat <<EOF |
| +drafted $out |
| +$(cat "$out") |
| + |
| +Next: read it as a stranger would (it is published, forever), fix the window, |
| +then commit it as a tiny standalone PR — review is the redaction gate. |
| +EOF |
diff --git a/deploy/ops-release.sh b/deploy/ops-release.sh| new file mode 100755 |
| index 0000000..b8566dc |
| --- /dev/null |
| +++ b/deploy/ops-release.sh |
| @@ -0,0 +1,101 @@ |
| +#!/usr/bin/env sh |
| +# 🤖 ops publish / promote / ship — the release path, one verb per stage. |
| +# |
| +# ./deploy/ops publish "release notes" # build HEAD -> server-<sha> release -> edge |
| +# ./deploy/ops promote <ver> beta # move a channel pointer (re-signs the feed) |
| +# ./deploy/ops ship # every fleet box catches up NOW |
| +# |
| +# The machinery is deploy/release-server.sh (which pulls the Bitwarden signing |
| +# key itself, unlocking interactively when the vault is locked — no key setup). |
| +# This wrapper adds only the rules that used to live in prose: |
| +# |
| +# - a release must correspond to LANDED MAIN CODE — publish refuses a HEAD |
| +# that origin/main doesn't contain, so a feature branch can't become the |
| +# fleet's edge build by accident; |
| +# - there are exactly three channels, edge / beta / stable. There is NO dev |
| +# channel — the dev mesh is deployed by `ops dev deploy`, never the feed; |
| +# - `ship` means every box, not one: it loops the fleet (same aliases doctor |
| +# checks), so a second box can't quietly stay behind; |
| +# - box SSH sessions incur a why-entry in the public access ledger. |
| +set -eu |
| + |
| +here="$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)" |
| +root="$(CDPATH= cd -- "$here/.." && pwd)" |
| +# Fleet box ssh aliases — user/hostname come from ~/.ssh/config (operator SSH |
| +# rides WireGuard; specifics in CLAUDE.local.md). Same default as doctor.sh. |
| +BOXES="${LCHAT_BOXES:-elevenbox kepler}" |
| + |
| +usage() { |
| + echo "usage: ops publish [notes...] | ops promote <ver> edge|beta|stable | ops ship [box...]" >&2 |
| + exit 2 |
| +} |
| + |
| +cmd="${1:-}" |
| +[ $# -gt 0 ] && shift |
| +case "$cmd" in |
| + |
| +publish) |
| + # Refuse to publish code main doesn't have: edge tracks main, and a branch |
| + # build published by accident would auto-update the whole edge fleet onto |
| + # unreviewed code within the hour. (Branch builds belong on a pinned canary: |
| + # `ops canary deploy`.) Fetch first so the check is against today's main, |
| + # not a stale ref; offline, fall back to the last-known one. |
| + git -C "$root" fetch -q origin main 2>/dev/null \ |
| + || echo "ops publish: warning: could not fetch origin/main — checking against the last-known ref" >&2 |
| + if ! git -C "$root" merge-base --is-ancestor HEAD origin/main 2>/dev/null; then |
| + echo "ops publish: HEAD is not on origin/main — a release must be landed main code." >&2 |
| + echo " merge the PR, pull main, and publish from there" >&2 |
| + echo " (branch builds go to a pinned canary instead: ops canary deploy <slug>)" >&2 |
| + exit 1 |
| + fi |
| + # release-server.sh re-checks this, but 20 minutes earlier is kinder. |
| + if [ -n "$(git -C "$root" status --porcelain 2>/dev/null)" ]; then |
| + echo "ops publish: working tree is dirty — publish from a clean main" >&2 |
| + exit 1 |
| + fi |
| + exec "$here/release-server.sh" publish "$@" |
| + ;; |
| + |
| +promote) |
| + ver="${1:-}"; chan="${2:-}" |
| + # Catch the swapped-argument habit before release-server's terser usage line. |
| + case "$ver" in edge|beta|stable) |
| + echo "ops promote: arguments are <version> then <channel> — try: ops promote $chan $ver" >&2 |
| + exit 2 ;; |
| + esac |
| + case "$chan" in |
| + edge|beta|stable) ;; |
| + dev) |
| + echo "ops promote: there is no 'dev' channel — the channels are edge / beta / stable." >&2 |
| + echo " the dev mesh is not on the release feed at all; deploy it with: ops dev deploy" >&2 |
| + exit 2 ;; |
| + *) usage ;; |
| + esac |
| + [ -n "$ver" ] || usage |
| + exec "$here/release-server.sh" promote "$ver" "$chan" |
| + ;; |
| + |
| +ship) |
| + # Ship converges the WHOLE fleet: one release-server.sh ship pass per box |
| + # (each box runs its own updater). Naming boxes limits the sweep; the |
| + # default is every fleet box, so nobody has to remember there are two. |
| + [ $# -gt 0 ] && BOXES="$*" |
| + failed="" |
| + for box in $BOXES; do |
| + echo "ops ship: converging $box" |
| + # One box lagging (GitHub's CDN serves fresh manifests stale for a few |
| + # minutes) must not stop the others from converging — finish the sweep, |
| + # then report. |
| + LCHAT_BOX="$box" "$here/release-server.sh" ship || failed="$failed $box" |
| + done |
| + echo |
| + echo "note: this opened operator SSH sessions — the public access ledger wants a" |
| + echo " why-entry for this session: ops ledger \"shipped <ver>\" (web/status/ssh/)" |
| + if [ -n "$failed" ]; then |
| + echo "ops ship: still behind:$failed — usually the CDN window; retry in a few minutes" >&2 |
| + exit 1 |
| + fi |
| + ;; |
| + |
| +*) usage ;; |
| +esac |
diff --git a/deploy/release-server.sh b/deploy/release-server.sh| index bd51616..58ca4ec 100755 |
| --- a/deploy/release-server.sh |
| +++ b/deploy/release-server.sh |
| @@ -51,6 +51,10 @@ |
| # never trusted. Unset, this script is unchanged. |
| set -eu |
| +# The Bitwarden item names this script reads — the vault contract. |
| +# shellcheck source=deploy/vault-items.sh |
| +. "$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)/vault-items.sh" |
| + |
| REPO="${LCHAT_RELEASES_REPO:-elevenmessenger/releases}" |
| KEY="${LCHAT_RELEASE_KEY_FILE:-}" |
| MIRROR_BUCKET="${LCHAT_RELEASE_MIRROR_BUCKET:-}" |
| @@ -79,7 +83,7 @@ case "${1:-}" in publish|promote) |
| && command -v bw >/dev/null 2>&1 && command -v jq >/dev/null 2>&1; then |
| # The item has no password field — the seed rides a hidden custom field, |
| # picked out as the only >40-char value (matches the header's incantation). |
| - bw_key() { bw get item "Eleven Messenger server release signing key" 2>/dev/null \ |
| + bw_key() { bw get item "$BW_ITEM_SIGNING_KEY" 2>/dev/null \ |
| | jq -r '.fields[]? | select((.value|length)>40).value' | head -1; } |
| LCHAT_RELEASE_KEY="$(bw_key)" |
| if [ -z "$LCHAT_RELEASE_KEY" ]; then |
| @@ -91,7 +95,7 @@ case "${1:-}" in publish|promote) |
| LCHAT_RELEASE_KEY="$(bw_key)" |
| fi |
| [ -n "$LCHAT_RELEASE_KEY" ] || { |
| - echo "release-server: couldn't read 'Eleven Messenger server release signing key' from Bitwarden" >&2; exit 1; } |
| + echo "release-server: couldn't read '$BW_ITEM_SIGNING_KEY' from Bitwarden" >&2; exit 1; } |
| export LCHAT_RELEASE_KEY # the signing tool (a child process) reads it from the env |
| echo "release-server: signing key sourced from Bitwarden" >&2 |
| fi |
diff --git a/deploy/release.sh b/deploy/release.sh| deleted file mode 100755 |
| index d58c02d..0000000 |
| --- a/deploy/release.sh |
| +++ /dev/null |
| @@ -1,34 +0,0 @@ |
| -#!/usr/bin/env sh |
| -# One-command release: fetch the signing key from Bitwarden and drive |
| -# release-server.sh — no key file to manage, runnable from anywhere: |
| -# |
| -# ./deploy/release.sh "release notes" # = publish (the common case) |
| -# ./deploy/release.sh promote <ver> <chan> # move a channel pointer |
| -# ./deploy/release.sh ship # box catch-up (no key involved) |
| -# |
| -# This wrapper is deliberately the ONLY place that knows where the signing key |
| -# lives (the Bitwarden item below); release-server.sh itself stays |
| -# secret-manager-agnostic — the fetch happens out here and the key rides |
| -# stdin (LCHAT_RELEASE_KEY_FILE=-), so it never lands on disk or in the |
| -# environment. bw prompts to unlock when the vault is locked. |
| -set -eu |
| -root="$(CDPATH= cd -- "$(dirname -- "$0")/.." && pwd)" |
| -ITEM="Eleven Messenger server release signing key" |
| - |
| -# Bare arguments are publish notes; ship never needs the key. |
| -case "${1:-}" in publish|promote|ship) ;; *) set -- publish "$@" ;; esac |
| -if [ "$1" = ship ]; then exec "$root/deploy/release-server.sh" "$@"; fi |
| - |
| -key="$(bw get item "$ITEM" | jq -r '(.fields[]? | select(.name=="credential").value) // .notes // empty')" |
| -nl=' |
| -' |
| -case "$key" in |
| -"") |
| - echo "release: no signing key in the Bitwarden item \"$ITEM\" (checked the credential field, then notes)" >&2 |
| - exit 1 ;; |
| -*"$nl"*) |
| - echo "release: the Bitwarden item \"$ITEM\" yielded a multi-line value — that isn't the key. Inspect with:" >&2 |
| - echo " bw get item \"$ITEM\" | jq '{fields:[.fields[]?.name]}'" >&2 |
| - exit 1 ;; |
| -esac |
| -printf '%s\n' "$key" | LCHAT_RELEASE_KEY_FILE=- "$root/deploy/release-server.sh" "$@" |
diff --git a/deploy/vault-items.sh b/deploy/vault-items.sh| new file mode 100644 |
| index 0000000..8bcb33b |
| --- /dev/null |
| +++ b/deploy/vault-items.sh |
| @@ -0,0 +1,28 @@ |
| +# 🤖 The Bitwarden vault contract — the ONE place the scripts and the vault |
| +# agree on item names and where each value lives. Scripts read exactly what |
| +# this file says and fail crisply when the vault doesn't match; they never |
| +# guess. The vault side keeps these items named exactly this way — renaming |
| +# one is a reviewed PR here plus the vault edit, in step, never one without |
| +# the other (the sudoers-operators rule, applied to the vault). |
| +# |
| +# Sourced by release-server.sh, doctor.sh, ops-dev.sh. POSIX sh; constants |
| +# only — no logic, no side effects, nothing here is secret (names, not |
| +# values). |
| +# shellcheck disable=SC2034 # consumed by the sourcing scripts |
| + |
| +# The Ed25519 seed that signs the server release feed. VALUE: a hidden custom |
| +# field — the item has no password slot, and the field carries no agreed name |
| +# yet, so readers pick the only >40-char field value. Naming that field (e.g. |
| +# `credential`) and reading it exactly is the remaining half of this |
| +# contract; doing so is a coordinated vault edit + script change. |
| +BW_ITEM_SIGNING_KEY="Eleven Messenger server release signing key" |
| + |
| +# The DNSimple API token (Eleven account 176940) — the tofu dnsimple provider |
| +# and DNS-01 issuance. VALUE: the login PASSWORD slot, nothing else. |
| +BW_ITEM_DNSIMPLE="lchat DNSimple DNS-01 token (11ch.at — Eleven Messenger acct 176940)" |
| + |
| +# Per-operator dev agent credentials (docs/secrets.md § AWS): every operator |
| +# holds EXACTLY ONE item whose name starts with this prefix — theirs, named |
| +# for their revocable IAM identity (aws-dev-agent-<name>). VALUE: the sealed |
| +# env file — KEY=VALUE lines in the secure note (or custom fields). |
| +BW_ITEM_DEV_AGENT_PREFIX="aws-dev-agent-" |
diff --git a/docs/deploying-instances.md b/docs/deploying-instances.md| index 9b70ba3..3d0cc91 100644 |
| --- a/docs/deploying-instances.md |
| +++ b/docs/deploying-instances.md |
| @@ -1,13 +1,15 @@ |
| -# 🤖 Deploying instances — the operator's guide |
| - |
| -How to run messenger instances on a box that uses the `lchat` router |
| -(`docs/instance-router.md` is the design; `deploy/README.md` is the one-page |
| -fresh-box bootstrap). This doc is the day-to-day operator reference: adding |
| -instances, giving them bots, shipping new builds safely (canary → promote → |
| -rollback), standing up throwaway canary instances for branch builds, and tearing |
| -things down. Everything here is generic — hosts are `chat.example.com`, the box |
| -is `<box>`; your own hostnames, IPs, and secrets live wherever you keep operator |
| -notes, not in this repo. |
| +# 🤖 Deploying instances — the mental model |
| + |
| +> **Not a procedure.** Day-to-day operation goes through **`./deploy/ops`** |
| +> (releases: `publish`/`promote`/`ship`; branch canaries: |
| +> `canary create|deploy|destroy`; preflight: `doctor`). Where this page shows |
| +> command sequences, they document what the verbs do — run the verb, and if |
| +> no verb covers your task, **extend ops** rather than following prose here. |
| + |
| +What a box that uses the `lchat` router is made of, and why the pieces are |
| +shaped this way (`docs/instance-router.md` is the design; `deploy/README.md` |
| +is the one-page fresh-box bootstrap). Everything here is generic — hosts are |
| +`chat.example.com`, the box is `<box>`. |
| ## The mental model |
diff --git a/docs/instance-migration.md b/docs/instance-migration.md| index 1e4df79..2048a47 100644 |
| --- a/docs/instance-migration.md |
| +++ b/docs/instance-migration.md |
| @@ -1,6 +1,12 @@ |
| # 🤖 Migrating an instance from one server to another |
| -Status: **procedure** (validated once in practice). Moving a running messenger |
| +> 🤖 **Not yet encoded — do not run from this page.** There is no |
| +> `ops` verb for instance migration yet. When a migration is next needed, |
| +> the move is to build one from this record (`ops migrate <host> <box>` or |
| +> similar, per `deploy/README.md § The operator surface`'s no-cookbooks |
| +> rule), then run the verb — and mark this page historical. |
| + |
| +Status: **design record** (the shape below was validated once in practice). Moving a running messenger |
| instance to a new box is **data movement + a DNS flip**, done so it's reversible: |
| copy the DB, keep the old instance running as a fallback, and only cut DNS over |
| once the new box is verified. The flip itself is reversible too. |
diff --git a/docs/instance-router.md b/docs/instance-router.md| index d108be7..f260997 100644 |
| --- a/docs/instance-router.md |
| +++ b/docs/instance-router.md |
| @@ -1,7 +1,10 @@ |
| # 🤖 The instance router — a single-machine bootstrap |
| -Status: **proposal for discussion** (no code yet). This is the design note; when |
| -it ships the code becomes the source of truth. |
| +Status: **shipped — this page is the design rationale, not a procedure.** The |
| +machinery described here is live (`lchat bootstrap`/`router`/`add-instance`…), |
| +and day-to-day operation of it goes through **`./deploy/ops`** — if something |
| +here reads like steps to run and no ops verb covers it (e.g. hardening a brand |
| +new box), the move is to add the verb, not to follow the prose. |
| ## The problem |
diff --git a/docs/multi-edge-dev-cutover.md b/docs/multi-edge-dev-cutover.md| index c654c5e..cb31e91 100644 |
| --- a/docs/multi-edge-dev-cutover.md |
| +++ b/docs/multi-edge-dev-cutover.md |
| @@ -9,6 +9,14 @@ returns **200** (not the 503 the country-scoped mesh gave). Design + rationale: |
| This is a **dev-only, reversible** change. Prod (`11ch.at` on DNSimple) is |
| untouched except for one scoped `dev` NS delegation record. |
| +> 🤖 **Historical record (2026-07) — not a procedure.** This page documents |
| +> the cutover as it was performed and the gotchas learned doing it; it is |
| +> kept as design history. **Do not operate from it.** The dev mesh is |
| +> operated by `./deploy/ops dev plan|apply|deploy|rebuild` — `rebuild` now |
| +> writes the env files below, pairs the mesh token, and seeds routes itself. |
| +> Anything the verbs can't do gets **added to them** (extend ops, then run |
| +> it), never hand-run from this page. |
| + |
| --- |
| ## The end state (what "done" looks like) |
| @@ -94,11 +102,10 @@ prod DNSimple zone, and the DNS-01 IAM on both boxes' role/user. The DNSimple |
| 176940 token comes from the vault at apply time — **never a box**. |
| ```sh |
| -cd infra/dev-mesh |
| -set -a; . <(cd ~/elevenmessenger/secrets-vault && sops -d aws-dev-agent-james.env | grep '^DNSIMPLE_'); set +a |
| -AWS_PROFILE=eleven-dev tofu init |
| -AWS_PROFILE=eleven-dev tofu plan # expect: 10 to add, 0 to change, 2 to destroy |
| -AWS_PROFILE=eleven-dev tofu apply |
| +# `ops dev plan`/`apply` sort the credentials out themselves (AWS_* + |
| +# DNSIMPLE_* from Bitwarden — the sops vault is retired): |
| +./deploy/ops dev plan # expect: 10 to add, 0 to change, 2 to destroy |
| +./deploy/ops dev apply |
| ``` |
| The plan (validated 2026-07-18): **+** zone, `*.dev`/`*.ie.dev`/`*.fi.dev` |
diff --git a/docs/operator-vault-migration.md b/docs/operator-vault-migration.md| index d7c906d..a5b49cf 100644 |
| --- a/docs/operator-vault-migration.md |
| +++ b/docs/operator-vault-migration.md |
| @@ -1,5 +1,9 @@ |
| # 🤖 Retiring sops → the shared operator vault on Eleven |
| +> 🤖 **Historical record — not a procedure.** The sops vault is retired and |
| +> the standing store is Bitwarden; what scripts read from it is pinned by |
| +> the vault contract (`deploy/vault-items.sh`). Operate via `./deploy/ops`. |
| + |
| The plan of record for moving the fleet's operator secrets off the sops+age |
| S3-bundle vault (`docs/secrets-vault.md`, RUNBOOK.md "Secrets vault") and onto a |
| **shared, locked Eleven vault** — the Vault mini-app with the biometric lock |
diff --git a/docs/planet-scale-guide/deploys.html b/docs/planet-scale-guide/deploys.html| index 6af3036..9908898 100644 |
| --- a/docs/planet-scale-guide/deploys.html |
| +++ b/docs/planet-scale-guide/deploys.html |
| @@ -70,7 +70,7 @@ |
| </ol> |
| <h2 id="manual-deploy">A manual deploy, end to end: build, stage, canary, promote</h2> |
| -<p>This is the full ceremony for shipping a build you compiled yourself. It is the procedure behind every “deploy the new version” task, and each step exists for a reason given alongside it.</p> |
| +<p>This is the full ceremony for shipping a build you compiled yourself. It is the procedure behind every “deploy the new version” task, and each step exists for a reason given alongside it. <span class="badge live">running today</span> The repository’s operator surface wraps the whole ceremony: <code>./deploy/ops canary create|deploy|destroy <slug></code> runs build → stage → pinned-deploy for a per-session test instance (and the release path is <code>./deploy/ops publish / promote / ship</code>) — the steps below are what those verbs do, worth understanding even when a verb runs them for you.</p> |
| <ol class="procedure"> |
| <li><p class="step-title">Build the Linux binary on your own machine, stamped with its version.</p> |
| <figure class="cmd"> |
diff --git a/docs/server-releases.md b/docs/server-releases.md| index a8e2b05..c246fb2 100644 |
| --- a/docs/server-releases.md |
| +++ b/docs/server-releases.md |
| @@ -1,9 +1,10 @@ |
| # 🤖 Server releases: channels, publishing, and the signed feed |
| How server builds get released, how the three channels work, and the trust |
| -model behind instance auto-updating. The publishing tool is |
| -`deploy/release-server.sh`; the format/signing code is `release.go` (shared by |
| -the publisher and the box updater so they can't drift). The box-side updater |
| +model behind instance auto-updating. The operator entry point is |
| +`./deploy/ops publish|promote|ship`; the machinery those verbs drive is |
| +`deploy/release-server.sh`, and the format/signing code is `release.go` |
| +(shared by the publisher and the box updater so they can't drift). The box-side updater |
| that consumes this feed is described in `docs/instance-router.md` |
| (§ Auto-updating). |
| @@ -306,23 +307,17 @@ mirror can only cause a refusal, not an install. |
| ## Publishing |
| ```sh |
| -# per session: fetch the seed from Bitwarden into the environment — it never |
| -# touches disk, and one fetch covers a whole publish && promote chain. The |
| -# item has no password field; the seed is its hidden custom field (the only |
| -# long value, which is what the jq picks out). |
| -export LCHAT_RELEASE_KEY="$(bw get item "Eleven Messenger server release signing key" \ |
| - | jq -r '.fields[]? | select((.value|length)>40).value' | head -1)" |
| +# cut a release from clean, landed main and point edge at it — the signing |
| +# key comes from Bitwarden automatically (deploy/vault-items.sh), unlocking |
| +# interactively if the vault is locked; no key export, nothing on disk |
| +./deploy/ops publish "what changed, one line" |
| -# cut a release from clean main and point edge at it |
| -./deploy/release-server.sh publish "what changed, one line" |
| - |
| -# make the fleet catch up NOW instead of on the hourly timer (the manual |
| -# "deploy" button; LCHAT_BOX names the box, e.g. root@<ip>): |
| -./deploy/release-server.sh ship |
| +# make every fleet box catch up NOW instead of on the hourly timer |
| +./deploy/ops ship |
| # promote when it's proven |
| -./deploy/release-server.sh promote a1b2c3d beta |
| -./deploy/release-server.sh promote a1b2c3d stable |
| +./deploy/ops promote a1b2c3d beta |
| +./deploy/ops promote a1b2c3d stable |
| ``` |
| `ship` loops the box's `lchat update run` until every un-pinned instance is |
diff --git a/infra/dev-mesh/README.md b/infra/dev-mesh/README.md| index 50b6b84..e4b2e33 100644 |
| --- a/infra/dev-mesh/README.md |
| +++ b/infra/dev-mesh/README.md |
| @@ -24,11 +24,13 @@ NS delegation and the `home.dev.elevenmessenger.com` A record. Cutover + |
| box-reconfig runbook: `docs/multi-edge-dev-cutover.md`. |
| ```sh |
| -cd infra/dev-mesh |
| -# DNSimple lives in the Eleven account (176940) — token from the vault: |
| -set -a; . <(cd ~/elevenmessenger/secrets-vault && sops -d aws-dev-agent-james.env | grep '^DNSIMPLE_'); set +a |
| -AWS_PROFILE=eleven-dev tofu init |
| -AWS_PROFILE=eleven-dev tofu apply |
| +# plan (with the AMI-replacement guard) — credentials come from your |
| +# per-operator dev agent item in Bitwarden automatically (AWS_* + DNSIMPLE_*; |
| +# the DNSimple token lives in the Eleven account, 176940): |
| +./deploy/ops dev plan |
| + |
| +# apply behind the same guard (tofu's interactive approve remains the gate): |
| +./deploy/ops dev apply |
| ``` |
| The Helsinki node's scoped S3 secret is a sensitive output — seal it into the |
diff --git a/infra/dev-mesh/compute.tf b/infra/dev-mesh/compute.tf| index 00170fe..f68f1e8 100644 |
| --- a/infra/dev-mesh/compute.tf |
| +++ b/infra/dev-mesh/compute.tf |
| @@ -22,10 +22,14 @@ resource "aws_key_pair" "ie_node" { |
| public_key = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILReNBCz+nwYvJ7x/RfgW/kbvSFe3pzPhOBM1SOqbjSg eleven-dev-box" |
| } |
| +# One /32 per operator. Add yourself via PR + `ops dev apply` (an in-place |
| +# security-group update — no instance touch); removing an entry revokes that |
| +# operator, so only ever prune with their say-so. Keep literal IPs out of the |
| +# description text — doctor/ops extract every a.b.c.d/nn inside this block. |
| variable "ssh_ingress_cidr" { |
| - description = "Where SSH is allowed from (the operator's current IP/32)" |
| - type = string |
| - default = "109.255.91.23/32" |
| + description = "Operator IPs allowed to SSH (a /32 each)" |
| + type = list(string) |
| + default = ["109.255.91.23/32", "51.155.205.89/32"] # paul, james |
| } |
| resource "aws_security_group" "ie_node" { |
| @@ -33,11 +37,11 @@ resource "aws_security_group" "ie_node" { |
| description = "dev-mesh Dublin node: ssh from the operator, web open" |
| ingress { |
| - description = "ssh (operator only)" |
| + description = "ssh (operators only)" |
| from_port = 22 |
| to_port = 22 |
| protocol = "tcp" |
| - cidr_blocks = [var.ssh_ingress_cidr] |
| + cidr_blocks = var.ssh_ingress_cidr |
| } |
| ingress { |
| description = "http (autocert HTTP-01 + serving)" |
diff --git a/infra/dev-mesh/main.tf b/infra/dev-mesh/main.tf| index 23c5a71..d172349 100644 |
| --- a/infra/dev-mesh/main.tf |
| +++ b/infra/dev-mesh/main.tf |
| @@ -42,8 +42,8 @@ provider "aws" { |
| } |
| # 11ch.at lives in the Eleven DNSimple account (176940), not Paul's personal |
| -# one — DNSIMPLE_TOKEN + DNSIMPLE_ACCOUNT come from the vault at apply time |
| -# (aws-dev-agent-james.env), never committed. |
| +# one — DNSIMPLE_TOKEN + DNSIMPLE_ACCOUNT come from the operator's dev agent |
| +# item in Bitwarden at run time (`ops dev` pulls them itself), never committed. |
| provider "dnsimple" {} |
| data "aws_caller_identity" "current" {} |