amadan
deploy: `ops validate` + `ops provision` (re-land of #1228, which merged to the wrong base) (#1234)
* deploy: `ops validate` + `ops dev provision` — stand the links redirector up on dev Three pieces, all aimed at being able to REHEARSE the 11ch.at fix on the dev mesh before touching production. `ops validate` — the code-side twin of `ops doctor`: every local check in one pass (Go build/vet/tests, node --check + npm test, script parse + shellcheck, tofu fmt/validate). The checks span four toolchains and the shell and tofu halves have no test suite at all, so "what do I run before pushing" was reconstructed from CLAUDE.md every time. Needs no credentials and touches no box. Each script is parsed with the interpreter its OWN shebang names — deploy/ mixes sh and bash, and forcing `sh -n` on the bash collectors reports syntax errors that aren't there (which is exactly what the first version of this did). `bootstrap --no-update-timer` — bootstrap enables the hourly auto-updater, which on the dev mesh would replace the unmerged code under test with the current stable build. Today that invariant is held only by the timer happening to be disabled, which one full bootstrap would silently undo. The flag DISABLES rather than skips, so the end state follows the flag rather than the box's history. `ops dev provision` — refreshes the dev boxes' systemd units and routes the short-link redirector at the dev apex. It exists because `ops dev deploy` only swaps the binary and restarts units that already exist, so a NEW unit never appears on a dev box however often you deploy — only bootstrap writes units. It asserts the update timer really is off before continuing, checks lchat-links came up, registers the route only if absent, and then proves the apex redirects end to end. Plus the Route53 apex A record for dev.11ch.at. A wildcard does not cover the bare name it wildcards, so the apex resolved nowhere — the dev-side mirror of the production gap, which is what makes this a real rehearsal. Co-Authored-By: Claude <noreply@anthropic.com> * deploy: `ops provision` — the fleet-side counterpart to `ops dev provision` No new systemd unit ever reaches an existing box on its own. `deploy`/`ship` swap binaries and restart units that already exist; the hourly updater does the same. Only `lchat bootstrap` writes unit files, and nothing re-runs it after the first provision — which is exactly how the short-link redirector shipped in the binary and ran on no box at all. Confirmed empirically while building this: `ops provision status` reports lchat-links as `not-found` on BOTH fleet boxes — the unit file has never been installed there, not merely left disabled. ops provision units [box...] re-run bootstrap so new units land ops provision apex <host> [box] ...and declare that box the short-link apex ops provision status [box...] read-only: who runs what `apex` passes bootstrap's new --links-apex, so declaring the apex is one step (start the redirector, register the route) rather than bootstrap + add-service, and it states the DNS prerequisite up front — an apex needs its own A record, since a wildcard doesn't cover it — because that is the failure this whole change came from and it surfaces as a confusing TLS error, not a 404. `ops dev provision` moves to the same flag: Dublin bootstraps as the dev apex, Helsinki bootstraps without it, so the dev run now exercises BOTH sides of the apex/not-apex behaviour rather than expecting the redirector everywhere. Its separate add-service step and idempotency check are gone — bootstrap owns that. Co-Authored-By: Claude <noreply@anthropic.com> * deploy: sharpen `ops provision apex` guidance Two small things found reviewing the verb: The DNS prerequisite printed "a *.ch.at wildcard does not cover 11ch.at" for a registrable apex — nonsense. The wildcard clause now only prints for a host under a parent domain (dev.11ch.at), where a covering wildcard plausibly exists and is exactly the trap. And a guardrail note in the header: this verb runs PLAIN bootstrap, which enables the hourly auto-updater — so pointing it at a dev-mesh box would replace the unmerged code under test with the current stable build. `ops dev provision` is the one for those boxes. The only thing standing in the way is the $LCHAT_BOXES default, so the header says so. Co-Authored-By: Claude <noreply@anthropic.com> * deploy: `ops dev provision` refuses to run against a stale binary provision drives bootstrap flags (--links-apex, --no-update-timer) that only exist in a new binary, and `deploy` is what puts one on the box. Run in the wrong order it failed deep inside flag parsing with "flag provided but not defined", which reads like a bug in the script rather than a missing step. Check the box's bootstrap actually offers --links-apex first, and name the verb to run if it doesn't. deploy ships the code; provision configures the box — a distinction worth making loudly, since the two verbs are separate. Co-Authored-By: Claude <noreply@anthropic.com> ---------
Co-authored-by: Claude <noreply@anthropic.com>
9 files changed,
+564
−9
bootstrap.go+31 −4bootstrap_test.go+31 −0deploy/README.md+5 −0deploy/ops+9 −2deploy/ops-dev.sh+132 −1deploy/ops-provision.sh+151 −0deploy/ops-validate.sh+182 −0docs/planet-scale-guide/reference.html+2 −2infra/dev-mesh/route53.tf+21 −0
diff --git a/bootstrap.go b/bootstrap.go| index cd93916..a24ef71 100644 |
| --- a/bootstrap.go |
| +++ b/bootstrap.go |
| @@ -274,6 +274,30 @@ func enableCollectorTimers() { |
| } |
| } |
| +// setUpdateTimer turns the hourly auto-updater on or off, and is deliberately |
| +// two-sided rather than a "skip enabling" flag. |
| +// |
| +// Some boxes must NEVER track a release channel — the dev mesh runs unmerged |
| +// code, so an updater pass would quietly replace it with the current stable |
| +// build. Today that invariant is held only by the timer happening to be |
| +// disabled, which a single full `bootstrap` would silently undo. Skipping the |
| +// enable would not be enough either: a box that was bootstrapped normally once |
| +// already has the timer on, and a later `--no-update-timer` run must fix that |
| +// rather than leave it. So the off path disables explicitly, making the end |
| +// state a property of the flag instead of a property of the box's history. |
| +func setUpdateTimer(on bool) { |
| + if on { |
| + if err := systemctl("enable", "--now", "lchat-update.timer"); err != nil { |
| + die("bootstrap: enable update timer: %v", err) |
| + } |
| + return |
| + } |
| + if err := systemctl("disable", "--now", "lchat-update.timer"); err != nil { |
| + die("bootstrap: disable update timer: %v (this box must not auto-update — refusing to finish in an unknown state)", err) |
| + } |
| + fmt.Println("bootstrap: auto-update timer DISABLED (--no-update-timer) — this box will not track a release channel") |
| +} |
| + |
| // linksSocket is where the redirector listens — the contract between |
| // lchat-links.service's ExecStart and the registry route that fronts it. |
| const linksSocket = "/run/lchat/links.sock" |
| @@ -414,6 +438,7 @@ func runBootstrap(args []string) { |
| noStart := fs.Bool("no-start", false, "install units but don't enable/start the router") |
| collectorsOnly := fs.Bool("collectors-only", false, "refresh ONLY the status collectors (scripts, units, viewer pages) + enable their timers — no service user, no control token, no router restart") |
| linksApex := fs.String("links-apex", "", "make this box the short-link apex for this host (starts the redirector and routes the host at it); without it, a box that isn't already routed as the apex doesn't run the redirector at all") |
| + noUpdateTimer := fs.Bool("no-update-timer", false, "ensure the hourly auto-updater is DISABLED instead of enabled (for boxes that must never track a release channel, e.g. the dev mesh)") |
| _ = fs.Parse(args) |
| // --collectors-only is the no-root-shell refresh path: with the fleet |
| @@ -497,9 +522,7 @@ func runBootstrap(args []string) { |
| if err := systemctl("restart", "lchat-router"); err != nil { |
| die("bootstrap: start router: %v", err) |
| } |
| - if err := systemctl("enable", "--now", "lchat-update.timer"); err != nil { |
| - die("bootstrap: enable update timer: %v", err) |
| - } |
| + setUpdateTimer(!*noUpdateTimer) |
| // Short links: --links-apex declares this box the apex (and records that in |
| // the registry); otherwise converge to whatever the registry already says, |
| // so a re-run neither turns a live apex off nor starts the redirector on a |
| @@ -514,6 +537,10 @@ func runBootstrap(args []string) { |
| // enable their 5-minute timers (safe everywhere — a pass with no /data/status.env |
| // creds is a near no-op). |
| enableCollectorTimers() |
| - fmt.Println("bootstrap: router installed and started (auto-update + status-collector timers enabled)") |
| + autoUpdate := "auto-update" |
| + if *noUpdateTimer { |
| + autoUpdate = "NO auto-update" |
| + } |
| + fmt.Printf("bootstrap: router + links installed and started (%s; status-collector timers enabled)\n", autoUpdate) |
| fmt.Println(" next: lchat add-instance <host>") |
| } |
diff --git a/bootstrap_test.go b/bootstrap_test.go| index b67adbf..36a96b4 100644 |
| --- a/bootstrap_test.go |
| +++ b/bootstrap_test.go |
| @@ -324,6 +324,37 @@ func TestRestartControlUnitsIncludesLinks(t *testing.T) { |
| } |
| } |
| +// TestSetUpdateTimer: the off path must DISABLE, not merely skip enabling. A |
| +// box bootstrapped normally once already has the timer on, and on the dev mesh |
| +// an updater pass would replace unmerged code with the current stable build — |
| +// so the end state has to follow the flag, not the box's history. |
| +func TestSetUpdateTimer(t *testing.T) { |
| + for _, tc := range []struct { |
| + name string |
| + on bool |
| + want string |
| + }{ |
| + {"on enables", true, "enable --now lchat-update.timer"}, |
| + {"off disables", false, "disable --now lchat-update.timer"}, |
| + } { |
| + t.Run(tc.name, func(t *testing.T) { |
| + oldSysctl := systemctl |
| + t.Cleanup(func() { systemctl = oldSysctl }) |
| + var calls []string |
| + systemctl = func(args ...string) error { |
| + calls = append(calls, strings.Join(args, " ")) |
| + return nil |
| + } |
| + |
| + setUpdateTimer(tc.on) |
| + |
| + if len(calls) != 1 || calls[0] != tc.want { |
| + t.Errorf("systemctl calls = %v, want [%q]", calls, tc.want) |
| + } |
| + }) |
| + } |
| +} |
| + |
| // TestLinksUnitShape guards the embedded unit against drift: it must exec the |
| // links subcommand on the socket bootstrap's route registration expects, and |
| // must NOT carry an EnvironmentFile (it holds no secrets — the reason enabling |
diff --git a/deploy/README.md b/deploy/README.md| index 77ebf22..1a66f29 100644 |
| --- a/deploy/README.md |
| +++ b/deploy/README.md |
| @@ -7,6 +7,10 @@ Bare `./deploy/ops` lists the verbs: |
| ```sh |
| ./deploy/ops doctor # can I deploy right now? (read-only preflight) |
| +./deploy/ops validate # run every local check (go|web|shell|infra) |
| +./deploy/ops provision units # re-run bootstrap so NEW systemd units land on the fleet |
| +./deploy/ops provision apex 11ch.at # declare a box the short-link apex |
| +./deploy/ops provision status # read-only: which box runs the short-link redirector |
| ./deploy/ops publish "notes" # build main -> release -> point edge at it |
| ./deploy/ops promote <ver> beta # move a channel pointer (edge|beta|stable) |
| ./deploy/ops ship # every fleet box catches up NOW |
| @@ -16,6 +20,7 @@ Bare `./deploy/ops` lists the verbs: |
| ./deploy/ops dev plan # dev-mesh tofu plan, AMI-replacement guarded |
| ./deploy/ops dev apply # tofu apply behind the same guard |
| ./deploy/ops dev deploy # dev mesh: build -> scp -> swap -> restart |
| +./deploy/ops dev provision # dev mesh: refresh systemd units + route short links |
| ./deploy/ops dev rebuild # destroy + recreate the Dublin box (new keys) |
| ./deploy/ops ledger "why..." # draft the public SSH-access why-entry |
| ``` |
diff --git a/deploy/ops b/deploy/ops| index 22e1e95..85cf7f0 100755 |
| --- a/deploy/ops |
| +++ b/deploy/ops |
| @@ -18,6 +18,9 @@ usage() { |
| ops — the operator command surface |
| ./deploy/ops doctor [section] can I deploy right now? (read-only preflight) |
| + ./deploy/ops validate [section] run every local check (go|web|shell|infra) |
| + ./deploy/ops provision <units|apex|status> [args] |
| + refresh box systemd units / set the short-link apex |
| ./deploy/ops publish "notes" build main -> release -> point edge at it |
| ./deploy/ops promote <ver> <chan> move beta/stable to a published version |
| ./deploy/ops ship [box...] make every box's instances catch up NOW |
| @@ -28,7 +31,9 @@ ops — the operator command surface |
| ./deploy/ops ledger "why..." draft the public SSH-access why-entry |
| Start with `ops doctor` — it checks every credential and access path the other |
| -verbs need, and prints the fix under each failure. Verbs that open box SSH |
| +verbs need, and prints the fix under each failure. `ops validate` is its |
| +code-side twin: every local check (Go, web, shell, tofu) in one pass, needing |
| +no credentials and touching no box — run it before opening a PR. Verbs that open box SSH |
| sessions (ship, canary, dev deploy) incur a why-entry in the public access |
| ledger; `ops ledger` drafts it. |
| @@ -46,7 +51,9 @@ EOF |
| verb="${1:-}" |
| [ $# -gt 0 ] && shift |
| case "$verb" in |
| - doctor) exec "$here/doctor.sh" "$@" ;; |
| + doctor) exec "$here/doctor.sh" "$@" ;; |
| + validate) exec "$here/ops-validate.sh" "$@" ;; |
| + provision) exec "$here/ops-provision.sh" "$@" ;; |
| publish|promote|ship) |
| exec "$here/ops-release.sh" "$verb" "$@" ;; |
| canary) exec "$here/ops-canary.sh" "$@" ;; |
diff --git a/deploy/ops-dev.sh b/deploy/ops-dev.sh| index 0171e0d..9493b5b 100755 |
| --- a/deploy/ops-dev.sh |
| +++ b/deploy/ops-dev.sh |
| @@ -5,6 +5,7 @@ |
| # ./deploy/ops dev plan # tofu plan, with the AMI-replacement guard |
| # ./deploy/ops dev deploy [ie|fi] # build -> scp -> swap -> restart (both by default) |
| # ./deploy/ops dev deploy --stage-only # stop after scp (pair the swap with env edits) |
| +# ./deploy/ops dev provision [ie|fi] # refresh systemd units + route the short-link apex |
| # ./deploy/ops dev rebuild --yes # destroy + recreate the Dublin box (new keys) |
| # |
| # What this encodes that prose used to carry: |
| @@ -24,6 +25,13 @@ |
| # as a .bak for one-command rollback. |
| # - Dublin runs router + cloud + control (restart all three); Helsinki runs |
| # only the router. |
| +# - `deploy` swaps the BINARY; `provision` refreshes the UNITS. They are |
| +# separate because deploy restarts only units that already exist, so a |
| +# newly-added unit never appears on a dev box however often you deploy — |
| +# only `lchat bootstrap` writes unit files. provision runs it with |
| +# --no-update-timer and then asserts the timer really is off, because a |
| +# plain bootstrap would enable the hourly updater and quietly replace the |
| +# unmerged code under test with the current stable build. |
| # - compute.tf's aws_ami is `most_recent = true` with no lifecycle guard: the |
| # day Canonical publishes a newer image, a plain `tofu apply` DESTROYS AND |
| # RECREATES the Dublin box (its /data dies with it; only the EIP survives). |
| @@ -56,13 +64,20 @@ BOX_IE="${LCHAT_DEV_BOX_IE:-ubuntu@54.154.95.205}" |
| BOX_FI="${LCHAT_DEV_BOX_FI:-root@157.180.123.183}" |
| KEY_IE="$HOME/.ssh/eleven-dev-box" |
| AWS_PROFILE="${LCHAT_DEV_AWS_PROFILE:-eleven-dev}" |
| +# The dev mesh's short-link apex — the dev-side stand-in for 11ch.at. It is the |
| +# APEX of the dev subtree, not a host under it, because that is what production |
| +# is: the wildcard cert that covers every space does not cover this name, so |
| +# rehearsing short links here exercises the same apex DNS + certificate path |
| +# that production needs. Its A record is aws_route53_record.dev_apex. |
| +DEV_LINKS_HOST="${LCHAT_DEV_LINKS_HOST:-dev.11ch.at}" |
| ssh_ie() { if [ -f "$KEY_IE" ]; then ssh -i "$KEY_IE" -o ConnectTimeout=10 "$@"; else ssh -o ConnectTimeout=10 "$@"; fi; } |
| scp_ie() { if [ -f "$KEY_IE" ]; then scp -i "$KEY_IE" "$@"; else scp "$@"; fi; } |
| usage() { |
| echo "usage: ops dev plan [tofu args] | ops dev apply [tofu args] |" >&2 |
| - echo " ops dev deploy [ie|fi] [--stage-only] | ops dev rebuild [--yes]" >&2 |
| + echo " ops dev deploy [ie|fi] [--stage-only] | ops dev provision [ie|fi] |" >&2 |
| + echo " ops dev rebuild [--yes]" >&2 |
| exit 2 |
| } |
| @@ -360,6 +375,122 @@ deploy) |
| echo " access-ledger why-entry yet: ./deploy/ops ledger \"...\"" |
| ;; |
| +provision) |
| + # Refresh the dev boxes' systemd units from the binary they are already |
| + # running, and route the short-link redirector at the dev apex. |
| + # |
| + # Why this is a verb and not a documented ssh sequence: `deploy` only swaps |
| + # the binary and restarts the units that already exist, so a NEW unit (like |
| + # lchat-links) never appears on a dev box no matter how many times you |
| + # deploy. Only `lchat bootstrap` writes units — and running plain bootstrap |
| + # here would be a trap, because it enables the hourly auto-updater, which |
| + # would replace this box's unmerged code with the current stable build. The |
| + # whole point of the dev mesh is that it is NOT a release channel. So this |
| + # passes --no-update-timer (which disables the timer outright rather than |
| + # merely skipping it) and then VERIFIES the timer really is off before |
| + # touching anything else. |
| + targets="ie fi" |
| + for a in "$@"; do |
| + case "$a" in |
| + ie|fi) targets="$a" ;; |
| + *) usage ;; |
| + esac |
| + done |
| + |
| + for t in $targets; do |
| + if [ "$t" = ie ]; then box="$BOX_IE"; run() { ssh_ie "$box" "$@"; } |
| + else box="$BOX_FI"; run() { ssh -o ConnectTimeout=10 "$box" "$@"; } |
| + fi |
| + # Dublin holds the EIP that dev.11ch.at points at, so it is the dev apex |
| + # and bootstraps AS it (one step: starts the redirector and registers the |
| + # route). Helsinki is not the apex, so its bootstrap leaves the redirector |
| + # disabled — which is the behaviour worth exercising here too. |
| + if [ "$t" = ie ]; then apexflag="--links-apex $DEV_LINKS_HOST"; else apexflag=""; fi |
| + |
| + # Ordering guard: provision drives flags that only exist in a NEW binary, |
| + # and `deploy` is what puts one on the box. Running these in the wrong |
| + # order otherwise fails deep inside bootstrap's flag parsing with |
| + # "flag provided but not defined", which reads like a bug in this script |
| + # rather than a missing step. |
| + if ! run 'sudo /opt/lchat/lchat bootstrap -h 2>&1' | grep -q 'links-apex'; then |
| + echo "ops dev: ABORT — the binary on $t predates --links-apex." >&2 |
| + echo " provision configures the box; deploy is what ships the code. Run:" >&2 |
| + echo " ./deploy/ops dev deploy $t" >&2 |
| + exit 1 |
| + fi |
| + |
| + echo "ops dev: bootstrapping units on $t ($box)${apexflag:+ (as short-link apex $DEV_LINKS_HOST)}" |
| + run "sudo /opt/lchat/lchat bootstrap --no-update-timer $apexflag" |
| + |
| + # The invariant, asserted rather than assumed: if bootstrap ever regains a |
| + # path that leaves the updater on, this box starts silently overwriting the |
| + # unmerged code under test and the mesh stops meaning anything. |
| + timer="$(run 'systemctl is-enabled lchat-update.timer 2>/dev/null || true')" |
| + case "$timer" in |
| + enabled) |
| + echo "ops dev: ABORT — $t's auto-update timer is ENABLED after bootstrap." >&2 |
| + echo " The dev mesh runs unmerged code; an updater pass would replace it with" >&2 |
| + echo " the current stable build. Disable it before using this box:" >&2 |
| + echo " ssh $box 'sudo systemctl disable --now lchat-update.timer'" >&2 |
| + exit 1 ;; |
| + *) echo " $t: auto-update timer is '$timer' (not enabled — correct for dev)" ;; |
| + esac |
| + |
| + links="$(run 'systemctl is-active lchat-links 2>/dev/null || true')" |
| + if [ -n "$apexflag" ]; then |
| + echo " $t: lchat-links is $links (apex — expected active)" |
| + [ "$links" = active ] || { |
| + echo "ops dev: $t is the apex but its links unit did not start — inspect:" >&2 |
| + echo " ssh $box journalctl -u lchat-links -n 40 --no-pager" >&2 |
| + exit 1 |
| + } |
| + else |
| + echo " $t: lchat-links is $links (not the apex — expected inactive)" |
| + [ "$links" != active ] || echo "ops dev: note — $t runs the redirector but isn't the apex" >&2 |
| + fi |
| + done |
| + |
| + # bootstrap --links-apex registered the route above, so there is no separate |
| + # add-service step: prove the result instead. |
| + case "$targets" in *ie*) |
| + # Prove it end to end. The certificate is issued lazily on first handshake, |
| + # so a cold apex can take a few seconds — and a TLS failure here is the |
| + # exact symptom of the production bug this rehearses (an unrouted host gets |
| + # no certificate at all, so it fails the handshake rather than 404ing). |
| + echo "ops dev: checking https://$DEV_LINKS_HOST/ ..." |
| + apex="" |
| + for _ in 1 2 3 4 5 6; do |
| + apex="$(curl -sS -o /dev/null -m 15 -w '%{http_code} -> %{redirect_url}' "https://$DEV_LINKS_HOST/" 2>&1 || true)" |
| + case "$apex" in 30*) break ;; esac |
| + sleep 5 |
| + done |
| + echo " apex: $apex" |
| + echo " (expect a 30x to elevenmessenger.com — that is the redirector answering)" |
| + case "$apex" in |
| + 30*) ;; |
| + *) echo "ops dev: the apex did not redirect. If this is a TLS error, check the" >&2 |
| + echo " A record exists (ops dev apply lands aws_route53_record.dev_apex) —" >&2 |
| + echo " a *.dev.11ch.at wildcard does NOT cover the bare apex." >&2 |
| + exit 1 ;; |
| + esac |
| + |
| + # /l/<host> only redirects to a host that verifiably runs lchat, so point it |
| + # at a real dev space rather than a made-up name. |
| + probe="${LCHAT_DEV_LINK_PROBE:-}" |
| + if [ -n "$probe" ]; then |
| + echo "ops dev: checking https://$DEV_LINKS_HOST/l/$probe ..." |
| + echo " /l/: $(curl -sS -o /dev/null -m 20 -w '%{http_code} -> %{redirect_url}' "https://$DEV_LINKS_HOST/l/$probe" 2>&1 || true)" |
| + echo " (expect a 30x to https://$probe — a 404 means the target failed verification)" |
| + else |
| + echo "ops dev: set LCHAT_DEV_LINK_PROBE=<a live dev space host> to also test /l/<host>" |
| + fi |
| + ;; esac |
| + |
| + echo |
| + echo "note: this opened dev-box SSH sessions — if this session hasn't written its" |
| + echo " access-ledger why-entry yet: ./deploy/ops ledger \"...\"" |
| + ;; |
| + |
| rebuild) |
| # Destroy + recreate the Dublin box — the deliberate key rotation. EC2 |
| # injects the keypair only at first boot, so this is how you take over a box |
diff --git a/deploy/ops-provision.sh b/deploy/ops-provision.sh| new file mode 100755 |
| index 0000000..ff2544a |
| --- /dev/null |
| +++ b/deploy/ops-provision.sh |
| @@ -0,0 +1,151 @@ |
| +#!/usr/bin/env sh |
| +# 🤖 ops provision — refresh a fleet box's systemd units, and declare which box |
| +# is the short-link apex. |
| +# |
| +# ./deploy/ops provision units [box...] # refresh unit files (lchat bootstrap) |
| +# ./deploy/ops provision apex <host> [box] # ...and make that box the apex for <host> |
| +# ./deploy/ops provision status [box...] # read-only: who runs what |
| +# |
| +# Why this verb exists: **no new systemd unit ever reaches an existing box on |
| +# its own.** `deploy` and `ship` swap binaries and restart units that already |
| +# exist; the hourly auto-updater does the same. Only `lchat bootstrap` writes |
| +# unit files, and nothing re-runs it after the first provision. So a unit added |
| +# to the repo lands on fresh boxes and silently never appears on the fleet — |
| +# which is exactly how the short-link redirector shipped in the binary and ran |
| +# on no box at all (issue #905). |
| +# |
| +# `provision units` is therefore the missing fleet-side counterpart to |
| +# `ops dev provision`, and the thing to run after merging any change that adds |
| +# or edits a unit file. |
| +# |
| +# What it does NOT do: touch /data, instances, or versions. `bootstrap` is |
| +# idempotent and data-safe — it writes units, ensures directories and the |
| +# service user, and restarts the ROUTER. That router restart is the one |
| +# member-visible effect (a sub-second blip on a retrying router), which is why |
| +# this is a deliberate verb you run, never something folded into a deploy. |
| +# |
| +# NEVER point this at a dev-mesh box. Plain `bootstrap` ENABLES the hourly |
| +# auto-updater, which would replace the dev mesh's unmerged code with the |
| +# current stable build — `ops dev provision` exists for those boxes and passes |
| +# --no-update-timer. The only guardrail here is the $LCHAT_BOXES default, so |
| +# be deliberate when overriding it or naming boxes explicitly. |
| +# |
| +# `provision apex` additionally passes --links-apex, which starts the |
| +# redirector and registers <host> as a route pointing at its socket. Exactly |
| +# one box in a fleet is the apex; every other box keeps the unit written and |
| +# disabled. After the first time, the registry route is the source of truth — |
| +# plain `provision units` converges to it, so the apex survives later runs |
| +# without anyone remembering a flag. |
| +set -eu |
| + |
| +# Same default fleet as doctor/ship, same override. |
| +BOXES="${LCHAT_BOXES:-elevenbox kepler}" |
| +# The box that fronts the short-link apex, when `apex` is given no box. |
| +APEX_BOX="${LCHAT_APEX_BOX:-elevenbox}" |
| + |
| +usage() { |
| + echo "usage: ops provision units [box...] | ops provision apex <host> [box] |" >&2 |
| + echo " ops provision status [box...]" >&2 |
| + echo >&2 |
| + echo " units re-run 'lchat bootstrap' so newly-added systemd units land (restarts the router)" >&2 |
| + echo " apex the same, declaring that box the short-link apex for <host>" >&2 |
| + echo " status read-only: unit + route state, no changes" >&2 |
| + exit 2 |
| +} |
| + |
| +# on <box> <command>: privileged box command through the committed sudo |
| +# allowlist (`/opt/lchat/lchat *`, deploy/sudoers-operators). Operator SSH |
| +# rides WireGuard and the username comes from ~/.ssh/config — root SSH is |
| +# bootstrap-only, so no user is baked in here. |
| +on() { box="$1"; shift; ssh -o ConnectTimeout=10 "$box" "$@"; } |
| + |
| +# report <box>: the read-only picture — is the redirector running here, and |
| +# which host (if any) routes to it. |
| +report() { |
| + box="$1" |
| + printf ' %s:\n' "$box" |
| + # is-enabled/is-active exit non-zero for the states we most want to SEE |
| + # (disabled, inactive, not-found), and is-enabled adds a stderr explanation |
| + # for a missing unit — so discard stderr, keep the first line, and only |
| + # substitute a word when there was genuinely no output. |
| + enabled="$(on "$box" 'systemctl is-enabled lchat-links 2>/dev/null | head -1' || true)" |
| + active="$(on "$box" 'systemctl is-active lchat-links 2>/dev/null | head -1' || true)" |
| + printf ' lchat-links %s / %s\n' "${enabled:-not-found}" "${active:-inactive}" |
| + route="$(on "$box" "sudo /opt/lchat/lchat instances 2>/dev/null | grep -F '/run/lchat/links.sock' || true")" |
| + if [ -n "$route" ]; then |
| + printf ' apex route %s\n' "$(printf '%s' "$route" | awk '{print $1}')" |
| + else |
| + printf ' apex route (none — not the short-link apex)\n' |
| + fi |
| +} |
| + |
| +cmd="${1:-}" |
| +[ $# -gt 0 ] && shift |
| +case "$cmd" in |
| + |
| +units) |
| + boxes="${*:-$BOXES}" |
| + echo "ops provision: refreshing systemd units on: $boxes" |
| + echo " (this restarts the router on each — a sub-second blip, not a data operation)" |
| + for box in $boxes; do |
| + echo "ops provision: $box" |
| + on "$box" 'sudo /opt/lchat/lchat bootstrap' |
| + report "$box" |
| + done |
| + echo |
| + echo "note: this opened box SSH sessions — if this session hasn't written its" |
| + echo " access-ledger why-entry yet: ./deploy/ops ledger \"...\"" |
| + ;; |
| + |
| +apex) |
| + host="${1:-}"; [ -n "$host" ] || usage |
| + shift |
| + box="${1:-$APEX_BOX}" |
| + |
| + # The apex is a real, public hostname whose DNS must already point at this |
| + # box — the router issues its certificate lazily on first handshake, and an |
| + # apex is NOT covered by a *.domain wildcard, so this is the one part the |
| + # script can't do for you. Say so plainly rather than failing later at TLS. |
| + echo "ops provision: making $box the short-link apex for $host" |
| + echo " prerequisite: $host must already resolve to $box, with its OWN A record." |
| + # Only worth spelling out where a covering wildcard plausibly exists: a host |
| + # under a parent domain (dev.11ch.at), not a registrable apex (11ch.at), |
| + # where "a *.ch.at wildcard" would be nonsense. |
| + case "$host" in |
| + *.*.*) echo " A *.${host#*.} wildcard does NOT cover $host — an apex never is." ;; |
| + esac |
| + on "$box" "sudo /opt/lchat/lchat bootstrap --links-apex $host" |
| + report "$box" |
| + |
| + echo "ops provision: checking https://$host/ ..." |
| + # The certificate is issued on the first handshake, so a cold apex can take a |
| + # few seconds. A TLS failure here (rather than an HTTP status) almost always |
| + # means DNS doesn't point at this box, or the route didn't register. |
| + apex="" |
| + for _ in 1 2 3 4 5 6; do |
| + apex="$(curl -sS -o /dev/null -m 15 -w '%{http_code} -> %{redirect_url}' "https://$host/" 2>&1 || true)" |
| + case "$apex" in 30*) break ;; esac |
| + sleep 5 |
| + done |
| + echo " apex: $apex" |
| + case "$apex" in |
| + 30*) echo " (a 30x to the marketing site is the redirector answering — short links are live)" ;; |
| + *) echo "ops provision: $host did not redirect." >&2 |
| + echo " TLS error -> DNS probably doesn't point at $box, or the route didn't register" >&2 |
| + echo " 404/timeout -> check: ssh $box journalctl -u lchat-links -n 40 --no-pager" >&2 |
| + exit 1 ;; |
| + esac |
| + echo |
| + echo "note: this opened box SSH sessions — if this session hasn't written its" |
| + echo " access-ledger why-entry yet: ./deploy/ops ledger \"...\"" |
| + ;; |
| + |
| +status) |
| + boxes="${*:-$BOXES}" |
| + echo "ops provision: short-link state" |
| + for box in $boxes; do report "$box"; done |
| + ;; |
| + |
| +-h|--help|help|"") usage ;; |
| +*) echo "ops provision: unknown subcommand '$cmd'" >&2; usage ;; |
| +esac |
diff --git a/deploy/ops-validate.sh b/deploy/ops-validate.sh| new file mode 100755 |
| index 0000000..9a7dc8c |
| --- /dev/null |
| +++ b/deploy/ops-validate.sh |
| @@ -0,0 +1,182 @@ |
| +#!/usr/bin/env sh |
| +# 🤖 ops validate — run every check this repo can run locally, in one command. |
| +# |
| +# ./deploy/ops validate # everything (what to run before opening a PR) |
| +# ./deploy/ops validate go # Go build + vet + tests |
| +# ./deploy/ops validate web # node --check on the ES modules + npm test |
| +# ./deploy/ops validate shell # sh -n (+ shellcheck if present) on deploy/ |
| +# ./deploy/ops validate infra # tofu fmt + validate on the dev-mesh stack |
| +# |
| +# Why this is a verb: the checks are spread across four toolchains, and every |
| +# one of them is easy to forget — the shell scripts and the tofu stack in |
| +# particular have no test suite and are only ever caught by running these by |
| +# hand. A contributor (human or agent) should not have to reconstruct the list |
| +# from CLAUDE.md; ops is where operational knowledge lives, and "what do I run |
| +# before pushing" is exactly that. |
| +# |
| +# Non-goals: this never touches a box, a credential, or the network beyond what |
| +# `go test` and `npm test` already do (both are hermetic — httptest servers and |
| +# Node's test runner). It is safe to run at any time, on any branch. |
| +# |
| +# Exit status is the number of FAILED sections (0 = all good), and every section |
| +# runs even when an earlier one fails — one pass tells you everything that is |
| +# broken, rather than making you re-run after each fix. |
| +set -eu |
| + |
| +here="$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)" |
| +root="$(CDPATH= cd -- "$here/.." && pwd)" |
| +cd "$root" |
| + |
| +bold=''; red=''; green=''; yellow=''; off='' |
| +if [ -t 1 ]; then |
| + bold=$(printf '\033[1m'); red=$(printf '\033[31m'); green=$(printf '\033[32m') |
| + yellow=$(printf '\033[33m'); off=$(printf '\033[0m') |
| +fi |
| + |
| +failed=0 |
| +skipped=0 |
| + |
| +section() { printf '\n%s%s%s\n' "$bold" "$1" "$off"; } |
| +ok() { printf ' %sok%s %s\n' "$green" "$off" "$1"; } |
| +skip() { printf ' %sskip%s %s\n' "$yellow" "$off" "$1"; skipped=$((skipped + 1)); } |
| +bad() { printf ' %sFAIL%s %s\n' "$red" "$off" "$1"; failed=$((failed + 1)); } |
| + |
| +# run <label> <command...>: run it, show output only when it fails. Keeping a |
| +# passing run silent is what makes the summary readable; a failure prints |
| +# everything, because that is the moment you need it. |
| +run() { |
| + label="$1"; shift |
| + out="$(mktemp -t lchat-validate.XXXXXX)" |
| + if "$@" >"$out" 2>&1; then |
| + ok "$label" |
| + else |
| + bad "$label" |
| + sed 's/^/ /' "$out" |
| + fi |
| + rm -f "$out" |
| +} |
| + |
| +validate_go() { |
| + section "Go" |
| + if ! command -v go >/dev/null 2>&1; then skip "go not installed"; return; fi |
| + run "build (all packages)" go build ./... |
| + # vet is advisory here: the tree carries a pre-existing finding in |
| + # checkpoint_test.go, so a failure must not mask a real regression elsewhere. |
| + # Shown, not counted. |
| + if go vet ./... >/dev/null 2>&1; then |
| + ok "vet" |
| + else |
| + printf ' %swarn%s vet reported findings:\n' "$yellow" "$off" |
| + go vet ./... 2>&1 | sed 's/^/ /' || true |
| + fi |
| + run "tests" go test ./... |
| +} |
| + |
| +validate_web() { |
| + section "Web" |
| + if command -v node >/dev/null 2>&1; then |
| + # The ES modules are never executed by the test suite, so a syntax error |
| + # ships silently — `node --check` is the only thing standing in the way. |
| + bad_files='' |
| + for f in web/static/*.js web/static/apps/*.js; do |
| + [ -f "$f" ] || continue |
| + node --check "$f" >/dev/null 2>&1 || bad_files="$bad_files $f" |
| + done |
| + if [ -n "$bad_files" ]; then |
| + bad "node --check —$bad_files" |
| + for f in $bad_files; do node --check "$f" 2>&1 | sed 's/^/ /' || true; done |
| + else |
| + ok "node --check (every web/static ES module parses)" |
| + fi |
| + else |
| + skip "node not installed" |
| + fi |
| + if [ -f package.json ] && command -v npm >/dev/null 2>&1; then |
| + run "npm test (crypto, ui, format, event vectors)" npm test --silent |
| + else |
| + skip "npm test (no npm or package.json)" |
| + fi |
| +} |
| + |
| +validate_shell() { |
| + section "Shell" |
| + # These scripts have no test suite, so a parse check is the floor. deploy/ |
| + # deliberately mixes sh and bash (ops and its verbs are POSIX sh; the |
| + # collectors are bash and use arrays and `local`), so each file is checked |
| + # with the interpreter its OWN shebang names — forcing `sh -n` on a bash |
| + # script reports syntax errors that are not there. |
| + bad_files='' |
| + for f in "$here"/ops "$here"/*.sh; do |
| + [ -f "$f" ] || continue |
| + case "$(head -1 "$f")" in |
| + *bash) shell='bash' ;; |
| + *) shell='sh' ;; |
| + esac |
| + "$shell" -n "$f" >/dev/null 2>&1 || bad_files="$bad_files $(basename "$f")" |
| + done |
| + if [ -n "$bad_files" ]; then |
| + bad "parse —$bad_files" |
| + for f in $bad_files; do |
| + case "$(head -1 "$here/$f")" in *bash) shell='bash' ;; *) shell='sh' ;; esac |
| + "$shell" -n "$here/$f" 2>&1 | sed 's/^/ /' || true |
| + done |
| + else |
| + ok "parse (every deploy/ script, per its own shebang)" |
| + fi |
| + |
| + if command -v shellcheck >/dev/null 2>&1; then |
| + # Advisory, and shebang-driven (no -s override, same reason as above). The |
| + # tree carries known pre-existing findings (SC1007/SC1091/SC2016), so |
| + # counting them would make this verb useless on day one: it reports and |
| + # never fails. Read it when you touch a script. |
| + if shellcheck "$here"/ops "$here"/*.sh >/dev/null 2>&1; then |
| + ok "shellcheck" |
| + else |
| + printf ' %swarn%s shellcheck findings (advisory — pre-existing ones exist):\n' "$yellow" "$off" |
| + shellcheck "$here"/ops "$here"/*.sh 2>&1 | grep -E '^In ' | sed "s|$root/||" | head -30 | sed 's/^/ /' || true |
| + fi |
| + else |
| + skip "shellcheck not installed (brew install shellcheck)" |
| + fi |
| +} |
| + |
| +validate_infra() { |
| + section "Infra (dev-mesh tofu)" |
| + stack="$root/infra/dev-mesh" |
| + if [ ! -d "$stack" ]; then skip "no infra/dev-mesh"; return; fi |
| + if ! command -v tofu >/dev/null 2>&1; then skip "tofu not installed"; return; fi |
| + # fmt + validate only: both are offline and read-only. `plan` needs AWS |
| + # credentials and talks to the provider, so it stays in `ops dev plan` where |
| + # the credential handling lives — validate must never need a secret. |
| + run "tofu fmt" tofu -chdir="$stack" fmt -check -diff |
| + if [ -d "$stack/.terraform" ]; then |
| + run "tofu validate" tofu -chdir="$stack" validate |
| + else |
| + skip "tofu validate (not initialised — run: ops dev plan)" |
| + fi |
| +} |
| + |
| +what="${1:-all}" |
| +case "$what" in |
| + all) validate_go; validate_web; validate_shell; validate_infra ;; |
| + go) validate_go ;; |
| + web) validate_web ;; |
| + shell) validate_shell ;; |
| + infra) validate_infra ;; |
| + -h|--help|help) |
| + sed -n '2,8p' "$0" | sed 's/^# \{0,1\}//' |
| + exit 0 ;; |
| + *) |
| + echo "ops validate: unknown section '$what' (all|go|web|shell|infra)" >&2 |
| + exit 2 ;; |
| +esac |
| + |
| +printf '\n' |
| +if [ "$failed" -eq 0 ]; then |
| + printf '%s%s%s' "$green" "all checks passed" "$off" |
| + [ "$skipped" -gt 0 ] && printf ' (%d skipped)' "$skipped" |
| + printf '\n' |
| + exit 0 |
| +fi |
| +printf '%s%d section(s) FAILED%s\n' "$red" "$failed" "$off" |
| +exit "$failed" |
diff --git a/docs/planet-scale-guide/reference.html b/docs/planet-scale-guide/reference.html| index ee3a679..a06b073 100644 |
| --- a/docs/planet-scale-guide/reference.html |
| +++ b/docs/planet-scale-guide/reference.html |
| @@ -27,8 +27,8 @@ |
| <h3 id="cmd-lifecycle">Box and instance lifecycle</h3> |
| <dl> |
| - <dt><code>lchat bootstrap</code></dt> |
| - <dd>One-time box setup: writes every systemd unit file, creates the <code>lchat</code> system user and the <code>/data</code>, <code>/run/lchat</code>, and certificate-cache directories, mints the root-only control token, and enables and starts the router and the hourly update timer. Idempotent — safe to re-run to refresh unit files, and re-running is the <em>only</em> thing that installs a newly-added unit on an existing box (<code>deploy</code> restarts only units that already exist). <code>--links-apex <host></code> additionally makes this box the <a href="#cmd-links">short-link apex</a>. Explained in <a href="deploys.html#bootstrap">part 04</a>.</dd> |
| + <dt><code>lchat bootstrap [--links-apex <host>] [--no-update-timer] [--no-start] [--collectors-only]</code></dt> |
| + <dd>One-time box setup: writes every systemd unit file, creates the <code>lchat</code> system user and the <code>/data</code>, <code>/run/lchat</code>, and certificate-cache directories, mints the root-only control token, and enables and starts the router and the hourly update timer. Idempotent — safe to re-run to refresh unit files, and re-running is the <em>only</em> thing that installs a newly-added unit on an existing box (<code>deploy</code> restarts only units that already exist). <code>--links-apex <host></code> makes this box the <a href="#cmd-links">short-link apex</a> for that host; without it, the box converges to whatever the registry already says. <code>--no-update-timer</code> <strong>disables</strong> the hourly auto-updater rather than enabling it, for boxes that must never track a release channel — the dev mesh runs unmerged code, so an updater pass would replace it with the current stable build. It disables rather than merely skips, so the end state follows the flag rather than the box’s history. Explained in <a href="deploys.html#bootstrap">part 04</a>.</dd> |
| <dt><code>lchat add-instance <host> [--admin-token <tok>] [--hibernate] [--home-region <cc>] [--app-store-url <url>] [--home-origin <url>] [--push-relay-url <url>] [--push-relay-token <tok>]</code></dt> |
| <dd>Creates a space: mints (or, with <code>--admin-token</code>, imports) the admin token, writes <code>/data/<host>.env</code>, registers the route, points the version link, starts and health-checks the unit (skipped with <code>--hibernate</code>), reloads the router, warms the certificate, and prints the one-time setup link and the admin link. Explained in <a href="spaces.html#adding-manual">part 05</a>.</dd> |
| <dt><code>lchat add-service <host> --socket <path></code></dt> |
diff --git a/infra/dev-mesh/route53.tf b/infra/dev-mesh/route53.tf| index 2c149dc..bb1b0ab 100644 |
| --- a/infra/dev-mesh/route53.tf |
| +++ b/infra/dev-mesh/route53.tf |
| @@ -32,6 +32,27 @@ resource "aws_route53_record" "dev_wildcard" { |
| records = [aws_eip.ie_node.public_ip] |
| } |
| +# The APEX of the dev subtree — dev.11ch.at itself, with no label in front. A |
| +# wildcard does NOT cover the bare name it wildcards, so without this record |
| +# dev.11ch.at resolves nowhere no matter how many *.dev.11ch.at hosts exist. |
| +# |
| +# It exists for the short-link redirector (lchat-links.service, issue #905), |
| +# which is an apex service by nature: production serves 11ch.at itself, so the |
| +# dev mesh has to serve dev.11ch.at itself to rehearse it at all. Nothing else |
| +# needs the apex today. |
| +# |
| +# This is the dev-side mirror of a real production gap: 11ch.at's own apex A |
| +# record and route are what were missing, so the redirector had nowhere to be |
| +# reached even after its code merged — a wildcard cert on *.11ch.at made it |
| +# look configured while the apex failed the TLS handshake outright. |
| +resource "aws_route53_record" "dev_apex" { |
| + zone_id = aws_route53_zone.dev.zone_id |
| + name = "dev.11ch.at" |
| + type = "A" |
| + ttl = 60 |
| + records = [aws_eip.ie_node.public_ip] |
| +} |
| + |
| # The two per-country wildcards, MOVED here from DNSimple (they were in the old |
| # dns.tf). The delegation makes DNSimple non-authoritative for the whole |
| # dev.11ch.at subtree, so these must live in THIS zone or every existing |