| 1 | # 🤖 Multi-edge Phase 1 — dev cutover runbook (region-neutral addressing) |
| 2 | |
| 3 | The ordered, hand-runnable steps to flip the two-node dev mesh onto |
| 4 | **region-neutral addressing**: one shared `dev.11ch.at` namespace, a shared |
| 5 | `*.dev.11ch.at` wildcard cert over Route53 DNS-01, and cross-region serving that |
| 6 | returns **200** (not the 503 the country-scoped mesh gave). Design + rationale: |
| 7 | `docs/multi-edge.md`. The tofu it drives: `infra/dev-mesh/`. |
| 8 | |
| 9 | This is a **dev-only, reversible** change. Prod (`11ch.at` on DNSimple) is |
| 10 | untouched except for one scoped `dev` NS delegation record. |
| 11 | |
| 12 | > 🤖 **Historical record (2026-07) — not a procedure.** This page documents |
| 13 | > the cutover as it was performed and the gotchas learned doing it; it is |
| 14 | > kept as design history. **Do not operate from it.** The dev mesh is |
| 15 | > operated by `./deploy/ops dev plan|apply|deploy|rebuild` — `rebuild` now |
| 16 | > writes the env files below, pairs the mesh token, and seeds routes itself. |
| 17 | > Anything the verbs can't do gets **added to them** (extend ops, then run |
| 18 | > it), never hand-run from this page. |
| 19 | |
| 20 | --- |
| 21 | |
| 22 | ## The end state (what "done" looks like) |
| 23 | |
| 24 | - New spaces mint as `<sqid>.dev.11ch.at` — no country label — their ids drawn |
| 25 | from the shared global ledger (`internal/idalloc`), so two region clouds never |
| 26 | collide. |
| 27 | - **Both** boxes serve the single mesh domain `dev.11ch.at`, hold the shared |
| 28 | `*.dev.11ch.at` wildcard cert, and place every space by its replicated |
| 29 | `home_region` (mesh.go). Either box can receive any region-neutral host and |
| 30 | either serve it (its region owns it) or reverse-proxy to the box that does. |
| 31 | - Cross-region serve returns 200: a `home_region=fi` host hit at Dublin is |
| 32 | proxied to Helsinki, which terminates TLS with the shared wildcard, finds the |
| 33 | replicated route, and serves. |
| 34 | |
| 35 | Why this fixes the 503: today the two boxes serve **different** |
| 36 | `LCHAT_MESH_DOMAINS` (`ie.dev.11ch.at` vs `fi.dev.11ch.at`). A cross-region host |
| 37 | is in-scope on the sender but out-of-scope on the receiver, so the receiver |
| 38 | serves locally with no route and no cert → "acme: host not in registry". One |
| 39 | shared domain makes it in-scope on both, plain domain-scoped route replication |
| 40 | copies every `*.dev.11ch.at` route to both boxes, and the shared wildcard lets |
| 41 | the receiver terminate the proxied HTTPS hop. (Region-*aware* replication — |
| 42 | branch `multi-edge-p3-repl` — is only a scale optimisation for many boxes on |
| 43 | different domains; two boxes on one domain don't need it.) |
| 44 | |
| 45 | --- |
| 46 | |
| 47 | ## Boxes, as found (2026-07-18) |
| 48 | |
| 49 | | | Dublin (ie) | Helsinki (fi) | |
| 50 | |---|---|---| |
| 51 | | reach | `ubuntu@54.154.95.205 -i ~/.ssh/eleven-dev-box` (passwordless sudo) | `root@157.180.123.183` (1Password hetzner agent key) | |
| 52 | | arch | arm64 (t4g.small) | amd64 | |
| 53 | | AWS creds | EC2 **instance role** `eleven-dev-mesh-ie` (no keys on box) | scoped IAM **user** `eleven-dev-mesh-fi` (`AWS_ACCESS_KEY_ID`/`_SECRET` in router.env) | |
| 54 | | router svc | `lchat-router` → `/opt/lchat/lchat` | `lchat-router` → `/opt/lchat/lchat` | |
| 55 | | cloud/control | `lchat-cloud`/`lchat-control` → `/opt/lchat/lchat-banner` (**separate binary!**) | → `/opt/lchat/lchat` (unified) | |
| 56 | |
| 57 | ⚠️ **Both boxes' `/opt/lchat/lchat` were rebuilt by hand on Jul 18** with |
| 58 | ambiguous stamps (Dublin's reports version `dev`). Don't trust the in-place |
| 59 | binaries — **Step 0 rebuilds both from a known main SHA** so the running code is |
| 60 | certain. On Dublin, `lchat-cloud`/`lchat-control` exec a *different* binary |
| 61 | (`lchat-banner`, `f3fa623b`); this cutover unifies them onto the fresh binary so |
| 62 | the cloud has the id-ledger + region-neutral minting code (that binary predates |
| 63 | `#1103`/`#1109`). Backups already on each box: `lchat.bak-mesh` (dcc83d0a), |
| 64 | `lchat.bak-multiedge` (90a6dee0). |
| 65 | |
| 66 | All of `#1093 #1094 #1103 #1106 #1109` (placement, provisioner stamp, id ledger, |
| 67 | Route53 provider, region-neutral minting) are on **main** — a plain build of |
| 68 | current main has everything. |
| 69 | |
| 70 | --- |
| 71 | |
| 72 | ## Step 0 — build fresh binaries from main (both arches) |
| 73 | |
| 74 | From a clean `main` checkout (record the SHA — it stamps the build): |
| 75 | |
| 76 | ```sh |
| 77 | # Dublin is arm64, Helsinki amd64 — build both. |
| 78 | GOOS=linux GOARCH=arm64 CGO_ENABLED=0 go build -trimpath \ |
| 79 | -ldflags "-X main.buildVersion=$(git rev-parse --short HEAD)" -o lchat-linux-arm64 . |
| 80 | GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -trimpath \ |
| 81 | -ldflags "-X main.buildVersion=$(git rev-parse --short HEAD)" -o lchat-linux-amd64 . |
| 82 | # (deploy/build-linux.sh builds amd64 the same way; arm64 needs the explicit GOARCH.) |
| 83 | ``` |
| 84 | |
| 85 | Ship + stage (do NOT overwrite the live binary yet — stage beside it): |
| 86 | |
| 87 | ```sh |
| 88 | scp lchat-linux-arm64 ubuntu@54.154.95.205:/opt/lchat/incoming/lchat-new |
| 89 | scp lchat-linux-amd64 root@157.180.123.183:/opt/lchat/incoming/lchat-new # (mkdir -p /opt/lchat/incoming first if absent) |
| 90 | ``` |
| 91 | |
| 92 | The binary swap itself happens in Step 3 (after the tofu + DNS are ready), so |
| 93 | router and cloud restart once, onto both the new binary *and* the new env. |
| 94 | |
| 95 | --- |
| 96 | |
| 97 | ## Step 1 — apply the Route53 tofu (from your Mac, with the vault token) |
| 98 | |
| 99 | `infra/dev-mesh/` now creates the `dev.11ch.at` Route53 zone, its three |
| 100 | wildcards (`*.dev`, `*.ie.dev`, `*.fi.dev`), the `dev` NS delegation into the |
| 101 | prod DNSimple zone, and the DNS-01 IAM on both boxes' role/user. The DNSimple |
| 102 | 176940 token comes from the vault at apply time — **never a box**. |
| 103 | |
| 104 | ```sh |
| 105 | # `ops dev plan`/`apply` sort the credentials out themselves (AWS_* + |
| 106 | # DNSIMPLE_* from Bitwarden — the sops vault is retired): |
| 107 | ./deploy/ops dev plan # expect: 10 to add, 0 to change, 2 to destroy |
| 108 | ./deploy/ops dev apply |
| 109 | ``` |
| 110 | |
| 111 | The plan (validated 2026-07-18): **+** zone, `*.dev`/`*.ie.dev`/`*.fi.dev` |
| 112 | records, 4×`dev` NS, 2× DNS-01 IAM policies; **−** the two old DNSimple |
| 113 | `*.ie.dev`/`*.fi.dev` A records (superseded by the ones now inside the delegated |
| 114 | zone). Nothing else changes — the box, bucket, existing IAM, and the |
| 115 | `home.dev.elevenmessenger.com` record are untouched. |
| 116 | |
| 117 | Note the zone nameservers for the next check: |
| 118 | |
| 119 | ```sh |
| 120 | AWS_PROFILE=eleven-dev tofu output dev_zone_nameservers |
| 121 | ``` |
| 122 | |
| 123 | --- |
| 124 | |
| 125 | ## Step 2 — verify delegation propagates |
| 126 | |
| 127 | Delegation TTL is short but resolvers cache the parent NS; give it a few minutes. |
| 128 | |
| 129 | ```sh |
| 130 | dig +short NS dev.11ch.at # must return the 4 Route53 nameservers |
| 131 | dig +short one.ie.dev.11ch.at # must still be 54.154.95.205 (now via Route53) |
| 132 | dig +short one.fi.dev.11ch.at # must still be 157.180.123.183 |
| 133 | dig +short anything.dev.11ch.at # must be 54.154.95.205 (the *.dev wildcard → Dublin) |
| 134 | ``` |
| 135 | |
| 136 | Do NOT proceed to the wildcard cert (Step 3) until `dig NS dev.11ch.at` shows |
| 137 | Route53 — Let's Encrypt validates the DNS-01 TXT against the authoritative |
| 138 | nameservers, which are Route53 only after delegation lands. |
| 139 | |
| 140 | --- |
| 141 | |
| 142 | ## Step 3 — reconfigure + swap the boxes |
| 143 | |
| 144 | Env deltas (add/modify; everything else stays). Edit `/data/router.env` and, on |
| 145 | Dublin, `/data/cloud.env`; then swap the binary and restart. |
| 146 | |
| 147 | ### Dublin (ie) — `/data/router.env` |
| 148 | ``` |
| 149 | LCHAT_MESH_DOMAINS=dev.11ch.at # was ie.dev.11ch.at |
| 150 | LCHAT_DNS_PROVIDER=route53 # new — the AWS DNS-01 off-ramp |
| 151 | LCHAT_WILDCARD_DOMAINS=dev.11ch.at # new — obtain *.dev.11ch.at |
| 152 | # unchanged: AWS_REGION=eu-west-1, LCHAT_HIBERNATE_BUCKET, LCHAT_MESH_REGION=ie, |
| 153 | # LCHAT_MESH_SELF=https://one.ie.dev.11ch.at, LCHAT_MESH_TOKEN |
| 154 | ``` |
| 155 | |
| 156 | ### Dublin (ie) — `/data/cloud.env` |
| 157 | ``` |
| 158 | LCHAT_CLOUD_BASE_DOMAIN=dev.11ch.at # was ie.dev.11ch.at — mint <sqid>.dev.11ch.at |
| 159 | LCHAT_CLOUD_ID_LEDGER=1 # new — draw ids from the global ledger |
| 160 | LCHAT_HIBERNATE_BUCKET=eleven-dev-mesh-427880590669 # new — where the ledger object lives |
| 161 | LCHAT_HIBERNATE_REGION=eu-west-1 # new — so idalloc's S3 client resolves the region |
| 162 | # unchanged: LCHAT_CLOUD_REGION=ie, LCHAT_ADMIN_TOKEN, demo/pool vars |
| 163 | ``` |
| 164 | > `dev.11ch.at` is a **fresh** namespace (no spaces minted under it), so the |
| 165 | > ledger counter starts at 0 → first id 1 → no collision. **No counter seed |
| 166 | > needed.** (Seeding is only required when flipping the ledger on for a |
| 167 | > namespace that already has spaces — see `newCloudLedger`'s warning.) |
| 168 | |
| 169 | ### Helsinki (fi) — `/data/router.env` |
| 170 | ``` |
| 171 | LCHAT_MESH_DOMAINS=dev.11ch.at # was fi.dev.11ch.at |
| 172 | LCHAT_DNS_PROVIDER=route53 # new |
| 173 | LCHAT_WILDCARD_DOMAINS=dev.11ch.at # new |
| 174 | # unchanged: AWS_ACCESS_KEY_ID/_SECRET (scoped user, now also Route53-DNS01-scoped), |
| 175 | # AWS_REGION=eu-west-1, LCHAT_HIBERNATE_BUCKET, LCHAT_MESH_REGION=fi, |
| 176 | # LCHAT_MESH_SELF=https://one.fi.dev.11ch.at, LCHAT_MESH_TOKEN |
| 177 | # (no cloud on Helsinki — nothing else to change.) |
| 178 | ``` |
| 179 | |
| 180 | ### Swap + restart (each box) |
| 181 | ```sh |
| 182 | # back up the current live binary, move the fresh one in: |
| 183 | sudo cp /opt/lchat/lchat /opt/lchat/lchat.bak-preroute53 |
| 184 | sudo mv /opt/lchat/incoming/lchat-new /opt/lchat/lchat |
| 185 | sudo chmod +x /opt/lchat/lchat |
| 186 | # Dublin only: point cloud/control at the fresh unified binary (drop lchat-banner) |
| 187 | # — edit the units' ExecStart to /opt/lchat/lchat, or symlink lchat-banner→lchat, |
| 188 | # then `sudo systemctl daemon-reload`. |
| 189 | sudo systemctl restart lchat-router |
| 190 | sudo systemctl restart lchat-cloud lchat-control # Dublin (Helsinki has none) |
| 191 | ``` |
| 192 | |
| 193 | The router obtains the `*.dev.11ch.at` wildcard on startup (its `run()` loop |
| 194 | checks immediately). Watch for it: |
| 195 | |
| 196 | ```sh |
| 197 | journalctl -u lchat-router -n 60 --no-pager | grep -iE "wildcard|route53|acme" |
| 198 | # want: "router: wildcard *.dev.11ch.at obtained (valid until …)" |
| 199 | ``` |
| 200 | |
| 201 | --- |
| 202 | |
| 203 | ## Step 4 — verify the wildcard cert (both boxes) |
| 204 | |
| 205 | ```sh |
| 206 | # The cert both boxes present for a dev host must be the *.dev.11ch.at wildcard: |
| 207 | echo | openssl s_client -connect 54.154.95.205:443 -servername probe.dev.11ch.at 2>/dev/null \ |
| 208 | | openssl x509 -noout -subject # want: subject=CN=*.dev.11ch.at |
| 209 | echo | openssl s_client -connect 157.180.123.183:443 -servername probe.dev.11ch.at 2>/dev/null \ |
| 210 | | openssl x509 -noout -subject # same on Helsinki |
| 211 | ``` |
| 212 | |
| 213 | If a box still serves a per-host cert, DNS-01 didn't complete — check its IAM |
| 214 | (the tofu attaches the Route53 policy to Dublin's role / Helsinki's user) and the |
| 215 | `journalctl` line above; per-host issuance keeps the box serving meanwhile, so |
| 216 | this degrades safely. |
| 217 | |
| 218 | --- |
| 219 | |
| 220 | ## Step 5 — the proof (cross-region 200) |
| 221 | |
| 222 | Two synthetic hosts under the shared domain, one homed to each region, plus one |
| 223 | real minted space to exercise the full ledger path. |
| 224 | |
| 225 | ⚠️ **Two things that WILL bite (both learned the hard way on the first run):** |
| 226 | |
| 227 | 1. **The cross-region host MUST be `--hibernate`.** Route replication (the |
| 228 | fan-out that gives the *other* box a row to proxy to) is gated by |
| 229 | `route.Hibernate` in `control.go` — a plain `add-instance` makes a live |
| 230 | systemd instance pinned to one box, which deliberately does NOT replicate, |
| 231 | so the receiving box 404s ("no route"). Cross-region serving is inherently |
| 232 | hibernation-backed: the state lives in S3 so the owning box restores it, and |
| 233 | only then does the route fan out. |
| 234 | 2. **The CLI needs the mesh env sourced.** `sudo` strips the environment, so |
| 235 | `sudo /opt/lchat/lchat add-instance …` has no `LCHAT_MESH_*` / |
| 236 | `LCHAT_HIBERNATE_BUCKET` and silently skips the fan-out ("mesh: skipping route |
| 237 | replication"). Wrap it so it reads `/data/router.env`. |
| 238 | |
| 239 | Create the fi-homed host **on Helsinki** (its home region — so the box that owns |
| 240 | it also holds the instance to wake), hibernating, with the env sourced: |
| 241 | |
| 242 | ```sh |
| 243 | # on Helsinki (root): |
| 244 | sudo bash -c 'set -a; . /data/router.env; set +a; \ |
| 245 | /opt/lchat/lchat add-instance --home-region fi --hibernate mxproof-fi.dev.11ch.at' |
| 246 | # ie-homed control on Dublin (served locally — a live instance is fine here): |
| 247 | sudo /opt/lchat/lchat add-instance --home-region ie mxproof-ie.dev.11ch.at |
| 248 | ``` |
| 249 | |
| 250 | Both resolve to Dublin via `*.dev.11ch.at`. Hit them from anywhere: |
| 251 | |
| 252 | ```sh |
| 253 | curl -s -o /dev/null -w "%{http_code}\n" https://mxproof-fi.dev.11ch.at/api/version # want 200 |
| 254 | curl -s -o /dev/null -w "%{http_code}\n" https://mxproof-ie.dev.11ch.at/api/version # want 200 |
| 255 | ``` |
| 256 | |
| 257 | Confirm it truly served cross-region (a successful proxy does NOT log — only |
| 258 | errors do, so don't look for a proxy line): check that **Dublin's** registry has |
| 259 | the fanned-out `fi` route, and that **Helsinki's** log shows it restoring the DB |
| 260 | from S3 for that host: |
| 261 | |
| 262 | ```sh |
| 263 | # Dublin: the fi route arrived (home_region=fi, hibernate=1) → Dublin proxies, never owns it: |
| 264 | sudo sqlite3 -column /data/registry.db "SELECT host,home_region,hibernate FROM routes WHERE host LIKE 'mxproof%';" |
| 265 | # Helsinki: it woke the instance from S3 (this is the cross-region serve): |
| 266 | journalctl -u lchat-router --since -3min | grep 'replica sync .*mxproof-fi' |
| 267 | ``` |
| 268 | |
| 269 | Then the real ledger path — mint a space through the ie cloud (uses home's caller |
| 270 | token) and confirm the hostname is region-neutral and ledger-issued: |
| 271 | |
| 272 | ```sh |
| 273 | # on Dublin: |
| 274 | TOK=$(sudo grep '^LCHAT_CLOUD_TOKEN=' /data/home.env | cut -d= -f2-) |
| 275 | curl -s --unix-socket /run/lchat/cloud.sock -H "Authorization: Bearer $TOK" \ |
| 276 | -H 'Content-Type: application/json' -d '{"requester":"ledger-proof"}' \ |
| 277 | http://cloud/api/instances |
| 278 | # → {"host":"<sqid>.dev.11ch.at", …} (NOT .ie.dev.11ch.at), serves 200 on first hit. |
| 279 | # The S3 counter mesh/id-sequence now holds the block high-water (e.g. 100 after |
| 280 | # the first mint reserves a block of 100 — ids are gap-tolerant). |
| 281 | ``` |
| 282 | |
| 283 | Tear the throwaways down when done — delete the cloud-minted space via the cloud |
| 284 | API (`DELETE /api/instances/<host>?requester=<the-requester>`, so cloud.db's quota |
| 285 | ledger is freed) and the synthetic hosts with `lchat remove … --stop`. Route |
| 286 | **deletes do NOT fan out**, so also clear the replicated row from the *other* |
| 287 | box's registry by hand (`DELETE FROM routes WHERE host=…` + `systemctl reload |
| 288 | lchat-router`), and purge each hibernating host's `s3://<bucket>/<host>/` prefix. |
| 289 | |
| 290 | Tear the synthetic hosts down when done: |
| 291 | ```sh |
| 292 | sudo /opt/lchat/lchat remove mxproof-fi.dev.11ch.at --stop && sudo rm -f /data/mxproof-fi.dev.11ch.at.* |
| 293 | sudo /opt/lchat/lchat remove mxproof-ie.dev.11ch.at --stop && sudo rm -f /data/mxproof-ie.dev.11ch.at.* |
| 294 | ``` |
| 295 | |
| 296 | --- |
| 297 | |
| 298 | ## Rollback |
| 299 | |
| 300 | - **Binary**: `sudo mv /opt/lchat/lchat.bak-preroute53 /opt/lchat/lchat && sudo systemctl restart lchat-router lchat-cloud lchat-control`. |
| 301 | - **Env**: revert `LCHAT_MESH_DOMAINS` to the per-country value, drop the two |
| 302 | wildcard/DNS lines, revert cloud.env — restart. The boxes are back to |
| 303 | per-country placement immediately. |
| 304 | - **DNS/tofu**: `tofu destroy` targets only the new resources; but the fast |
| 305 | revert is to re-add the DNSimple `*.ie.dev`/`*.fi.dev` A records and remove the |
| 306 | `dev` NS delegation (DNSimple becomes authoritative again). The Route53 zone can |
| 307 | linger harmlessly. Because the delegation is the switch, removing it reverts |
| 308 | resolution wholesale. |
| 309 | - Nothing here touches instance DBs or member data — the dev instances keep their |
| 310 | `/data/<host>.db`, and the change is DNS + router/cloud config only. |
| 311 | |
| 312 | --- |
| 313 | |
| 314 | ## Follow-on (not in this cutover) |
| 315 | |
| 316 | - **Slice 5 — the relocate action**: an admin "change region" that flips a live |
| 317 | space's `home_region`, copies its store to the destination region, and moves |
| 318 | its `<sqid>.dev.11ch.at` A record — the proxy covers the DNS-TTL window. This |
| 319 | cutover's synthetic fi/ie hosts prove the *serving* half relocation lands on. |
| 320 | - **Fleet guide**: once this is stable, document `LCHAT_MESH_REGION` / |
| 321 | `LCHAT_CLOUD_REGION` / `LCHAT_CLOUD_ID_LEDGER` / `LCHAT_DNS_PROVIDER` / |
| 322 | `add-instance --home-region` in `docs/planet-scale-guide/`. |
| 323 | - **Prod `11ch.at` → Route53** is a separate, deliberate, deferred call — DNSimple |
| 324 | serves prod fine; consolidate later with a low-TTL dual-run if wanted. |
| 325 | |
amadan