🤖 Multi-edge Phase 1 — dev cutover runbook (region-neutral addressing)

The ordered, hand-runnable steps to flip the two-node dev mesh onto region-neutral addressing: one shared dev.11ch.at namespace, a shared *.dev.11ch.at wildcard cert over Route53 DNS-01, and cross-region serving that returns 200 (not the 503 the country-scoped mesh gave). Design + rationale: docs/multi-edge.md. The tofu it drives: infra/dev-mesh/.

This is a dev-only, reversible change. Prod (11ch.at on DNSimple) is untouched except for one scoped dev NS delegation record.

> 🤖 Historical record (2026-07) — not a procedure. This page documents > the cutover as it was performed and the gotchas learned doing it; it is > kept as design history. Do not operate from it. The dev mesh is > operated by ./deploy/ops dev plan|apply|deploy|rebuild — rebuild now > writes the env files below, pairs the mesh token, and seeds routes itself. > Anything the verbs can't do gets added to them (extend ops, then run > it), never hand-run from this page.

---

The end state (what "done" looks like)

• New spaces mint as <sqid>.dev.11ch.at — no country label — their ids drawn from the shared global ledger (internal/idalloc), so two region clouds never collide.
• Both boxes serve the single mesh domain dev.11ch.at, hold the shared *.dev.11ch.at wildcard cert, and place every space by its replicated home_region (mesh.go). Either box can receive any region-neutral host and either serve it (its region owns it) or reverse-proxy to the box that does.
• Cross-region serve returns 200: a home_region=fi host hit at Dublin is proxied to Helsinki, which terminates TLS with the shared wildcard, finds the replicated route, and serves.

Why this fixes the 503: today the two boxes serve different LCHAT_MESH_DOMAINS (ie.dev.11ch.at vs fi.dev.11ch.at). A cross-region host is in-scope on the sender but out-of-scope on the receiver, so the receiver serves locally with no route and no cert → "acme: host not in registry". One shared domain makes it in-scope on both, plain domain-scoped route replication copies every *.dev.11ch.at route to both boxes, and the shared wildcard lets the receiver terminate the proxied HTTPS hop. (Region-aware replication — branch multi-edge-p3-repl — is only a scale optimisation for many boxes on different domains; two boxes on one domain don't need it.)

---

Boxes, as found (2026-07-18)

| | Dublin (ie) | Helsinki (fi) | |---|---|---| | reach | ubuntu@54.154.95.205 -i ~/.ssh/eleven-dev-box (passwordless sudo) | root@157.180.123.183 (1Password hetzner agent key) | | arch | arm64 (t4g.small) | amd64 | | AWS creds | EC2 instance role eleven-dev-mesh-ie (no keys on box) | scoped IAM user eleven-dev-mesh-fi (AWS_ACCESS_KEY_ID/_SECRET in router.env) | | router svc | lchat-router → /opt/lchat/lchat | lchat-router → /opt/lchat/lchat | | cloud/control | lchat-cloud/lchat-control → /opt/lchat/lchat-banner (separate binary!) | → /opt/lchat/lchat (unified) |

⚠️ Both boxes' /opt/lchat/lchat were rebuilt by hand on Jul 18 with ambiguous stamps (Dublin's reports version dev). Don't trust the in-place binaries — Step 0 rebuilds both from a known main SHA so the running code is certain. On Dublin, lchat-cloud/lchat-control exec a different binary (lchat-banner, f3fa623b); this cutover unifies them onto the fresh binary so the cloud has the id-ledger + region-neutral minting code (that binary predates #1103/#1109). Backups already on each box: lchat.bak-mesh (dcc83d0a), lchat.bak-multiedge (90a6dee0).

All of #1093 #1094 #1103 #1106 #1109 (placement, provisioner stamp, id ledger, Route53 provider, region-neutral minting) are on main — a plain build of current main has everything.

---

Step 0 — build fresh binaries from main (both arches)

From a clean main checkout (record the SHA — it stamps the build):

# Dublin is arm64, Helsinki amd64 — build both.
GOOS=linux GOARCH=arm64 CGO_ENABLED=0 go build -trimpath \
  -ldflags "-X main.buildVersion=$(git rev-parse --short HEAD)" -o lchat-linux-arm64 .
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -trimpath \
  -ldflags "-X main.buildVersion=$(git rev-parse --short HEAD)" -o lchat-linux-amd64 .
# (deploy/build-linux.sh builds amd64 the same way; arm64 needs the explicit GOARCH.)

Ship + stage (do NOT overwrite the live binary yet — stage beside it):

scp lchat-linux-arm64 ubuntu@54.154.95.205:/opt/lchat/incoming/lchat-new
scp lchat-linux-amd64 root@157.180.123.183:/opt/lchat/incoming/lchat-new   # (mkdir -p /opt/lchat/incoming first if absent)

The binary swap itself happens in Step 3 (after the tofu + DNS are ready), so router and cloud restart once, onto both the new binary and the new env.

---

Step 1 — apply the Route53 tofu (from your Mac, with the vault token)

infra/dev-mesh/ now creates the dev.11ch.at Route53 zone, its three wildcards (*.dev, *.ie.dev, *.fi.dev), the dev NS delegation into the prod DNSimple zone, and the DNS-01 IAM on both boxes' role/user. The DNSimple 176940 token comes from the vault at apply time — never a box.

# `ops dev plan`/`apply` sort the credentials out themselves (AWS_* +
# DNSIMPLE_* from Bitwarden — the sops vault is retired):
./deploy/ops dev plan                # expect: 10 to add, 0 to change, 2 to destroy
./deploy/ops dev apply

The plan (validated 2026-07-18): + zone, *.dev/*.ie.dev/*.fi.dev records, 4×dev NS, 2× DNS-01 IAM policies; − the two old DNSimple *.ie.dev/*.fi.dev A records (superseded by the ones now inside the delegated zone). Nothing else changes — the box, bucket, existing IAM, and the home.dev.elevenmessenger.com record are untouched.

Note the zone nameservers for the next check:

AWS_PROFILE=eleven-dev tofu output dev_zone_nameservers

---

Step 2 — verify delegation propagates

Delegation TTL is short but resolvers cache the parent NS; give it a few minutes.

dig +short NS dev.11ch.at            # must return the 4 Route53 nameservers
dig +short one.ie.dev.11ch.at        # must still be 54.154.95.205 (now via Route53)
dig +short one.fi.dev.11ch.at        # must still be 157.180.123.183
dig +short anything.dev.11ch.at      # must be 54.154.95.205 (the *.dev wildcard → Dublin)

Do NOT proceed to the wildcard cert (Step 3) until dig NS dev.11ch.at shows Route53 — Let's Encrypt validates the DNS-01 TXT against the authoritative nameservers, which are Route53 only after delegation lands.

---

Step 3 — reconfigure + swap the boxes

Env deltas (add/modify; everything else stays). Edit /data/router.env and, on Dublin, /data/cloud.env; then swap the binary and restart.

Dublin (ie) — /data/router.env

LCHAT_MESH_DOMAINS=dev.11ch.at            # was ie.dev.11ch.at
LCHAT_DNS_PROVIDER=route53                # new — the AWS DNS-01 off-ramp
LCHAT_WILDCARD_DOMAINS=dev.11ch.at        # new — obtain *.dev.11ch.at
# unchanged: AWS_REGION=eu-west-1, LCHAT_HIBERNATE_BUCKET, LCHAT_MESH_REGION=ie,
#            LCHAT_MESH_SELF=https://one.ie.dev.11ch.at, LCHAT_MESH_TOKEN

Dublin (ie) — /data/cloud.env

LCHAT_CLOUD_BASE_DOMAIN=dev.11ch.at       # was ie.dev.11ch.at — mint <sqid>.dev.11ch.at
LCHAT_CLOUD_ID_LEDGER=1                    # new — draw ids from the global ledger
LCHAT_HIBERNATE_BUCKET=eleven-dev-mesh-427880590669  # new — where the ledger object lives
LCHAT_HIBERNATE_REGION=eu-west-1          # new — so idalloc's S3 client resolves the region
# unchanged: LCHAT_CLOUD_REGION=ie, LCHAT_ADMIN_TOKEN, demo/pool vars

> dev.11ch.at is a fresh namespace (no spaces minted under it), so the > ledger counter starts at 0 → first id 1 → no collision. No counter seed > needed. (Seeding is only required when flipping the ledger on for a > namespace that already has spaces — see newCloudLedger's warning.)

Helsinki (fi) — /data/router.env

LCHAT_MESH_DOMAINS=dev.11ch.at            # was fi.dev.11ch.at
LCHAT_DNS_PROVIDER=route53                # new
LCHAT_WILDCARD_DOMAINS=dev.11ch.at        # new
# unchanged: AWS_ACCESS_KEY_ID/_SECRET (scoped user, now also Route53-DNS01-scoped),
#            AWS_REGION=eu-west-1, LCHAT_HIBERNATE_BUCKET, LCHAT_MESH_REGION=fi,
#            LCHAT_MESH_SELF=https://one.fi.dev.11ch.at, LCHAT_MESH_TOKEN
# (no cloud on Helsinki — nothing else to change.)

Swap + restart (each box)

# back up the current live binary, move the fresh one in:
sudo cp /opt/lchat/lchat /opt/lchat/lchat.bak-preroute53
sudo mv /opt/lchat/incoming/lchat-new /opt/lchat/lchat
sudo chmod +x /opt/lchat/lchat
# Dublin only: point cloud/control at the fresh unified binary (drop lchat-banner)
#   — edit the units' ExecStart to /opt/lchat/lchat, or symlink lchat-banner→lchat,
#     then `sudo systemctl daemon-reload`.
sudo systemctl restart lchat-router
sudo systemctl restart lchat-cloud lchat-control   # Dublin (Helsinki has none)

The router obtains the *.dev.11ch.at wildcard on startup (its run() loop checks immediately). Watch for it:

journalctl -u lchat-router -n 60 --no-pager | grep -iE "wildcard|route53|acme"
# want: "router: wildcard *.dev.11ch.at obtained (valid until …)"

---

Step 4 — verify the wildcard cert (both boxes)

# The cert both boxes present for a dev host must be the *.dev.11ch.at wildcard:
echo | openssl s_client -connect 54.154.95.205:443  -servername probe.dev.11ch.at 2>/dev/null \
  | openssl x509 -noout -subject           # want: subject=CN=*.dev.11ch.at
echo | openssl s_client -connect 157.180.123.183:443 -servername probe.dev.11ch.at 2>/dev/null \
  | openssl x509 -noout -subject           # same on Helsinki

If a box still serves a per-host cert, DNS-01 didn't complete — check its IAM (the tofu attaches the Route53 policy to Dublin's role / Helsinki's user) and the journalctl line above; per-host issuance keeps the box serving meanwhile, so this degrades safely.

---

Step 5 — the proof (cross-region 200)

Two synthetic hosts under the shared domain, one homed to each region, plus one real minted space to exercise the full ledger path.

⚠️ Two things that WILL bite (both learned the hard way on the first run):

1. The cross-region host MUST be --hibernate. Route replication (the fan-out that gives the other box a row to proxy to) is gated by route.Hibernate in control.go — a plain add-instance makes a live systemd instance pinned to one box, which deliberately does NOT replicate, so the receiving box 404s ("no route"). Cross-region serving is inherently hibernation-backed: the state lives in S3 so the owning box restores it, and only then does the route fan out.
2. The CLI needs the mesh env sourced. sudo strips the environment, so sudo /opt/lchat/lchat add-instance … has no LCHAT_MESH_* / LCHAT_HIBERNATE_BUCKET and silently skips the fan-out ("mesh: skipping route replication"). Wrap it so it reads /data/router.env.

Create the fi-homed host on Helsinki (its home region — so the box that owns it also holds the instance to wake), hibernating, with the env sourced:

# on Helsinki (root):
sudo bash -c 'set -a; . /data/router.env; set +a; \
  /opt/lchat/lchat add-instance --home-region fi --hibernate mxproof-fi.dev.11ch.at'
# ie-homed control on Dublin (served locally — a live instance is fine here):
sudo /opt/lchat/lchat add-instance --home-region ie mxproof-ie.dev.11ch.at

Both resolve to Dublin via *.dev.11ch.at. Hit them from anywhere:

curl -s -o /dev/null -w "%{http_code}\n" https://mxproof-fi.dev.11ch.at/api/version   # want 200
curl -s -o /dev/null -w "%{http_code}\n" https://mxproof-ie.dev.11ch.at/api/version   # want 200

Confirm it truly served cross-region (a successful proxy does NOT log — only errors do, so don't look for a proxy line): check that Dublin's registry has the fanned-out fi route, and that Helsinki's log shows it restoring the DB from S3 for that host:

# Dublin: the fi route arrived (home_region=fi, hibernate=1) → Dublin proxies, never owns it:
sudo sqlite3 -column /data/registry.db "SELECT host,home_region,hibernate FROM routes WHERE host LIKE 'mxproof%';"
# Helsinki: it woke the instance from S3 (this is the cross-region serve):
journalctl -u lchat-router --since -3min | grep 'replica sync .*mxproof-fi'

Then the real ledger path — mint a space through the ie cloud (uses home's caller token) and confirm the hostname is region-neutral and ledger-issued:

# on Dublin:
TOK=$(sudo grep '^LCHAT_CLOUD_TOKEN=' /data/home.env | cut -d= -f2-)
curl -s --unix-socket /run/lchat/cloud.sock -H "Authorization: Bearer $TOK" \
  -H 'Content-Type: application/json' -d '{"requester":"ledger-proof"}' \
  http://cloud/api/instances
# → {"host":"<sqid>.dev.11ch.at", …}  (NOT .ie.dev.11ch.at), serves 200 on first hit.
# The S3 counter mesh/id-sequence now holds the block high-water (e.g. 100 after
# the first mint reserves a block of 100 — ids are gap-tolerant).

Tear the throwaways down when done — delete the cloud-minted space via the cloud API (DELETE /api/instances/<host>?requester=<the-requester>, so cloud.db's quota ledger is freed) and the synthetic hosts with lchat remove … --stop. Route deletes do NOT fan out, so also clear the replicated row from the other box's registry by hand (DELETE FROM routes WHERE host=… + systemctl reload lchat-router), and purge each hibernating host's s3://<bucket>/<host>/ prefix.

Tear the synthetic hosts down when done:

sudo /opt/lchat/lchat remove mxproof-fi.dev.11ch.at --stop && sudo rm -f /data/mxproof-fi.dev.11ch.at.*
sudo /opt/lchat/lchat remove mxproof-ie.dev.11ch.at --stop && sudo rm -f /data/mxproof-ie.dev.11ch.at.*

---

Rollback

• Binary: sudo mv /opt/lchat/lchat.bak-preroute53 /opt/lchat/lchat && sudo systemctl restart lchat-router lchat-cloud lchat-control.
• Env: revert LCHAT_MESH_DOMAINS to the per-country value, drop the two wildcard/DNS lines, revert cloud.env — restart. The boxes are back to per-country placement immediately.
• DNS/tofu: tofu destroy targets only the new resources; but the fast revert is to re-add the DNSimple *.ie.dev/*.fi.dev A records and remove the dev NS delegation (DNSimple becomes authoritative again). The Route53 zone can linger harmlessly. Because the delegation is the switch, removing it reverts resolution wholesale.
• Nothing here touches instance DBs or member data — the dev instances keep their /data/<host>.db, and the change is DNS + router/cloud config only.

---

Follow-on (not in this cutover)

• Slice 5 — the relocate action: an admin "change region" that flips a live space's home_region, copies its store to the destination region, and moves its <sqid>.dev.11ch.at A record — the proxy covers the DNS-TTL window. This cutover's synthetic fi/ie hosts prove the serving half relocation lands on.
• Fleet guide: once this is stable, document LCHAT_MESH_REGION / LCHAT_CLOUD_REGION / LCHAT_CLOUD_ID_LEDGER / LCHAT_DNS_PROVIDER / add-instance --home-region in docs/planet-scale-guide/.
• Prod 11ch.at → Route53 is a separate, deliberate, deferred call — DNSimple serves prod fine; consolidate later with a low-TTL dual-run if wanted.