amadan
infra+docs: dev.11ch.at Route53 zone + region-neutral cutover (multi-edge P1) (#1123)
* infra+docs: dev.11ch.at Route53 zone + region-neutral cutover runbook Brings the (state-live, code-stranded) dev-mesh tofu forward onto main and adds the Route53 half of multi-edge Phase 1: - route53.tf: a delegated dev.11ch.at hosted zone in the dev AWS account, the region-neutral *.dev.11ch.at wildcard (→ Dublin, the mesh front door), the two per-country *.ie.dev / *.fi.dev wildcards moved in from DNSimple, and the DNS-01 IAM both boxes use to obtain the shared *.dev.11ch.at cert (Dublin's instance role + Helsinki's scoped user). ChangeResourceRecordSets is locked to the one zone; the account-wide list/GetChange ops can't be resource-scoped. - dns.tf: the DNSimple `dev` NS delegation (4 records, count-based since the Route53 nameservers aren't known at plan time) + the kept home.dev A record. The old *.ie.dev / *.fi.dev DNSimple A records are removed (superseded by the ones now inside the delegated zone). Route53 is the additive AWS off-ramp (LCHAT_DNS_PROVIDER=route53, route53dns.go); DNSimple stays the default provider everywhere else. Prod 11ch.at is untouched except the one scoped dev NS record. docs/multi-edge-dev-cutover.md: the ordered, hand-runnable cutover — build fresh binaries, apply the tofu, verify delegation, widen both boxes to the shared dev.11ch.at domain + the wildcard cert, turn on the id ledger for minting, and prove cross-region serve returns 200 (not the country-scoped 503). Reversible; touches no member data. tofu plan validated against live state: 10 to add, 0 to change, 2 to destroy. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: correct the cutover proof — cross-region serve needs --hibernate Learned on the live run: route fan-out is gated by route.Hibernate (control.go), so a plain add-instance makes a box-pinned live instance that never replicates — the receiving box 404s. Cross-region serving is hibernation-backed (state in S3, owner restores). Also: the CLI must source /data/router.env or sudo strips the mesh env and the fan-out silently no-ops. Step 5 now creates the fi host on Helsinki with --hibernate + sourced env, verifies via the S3 replica-sync log (successful proxies don't log), and documents the deletes-don't-fan-out cleanup. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> ---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
9 files changed,
+853
−0
docs/multi-edge-dev-cutover.md+317 −0infra/dev-mesh/.terraform.lock.hcl+75 −0infra/dev-mesh/README.md+39 −0infra/dev-mesh/compute.tf+116 −0infra/dev-mesh/dns.tf+47 −0infra/dev-mesh/main.tf+53 −0infra/dev-mesh/outputs.tf+33 −0infra/dev-mesh/route53.tf+91 −0infra/dev-mesh/storage.tf+82 −0
diff --git a/docs/multi-edge-dev-cutover.md b/docs/multi-edge-dev-cutover.md| new file mode 100644 |
| index 0000000..c654c5e |
| --- /dev/null |
| +++ b/docs/multi-edge-dev-cutover.md |
| @@ -0,0 +1,317 @@ |
| +# 🤖 Multi-edge Phase 1 — dev cutover runbook (region-neutral addressing) |
| + |
| +The ordered, hand-runnable steps to flip the two-node dev mesh onto |
| +**region-neutral addressing**: one shared `dev.11ch.at` namespace, a shared |
| +`*.dev.11ch.at` wildcard cert over Route53 DNS-01, and cross-region serving that |
| +returns **200** (not the 503 the country-scoped mesh gave). Design + rationale: |
| +`docs/multi-edge.md`. The tofu it drives: `infra/dev-mesh/`. |
| + |
| +This is a **dev-only, reversible** change. Prod (`11ch.at` on DNSimple) is |
| +untouched except for one scoped `dev` NS delegation record. |
| + |
| +--- |
| + |
| +## The end state (what "done" looks like) |
| + |
| +- New spaces mint as `<sqid>.dev.11ch.at` — no country label — their ids drawn |
| + from the shared global ledger (`internal/idalloc`), so two region clouds never |
| + collide. |
| +- **Both** boxes serve the single mesh domain `dev.11ch.at`, hold the shared |
| + `*.dev.11ch.at` wildcard cert, and place every space by its replicated |
| + `home_region` (mesh.go). Either box can receive any region-neutral host and |
| + either serve it (its region owns it) or reverse-proxy to the box that does. |
| +- Cross-region serve returns 200: a `home_region=fi` host hit at Dublin is |
| + proxied to Helsinki, which terminates TLS with the shared wildcard, finds the |
| + replicated route, and serves. |
| + |
| +Why this fixes the 503: today the two boxes serve **different** |
| +`LCHAT_MESH_DOMAINS` (`ie.dev.11ch.at` vs `fi.dev.11ch.at`). A cross-region host |
| +is in-scope on the sender but out-of-scope on the receiver, so the receiver |
| +serves locally with no route and no cert → "acme: host not in registry". One |
| +shared domain makes it in-scope on both, plain domain-scoped route replication |
| +copies every `*.dev.11ch.at` route to both boxes, and the shared wildcard lets |
| +the receiver terminate the proxied HTTPS hop. (Region-*aware* replication — |
| +branch `multi-edge-p3-repl` — is only a scale optimisation for many boxes on |
| +different domains; two boxes on one domain don't need it.) |
| + |
| +--- |
| + |
| +## Boxes, as found (2026-07-18) |
| + |
| +| | Dublin (ie) | Helsinki (fi) | |
| +|---|---|---| |
| +| reach | `ubuntu@54.154.95.205 -i ~/.ssh/eleven-dev-box` (passwordless sudo) | `root@157.180.123.183` (1Password hetzner agent key) | |
| +| arch | arm64 (t4g.small) | amd64 | |
| +| AWS creds | EC2 **instance role** `eleven-dev-mesh-ie` (no keys on box) | scoped IAM **user** `eleven-dev-mesh-fi` (`AWS_ACCESS_KEY_ID`/`_SECRET` in router.env) | |
| +| router svc | `lchat-router` → `/opt/lchat/lchat` | `lchat-router` → `/opt/lchat/lchat` | |
| +| cloud/control | `lchat-cloud`/`lchat-control` → `/opt/lchat/lchat-banner` (**separate binary!**) | → `/opt/lchat/lchat` (unified) | |
| + |
| +⚠️ **Both boxes' `/opt/lchat/lchat` were rebuilt by hand on Jul 18** with |
| +ambiguous stamps (Dublin's reports version `dev`). Don't trust the in-place |
| +binaries — **Step 0 rebuilds both from a known main SHA** so the running code is |
| +certain. On Dublin, `lchat-cloud`/`lchat-control` exec a *different* binary |
| +(`lchat-banner`, `f3fa623b`); this cutover unifies them onto the fresh binary so |
| +the cloud has the id-ledger + region-neutral minting code (that binary predates |
| +`#1103`/`#1109`). Backups already on each box: `lchat.bak-mesh` (dcc83d0a), |
| +`lchat.bak-multiedge` (90a6dee0). |
| + |
| +All of `#1093 #1094 #1103 #1106 #1109` (placement, provisioner stamp, id ledger, |
| +Route53 provider, region-neutral minting) are on **main** — a plain build of |
| +current main has everything. |
| + |
| +--- |
| + |
| +## Step 0 — build fresh binaries from main (both arches) |
| + |
| +From a clean `main` checkout (record the SHA — it stamps the build): |
| + |
| +```sh |
| +# Dublin is arm64, Helsinki amd64 — build both. |
| +GOOS=linux GOARCH=arm64 CGO_ENABLED=0 go build -trimpath \ |
| + -ldflags "-X main.buildVersion=$(git rev-parse --short HEAD)" -o lchat-linux-arm64 . |
| +GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -trimpath \ |
| + -ldflags "-X main.buildVersion=$(git rev-parse --short HEAD)" -o lchat-linux-amd64 . |
| +# (deploy/build-linux.sh builds amd64 the same way; arm64 needs the explicit GOARCH.) |
| +``` |
| + |
| +Ship + stage (do NOT overwrite the live binary yet — stage beside it): |
| + |
| +```sh |
| +scp lchat-linux-arm64 ubuntu@54.154.95.205:/opt/lchat/incoming/lchat-new |
| +scp lchat-linux-amd64 root@157.180.123.183:/opt/lchat/incoming/lchat-new # (mkdir -p /opt/lchat/incoming first if absent) |
| +``` |
| + |
| +The binary swap itself happens in Step 3 (after the tofu + DNS are ready), so |
| +router and cloud restart once, onto both the new binary *and* the new env. |
| + |
| +--- |
| + |
| +## Step 1 — apply the Route53 tofu (from your Mac, with the vault token) |
| + |
| +`infra/dev-mesh/` now creates the `dev.11ch.at` Route53 zone, its three |
| +wildcards (`*.dev`, `*.ie.dev`, `*.fi.dev`), the `dev` NS delegation into the |
| +prod DNSimple zone, and the DNS-01 IAM on both boxes' role/user. The DNSimple |
| +176940 token comes from the vault at apply time — **never a box**. |
| + |
| +```sh |
| +cd infra/dev-mesh |
| +set -a; . <(cd ~/elevenmessenger/secrets-vault && sops -d aws-dev-agent-james.env | grep '^DNSIMPLE_'); set +a |
| +AWS_PROFILE=eleven-dev tofu init |
| +AWS_PROFILE=eleven-dev tofu plan # expect: 10 to add, 0 to change, 2 to destroy |
| +AWS_PROFILE=eleven-dev tofu apply |
| +``` |
| + |
| +The plan (validated 2026-07-18): **+** zone, `*.dev`/`*.ie.dev`/`*.fi.dev` |
| +records, 4×`dev` NS, 2× DNS-01 IAM policies; **−** the two old DNSimple |
| +`*.ie.dev`/`*.fi.dev` A records (superseded by the ones now inside the delegated |
| +zone). Nothing else changes — the box, bucket, existing IAM, and the |
| +`home.dev.elevenmessenger.com` record are untouched. |
| + |
| +Note the zone nameservers for the next check: |
| + |
| +```sh |
| +AWS_PROFILE=eleven-dev tofu output dev_zone_nameservers |
| +``` |
| + |
| +--- |
| + |
| +## Step 2 — verify delegation propagates |
| + |
| +Delegation TTL is short but resolvers cache the parent NS; give it a few minutes. |
| + |
| +```sh |
| +dig +short NS dev.11ch.at # must return the 4 Route53 nameservers |
| +dig +short one.ie.dev.11ch.at # must still be 54.154.95.205 (now via Route53) |
| +dig +short one.fi.dev.11ch.at # must still be 157.180.123.183 |
| +dig +short anything.dev.11ch.at # must be 54.154.95.205 (the *.dev wildcard → Dublin) |
| +``` |
| + |
| +Do NOT proceed to the wildcard cert (Step 3) until `dig NS dev.11ch.at` shows |
| +Route53 — Let's Encrypt validates the DNS-01 TXT against the authoritative |
| +nameservers, which are Route53 only after delegation lands. |
| + |
| +--- |
| + |
| +## Step 3 — reconfigure + swap the boxes |
| + |
| +Env deltas (add/modify; everything else stays). Edit `/data/router.env` and, on |
| +Dublin, `/data/cloud.env`; then swap the binary and restart. |
| + |
| +### Dublin (ie) — `/data/router.env` |
| +``` |
| +LCHAT_MESH_DOMAINS=dev.11ch.at # was ie.dev.11ch.at |
| +LCHAT_DNS_PROVIDER=route53 # new — the AWS DNS-01 off-ramp |
| +LCHAT_WILDCARD_DOMAINS=dev.11ch.at # new — obtain *.dev.11ch.at |
| +# unchanged: AWS_REGION=eu-west-1, LCHAT_HIBERNATE_BUCKET, LCHAT_MESH_REGION=ie, |
| +# LCHAT_MESH_SELF=https://one.ie.dev.11ch.at, LCHAT_MESH_TOKEN |
| +``` |
| + |
| +### Dublin (ie) — `/data/cloud.env` |
| +``` |
| +LCHAT_CLOUD_BASE_DOMAIN=dev.11ch.at # was ie.dev.11ch.at — mint <sqid>.dev.11ch.at |
| +LCHAT_CLOUD_ID_LEDGER=1 # new — draw ids from the global ledger |
| +LCHAT_HIBERNATE_BUCKET=eleven-dev-mesh-427880590669 # new — where the ledger object lives |
| +LCHAT_HIBERNATE_REGION=eu-west-1 # new — so idalloc's S3 client resolves the region |
| +# unchanged: LCHAT_CLOUD_REGION=ie, LCHAT_ADMIN_TOKEN, demo/pool vars |
| +``` |
| +> `dev.11ch.at` is a **fresh** namespace (no spaces minted under it), so the |
| +> ledger counter starts at 0 → first id 1 → no collision. **No counter seed |
| +> needed.** (Seeding is only required when flipping the ledger on for a |
| +> namespace that already has spaces — see `newCloudLedger`'s warning.) |
| + |
| +### Helsinki (fi) — `/data/router.env` |
| +``` |
| +LCHAT_MESH_DOMAINS=dev.11ch.at # was fi.dev.11ch.at |
| +LCHAT_DNS_PROVIDER=route53 # new |
| +LCHAT_WILDCARD_DOMAINS=dev.11ch.at # new |
| +# unchanged: AWS_ACCESS_KEY_ID/_SECRET (scoped user, now also Route53-DNS01-scoped), |
| +# AWS_REGION=eu-west-1, LCHAT_HIBERNATE_BUCKET, LCHAT_MESH_REGION=fi, |
| +# LCHAT_MESH_SELF=https://one.fi.dev.11ch.at, LCHAT_MESH_TOKEN |
| +# (no cloud on Helsinki — nothing else to change.) |
| +``` |
| + |
| +### Swap + restart (each box) |
| +```sh |
| +# back up the current live binary, move the fresh one in: |
| +sudo cp /opt/lchat/lchat /opt/lchat/lchat.bak-preroute53 |
| +sudo mv /opt/lchat/incoming/lchat-new /opt/lchat/lchat |
| +sudo chmod +x /opt/lchat/lchat |
| +# Dublin only: point cloud/control at the fresh unified binary (drop lchat-banner) |
| +# — edit the units' ExecStart to /opt/lchat/lchat, or symlink lchat-banner→lchat, |
| +# then `sudo systemctl daemon-reload`. |
| +sudo systemctl restart lchat-router |
| +sudo systemctl restart lchat-cloud lchat-control # Dublin (Helsinki has none) |
| +``` |
| + |
| +The router obtains the `*.dev.11ch.at` wildcard on startup (its `run()` loop |
| +checks immediately). Watch for it: |
| + |
| +```sh |
| +journalctl -u lchat-router -n 60 --no-pager | grep -iE "wildcard|route53|acme" |
| +# want: "router: wildcard *.dev.11ch.at obtained (valid until …)" |
| +``` |
| + |
| +--- |
| + |
| +## Step 4 — verify the wildcard cert (both boxes) |
| + |
| +```sh |
| +# The cert both boxes present for a dev host must be the *.dev.11ch.at wildcard: |
| +echo | openssl s_client -connect 54.154.95.205:443 -servername probe.dev.11ch.at 2>/dev/null \ |
| + | openssl x509 -noout -subject # want: subject=CN=*.dev.11ch.at |
| +echo | openssl s_client -connect 157.180.123.183:443 -servername probe.dev.11ch.at 2>/dev/null \ |
| + | openssl x509 -noout -subject # same on Helsinki |
| +``` |
| + |
| +If a box still serves a per-host cert, DNS-01 didn't complete — check its IAM |
| +(the tofu attaches the Route53 policy to Dublin's role / Helsinki's user) and the |
| +`journalctl` line above; per-host issuance keeps the box serving meanwhile, so |
| +this degrades safely. |
| + |
| +--- |
| + |
| +## Step 5 — the proof (cross-region 200) |
| + |
| +Two synthetic hosts under the shared domain, one homed to each region, plus one |
| +real minted space to exercise the full ledger path. |
| + |
| +⚠️ **Two things that WILL bite (both learned the hard way on the first run):** |
| + |
| +1. **The cross-region host MUST be `--hibernate`.** Route replication (the |
| + fan-out that gives the *other* box a row to proxy to) is gated by |
| + `route.Hibernate` in `control.go` — a plain `add-instance` makes a live |
| + systemd instance pinned to one box, which deliberately does NOT replicate, |
| + so the receiving box 404s ("no route"). Cross-region serving is inherently |
| + hibernation-backed: the state lives in S3 so the owning box restores it, and |
| + only then does the route fan out. |
| +2. **The CLI needs the mesh env sourced.** `sudo` strips the environment, so |
| + `sudo /opt/lchat/lchat add-instance …` has no `LCHAT_MESH_*` / |
| + `LCHAT_HIBERNATE_BUCKET` and silently skips the fan-out ("mesh: skipping route |
| + replication"). Wrap it so it reads `/data/router.env`. |
| + |
| +Create the fi-homed host **on Helsinki** (its home region — so the box that owns |
| +it also holds the instance to wake), hibernating, with the env sourced: |
| + |
| +```sh |
| +# on Helsinki (root): |
| +sudo bash -c 'set -a; . /data/router.env; set +a; \ |
| + /opt/lchat/lchat add-instance --home-region fi --hibernate mxproof-fi.dev.11ch.at' |
| +# ie-homed control on Dublin (served locally — a live instance is fine here): |
| +sudo /opt/lchat/lchat add-instance --home-region ie mxproof-ie.dev.11ch.at |
| +``` |
| + |
| +Both resolve to Dublin via `*.dev.11ch.at`. Hit them from anywhere: |
| + |
| +```sh |
| +curl -s -o /dev/null -w "%{http_code}\n" https://mxproof-fi.dev.11ch.at/api/version # want 200 |
| +curl -s -o /dev/null -w "%{http_code}\n" https://mxproof-ie.dev.11ch.at/api/version # want 200 |
| +``` |
| + |
| +Confirm it truly served cross-region (a successful proxy does NOT log — only |
| +errors do, so don't look for a proxy line): check that **Dublin's** registry has |
| +the fanned-out `fi` route, and that **Helsinki's** log shows it restoring the DB |
| +from S3 for that host: |
| + |
| +```sh |
| +# Dublin: the fi route arrived (home_region=fi, hibernate=1) → Dublin proxies, never owns it: |
| +sudo sqlite3 -column /data/registry.db "SELECT host,home_region,hibernate FROM routes WHERE host LIKE 'mxproof%';" |
| +# Helsinki: it woke the instance from S3 (this is the cross-region serve): |
| +journalctl -u lchat-router --since -3min | grep 'replica sync .*mxproof-fi' |
| +``` |
| + |
| +Then the real ledger path — mint a space through the ie cloud (uses home's caller |
| +token) and confirm the hostname is region-neutral and ledger-issued: |
| + |
| +```sh |
| +# on Dublin: |
| +TOK=$(sudo grep '^LCHAT_CLOUD_TOKEN=' /data/home.env | cut -d= -f2-) |
| +curl -s --unix-socket /run/lchat/cloud.sock -H "Authorization: Bearer $TOK" \ |
| + -H 'Content-Type: application/json' -d '{"requester":"ledger-proof"}' \ |
| + http://cloud/api/instances |
| +# → {"host":"<sqid>.dev.11ch.at", …} (NOT .ie.dev.11ch.at), serves 200 on first hit. |
| +# The S3 counter mesh/id-sequence now holds the block high-water (e.g. 100 after |
| +# the first mint reserves a block of 100 — ids are gap-tolerant). |
| +``` |
| + |
| +Tear the throwaways down when done — delete the cloud-minted space via the cloud |
| +API (`DELETE /api/instances/<host>?requester=<the-requester>`, so cloud.db's quota |
| +ledger is freed) and the synthetic hosts with `lchat remove … --stop`. Route |
| +**deletes do NOT fan out**, so also clear the replicated row from the *other* |
| +box's registry by hand (`DELETE FROM routes WHERE host=…` + `systemctl reload |
| +lchat-router`), and purge each hibernating host's `s3://<bucket>/<host>/` prefix. |
| + |
| +Tear the synthetic hosts down when done: |
| +```sh |
| +sudo /opt/lchat/lchat remove mxproof-fi.dev.11ch.at --stop && sudo rm -f /data/mxproof-fi.dev.11ch.at.* |
| +sudo /opt/lchat/lchat remove mxproof-ie.dev.11ch.at --stop && sudo rm -f /data/mxproof-ie.dev.11ch.at.* |
| +``` |
| + |
| +--- |
| + |
| +## Rollback |
| + |
| +- **Binary**: `sudo mv /opt/lchat/lchat.bak-preroute53 /opt/lchat/lchat && sudo systemctl restart lchat-router lchat-cloud lchat-control`. |
| +- **Env**: revert `LCHAT_MESH_DOMAINS` to the per-country value, drop the two |
| + wildcard/DNS lines, revert cloud.env — restart. The boxes are back to |
| + per-country placement immediately. |
| +- **DNS/tofu**: `tofu destroy` targets only the new resources; but the fast |
| + revert is to re-add the DNSimple `*.ie.dev`/`*.fi.dev` A records and remove the |
| + `dev` NS delegation (DNSimple becomes authoritative again). The Route53 zone can |
| + linger harmlessly. Because the delegation is the switch, removing it reverts |
| + resolution wholesale. |
| +- Nothing here touches instance DBs or member data — the dev instances keep their |
| + `/data/<host>.db`, and the change is DNS + router/cloud config only. |
| + |
| +--- |
| + |
| +## Follow-on (not in this cutover) |
| + |
| +- **Slice 5 — the relocate action**: an admin "change region" that flips a live |
| + space's `home_region`, copies its store to the destination region, and moves |
| + its `<sqid>.dev.11ch.at` A record — the proxy covers the DNS-TTL window. This |
| + cutover's synthetic fi/ie hosts prove the *serving* half relocation lands on. |
| +- **Fleet guide**: once this is stable, document `LCHAT_MESH_REGION` / |
| + `LCHAT_CLOUD_REGION` / `LCHAT_CLOUD_ID_LEDGER` / `LCHAT_DNS_PROVIDER` / |
| + `add-instance --home-region` in `docs/planet-scale-guide/`. |
| +- **Prod `11ch.at` → Route53** is a separate, deliberate, deferred call — DNSimple |
| + serves prod fine; consolidate later with a low-TTL dual-run if wanted. |
diff --git a/infra/dev-mesh/.terraform.lock.hcl b/infra/dev-mesh/.terraform.lock.hcl| new file mode 100644 |
| index 0000000..fe46238 |
| --- /dev/null |
| +++ b/infra/dev-mesh/.terraform.lock.hcl |
| @@ -0,0 +1,75 @@ |
| +# This file is maintained automatically by "tofu init". |
| +# Manual edits may be lost in future updates. |
| + |
| +provider "registry.opentofu.org/dnsimple/dnsimple" { |
| + version = "1.10.0" |
| + constraints = "~> 1.0" |
| + hashes = [ |
| + "h1:EVDDjVKBij3ivkK6+tyS9jWxSExxVvOoGtMS8/M43f8=", |
| + "h1:HK5nqc5bZFdth/BcNc9GqUfXAYsDn8Ev/tgfmB0jMYI=", |
| + "h1:Hhbv3mnLhgY9NuvjzUqDo0PQKNTfStULx+m+S3wHKQg=", |
| + "h1:IHnPrj3/ayQ7PmKNkaq2nzoT87F+hnoSp5c/xdsPjnY=", |
| + "h1:O13KzReeaEXMiUe6qx2HDgHVVhBan2FnSiq41BMtSgo=", |
| + "h1:RN2gojQH/bA1zTRUOGdGH8w03uIa2agezDZyMY1VhV8=", |
| + "h1:UOCA/dFp2ORSVi/WlCCZrjjfXPUy9vpgk4PEBv6h4w0=", |
| + "h1:YyaCa5mDiLhM8lddBZyjKo5iM9lfunk0uA1X0uOSvoo=", |
| + "h1:ZNsEnMVMw39wrFufI7N5ypjnl92awiUlZ97z92MUPrs=", |
| + "h1:cQvKfEDJeGr3a0wWbGLoS+x0Orx8+9mWwSNAYfnqljM=", |
| + "h1:oV3AqgcHjXhMmoYmdHAerNEDRCkVq9QKvMnsyodsnSc=", |
| + "h1:r9RpaCOvjJjEZ6S3v3KPB6eNqf/STR9dYc94ippxhGk=", |
| + "h1:rIdsl1pgd2fqiSyB7lyn6gRh8RPq3MfCA5bt7q4NTWc=", |
| + "h1:zROoguvUNCxV46qrJGwbH7vEoUwxQJIfZrxb20teJa0=", |
| + "zh:025dbad287c9a9ca61b84fc4a14d3f595dbbc1139d5cf034597922264b756f6f", |
| + "zh:0c2fa8513004732f3aae6a408eb68ff9c2ad119f8bcefe491f0420ad0f4fe7d5", |
| + "zh:1ebb4cebfc29dfd8dd7f4a03f57aebe118a9dda15caf17eace06a16b40507317", |
| + "zh:5585c10af512519f006491bec5e63504e3682f61af4ab4c4807721de7764bf0f", |
| + "zh:571ead8072d43334d8ac169c86da21c9b325439508344898988f925f89ea7593", |
| + "zh:5bdd4d83e0831f0094f06c9d56aed7e1388967e9f4f4144d69c5c37d3d170e45", |
| + "zh:67982e17e937d71c1b2c83d740cd37a28c9c6ad16d325dddb91327303d6e96ff", |
| + "zh:7de0afaaed2668c27c7189180ceda3bc73e17b21a150ecec8eb3979e95875323", |
| + "zh:88911647fc4f8eaac3699b72ca7a0bee8f33521725b839b20235138f5072bfb7", |
| + "zh:8ad2a7f599864602e3126a9c31ec28b8c277090e98f80cd0d5e29eddb1bebae1", |
| + "zh:ac47ee0fd0837b57945fe961656f1f8e3c20e31178c49aafa7836b93aa5966a6", |
| + "zh:acdc1a9a5553d8d0e7aa887acfa0cf440f9c428dabc0c46b61d1b3eae6443e09", |
| + "zh:b3e3d236b72b5d975098f29e42a8eff071064de99cd81b27e264e5641aed7365", |
| + "zh:d21daf975282bdddc7c66580d0fde5395a0ed67fdfe3c3c87aec952a6604ba24", |
| + "zh:f809ab383cca0a5f83072981c64208cbd7fa67e986a86ee02dd2c82333221e32", |
| + ] |
| +} |
| + |
| +provider "registry.opentofu.org/hashicorp/aws" { |
| + version = "6.55.0" |
| + constraints = "~> 6.0" |
| + hashes = [ |
| + "h1:+G+qu+sZSsc6e4qCUwXWNtRKkwjpjPifRww5DsuTHdo=", |
| + "h1:1SXPVJbjeofrYsc/ykMMo8n4hMHhSfo9g04kKk021Ls=", |
| + "h1:9WBgWIOdQ7ahIkPtaoJnDMDp1gQ0C4duyb/Ky0YLRPA=", |
| + "h1:BzbhNaTSZ+THqY+qW2gQ+4BN4L2ktOppoySPqTuK/n4=", |
| + "h1:CWzHw09w02N7/srODY75wtKeBCn99tIg1/JOR349vpU=", |
| + "h1:D2dG84nEJX5t6D40dg+wjaYn84hdDEKIRNZjRbw/K20=", |
| + "h1:T+/L7WWTNCEroeqg9CqttycX7cTS/w1MYsWmNMIehxU=", |
| + "h1:W8/qF1ERN6TRvRZ1ibJ0vDfFFWmsW4oDDeKIB68CFUE=", |
| + "h1:bIGF7sF6iXj3UzsFgJsJmoh2u1BqXRHvE7OVXL2dPrU=", |
| + "h1:gKAQuYX5TYXLdAHHFzFl4sUY73kvt71tWHTdQS0qgfk=", |
| + "h1:hXkvjHUIlMo9Mx7wnQ/YnDCiFbMkZvP5l5d0napXEro=", |
| + "h1:jwx90C/f0W/WSzP/J5Ce1ZgVKD9Wufauy+pFQpuucFM=", |
| + "h1:l0r4f1CRMiQL1CS8iA0GHmxU7UtFS7lTzlJa5xc2Bl0=", |
| + "h1:s0vAe8fADm5UOlyHLyO5rQ3EOfAxqD3aYYHrziqZoRw=", |
| + "h1:toIB+e3kXr7RyBDSq0fuNXRQI1OlSb9CANqxWeizb2M=", |
| + "zh:01040047b0d83d38c42fc0135b27127da99a86ae201bbed76404bc099678a3f8", |
| + "zh:17286b6a9d51417f237f052f21e2b634505986e0f5374b4141db7415c25dac61", |
| + "zh:17e119fdf057ecca1b5b984effad80bfaa703507f426c8eb9804c1be0d798a73", |
| + "zh:2e60d1bc96cb8ab84959699f80d8e06a26d48bf923e095d5dca2060b2cd02149", |
| + "zh:628df232a979eb1343cfdfb8808b32808eff3703862d622baf72686e9d2418fe", |
| + "zh:7324fead0db670667070f005ff381cc4e8cdccc67e1da47a5905131e60eef2b3", |
| + "zh:7956da45218866045a5a9f8b2aa15567d48461b2102c54af4d23952d79b3b049", |
| + "zh:842eb5192b8f2d96a5a2ecc23a7a4c62dc19d6ba1e95eec544d0692cd9ba9d2a", |
| + "zh:9a4e7f197bb00e0b6286535629a285c3c08736bb363dc12f388f52ebfa47a7ea", |
| + "zh:aeedd0f8b583955d947b0b51b2fa0fbfaaa3cab4ad913aa3b3b1a97bd453161c", |
| + "zh:b5903f3a41ad3843fdc4d54c5ba59e776940378afc9759d3b82403e3b34f190a", |
| + "zh:bca838dfa5b141828df9760bed118a3aeb7f91570f3316896fe7af2493418894", |
| + "zh:c31c408632d94b1142a7e6c5cf81273378343b4b91bcfb408282cd6ea4478d34", |
| + "zh:de80dc1fbe0def1d93b4aaf8bd0facb27adead885dfda9e4934ae7698a0aef13", |
| + "zh:ea482ccdd358d2beec89c07135468f366a5c99f8e74c4b0aab4902ec6bec62e7", |
| + ] |
| +} |
diff --git a/infra/dev-mesh/README.md b/infra/dev-mesh/README.md| new file mode 100644 |
| index 0000000..50b6b84 |
| --- /dev/null |
| +++ b/infra/dev-mesh/README.md |
| @@ -0,0 +1,39 @@ |
| +# 🤖 infra/dev-mesh — the AWS half of the two-node dev mesh |
| + |
| +The Dublin (`ie`) node of the relaxed, cut-off staging fleet. Design + the |
| +prod per-country model it rehearses: `docs/dev-mesh.md`. Its own tofu state |
| +(`dev-mesh/terraform.tfstate`) so it tears down independently of the `#550` |
| +activator box in `infra/dev`. |
| + |
| +Manages the static skeleton only: the EC2 box + EIP, its security group, the |
| +instance role, the shared `eleven-dev-mesh-*` S3 bucket, the Helsinki node's |
| +scoped IAM user, the **Route53 `dev.11ch.at` hosted zone** (delegated from the |
| +prod `11ch.at` DNSimple zone) and its wildcards, and the DNS-01 IAM the boxes |
| +use to obtain the shared `*.dev.11ch.at` cert. The Helsinki (`fi`) node itself |
| +is Hetzner, hand-provisioned like prod's Hetzner boxes. |
| + |
| +**DNS lives in Route53 now** (`route53.tf`): `dev.11ch.at` is its own hosted |
| +zone in the dev AWS account, NS-delegated from DNSimple (`dns.tf`). That is the |
| +region-neutral base domain for multi-edge Phase 1 — new spaces mint as |
| +`<sqid>.dev.11ch.at` and both boxes obtain the `*.dev.11ch.at` wildcard cert |
| +over Route53 DNS-01 with their AWS role/user, no DNS token on a box |
| +(`route53dns.go`, `LCHAT_DNS_PROVIDER=route53`). The per-country |
| +`*.ie.dev` / `*.fi.dev` wildcards moved into this zone too (the delegation makes |
| +DNSimple non-authoritative below `dev.11ch.at`). DNSimple keeps only the `dev` |
| +NS delegation and the `home.dev.elevenmessenger.com` A record. Cutover + |
| +box-reconfig runbook: `docs/multi-edge-dev-cutover.md`. |
| + |
| +```sh |
| +cd infra/dev-mesh |
| +# DNSimple lives in the Eleven account (176940) — token from the vault: |
| +set -a; . <(cd ~/elevenmessenger/secrets-vault && sops -d aws-dev-agent-james.env | grep '^DNSIMPLE_'); set +a |
| +AWS_PROFILE=eleven-dev tofu init |
| +AWS_PROFILE=eleven-dev tofu apply |
| +``` |
| + |
| +The Helsinki node's scoped S3 secret is a sensitive output — seal it into the |
| +vault, never commit it: |
| + |
| +```sh |
| +AWS_PROFILE=eleven-dev tofu output -raw fi_node_secret_key # → vault, in your terminal |
| +``` |
diff --git a/infra/dev-mesh/compute.tf b/infra/dev-mesh/compute.tf| new file mode 100644 |
| index 0000000..00170fe |
| --- /dev/null |
| +++ b/infra/dev-mesh/compute.tf |
| @@ -0,0 +1,116 @@ |
| +# 🤖 The Dublin (ie) node — a Graviton box that reads the dev-mesh bucket via an |
| +# INSTANCE ROLE (no AWS keys on the box, the clean path). Ubuntu 26.04 arm64 to |
| +# mirror prod's OS; arm64 pairs with the amd64 Helsinki node to prove the mesh |
| +# works cross-arch (the release feed builds both every publish). |
| + |
| +data "aws_ami" "ubuntu_2604_arm" { |
| + most_recent = true |
| + owners = ["099720109477"] # Canonical |
| + |
| + filter { |
| + name = "name" |
| + values = ["ubuntu/images/hvm-ssd*/ubuntu-*-26.04-arm64-server-*"] |
| + } |
| + filter { |
| + name = "architecture" |
| + values = ["arm64"] |
| + } |
| +} |
| + |
| +resource "aws_key_pair" "ie_node" { |
| + key_name = "eleven-dev-mesh-ie" |
| + public_key = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILReNBCz+nwYvJ7x/RfgW/kbvSFe3pzPhOBM1SOqbjSg eleven-dev-box" |
| +} |
| + |
| +variable "ssh_ingress_cidr" { |
| + description = "Where SSH is allowed from (the operator's current IP/32)" |
| + type = string |
| + default = "109.255.91.23/32" |
| +} |
| + |
| +resource "aws_security_group" "ie_node" { |
| + name = "eleven-dev-mesh-ie" |
| + description = "dev-mesh Dublin node: ssh from the operator, web open" |
| + |
| + ingress { |
| + description = "ssh (operator only)" |
| + from_port = 22 |
| + to_port = 22 |
| + protocol = "tcp" |
| + cidr_blocks = [var.ssh_ingress_cidr] |
| + } |
| + ingress { |
| + description = "http (autocert HTTP-01 + serving)" |
| + from_port = 80 |
| + to_port = 80 |
| + protocol = "tcp" |
| + cidr_blocks = ["0.0.0.0/0"] |
| + } |
| + ingress { |
| + description = "https" |
| + from_port = 443 |
| + to_port = 443 |
| + protocol = "tcp" |
| + cidr_blocks = ["0.0.0.0/0"] |
| + } |
| + egress { |
| + from_port = 0 |
| + to_port = 0 |
| + protocol = "-1" |
| + cidr_blocks = ["0.0.0.0/0"] |
| + } |
| +} |
| + |
| +resource "aws_iam_role" "ie_node" { |
| + name = "eleven-dev-mesh-ie" |
| + assume_role_policy = jsonencode({ |
| + Version = "2012-10-17" |
| + Statement = [{ |
| + Action = "sts:AssumeRole" |
| + Effect = "Allow" |
| + Principal = { Service = "ec2.amazonaws.com" } |
| + }] |
| + }) |
| +} |
| + |
| +resource "aws_iam_role_policy" "ie_node" { |
| + name = "eleven-dev-mesh-bucket" |
| + role = aws_iam_role.ie_node.id |
| + policy = data.aws_iam_policy_document.mesh_rw.json # shared with the fi user (storage.tf) |
| +} |
| + |
| +resource "aws_iam_instance_profile" "ie_node" { |
| + name = "eleven-dev-mesh-ie" |
| + role = aws_iam_role.ie_node.name |
| +} |
| + |
| +resource "aws_instance" "ie_node" { |
| + ami = data.aws_ami.ubuntu_2604_arm.id |
| + instance_type = "t4g.small" # 2 vCPU / 2 GB — the sizing floor for a node that runs a sidecar |
| + key_name = aws_key_pair.ie_node.key_name |
| + vpc_security_group_ids = [aws_security_group.ie_node.id] |
| + iam_instance_profile = aws_iam_instance_profile.ie_node.name |
| + |
| + # IMDSv2 required: an SSRF on the box can't read the role credentials with a |
| + # bare GET. Hop limit 1 (nothing runs in containers here). |
| + metadata_options { |
| + http_endpoint = "enabled" |
| + http_tokens = "required" |
| + } |
| + |
| + root_block_device { |
| + volume_size = 20 |
| + volume_type = "gp3" |
| + encrypted = true |
| + } |
| + |
| + tags = { Name = "eleven-dev-mesh-ie" } |
| +} |
| + |
| +# A stable address the DNS wildcard points at, so a stop/start (or a -replace of |
| +# the box) doesn't move the record. |
| +resource "aws_eip" "ie_node" { |
| + instance = aws_instance.ie_node.id |
| + domain = "vpc" |
| + tags = { Name = "eleven-dev-mesh-ie" } |
| +} |
diff --git a/infra/dev-mesh/dns.tf b/infra/dev-mesh/dns.tf| new file mode 100644 |
| index 0000000..c2b4de7 |
| --- /dev/null |
| +++ b/infra/dev-mesh/dns.tf |
| @@ -0,0 +1,47 @@ |
| +# 🤖 The DNSimple side of the dev mesh: the NS delegation that hands the |
| +# dev.11ch.at subtree to the Route53 zone (route53.tf), plus the one dev record |
| +# that stays on elevenmessenger.com. The two *.<cc>.dev.11ch.at wildcards that |
| +# used to live here MOVED into the Route53 zone (route53.tf) — see there for why. |
| + |
| +variable "fi_node_ip" { |
| + description = "The Helsinki (Hetzner) node's public IP (the *.fi.dev.11ch.at target)" |
| + type = string |
| + default = "157.180.123.183" |
| +} |
| + |
| +# Delegate dev.11ch.at from the prod 11ch.at DNSimple zone (Eleven account |
| +# 176940) to the Route53 hosted zone. One NS record per Route53 nameserver, all |
| +# named "dev". After this propagates, DNSimple answers anything under dev.11ch.at |
| +# with an NS referral and Route53 is authoritative — which is the whole reason the |
| +# per-country wildcards had to move into that zone. |
| +# |
| +# ⚠️ This is the ONLY record this dev stack writes into the prod DNSimple zone, |
| +# and it touches ONLY the dev subtree — it cannot affect 11ch.at itself or any |
| +# member space. The 176940 token comes from the vault at apply time |
| +# (aws-dev-agent-james.env), never a dev box, never committed. |
| +# count (not for_each): a Route53 hosted zone's delegation set is ALWAYS exactly |
| +# four nameservers, and their names are known only after the zone is created — a |
| +# for_each keyed on them can't plan. A static count of 4 sidesteps that and still |
| +# creates one `dev` NS record per Route53 nameserver. |
| +resource "dnsimple_zone_record" "dev_delegation" { |
| + count = 4 |
| + |
| + zone_name = "11ch.at" |
| + name = "dev" |
| + type = "NS" |
| + value = aws_route53_zone.dev.name_servers[count.index] |
| + ttl = 3600 |
| +} |
| + |
| +# The dev home server lives on elevenmessenger.com (mirroring prod's |
| +# home.elevenmessenger.com), not 11ch.at — the WebAuthn RP identity, and where |
| +# native dev alts point HomeClient. It is OUTSIDE the dev.11ch.at delegation, so |
| +# it stays a plain DNSimple A record on the Dublin (ie) box (the dev "primary" |
| +# that also hosts the global home + the ie cloud provisioner). |
| +resource "dnsimple_zone_record" "home_dev" { |
| + zone_name = "elevenmessenger.com" |
| + name = "home.dev" |
| + type = "A" |
| + value = aws_eip.ie_node.public_ip |
| + ttl = 60 |
| +} |
diff --git a/infra/dev-mesh/main.tf b/infra/dev-mesh/main.tf| new file mode 100644 |
| index 0000000..23c5a71 |
| --- /dev/null |
| +++ b/infra/dev-mesh/main.tf |
| @@ -0,0 +1,53 @@ |
| +# 🤖 The dev mesh — the AWS (Dublin, ie) half of the two-node staging fleet |
| +# (docs/dev-mesh.md). Its own state key so it tears down independently of the |
| +# #550 activator-proof box in infra/dev. The Helsinki (fi) node is Hetzner, |
| +# hand-provisioned (like prod's Hetzner boxes) — only its scoped S3 credential |
| +# lives here. |
| +# |
| +# Boundary rule (infra/README.md) still holds: this manages the static |
| +# skeleton — box, bucket, IAM, DNS wildcards — never an instance hostname. |
| + |
| +terraform { |
| + required_version = ">= 1.10" |
| + |
| + required_providers { |
| + aws = { |
| + source = "hashicorp/aws" |
| + version = "~> 6.0" |
| + } |
| + dnsimple = { |
| + source = "dnsimple/dnsimple" |
| + version = "~> 1.0" |
| + } |
| + } |
| + |
| + backend "s3" { |
| + bucket = "eleven-tofu-state-427880590669" |
| + key = "dev-mesh/terraform.tfstate" |
| + region = "eu-west-1" |
| + use_lockfile = true |
| + } |
| +} |
| + |
| +provider "aws" { |
| + region = "eu-west-1" |
| + |
| + default_tags { |
| + tags = { |
| + project = "eleven" |
| + env = "dev-mesh" |
| + managed = "opentofu" |
| + } |
| + } |
| +} |
| + |
| +# 11ch.at lives in the Eleven DNSimple account (176940), not Paul's personal |
| +# one — DNSIMPLE_TOKEN + DNSIMPLE_ACCOUNT come from the vault at apply time |
| +# (aws-dev-agent-james.env), never committed. |
| +provider "dnsimple" {} |
| + |
| +data "aws_caller_identity" "current" {} |
| + |
| +locals { |
| + account_id = data.aws_caller_identity.current.account_id |
| +} |
diff --git a/infra/dev-mesh/outputs.tf b/infra/dev-mesh/outputs.tf| new file mode 100644 |
| index 0000000..69d9cc8 |
| --- /dev/null |
| +++ b/infra/dev-mesh/outputs.tf |
| @@ -0,0 +1,33 @@ |
| +# 🤖 |
| + |
| +output "ie_node_ip" { |
| + description = "The Dublin node's stable public IP (the *.ie.dev.11ch.at target)" |
| + value = aws_eip.ie_node.public_ip |
| +} |
| + |
| +output "mesh_bucket" { |
| + value = aws_s3_bucket.mesh.bucket |
| +} |
| + |
| +# The Route53 zone's authoritative nameservers — the values the DNSimple `dev` |
| +# NS delegation points at (dns.tf wires this automatically). Handy to eyeball |
| +# delegation propagation: `dig NS dev.11ch.at` should return these. |
| +output "dev_zone_id" { |
| + value = aws_route53_zone.dev.zone_id |
| +} |
| + |
| +output "dev_zone_nameservers" { |
| + value = aws_route53_zone.dev.name_servers |
| +} |
| + |
| +# The Helsinki node's scoped S3 credential. Secret is sensitive — read it with |
| +# `tofu output -raw fi_node_secret_key` and seal it straight into the vault |
| +# (never commit, never echo into a shared transcript). |
| +output "fi_node_access_key_id" { |
| + value = aws_iam_access_key.fi_node.id |
| +} |
| + |
| +output "fi_node_secret_key" { |
| + value = aws_iam_access_key.fi_node.secret |
| + sensitive = true |
| +} |
diff --git a/infra/dev-mesh/route53.tf b/infra/dev-mesh/route53.tf| new file mode 100644 |
| index 0000000..2c149dc |
| --- /dev/null |
| +++ b/infra/dev-mesh/route53.tf |
| @@ -0,0 +1,91 @@ |
| +# 🤖 The delegated Route53 hosted zone for dev.11ch.at — the region-neutral base |
| +# domain for multi-edge Phase 1 (docs/multi-edge.md). It is NS-delegated from the |
| +# prod 11ch.at DNSimple zone (176940) in dns.tf, so DNSimple stays the registrar |
| +# and authoritative for everything ELSE under 11ch.at; only the dev.11ch.at |
| +# subtree lives here, in the dev AWS account. |
| +# |
| +# Why a Route53 zone at all: the dev boxes need a *.dev.11ch.at WILDCARD cert |
| +# (both boxes must terminate TLS for any region-neutral host, since either may own |
| +# it by home_region or proxy to the box that does — mesh.go). A wildcard cert is |
| +# DNS-01 only, and Route53's per-zone IAM lets a box solve that challenge with its |
| +# AWS role/user — NO DNS token on disk (route53dns.go, LCHAT_DNS_PROVIDER=route53). |
| +# DNSimple's account-scoped tokens can't be confined to one zone, which is the |
| +# friction this sidesteps. DNSimple remains a fully-supported provider everywhere |
| +# else — this is the AWS off-ramp, additive, never a replacement. |
| + |
| +resource "aws_route53_zone" "dev" { |
| + name = "dev.11ch.at" |
| + comment = "dev mesh — region-neutral space addressing (docs/multi-edge.md)" |
| +} |
| + |
| +# The region-neutral wildcard: new spaces mint as <sqid>.dev.11ch.at — ONE |
| +# namespace, no country label (the global id ledger keeps ids fleet-unique, |
| +# idalloc). Points at the Dublin (ie) box, the mesh front door: it receives any |
| +# region-neutral host and either serves it (home_region == ie) or reverse-proxies |
| +# to the box whose region owns it (mesh.go proxyTo). A single-label wildcard, so |
| +# it never shadows the deeper two-label *.ie.dev / *.fi.dev records below. |
| +resource "aws_route53_record" "dev_wildcard" { |
| + zone_id = aws_route53_zone.dev.zone_id |
| + name = "*.dev.11ch.at" |
| + type = "A" |
| + ttl = 60 |
| + records = [aws_eip.ie_node.public_ip] |
| +} |
| + |
| +# The two per-country wildcards, MOVED here from DNSimple (they were in the old |
| +# dns.tf). The delegation makes DNSimple non-authoritative for the whole |
| +# dev.11ch.at subtree, so these must live in THIS zone or every existing |
| +# *.ie.dev / *.fi.dev host — including each box's own one.<cc>.dev.11ch.at self |
| +# URL (LCHAT_MESH_SELF) — would stop resolving. Kept so the existing per-country |
| +# hosts and the mesh peer-dial addresses survive the cutover unchanged. |
| +resource "aws_route53_record" "ie_wildcard" { |
| + zone_id = aws_route53_zone.dev.zone_id |
| + name = "*.ie.dev.11ch.at" |
| + type = "A" |
| + ttl = 60 |
| + records = [aws_eip.ie_node.public_ip] |
| +} |
| + |
| +resource "aws_route53_record" "fi_wildcard" { |
| + zone_id = aws_route53_zone.dev.zone_id |
| + name = "*.fi.dev.11ch.at" |
| + type = "A" |
| + ttl = 60 |
| + records = [var.fi_node_ip] |
| +} |
| + |
| +# ---- DNS-01 permissions for the *.dev.11ch.at wildcard cert -------------------- |
| +# The boxes publish then delete an _acme-challenge TXT in THIS zone |
| +# (route53dns.go). ListHostedZonesByName and GetChange take no resource-level |
| +# permissions in AWS (they're account-wide by design), so they're granted on "*"; |
| +# the one MUTATING op — ChangeResourceRecordSets — is locked to this single zone. |
| +# A leaked box credential could therefore touch dev.11ch.at records and nothing |
| +# else in the account: not prod DNS (it's in DNSimple), not any other zone. |
| +data "aws_iam_policy_document" "route53_dns01" { |
| + statement { |
| + sid = "DiscoverZoneAndChange" |
| + actions = ["route53:ListHostedZonesByName", "route53:GetChange"] |
| + resources = ["*"] |
| + } |
| + statement { |
| + sid = "WriteChallengeInDevZoneOnly" |
| + actions = ["route53:ChangeResourceRecordSets"] |
| + resources = [aws_route53_zone.dev.arn] |
| + } |
| +} |
| + |
| +# Dublin (ie) reaches Route53 through its EC2 instance role (no keys on the box). |
| +resource "aws_iam_role_policy" "ie_node_route53" { |
| + name = "eleven-dev-mesh-route53-dns01" |
| + role = aws_iam_role.ie_node.id |
| + policy = data.aws_iam_policy_document.route53_dns01.json |
| +} |
| + |
| +# Helsinki (fi) is Hetzner — it can't assume an EC2 role, so the same policy |
| +# attaches to its scoped IAM user (the S3 credential it already carries lives in |
| +# the vault; this just widens that user to the one DNS zone). |
| +resource "aws_iam_user_policy" "fi_node_route53" { |
| + name = "eleven-dev-mesh-route53-dns01" |
| + user = aws_iam_user.fi_node.name |
| + policy = data.aws_iam_policy_document.route53_dns01.json |
| +} |
diff --git a/infra/dev-mesh/storage.tf b/infra/dev-mesh/storage.tf| new file mode 100644 |
| index 0000000..13fa84f |
| --- /dev/null |
| +++ b/infra/dev-mesh/storage.tf |
| @@ -0,0 +1,82 @@ |
| +# 🤖 The dev mesh's object store — deliberately SEPARATE from prod's buckets and |
| +# from infra/dev's proof buckets. Mesh membership is S3 heartbeats under a |
| +# prefix, so a distinct bucket is what physically stops a dev node from ever |
| +# joining the prod mesh (docs/dev-mesh.md § Cut-off). One bucket holds it all: |
| +# membership, the hibernation lease + replicas, and instance DBs. |
| + |
| +resource "aws_s3_bucket" "mesh" { |
| + bucket = "eleven-dev-mesh-${local.account_id}" |
| +} |
| + |
| +# Versioning is the ransomware/oops posture: an overwrite or delete with the |
| +# box's own credentials leaves recoverable versions behind. Dev data is |
| +# disposable, but the machinery must match prod so we exercise the real thing. |
| +resource "aws_s3_bucket_versioning" "mesh" { |
| + bucket = aws_s3_bucket.mesh.id |
| + versioning_configuration { |
| + status = "Enabled" |
| + } |
| +} |
| + |
| +resource "aws_s3_bucket_lifecycle_configuration" "mesh" { |
| + bucket = aws_s3_bucket.mesh.id |
| + rule { |
| + id = "prune-noncurrent" |
| + status = "Enabled" |
| + filter {} |
| + noncurrent_version_expiration { |
| + noncurrent_days = 5 |
| + } |
| + } |
| +} |
| + |
| +resource "aws_s3_bucket_public_access_block" "mesh" { |
| + bucket = aws_s3_bucket.mesh.id |
| + block_public_acls = true |
| + block_public_policy = true |
| + ignore_public_acls = true |
| + restrict_public_buckets = true |
| +} |
| + |
| +# ---- the Helsinki (fi) node's credential -------------------------------------- |
| +# Hetzner can't assume an EC2 role, so the fi node gets a scoped IAM user whose |
| +# policy is confined to this one bucket — the shape a non-AWS provider's |
| +# credential takes in the co-op model. The access key is created here but its |
| +# secret is written to tofu STATE only (S3, private); it must be pulled from |
| +# there into the vault, never committed. See outputs.tf. |
| + |
| +resource "aws_iam_user" "fi_node" { |
| + name = "eleven-dev-mesh-fi" |
| +} |
| + |
| +data "aws_iam_policy_document" "mesh_rw" { |
| + statement { |
| + sid = "ListMeshBucket" |
| + # GetBucketLocation: the hibernation restore (embedded Litestream) resolves |
| + # the bucket's region before reading replicas — without it a woken instance |
| + # 503s with "cannot lookup bucket region" even though object read/write is |
| + # allowed. Bucket-level, so it sits in this ARN-of-the-bucket statement. |
| + actions = ["s3:ListBucket", "s3:ListBucketVersions", "s3:GetBucketLocation"] |
| + resources = [aws_s3_bucket.mesh.arn] |
| + } |
| + statement { |
| + sid = "ReadWriteMeshObjects" |
| + actions = [ |
| + "s3:GetObject", |
| + "s3:PutObject", |
| + "s3:DeleteObject", |
| + "s3:DeleteObjectVersion", |
| + ] |
| + resources = ["${aws_s3_bucket.mesh.arn}/*"] |
| + } |
| +} |
| + |
| +resource "aws_iam_user_policy" "fi_node" { |
| + name = "eleven-dev-mesh-bucket" |
| + user = aws_iam_user.fi_node.name |
| + policy = data.aws_iam_policy_document.mesh_rw.json |
| +} |
| + |
| +resource "aws_iam_access_key" "fi_node" { |
| + user = aws_iam_user.fi_node.name |
| +} |