amadan
spec: multi-region ingress — TLS at every edge (#238)
Rung-2+ of the HA ladder, designed with Paul 2026-08-20: any edge any app; one elected issuer with bucket-distributed certs; edge-to-edge backhaul on 443 with a one-hop guard; per-route latency steering opt-in; flagship first, tito UK/SE lands as edges (stage 6). The writer map fixes #158 (per-box snapshot keys) and puts the releases prune behind the housekeeper lease. Read-local/write-home and per-instance residence are anticipated as committed shapes, not built.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 file changed,
+272
−0
docs/superpowers/specs/2026-08-20-multi-region-ingress-design.md+272 −0
diff --git a/docs/superpowers/specs/2026-08-20-multi-region-ingress-design.md b/docs/superpowers/specs/2026-08-20-multi-region-ingress-design.md| new file mode 100644 |
| index 000000000..264f263fb |
| --- /dev/null |
| +++ b/docs/superpowers/specs/2026-08-20-multi-region-ingress-design.md |
| @@ -0,0 +1,272 @@ |
| +# Multi-region ingress — TLS termination at every edge (#238) |
| + |
| +Status: designed with Paul in-session 2026-08-20; each section approved as |
| +presented. Successor to the HA ladder's rung-2 sketch |
| +(`2026-08-12-ha-posture-and-multi-upstream-routes-design.md` Part I) — this |
| +is the "distribution, not just recovery" half. Closes the design half of |
| +issue #238; the #158 fix ships inside stage 1. |
| + |
| +## Decisions (Paul, 2026-08-20) |
| + |
| +- **Ordering**: multi-region ingress designs and ships BEFORE UK/SE tito |
| + fleet members. Placing instances regionally while every client routes |
| + through Dublin delivers capacity, not latency; the tito capacity crisis |
| + was solved separately (t4g.xlarge resize, same day). |
| +- **Traffic model: any edge, any app.** Every edge terminates TLS for |
| + every route. Per-app regional homing alone was rejected; "homing now, |
| + any-edge later" was rejected in favour of doing the invariants once. |
| +- **Certs: one elected issuer, distributed via the control bucket.** |
| + Every-edge-mints and hybrid were rejected (shared ACME account budget, |
| + order-state collisions, two validation paths forever). |
| +- **Backhaul: edge-to-edge over public 443.** Tunnel-hub transport and |
| + VPC peering rejected (hub hops for publicly-addressable boxes; cloud |
| + coupling that breaks customer-boxes-anywhere). |
| +- **First target: flagship** (au/uk/us boxes exist, idle since 2026-08-11; |
| + a steering or cert mistake burns nobody's ticket sales). Tito UK/SE is |
| + the second, boring deployment of a proven pattern. |
| +- **Steering is per-route opt-in; serving capability is automatic.** |
| + Default = today's single-region DNS. `steering` is a string field |
| + (`"latency"` in v1), never a bool. |
| +- **Bucket topology: one control bucket per deployment, forever; data |
| + buckets are regional.** CAS/leases/serial demand a single linearizable |
| + home; instance data lives in its region's data bucket. |
| +- **Anticipated, not built** (shapes committed now, features later): |
| + read-local/write-home routing; per-instance residence choice with |
| + data-following moves. |
| + |
| +## Vocabulary |
| + |
| +- **Edge / platform box**: a box converging the deployment bucket and |
| + terminating TLS. After this spec, a deployment has N of them (the |
| + original box plus armed region hosts). Distinct from fleet members, |
| + which reach the bucket only through the store door and never terminate |
| + deployment TLS. |
| +- **Home**: the box where a home-boxed route's process lives. Exactly one |
| + per route, unchanged. |
| +- **Steered**: a hostname whose DNS answers with latency records across |
| + all armed edges. |
| +- **Housekeeper**: the single lease-elected box performing |
| + deployment-global duties (cert issuance, releases prune). |
| + |
| +## Part 1 — the writer map |
| + |
| +Rule: **data that is per-box gets per-box keys; duties that are |
| +deployment-global get a lease.** The status publisher's CAS-lease is the |
| +one election mechanism, reused; no second election pattern may be |
| +introduced. |
| + |
| +| Prefix | Writer today | Ruling | |
| +|---|---|---| |
| +| `apps/**`, `domains/*`, `control/serial`, restart-gen | console | unchanged — console stays the single control-plane writer, homed on one box; remote edges backhaul console traffic to it | |
| +| `control/registry-snapshot*` (fixed keys) | any platform box — **#158** | **per-box keys**: `control/registry-snapshot/<box>.db` + `<box>.json`; generations `control/registry-snapshots/<box>/<stamp>.db`. Every box snapshots its own registry. Closes #158; retires the pinfra #15 blocker | |
| +| `status/**` | lease-elected publisher | unchanged; verify the lease is deployment-scoped (it is bucket-scoped, which is the same thing) | |
| +| `releases/**` retention prune | every box | moves behind the **housekeeper lease** — five concurrent pruners racing a promote is a bug class we refuse to have | |
| +| usage metering day objects | the (single) edge | **audit**: if day-object keys are not box-scoped, they become `…/<box>/…` in the same shape as snapshots | |
| +| `control/certs/**`, `control/acme-challenges/**` (new) | housekeeper | Part 2 | |
| +| `control/edges/<box>.json` (new) | each box, own key only | Part 3 | |
| +| `control/edge-mesh-key` (new) | housekeeper mints if absent (create-only CAS) | Part 3 | |
| + |
| +Stage-1 audit deliverable: an enumerated list of EVERY bucket writer in |
| +the codebase with its ruling from the table above, checked into the spec's |
| +plan; anything discovered outside this table gets a ruling before stage 2. |
| + |
| +**Snapshot key migration** is a storage-shape change; the consumer list is |
| +what PARSES it: the DR runbook (teamtito/carlos-infrastructure |
| +`RUNBOOK-edge-failover.md`) and nothing else — flagship has no DR runbook |
| +today, a gap this migration's PR records rather than hides. New keys and runbook updates land in the same PR; the old fixed |
| +keys are deleted only after both deployments run the new binary. While |
| +both key shapes exist the runbooks name the new keys only. |
| + |
| +**Bucket topology.** One control bucket per deployment, in the |
| +deployment's home region, forever: serial bumps, leases, create-only |
| +claims and record CAS are linearizable only within one bucket, and |
| +cross-region S3 replication is async — leases across replicas are broken |
| +by construction. The control plane is never on the request path (edges |
| +serve from local converged state), so remote boxes pay ~150ms on a 2s |
| +background poll, which is irrelevant. Accepted consequence: a control- |
| +bucket-region S3 outage freezes the control plane globally (no |
| +converges, no cert renewals) while every edge keeps serving |
| +last-converged state. Cross-region control-bucket replicas are an |
| +explicit non-goal of this spec. |
| + |
| +Regional **data** buckets already exist (flagship `carlos-data-{ie,au,uk,us}`) |
| +and keep their role: a box's litestream/instance-data replication and |
| +region-homed object stores write to the box's own region's data bucket. |
| +Tito's UK/SE boxes get `carlos-tito-{uk,se}` data buckets when they |
| +arrive. One deliberate cross-over: **registry snapshots go to the control |
| +bucket, not the regional data bucket** — the AU box's snapshot exists to |
| +rebuild the AU box after an AU-region failure and must not be stored in |
| +the blast radius it insures against. Certs never appear in data buckets. |
| + |
| +## Part 2 — certificates |
| + |
| +**Issuer.** The housekeeper runs today's issuance machinery unmodified: |
| +per-host ACME HTTP-01, DNS-01 wildcards, the delegation zone |
| +(`CARLOS_ACME_DELEGATION_ZONE`), and the HostPolicy rulings from PR #180 |
| +(EXACT-only per-host orders; a catch-all never enables per-host issuance). |
| +One ACME account, one order stream — the shared-account blast-radius |
| +rulings hold by construction. |
| + |
| +**Distribution.** Minted and renewed certs publish to |
| +`control/certs/<host>.json` — chain, private key, `expires_at` — and the |
| +write bumps `control/serial`. Every edge mirrors the prefix on the |
| +existing 2s serial cadence; `GetCertificate` serves from the local |
| +mirror. Renewals run ~30 days early, so distribution latency is never on |
| +a deadline. |
| + |
| +**Keys in the bucket — the trust statement.** Any box with deployment- |
| +bucket read already serves this deployment's traffic and reads its |
| +releases; holding TLS keys does not extend what a compromised box can |
| +do. Hard guards: the fleet store door's ACL is an allowlist that never |
| +includes `control/certs/**` or `control/edge-mesh-key` (fleet members and |
| +their tokens can never read keys — this gets a permanent test), and |
| +certs never replicate to data buckets. |
| + |
| +**HTTP-01 under latency DNS** — the one new mechanism. LE's validators |
| +hit whichever edge is nearest to them, not the issuer. Before answering |
| +an order, the issuer writes each active token to |
| +`control/acme-challenges/<token>`; EVERY edge serves |
| +`/.well-known/acme-challenge/*` from its mirror of that prefix. Tokens |
| +are deleted after validation and carry a TTL stamp so a crashed order |
| +never leaves permanent litter. The issuer remains the only box that |
| +talks to the CA. |
| + |
| +**Issuer death**: the lease expires and the next box's housekeeper takes |
| +over with identical machinery. Distributed certs keep every edge serving |
| +through the gap; a dead issuer costs nothing until the earliest renewal |
| +window ~30 days out. |
| + |
| +## Part 3 — serving classes and backhaul |
| + |
| +| Class | Kinds | Behaviour on a non-home edge | |
| +|---|---|---| |
| +| Everywhere | static, bake-served | adopt and serve locally from converged artifacts — zero backhaul | |
| +| Everywhere, probed | addr routes with publicly-reachable upstreams (incl. #152 multi-upstream) | each edge runs its own prober and dials upstreams directly; per-edge live sets are per-edge by design. Loopback/link-local addrs classify as home-boxed (another box's 127.0.0.1 is not dialable) | |
| +| Home-boxed | exec, unit, socket-activated, tunnel-backed | backhaul to the home edge. The process exists on exactly one box — unchanged invariant | |
| + |
| +**Home resolution.** Each box maintains `control/edges/<box>.json`: its |
| +EIP and a map `host → {role}` for the routes it homes, written by its |
| +converge loop, serial-bumped on change. v1 writes only `role: "home"`. |
| +Every edge folds these files into a deployment-wide host→home map. The |
| +file doubles as the DR runbook's explicit statement of what a dead box |
| +was serving. |
| + |
| +**The hop.** A remote edge dials `home-EIP:443` with ordinary SNI — the |
| +home edge presents the same bucket-distributed cert. Two headers make it |
| +a trusted hop: an HMAC over (host, timestamp) keyed by |
| +`control/edge-mesh-key`, and `X-Carlos-Hop: 1`. A receiving edge with a |
| +valid HMAC serves the route locally, honours the appended |
| +`X-Forwarded-For`, and hard-refuses to forward again — **one hop, ever**. |
| +If its map says the host homes elsewhere (stale map mid-move), it |
| +answers 503 + `Retry-After: 2` and lets converge heal it; it never |
| +re-forwards. Invalid or absent HMAC → the request is an ordinary client: |
| +forwarded headers stripped, normal serving rules. Timestamp skew bound |
| +±30s (boxes run NTP). Key rotation: dual-accept window, old + new both |
| +verify until every edge converges the new key. |
| + |
| +**Hibernation** composes unchanged: the home edge owns wake (wake-once |
| +singleflight, #199), and a backhaul hop is just another first client. |
| + |
| +**Steering opt-in.** New additive record field `steering` (string; |
| +`"latency"` is the only v1 value; absent = today's single-region DNS). |
| +Member-settable via console and CLI like any record field; |
| +fingerprint-covered; rides the #137 record shape. Serving capability |
| +exists on every edge regardless — steering only controls whether DNS |
| +sends anyone there. |
| + |
| +Consequences stated honestly: |
| +- an opted-in addr route is probed by N edges — N× probe cadence on the |
| + member's health endpoint. Trivial at our N; documented for members |
| + (Eleven). |
| +- **#170's blast-radius statement updates**: member-stated upstreams are |
| + dialled from every edge, not one. The trust ruling that gates the |
| + member addrs door gates steering of member addr routes identically. |
| + |
| +## Part 4 — steering mechanics, failure, rollout, testing |
| + |
| +**DNS.** Route53 latency records (both production zones are already |
| +Route53). A steered apps-domain hostname gets one latency record per |
| +armed edge, each bound to a Route53 health check on that edge's EIP |
| +(HTTPS + SNI), TTL 60. Console-driven through the existing zone |
| +machinery — never tofu-per-app. Custom domains: CNAME to a steering |
| +hostname (shape: `edge.<apps-domain>`, itself latency-routed) gets full |
| +steering; apexes cannot CNAME, so apex guidance remains |
| +A-records-to-all-EIPs — round-robin without latency bias, documented as |
| +such in attach guidance. Edge death: health checks withdraw its records |
| +in ~90s globally — steered routes thereby get most of rung-2 warm |
| +standby for free. |
| + |
| +**Failure modes.** |
| +- Edge dies: steered → fails away via health checks; unsteered routes |
| + homed there are down until rung-1 recovery (unchanged today). |
| +- Issuer dies: Part 2 — costs nothing for ~30 days. |
| +- Control-bucket region outage: global control freeze, edges serve |
| + last-converged state. Accepted. |
| +- Stale home map during a move: one 503 + Retry-After, healed in |
| + seconds; the hop guard makes loops structurally impossible. |
| +- Mesh-key rotation: dual-accept window. |
| + |
| +**Rollout.** Stages 1–4 land dormant on single-box deployments |
| +(one-candidate elections; per-box keys with one box; no steered routes): |
| +1. Writer map: per-box snapshot keys (+ runbooks), prune behind |
| + housekeeper lease, usage-key audit, full bucket-writer enumeration. |
| +2. Issuer election + cert distribution + challenge-anywhere. |
| +3. Backhaul: edges files, mesh key, hop serving + guard. |
| +4. Steering: record field, console/CLI setter, Route53 latency records + |
| + health checks (tofu for the health-check scaffolding, console for |
| + per-host records). |
| +5. Arm flagship AU alone; steer one static site and one exec app; |
| + verify from real vantage points; then UK/US. |
| +6. Tito UK/SE: new region hosts + `carlos-tito-{uk,se}` data buckets in |
| + teamtito/carlos-infrastructure; second deployment of a proven |
| + pattern. (This is where the original UK/SE member request lands — |
| + as edges, not door-joined members.) |
| + |
| +**Testing.** In-process two-edge harness (pattern: `fleet_e2e`, |
| +`tunnel_e2e`): fake bucket + fake ACME prove cert distribution, |
| +challenge-served-anywhere, HMAC + hop guard (including the |
| +stale-map-503 and never-double-forward cases), home-map convergence, and |
| +the door-ACL-never-reads-certs guard. Steering verified live per edge |
| +with `curl --resolve` and geo vantage checks; the Route53 record shape |
| +gets a tofu-plan golden. Every stage keeps the full `go build/vet/test` |
| +gate green with stages dormant. |
| + |
| +## Anticipated, not built |
| + |
| +Committed shapes so neither feature requires unwinding this spec: |
| + |
| +1. **Read-local / write-home** (per-app opt-in, never inferred): the |
| + home map's `{role}` object grows a `read` role; edges route safe |
| + methods (GET/HEAD/OPTIONS) to a near `read` box and mutating methods |
| + to `home`. Deferred with it: replica materialization off |
| + `replication`/`backing`, read-your-writes (sticky-to-primary window |
| + after a mutation), and the honesty problem that GET-is-read-only is |
| + an app contract. v1 routes ALL methods to `home`; the role field and |
| + the string-typed `steering` field are the anticipation. |
| +2. **Per-instance residence** (titogo customers choosing regions): an |
| + instance IS a route, and every mechanism here is route-scoped — |
| + stated as invariant: *residence is a property of a route; an app is |
| + just the default its instances inherit* (instance beats app, |
| + precedence fixed now). The registry/record `region` field is the |
| + residence carrier. Deferred to its own spec: the member-facing |
| + residence door, data-following moves (litestream-assisted replicate- |
| + then-cutover — the opposite of fleet-placement's D6 |
| + state-does-not-follow ruling, deliberately), and region-aware unit |
| + economics. |
| + |
| +## Non-goals |
| + |
| +- Anycast. DNS latency records are the steering mechanism; revisit only |
| + if measured DNS behaviour fails real users. |
| +- Cross-region control-bucket replicas. |
| +- Multi-writer console. |
| +- Method-aware routing and residence moves (anticipated above). |
| +- Latency-aware upstream selection within an addr route's `Addrs` set |
| + (round-robin stands; tags/filters are the fleets-v2.1 lane). |
| + |
| +## Issue map |
| + |
| +Closes the design half of #238. Stage 1 closes #158 and unblocks pinfra |
| +#15. Updates #170's blast-radius statement. Touches #207's runbook text |
| +(generation-window note survives per-box keys). The UK/SE tito request |
| +resolves as stage 6 edges, not fleet members. |