amadan

carlos / platform Public

Merge pull request #239 from carlosframework/multi-region-ingress-spec

spec: multi-region ingress — TLS at every edge (#238)
Paul Campbell pushed by paul@keymail.dev 67fb71d562717c911e55df4ee42877e97d5993b2 parent d9a643fparent 13e4d45
2 files changed, +866 −0
  • docs/superpowers/plans/2026-08-20-multi-region-writer-map-and-certs.md +593 −0
  • docs/superpowers/specs/2026-08-20-multi-region-ingress-design.md +273 −0
diff --git a/docs/superpowers/plans/2026-08-20-multi-region-writer-map-and-certs.md b/docs/superpowers/plans/2026-08-20-multi-region-writer-map-and-certs.md
new file mode 100644
index 000000000..046849599
--- /dev/null
+++ b/docs/superpowers/plans/2026-08-20-multi-region-writer-map-and-certs.md
@@ -0,0 +1,593 @@
+# Multi-region ingress, plan 1 of 2: writer map + cert distribution
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use
+> superpowers:subagent-driven-development (recommended) or
+> superpowers:executing-plans to implement this plan task-by-task. Steps
+> use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Ship spec stages 1–2 (writer map fixes + elected-issuer cert
+distribution), fully dormant on today's single-box deployments.
+
+**Architecture:** Per-box snapshot keys close #158; the releases prune
+moves behind a new housekeeper lease (same election pattern as the status
+publisher); cert issuance stays exactly today's autocert machinery but
+only the housekeeper mints, publishing cert material and HTTP-01 tokens
+through the control bucket so every edge can serve both. Plan 2
+(backhaul + steering, spec Parts 3–4) starts only after this plan merges.
+
+**Tech Stack:** Go (platform repo), existing `release.Store` bucket
+interface, `golang.org/x/crypto/acme/autocert` (already vendored via
+edge), existing fake-store test harness.
+
+## Global Constraints
+
+- Spec: `docs/superpowers/specs/2026-08-20-multi-region-ingress-design.md`.
+ Key shapes verbatim from it: `control/registry-snapshot/<box>.db`,
+ `control/registry-snapshot/<box>.json`,
+ `control/registry-snapshots/<box>/<stamp>.db`, `control/certs/**`,
+ `control/housekeeper.json`.
+- **Wire-shape changes are additive.** Old fixed snapshot keys are NOT
+ deleted by this plan; deletion happens after both deployments roll
+ (recorded in the rollout notes, not code).
+- **`control/serial` stays console-written.** Nothing in this plan may
+ write it. The certs mirror refreshes on its own 30s list-poll.
+- **One election pattern.** The housekeeper reuses the status publisher's
+ election approach (CAS-lease in the bucket, renew at half-TTL); no new
+ election mechanism.
+- **The fleet store door ACL never includes `control/certs/**` or
+ `control/edge-mesh-key`** — Task 6 pins this with a permanent test.
+- Gate before every commit: `go build ./... && go vet ./... &&
+ go test ./... -count=1` plus `gofmt -l` clean on touched files, with
+ `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off`
+ (sandbox may need disabling — "read-only file system" is the tell).
+- Tests that shell out to the carlos binary build env through
+ `carlosTestEnv` and set `cmd.Dir` to a scratch dir (standing rule; no
+ task here should need it, but fix loops sometimes reach for the CLI).
+- Production channel names are pinned in
+ `internal/release/laddercheck_test.go` — nothing here may alter ladder
+ behaviour.
+
+## File Structure
+
+- `internal/host/snapshot_loop.go` — per-box key derivation (Task 1)
+- `internal/host/snapshot_loop_test.go` — updated + two-box test (Task 1)
+- `docs/superpowers/specs/2026-08-20-bucket-writer-audit.md` — audit
+ deliverable (Task 2)
+- `internal/host/housekeeper.go` + `_test.go` — the lease (Task 3)
+- `internal/host/agent.go` — retention call gated by housekeeper (Task 3)
+- `internal/edge/certsync.go` + `_test.go` — publishing cache wrapper +
+ edge mirror (Tasks 4–5)
+- `internal/edge/edge.go` — issuer gating + mirror wiring + challenge
+ route (Tasks 4–5)
+- `internal/edge/store_door_test.go` (or the door's existing test file) —
+ ACL guard (Task 6)
+- `internal/edge/certsync_e2e_test.go` — two-edge proof (Task 7)
+
+---
+
+### Task 1: Per-box registry snapshot keys (closes #158)
+
+**Files:**
+- Modify: `internal/host/snapshot_loop.go` (consts at lines 24–26; every
+ use of them in `registrySnapshotPass`, `seedSnapshotFingerprint`,
+ `readSnapshotMeta`)
+- Test: `internal/host/snapshot_loop_test.go`
+
+**Interfaces:**
+- Consumes: existing `registrySnapshotPass(ctx, s, reg, box, now, lastFP)`
+ — the `box` parameter already exists and becomes the key discriminator.
+- Produces: `snapshotDBKey(box string) string`,
+ `snapshotMetaKey(box string) string`,
+ `snapshotGenPrefix(box string) string` — Task 7's e2e and the DR
+ runbook update rely on these exact shapes.
+
+- [ ] **Step 1: Write the failing tests** — replace fixed-key
+ expectations and add the two-box isolation test:
+
+```go
+func TestSnapshotKeysArePerBox(t *testing.T) {
+ if got, want := snapshotDBKey("annie"), "control/registry-snapshot/annie.db"; got != want {
+ t.Fatalf("db key = %q, want %q", got, want)
+ }
+ if got, want := snapshotMetaKey("annie"), "control/registry-snapshot/annie.json"; got != want {
+ t.Fatalf("meta key = %q, want %q", got, want)
+ }
+ if got, want := snapshotGenPrefix("annie"), "control/registry-snapshots/annie/"; got != want {
+ t.Fatalf("gen prefix = %q, want %q", got, want)
+ }
+}
+
+func TestTwoBoxesSnapshotWithoutColliding(t *testing.T) {
+ s := newFakeStore(t) // the fake used by the existing pass tests
+ regA, regB := scratchRegistry(t), scratchRegistry(t)
+ var fpA, fpB string
+ now := time.Date(2026, 8, 20, 12, 0, 0, 0, time.UTC)
+ if err := registrySnapshotPass(context.Background(), s, regA, "annie", now, &fpA); err != nil {
+ t.Fatal(err)
+ }
+ if err := registrySnapshotPass(context.Background(), s, regB, "betty", now, &fpB); err != nil {
+ t.Fatal(err)
+ }
+ for _, key := range []string{
+ "control/registry-snapshot/annie.db", "control/registry-snapshot/annie.json",
+ "control/registry-snapshot/betty.db", "control/registry-snapshot/betty.json",
+ } {
+ if _, err := s.Get(context.Background(), key); err != nil {
+ t.Fatalf("missing %s: %v", key, err)
+ }
+ }
+ // betty's pass must not have disturbed annie's generations.
+ gens, err := s.List(context.Background(), "control/registry-snapshots/annie/")
+ if err != nil || len(gens) != 1 {
+ t.Fatalf("annie generations = %v, %v; want exactly 1", gens, err)
+ }
+}
+```
+
+(Adopt the existing test file's fake-store and scratch-registry helpers —
+they exist for the current pass tests; do not invent new ones.)
+
+- [ ] **Step 2: Run to verify failure**
+
+Run: `go test ./internal/host/ -run 'TestSnapshotKeys|TestTwoBoxes' -count=1`
+Expected: FAIL — `snapshotDBKey` undefined.
+
+- [ ] **Step 3: Implement.** Replace the three consts with:
+
+```go
+// Per-box keys (spec 2026-08-20 Part 1, closes #158): every platform box
+// snapshots ITS OWN registry under its own name; boxes never contend.
+// The pre-multi-region fixed keys ("control/registry-snapshot.db" etc.)
+// are still present in production buckets until both deployments roll —
+// they are abandoned, not rewritten; deletion is a rollout step, not code.
+func snapshotDBKey(box string) string { return "control/registry-snapshot/" + box + ".db" }
+func snapshotMetaKey(box string) string { return "control/registry-snapshot/" + box + ".json" }
+func snapshotGenPrefix(box string) string {
+ return "control/registry-snapshots/" + box + "/"
+}
+```
+
+Thread `box` to every call site inside the pass, the generation prune
+(which now lists `snapshotGenPrefix(box)`), `seedSnapshotFingerprint`
+(reads own box's sidecar — takes `box string` as a new parameter), and
+`readSnapshotMeta` (same). The foreign-box "OVERWRITING a snapshot
+written by box …" guard becomes unreachable for differently-named boxes;
+keep it — same-named boxes (a rebuilt standby reusing the dead box's
+name per the runbook) still hit it, and that is exactly when it should
+be loud.
+
+- [ ] **Step 4: Run the focused tests, then the package**
+
+Run: `go test ./internal/host/ -count=1`
+Expected: PASS, including all pre-existing snapshot tests updated to the
+per-box shapes.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add internal/host/snapshot_loop.go internal/host/snapshot_loop_test.go
+git commit -m "host: per-box registry snapshot keys — boxes never contend (#158)"
+```
+
+**Cross-repo deliverable (report, don't do):** the DR runbook
+(`teamtito/carlos-infrastructure/RUNBOOK-edge-failover.md`) parses these
+keys; its update is a separate PR listed in this plan's rollout notes.
+The controller opens it after this plan's platform PR merges. Flagship
+has no DR runbook (spec records the gap).
+
+---
+
+### Task 2: Bucket-writer audit (stage-1 deliverable)
+
+**Files:**
+- Create: `docs/superpowers/specs/2026-08-20-bucket-writer-audit.md`
+
+**Interfaces:**
+- Consumes: nothing from other tasks (pure survey; do it against the
+ Task-1 tree).
+- Produces: the audit table later tasks and plan 2 cite; any writer
+ discovered outside the spec's table gets a ruling row here.
+
+- [ ] **Step 1: Enumerate every bucket write.** Survey, verbatim recipe:
+
+```bash
+grep -rn "\.Put(\|\.Delete(\|putReplacing(" internal/ --include='*.go' | grep -v _test
+```
+
+For each call site record: file:line, the key/prefix written, which
+process runs it (console / every box / housekeeper / edge), and the
+spec ruling that covers it. Known-expected rows (verify, don't assume):
+`usage/edge/<box>/days/` (already box-scoped — `internal/usage/edge.go:40`),
+`usage/census/` + `usage/storage/` + statements (console's
+`economicspass.go` — console-only, fine), status publisher (elected),
+snapshots (Task 1), retention deletes (Task 3), records/domains/serial
+(console).
+
+- [ ] **Step 2: Write the audit doc.** One table, same columns as the
+ spec's writer map, one row per call site, each with a ruling of:
+ `console-only` | `per-box keys` | `housekeeper` | `elected already` |
+ `NEEDS RULING`. A `NEEDS RULING` row is a finding: stop and surface it
+ to the controller rather than inventing a ruling.
+
+- [ ] **Step 3: Commit**
+
+```bash
+git add docs/superpowers/specs/2026-08-20-bucket-writer-audit.md
+git commit -m "docs: bucket-writer audit — every write call site ruled for N boxes"
+```
+
+---
+
+### Task 3: Housekeeper lease; releases prune moves behind it
+
+**Files:**
+- Create: `internal/host/housekeeper.go`
+- Test: `internal/host/housekeeper_test.go`
+- Modify: `internal/host/agent.go` (the tick that calls
+ `retentionPass` — find it with `grep -n retentionPass internal/host/agent.go`)
+
+**Interfaces:**
+- Consumes: `retentionPass(ctx, store, pub, reg)` from
+ `internal/host/retention.go:26` (unchanged signature); the status
+ publisher's election in `agent.go`'s statusLoop as the pattern to
+ mirror (read it first — reuse its store client and clock idioms, and
+ if a helper can be shared without contortion, share it).
+- Produces: `acquireHousekeeper(ctx context.Context, s release.Store,
+ box string, now time.Time) (held bool, err error)` — Task 4 gates
+ issuance on it.
+
+- [ ] **Step 1: Failing tests** — CAS contention and expiry:
+
+```go
+func TestHousekeeperSingleHolder(t *testing.T) {
+ s := newFakeStore(t)
+ now := time.Date(2026, 8, 20, 12, 0, 0, 0, time.UTC)
+ heldA, err := acquireHousekeeper(context.Background(), s, "annie", now)
+ if err != nil || !heldA {
+ t.Fatalf("annie should acquire: held=%v err=%v", heldA, err)
+ }
+ heldB, err := acquireHousekeeper(context.Background(), s, "betty", now.Add(10*time.Second))
+ if err != nil || heldB {
+ t.Fatalf("betty must NOT hold while annie's lease is live: held=%v err=%v", heldB, err)
+ }
+ // After expiry betty takes over.
+ heldB, err = acquireHousekeeper(context.Background(), s, "betty", now.Add(housekeeperTTL+time.Second))
+ if err != nil || !heldB {
+ t.Fatalf("betty should acquire after expiry: held=%v err=%v", heldB, err)
+ }
+}
+
+func TestHousekeeperRenewsOwnLease(t *testing.T) {
+ s := newFakeStore(t)
+ now := time.Date(2026, 8, 20, 12, 0, 0, 0, time.UTC)
+ if held, _ := acquireHousekeeper(context.Background(), s, "annie", now); !held {
+ t.Fatal("first acquire")
+ }
+ // Same box re-acquiring mid-TTL renews rather than being locked out.
+ if held, _ := acquireHousekeeper(context.Background(), s, "annie", now.Add(housekeeperTTL/2)); !held {
+ t.Fatal("holder must renew its own live lease")
+ }
+}
+```
+
+- [ ] **Step 2: Verify failure** — `go test ./internal/host/ -run
+ Housekeeper -count=1` → FAIL, undefined.
+
+- [ ] **Step 3: Implement `housekeeper.go`:**
+
+```go
+// Package-internal single election for deployment-global duties (spec
+// 2026-08-20 Part 1): releases retention prune (this plan) and cert
+// issuance (Task 4). ONE lease, several duties — never a second election.
+const (
+ housekeeperKey = "control/housekeeper.json"
+ housekeeperTTL = 90 * time.Second
+)
+
+type housekeeperLease struct {
+ Box string `json:"box"`
+ Expires time.Time `json:"expires"`
+}
+
+// acquireHousekeeper CAS-writes the lease if absent, expired, or already
+// ours (renewal). It never blocks; a false return means another box
+// holds a live lease and the caller simply skips housekeeper duties this
+// tick — duties are idempotent and re-run within a TTL of any handover.
+func acquireHousekeeper(ctx context.Context, s release.Store, box string, now time.Time) (bool, error)
+```
+
+Implementation mirrors the status publisher election: `Get` the key; if
+present, unmarshal, and return true early only when `Box == box ||
+now.After(Expires)` fails the other way — i.e. contend only when absent,
+expired, or self-owned; write with the store's conditional-PUT (the same
+CAS the publisher election and `putReplacing` use — reuse, don't
+duplicate; if extraction of a shared helper from the statusLoop election
+is clean, do it and note it in the report). Losing the CAS race returns
+`(false, nil)`.
+
+- [ ] **Step 4: Gate the prune.** In `agent.go`, wrap the existing
+ `retentionPass` call:
+
+```go
+held, err := acquireHousekeeper(ctx, store, boxName, time.Now())
+if err != nil {
+ log.Printf("housekeeper: election error (prune skipped this tick): %v", err)
+} else if held {
+ if err := retentionPass(ctx, store, pub, reg); err != nil { /* existing handling */ }
+}
+```
+
+(Exact surrounding names come from the call site — keep its logging
+style. On a single-box deployment the box always holds: provably inert.)
+
+- [ ] **Step 5: Package gate + commit**
+
+```bash
+go test ./internal/host/ -count=1
+git add internal/host/housekeeper.go internal/host/housekeeper_test.go internal/host/agent.go
+git commit -m "host: housekeeper lease; releases prune single-holder (spec Part 1)"
+```
+
+---
+
+### Task 4: Issuer-side cert publishing (housekeeper mints, bucket carries)
+
+**Files:**
+- Create: `internal/edge/certsync.go`
+- Test: `internal/edge/certsync_test.go`
+- Modify: `internal/edge/edge.go` (autocert manager construction at
+ ~line 1765; `RunWithActivatorReady`'s HTTP-01 handler wiring ~1222)
+
+**Interfaces:**
+- Consumes: `acquireHousekeeper` (Task 3) via a
+ `held func() bool` closure the host layer already knows how to
+ provide (edge receives it as a field, mirroring how existing
+ host-owned knobs reach edge construction — follow the pattern of the
+ edge's existing config fields); the edge's bucket store handle.
+- Produces: `publishingCache{inner autocert.Cache, s release.Store}`
+ satisfying `autocert.Cache`; bucket key shape
+ `certKey(name string) = "control/certs/" + name` — Task 5's mirror
+ and Task 6's ACL test rely on it.
+
+Key facts for the implementer (verified against the vendored autocert):
+`autocert.Manager` writes EVERYTHING through its `Cache` — account key,
+per-host cert+key PEM under the host name, and HTTP-01 challenge
+responses under `<token-path>+http-01` (cluster support is the
+documented purpose). Wrapping the cache captures certs AND tokens with
+zero autocert forks.
+
+- [ ] **Step 1: Failing tests:**
+
+```go
+func TestPublishingCacheMirrorsPutsToBucket(t *testing.T) {
+ s := newFakeStore(t)
+ pc := &publishingCache{inner: autocert.DirCache(t.TempDir()), s: s}
+ if err := pc.Put(context.Background(), "example.com", []byte("PEM")); err != nil {
+ t.Fatal(err)
+ }
+ got, err := s.Get(context.Background(), "control/certs/example.com")
+ if err != nil || string(got) != "PEM" {
+ t.Fatalf("bucket copy = %q, %v", got, err)
+ }
+ // Local cache also holds it (autocert's own read path keeps working).
+ if b, err := pc.Get(context.Background(), "example.com"); err != nil || string(b) != "PEM" {
+ t.Fatalf("local copy = %q, %v", b, err)
+ }
+}
+
+func TestPublishingCacheDeleteRemovesBucketCopy(t *testing.T) {
+ s := newFakeStore(t)
+ pc := &publishingCache{inner: autocert.DirCache(t.TempDir()), s: s}
+ _ = pc.Put(context.Background(), "tok+http-01", []byte("challenge"))
+ if err := pc.Delete(context.Background(), "tok+http-01"); err != nil {
+ t.Fatal(err)
+ }
+ if _, err := s.Get(context.Background(), "control/certs/tok+http-01"); err == nil {
+ t.Fatal("token must be deleted from bucket after validation")
+ }
+}
+```
+
+- [ ] **Step 2: Verify failure**, then **Step 3: implement** —
+ `publishingCache` methods delegate to `inner` first (local durability),
+ then best-effort mirror to the bucket with `putReplacing`/`Delete`; a
+ bucket error logs loudly (`certsync: bucket publish failed for %q`)
+ and does NOT fail the autocert operation (local issuance must survive
+ bucket blips; the 30s mirror poll self-heals). The account key entry
+ (`acme_account+key`) is mirrored like everything else — a failed-over
+ housekeeper reuses the account instead of registering a new one.
+
+- [ ] **Step 4: Gate issuance on the housekeeper.** In `edge.go` where
+ the manager's `HostPolicy` is set, wrap: when `e.housekeeperHeld()` is
+ false, return a sentinel `errNotIssuer` from HostPolicy — autocert
+ then refuses new orders on non-issuers, while `GetCertificate` still
+ serves anything already in cache/mirror (Task 5). Single-box
+ deployments always hold → behaviour identical to today. Renewals run
+ only where orders can start, so only on the issuer.
+
+- [ ] **Step 5: Package gate + commit**
+
+```bash
+go test ./internal/edge/ -count=1
+git add internal/edge/certsync.go internal/edge/certsync_test.go internal/edge/edge.go
+git commit -m "edge: housekeeper-gated issuance, certs+tokens published to bucket"
+```
+
+---
+
+### Task 5: Edge mirror — serve certs and challenges everywhere
+
+**Files:**
+- Modify: `internal/edge/certsync.go` (mirror type), `internal/edge/edge.go`
+ (GetCertificate layering at `wrapGetCertificate` — `wildcard.go:665` —
+ and the HTTP-01 path)
+- Test: `internal/edge/certsync_test.go`
+
+**Interfaces:**
+- Consumes: `certKey` shape from Task 4.
+- Produces: `certMirror` with `Refresh(ctx) error` (30s loop),
+ `Cert(name string) ([]byte, bool)`, `Challenge(token string) ([]byte,
+ bool)` — Task 7 exercises all three.
+
+- [ ] **Step 1: Failing tests:**
+
+```go
+func TestCertMirrorServesBucketCert(t *testing.T) {
+ s := newFakeStore(t)
+ _ = putReplacing(context.Background(), s, "control/certs/example.com", []byte("PEM"))
+ m := newCertMirror(s)
+ if err := m.Refresh(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ if b, ok := m.Cert("example.com"); !ok || string(b) != "PEM" {
+ t.Fatalf("mirror cert = %q, %v", b, ok)
+ }
+}
+
+func TestCertMirrorServesChallengeTokens(t *testing.T) {
+ s := newFakeStore(t)
+ _ = putReplacing(context.Background(), s, "control/certs/.well-known/acme-challenge/tok+http-01", []byte("resp"))
+ m := newCertMirror(s)
+ _ = m.Refresh(context.Background())
+ if b, ok := m.Challenge("tok"); !ok || string(b) != "resp" {
+ t.Fatalf("challenge = %q, %v", b, ok)
+ }
+}
+```
+
+(The exact cache key autocert uses for HTTP-01 entries must be read from
+the vendored source in Step 3 — the `+http-01` suffix is stable; the
+path prefix in the key mirrors the request path. Pin the REAL observed
+key shape in these tests during implementation, not this plan's guess —
+then update `Challenge()` to match. This is the one place the plan
+defers to the vendored code on purpose; getting it wrong fails Task 7
+loudly.)
+
+- [ ] **Step 2: Verify failure. Step 3: implement** — `certMirror` holds
+ a `map[string][]byte` behind a `sync.RWMutex`, rebuilt by `Refresh`
+ from `s.List("control/certs/")` + gets (list-then-get; sizes are small
+ and renewals rare — no delta cleverness). A refresh error keeps the
+ last-good map (degrade-not-die, the edge's standing pattern). Start a
+ 30s refresh loop next to the edge's existing background loops.
+
+- [ ] **Step 4: Wire serving.** `GetCertificate` layering becomes:
+ wildcard → local autocert → mirror (parse PEM with
+ `tls.X509KeyPair`, cache the parsed `tls.Certificate` keyed by
+ mirror-map generation so parsing isn't per-handshake). HTTP-01: in the
+ handler where autocert's `HTTPHandler` is consulted (`edge.go` ~1222),
+ fall back to `mirror.Challenge(token)` when local autocert has no
+ answer. Non-issuer edges thereby answer any edge's challenges.
+
+- [ ] **Step 5: Package gate + commit**
+
+```bash
+go test ./internal/edge/ -count=1
+git add internal/edge/certsync.go internal/edge/certsync_test.go internal/edge/edge.go internal/edge/wildcard.go
+git commit -m "edge: cert mirror — every edge serves every cert and challenge"
+```
+
+---
+
+### Task 6: Door ACL guard — keys never leave through the fleet door
+
+**Files:**
+- Test: add to the store door's existing ACL test file (find with
+ `grep -rln "store/o\|door" internal/edge/*_test.go`)
+
+**Interfaces:** consumes the door's existing test harness; produces the
+permanent guard the spec promises.
+
+- [ ] **Step 1: Write the test** (goes straight to green if the ACL is
+ the allowlist the spec says it is — that's fine; it's a tripwire, and
+ the review gate checks it FAILS if the allowlist is loosened):
+
+```go
+func TestDoorNeverServesCertMaterial(t *testing.T) {
+ // Use the door test harness's authenticated fleet client.
+ for _, key := range []string{
+ "control/certs/example.com",
+ "control/certs/acme_account+key",
+ "control/edge-mesh-key",
+ "control/housekeeper.json",
+ } {
+ resp := doorGet(t, key) // harness's existing helper for authed reads
+ if resp.StatusCode != http.StatusForbidden && resp.StatusCode != http.StatusNotFound {
+ t.Fatalf("door served %s: %d — cert material must never transit the door", key, resp.StatusCode)
+ }
+ }
+}
+```
+
+Also assert the `dirs`/`list` verbs refuse `prefix=control/certs/`.
+Adapt helper names to the harness's actual ones; do not weaken the
+assertion to "not 200".
+
+- [ ] **Step 2: Run** `go test ./internal/edge/ -run DoorNever -count=1`
+ → PASS (or a real finding — surface it, don't paper it).
+
+- [ ] **Step 3: Commit**
+
+```bash
+git add internal/edge/
+git commit -m "edge: door ACL tripwire — cert material never transits the fleet door"
+```
+
+---
+
+### Task 7: Two-edge e2e — mint on one, serve on the other
+
+**Files:**
+- Create: `internal/edge/certsync_e2e_test.go`
+
+**Interfaces:** consumes everything above; pattern-match the existing
+in-process e2e harnesses (`tunnel_e2e_test.go`, `fleet_e2e_test.go`) for
+fake-ACME and dual-edge scaffolding.
+
+- [ ] **Step 1: Write the test:** one fake store; edge A with
+ `housekeeperHeld = () => true` and a fake ACME CA (the wildcard tests
+ already carry one — reuse it); edge B with `housekeeperHeld = () =>
+ false` and the same store. Drive: A mints `app.example.test` (fake CA
+ validates via HTTP-01 against **B's** handler — the challenge-anywhere
+ proof); then a TLS handshake against **B** with SNI `app.example.test`
+ succeeds using the mirrored cert after one `Refresh`. Assert B started
+ zero ACME orders (fake CA order counter), and that A's
+ `errNotIssuer`-gated twin refuses to mint when `held` is false.
+
+- [ ] **Step 2: Run** `go test ./internal/edge/ -run CertSyncE2E -count=1 -v`
+ → PASS, output pristine.
+
+- [ ] **Step 3: Full gate + commit**
+
+```bash
+go build ./... && go vet ./... && go test ./... -count=1
+git add internal/edge/certsync_e2e_test.go
+git commit -m "edge: e2e — issuer mints, every edge serves cert and challenge"
+```
+
+---
+
+### Task 8: Spec/rollout bookkeeping
+
+**Files:**
+- Modify: `docs/superpowers/specs/2026-08-20-multi-region-ingress-design.md`
+ (mark stage 1–2 rows as implemented-dormant; the 30s-poll amendment is
+ already in)
+- Modify: `docs/superpowers/plans/2026-08-20-multi-region-writer-map-and-certs.md`
+ (rollout notes section below filled with actuals)
+
+- [ ] **Step 1:** Record in the plan's rollout notes: (a) old fixed
+ snapshot keys still live in both buckets — delete after both
+ deployments run this build (one `aws s3 rm` pair per bucket, listed
+ verbatim); (b) the teamtito runbook PR (key-shape update §Step 0/3)
+ is owed at that same moment; (c) #158 closes on merge — comment
+ linking the audit doc; pinfra #15's blocker note gets updated.
+- [ ] **Step 2:** Full gate, commit `docs: stage 1-2 rollout notes`.
+
+---
+
+## Rollout notes (filled by Task 8 at execution)
+
+- Old-key deletion commands: (recorded at execution)
+- Runbook PR: (recorded at execution)
+- Issue comments: #158 close, pinfra #15 unblock, #207 interaction —
+ per-box keys change the generation-window arithmetic note.
diff --git a/docs/superpowers/specs/2026-08-20-multi-region-ingress-design.md b/docs/superpowers/specs/2026-08-20-multi-region-ingress-design.md
new file mode 100644
index 000000000..7de41a805
--- /dev/null
+++ b/docs/superpowers/specs/2026-08-20-multi-region-ingress-design.md
@@ -0,0 +1,273 @@
+# Multi-region ingress — TLS termination at every edge (#238)
+
+Status: designed with Paul in-session 2026-08-20; each section approved as
+presented. Successor to the HA ladder's rung-2 sketch
+(`2026-08-12-ha-posture-and-multi-upstream-routes-design.md` Part I) — this
+is the "distribution, not just recovery" half. Closes the design half of
+issue #238; the #158 fix ships inside stage 1.
+
+## Decisions (Paul, 2026-08-20)
+
+- **Ordering**: multi-region ingress designs and ships BEFORE UK/SE tito
+ fleet members. Placing instances regionally while every client routes
+ through Dublin delivers capacity, not latency; the tito capacity crisis
+ was solved separately (t4g.xlarge resize, same day).
+- **Traffic model: any edge, any app.** Every edge terminates TLS for
+ every route. Per-app regional homing alone was rejected; "homing now,
+ any-edge later" was rejected in favour of doing the invariants once.
+- **Certs: one elected issuer, distributed via the control bucket.**
+ Every-edge-mints and hybrid were rejected (shared ACME account budget,
+ order-state collisions, two validation paths forever).
+- **Backhaul: edge-to-edge over public 443.** Tunnel-hub transport and
+ VPC peering rejected (hub hops for publicly-addressable boxes; cloud
+ coupling that breaks customer-boxes-anywhere).
+- **First target: flagship** (au/uk/us boxes exist, idle since 2026-08-11;
+ a steering or cert mistake burns nobody's ticket sales). Tito UK/SE is
+ the second, boring deployment of a proven pattern.
+- **Steering is per-route opt-in; serving capability is automatic.**
+ Default = today's single-region DNS. `steering` is a string field
+ (`"latency"` in v1), never a bool.
+- **Bucket topology: one control bucket per deployment, forever; data
+ buckets are regional.** CAS/leases/serial demand a single linearizable
+ home; instance data lives in its region's data bucket.
+- **Anticipated, not built** (shapes committed now, features later):
+ read-local/write-home routing; per-instance residence choice with
+ data-following moves.
+
+## Vocabulary
+
+- **Edge / platform box**: a box converging the deployment bucket and
+ terminating TLS. After this spec, a deployment has N of them (the
+ original box plus armed region hosts). Distinct from fleet members,
+ which reach the bucket only through the store door and never terminate
+ deployment TLS.
+- **Home**: the box where a home-boxed route's process lives. Exactly one
+ per route, unchanged.
+- **Steered**: a hostname whose DNS answers with latency records across
+ all armed edges.
+- **Housekeeper**: the single lease-elected box performing
+ deployment-global duties (cert issuance, releases prune).
+
+## Part 1 — the writer map
+
+Rule: **data that is per-box gets per-box keys; duties that are
+deployment-global get a lease.** The status publisher's CAS-lease is the
+one election mechanism, reused; no second election pattern may be
+introduced.
+
+| Prefix | Writer today | Ruling |
+|---|---|---|
+| `apps/**`, `domains/*`, `control/serial`, restart-gen | console | unchanged — console stays the single control-plane writer, homed on one box; remote edges backhaul console traffic to it |
+| `control/registry-snapshot*` (fixed keys) | any platform box — **#158** | **per-box keys**: `control/registry-snapshot/<box>.db` + `<box>.json`; generations `control/registry-snapshots/<box>/<stamp>.db`. Every box snapshots its own registry. Closes #158; retires the pinfra #15 blocker |
+| `status/**` | lease-elected publisher | unchanged; verify the lease is deployment-scoped (it is bucket-scoped, which is the same thing) |
+| `releases/**` retention prune | every box | moves behind the **housekeeper lease** — five concurrent pruners racing a promote is a bug class we refuse to have |
+| usage metering day objects | the (single) edge | **audit**: if day-object keys are not box-scoped, they become `…/<box>/…` in the same shape as snapshots |
+| `control/certs/**`, `control/acme-challenges/**` (new) | housekeeper | Part 2 |
+| `control/edges/<box>.json` (new) | each box, own key only | Part 3 |
+| `control/edge-mesh-key` (new) | housekeeper mints if absent (create-only CAS) | Part 3 |
+
+Stage-1 audit deliverable: an enumerated list of EVERY bucket writer in
+the codebase with its ruling from the table above, checked into the spec's
+plan; anything discovered outside this table gets a ruling before stage 2.
+
+**Snapshot key migration** is a storage-shape change; the consumer list is
+what PARSES it: the DR runbook (teamtito/carlos-infrastructure
+`RUNBOOK-edge-failover.md`) and nothing else — flagship has no DR runbook
+today, a gap this migration's PR records rather than hides. New keys and runbook updates land in the same PR; the old fixed
+keys are deleted only after both deployments run the new binary. While
+both key shapes exist the runbooks name the new keys only.
+
+**Bucket topology.** One control bucket per deployment, in the
+deployment's home region, forever: serial bumps, leases, create-only
+claims and record CAS are linearizable only within one bucket, and
+cross-region S3 replication is async — leases across replicas are broken
+by construction. The control plane is never on the request path (edges
+serve from local converged state), so remote boxes pay ~150ms on a 2s
+background poll, which is irrelevant. Accepted consequence: a control-
+bucket-region S3 outage freezes the control plane globally (no
+converges, no cert renewals) while every edge keeps serving
+last-converged state. Cross-region control-bucket replicas are an
+explicit non-goal of this spec.
+
+Regional **data** buckets already exist (flagship `carlos-data-{ie,au,uk,us}`)
+and keep their role: a box's litestream/instance-data replication and
+region-homed object stores write to the box's own region's data bucket.
+Tito's UK/SE boxes get `carlos-tito-{uk,se}` data buckets when they
+arrive. One deliberate cross-over: **registry snapshots go to the control
+bucket, not the regional data bucket** — the AU box's snapshot exists to
+rebuild the AU box after an AU-region failure and must not be stored in
+the blast radius it insures against. Certs never appear in data buckets.
+
+## Part 2 — certificates
+
+**Issuer.** The housekeeper runs today's issuance machinery unmodified:
+per-host ACME HTTP-01, DNS-01 wildcards, the delegation zone
+(`CARLOS_ACME_DELEGATION_ZONE`), and the HostPolicy rulings from PR #180
+(EXACT-only per-host orders; a catch-all never enables per-host issuance).
+One ACME account, one order stream — the shared-account blast-radius
+rulings hold by construction.
+
+**Distribution.** Minted and renewed certs publish under
+`control/certs/` (autocert cache-key naming: per-host PEM, the account
+key, and HTTP-01 tokens). Cert writes do NOT bump `control/serial` —
+the serial stays console-written per the writer map; edges refresh a
+local certs mirror on their own 30-second list-poll, and
+`GetCertificate` serves from that mirror. Renewals run ~30 days early,
+so a 30s propagation cadence is never on a deadline.
+
+**Keys in the bucket — the trust statement.** Any box with deployment-
+bucket read already serves this deployment's traffic and reads its
+releases; holding TLS keys does not extend what a compromised box can
+do. Hard guards: the fleet store door's ACL is an allowlist that never
+includes `control/certs/**` or `control/edge-mesh-key` (fleet members and
+their tokens can never read keys — this gets a permanent test), and
+certs never replicate to data buckets.
+
+**HTTP-01 under latency DNS** — the one new mechanism. LE's validators
+hit whichever edge is nearest to them, not the issuer. Before answering
+an order, the issuer writes each active token to
+`control/acme-challenges/<token>`; EVERY edge serves
+`/.well-known/acme-challenge/*` from its mirror of that prefix. Tokens
+are deleted after validation and carry a TTL stamp so a crashed order
+never leaves permanent litter. The issuer remains the only box that
+talks to the CA.
+
+**Issuer death**: the lease expires and the next box's housekeeper takes
+over with identical machinery. Distributed certs keep every edge serving
+through the gap; a dead issuer costs nothing until the earliest renewal
+window ~30 days out.
+
+## Part 3 — serving classes and backhaul
+
+| Class | Kinds | Behaviour on a non-home edge |
+|---|---|---|
+| Everywhere | static, bake-served | adopt and serve locally from converged artifacts — zero backhaul |
+| Everywhere, probed | addr routes with publicly-reachable upstreams (incl. #152 multi-upstream) | each edge runs its own prober and dials upstreams directly; per-edge live sets are per-edge by design. Loopback/link-local addrs classify as home-boxed (another box's 127.0.0.1 is not dialable) |
+| Home-boxed | exec, unit, socket-activated, tunnel-backed | backhaul to the home edge. The process exists on exactly one box — unchanged invariant |
+
+**Home resolution.** Each box maintains `control/edges/<box>.json`: its
+EIP and a map `host → {role}` for the routes it homes, written by its
+converge loop, serial-bumped on change. v1 writes only `role: "home"`.
+Every edge folds these files into a deployment-wide host→home map. The
+file doubles as the DR runbook's explicit statement of what a dead box
+was serving.
+
+**The hop.** A remote edge dials `home-EIP:443` with ordinary SNI — the
+home edge presents the same bucket-distributed cert. Two headers make it
+a trusted hop: an HMAC over (host, timestamp) keyed by
+`control/edge-mesh-key`, and `X-Carlos-Hop: 1`. A receiving edge with a
+valid HMAC serves the route locally, honours the appended
+`X-Forwarded-For`, and hard-refuses to forward again — **one hop, ever**.
+If its map says the host homes elsewhere (stale map mid-move), it
+answers 503 + `Retry-After: 2` and lets converge heal it; it never
+re-forwards. Invalid or absent HMAC → the request is an ordinary client:
+forwarded headers stripped, normal serving rules. Timestamp skew bound
+±30s (boxes run NTP). Key rotation: dual-accept window, old + new both
+verify until every edge converges the new key.
+
+**Hibernation** composes unchanged: the home edge owns wake (wake-once
+singleflight, #199), and a backhaul hop is just another first client.
+
+**Steering opt-in.** New additive record field `steering` (string;
+`"latency"` is the only v1 value; absent = today's single-region DNS).
+Member-settable via console and CLI like any record field;
+fingerprint-covered; rides the #137 record shape. Serving capability
+exists on every edge regardless — steering only controls whether DNS
+sends anyone there.
+
+Consequences stated honestly:
+- an opted-in addr route is probed by N edges — N× probe cadence on the
+ member's health endpoint. Trivial at our N; documented for members
+ (Eleven).
+- **#170's blast-radius statement updates**: member-stated upstreams are
+ dialled from every edge, not one. The trust ruling that gates the
+ member addrs door gates steering of member addr routes identically.
+
+## Part 4 — steering mechanics, failure, rollout, testing
+
+**DNS.** Route53 latency records (both production zones are already
+Route53). A steered apps-domain hostname gets one latency record per
+armed edge, each bound to a Route53 health check on that edge's EIP
+(HTTPS + SNI), TTL 60. Console-driven through the existing zone
+machinery — never tofu-per-app. Custom domains: CNAME to a steering
+hostname (shape: `edge.<apps-domain>`, itself latency-routed) gets full
+steering; apexes cannot CNAME, so apex guidance remains
+A-records-to-all-EIPs — round-robin without latency bias, documented as
+such in attach guidance. Edge death: health checks withdraw its records
+in ~90s globally — steered routes thereby get most of rung-2 warm
+standby for free.
+
+**Failure modes.**
+- Edge dies: steered → fails away via health checks; unsteered routes
+ homed there are down until rung-1 recovery (unchanged today).
+- Issuer dies: Part 2 — costs nothing for ~30 days.
+- Control-bucket region outage: global control freeze, edges serve
+ last-converged state. Accepted.
+- Stale home map during a move: one 503 + Retry-After, healed in
+ seconds; the hop guard makes loops structurally impossible.
+- Mesh-key rotation: dual-accept window.
+
+**Rollout.** Stages 1–4 land dormant on single-box deployments
+(one-candidate elections; per-box keys with one box; no steered routes):
+1. Writer map: per-box snapshot keys (+ runbooks), prune behind
+ housekeeper lease, usage-key audit, full bucket-writer enumeration.
+2. Issuer election + cert distribution + challenge-anywhere.
+3. Backhaul: edges files, mesh key, hop serving + guard.
+4. Steering: record field, console/CLI setter, Route53 latency records +
+ health checks (tofu for the health-check scaffolding, console for
+ per-host records).
+5. Arm flagship AU alone; steer one static site and one exec app;
+ verify from real vantage points; then UK/US.
+6. Tito UK/SE: new region hosts + `carlos-tito-{uk,se}` data buckets in
+ teamtito/carlos-infrastructure; second deployment of a proven
+ pattern. (This is where the original UK/SE member request lands —
+ as edges, not door-joined members.)
+
+**Testing.** In-process two-edge harness (pattern: `fleet_e2e`,
+`tunnel_e2e`): fake bucket + fake ACME prove cert distribution,
+challenge-served-anywhere, HMAC + hop guard (including the
+stale-map-503 and never-double-forward cases), home-map convergence, and
+the door-ACL-never-reads-certs guard. Steering verified live per edge
+with `curl --resolve` and geo vantage checks; the Route53 record shape
+gets a tofu-plan golden. Every stage keeps the full `go build/vet/test`
+gate green with stages dormant.
+
+## Anticipated, not built
+
+Committed shapes so neither feature requires unwinding this spec:
+
+1. **Read-local / write-home** (per-app opt-in, never inferred): the
+ home map's `{role}` object grows a `read` role; edges route safe
+ methods (GET/HEAD/OPTIONS) to a near `read` box and mutating methods
+ to `home`. Deferred with it: replica materialization off
+ `replication`/`backing`, read-your-writes (sticky-to-primary window
+ after a mutation), and the honesty problem that GET-is-read-only is
+ an app contract. v1 routes ALL methods to `home`; the role field and
+ the string-typed `steering` field are the anticipation.
+2. **Per-instance residence** (titogo customers choosing regions): an
+ instance IS a route, and every mechanism here is route-scoped —
+ stated as invariant: *residence is a property of a route; an app is
+ just the default its instances inherit* (instance beats app,
+ precedence fixed now). The registry/record `region` field is the
+ residence carrier. Deferred to its own spec: the member-facing
+ residence door, data-following moves (litestream-assisted replicate-
+ then-cutover — the opposite of fleet-placement's D6
+ state-does-not-follow ruling, deliberately), and region-aware unit
+ economics.
+
+## Non-goals
+
+- Anycast. DNS latency records are the steering mechanism; revisit only
+ if measured DNS behaviour fails real users.
+- Cross-region control-bucket replicas.
+- Multi-writer console.
+- Method-aware routing and residence moves (anticipated above).
+- Latency-aware upstream selection within an addr route's `Addrs` set
+ (round-robin stands; tags/filters are the fleets-v2.1 lane).
+
+## Issue map
+
+Closes the design half of #238. Stage 1 closes #158 and unblocks pinfra
+#15. Updates #170's blast-radius statement. Touches #207's runbook text
+(generation-window note survives per-box keys). The UK/SE tito request
+resolves as stage 6 edges, not fleet members.