carlos / platform Public

plan: multi-region plan 2 of 3 — backhaul (spec Part 3)

Seven tasks, dormant single-box: per-box edges files (spec amended —
NOT serial-bumped, 30s poll, same correction pattern as certs), the
housekeeper-minted mesh key with dual-accept rotation reads, the home
map mirror, one-hop acceptance with the structural no-reforward guard,
backhaul dialing over real TLS, the serving-class adoption rules, and
a two-edge e2e. Steering (Part 4) split into plan 3 per the scope
check — DNS is untouched here.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Paul Campbell pushed by paul@keymail.dev cea1911fca3ff1ffb8461f14d1c3f642b684d05c parent 66dbc05
1 file changed, +452 −0
  • docs/superpowers/plans/2026-08-22-multi-region-backhaul.md +452 −0
diff --git a/docs/superpowers/plans/2026-08-22-multi-region-backhaul.md b/docs/superpowers/plans/2026-08-22-multi-region-backhaul.md
new file mode 100644
index 000000000..f99fb432d
--- /dev/null
+++ b/docs/superpowers/plans/2026-08-22-multi-region-backhaul.md
@@ -0,0 +1,452 @@
+# Multi-region ingress, plan 2 of 3: backhaul (spec Part 3)
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use
+> superpowers:subagent-driven-development (recommended) or
+> superpowers:executing-plans to implement this plan task-by-task. Steps
+> use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Ship spec Part 3 — home resolution, the mesh key, edge-to-edge
+backhaul with a structural one-hop guard, and the serving-class rules —
+dormant on single-box deployments.
+
+**Architecture:** Each box publishes `control/edges/<box>.json` (its EIP
+and homed hosts); every edge folds these into a host→home map on a 30s
+poll (spec amended: NOT serial-bumped — boxes never write
+`control/serial`). A remote edge answers a hello for a host homed
+elsewhere by dialing the home EIP:443 with ordinary SNI plus two trust
+headers (mesh-key HMAC + `X-Carlos-Hop: 1`); the receiving edge serves
+locally and never forwards again. The housekeeper mints the mesh key
+create-only. Static/bake routes become adopt-everywhere; public-addr
+routes probe-everywhere; exec/unit/socket/tunnel stay home-boxed and
+backhaul. Steering (Part 4) is plan 3 — nothing here touches DNS.
+
+**Tech Stack:** Go; `release.Store` (create-only Put with empty ifMatch =
+the CAS); stage 1-2's mirror/poll patterns in `internal/edge/certsync.go`
+as the house idiom for bucket-fed edge state.
+
+## Global Constraints
+
+- Spec: `docs/superpowers/specs/2026-08-20-multi-region-ingress-design.md`
+ Part 3. Key shapes verbatim: `control/edges/<box>.json`,
+ `control/edge-mesh-key`. Hop headers verbatim: `X-Carlos-Hop: 1` and
+ `X-Carlos-Hop-Auth: <hex hmac>`.
+- **Nothing writes `control/serial`.** Task 1 amends the spec's Part 3
+ line that said the edges file is serial-bumped (same correction the
+ certs took in stage 1-2): edges files ride their own 30s poll.
+- **Wire-shape changes are additive.** The edges-file schema starts
+ versioned (`"v":1`) and its `hosts` values are objects
+ (`{"role":"home"}`) per the spec's read-local anticipation — the role
+ field ships now, only `"home"` is ever written.
+- **One hop, ever.** A received hop is NEVER re-forwarded; stale-map
+ refusal is `503` + `Retry-After: 2`.
+- **The door ACL never serves `control/edge-mesh-key`** — already
+ tripwired in `internal/edge/storedoor_test.go` (Task 6 of plan 1);
+ Task 3 must not weaken that test.
+- Mesh-key HMAC: SHA-256, over `host + "\n" + unix-timestamp`, ±30s skew
+ bound, hex-encoded. Rotation is dual-accept and the READ side ships in
+ v1 (spec Part 3): the mirror reads both `control/edge-mesh-key` and
+ `control/edge-mesh-key.next`; `verifyHop` accepts either; `signHop`
+ always signs with `.next` when present. Promotion (operator copies
+ `.next` over the key and deletes `.next`) is a documented runbook
+ step, not code — but a fleet mid-rotation must interoperate, which the
+ dual-accept reader guarantees. Task 2's tests cover the both-keys
+ case.
+- Gate before every commit: `go build ./... && go vet ./... &&
+ go test ./... -count=1`, `GOOS=darwin go build ./...`, `gofmt -l` clean
+ on touched files, `-race` on `internal/edge` and `internal/host`, env
+ `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off`.
+- Production channel names in `internal/release/laddercheck_test.go`
+ untouched. Tests shelling out to the carlos binary use `carlosTestEnv`
+ + scratch `cmd.Dir`.
+- Dormancy: with one box, the home map contains only that box, every
+ route is local, no hop is ever dialed, and the mesh key mints once and
+ sits unused. Every task states its single-box inertness.
+
+## File Structure
+
+- `internal/host/edgesfile.go` + `_test.go` — the per-box publisher (T1)
+- `internal/host/housekeeper.go` — mesh-key mint duty (T2, additive)
+- `internal/edge/mesh.go` + `_test.go` — key mirror, HMAC sign/verify,
+ hop-header vocabulary (T2/T3)
+- `internal/edge/homemap.go` + `_test.go` — edges-prefix mirror + lookup
+ (T3)
+- `internal/edge/edge.go` — hop acceptance in the request path (T3),
+ backhaul dial in the miss path (T4)
+- `internal/provision/reconcile.go` — everywhere-class adoption (T5)
+- `internal/edge/backhaul_e2e_test.go` — two-edge proof (T6)
+
+---
+
+### Task 1: The edges file — each box publishes what it homes
+
+**Files:**
+- Create: `internal/host/edgesfile.go`
+- Test: `internal/host/edgesfile_test.go`
+- Modify: `internal/host/agent.go` (arm the loop where
+ `registrySnapshotLoop` is armed — find it with
+ `grep -n registrySnapshotLoop internal/host/agent.go`)
+- Modify: `docs/superpowers/specs/2026-08-20-multi-region-ingress-design.md`
+ (the Part 3 "serial-bumped on change" sentence → 30s-poll amendment,
+ dated, same style as the certs amendment)
+
+**Interfaces:**
+- Consumes: `readPublicIP()` (`internal/host/agent.go:1421-1426`, IMDSv2
+ best-effort — reuse, don't duplicate; empty string means "no public
+ IP", publish the file anyway with `"eip":""` so the DR runbook still
+ gets the hosts list); the box label the snapshot pass uses (same
+ derivation — find `registrySnapshotLoop`'s label source and share it);
+ `reg.List()` for the routes this box serves.
+- Produces: key `control/edges/<box>.json`, schema (exact):
+
+```json
+{"v":1,"box":"annie-welding","eip":"99.81.104.219","generation":"2026-08-22T09:00:00Z","hosts":{"app.example.com":{"role":"home"}}}
+```
+
+ and `edgesFileKey(box string) string` returning
+ `"control/edges/" + box + ".json"` — Task 3's mirror parses this
+ schema; the `role` object form is load-bearing (read-local
+ anticipation).
+
+- [ ] **Step 1: Failing tests.** In `edgesfile_test.go` (reuse the
+ package's existing fake store + scratch registry helpers — the
+ snapshot tests show the idiom):
+
+```go
+func TestEdgesFilePublishesHomedHosts(t *testing.T) {
+ s, reg := edgesTestStore(t), edgesTestRegistry(t) // adapt to real helpers
+ seedRoute(t, reg, "app.example.com") // any served kind
+ var lastBody []byte
+ if err := edgesFilePass(context.Background(), s, reg, "annie", "1.2.3.4", &lastBody); err != nil {
+ t.Fatal(err)
+ }
+ got, err := s.Get(context.Background(), "control/edges/annie.json")
+ if err != nil {
+ t.Fatal(err)
+ }
+ var f edgesFile
+ if err := json.Unmarshal(got, &f); err != nil {
+ t.Fatal(err)
+ }
+ if f.V != 1 || f.Box != "annie" || f.EIP != "1.2.3.4" {
+ t.Fatalf("file = %+v", f)
+ }
+ if r, ok := f.Hosts["app.example.com"]; !ok || r.Role != "home" {
+ t.Fatalf("hosts = %+v", f.Hosts)
+ }
+}
+
+func TestEdgesFileSkipsWriteWhenUnchanged(t *testing.T) {
+ // Second pass with identical registry state writes nothing:
+ // compare the store's put-count across two passes (fake store
+ // exposes writes; adapt). The change-gate is the marshaled body,
+ // EXCLUDING generation (a timestamp-only refresh is not a change).
+}
+```
+
+- [ ] **Step 2: Verify failure** —
+ `go test ./internal/host/ -run EdgesFile -count=1` → undefined.
+
+- [ ] **Step 3: Implement.** `edgesFile` struct mirroring the schema;
+ `edgesFilePass(ctx, s, reg, box, eip string, last *[]byte) error`
+ building hosts from `reg.List()` (every route row this box serves —
+ ALL kinds; the serving-class split is the READER's business, Task 3),
+ marshaling with `generation` = `time.Now().UTC().Format(time.RFC3339)`,
+ change-gated on the body-minus-generation (marshal a copy with
+ `Generation: ""` for comparison), written with the package's
+ `putReplacing`. Loop `edgesFileLoop(reg)` at the snapshot loop's
+ cadence and arming site (5min + jitter is fine — home moves are
+ placement events, not per-request events; the hop's stale-map 503 +
+ converge heals faster paths). Single-box inertness: the file exists
+ and nobody reads it until Task 3 ships.
+- [ ] **Step 4:** `go test ./internal/host/ -count=1` → PASS.
+- [ ] **Step 5: Spec amendment** — replace the Part 3 sentence claiming
+ the edges file is "serial-bumped on change" with: written by each box's
+ own loop, change-gated; edges MIRROR the prefix on a 30-second poll
+ (`control/serial` stays console-written; dated amendment note).
+- [ ] **Step 6: Commit**
+
+```bash
+git add internal/host/edgesfile.go internal/host/edgesfile_test.go internal/host/agent.go docs/superpowers/specs/2026-08-20-multi-region-ingress-design.md
+git commit -m "host: per-box edges file — what this box homes, for the fleet's home map"
+```
+
+---
+
+### Task 2: The mesh key — housekeeper mints, edges mirror, HMAC vocabulary
+
+**Files:**
+- Create: `internal/edge/mesh.go`
+- Test: `internal/edge/mesh_test.go`
+- Modify: `internal/host/agent.go` (the `runHousekeeperDuties` call at
+ ~:1632 gains the mint duty beside `prune`)
+- Modify: `internal/host/housekeeper_test.go` (duty-count expectations,
+ if any assert exact duty sets)
+
+**Interfaces:**
+- Consumes: `runHousekeeperDuties(gate, duties...)` shape from
+ `internal/host/housekeeper.go` (read it first — add the mint as a
+ second duty callback, same quiet-when-not-held contract);
+ `release.Store.Put` with empty `ifMatch` (create-only — this IS the
+ CAS; a lost race is success-by-other).
+- Produces (in `internal/edge/mesh.go`, exported within the package):
+ - `const meshKeyKey = "control/edge-mesh-key"`
+ - `mintMeshKey(ctx, s) error` — 32 random bytes hex-encoded,
+ create-only Put; `os.ErrExist`-shaped refusal = already minted =
+ success (log at debug, not error). Called from host via a tiny
+ exported wrapper (`edge.MintMeshKey`) so the duty lives beside the
+ housekeeper.
+ - `signHop(key []byte, host string, now time.Time) string` — hex HMAC
+ SHA-256 over `host + "\n" + strconv.FormatInt(now.Unix(), 10)`,
+ returned as `"<unix>:<hex>"` so the verifier gets the timestamp.
+ - `verifyHop(key []byte, host, header string, now time.Time) bool` —
+ parse, ±30s skew bound, `hmac.Equal`.
+- Header names (Task 3/4 use verbatim): `X-Carlos-Hop: 1`,
+ `X-Carlos-Hop-Auth: <unix>:<hex>`.
+
+- [ ] **Step 1: Failing tests:**
+
+```go
+func TestMintMeshKeyIsCreateOnly(t *testing.T) {
+ s := meshTestStore(t)
+ if err := mintMeshKey(context.Background(), s); err != nil {
+ t.Fatal(err)
+ }
+ first, _ := s.Get(context.Background(), meshKeyKey)
+ if err := mintMeshKey(context.Background(), s); err != nil {
+ t.Fatalf("second mint must be success-by-other: %v", err)
+ }
+ second, _ := s.Get(context.Background(), meshKeyKey)
+ if !bytes.Equal(first, second) {
+ t.Fatal("second mint replaced the key")
+ }
+}
+
+func TestHopSignatureRoundTripsAndBoundsSkew(t *testing.T) {
+ key := []byte("0123456789abcdef0123456789abcdef")
+ now := time.Unix(1_766_000_000, 0)
+ h := signHop(key, "app.example.com", now)
+ if !verifyHop(key, "app.example.com", h, now.Add(29*time.Second)) {
+ t.Fatal("within skew must verify")
+ }
+ if verifyHop(key, "app.example.com", h, now.Add(31*time.Second)) {
+ t.Fatal("past skew must refuse")
+ }
+ if verifyHop(key, "other.example.com", h, now) {
+ t.Fatal("host is bound into the mac")
+ }
+ if verifyHop([]byte("wrong-key-wrong-key-wrong-key-00"), "app.example.com", h, now) {
+ t.Fatal("wrong key must refuse")
+ }
+}
+```
+
+- [ ] **Step 2: verify failure. Step 3: implement** per the interfaces
+ above; wire the host-side duty:
+
+```go
+runHousekeeperDuties(housekeeperGate, prune, mintMesh)
+```
+
+ where `mintMesh` closes over the deployment store and calls
+ `edge.MintMeshKey`; a mint error logs and does not fail the tick
+ (idempotent next tick). Single-box inertness: key mints once, nothing
+ reads it until Tasks 3-4.
+- [ ] **Step 4:** package gates both packages. **Step 5: Commit**
+ `edge+host: mesh key — housekeeper mints create-only, HMAC hop vocabulary`.
+
+---
+
+### Task 3: Hop acceptance + the home map mirror
+
+**Files:**
+- Create: `internal/edge/homemap.go`
+- Test: `internal/edge/homemap_test.go`, additions to `mesh_test.go`
+- Modify: `internal/edge/edge.go` (request path — where the edge has
+ resolved `lookupUpstream(host)` successfully, BEFORE proxying: the
+ hop-header handling; find the request entry with
+ `grep -n "func (e \*Edge) ServeHTTP\|lookupUpstream(host)" internal/edge/edge.go`)
+
+**Interfaces:**
+- Consumes: Task 1's `edgesFile` schema + `control/edges/` prefix; Task
+ 2's `verifyHop` + mesh-key mirror (a `meshKey()` accessor on the same
+ 30s poll — fold the key read into the home-map mirror's refresh, one
+ loop for both).
+- Produces:
+ - `homeMap` mirror: `Refresh(ctx) error` (list `control/edges/`, get
+ each, last-good on partial failure — copy `certMirror`'s
+ carry-forward idiom from `certsync.go`), `Home(host string) (eip
+ string, box string, ok bool)` — hosts map folded across files;
+ OWN-BOX entries excluded from backhaul candidacy (a host homed here
+ is served here).
+ - Hop acceptance in the request path (exact behavior, Task 6 proves
+ it): if `X-Carlos-Hop` present AND `verifyHop` passes → serve the
+ route locally; append client IP to `X-Forwarded-For` (trust the
+ inbound XFF only on a verified hop); if the host is NOT locally
+ served (stale map on the sender) → `503` + `Retry-After: 2`, NEVER
+ a re-forward. If the header is present but verification fails →
+ strip `X-Forwarded-*`, treat as an ordinary client.
+- Single-box inertness: the home map holds only this box → `Home()`
+ never offers a remote target; hop headers never arrive (nothing sends
+ them).
+
+- [ ] **Step 1: Failing tests** — homemap fold/exclude-own/last-good
+ (mirror `certsync_test.go`'s fake-store patterns):
+
+```go
+func TestHomeMapFoldsAndExcludesOwnBox(t *testing.T) {
+ s := meshTestStore(t)
+ putJSON(t, s, "control/edges/annie.json", `{"v":1,"box":"annie","eip":"1.1.1.1","hosts":{"a.test":{"role":"home"}}}`)
+ putJSON(t, s, "control/edges/betty.json", `{"v":1,"box":"betty","eip":"2.2.2.2","hosts":{"b.test":{"role":"home"}}}`)
+ m := newHomeMap(s, "annie")
+ if err := m.Refresh(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ if eip, box, ok := m.Home("b.test"); !ok || eip != "2.2.2.2" || box != "betty" {
+ t.Fatalf("b.test home = %s/%s/%v", eip, box, ok)
+ }
+ if _, _, ok := m.Home("a.test"); ok {
+ t.Fatal("own-box hosts must not offer a backhaul target")
+ }
+}
+```
+
+ plus hop-acceptance handler tests: verified hop on a served host →
+ 200 + XFF honored; verified hop on an unserved host → 503 with
+ `Retry-After: 2` and NO outbound dial (fake dialer asserts zero
+ calls); unverified hop header → served as ordinary client with
+ forwarded headers stripped.
+- [ ] **Step 2: verify failure. Step 3: implement**; the mirror loop
+ runs beside the cert mirror's (same 30s cadence, same
+ start-after-boot-refresh ordering, same stop-channel shape — fix the
+ discarded-literal lifecycle the cert mirror still has ONLY if you
+ touch that line anyway; otherwise leave it, it's ledgered).
+- [ ] **Step 4:** `go test ./internal/edge/ -count=1 -race`.
+- [ ] **Step 5: Commit** `edge: home map mirror + one-hop acceptance`.
+
+---
+
+### Task 4: Backhaul dialing — the miss path reaches the home edge
+
+**Files:**
+- Modify: `internal/edge/edge.go` (the point where a hello/request for a
+ KNOWN-remote host currently dead-ends — after `lookupUpstream` misses
+ locally; and the proxy construction for the hop)
+- Test: `internal/edge/homemap_test.go` additions
+
+**Interfaces:**
+- Consumes: `homeMap.Home(host)`, `signHop`, the mesh key accessor;
+ Go's `httputil.ReverseProxy` per the file's existing proxy idioms
+ (`multiProxy`/`tcpProxy` show the house style — read them first).
+- Produces: `backhaulProxy(host, eip string)` — an HTTP proxy dialing
+ `eip:443` with TLS `ServerName: host` and standard verification
+ (the home edge presents a real cert — the mirror distributed it),
+ setting `X-Carlos-Hop: 1` + `X-Carlos-Hop-Auth: signHop(...)`,
+ appending XFF. Timeouts: dial 5s, response-header 30s, no whole-body
+ deadline (streaming; matches the fleetapi client ruling). WebSockets:
+ the existing proxy layer's upgrade handling applies unchanged —
+ verify, don't reinvent.
+- Routing precedence (exact): local `lookupUpstream` hit → serve
+ locally (unchanged, always). Miss + `Home(host)` hit → backhaul. Miss
+ + no home → today's 404/refusal path unchanged. A request that
+ ARRIVED as a verified hop never consults `Home` (Task 3's guard).
+- Single-box inertness: `Home` never offers a target → the new branch
+ is dead code until a second box publishes.
+
+- [ ] **Step 1: failing tests** — precedence table driven: local-hit
+ ignores home map (fake home map poisoned to prove it); miss+home
+ dials the fake home edge (httptest TLS server asserting SNI, hop
+ headers present and verifying, XFF appended); miss+no-home = 404
+ path unchanged; hop-arrived request never dials (zero-call fake).
+- [ ] **Step 2: verify failure. Step 3: implement. Step 4:**
+ `-race` package run. **Step 5: Commit**
+ `edge: backhaul — remote-homed hosts dial the home edge, one hop, real TLS`.
+
+---
+
+### Task 5: Serving classes — static/bake adopt everywhere
+
+**Files:**
+- Modify: `internal/provision/reconcile.go` (the box-side gate deciding
+ which records this box materializes — find it via `routeFor` /
+ the tenancy gate; the change: records whose kind is `static` or
+ bake-served are adopted by EVERY platform box, not only their home)
+- Test: `internal/provision/reconcile_test.go`
+
+**Interfaces:**
+- Consumes: the record kind vocabulary already in
+ `internal/provision/provision.go` (read the Kind constants — do NOT
+ invent names); the reconciler's existing region/pool/home gates.
+- Produces: the class rule, stated in one place with the spec's table
+ cited: static/bake = everywhere; addr with public IPs = everywhere
+ (this is ALREADY true if addr records pass the tenancy gate on all
+ boxes — VERIFY and add the pinning test rather than assuming;
+ loopback/link-local addrs are home-boxed — the #177 validAddr rules
+ already refuse link-local in Addrs, and loopback single-Addr rows are
+ legacy: they stay wherever they are, gate unchanged);
+ exec/unit/socket/tunnel = home-boxed (unchanged — the pinning test
+ proves the gate still refuses them off-home).
+- Single-box inertness: "every box" = the one box; adoption outcomes
+ identical.
+
+- [ ] **Step 1: failing tests** — a static record with a fleet_box/home
+ naming ANOTHER box still materializes here; an exec record homed
+ elsewhere still does NOT; an addr record with public upstreams
+ materializes here regardless of home (or, if the current gate already
+ admits it, the test pins that fact).
+- [ ] **Step 2: verify failure (or document already-green as a pin).
+ Step 3: implement the gate change with the class table as the comment.
+ Step 4:** full `go test ./internal/provision/ ./internal/host/ -count=1`
+ (host consumes reconcile). **Step 5: Commit**
+ `provision: serving classes — static and bake adopt on every edge`.
+
+---
+
+### Task 6: Two-edge backhaul e2e
+
+**Files:**
+- Create: `internal/edge/backhaul_e2e_test.go`
+
+**Interfaces:** consumes everything above; harness pattern =
+`certsync_e2e_test.go` (in-process edges, one fake store).
+
+- [ ] **Step 1: the test.** One fake store. Edge A homes
+ `app.backhaul.test` (a local upstream serving a marker body); both
+ boxes publish edges files (write them directly into the store per Task
+ 1's schema — the host loop isn't in-process here); mesh key minted;
+ both edges' mirrors refreshed. Drive a request for `app.backhaul.test`
+ into edge B:
+ - B backhauls to A (assert the marker body served through B, XFF
+ carries the client, exactly ONE dial on A's counter).
+ - Poison the map (point the host at a third, dead EIP) → B answers
+ 502/504-class from the dial failure path, never loops.
+ - Stale-map case: B's map says A homes a host A no longer serves →
+ A answers 503 + `Retry-After: 2`, and B does NOT re-dial (one hop).
+ - Tampered hop: flip a byte in the auth header → A treats it as an
+ ordinary client (serves its 404 path for an unserved host, no hop
+ semantics), forwarded headers stripped.
+ - Single-box regression pin: an edge with only its OWN edges file
+ serves local routes exactly as before and never dials (zero-call
+ fake dialer).
+- [ ] **Step 2:** `-race -count=3` on the new test; full gate.
+- [ ] **Step 3: Commit** `edge: e2e — one-hop backhaul across two edges`.
+
+---
+
+### Task 7: Rollout bookkeeping
+
+**Files:**
+- Modify: the spec (Part 3 rows → implemented-dormant with commits);
+ this plan (rollout notes below).
+
+- [ ] Fill the rollout notes: stage-5 arming now needs — per-box edges
+ files verified present for every armed box BEFORE steering (plan 3)
+ ever points DNS at a second edge; mesh-key presence check; the
+ backhaul path exercised on the dev/Kass deployment before flagship AU
+ (first two-real-boxes deployment); #244's bare-edge note extends to
+ the mesh key (a bare edge with a bucket can read it — same trust
+ envelope as certs, record it).
+- [ ] Full gate; commit `docs: backhaul rollout notes`.
+
+## Rollout notes (filled by Task 7 at execution)
+
+- (filled at execution)