amadan

carlos / platform Public

Merge pull request #252 from carlosframework/multi-region-backhaul

Multi-region backhaul: home maps, mesh key, one-hop dialing, serving classes (#238 Part 3)
Paul Campbell pushed by paul@keymail.dev 920d599480e2bdea157e6fc2bef4149f75ec011d parent eab8c78parent 3aea65c
16 files changed, +5010 −36
  • docs/superpowers/plans/2026-08-22-multi-region-backhaul.md +377 −11
  • docs/superpowers/specs/2026-08-20-multi-region-ingress-design.md +65 −7
  • internal/edge/backhaul_e2e_test.go +870 −0
  • internal/edge/edge.go +346 −9
  • internal/edge/homemap.go +602 −0
  • internal/edge/homemap_test.go +1041 −0
  • internal/edge/mesh.go +303 −0
  • internal/edge/mesh_test.go +389 −0
  • internal/edge/storedoor_test.go +12 −5
  • internal/host/agent.go +16 −1
  • internal/host/edgesfile.go +153 −0
  • internal/host/edgesfile_test.go +270 −0
  • internal/host/housekeeper.go +48 −0
  • internal/host/housekeeper_test.go +49 −0
  • internal/provision/reconcile.go +183 −3
  • internal/provision/reconcile_test.go +286 −0
diff --git a/docs/superpowers/plans/2026-08-22-multi-region-backhaul.md b/docs/superpowers/plans/2026-08-22-multi-region-backhaul.md
index f99fb432d..79e338ea9 100644
--- a/docs/superpowers/plans/2026-08-22-multi-region-backhaul.md
+++ b/docs/superpowers/plans/2026-08-22-multi-region-backhaul.md
@@ -29,7 +29,11 @@ as the house idiom for bucket-fed edge state.
- Spec: `docs/superpowers/specs/2026-08-20-multi-region-ingress-design.md`
Part 3. Key shapes verbatim: `control/edges/<box>.json`,
`control/edge-mesh-key`. Hop headers verbatim: `X-Carlos-Hop: 1` and
- `X-Carlos-Hop-Auth: <hex hmac>`.
+ `X-Carlos-Hop-Auth: <unix>:<hex>` (**amendment, 2026-08-22, T7**: this
+ line originally read `<hex hmac>`, omitting the leading unix timestamp
+ `signHop` actually prepends — File Structure's own Task 2 Interfaces
+ section below had the right shape all along; ledgered as a Task 2 minor
+ finding, fixed here rather than left to drift from the code).
- **Nothing writes `control/serial`.** Task 1 amends the spec's Part 3
line that said the edges file is serial-bumped (same correction the
certs took in stage 1-2): edges files ride their own 30s poll.
@@ -177,8 +181,18 @@ git commit -m "host: per-box edges file — what this box homes, for the fleet's
**Files:**
- Create: `internal/edge/mesh.go`
- Test: `internal/edge/mesh_test.go`
-- Modify: `internal/host/agent.go` (the `runHousekeeperDuties` call at
- ~:1632 gains the mint duty beside `prune`)
+- Modify: `internal/host/housekeeper.go` (a standalone `mintMeshKeyLoop`,
+ armed beside `edgesFileLoop` in `RunAgent` — **Amendment (2026-08-22,
+ review):** the line above originally read "the `runHousekeeperDuties`
+ call at ~:1632 gains the mint duty beside `prune`", i.e. folded into
+ `retentionLoop`'s tick; that is wrong and this File Structure section's
+ own `internal/host/housekeeper.go` entry ("mesh-key mint duty, T2,
+ additive") was the plan's real intent all along. `retentionLoop` returns
+ early — never arming at all — on a box without `CARLOS_RELEASE_PUBKEY`,
+ and such a box still wins the housekeeper election, so a mint duty living
+ only inside it would silently never run fleet-wide. The mint needs its
+ own loop, gated on nothing but deployment-store presence, mirroring
+ `edgesFileLoop`'s shape.)
- Modify: `internal/host/housekeeper_test.go` (duty-count expectations,
if any assert exact duty sets)
@@ -241,15 +255,22 @@ func TestHopSignatureRoundTripsAndBoundsSkew(t *testing.T) {
```
- [ ] **Step 2: verify failure. Step 3: implement** per the interfaces
- above; wire the host-side duty:
+ above; wire the host-side duty as its own loop (**Amendment
+ (2026-08-22, review):** NOT folded into `retentionLoop`'s
+ `runHousekeeperDuties(housekeeperGate, prune, mintMesh)` call as this
+ step originally showed — see the Files section above for why):
```go
-runHousekeeperDuties(housekeeperGate, prune, mintMesh)
+runHousekeeperDuties(housekeeperGate, mint)
```
- where `mintMesh` closes over the deployment store and calls
- `edge.MintMeshKey`; a mint error logs and does not fail the tick
- (idempotent next tick). Single-box inertness: key mints once, nothing
+ called from `mintMeshKeyLoop`'s own `for { sleep; ... }`, armed
+ unconditionally on store presence (mirroring `edgesFileLoop`) rather
+ than riding `retentionLoop`'s `CARLOS_RELEASE_PUBKEY` gate.
+
+ `mint` closes over the deployment store and calls `edge.MintMeshKey`;
+ a mint error logs and does not fail the tick (idempotent next tick).
+ Single-box inertness: key mints once, nothing
reads it until Tasks 3-4.
- [ ] **Step 4:** package gates both packages. **Step 5: Commit**
`edge+host: mesh key — housekeeper mints create-only, HMAC hop vocabulary`.
@@ -438,15 +459,360 @@ func TestHomeMapFoldsAndExcludesOwnBox(t *testing.T) {
- Modify: the spec (Part 3 rows → implemented-dormant with commits);
this plan (rollout notes below).
-- [ ] Fill the rollout notes: stage-5 arming now needs — per-box edges
+- [x] Fill the rollout notes: stage-5 arming now needs — per-box edges
files verified present for every armed box BEFORE steering (plan 3)
ever points DNS at a second edge; mesh-key presence check; the
backhaul path exercised on the dev/Kass deployment before flagship AU
(first two-real-boxes deployment); #244's bare-edge note extends to
the mesh key (a bare edge with a bucket can read it — same trust
envelope as certs, record it).
-- [ ] Full gate; commit `docs: backhaul rollout notes`.
+- [x] Full gate; commit `docs: backhaul rollout notes`.
## Rollout notes (filled by Task 7 at execution)
-- (filled at execution)
+Synthesized from the SDD ledger (`progress.md`) across all six
+implementation tasks and their reviews. This plan ships stages 1–3 (the
+spec's Part 3 rows) dormant on single-box deployments; nothing below is
+armed by merging this branch — it is the checklist and the open-questions
+list for the rollout that follows, including the stage-5/6 arming this
+plan's own spec Part 4 describes.
+
+### Gates from the whole-branch final review (2026-08-22)
+
+The whole-branch final review (2026-08-22) marked two items
+MUST-FIX-BEFORE-STAGE-5. Recorded here as gates on the checklist below,
+not left to stand alone in the review record:
+
+a. **Seed the edges-file change-gate (`*last`) from the bucket at loop
+ arm.** `edgesFilePass`'s change-gate (`internal/host/edgesfile.go`,
+ Task 1) compares each pass's body against `*last`, but `*last` starts
+ nil at process start, in memory only. Two consequences follow from
+ that until it is seeded: a box that restarts during a transient
+ `readPublicIP()` miss (an IMDS blip) publishes `"eip":""` over its own
+ last-good file instead of falling back to it — the `eip == "" && *last
+ != nil` carry-forward this file already has only helps once `*last`
+ is populated, and right after a restart it never is — and separately,
+ every restart forces one gate-crossing write regardless of whether
+ anything about the box's routes actually changed, churning the
+ object's ETag on every box bounce. Seeding `*last` from the bucket's
+ current `control/edges/<box>.json` at loop arm (read-old-then-loop,
+ the same shape the loop already uses for its steady-state reads)
+ closes both. Land this as the FIRST commit of the stage-5 arming
+ branch, before anything else in the checklist below runs against a
+ real second box.
+b. **Visibility for the two stuck-state paths before steering sends
+ sustained traffic.** Both are silent today. `staleHopRefusal`
+ (`internal/edge/homemap.go:551`) answers a sender stuck dialing a home
+ box whose map has moved on with a correct 503, but nothing is LOGGED
+ on either end, so a sender looping into it during a stuck map
+ propagation produces zero signal. The backhaul 502 `ErrorHandler` is
+ unthrottled the other direction — one log line per failed dial, not a
+ throttled cadence (detail and cross-ref: item 14 below). Neither is a
+ correctness bug, but both need eyes before Part 4 (steering) can
+ legitimately point real, sustained traffic at this path — the
+ dormant/low-volume window this plan ships into is where to add the
+ logging, not after a stuck-move incident goes unseen in production.
+
+### Stage-5 arming checklist (before flagship AU, before any real second box)
+
+1. **Per-box edges files verified present for every armed box, BEFORE
+ plan 3 (steering) ever points DNS at a second edge.** The home map is
+ built by folding `control/edges/<box>.json` for every box the mirror
+ can list (`internal/edge/homemap.go`, Task 3); a box that never
+ published one is invisible to every other edge's map, so a route
+ homed there is neither served locally by strangers' hellos (fine) nor
+ backhaul-reachable (NOT fine, once steering can legitimately land
+ traffic on a box that has nothing to route it with). No CLI/console
+ surface exists for this today — same box-local-only shape as the
+ registry-snapshot check plan 1's rollout notes carry — so it is a
+ direct-bucket list, consistent with CLAUDE.md's operate-via-API-and-CLI
+ carve-out for platform-internal work:
+
+ ```bash
+ aws --profile <profile> s3 ls s3://<control-bucket>/control/edges/
+ ```
+
+ Expect exactly one `<box>.json` per box that is supposed to be armed.
+ A missing entry means that box's `edgesFileLoop` has not ticked yet
+ (5min + jitter cadence) or is not running — resolve before arming
+ steering, not after.
+2. **Mesh-key presence check.** `control/edge-mesh-key` must exist and
+ every armed box's mirror must have it loaded before any hop can verify
+ (`internal/edge/mesh.go`, Task 2). Same direct-bucket check:
+ `aws s3 ls s3://<control-bucket>/control/edge-mesh-key`. Absence means
+ no box has yet won the housekeeper election with `mintMeshKeyLoop`
+ armed — check the housekeeper's acquisition log line
+ (`internal/host/housekeeper.go`) on whichever box should hold it.
+ **Rotation runbook pointer:** there is no standalone rotation runbook
+ document yet — the only written procedure is this plan's own Task 2
+ Global Constraints section above (`control/edge-mesh-key.next`,
+ dual-accept read on both `verifyHop` callers, promotion = operator
+ copies `.next` over the key and deletes `.next`). File a real runbook
+ doc before the first live rotation rather than pointing operators at a
+ plan file; until then, this paragraph IS the pointer. **Propagation
+ window is ≤1 poll** (30s, the home-map/mesh-key mirror's shared
+ cadence) per Task 2's ledgered honest-header fix (`0c07490`) — during
+ that window some edges verify hops under the old key, some under
+ `.next`; dual-accept on both is what makes that safe, not a race to
+ avoid. **A corrupt key is operator-delete only.** `mintMeshKey` is
+ create-only (Task 2) — it never overwrites an existing
+ `control/edge-mesh-key`, so a key that gets corrupted in the bucket
+ (a bad manual edit, a truncated write) is never self-repaired by any
+ box's `mintMeshKeyLoop`. Recovery is manual: delete the object; the
+ next box to hold the housekeeper lease re-mints on its next tick.
+ Worth knowing before touching that key by hand during an arming
+ window, not just at a real rotation.
+3. **Exercise the backhaul path on dev/Kass before flagship AU.**
+ Flagship AU is the spec's designated first target because a steering
+ mistake there "burns nobody's ticket sales" (spec Part 1 decisions) —
+ but nothing in this branch has run across two REAL boxes; Task 6's
+ e2e (`67de75f`) is in-process with edges files written directly into a
+ fake store (see the spec's Part 3 residual note, and Task 6's report
+ §4.1). The dev/Kass deployment is the first two-real-boxes proving
+ ground: arm a second box there, verify items 1–2 above hold live (not
+ just in a fake store), drive one real cross-box hop with `curl
+ --resolve` against the far box's EIP for a host homed on the near box,
+ and confirm the marker/XFF/one-hop behaviour Task 6 pinned in-process
+ also holds over a real network hop — before repeating any of this on
+ flagship.
+4. **#244's bare-edge note extends to the mesh key.** Plan 1's rollout
+ notes (item 5) ruled that a standalone `carlos edge` run outside the
+ agent on a bucket-backed box is ungated and publishes to
+ `control/certs/**` unconditionally — "don't run a bare `carlos edge`
+ against a deployment bucket on a multi-edge deployment," tracked as
+ issue #244. The same box, run the same way, also mirrors
+ `control/edges/` and `control/edge-mesh-key` (Task 3's home-map mirror
+ has no issuer gate — it is read-only bucket state, not the issuance
+ path) — so a bare edge can read the mesh key and the whole
+ deployment's home map exactly as it can read certs. This is not a new
+ hole: it is the same trust statement Part 2 already makes explicit
+ ("any box with deployment-bucket read already serves this deployment's
+ traffic... holding TLS keys does not extend what a compromised box can
+ do") extended to the one new key type Part 3 adds. Recorded here so
+ #244's operational rule ("don't run a bare `carlos edge` against a
+ deployment bucket on a multi-edge deployment") is read as covering the
+ mesh key too, not just certs.
+
+### Open ruling requested for Paul
+
+5. **T7 egress ruling — under-metering of backhauled egress, including
+ an abuse case.** By Task 3's hard constraint (T3 review) the ONLY
+ usage meter (`edge.go:840`) sits below a successful local
+ `lookupUpstream`; a backhauling edge misses it by construction, so a
+ route's cross-region backhaul traffic is billed (AWS egress) on the
+ SENDING box's account but never metered (CARLOS usage/billing) on
+ either box — the bytes leave the sending box's NIC twice as expensive
+ as a locally-served request and the platform's own metering sees none
+ of it. Task 4 (`bdcf6f6`..`0193912`) confirmed this is not merely a
+ missed-revenue gap: it is an **abuse surface**. Backhaul dials any
+ host the home map names, on an unverified hello, before any hop trust
+ check applies to the INBOUND request that triggered the dial — so a
+ scanner sending forged `Host:` headers at a non-home edge for hosts
+ that happen to be real elsewhere in the deployment generates real
+ cross-region egress cost on the sending box's AWS bill, with no
+ member, no account, and no CARLOS usage row to attribute it to.
+ **Containment option on the table, not yet decided:** a per-home-box
+ in-flight semaphore bounding concurrent backhaul dials toward any one
+ home box — cheap (one counter, no new storage), does not fix
+ attribution, but bounds the blast radius of a scanning flood to a
+ fixed number of concurrent cross-region dials rather than unbounded.
+ This needs a ruling before Part 4 (steering) sends real, sustained
+ traffic through this path — today it is dormant/low-volume by
+ construction (no steering, no DNS pointing anywhere but the home
+ region), which is the window in which to decide, not after. Cross-refs
+ unit-economics (the meter placement is `internal/usage`'s territory,
+ not `internal/edge`'s to relocate unilaterally) and the object of this
+ ruling should be recorded as a numbered issue once Paul rules on the
+ containment approach.
+
+### Constraint for Part 4 / plan 3 (steering) to honor
+
+6. **Pool narrowing: steered non-shared-pool routes must not get
+ out-of-pool latency records.** Task 5 (`15c3673`..`af006be`) confirmed
+ `inScope` tests `poolServed` FIRST and unchanged — the everywhere-class
+ widening in this plan only ever widens the REGION axis, never the pool
+ axis, so a `tito`-pool static record is still refused by a box serving
+ a different pool. Part 4's Route53 record generation must respect the
+ same boundary: a steered route in a non-shared pool must only get
+ latency records for the armed edges that actually serve its pool, not
+ every armed edge in the deployment — otherwise DNS would advertise a
+ path to a box that has never adopted the route and cannot serve it
+ (the everywhere-class widening's pool refusal would make that box 404
+ or refuse the hop, not backhaul it, since the route never landed
+ there). Stated as a plan-3 design constraint, not implemented here.
+
+### Operator/dashboard notes at second-region arming
+
+7. **`DNSStatus: "elsewhere"` false-alarm risk at second-region arming.**
+ `dnsStatus` (`internal/domains/reconcile.go:465`) classifies a route's
+ resolved IPs against `cfg.EdgeIPs` — the deployment's edge address list,
+ sourced from `PLATFORM_EDGE_IPS` per box (`internal/host/agent.go:1371`).
+ Each box runs its OWN `domains.Reconcile` pass and stamps its OWN
+ registry. Once Task 5's everywhere-class adoption puts the SAME static
+ routes on every box (not just the home box), every armed box now
+ independently DNS-checks those hosts — and any box whose own
+ `PLATFORM_EDGE_IPS` has not been updated to include the new region's
+ EIP will read a real, correctly-resolving DNS answer (still pointing
+ only at the original region, since Part 4 steering has not run yet) as
+ `"elsewhere"`, because the resolved IP is not in that box's edge set.
+ **Rollout action:** update `PLATFORM_EDGE_IPS` on every box to the
+ full deployment-wide edge IP list at the same time a new region is
+ armed, not after — otherwise the fleet dashboard (which aggregates
+ per-box readings) shows real DNS noise for every everywhere-class host
+ the instant the second box starts reconciling them, indistinguishable
+ at a glance from an actual DNS problem.
+8. **Legacy public-IP single-`Addr` rows: enumerate before regions
+ apply.** Task 5's `publiclyDialable` gate reclassifies any addr record
+ whose upstreams are ALL public IP literals as everywhere-class the
+ moment a second box exists — no code deploy required, just another
+ box appearing. Loopback and RFC1918/ULA private addrs are excluded by
+ design (home-boxed, unchanged), but a legacy single-`Addr` row that
+ happens to carry a real public IP literal would silently start being
+ probed and served from every armed box the day regions apply. Expected
+ empty in both production deployments today, but unverified — enumerate
+ before the first regions-apply:
+ ```sql
+ -- against each deployment's registry (read-only)
+ SELECT host, addr FROM routes WHERE addr <> '' AND kind = '';
+ ```
+ then eyeball each `addr` against `publiclyDialable`'s rule
+ (`net.ParseIP` non-nil, `IsGlobalUnicast()`, not private/loopback/
+ link-local) — any hit is a route to look at BEFORE arming, not after.
+
+ The same before-first-regions-apply window wants a second, adjacent
+ scan that the SQL above cannot reach: bucket **records**
+ (`apps/<sqid>/<app>/instances/<host>.json`, `Region` field —
+ `internal/provision/provision.go:40`) whose region does not match the
+ deployment's own single region. That mismatch is the only shape that
+ would make Task 5's widened everywhere-class adoption gate fire on the
+ very first roll, on a record no one has looked at since it was
+ written, since adoption reads the ROW's region, not the box's
+ assumption of it. Expect empty, same as the addr scan above, but
+ unverified — enumerate with a bucket-wide key scan in the spirit of
+ the bucket-writer audit's own enumeration method
+ (`docs/superpowers/specs/2026-08-20-bucket-writer-audit.md`):
+
+ ```bash
+ aws --profile <profile> s3 sync s3://<control-bucket>/apps/ /tmp/apps-scan/ \
+ --exclude "*" --include "*/instances/*.json"
+ grep -rl '"region"' /tmp/apps-scan | xargs -r grep -L '"region":"<deployment-region>"'
+ ```
+
+ Any file the second `grep` prints carries a region other than the
+ deployment's own — a record to look at BEFORE arming, not after.
+9. **Edges-file `role: "home"` over-claims for static hosts — wants a
+ `"replica"` role before Part 4 leans on the map.** Task 5's own
+ concern (report §4.1): once static routes are everywhere-class, EVERY
+ box that adopts one publishes it with `role: "home"` in its edges
+ file (Task 1's schema has only ever written `"home"`), so the home map
+ folds multiple boxes all claiming to home the same static host. The
+ fold's duplicate rule (first box in sorted order wins) makes this
+ harmless TODAY, because static routes never backhaul (Part 3's table:
+ Everywhere class is "adopt and serve locally... zero backhaul") — the
+ role is never consulted for a routing decision on this class. It stops
+ being harmless the moment Part 4's read-local anticipation (spec
+ "Anticipated, not built" §1) starts reading the `role` field for
+ real — a reader that cannot distinguish "the box that actually homes
+ this process" from "a box that adopted a static copy" cannot implement
+ read-local correctly. The role vocabulary already has the room (spec:
+ `{"role":"home"}` object form, not a bool) — this is a note for
+ whoever designs Part 4's writer change, not a code change here.
+10. **Domains: a static app's apex row appears on far boxes — operator-
+ surprise note, not a DNS-writer change.** Task 5 confirmed (report
+ §4.1, item 3) that `domains.Reconcile` mints the platform apex on a
+ far box too, since Task 5 makes its source route exist there. DNS
+ itself still points only at the home region until Part 4 steering —
+ **this is NOT a new DNS writer and does not change what answers a
+ real client sees**, disproving any read of it as a DNS-authority
+ change. It is purely a local-registry/console-visibility surprise: an
+ operator inspecting a far box's own domain rows will now see an apex
+ entry for an app they may not expect that box to know about at all.
+ Worth a line in the fleet dashboard / `carlos apps` docs before
+ stage 5, not a blocker.
+
+### Cross-referenced issues
+
+11. **#250 — everywhere-class rows with operator-touched
+ Backing/Tunnel/Caps survive record deletion on far boxes ("far-box
+ survivors").** Filed during Task 5's review round 1 (fix commit
+ `af006be`). Since `15c3673`, a static or public-addr record is
+ materialized on every platform box, and the removal phase reads the
+ class off the ROW (`everywhereRoute`), which deliberately answers
+ "home-boxed" for any row carrying `Backing`, `Tunnel` or `Caps` — a
+ conservative choice, but it means such a row is never retired once
+ touched. **Corrected reproduction** (the plain `-grant`/`-backing`
+ phrasing this note originally carried does not hold: both rails
+ refuse non-instance rows — `grantRail`, `cli.go:1197`, refuses
+ anything but a unit-backed instance route; `backingRail`, `cli.go:1370`,
+ refuses any non-instance kind outright — so neither reaches a static
+ row directly):
+ - a public-addr **instance** route (the Everywhere-probed class) on a
+ far box: `carlos route -backing unit` succeeds (kind `instance`,
+ not Platform), then `-grant <token>` succeeds (kind `instance`,
+ `Backing == unit`) — both rails clear because the row really is an
+ instance row, just a far-box copy of one;
+ - **or any everywhere-class row at all, static included**, touched
+ with `-tunnel <name>`: there is no `tunnelRail` — the tunnel axis
+ (`cli.go:951-974`) checks only `platformCanaryOwnedRow`, no kind
+ check exists, so a far box's static copy takes a tunnel stamp as
+ readily as an instance row.
+ Either path leaves `Backing`/`Tunnel` non-empty on the far copy, and
+ deleting the record then retires the home box's route while the far
+ box keeps serving the retired site forever, grants intact, with no
+ record left anywhere to explain why. The gate stays conservative on
+ purpose (the alternative hands an unattended bucket-record deletion
+ the capability-grant purge's human-gated contract), so the fix is
+ visibility + a supported cleanup verb, not a gate change. Relevant to
+ this rollout because everywhere-class adoption (this plan) is what
+ makes far-box copies routine for the first time — #250 was inert
+ before this branch.
+12. **#179 — deployment-wide secrets key, tracked as a mesh blocker.**
+ Task 5's concern (report §4.1, item 2): `appconfig.Sync` now renders
+ an everywhere-class app's `<host>.env` on every box that adopts it,
+ which is more copies of a tenant's rendered config at rest — not a
+ new READ capability, since every box already holds the
+ deployment-wide secrets key and could always read the whole store,
+ the exact coupling #179 tracks. #179 is a pre-existing blocker (also
+ surfaced on the trust-placement branch's mesh design work), not
+ something this branch introduces; recorded here as the cross-ref
+ Task 5's review asked for, not as new scope for this plan.
+
+### Deferred code-quality items worth a rollout-time watch
+
+13. **`backhaulTransports` is keyed by EIP and never evicts.** Task 4's
+ review flagged this as a minor, deferred item, and it turns real once
+ boxes actually get replaced: an EIP re-association (box replacement,
+ which the fleet already does routinely — see the IMDS-blip EIP-hold
+ logic Task 1 built) leaves the OLD EIP's cached `*http.Transport`
+ (`internal/edge/edge.go:184`, `backhaulTransports sync.Map`) stranded
+ in the map forever rather than pruned — bounded by the number of
+ distinct EIPs a deployment's boxes have ever held over its lifetime,
+ not by anything that self-cleans. Not a correctness bug (a new EIP
+ gets its own entry; the stale one is just dead weight), but worth
+ revisiting once box replacement during multi-region operation is a
+ routine event rather than a hypothetical.
+14. **The backhaul 502 `ErrorHandler` is unthrottled**
+ (`logThrottled` key `"backhaul-error "+box`, per-box but every dial
+ failure still logs). During the dev/Kass and flagship-AU rollout
+ exercises (item 3 above), a misconfigured home map or a genuinely
+ dead box will produce one line per failed request rather than a
+ throttled cadence — expected noise during the arming window, not a
+ defect, but flagged so it isn't mistaken for something worse the
+ first time it's seen live. **`staleHopRefusal` (`internal/edge/homemap.go:551`)
+ is silent in the same neighbourhood, the other direction** — a
+ sender stuck dialing a home box whose map has moved on gets its 503
+ answered correctly but nothing is LOGGED on either end, so a sender
+ looping into that refusal (map propagation stuck, a box that never
+ converged the move) produces zero signal that anything is wrong.
+ Task 3's review named this the "converge is not healing this" case
+ and wanted the same first-and-every-tenth cadence the mirror's
+ partial-poll logging already uses elsewhere in this branch — not
+ fixed here, but worth doing the next time anyone is in that file
+ rather than left to be rediscovered blind during a real stuck-move
+ incident.
+
+**Full gate** (env `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod
+GOPROXY=off`): `go build ./...`, `go vet ./...`, `GOOS=darwin go build
+./...`, `go test ./... -count=1`, `gofmt -l` on touched files — all
+clean; this task's changes are docs-only (spec + this plan file), so
+the gate is trivially green (no `.go` file in the diff).
diff --git a/docs/superpowers/specs/2026-08-20-multi-region-ingress-design.md b/docs/superpowers/specs/2026-08-20-multi-region-ingress-design.md
index 0cd27bb3a..19e15f475 100644
--- a/docs/superpowers/specs/2026-08-20-multi-region-ingress-design.md
+++ b/docs/superpowers/specs/2026-08-20-multi-region-ingress-design.md
@@ -74,8 +74,8 @@ in the plan's rollout notes, not code.
| `releases/**` retention prune | every box | moves behind the **housekeeper lease** — five concurrent pruners racing a promote is a bug class we refuse to have. **Implemented, dormant** (`45a2ea6`..`aadd18e`) |
| usage metering day objects | the (single) edge | **audit**: if day-object keys are not box-scoped, they become `…/<box>/…` in the same shape as snapshots. **Confirmed already box-scoped** (`usage/edge/<box>/days/`, bucket-writer audit) |
| `control/certs/**` (new; carries HTTP-01 tokens as `<token>+http-01` and tls-alpn-01 token certs as `<host>+token` — no separate acme-challenges prefix) | housekeeper | Part 2 — **implemented, dormant** (`7eff14b`..`f7e67fe`) |
-| `control/edges/<box>.json` (new) | each box, own key only | Part 3 |
-| `control/edge-mesh-key` (new) | housekeeper mints if absent (create-only CAS) | Part 3 |
+| `control/edges/<box>.json` (new) | each box, own key only | Part 3 — **implemented, dormant** (`23a2b0c`..`e964df3`) |
+| `control/edge-mesh-key` (new) | housekeeper mints if absent (create-only CAS) | Part 3 — **implemented, dormant** (`fc2bad7`, `0c07490`) |
Stage-1 audit deliverable: an enumerated list of EVERY bucket writer in
the codebase with its ruling from the table above, checked into the spec's
@@ -173,8 +173,49 @@ over with identical machinery. Distributed certs keep every edge serving
through the gap; a dead issuer costs nothing until the earliest renewal
window ~30 days out.
+**The serving chain, and the last layer's boot-outage rationale.**
+`GetCertificate` on a non-issuer edge tries, in order: the wildcard
+handler, then this mirror, then the box's own local autocert manager,
+then — last — a direct read of that box's own on-disk autocert cache
+(`internal/edge/certsync.go`, `localCertificate`). That last layer exists
+because it is not redundant with the mirror: it is what a box serves when
+the bucket is unreachable at boot (an empty mirror) but its own disk
+still holds yesterday's certs, and what covers the ~20 minute window
+after a boot or a crash while the housekeeper election has not yet
+resolved. **Amendment (2026-08-22, T7):** that last-resort read is
+negatively memoized for `localCertTTL` (30s) — a name whose disk lookup
+comes back empty is not re-read from disk for 30s. Harmless in practice:
+the mirror is consulted first and answers ahead of this layer whenever it
+has anything, so a memoized nil here only matters in the narrow window
+where BOTH the mirror and this box's own disk have nothing for the name —
+it cannot cost a handshake the mirror could have served. Recorded because
+Task 6's two-edge e2e (`67de75f`) is the first thing to demonstrate the
+negative memoization directly (its P1 subtest), not because it changed
+behaviour.
+
## Part 3 — serving classes and backhaul
+**Implementation status (2026-08-22): implemented, dormant.** The per-box
+edges file (commits `23a2b0c`..`e964df3`) and the housekeeper-minted,
+dual-accept mesh key (`fc2bad7`, `0c07490`) are Part 1's writer-map rows
+above. The home-map mirror and one-hop request-path acceptance
+(`42844ae`..`67c2034`), backhaul dialing with a real TLS hop and a
+per-home-box cached transport (`bdcf6f6`..`0193912`), everywhere-class
+adoption for static/bake routes plus the paired `:80` remote-homed
+redirect (`15c3673`..`af006be`), and the two-edge e2e proving one
+marker-body backhaul, the poisoned-map 502, the stale-map 503 with no
+re-dial, and a tampered-auth hop degrading to an ordinary client
+(`67de75f`) are Part 3's own rows above. Single-box deployments: the home
+map holds only the local box, `Home()` never offers a backhaul target,
+the mesh key mints once and sits unused, and everywhere-class adoption
+outcomes are byte-identical to today's (see each task's single-box
+inertness note). Residual: nothing in this branch has run across two
+REAL boxes — the e2e is in-process with edges files written directly into
+a fake store — tracked as a rollout gate, not a blocker (see the plan's
+rollout notes). Egress attribution for backhauled bytes is an open
+ruling, not a defect: see the plan's rollout notes, T7 egress ruling
+request.
+
| Class | Kinds | Behaviour on a non-home edge |
|---|---|---|
| Everywhere | static, bake-served | adopt and serve locally from converged artifacts — zero backhaul |
@@ -182,11 +223,22 @@ window ~30 days out.
| Home-boxed | exec, unit, socket-activated, tunnel-backed | backhaul to the home edge. The process exists on exactly one box — unchanged invariant |
**Home resolution.** Each box maintains `control/edges/<box>.json`: its
-EIP and a map `host → {role}` for the routes it homes, written by its
-converge loop, serial-bumped on change. v1 writes only `role: "home"`.
-Every edge folds these files into a deployment-wide host→home map. The
-file doubles as the DR runbook's explicit statement of what a dead box
-was serving.
+EIP and a map `host → {role}` for the routes it homes. v1 writes only
+`role: "home"`. **Amendment (2026-08-22):** written by each box's own
+loop (the registry snapshot loop's arming site and cadence, 5min +
+jitter), change-gated — a same-content refresh with only a new
+timestamp writes nothing. Edge writes do NOT bump `control/serial` —
+the serial stays console-written per the writer map, matching Part 2's
+certs ruling; edges mirror the `control/edges/` prefix on their own
+30-second poll, the same cadence the certs mirror uses. Every edge
+folds these files into a deployment-wide host→home map. The file
+doubles as the DR runbook's explicit statement of what a dead box was
+serving. **Keys are registry row hosts verbatim, including `*.parent`
+catch-all rows** — this writer performs no wildcard expansion, so a
+reader that needs exact-host answers must apply `lookupUpstream`'s
+one-label wildcard rule (`internal/edge/edge.go`) itself: an exact
+match wins, and on a miss the first label's leading dot is swapped for
+`*` before trying again.
**The hop.** A remote edge dials `home-EIP:443` with ordinary SNI — the
home edge presents the same bucket-distributed cert. Two headers make it
@@ -201,6 +253,12 @@ forwarded headers stripped, normal serving rules. Timestamp skew bound
±30s (boxes run NTP). Key rotation: dual-accept window, old + new both
verify until every edge converges the new key.
+**Amendment (2026-08-22):** Task 3 shipped verbatim-restore, not append —
+on a verified hop the receiving edge replaces its own computed
+`X-Forwarded-For` with the sending edge's inbound chain unchanged (the
+last element stays the client); the sending edge's own address is never
+appended (`internal/edge/homemap.go`'s `setForwarded`).
+
**Hibernation** composes unchanged: the home edge owns wake (wake-once
singleflight, #199), and a backhaul hop is just another first client.
diff --git a/internal/edge/backhaul_e2e_test.go b/internal/edge/backhaul_e2e_test.go
new file mode 100644
index 000000000..98eca40ba
--- /dev/null
+++ b/internal/edge/backhaul_e2e_test.go
@@ -0,0 +1,870 @@
+package edge
+
+// Task 6 of the multi-region backhaul plan: the acceptance test the whole of
+// Part 3 exists for — ONE control bucket, TWO edges, and a request that lands
+// in the wrong region and is served anyway.
+//
+// Edge A ("annie") homes app.backhaul.test and runs the tenant's app behind it.
+// Edge B ("betty") has never heard of that host: no registry route, no
+// certificate of its own, no way to reach the app except through A. Everything
+// between the two is the bucket — the per-box edges files (Task 1's schema,
+// written here directly because the publishing loop lives in internal/host),
+// the mesh key (Task 2), the certs Part 2 distributes — and one TLS hop.
+//
+// What makes this an e2e rather than a bigger unit test is that BOTH doors are
+// real and neither is stubbed:
+//
+// - the client completes a real TLS handshake with the SENDING edge, with a
+// real x509 verification against the test CA, for a host that edge does not
+// route. That is the invariant the whole feature rides on and the one the
+// unit tests can only assert one branch of at a time: certMirror must
+// answer a hello for a host this box has no route for, or the client never
+// gets far enough to be backhauled at all (P1 below);
+// - the hop itself is a second real TLS handshake, from B's own transport to
+// A's front door, verified the same way.
+//
+// The two edges are wired exactly as RunWithActivatorReady wires one (raw local
+// cache kept aside, publishing wrapper over it, mirror built from the RAW
+// handle, autocert behind the mirror, the :80 chain composed in the same
+// order), and every poll is DRIVEN — Refresh is called, never waited for. There
+// are no sleeps in this file.
+//
+// Neither edge is the deployment's certificate issuer. That is deliberate and
+// it is the sharp case, not a convenience: issuance is Part 2's subject, while
+// what Part 3 needs is that a box which may never talk to a CA can still answer
+// handshakes for hosts it does not serve. Both managers point at a dead ACME
+// directory as well, so nothing here can reach a CA even by accident.
+//
+// Conventions are the neighbouring harnesses' (certsync_e2e_test.go for the
+// two-edge shape and the e2e* naming, homemap_test.go for the hop vocabulary
+// helpers): real loopback listeners, helpers reused rather than re-cut, bounds
+// asserted rather than narrated. No t.Parallel anywhere — this package's tests
+// swap package-level seams (backhaulPort, backhaulRootCAs) and the log writer.
+
+import (
+ "context"
+ "crypto/ecdsa"
+ "crypto/elliptic"
+ "crypto/rand"
+ "crypto/tls"
+ "crypto/x509"
+ "crypto/x509/pkix"
+ "fmt"
+ "io"
+ "math/big"
+ "net"
+ "net/http"
+ "net/http/httptest"
+ "strings"
+ "sync/atomic"
+ "testing"
+ "time"
+
+ "golang.org/x/crypto/acme/autocert"
+
+ "github.com/carlosframework/platform/internal/registry"
+ "github.com/carlosframework/platform/internal/release"
+)
+
+const (
+ // e2eHomedHost is the host the whole scenario turns on: routed on edge A,
+ // unknown to edge B, homed on A in every box's map.
+ e2eHomedHost = "app.backhaul.test"
+ // e2eMovedHost is homed on A in the MAP but no longer routed there — the
+ // stale-map case, which is what a placement move looks like from the
+ // sending edge for one publish cycle.
+ e2eMovedHost = "moved.backhaul.test"
+ // e2eDeadHost is homed on a box that answers nothing: the poisoned map.
+ e2eDeadHost = "dead.backhaul.test"
+ // e2eUnhomedHost is in nobody's edges file and nobody's registry — the
+ // answer for it must stay the plain unknown-host 404 on every box.
+ e2eUnhomedHost = "nothere.backhaul.test"
+ // e2eSenderHost is edge B's OWN route, so B is a real box with traffic of
+ // its own rather than a bare forwarder.
+ e2eSenderHost = "local.betty.test"
+
+ e2eHomeMarker = "served by the home edge's own app"
+ e2eSenderMarker = "served by the sending edge's own app"
+)
+
+// ---- the tenant app ----------------------------------------------------------
+
+// e2eApp is a tenant's app behind an edge: a marker body that says WHICH box's
+// upstream produced it, plus the trust-bearing headers it was handed. One
+// response therefore proves both halves at once — that the bytes crossed the
+// hop, and what the app was told about who called.
+//
+// X-Forwarded-For is rendered from Values(), joined visibly, because the
+// property under test is the LENGTH of the chain as much as its content: a
+// backhaul that appended the sending edge would show up here as a second
+// element (or a second header line), and Get() would hide it.
+func e2eApp(t *testing.T, marker string) *httptest.Server {
+ t.Helper()
+ ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ fmt.Fprintf(w, "%s|xff=%s|xfh=%s|hop=%s|auth=%s", marker,
+ strings.Join(r.Header.Values("X-Forwarded-For"), " + "),
+ r.Header.Get("X-Forwarded-Host"),
+ r.Header.Get(hopHeader), r.Header.Get(hopAuthHeader))
+ }))
+ t.Cleanup(ts.Close)
+ return ts
+}
+
+// e2eSeen is one parsed e2eApp response.
+type e2eSeen struct {
+ marker, xff, xfh, hop, auth string
+}
+
+func e2eParseApp(t *testing.T, body string) e2eSeen {
+ t.Helper()
+ parts := strings.Split(body, "|")
+ if len(parts) != 5 {
+ t.Fatalf("body %q did not come from the tenant app at all", body)
+ }
+ seen := e2eSeen{marker: parts[0]}
+ for _, p := range parts[1:] {
+ k, v, _ := strings.Cut(p, "=")
+ switch k {
+ case "xff":
+ seen.xff = v
+ case "xfh":
+ seen.xfh = v
+ case "hop":
+ seen.hop = v
+ case "auth":
+ seen.auth = v
+ }
+ }
+ return seen
+}
+
+// ---- the deployment's certificate authority ----------------------------------
+
+// e2eCA stands in for the public CA Part 2's issuer orders from. A real CA
+// rather than a self-signed leaf per host, because every handshake in this file
+// is verified the way a browser verifies one — RootCAs, hostname, expiry — and
+// a leaf trusted as its own root would quietly excuse a chain that production
+// would reject.
+type e2eCA struct {
+ root *x509.Certificate
+ key *ecdsa.PrivateKey
+ pool *x509.CertPool
+}
+
+func newE2ECA(t *testing.T) *e2eCA {
+ t.Helper()
+ key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
+ if err != nil {
+ t.Fatal(err)
+ }
+ tmpl := &x509.Certificate{
+ SerialNumber: big.NewInt(1),
+ Subject: pkix.Name{CommonName: "CARLOS backhaul test CA"},
+ NotBefore: time.Now().Add(-time.Hour),
+ NotAfter: time.Now().Add(365 * 24 * time.Hour),
+ KeyUsage: x509.KeyUsageCertSign,
+ BasicConstraintsValid: true,
+ IsCA: true,
+ }
+ der, err := x509.CreateCertificate(rand.Reader, tmpl, tmpl, &key.PublicKey, key)
+ if err != nil {
+ t.Fatal(err)
+ }
+ root, err := x509.ParseCertificate(der)
+ if err != nil {
+ t.Fatal(err)
+ }
+ pool := x509.NewCertPool()
+ pool.AddCert(root)
+ return &e2eCA{root: root, key: key, pool: pool}
+}
+
+// publish mints a certificate for host and writes it to control/certs/<host>,
+// in autocert's own cache-entry shape and through putCert — byte-for-byte what
+// the issuer's publishingCache leaves in the bucket. Every edge in the
+// deployment then serves it; none of them issued it.
+func (ca *e2eCA) publish(t *testing.T, s release.Store, host string) *x509.Certificate {
+ t.Helper()
+ key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
+ if err != nil {
+ t.Fatal(err)
+ }
+ serial, err := rand.Int(rand.Reader, new(big.Int).Lsh(big.NewInt(1), 96))
+ if err != nil {
+ t.Fatal(err)
+ }
+ tmpl := &x509.Certificate{
+ SerialNumber: serial,
+ Subject: pkix.Name{CommonName: host},
+ DNSNames: []string{host},
+ NotBefore: time.Now().Add(-time.Hour),
+ NotAfter: time.Now().Add(90 * 24 * time.Hour),
+ KeyUsage: x509.KeyUsageDigitalSignature,
+ ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
+ }
+ der, err := x509.CreateCertificate(rand.Reader, tmpl, ca.root, &key.PublicKey, ca.key)
+ if err != nil {
+ t.Fatal(err)
+ }
+ leaf, err := x509.ParseCertificate(der)
+ if err != nil {
+ t.Fatal(err)
+ }
+ raw, err := encodeWildcardPEM(&tls.Certificate{Certificate: [][]byte{der}, PrivateKey: key, Leaf: leaf}, key)
+ if err != nil {
+ t.Fatal(err)
+ }
+ putBucket(t, s, host, raw)
+ return leaf
+}
+
+// ---- one edge ----------------------------------------------------------------
+
+// backhaulEdge is one box: a real Edge over its OWN handle on the shared
+// bucket, with the full handshake chain and the full :80 chain composed the way
+// RunWithActivatorReady composes them, on real loopback listeners.
+type backhaulEdge struct {
+ edge *Edge
+ store release.Store
+ homes *homeMap
+ mirror *certMirror
+ mgr *autocert.Manager
+ // local is the RAW autocert cache — this box's own disk, which starts
+ // empty and is only ever written by the mirror.
+ local autocert.Cache
+
+ tlsAddr, plainAddr string
+ // calls counts requests that reached this box's 443 door, refusals
+ // included: "exactly one dial" and "never dialled at all" are both this
+ // number moving, or not, on the OTHER box.
+ calls atomic.Int64
+ // plainCalls is the same for :80.
+ plainCalls atomic.Int64
+
+ client *http.Client
+}
+
+// newBackhaulEdge builds one box. routes are host → upstream address; anything
+// not listed is a host this box genuinely does not serve.
+func newBackhaulEdge(t *testing.T, bucketDir, box string, ca *e2eCA, from *net.TCPAddr, routes map[string]string) *backhaulEdge {
+ t.Helper()
+ be := &backhaulEdge{store: release.NewDirStore(bucketDir)}
+ reg := openReg(t)
+ for host, addr := range routes {
+ if err := reg.Upsert(registry.Route{Host: host, Kind: "service", Addr: addr, Status: "ready"}); err != nil {
+ t.Fatalf("upsert %s: %v", host, err)
+ }
+ }
+ e, err := New(reg, nil)
+ if err != nil {
+ t.Fatalf("New: %v", err)
+ }
+ t.Cleanup(e.probers.Stop)
+ be.edge = e
+ e.staticStore = be.store
+ WithBoxLabel(box)(e)
+ // Not the issuer, and never able to become one: the gate is shut and the
+ // manager below points at a dead directory. Everything this box serves on
+ // 443 therefore came out of the bucket.
+ WithIssuerGate(func() (context.Context, bool) { return nil, false })(e)
+
+ // The RAW cache, kept before the publishing wrapper goes on (edge.go).
+ be.local = autocert.DirCache(t.TempDir())
+ cache := e.armCertIssuance(be.local)
+ be.mirror = e.newCertMirror(be.local)
+ if be.mirror == nil {
+ t.Fatal("a store-backed edge must have a mirror to serve from")
+ }
+ be.mgr = offlineManager(t, cache, e.issuerHostPolicy())
+ tlsCfg := be.mgr.TLSConfig()
+ tlsCfg.GetCertificate = be.mirror.wrapGetCertificate(tlsCfg.GetCertificate)
+
+ be.homes = e.newHomeMap()
+ if be.homes == nil {
+ t.Fatal("a store-backed edge must have a home map")
+ }
+ e.homes.Store(be.homes)
+
+ // 443: the edge itself, counted.
+ https := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ be.calls.Add(1)
+ e.ServeHTTP(w, r)
+ }))
+ https.TLS = tlsCfg
+ https.StartTLS()
+ t.Cleanup(https.Close)
+ be.tlsAddr = https.Listener.Addr().String()
+
+ // :80, outermost first, exactly as RunWithActivatorReady composes it: the
+ // mirror's challenge handler, then autocert's, then the redirect fallback.
+ // Composed ONCE, before anything listens, for the reason edge.go composes
+ // it there — HTTPHandler is what arms autocert's HTTP-01 path.
+ acmeAndFallback := be.mirror.challengeHandler(be.mgr.HTTPHandler(e.httpFallbackHandler()))
+ plain := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ be.plainCalls.Add(1)
+ acmeAndFallback.ServeHTTP(w, r)
+ }))
+ t.Cleanup(plain.Close)
+ be.plainAddr = plain.Listener.Addr().String()
+
+ be.client = e2eClientFor(be, ca, from)
+ return be
+}
+
+// e2eClientFor is a client that resolves every hostname to ONE box's two
+// listeners — the deployment's DNS, in other words, pointed wherever the test
+// wants this request to land. https connections are handshaken here rather than
+// by the transport so that the client verifies the edge's certificate against
+// the test CA, by name, exactly as a browser would: a mirrored certificate that
+// did not actually cover the host would fail here and not quietly downstream.
+func e2eClientFor(be *backhaulEdge, ca *e2eCA, from *net.TCPAddr) *http.Client {
+ dialer := &net.Dialer{Timeout: 10 * time.Second, LocalAddr: from}
+ return &http.Client{
+ Timeout: 30 * time.Second,
+ Transport: &http.Transport{
+ DialContext: func(ctx context.Context, network, _ string) (net.Conn, error) {
+ return dialer.DialContext(ctx, network, be.plainAddr)
+ },
+ DialTLSContext: func(ctx context.Context, network, addr string) (net.Conn, error) {
+ name, _, err := net.SplitHostPort(addr)
+ if err != nil {
+ return nil, err
+ }
+ raw, err := dialer.DialContext(ctx, network, be.tlsAddr)
+ if err != nil {
+ return nil, err
+ }
+ conn := tls.Client(raw, &tls.Config{ServerName: name, RootCAs: ca.pool, MinVersion: tls.VersionTLS12})
+ if err := conn.HandshakeContext(ctx); err != nil {
+ raw.Close()
+ return nil, err
+ }
+ return conn, nil
+ },
+ },
+ }
+}
+
+// ---- the scene ---------------------------------------------------------------
+
+type backhaulScene struct {
+ ca *e2eCA
+ seed release.Store // the housekeeper's handle: what writes the control plane
+ a, b *backhaulEdge
+ key []byte // the deployment's mesh key, as both edges mirror it
+ // leaf is the certificate the bucket carries per host.
+ leaf map[string]*x509.Certificate
+ // clientIP is the source address the test client dials from, or "" if the
+ // platform would not give us a second loopback address.
+ clientIP string
+ deadEIP string
+}
+
+func newBackhaulScene(t *testing.T) *backhaulScene {
+ t.Helper()
+ e2eQuietLogs(t)
+ bucket := t.TempDir()
+ s := &backhaulScene{
+ seed: release.NewDirStore(bucket),
+ ca: newE2ECA(t),
+ leaf: map[string]*x509.Certificate{},
+ deadEIP: e2eDeadEIP(t),
+ }
+
+ // The housekeeper's one-time mint (Task 2), and the certs Part 2's issuer
+ // publishes. Both edges read them; neither wrote them.
+ if err := MintMeshKey(context.Background(), s.seed); err != nil {
+ t.Fatalf("mint mesh key: %v", err)
+ }
+ for _, host := range []string{e2eHomedHost, e2eMovedHost, e2eDeadHost, e2eUnhomedHost, e2eSenderHost} {
+ s.leaf[host] = s.ca.publish(t, s.seed, host)
+ }
+
+ from := e2eClientAddr(t)
+ if from != nil {
+ s.clientIP = from.IP.String()
+ }
+ s.a = newBackhaulEdge(t, bucket, "annie", s.ca, from, map[string]string{
+ e2eHomedHost: e2eApp(t, e2eHomeMarker).Listener.Addr().String(),
+ })
+ s.b = newBackhaulEdge(t, bucket, "betty", s.ca, from, map[string]string{
+ e2eSenderHost: e2eApp(t, e2eSenderMarker).Listener.Addr().String(),
+ })
+
+ // The dial seams, armed for the test's lifetime: 443 is an ephemeral port
+ // here, and the home edge's certificate is signed by the test CA rather
+ // than a public one. Nothing about verification is weakened — the hop still
+ // checks the name and the chain (see backhaulRootCAs' own doc).
+ //
+ // One consequence of backhaulPort being a single global: every box in this
+ // scene is dialled on the SAME port, so a box's EIP is the whole of its
+ // identity on the wire. That is why the dead box below gets an address of
+ // its own rather than a port of its own.
+ eip, port, err := net.SplitHostPort(s.a.tlsAddr)
+ if err != nil {
+ t.Fatalf("split home edge address: %v", err)
+ }
+ oldPort, oldCAs := backhaulPort, backhaulRootCAs
+ backhaulPort, backhaulRootCAs = port, s.ca.pool
+ t.Cleanup(func() { backhaulPort, backhaulRootCAs = oldPort, oldCAs })
+
+ // The per-box edges files, Task 1's schema, written straight into the store
+ // (the publishing loop is internal/host's and is not in this process).
+ bEIP, _, err := net.SplitHostPort(s.b.tlsAddr)
+ if err != nil {
+ t.Fatalf("split sending edge address: %v", err)
+ }
+ s.publishEdgesFile(t, "annie", eip, e2eHomedHost, e2eMovedHost)
+ s.publishEdgesFile(t, "betty", bEIP, e2eSenderHost)
+ s.publishEdgesFile(t, "carol", s.deadEIP, e2eDeadHost)
+ return s
+}
+
+// publishEdgesFile writes one box's control/edges/<box>.json — homedOn's shape,
+// written through the seed handle so neither edge is the publisher of its own
+// map (which is the production writer arrangement: the box's own host loop
+// writes it, and every edge only ever READS the prefix).
+func (s *backhaulScene) publishEdgesFile(t *testing.T, box, eip string, hosts ...string) {
+ t.Helper()
+ entries := make([]string, 0, len(hosts))
+ for _, h := range hosts {
+ entries = append(entries, fmt.Sprintf("%q:{\"role\":\"home\"}", h))
+ }
+ putJSON(t, s.seed, "control/edges/"+box+".json", fmt.Sprintf(
+ `{"v":1,"box":%q,"eip":%q,"hosts":{%s}}`, box, eip, strings.Join(entries, ",")))
+}
+
+// converge drives every mirror and every map ONE poll forward. Explicit, never
+// waited for: the loops (certMirror.run, homeMap.run) are 30s tickers, and a
+// test that slept for them would be testing the clock.
+func (s *backhaulScene) converge(t *testing.T) {
+ t.Helper()
+ for _, be := range []*backhaulEdge{s.a, s.b} {
+ if err := be.mirror.Refresh(context.Background()); err != nil {
+ t.Fatalf("cert mirror refresh: %v", err)
+ }
+ refreshHomeMap(t, be.homes)
+ }
+ key, next := s.a.homes.meshKeys()
+ if len(key) == 0 || len(next) != 0 {
+ t.Fatalf("mesh keys mirrored as current=%d next=%d bytes, want one current key", len(key), len(next))
+ }
+ s.key = key
+}
+
+// get drives one client request at one box and reads the whole body. follow
+// says whether redirects are followed — the :80 leg wants both answers.
+func (s *backhaulScene) get(t *testing.T, be *backhaulEdge, url string, headers map[string]string, follow bool) (*http.Response, string) {
+ t.Helper()
+ resp, err := s.try(be, url, headers, follow)
+ if err != nil {
+ t.Fatalf("GET %s: %v", url, err)
+ }
+ body, err := io.ReadAll(resp.Body)
+ resp.Body.Close()
+ if err != nil {
+ t.Fatalf("read body of %s: %v", url, err)
+ }
+ return resp, string(body)
+}
+
+func (s *backhaulScene) try(be *backhaulEdge, url string, headers map[string]string, follow bool) (*http.Response, error) {
+ req, err := http.NewRequest(http.MethodGet, url, nil)
+ if err != nil {
+ return nil, err
+ }
+ for k, v := range headers {
+ req.Header.Set(k, v)
+ }
+ client := be.client
+ if !follow {
+ c := *be.client
+ c.CheckRedirect = func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse }
+ client = &c
+ }
+ return client.Do(req)
+}
+
+// e2eClientAddr is the source address the test client dials FROM: a second
+// loopback address, so the client is distinguishable ON THE WIRE from the
+// sending edge, which dials the home edge from the default one. That is what
+// turns "X-Forwarded-For names the client, not the edge that forwarded it" into
+// an assertion rather than a coincidence of both being 127.0.0.1.
+//
+// nil on a platform that will not hand out a second loopback address; the
+// chain-LENGTH assertions still hold there, and the identity ones are skipped
+// explicitly rather than silently passing.
+func e2eClientAddr(t *testing.T) *net.TCPAddr {
+ t.Helper()
+ ln, err := net.Listen("tcp", "127.0.0.2:0")
+ if err != nil {
+ return nil
+ }
+ ln.Close()
+ return &net.TCPAddr{IP: net.ParseIP("127.0.0.2")}
+}
+
+// e2eDeadEIP is an address nothing listens on, for the poisoned-map case: a
+// loopback one when the platform allows it, so the dial is REFUSED immediately
+// rather than hanging. The documentation address is the fallback, and costs the
+// backhaul dial timeout instead — the same 502, more slowly.
+func e2eDeadEIP(t *testing.T) string {
+ t.Helper()
+ if ln, err := net.Listen("tcp", "127.0.0.9:0"); err == nil {
+ ln.Close()
+ return "127.0.0.9"
+ }
+ return "192.0.2.1" // TEST-NET-1, RFC 5737
+}
+
+// ---- the scenario ------------------------------------------------------------
+
+func TestBackhaulE2EAcrossTwoEdges(t *testing.T) {
+ s := newBackhaulScene(t)
+
+ // Before either mirror has polled, edge B holds nothing for the homed host:
+ // empty mirror, empty local cache, autocert gated shut. The handshake must
+ // FAIL — otherwise every success below could be autocert's own doing, or a
+ // certificate that was on B's disk all along.
+ if _, err := s.try(s.b, "https://"+e2eHomedHost+"/", nil, true); err == nil {
+ t.Fatal("the sending edge completed a handshake for a certificate no box has published to it yet")
+ }
+ s.converge(t)
+
+ t.Run("P1 the sending edge answers a hello for a host it does not route", func(t *testing.T) {
+ // The premise, asserted rather than assumed: B has no route for this
+ // host, will not issue for it, and its own manager refuses it outright.
+ if _, ok := s.b.edge.lookupUpstream(e2eHomedHost); ok {
+ t.Fatal("test bug: the sending edge must not route the host it is supposed to backhaul")
+ }
+ if s.b.edge.Allowed(e2eHomedHost) {
+ t.Fatal("test bug: the sending edge must not claim the host for issuance either")
+ }
+ if _, err := s.b.mgr.GetCertificate(ecdsaHello(e2eHomedHost)); err == nil {
+ t.Fatal("the sending edge's own autocert manager must refuse a host it does not serve")
+ }
+
+ // And yet a real client, verifying against the CA by name, completes the
+ // handshake — with the certificate the BUCKET carries. This is the
+ // invariant the feature rides on: without it a remote-homed host dies at
+ // the TLS layer on every edge but its home, and nothing downstream ever
+ // runs.
+ resp, _ := s.get(t, s.b, "https://"+e2eHomedHost+"/", nil, true)
+ if resp.TLS == nil || len(resp.TLS.PeerCertificates) == 0 {
+ t.Fatal("no peer certificate — the handshake did not happen")
+ }
+ if got := resp.TLS.PeerCertificates[0]; !got.Equal(s.leaf[e2eHomedHost]) {
+ t.Fatalf("the sending edge served a certificate that is not the mirrored one (subject %q)", got.Subject.CommonName)
+ }
+ if c := s.b.mirror.certificate(e2eHomedHost, false); c == nil {
+ t.Fatal("the mirror map holds no certificate for the host — it cannot have been the source")
+ }
+ })
+
+ t.Run("the marker body backhauls, the chain names the client, one dial", func(t *testing.T) {
+ home, sender := s.a.calls.Load(), s.b.calls.Load()
+ resp, body := s.get(t, s.b, "https://"+e2eHomedHost+"/deep?q=1", nil, true)
+ if resp.StatusCode != http.StatusOK {
+ t.Fatalf("status = %d, want 200 — a remote-homed host is served through its home edge", resp.StatusCode)
+ }
+ seen := e2eParseApp(t, body)
+ if seen.marker != e2eHomeMarker {
+ t.Fatalf("marker = %q, want the HOME edge's app (%q)", seen.marker, e2eHomeMarker)
+ }
+ if got := s.a.calls.Load() - home; got != 1 {
+ t.Fatalf("the home edge saw %d request(s), want exactly one — one hop, ever", got)
+ }
+ if got := s.b.calls.Load() - sender; got != 1 {
+ t.Fatalf("the sending edge saw %d request(s) for one client request", got)
+ }
+ // The chain the app sees is a single element and it is the CLIENT's
+ // address — not the sending edge's, which is a different loopback
+ // address here precisely so the two can be told apart. Every consumer in
+ // this deployment takes the last element as the caller
+ // (internal/console/authcli.go's clientIP), so an appended edge would
+ // bill, rate-limit and log a real visitor as infrastructure.
+ if strings.Contains(seen.xff, "+") || strings.Contains(seen.xff, ",") {
+ t.Fatalf("X-Forwarded-For = %q, want a single-element chain", seen.xff)
+ }
+ if s.clientIP != "" && seen.xff != s.clientIP {
+ t.Fatalf("X-Forwarded-For = %q, want the client's own address %q", seen.xff, s.clientIP)
+ }
+ if seen.xfh != e2eHomedHost {
+ t.Fatalf("X-Forwarded-Host = %q, want the host the client asked for", seen.xfh)
+ }
+ // The hop vocabulary is the edges' own and stops at the home edge.
+ if seen.hop != "" || seen.auth != "" {
+ t.Fatalf("hop=%q auth=%q reached the tenant's app", seen.hop, seen.auth)
+ }
+ })
+
+ t.Run("the sending edge still serves its own routes without dialling", func(t *testing.T) {
+ home := s.a.calls.Load()
+ resp, body := s.get(t, s.b, "https://"+e2eSenderHost+"/", nil, true)
+ if resp.StatusCode != http.StatusOK {
+ t.Fatalf("status = %d, want 200", resp.StatusCode)
+ }
+ if seen := e2eParseApp(t, body); seen.marker != e2eSenderMarker {
+ t.Fatalf("marker = %q, want the sending edge's OWN app", seen.marker)
+ }
+ if got := s.a.calls.Load() - home; got != 0 {
+ t.Fatalf("the home edge saw %d request(s) for a host the sending edge serves itself", got)
+ }
+ })
+
+ t.Run("the :80 leg redirects, and the redirect it hands out backhauls", func(t *testing.T) {
+ // A first-time visitor typing a bare hostname lands on whichever edge
+ // DNS picked, on :80. Task 5's fix: a remote-homed host is routable, so
+ // it gets the ordinary redirect rather than a 404 for an app that is up.
+ plain := s.b.plainCalls.Load()
+ resp, _ := s.get(t, s.b, "http://"+e2eHomedHost+"/deep?q=1", nil, false)
+ if resp.StatusCode != http.StatusFound {
+ t.Fatalf("status = %d, want 302 — :80 must send a remote-homed host to 443", resp.StatusCode)
+ }
+ if got := s.b.plainCalls.Load() - plain; got != 1 {
+ t.Fatalf("the sending edge's :80 door saw %d request(s), want one — this leg must be plain HTTP", got)
+ }
+ if got, want := resp.Header.Get("Location"), "https://"+e2eHomedHost+"/deep?q=1"; got != want {
+ t.Fatalf("Location = %q, want %q", got, want)
+ }
+ // A host nothing in the deployment homes keeps the unchanged 404 on the
+ // same handler, so the redirect is a routing answer and not a blanket one.
+ if got, _ := s.get(t, s.b, "http://"+e2eUnhomedHost+"/", nil, false); got.StatusCode != http.StatusNotFound {
+ t.Fatalf("status = %d, want the unchanged 404 for a host nothing homes", got.StatusCode)
+ }
+
+ // Both legs, one client call: the visitor follows the redirect and ends
+ // on the home edge's app, having crossed :80 here and 443 there.
+ home := s.a.calls.Load()
+ final, body := s.get(t, s.b, "http://"+e2eHomedHost+"/deep?q=1", nil, true)
+ if final.StatusCode != http.StatusOK {
+ t.Fatalf("status = %d after following the redirect, want 200", final.StatusCode)
+ }
+ if final.Request.URL.Scheme != "https" {
+ t.Fatalf("the followed request ended on %q, want https", final.Request.URL.Scheme)
+ }
+ if final.TLS == nil {
+ t.Fatal("the second leg was not TLS")
+ }
+ if seen := e2eParseApp(t, body); seen.marker != e2eHomeMarker {
+ t.Fatalf("marker = %q, want the home edge's app at the end of both legs", seen.marker)
+ }
+ if got := s.a.calls.Load() - home; got != 1 {
+ t.Fatalf("the home edge saw %d request(s) across the two legs, want one", got)
+ }
+ })
+
+ t.Run("a poisoned map is a bad gateway, and never a loop", func(t *testing.T) {
+ if _, box, ok := s.b.homes.Home(e2eDeadHost); !ok || box != "carol" {
+ t.Fatalf("test bug: the map must name the dead box for %s (got %q, %v)", e2eDeadHost, box, ok)
+ }
+ home, sender := s.a.calls.Load(), s.b.calls.Load()
+ resp, body := s.get(t, s.b, "https://"+e2eDeadHost+"/", nil, true)
+ if resp.StatusCode != http.StatusBadGateway {
+ t.Fatalf("status = %d, want 502 — this edge believed a specific box was there", resp.StatusCode)
+ }
+ if got := resp.Header.Get("Retry-After"); got != "" {
+ t.Fatalf("Retry-After = %q — that answer is reserved for KNOWING nobody is home", got)
+ }
+ if strings.Contains(body, e2eHomeMarker) {
+ t.Fatal("a dead home somehow served the app")
+ }
+ if got := s.b.calls.Load() - sender; got != 1 {
+ t.Fatalf("the sending edge handled %d request(s) for one client request — a failed dial must not be retried around the fleet", got)
+ }
+ if got := s.a.calls.Load() - home; got != 0 {
+ t.Fatalf("the live home edge saw %d request(s) for a host it does not home — a failed dial must not scan for a second target", got)
+ }
+ })
+
+ t.Run("a stale map gets the home edge's 503, and the sender does not re-dial", func(t *testing.T) {
+ // The map still homes this host on A; A no longer routes it. That is
+ // what one publish cycle of a placement move looks like from here.
+ if _, box, ok := s.b.homes.Home(e2eMovedHost); !ok || box != "annie" {
+ t.Fatalf("test bug: the map must still name the home edge for %s (got %q, %v)", e2eMovedHost, box, ok)
+ }
+ if _, ok := s.a.edge.lookupUpstream(e2eMovedHost); ok {
+ t.Fatal("test bug: the home edge must NOT serve the moved host")
+ }
+ home, sender := s.a.calls.Load(), s.b.calls.Load()
+ resp, body := s.get(t, s.b, "https://"+e2eMovedHost+"/", nil, true)
+ if resp.StatusCode != http.StatusServiceUnavailable {
+ t.Fatalf("status = %d, want the home edge's 503", resp.StatusCode)
+ }
+ if got := resp.Header.Get("Retry-After"); got != "2" {
+ t.Fatalf("Retry-After = %q, want 2 — converge heals this within a publish cycle", got)
+ }
+ // The refusal reaches the client as the home edge wrote it, and says
+ // nothing about where the host went.
+ if strings.Contains(body, e2eMovedHost) || strings.Contains(body, "annie") || strings.Contains(body, "betty") {
+ t.Fatalf("body = %q — the refusal must not name the host or any box", body)
+ }
+ if got := s.a.calls.Load() - home; got != 1 {
+ t.Fatalf("the home edge saw %d request(s), want exactly one — a 503 is an answer, not a reason to try again", got)
+ }
+ if got := s.b.calls.Load() - sender; got != 1 {
+ t.Fatalf("the sending edge handled %d request(s) for one client request", got)
+ }
+ })
+
+ t.Run("a tampered hop reaches the home edge as an ordinary client", func(t *testing.T) {
+ // Tampering happens on the wire INTO the home edge: a genuine signature
+ // this deployment's own key produced, with one hex nibble flipped, plus
+ // the forwarding claims a tamperer would want believed.
+ tampered := []byte(signHop(s.key, e2eHomedHost, time.Now()))
+ tampered[len(tampered)-1] ^= 'f' ^ '0'
+ headers := map[string]string{
+ hopHeader: "1",
+ hopAuthHeader: string(tampered),
+ "X-Forwarded-For": "203.0.113.9",
+ "X-Forwarded-Host": "evil.test",
+ "X-Forwarded-Proto": "https",
+ }
+
+ resp, body := s.get(t, s.a, "https://"+e2eHomedHost+"/", headers, true)
+ if resp.StatusCode != http.StatusOK {
+ t.Fatalf("status = %d, want 200 — a failed verification is 'you are an ordinary client', not a rejection", resp.StatusCode)
+ }
+ seen := e2eParseApp(t, body)
+ if seen.marker != e2eHomeMarker {
+ t.Fatalf("marker = %q, want the home edge's app", seen.marker)
+ }
+ if strings.Contains(seen.xff, "203.0.113.9") {
+ t.Fatalf("X-Forwarded-For = %q — a chain nothing vouched for must be discarded", seen.xff)
+ }
+ if s.clientIP != "" && seen.xff != s.clientIP {
+ t.Fatalf("X-Forwarded-For = %q, want the tamperer's own address %q", seen.xff, s.clientIP)
+ }
+ if seen.xfh == "evil.test" {
+ t.Fatalf("X-Forwarded-Host = %q — an unverified sender does not get to state it", seen.xfh)
+ }
+ if seen.hop != "" || seen.auth != "" {
+ t.Fatalf("hop=%q auth=%q reached the app — hop vocabulary is stripped either way", seen.hop, seen.auth)
+ }
+
+ // And no hop SEMANTICS at all: for a host the home edge does not serve,
+ // a tampered hop gets the ordinary unknown-host answer. The same request
+ // with an intact signature gets the stale-map refusal instead — the
+ // signature is the whole of the difference, which is what makes the 503
+ // unreachable to a stranger who cannot mint one.
+ bad := map[string]string{hopHeader: "1", hopAuthHeader: string(tampered)}
+ if got, _ := s.get(t, s.a, "https://"+e2eUnhomedHost+"/", bad, true); got.StatusCode != http.StatusNotFound {
+ t.Fatalf("status = %d, want the unchanged 404 — a tampered hop earns no hop semantics", got.StatusCode)
+ } else if ra := got.Header.Get("Retry-After"); ra != "" {
+ t.Fatalf("Retry-After = %q — a stranger must not be told to come back", ra)
+ }
+ good := map[string]string{hopHeader: "1", hopAuthHeader: signHop(s.key, e2eUnhomedHost, time.Now())}
+ if got, _ := s.get(t, s.a, "https://"+e2eUnhomedHost+"/", good, true); got.StatusCode != http.StatusServiceUnavailable {
+ t.Fatalf("status = %d, want 503 — an INTACT signature is what buys the stale-map refusal", got.StatusCode)
+ }
+ })
+}
+
+// ---- the single-box regression pin -------------------------------------------
+
+// Everything above is dormant on a deployment with one platform box, and this
+// is the pin that says so: an edge whose control/edges/ prefix holds only its
+// OWN file serves its own routes exactly as it always has, answers the
+// unchanged 404 for everything else, and never opens a connection to anything.
+//
+// The witness is a zero-call dialer — a listener that counts every connection
+// made to the backhaul port — with the single box's own edges file naming the
+// listener's ADDRESS as its EIP. So the one failure this pin exists to catch,
+// an edge that offers itself as a backhaul target, lands ON the counter rather
+// than merely failing somewhere else.
+//
+// The file lists a host the box does not route, which is what makes the counter
+// a real witness rather than a formality: a route removed between two publishes
+// leaves exactly that state for one cycle, it is the ONLY shape in which a
+// single box could dial at all (a host it serves is served locally, before the
+// map is ever consulted), and dropping own-box exclusion turns it into a dial to
+// this box's own front door.
+func TestBackhaulE2ESingleBoxNeverDials(t *testing.T) {
+ e2eQuietLogs(t)
+ bucket := t.TempDir()
+ seed := release.NewDirStore(bucket)
+ if err := MintMeshKey(context.Background(), seed); err != nil {
+ t.Fatalf("mint mesh key: %v", err)
+ }
+
+ const soloHost = "solo.backhaul.test"
+ // A host this box's own file still claims but its registry no longer
+ // serves — one publish cycle after a route was removed.
+ const soloStaleHost = "stale.backhaul.test"
+ app := e2eApp(t, e2eHomeMarker)
+ reg := openReg(t)
+ if err := reg.Upsert(registry.Route{Host: soloHost, Kind: "service", Addr: app.Listener.Addr().String(), Status: "ready"}); err != nil {
+ t.Fatalf("upsert: %v", err)
+ }
+ e, err := New(reg, nil)
+ if err != nil {
+ t.Fatalf("New: %v", err)
+ }
+ t.Cleanup(e.probers.Stop)
+ e.staticStore = release.NewDirStore(bucket)
+ WithBoxLabel("solo")(e)
+
+ eip, dials := e2eDialCounter(t)
+ putJSON(t, seed, "control/edges/solo.json", fmt.Sprintf(
+ `{"v":1,"box":"solo","eip":%q,"hosts":{%q:{"role":"home"},%q:{"role":"home"}}}`,
+ eip, soloHost, soloStaleHost))
+
+ m := e.newHomeMap()
+ refreshHomeMap(t, m)
+ e.homes.Store(m)
+ for _, host := range []string{soloHost, soloStaleHost} {
+ if _, _, ok := m.Home(host); ok {
+ t.Fatalf("%s: the only box in a deployment must never be offered as a backhaul target for its own host", host)
+ }
+ }
+
+ if resp := hopRequest(t, e, soloHost, nil); resp.StatusCode != http.StatusOK {
+ t.Fatalf("status = %d, want 200 — a single box serves its own routes as before", resp.StatusCode)
+ } else if seen := e2eParseApp(t, readBody(t, resp)); seen.marker != e2eHomeMarker {
+ t.Fatalf("marker = %q, want the box's own app", seen.marker)
+ }
+ for _, host := range []string{soloStaleHost, "nothing.backhaul.test"} {
+ if resp := hopRequest(t, e, host, nil); resp.StatusCode != http.StatusNotFound {
+ t.Fatalf("%s: status = %d, want the unchanged 404", host, resp.StatusCode)
+ }
+ }
+ if n := dials.Load(); n != 0 {
+ t.Fatalf("%d backhaul dial(s) from a single-box deployment, want none", n)
+ }
+ n := 0
+ e.backhaulTransports.Range(func(_, _ any) bool { n++; return true })
+ if n != 0 {
+ t.Fatalf("%d backhaul transport(s) built, want none — nothing was ever a target", n)
+ }
+}
+
+// e2eDialCounter is the zero-call witness: a listener on the backhaul port that
+// counts connections and answers nothing. It returns the address a dial would
+// have to name to reach it, and arms backhaulPort for the test's lifetime.
+func e2eDialCounter(t *testing.T) (string, *atomic.Int64) {
+ t.Helper()
+ ln, err := net.Listen("tcp", "127.0.0.1:0")
+ if err != nil {
+ t.Fatalf("listen: %v", err)
+ }
+ t.Cleanup(func() { ln.Close() })
+ var n atomic.Int64
+ go func() {
+ for {
+ conn, err := ln.Accept()
+ if err != nil {
+ return
+ }
+ n.Add(1)
+ conn.Close()
+ }
+ }()
+ host, port, err := net.SplitHostPort(ln.Addr().String())
+ if err != nil {
+ t.Fatalf("split: %v", err)
+ }
+ old := backhaulPort
+ backhaulPort = port
+ t.Cleanup(func() { backhaulPort = old })
+ return host, &n
+}
diff --git a/internal/edge/edge.go b/internal/edge/edge.go
index 29c0a6568..bb69f7d43 100644
--- a/internal/edge/edge.go
+++ b/internal/edge/edge.go
@@ -20,6 +20,8 @@ import (
"bufio"
"context"
"crypto/ed25519"
+ "crypto/tls"
+ "crypto/x509"
"encoding/json"
"errors"
"flag"
@@ -168,6 +170,24 @@ type Edge struct {
// every single-box deployment's behaviour to date: mint as always.
// Serving is never gated by it.
issuer IssuerGate
+ // box is this box's own label — the SAME derivation its edges file is
+ // published under (internal/host's metricsBoxLabel), handed down through
+ // WithBoxLabel for the reason the issuer gate is: that package imports
+ // this one and never the reverse. It is what the home map excludes its own
+ // hosts by; "" (bare `carlos edge`) leaves the map offering nothing.
+ box string
+ // homes, when non-nil, is the mirror of control/edges/ plus the mesh keys
+ // (homemap.go): who homes a host this box does not serve, and what proves
+ // an inbound hop. Atomic because it is armed in RunWithActivatorReady after
+ // construction, and read on the request path while that is happening.
+ homes atomic.Pointer[homeMap]
+ // backhaulTransports caches the outbound TLS transport of each backhaul
+ // target, keyed by the home box's EIP — see backhaulTransport. Bounded by
+ // the number of BOXES in the deployment, never by hostnames: a catch-all
+ // row in another box's edges file makes every subdomain under it a
+ // backhaul target, and keying per host would let a stranger's stream of
+ // invented hostnames grow this map without limit.
+ backhaulTransports sync.Map
}
// RunOption configures the Edge that Run/RunWithActivator* builds, before it
@@ -181,6 +201,14 @@ type RunOption func(*Edge)
// as it always has. See IssuerGate (certsync.go).
func WithIssuerGate(g IssuerGate) RunOption { return func(e *Edge) { e.issuer = g } }
+// WithBoxLabel hands the edge this box's own label, the one its edges file is
+// published under (internal/host's metricsBoxLabel — the same call the edges
+// publisher makes, so the two can never disagree about which file is ours).
+// Without it the home map excludes nothing and therefore offers nothing: an
+// edge that cannot recognise its own hosts must not hand out dial targets.
+// See homemap.go.
+func WithBoxLabel(label string) RunOption { return func(e *Edge) { e.box = label } }
+
// upstream is one served host: the handler for its backend, plus — when the
// activator manages it — the route itself, so a request can trigger the wake.
type upstream struct {
@@ -782,8 +810,38 @@ func (e *Edge) ServeHTTP(w http.ResponseWriter, r *http.Request) {
e.tunnels.ServeHTTP(w, r)
return
}
+ // Backhaul hop acceptance (homemap.go). Deliberately placed HERE — after
+ // the reserved tunnel hostname's branch above, which stays byte-for-byte
+ // what it was (that hostname is not a registry route, so it can never
+ // appear in any box's edges file and can never be a hop's target), and
+ // before the route lookup, so that a verified hop naming a host we do not
+ // serve is answered by the stale-map refusal rather than the ordinary
+ // unknown-host answer.
+ //
+ // On a request carrying no X-Carlos-Hop this is one Header.Get and
+ // nothing else.
+ r, hopped := e.acceptHop(r, host)
+
up, ok := e.lookupUpstream(host)
if !ok {
+ if hopped {
+ // A PROVEN hop from a sibling edge, for a host that has moved (or
+ // never lived) here: its map is stale. 503 + Retry-After: 2, and
+ // this return is the structural half of "one hop, ever" — the
+ // backhaul dial below lives further down this same branch, so an
+ // accepted hop physically cannot reach it. (e.backhaul opens with
+ // the same refusal anyway, so moving this branch cannot open a
+ // forwarding loop either — see its doc.)
+ staleHopRefusal(w)
+ return
+ }
+ // Nothing here serves this host — but another box in the deployment
+ // may (backhaul, below). It answers false when no box homes the host,
+ // or when nothing could prove the hop, and today's unknown-host answer
+ // then follows exactly as it always has.
+ if e.backhaul(w, r, host) {
+ return
+ }
e.unknownHost(w, r)
return
}
@@ -1242,10 +1300,11 @@ func (e *Edge) unknownHost(w http.ResponseWriter, r *http.Request) {
// fallback URL instead of a redirect into a doomed TLS handshake (no cert
// will ever exist for them).
//
-// This gate uses lookupUpstream, NOT Allowed: "does something answer this
-// host on 443?" is a ROUTING question (lookupUpstream's job, including its
-// one-label wildcard-alias fallback), not an issuance question (Allowed's
-// job, deliberately exact-only — see its own doc comment). A wildcard-alias
+// This gate uses lookupUpstream (and, since the backhaul, the home map), NOT
+// Allowed: "does something answer this host on 443?" is a ROUTING question
+// (lookupUpstream's job, including its one-label wildcard-alias fallback, plus
+// the deployment's other edges), not an issuance question (Allowed's job,
+// deliberately exact-only — see its own doc comment). A wildcard-alias
// host like paul.woodstar.app resolves fine on 443 via the wildcard cert, so
// plain http:// must redirect it rather than 404. This is safe to widen past
// Allowed's exact match because a redirect grants nothing: it's a 302 to the
@@ -1271,6 +1330,20 @@ func (e *Edge) httpFallbackHandler() http.Handler {
if !routable {
_, routable = e.lookupUpstream(host)
}
+ if !routable {
+ // A host this box does not serve but another box HOMES: 443
+ // backhauls it (the hop, edge.go's backhaul), so :80 must send the
+ // visitor there rather than 404 an app that is up — a first-time
+ // visitor typing a bare hostname lands on whichever edge DNS
+ // picked, which in a steered deployment is routinely not the home.
+ // Home() is the same wildcard-capable lookup the dial uses, is
+ // nil-safe (no deployment store ⇒ no answer), and excludes this
+ // box's own hosts, so a single-box deployment is untouched.
+ //
+ // No hop logic here on purpose: the hop dials 443, so a verified
+ // hop can never arrive on this handler at all.
+ _, _, routable = e.homes.Load().Home(host)
+ }
if !routable {
e.unknownHost(w, r)
return
@@ -1364,7 +1437,7 @@ func unixProxy(socketPath string) *httputil.ReverseProxy {
dialer := &net.Dialer{}
return &httputil.ReverseProxy{
Rewrite: func(pr *httputil.ProxyRequest) {
- pr.SetXForwarded()
+ setForwarded(pr)
pr.Out.URL.Scheme = "http"
pr.Out.URL.Host = "carlos" // placeholder; the unix dialer ignores it
pr.Out.Host = pr.In.Host // backend sees the real hostname
@@ -1389,7 +1462,7 @@ func unixProxy(socketPath string) *httputil.ReverseProxy {
func tcpProxy(addr string) *httputil.ReverseProxy {
return &httputil.ReverseProxy{
Rewrite: func(pr *httputil.ProxyRequest) {
- pr.SetXForwarded()
+ setForwarded(pr)
pr.Out.URL.Scheme = "http"
pr.Out.URL.Host = addr
pr.Out.Host = pr.In.Host // backend sees the real hostname
@@ -1476,7 +1549,7 @@ func multiProxy(host string, p *proberSet) *httputil.ReverseProxy {
dialer := &net.Dialer{Timeout: multiDialTimeout}
return &httputil.ReverseProxy{
Rewrite: func(pr *httputil.ProxyRequest) {
- pr.SetXForwarded()
+ setForwarded(pr)
pr.Out.URL.Scheme = "http"
pr.Out.URL.Host = multiDialHost // placeholder; the dialer below ignores it
pr.Out.Host = pr.In.Host // backend sees the real hostname
@@ -1561,7 +1634,7 @@ func multiProxy(host string, p *proberSet) *httputil.ReverseProxy {
func tunnelProxy(name string, hub *TunnelHub) *httputil.ReverseProxy {
return &httputil.ReverseProxy{
Rewrite: func(pr *httputil.ProxyRequest) {
- pr.SetXForwarded()
+ setForwarded(pr)
pr.Out.URL.Scheme = "https"
pr.Out.URL.Host = pr.In.Host // h2 :authority = the real hostname
pr.Out.Host = pr.In.Host
@@ -1615,7 +1688,7 @@ func fleetChannelFor(fleet, boxLabel string, hub *TunnelHub) string {
func fleetProxy(fleet, boxLabel string, hub *TunnelHub) *httputil.ReverseProxy {
return &httputil.ReverseProxy{
Rewrite: func(pr *httputil.ProxyRequest) {
- pr.SetXForwarded()
+ setForwarded(pr)
pr.Out.URL.Scheme = "https"
pr.Out.URL.Host = pr.In.Host // h2 :authority = the real hostname
pr.Out.Host = pr.In.Host
@@ -1643,6 +1716,255 @@ func fleetProxy(fleet, boxLabel string, hub *TunnelHub) *httputil.ReverseProxy {
}
}
+// ---- backhaul: the SENDING half of one-hop edge-to-edge forwarding ----------
+//
+// homemap.go holds the map (which OTHER box homes a host) and the RECEIVING
+// half (what proves an inbound hop, and what a stale one is answered with).
+// This is the other end: a request that landed on this edge for a host homed
+// elsewhere is forwarded to that box's edge — over ordinary, publicly-verified
+// TLS, exactly once — instead of dead-ending in the unknown-host answer.
+//
+// Three things are deliberately NOT done here, and each is a decision rather
+// than an omission:
+//
+// - NO METERING. The one usage meter (ServeHTTP, above) sits BELOW a
+// successful lookupUpstream, so a backhauled request misses it here by
+// construction while the HOME edge — which does serve the route — records
+// the request and its bytes exactly once. A second Record at this site
+// would double-count every backhauled request against the tenant. What
+// that leaves genuinely open is EGRESS ATTRIBUTION: the bytes cross the
+// internet twice and the second leg is billed to this box's account while
+// the meter runs on the home box's route. That is a unit-economics
+// question — the plan's Task 7 ruling — and not one to guess at from
+// inside a proxy.
+//
+// - NO CACHING AND NO COMPRESSION. This branch returns before ServeHTTP's
+// cache and gzip wrapping, so a backhauled response is neither stored nor
+// re-encoded here. That is load-bearing, not lazy: a tenant's Carlos-Purge
+// is honoured and stripped by the HOME edge's purgeModifier
+// (purge_header.go), so a copy cached on THIS box could never be
+// invalidated by the app that owns it. The home edge's own cache and
+// compression still apply and reach the client through the hop — as does
+// X-Carlos-Version, which the home edge stamps from the route's own row
+// (versionHeader is per-route, and this box has no row for the host), so
+// `carlos deploy`'s verification works through a backhaul with nothing
+// added here.
+//
+// - NO RETRY, AND NO SECOND TARGET. Home() answers with one box or none.
+// A failed dial is a 502 (see backhaulProxy's ErrorHandler), not a
+// scan for somewhere else to try: the process backing a home-boxed route
+// exists on exactly ONE box, so there is no second copy a retry could
+// reach.
+//
+// Single-box inertness: with one platform box Home() never offers a target
+// (own-box exclusion, homemap.go's foldHomes), so e.backhaul returns false on
+// every request and this whole path is dead code until a second box publishes
+// its edges file.
+
+// backhaulPort is the port a home edge answers on. Always 443 in production —
+// an edges file publishes a bare EIP (Task 1) and every edge's front door is
+// 443 — and a var only so a test can point the dial at an httptest server, the
+// same reason homeMapInterval is one.
+var backhaulPort = "443"
+
+// backhaulRootCAs is the trust anchor set the hop's TLS handshake verifies the
+// home edge's certificate against. nil — the system trust store, standard
+// verification — is the production answer and the only one: the home edge
+// presents the SAME publicly-trusted certificate it presents to any browser
+// (Part 2's bucket-distributed certs), so a hop needs no private CA, no
+// pinning, and above all no skipped verification. A var only so a test can
+// trust a fake home edge's self-signed cert.
+//
+// The bootstrap window this implies is stated rather than papered over: a box
+// that has just joined and whose cert for a host has not yet been distributed
+// presents something the dialer will not verify, so the hop fails the
+// handshake and the client gets a 502 until the cert mirror catches up. That
+// is the correct failure — the alternative, dialing without verification,
+// would make the hop's TLS decorative — and it heals on its own within a
+// mirror poll.
+var backhaulRootCAs *x509.CertPool
+
+// The hop's timeouts, per the plan. Deliberately no whole-body deadline: a
+// backhauled response may be a stream or an upgraded WebSocket that is
+// supposed to stay open for hours (the fleetapi client's ruling, same
+// reasoning), so the only bounds are on getting connected and on the home edge
+// beginning to answer.
+const (
+ backhaulDialTimeout = 5 * time.Second
+ backhaulHeaderTimeout = 30 * time.Second
+)
+
+// backhaul forwards a request for a host this box does not serve to the box
+// that does, and reports whether it answered. false means "not mine to
+// forward" and the caller falls through to the unchanged unknown-host answer:
+// either no box homes the host, or nothing could prove the hop.
+//
+// The hopVerified guard is FIRST, before the map is even consulted. ServeHTTP
+// already refuses an arrived hop above this call (that is the structural half
+// of "one hop, ever"), so this line is unreachable today — which is exactly
+// why it is here: if this call is ever moved, or that branch reordered, a hop
+// that arrived from a sibling edge fails closed with the stale-map refusal
+// instead of being forwarded onward. A forwarding loop must take TWO mistakes,
+// not one.
+func (e *Edge) backhaul(w http.ResponseWriter, r *http.Request, host string) bool {
+ if hopVerified(r.Context()) {
+ staleHopRefusal(w)
+ return true
+ }
+ m := e.homes.Load()
+ eip, box, ok := m.Home(host)
+ if !ok {
+ return false
+ }
+ // Sign with the mirror's cached pair, .next preferred (signingKey, mesh.go)
+ // — never a store read on the request path. NO KEY MEANS NO DIAL: signHop
+ // would happily HMAC under an empty key, the home edge would refuse the
+ // result, and the request would have crossed a region to be served as an
+ // ordinary client with its forwarding chain stripped. Falling through to
+ // the unknown-host answer is both cheaper and honest about the deployment
+ // being mid-bootstrap.
+ key := signingKey(m.meshKeys())
+ if len(key) == 0 {
+ e.logThrottled("backhaul-no-mesh-key", fmt.Sprintf("%s is homed on %s but no mesh key is mirrored — not dialling (the request gets the unknown-host answer until the key lands)", host, box))
+ return false
+ }
+ // The MAC is over the CONCRETE request host — never the wildcard map key
+ // that may have matched it — because the home edge verifies against its own
+ // literal request host (hopMAC canonicalizes both, mesh.go).
+ auth := signHop(key, host, time.Now())
+ backhaulProxy(host, eip, box, auth, e.backhaulTransport(eip)).ServeHTTP(w, r)
+ return true
+}
+
+// backhaulTransport is the cached outbound transport for one home box. The
+// TRANSPORT is what is cached and the ReverseProxy above is what is rebuilt
+// per request: a proxy value is two closures, while a transport owns the
+// connection pool, and a cross-region TCP+TLS handshake on every backhauled
+// page is a real, visible cost.
+//
+// Keyed by EIP and NOT by host, which is both the cheaper and the safer half
+// of that trade:
+//
+// - Nothing about the connection is per-host. The dial goes to the box, and
+// the TLS ServerName is derived per request from the outbound URL's
+// authority — backhaulProxy sets it to the real hostname — so one
+// transport serves every host homed on one box while each still gets its
+// own SNI, its own certificate verification, and (the stdlib pools by that
+// same authority) its own pooled connections.
+// - Per host, this map would be unbounded. A catch-all row in another box's
+// edges file homes every subdomain under it, so a stranger asking for a
+// stream of invented hostnames would mint a transport apiece. Per box it
+// is bounded by the deployment, and the pooled connections inside it are
+// bounded by MaxIdleConns and IdleConnTimeout as usual.
+//
+// A host whose home MOVES therefore lands on the new box's transport, with the
+// old box's pooled connections left to idle out — no equivalent of the pinning
+// multiProxy pays DisableKeepAlives to avoid, because the authority a
+// connection is pooled under names the host and the transport it lives in
+// names the box.
+//
+// HTTP/2 is not attempted, and that is on purpose: ForceAttemptHTTP2 stays
+// false (the stdlib disables the automatic upgrade anyway once TLSClientConfig
+// or a custom dialer is set), so the hop speaks HTTP/1.1 and
+// httputil.ReverseProxy's protocol-switch path — the one that carries a
+// WebSocket through — works exactly as it does for every other proxy in this
+// file.
+func (e *Edge) backhaulTransport(eip string) *http.Transport {
+ if tr, ok := e.backhaulTransports.Load(eip); ok {
+ return tr.(*http.Transport)
+ }
+ dialer := &net.Dialer{Timeout: backhaulDialTimeout}
+ tr := &http.Transport{
+ // The address is ignored, multiProxy-style: it is the outbound URL's
+ // authority, which is the HOST (for SNI, verification and pooling) and
+ // never where the bytes go. The home edge is at its EIP.
+ DialContext: func(ctx context.Context, network, _ string) (net.Conn, error) {
+ return dialer.DialContext(ctx, network, net.JoinHostPort(eip, backhaulPort))
+ },
+ // No ServerName: the stdlib fills it from the outbound URL's authority,
+ // which is exactly the host being asked for. RootCAs nil is the system
+ // trust store — standard verification, see backhaulRootCAs.
+ TLSClientConfig: &tls.Config{
+ RootCAs: backhaulRootCAs,
+ MinVersion: tls.VersionTLS12,
+ },
+ TLSHandshakeTimeout: backhaulDialTimeout,
+ ResponseHeaderTimeout: backhaulHeaderTimeout,
+ // PER-HOST as well as in total, which is where this diverges from
+ // unixProxy/tcpProxy's bare MaxIdleConns: their pool key is one constant
+ // localhost authority, so the stdlib default of 2 idle conns per host is
+ // really 2 per BACKEND. Here the key is the hostname, so the default
+ // would cap each backhauled host at 2 — and the third concurrent request
+ // to a busy host would pay the cross-region TCP+TLS handshake this cache
+ // exists to avoid. Equal to MaxIdleConns so the total is the only bound.
+ MaxIdleConns: 32,
+ MaxIdleConnsPerHost: 32,
+ IdleConnTimeout: 90 * time.Second,
+ }
+ // Two requests for a box nobody has dialled yet can both build one; the
+ // loser's transport is simply dropped, having opened nothing and started no
+ // goroutine.
+ if existing, loaded := e.backhaulTransports.LoadOrStore(eip, tr); loaded {
+ return existing.(*http.Transport)
+ }
+ return tr
+}
+
+// backhaulProxy is the hop itself: an ordinary reverse proxy to the home box,
+// carrying the client's Host through unchanged so the home edge routes on the
+// real hostname, and carrying the two hop headers that make it a hop rather
+// than an anonymous request from a stranger with a data-centre IP.
+//
+// The outbound URL names the HOST, not the EIP, and the transport's dialer is
+// what turns that into bytes at the home box (backhaulTransport). The
+// authority is therefore doing three jobs at once and all three want the
+// hostname: the TLS ServerName the home edge selects a certificate by, the
+// name that certificate is verified against, and the key the connection is
+// pooled under.
+//
+// It takes the signature ready-made instead of minting one inside Rewrite, so
+// that the "no key, no dial" decision (backhaul, above) and the signing happen
+// at the SAME read of the mirror's key pair — a Rewrite that signed for itself
+// could still reach for a key that had gone away since the guard.
+//
+// X-Forwarded-For is setForwarded's single-element chain and nothing more: the
+// client that dialled THIS edge, with neither this edge's own address nor the
+// home EIP appended. That is the contract homemap.go's setForwarded documents
+// from the receiving end — the home edge restores this chain verbatim on a
+// verified hop, so the app's last XFF element is the real client, exactly as
+// it would be if the request had landed on the home box directly. (The restore
+// branch cannot fire on THIS side: backhaul only runs when the request did not
+// arrive as a verified hop.)
+func backhaulProxy(host, eip, box, auth string, tr http.RoundTripper) *httputil.ReverseProxy {
+ return &httputil.ReverseProxy{
+ Rewrite: func(pr *httputil.ProxyRequest) {
+ setForwarded(pr)
+ pr.Out.URL.Scheme = "https"
+ pr.Out.URL.Host = host // SNI + verification + pool key; the dial is the transport's
+ pr.Out.Host = pr.In.Host // the home edge routes on the real hostname
+ pr.Out.Header.Set(hopHeader, "1")
+ pr.Out.Header.Set(hopAuthHeader, auth)
+ // The same discipline every other Rewrite here keeps, and needed for
+ // the same reason one hop further along: the home edge scrubs it
+ // again before the app, but a header this edge forwards is a header
+ // this edge vouched for.
+ scrubInboundPurgeHeader(pr)
+ },
+ Transport: tr,
+ ErrorHandler: func(w http.ResponseWriter, r *http.Request, err error) {
+ // 502, not the 503 + Retry-After that multiProxy and fleetProxy
+ // answer with: those two KNOW nobody is home (an empty live set, no
+ // connected channel), while this one believed a specific box was
+ // there and could not reach it. A failed dial, a TLS handshake
+ // against a cert the mirror has not distributed yet, or a home edge
+ // that hung up mid-response are all genuinely bad gateways — and the
+ // cert case heals itself on the next mirror poll.
+ log.Printf("edge: backhaul %s → %s (%s): %v", host, box, eip, err)
+ w.WriteHeader(http.StatusBadGateway)
+ },
+ }
+}
+
// dialUnixWithRetry masks the brief gap during an instance restart: while the
// old process has exited and the new one hasn't rebound the socket, dials fail
// with ENOENT/ECONNREFUSED. Retry those for a short window so no in-flight
@@ -1818,6 +2140,21 @@ func RunWithActivatorReady(args []string, act Activator, cache *Cache, ready fun
cancel()
}
+ // The home map and the mesh keys (homemap.go), armed beside the cert
+ // mirror and on the same terms: nil when there is no deployment store to
+ // mirror from, one synchronous refresh before anything listens so the
+ // first request does not meet an empty map, then the 30s loop. Stored
+ // before either listener binds, so ServeHTTP never sees a half-armed edge.
+ if hm := e.newHomeMap(); hm != nil {
+ ctx, cancel := context.WithTimeout(context.Background(), homeMapBootTimeout)
+ if err := hm.Refresh(ctx); err != nil {
+ log.Printf("edge: first home map refresh failed: %v — hosts homed on another box are unreachable from here until a later poll lands", err)
+ }
+ cancel()
+ e.homes.Store(hm)
+ go hm.run(make(chan struct{})) // process-lifetime; never stopped, like the cert mirror's
+ }
+
m := &autocert.Manager{
Prompt: autocert.AcceptTOS,
Cache: certCache,
diff --git a/internal/edge/homemap.go b/internal/edge/homemap.go
new file mode 100644
index 000000000..dec82865a
--- /dev/null
+++ b/internal/edge/homemap.go
@@ -0,0 +1,602 @@
+package edge
+
+// The home map, and the RECEIVING half of edge-to-edge backhaul (spec
+// docs/superpowers/specs/2026-08-20-multi-region-ingress-design.md, Part 3,
+// Task 3).
+//
+// Two things live here because they are read on the same poll and used in the
+// same breath:
+//
+// - homeMap mirrors control/edges/ — the per-box files internal/host's
+// edgesFilePass publishes (its edgesFile shape, parsed here as
+// edgesFileView) — and folds them into one deployment-wide host→home
+// answer, Home(host). It is the same 30s list-then-get, carry-forward,
+// last-good mirror certsync.go's certMirror is; that shape survived heavy
+// review and is copied rather than reinvented. It NEVER writes: the
+// writer map gives control/edges/<box>.json to that box's own loop, and
+// nothing in this package bumps control/serial.
+//
+// - the mesh keys ride the SAME refresh (loadMeshKeys, mesh.go), so a
+// request path never reads the store: who homes a host and what key
+// proves a hop are answered from one poll's view of the bucket rather
+// than two.
+//
+// acceptHop is where a hop lands on the request path. The trust reasoning it
+// rests on — why the MAC binds the concrete host, why ±hopSkew is the replay
+// bound, and why dual-accept rotation means the request path must call
+// verifyHopEitherKey and never the single-key verifyHop — is written out in
+// mesh.go's file header; read that, this code does not re-derive it. Two
+// consequences of it are worth naming HERE because they are this file's
+// decisions rather than mesh.go's:
+//
+// - The replay envelope is deliberately loose and that is affordable. A
+// captured X-Carlos-Hop-Auth is good for at most 2×hopSkew against ONE
+// host, and what it buys a replayer is a request to a route the edge
+// already serves to the whole internet — no new reachability, no
+// privilege — PLUS, for that one host and that one window, XFF belief:
+// verification gates exactly one thing, whether the inbound
+// X-Forwarded-For is believed, and a valid MAC is what grants that
+// belief. What keeps the loose envelope affordable is not that XFF
+// belief is withheld — it is that capturing a valid Hop-Auth in the
+// first place requires breaking the hop's TLS or holding
+// control/edge-mesh-key outright, either of which already hands an
+// attacker strictly more than "who does this one host think the client
+// was for 2×hopSkew seconds."
+//
+// - XFF is trusted as an inbound chain to CARRY, never one to extend. On a
+// verified hop the chain the sending edge built is passed through
+// verbatim (setForwarded, below), so the app sees exactly what a
+// single-region app sees and the LAST element is still the client. The
+// backhaul leg is deliberately invisible in that header: this
+// deployment's own consumers take the last element as the caller
+// (internal/console/authcli.go's clientIP, and every tenant framework's
+// rightmost-public heuristic), so appending the sibling edge would make
+// every backhauled request look like it came from infrastructure. On
+// anything but a verified hop the chain is discarded outright: an
+// unverified sender is an ordinary client, and an ordinary client has
+// never been allowed to state its own address here.
+//
+// One hop, ever, is STRUCTURAL and not a policy check: a verified hop that
+// names a host this box does not serve leaves through staleHopRefusal (503 +
+// Retry-After: 2) from inside the lookupUpstream-miss branch, BEFORE the point
+// where Task 4 adds the backhaul dial. There is no path from an accepted hop
+// to Home() at all, so a re-forward cannot be introduced by adding a branch —
+// only by deleting one.
+//
+// Single-box inertness: with one platform box the only edges file is this
+// box's own, own-box exclusion drops it, and Home() therefore never offers a
+// target. Nothing signs hops either (Task 4 is the only signer, and it needs a
+// target), so acceptHop's verified branch is unreachable on a single-box
+// deployment — this task ships the receiving half, dormant.
+
+import (
+ "context"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "log"
+ "net/http"
+ "net/http/httputil"
+ "os"
+ "slices"
+ "strings"
+ "sync"
+ "time"
+
+ "github.com/carlosframework/platform/internal/release"
+)
+
+// edgesPrefix is where every box publishes what it homes — one flat object
+// per box, control/edges/<box>.json (internal/host's edgesFileKey).
+const edgesPrefix = "control/edges/"
+
+// The hop's two headers, verbatim from the spec's Part 3. Task 2 left them
+// prose-only; this file is their first consumer, and Task 4's signer imports
+// these constants rather than spelling the strings a second time.
+const (
+ hopHeader = "X-Carlos-Hop"
+ hopAuthHeader = "X-Carlos-Hop-Auth"
+)
+
+// edgeRoleHome is the only role v1 writes, and the only one that routes.
+// Unknown roles (the spec's read-local anticipation) are IGNORED rather than
+// refused: a newer box publishing a role this build has never heard of must
+// not take its whole file — or the poll — down with it.
+const edgeRoleHome = "home"
+
+// homeMapInterval / homeMapTimeout / homeMapBootTimeout mirror the cert
+// mirror's cadence and budgets exactly (certsync.go), for the same reasons: a
+// home move is a placement event that converge heals, the poll runs on its own
+// goroutine and blocks nothing, and the ONE boot refresh must delay the front
+// door by seconds rather than minutes. Vars so tests can shorten them.
+var (
+ homeMapInterval = 30 * time.Second
+ homeMapTimeout = 2 * time.Minute
+ homeMapBootTimeout = 20 * time.Second
+)
+
+// edgesFileView is the READER's half of internal/host's edgesFile wire shape
+// (edgesfile.go). Deliberately a separate declaration rather than a shared
+// type: internal/host imports this package and never the reverse, and the
+// fields this side needs are the stable three (box, eip, hosts) plus the
+// version. Additive growth on the writer's side lands here as fields this
+// decoder ignores, which is the point.
+type edgesFileView struct {
+ V int `json:"v"`
+ Box string `json:"box"`
+ EIP string `json:"eip"`
+ Hosts map[string]edgesHostView `json:"hosts"`
+}
+
+// edgesHostView is the per-host object. An object rather than a bare string is
+// what makes the role field addable without a wire break — see edgeRoleHome.
+type edgesHostView struct {
+ Role string `json:"role"`
+}
+
+// homeEntry is one folded answer: where to dial, and which box that is.
+type homeEntry struct {
+ eip string
+ box string
+}
+
+// homeSnapshot is one poll's result, swapped in whole and immutable once
+// published, so a reader holds no lock past the pointer read.
+type homeSnapshot struct {
+ // files is the parsed per-box file, keyed by its BUCKET key. This is the
+ // carry-forward unit: an object that could not be read (or could not be
+ // parsed) this poll keeps its previous parse, per box, so one bad file
+ // never costs the deployment its whole map.
+ files map[string]edgesFileView
+ // hosts is the fold: host → home, own box already excluded.
+ hosts map[string]homeEntry
+}
+
+var emptyHomeSnapshot = &homeSnapshot{files: map[string]edgesFileView{}, hosts: map[string]homeEntry{}}
+
+type homeMap struct {
+ s release.Store
+ // self is THIS box's label — the same derivation the box's own edges file
+ // is published under (internal/host's metricsBoxLabel), handed down at
+ // edge construction through WithBoxLabel. "" means this process does not
+ // know which box it is, and Home then offers nothing at all: see its doc.
+ self string
+
+ // refreshMu serializes Refresh's whole body, for the reason certMirror's
+ // does — two overlapping polls could otherwise publish out of order, the
+ // slower one clobbering the fresher.
+ refreshMu sync.Mutex
+
+ mu sync.RWMutex
+ snap *homeSnapshot
+ // partialPolls counts polls that could not read (or parse) every object,
+ // for the first-and-every-tenth log cadence.
+ partialPolls int
+
+ // The mirrored mesh keys, read on the same refresh. Replaced only on a
+ // SUCCESSFUL load: a bucket blip leaves the last good key verifying hops,
+ // exactly as a blip leaves the last good host map routing them.
+ keyMu sync.RWMutex
+ keyCurrent []byte
+ keyNext []byte
+}
+
+func newHomeMap(s release.Store, self string) *homeMap {
+ return &homeMap{s: s, self: self, snap: emptyHomeSnapshot}
+}
+
+// newHomeMap is the Edge's map, or nil when there is nothing to mirror FROM —
+// the same no-deployment-store case that leaves the cert mirror nil. A box
+// with no deployment bucket is not part of any mesh: it offers no backhaul
+// target AND verifies no inbound hop (no key to verify against), which is the
+// fail-closed answer either way.
+func (e *Edge) newHomeMap() *homeMap {
+ if e.staticStore == nil {
+ return nil
+ }
+ return newHomeMap(e.staticStore, e.box)
+}
+
+// Refresh rebuilds the host fold and re-reads the mesh keys. The two halves
+// fail INDEPENDENTLY — a bucket that answered the edges list but not the key
+// object should still land the fold — so both run and their errors join.
+// Whatever fails leaves its own last-good in place; the caller logs.
+func (m *homeMap) Refresh(ctx context.Context) error {
+ m.refreshMu.Lock()
+ defer m.refreshMu.Unlock()
+ return errors.Join(m.refreshHosts(ctx), m.refreshKeys(ctx))
+}
+
+// refreshHosts rebuilds the whole map from one list plus a get per key — no
+// delta cleverness, for certMirror.Refresh's reason: a full rebuild is what
+// makes a RETIRED box's hosts actually leave the map.
+//
+// An error leaves the last-good snapshot in place. That matters as much here
+// as it does for certs: an empty map on a bucket blip would stop every
+// backhaul in the deployment at once, turning a blip into a regional outage
+// for every host homed elsewhere.
+func (m *homeMap) refreshHosts(ctx context.Context) error {
+ keys, err := m.s.List(ctx, edgesPrefix)
+ if err != nil {
+ return fmt.Errorf("list %s: %w", edgesPrefix, err)
+ }
+ prev := m.snapshot()
+ next := make(map[string]edgesFileView, len(keys))
+ // failed counts both fault classes; unread counts only the objects the
+ // BUCKET would not hand over, which is the one that says "this is not an
+ // object fault at all" when it covers everything (see below).
+ var tried, failed, unread int
+ var lastErr error
+ for _, key := range keys {
+ // The prefix is flat and every member is <box>.json. Anything else
+ // under it was not written by this deployment's edges loop.
+ name, ok := strings.CutPrefix(key, edgesPrefix)
+ if !ok || !strings.HasSuffix(name, ".json") || strings.Contains(name, "/") {
+ continue
+ }
+ tried++
+ body, _, err := m.s.Get(ctx, key)
+ if errors.Is(err, os.ErrNotExist) {
+ // Raced a delete between the list and the get — the box was
+ // retired mid-poll. Not a fault; it is simply gone.
+ continue
+ }
+ // ONE bad object is not a bad poll, whichever way it went bad.
+ // Aborting would discard every other box's file that landed this tick,
+ // and a permanently unreadable (or permanently corrupt) key would
+ // freeze the map forever. Carry this box's previous parse forward and
+ // keep going.
+ if err != nil {
+ unread++
+ } else if f, perr := parseEdgesFile(key, body); perr != nil {
+ err = perr
+ } else {
+ next[key] = f
+ continue
+ }
+ failed++
+ lastErr = err
+ if old, ok := prev.files[key]; ok {
+ next[key] = old
+ }
+ }
+ if tried > 0 && unread == tried {
+ // Nothing at all could be READ: a bucket fault, not an object fault,
+ // reported like a failed List — last-good stands. Note the bound is
+ // unread and not failed: a deployment whose only edges file is a
+ // corrupt one has a bucket that answered perfectly well, and calling
+ // that a bucket fault would report a blip that is not happening.
+ return fmt.Errorf("get %s*: every one of %d object(s) was unreadable, last: %w", edgesPrefix, tried, lastErr)
+ }
+ if failed > 0 {
+ m.logPartialPoll(failed, tried, lastErr)
+ }
+ snap := &homeSnapshot{files: next, hosts: foldHomes(next, m.self)}
+ m.mu.Lock()
+ m.snap = snap
+ m.mu.Unlock()
+ return nil
+}
+
+// parseEdgesFile decodes one box's file. A parse failure gets the same
+// treatment an unreadable object gets — carry the last good parse forward —
+// because a half-written body is no more usable than a body that never
+// arrived, and the fix for both is the publisher's next pass.
+func parseEdgesFile(key string, body []byte) (edgesFileView, error) {
+ var f edgesFileView
+ if err := json.Unmarshal(body, &f); err != nil {
+ return edgesFileView{}, fmt.Errorf("%s: %w", key, err)
+ }
+ if f.V < 1 {
+ // No version at all: not a file this reader wrote a decoder for. A
+ // HIGHER version is accepted on purpose — the schema grows additively
+ // (the plan's global constraint), so a newer box's file still carries
+ // the box/eip/hosts this reader needs, and refusing it would black out
+ // backhaul to every box that upgraded first.
+ return edgesFileView{}, fmt.Errorf("%s: unversioned edges file", key)
+ }
+ return f, nil
+}
+
+// logPartialPoll reports unreadable or unparseable files on the loop's own
+// first-and-every-tenth cadence: a permanently bad object would otherwise say
+// the same thing every 30 seconds forever.
+func (m *homeMap) logPartialPoll(failed, tried int, err error) {
+ m.mu.Lock()
+ m.partialPolls++
+ n := m.partialPolls
+ m.mu.Unlock()
+ if n == 1 || n%10 == 0 {
+ log.Printf("edge: %d of %d object(s) under %s unreadable this poll (#%d, last: %v) — the previous copy of each is carried forward", failed, tried, edgesPrefix, n, err)
+ }
+}
+
+// foldHomes turns the per-box files into one host→home map.
+//
+// Three exclusions, each of which is a request this box must never make:
+//
+// - OWN BOX. A host homed here is SERVED here; offering it as a backhaul
+// target would have the edge dial itself. Excluded by the file's own box
+// field rather than by its object name, because the field is what the
+// publisher stamps and what self is derived to match.
+// - NO EIP. Task 1 publishes a box with no discoverable public IP as
+// eip:"" so the DR runbook still gets its hosts list. There is nothing to
+// dial, so it is not a target.
+// - A ROLE THIS BUILD DOES NOT KNOW. Only "home" routes. The role field
+// exists so read-local roles can be added later; a reader that treated
+// every role as a home would send traffic to a box that had declared it
+// serves only reads.
+//
+// Iteration is over SORTED file keys so a duplicate claim — two boxes both
+// saying they home one host, which is a placement bug mid-move — resolves the
+// same way on every edge in the deployment. Convergent-but-wrong beats
+// split-and-wrong: the whole fleet aiming at one box is a condition that heals
+// on the next publish, where half the fleet aiming at each is a condition that
+// looks fine from any single box.
+func foldHomes(files map[string]edgesFileView, self string) map[string]homeEntry {
+ keys := make([]string, 0, len(files))
+ for k := range files {
+ keys = append(keys, k)
+ }
+ slices.Sort(keys)
+ hosts := make(map[string]homeEntry)
+ for _, k := range keys {
+ f := files[k]
+ if f.Box == "" || f.Box == self || f.EIP == "" {
+ continue
+ }
+ for host, h := range f.Hosts {
+ if host == "" || h.Role != edgeRoleHome {
+ continue
+ }
+ if _, dup := hosts[host]; dup {
+ continue // first box in sorted order wins, everywhere
+ }
+ hosts[host] = homeEntry{eip: f.EIP, box: f.Box}
+ }
+ }
+ return hosts
+}
+
+func (m *homeMap) snapshot() *homeSnapshot {
+ m.mu.RLock()
+ defer m.mu.RUnlock()
+ return m.snap
+}
+
+// Home answers "which OTHER box homes this host, and where do I dial it?".
+//
+// The lookup is lookupUpstream's, rule for rule, and that is load-bearing
+// rather than tidy: Task 1 publishes registry row hosts VERBATIM, catch-all
+// "*.<parent>" rows included, so an exact match wins and a miss retries with
+// the leading dot of the FIRST label swapped for "*" — one label, never two,
+// and never the parent matching its own wildcard. Deliberately NOT
+// case-folded, for the same reason: a host lookupUpstream would miss on case
+// must miss here too, or this edge would backhaul a request the home edge
+// would then refuse with a 503.
+//
+// A nil map (no deployment store) and a map with no self label both answer
+// "nothing". The self case is fail-closed on purpose: without knowing which
+// box it is, this edge cannot exclude its own hosts, and an edge that cannot
+// exclude itself must not be handing out dial targets at all.
+func (m *homeMap) Home(host string) (eip, box string, ok bool) {
+ if m == nil || m.self == "" {
+ return "", "", false
+ }
+ hosts := m.snapshot().hosts
+ if h, ok := hosts[host]; ok {
+ return h.eip, h.box, true
+ }
+ if i := strings.IndexByte(host, '.'); i > 0 {
+ if h, ok := hosts["*"+host[i:]]; ok {
+ return h.eip, h.box, true
+ }
+ }
+ return "", "", false
+}
+
+// refreshKeys re-reads the mesh key pair. loadMeshKeys is already tolerant of
+// a malformed .next and fail-closed on the current key (mesh.go); this only
+// adds the mirror's own posture on top — the pair is replaced ONLY on a clean
+// read, so a bucket blip leaves the last good key verifying hops.
+//
+// A clean read that finds NOTHING does replace the pair with nil, and that is
+// not a blip: an absent control/edge-mesh-key means no hop can be proven, and
+// refusing every hop (they degrade to ordinary clients) is the only safe
+// reading of "the deployment has no mesh key".
+func (m *homeMap) refreshKeys(ctx context.Context) error {
+ current, next, err := loadMeshKeys(ctx, m.s)
+ if err != nil {
+ return err
+ }
+ m.keyMu.Lock()
+ m.keyCurrent, m.keyNext = current, next
+ m.keyMu.Unlock()
+ return nil
+}
+
+// meshKeys is the request path's cached view of the key pair — never a store
+// read per request. Both may be nil (nothing minted yet, or nothing mirrored
+// yet), which verifyHopEitherKey answers as "no".
+func (m *homeMap) meshKeys() (current, next []byte) {
+ if m == nil {
+ return nil, nil
+ }
+ m.keyMu.RLock()
+ defer m.keyMu.RUnlock()
+ return m.keyCurrent, m.keyNext
+}
+
+// run polls until stop closes — the edge's standing background-loop shape,
+// the same one certMirror.run uses: fixed ticker, errors counted and logged at
+// the first and then every tenth, recovery logged once.
+func (m *homeMap) run(stop <-chan struct{}) {
+ log.Printf("edge: home map armed (%s poll of %s*, this box is %q) — hosts homed on another box are reachable through it", homeMapInterval, edgesPrefix, m.self)
+ tick := time.NewTicker(homeMapInterval)
+ defer tick.Stop()
+ errs := 0
+ for {
+ select {
+ case <-stop:
+ return
+ case <-tick.C:
+ }
+ ctx, cancel := context.WithTimeout(context.Background(), homeMapTimeout)
+ err := m.Refresh(ctx)
+ cancel()
+ if err != nil {
+ errs++
+ if errs == 1 || errs%10 == 0 {
+ log.Printf("edge: home map refresh failed (#%d): %v — serving the last good map and key", errs, err)
+ }
+ continue
+ }
+ if errs > 0 {
+ log.Printf("edge: home map refresh recovered after %d failure(s)", errs)
+ errs = 0
+ }
+ }
+}
+
+// ---- hop acceptance on the request path -------------------------------------
+
+// hopVerifiedKey marks a request that arrived as a PROVEN hop from another
+// edge in this deployment's mesh. A context value rather than a header,
+// because a header is something a client can send: nothing outside this file
+// can put this mark on a request, which is exactly the property the XFF trust
+// below rests on.
+type hopVerifiedKey struct{}
+
+// hopVerified reports whether this request arrived as a verified hop. Task 4's
+// forwarding decision reads it too — a request that ARRIVED as a hop never
+// consults the home map.
+func hopVerified(ctx context.Context) bool {
+ v, _ := ctx.Value(hopVerifiedKey{}).(bool)
+ return v
+}
+
+// acceptHop is the whole of hop handling on the request path, and it runs
+// before the route lookup so that BOTH of its outcomes are decided ahead of
+// anything that serves:
+//
+// no hop header → nothing happens at all (the overwhelming case: one
+// Header.Get on a request that is not a hop).
+// verified hop → the mark goes on the context, the inbound
+// X-Forwarded-For is believed, and the caller serves
+// the route locally or refuses (staleHopRefusal).
+// unverified hop → every forwarded header is stripped and the sender is
+// an ordinary client from here on.
+//
+// The hop headers themselves are deleted in ALL hop cases, verified included:
+// they are the edge's vocabulary between boxes and have no business reaching a
+// tenant's app, and a stripped header cannot be laundered back out by an app
+// that reflects what it was sent (scrubInboundPurgeHeader, purge_header.go,
+// records what that class of bug already cost this package).
+//
+// host is the caller's already-port-stripped request host; canonicalization
+// into the exact bytes the MAC covers is hopMAC's job, in one place, so a
+// signer and a verifier can never disagree (mesh.go).
+func (e *Edge) acceptHop(r *http.Request, host string) (*http.Request, bool) {
+ if r.Header.Get(hopHeader) == "" {
+ // A signature with no marker beside it is not a hop — but it is still
+ // this edge's vocabulary, and the invariant above is "deleted in ALL
+ // hop cases" without an exception for the half a client made up.
+ // scrubInboundPurgeHeader deletes both purge headers unconditionally
+ // for exactly this reason (purge_header.go): an app that reflects what
+ // it was sent turns a client-supplied edge header into an
+ // app-DECLARED one, and no tenant's reflection bug should be able to
+ // put hop vocabulary back on the wire. The Get stays the fast path's
+ // single lookup — the Del only happens on a request that actually
+ // carries the stray header, which is never, in the ordinary case.
+ if r.Header.Get(hopAuthHeader) != "" {
+ r.Header.Del(hopAuthHeader)
+ }
+ return r, false
+ }
+ auth := r.Header.Get(hopAuthHeader)
+ r.Header.Del(hopHeader)
+ r.Header.Del(hopAuthHeader)
+
+ // verifyHopEitherKey, never verifyHop: the single-key form cannot accept a
+ // signer that has already moved to .next, and a rotation is precisely when
+ // refusing hops would hurt. Keys come from the mirror's cache — a hop must
+ // never cost a bucket read.
+ current, next := e.homes.Load().meshKeys()
+ if auth == "" || !verifyHopEitherKey(current, next, host, auth, time.Now()) {
+ stripForwardedHeaders(r)
+ return r, false
+ }
+ return r.WithContext(context.WithValue(r.Context(), hopVerifiedKey{}, true)), true
+}
+
+// stripForwardedHeaders discards every claim a request makes about who sent
+// it. httputil.ReverseProxy's Rewrite path already deletes these from the
+// OUTBOUND request, so for a proxied route this is belt and braces — but the
+// edge serves plenty that is not a ReverseProxy (static trees, the status
+// host, cached replays, the activator's dispatch), and all of them read this
+// same inbound request.
+func stripForwardedHeaders(r *http.Request) {
+ r.Header.Del("Forwarded")
+ r.Header.Del("X-Forwarded-For")
+ r.Header.Del("X-Forwarded-Host")
+ r.Header.Del("X-Forwarded-Proto")
+}
+
+// staleHopRefusal answers a VERIFIED hop that named a host this box does not
+// serve: the sending edge's home map is stale, mid-move. 503 + Retry-After: 2
+// is the same "come back shortly" answer a dark fleet gets (fleetProxy).
+// Retry-After: 2 is a keep-trying hint, not a heal estimate: there is no 2s
+// serial-converge path here (edges never bump control/serial), so what
+// actually heals a stale map is the edges-file cycle — the sending box's own
+// 5min+jitter publish loop (internal/host's edgesFileLoop) followed by this
+// box's 30s mirror poll — which can leave a sender retrying into this refusal
+// for minutes during a home move, not seconds.
+//
+// The body says nothing. It is reachable only by a party holding the mesh key
+// — an unauthenticated stranger asking for the same host gets the ordinary,
+// unchanged unknownHost answer — so it cannot tell an outsider that a host
+// exists somewhere else in the deployment. Keeping it contentless holds that
+// line for the ONE case where it could still leak: a peer whose key is valid
+// but whose map is wrong learns only "not here", never "here is where it
+// went".
+func staleHopRefusal(w http.ResponseWriter) {
+ w.Header().Set("Retry-After", "2")
+ http.Error(w, "not served here", http.StatusServiceUnavailable)
+}
+
+// setForwarded is what every proxy Rewrite in this package calls in place of
+// pr.SetXForwarded(), and it exists for one narrow case.
+//
+// ReverseProxy's Rewrite path deletes Forwarded and X-Forwarded-* from the
+// outbound request before Rewrite runs, so SetXForwarded always writes a
+// single address: the peer that dialled us. That is right for a client and
+// wrong for a verified hop, where the peer that dialled us is a sibling edge
+// and the address the app wants is the one that edge already recorded.
+//
+// So on a verified hop — and only there — SetXForwarded's answer is REPLACED
+// by the inbound chain, verbatim. Not appended to: the last element has to
+// stay the client. This deployment's own consumer says so in as many words
+// (internal/console/authcli.go's clientIP: "the LAST element is the one to
+// take"), and tenant frameworks' rightmost-public heuristics agree — a chain
+// of "<client>, <sending edge EIP>" would rate-limit every backhauled console
+// request as one bucket per region and show tenant apps infrastructure in
+// place of their caller. Passing the chain through unchanged means a
+// backhauled request is indistinguishable from a local one at the app, which
+// is the whole promise of the hop. Multiple inbound X-Forwarded-For lines are
+// joined, which is what list-valued header semantics already say they mean
+// (RFC 9110 §5.3).
+//
+// SetXForwarded still runs first, and is still what sets X-Forwarded-Host and
+// X-Forwarded-Proto: the hop carries the original Host header and is itself
+// TLS, so both are correct as computed here and need no restore. It touches
+// only pr.Out, so reading the inbound chain after it is the same read as
+// before it.
+func setForwarded(pr *httputil.ProxyRequest) {
+ pr.SetXForwarded()
+ if !hopVerified(pr.In.Context()) {
+ return
+ }
+ if chain := pr.In.Header.Values("X-Forwarded-For"); len(chain) > 0 {
+ pr.Out.Header.Set("X-Forwarded-For", strings.Join(chain, ", "))
+ }
+}
diff --git a/internal/edge/homemap_test.go b/internal/edge/homemap_test.go
new file mode 100644
index 000000000..687a85927
--- /dev/null
+++ b/internal/edge/homemap_test.go
@@ -0,0 +1,1041 @@
+package edge
+
+import (
+ "bufio"
+ "context"
+ "crypto/tls"
+ "crypto/x509"
+ "errors"
+ "fmt"
+ "io"
+ "net"
+ "net/http"
+ "net/http/httptest"
+ "os"
+ "strings"
+ "sync"
+ "testing"
+ "time"
+
+ "github.com/carlosframework/platform/internal/registry"
+ "github.com/carlosframework/platform/internal/release"
+)
+
+// putJSON writes a control-plane object body verbatim, replacing whatever is
+// there. release.Store.Put with an empty ifMatch is create-only, so a test
+// that rewrites a key (the last-good and stale-file cases below) has to carry
+// the etag — the same putReplacing shape certsync.go uses in production.
+func putJSON(t *testing.T, s release.Store, key, body string) {
+ t.Helper()
+ _, etag, err := s.Get(context.Background(), key)
+ switch {
+ case errors.Is(err, os.ErrNotExist):
+ etag = ""
+ case err != nil:
+ t.Fatalf("get %s: %v", key, err)
+ }
+ if _, err := s.Put(context.Background(), key, []byte(body), etag); err != nil {
+ t.Fatalf("put %s: %v", key, err)
+ }
+}
+
+func refreshHomeMap(t *testing.T, m *homeMap) {
+ t.Helper()
+ if err := m.Refresh(context.Background()); err != nil {
+ t.Fatalf("refresh: %v", err)
+ }
+}
+
+func TestHomeMapFoldsAndExcludesOwnBox(t *testing.T) {
+ s := meshTestStore(t)
+ putJSON(t, s, "control/edges/annie.json", `{"v":1,"box":"annie","eip":"1.1.1.1","hosts":{"a.test":{"role":"home"}}}`)
+ putJSON(t, s, "control/edges/betty.json", `{"v":1,"box":"betty","eip":"2.2.2.2","hosts":{"b.test":{"role":"home"}}}`)
+ m := newHomeMap(s, "annie")
+ refreshHomeMap(t, m)
+ if eip, box, ok := m.Home("b.test"); !ok || eip != "2.2.2.2" || box != "betty" {
+ t.Fatalf("b.test home = %s/%s/%v", eip, box, ok)
+ }
+ if _, _, ok := m.Home("a.test"); ok {
+ t.Fatal("own-box hosts must not offer a backhaul target")
+ }
+}
+
+// Task 1's writer publishes registry row hosts VERBATIM, catch-all rows
+// included, so the reader owes lookupUpstream's one-label wildcard rule: an
+// exact match wins, then "*.<parent>" — and only ONE label, so a two-label
+// host misses exactly as it does locally.
+func TestHomeMapAppliesOneLabelWildcardRule(t *testing.T) {
+ s := meshTestStore(t)
+ putJSON(t, s, "control/edges/betty.json",
+ `{"v":1,"box":"betty","eip":"2.2.2.2","hosts":{"*.apps.test":{"role":"home"},"exact.apps.test":{"role":"home"}}}`)
+ m := newHomeMap(s, "annie")
+ refreshHomeMap(t, m)
+
+ // Exact form.
+ if eip, _, ok := m.Home("exact.apps.test"); !ok || eip != "2.2.2.2" {
+ t.Fatalf("exact host = %s/%v, want the home edge", eip, ok)
+ }
+ // Wildcard form: one label under the catch-all parent.
+ if eip, box, ok := m.Home("paul.apps.test"); !ok || eip != "2.2.2.2" || box != "betty" {
+ t.Fatalf("one-label wildcard host = %s/%s/%v", eip, box, ok)
+ }
+ // Two labels: a miss here exactly as in lookupUpstream, which composes
+ // "*.b.apps.test" and finds nothing.
+ if _, _, ok := m.Home("a.b.apps.test"); ok {
+ t.Fatal("a two-label host must not match a one-label wildcard")
+ }
+ // The wildcard key itself is not a request host and must not resolve as
+ // one via the fallback ("*.apps.test" composes "*.apps.test" again).
+ if _, _, ok := m.Home("apps.test"); ok {
+ t.Fatal("the catch-all parent must not match its own wildcard")
+ }
+}
+
+// The role object is read-local anticipation (Task 1). Only "home" routes;
+// anything else is a role this build does not implement, and an unknown role
+// must neither route nor fail the poll.
+func TestHomeMapIgnoresUnknownRoles(t *testing.T) {
+ s := meshTestStore(t)
+ putJSON(t, s, "control/edges/betty.json",
+ `{"v":1,"box":"betty","eip":"2.2.2.2","hosts":{"read.test":{"role":"read"},"home.test":{"role":"home"},"blank.test":{}}}`)
+ m := newHomeMap(s, "annie")
+ refreshHomeMap(t, m) // an unknown role is not an error
+ if _, _, ok := m.Home("read.test"); ok {
+ t.Fatal("role:\"read\" is not a home — it must not offer a backhaul target")
+ }
+ if _, _, ok := m.Home("blank.test"); ok {
+ t.Fatal("an absent role is not a home")
+ }
+ if _, _, ok := m.Home("home.test"); !ok {
+ t.Fatal("role:\"home\" must still route alongside roles this build ignores")
+ }
+}
+
+// A box with no discoverable public IP publishes its hosts list with eip:""
+// (Task 1's DR-runbook case). There is nothing to dial, so it must never be
+// offered as a target — a "" EIP would otherwise become a dial to port 443 of
+// nothing.
+func TestHomeMapSkipsBoxesWithNoEIP(t *testing.T) {
+ s := meshTestStore(t)
+ putJSON(t, s, "control/edges/betty.json", `{"v":1,"box":"betty","eip":"","hosts":{"b.test":{"role":"home"}}}`)
+ m := newHomeMap(s, "annie")
+ refreshHomeMap(t, m)
+ if _, _, ok := m.Home("b.test"); ok {
+ t.Fatal("a box with no EIP has no dial address and must not be a backhaul target")
+ }
+}
+
+// Two boxes claiming the same host is a placement bug, but every edge must
+// still pick the SAME one — a coin flip per box would split traffic across a
+// move instead of converging on it.
+func TestHomeMapPicksTheSameHomeOnADuplicateClaim(t *testing.T) {
+ s := meshTestStore(t)
+ putJSON(t, s, "control/edges/betty.json", `{"v":1,"box":"betty","eip":"2.2.2.2","hosts":{"b.test":{"role":"home"}}}`)
+ putJSON(t, s, "control/edges/carol.json", `{"v":1,"box":"carol","eip":"3.3.3.3","hosts":{"b.test":{"role":"home"}}}`)
+ for i := 0; i < 5; i++ {
+ m := newHomeMap(s, "annie")
+ refreshHomeMap(t, m)
+ if _, box, _ := m.Home("b.test"); box != "betty" {
+ t.Fatalf("duplicate claim resolved to %q, want the deterministic first (betty)", box)
+ }
+ }
+}
+
+// A box that is retired (its file deleted) must LEAVE the map — the fold is a
+// full rebuild from each list, never a merge into the previous one.
+func TestHomeMapForgetsADeletedEdgesFile(t *testing.T) {
+ s := meshTestStore(t)
+ putJSON(t, s, "control/edges/betty.json", `{"v":1,"box":"betty","eip":"2.2.2.2","hosts":{"b.test":{"role":"home"}}}`)
+ m := newHomeMap(s, "annie")
+ refreshHomeMap(t, m)
+ if _, _, ok := m.Home("b.test"); !ok {
+ t.Fatal("precondition: b.test should be homed on betty")
+ }
+ if err := s.Delete(context.Background(), "control/edges/betty.json"); err != nil {
+ t.Fatalf("delete: %v", err)
+ }
+ refreshHomeMap(t, m)
+ if _, _, ok := m.Home("b.test"); ok {
+ t.Fatal("a retired box's hosts must leave the map on the next poll")
+ }
+}
+
+// One unreadable object is not a bad poll: the previous copy of that box's
+// file carries forward and every other box's file still lands, exactly as the
+// cert mirror does it.
+func TestHomeMapCarriesForwardOneUnreadableFile(t *testing.T) {
+ inner := meshTestStore(t)
+ s := &oneKeyFailingStore{Store: inner}
+ putJSON(t, inner, "control/edges/betty.json", `{"v":1,"box":"betty","eip":"2.2.2.2","hosts":{"b.test":{"role":"home"}}}`)
+ m := newHomeMap(s, "annie")
+ refreshHomeMap(t, m)
+
+ // betty's file goes unreadable; carol appears in the same poll.
+ putJSON(t, inner, "control/edges/carol.json", `{"v":1,"box":"carol","eip":"3.3.3.3","hosts":{"c.test":{"role":"home"}}}`)
+ s.failKey("control/edges/betty.json")
+ refreshHomeMap(t, m)
+
+ if eip, _, ok := m.Home("b.test"); !ok || eip != "2.2.2.2" {
+ t.Fatalf("b.test = %s/%v — the previous copy of an unreadable file must carry forward", eip, ok)
+ }
+ if _, _, ok := m.Home("c.test"); !ok {
+ t.Fatal("one bad object must not discard the rest of the rebuild")
+ }
+}
+
+// A corrupt body is the same class of fault as an unreadable one: carry the
+// last good parse forward rather than dropping the box off the map.
+func TestHomeMapCarriesForwardACorruptFile(t *testing.T) {
+ s := meshTestStore(t)
+ putJSON(t, s, "control/edges/betty.json", `{"v":1,"box":"betty","eip":"2.2.2.2","hosts":{"b.test":{"role":"home"}}}`)
+ m := newHomeMap(s, "annie")
+ refreshHomeMap(t, m)
+ putJSON(t, s, "control/edges/betty.json", `{"v":1,"box":"betty",`)
+ refreshHomeMap(t, m)
+ if eip, _, ok := m.Home("b.test"); !ok || eip != "2.2.2.2" {
+ t.Fatalf("b.test = %s/%v — a corrupt file must leave the last good parse standing", eip, ok)
+ }
+}
+
+// A failed List is a bucket fault: last-good stands and the error is returned
+// for the loop to log. An empty map here would silently stop every backhaul
+// in the deployment on one blip.
+func TestHomeMapKeepsLastGoodWhenTheBucketIsDown(t *testing.T) {
+ inner := meshTestStore(t)
+ s := &listFailingStore{Store: inner}
+ putJSON(t, inner, "control/edges/betty.json", `{"v":1,"box":"betty","eip":"2.2.2.2","hosts":{"b.test":{"role":"home"}}}`)
+ m := newHomeMap(s, "annie")
+ refreshHomeMap(t, m)
+
+ s.setFail(true)
+ if err := m.Refresh(context.Background()); err == nil {
+ t.Fatal("a failed List must be reported, not swallowed")
+ }
+ if _, _, ok := m.Home("b.test"); !ok {
+ t.Fatal("last-good must stand through a bucket blip")
+ }
+}
+
+// An edge with no box identity of its own cannot exclude itself, so it must
+// never offer a target at all — fail closed rather than backhaul to itself.
+func TestHomeMapWithoutASelfLabelNeverOffersATarget(t *testing.T) {
+ s := meshTestStore(t)
+ putJSON(t, s, "control/edges/betty.json", `{"v":1,"box":"betty","eip":"2.2.2.2","hosts":{"b.test":{"role":"home"}}}`)
+ m := newHomeMap(s, "")
+ refreshHomeMap(t, m)
+ if _, _, ok := m.Home("b.test"); ok {
+ t.Fatal("an edge that does not know its own box label must not backhaul")
+ }
+}
+
+// A nil map (no deployment store, so no mesh at all) answers like an empty
+// one rather than panicking on the request path.
+func TestNilHomeMapIsInert(t *testing.T) {
+ var m *homeMap
+ if _, _, ok := m.Home("b.test"); ok {
+ t.Fatal("a nil home map must offer nothing")
+ }
+ if cur, next := m.meshKeys(); cur != nil || next != nil {
+ t.Fatal("a nil home map holds no mesh keys")
+ }
+}
+
+// ---- hop acceptance ---------------------------------------------------------
+
+// forwardEcho is an upstream that reports back exactly the trust-bearing
+// headers it was handed, so a test can assert both what survived and what was
+// stripped on the way in.
+func forwardEcho(t *testing.T) *httptest.Server {
+ t.Helper()
+ ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ fmt.Fprintf(w, "xff=%s|xfh=%s|hop=%s|auth=%s",
+ r.Header.Get("X-Forwarded-For"), r.Header.Get("X-Forwarded-Host"),
+ r.Header.Get(hopHeader), r.Header.Get(hopAuthHeader))
+ }))
+ t.Cleanup(ts.Close)
+ return ts
+}
+
+// hopEdge builds an edge serving hosts from one echo backend, with a mesh key
+// minted and a home map armed over the same store — the receiving box of a
+// backhaul hop.
+func hopEdge(t *testing.T, self string, hosts ...string) (*Edge, *homeMap, []byte) {
+ t.Helper()
+ s := meshTestStore(t)
+ if err := mintMeshKey(context.Background(), s); err != nil {
+ t.Fatalf("mint: %v", err)
+ }
+ backend := forwardEcho(t)
+ reg := openReg(t)
+ for _, h := range hosts {
+ if err := reg.Upsert(registry.Route{Host: h, Kind: "service", Addr: backend.Listener.Addr().String(), Status: "ready"}); err != nil {
+ t.Fatalf("upsert %s: %v", h, err)
+ }
+ }
+ e, err := New(reg, nil)
+ if err != nil {
+ t.Fatalf("New: %v", err)
+ }
+ t.Cleanup(e.probers.Stop)
+ m := newHomeMap(s, self)
+ refreshHomeMap(t, m)
+ e.homes.Store(m)
+ key, _ := m.meshKeys()
+ if len(key) == 0 {
+ t.Fatal("precondition: the mesh key must be mirrored")
+ }
+ return e, m, key
+}
+
+// hopRequest drives one request at the edge with the given headers.
+func hopRequest(t *testing.T, e *Edge, host string, headers map[string]string) *http.Response {
+ t.Helper()
+ srv := httptest.NewServer(e)
+ t.Cleanup(srv.Close)
+ req, err := http.NewRequest("GET", srv.URL+"/", nil)
+ if err != nil {
+ t.Fatalf("new request: %v", err)
+ }
+ req.Host = host
+ for k, v := range headers {
+ req.Header.Set(k, v)
+ }
+ resp, err := http.DefaultClient.Do(req)
+ if err != nil {
+ t.Fatalf("do: %v", err)
+ }
+ t.Cleanup(func() { resp.Body.Close() })
+ return resp
+}
+
+func readBody(t *testing.T, resp *http.Response) string {
+ t.Helper()
+ b, err := io.ReadAll(resp.Body)
+ if err != nil {
+ t.Fatalf("read body: %v", err)
+ }
+ return string(b)
+}
+
+// A verified hop is served locally, and the chain the sending edge appended
+// is the one thing an ordinary client is never trusted for.
+func TestVerifiedHopIsServedLocallyAndHonoursTheInboundXFF(t *testing.T) {
+ e, _, key := hopEdge(t, "annie", "app.test")
+ resp := hopRequest(t, e, "app.test", map[string]string{
+ hopHeader: "1",
+ hopAuthHeader: signHop(key, "app.test", time.Now()),
+ "X-Forwarded-For": "203.0.113.9",
+ })
+ if resp.StatusCode != http.StatusOK {
+ t.Fatalf("status = %d, want 200", resp.StatusCode)
+ }
+ body := readBody(t, resp)
+ // Verbatim, with NO element appended: the last element must still be the
+ // client. internal/console/authcli.go's clientIP takes the last element as
+ // the caller and every tenant framework's rightmost-public heuristic does
+ // the same, so appending this edge's peer (the sibling edge) would make a
+ // backhauled request look as though infrastructure called it.
+ if !strings.HasPrefix(body, "xff=203.0.113.9|") {
+ t.Fatalf("body = %q — a verified hop's inbound XFF must pass through verbatim, last element still the client", body)
+ }
+ // The hop's own trust headers are the edge's vocabulary, never the app's.
+ if !strings.Contains(body, "|hop=|auth=") {
+ t.Fatalf("body = %q — the hop headers must not reach the app", body)
+ }
+}
+
+// The MAC binds the CONCRETE request host, and a catch-all route serves it
+// through lookupUpstream's wildcard fallback — the two halves of carry C1 on
+// the receiving side.
+func TestVerifiedHopServesAWildcardRoutedHost(t *testing.T) {
+ e, _, key := hopEdge(t, "annie", "*.apps.test")
+ resp := hopRequest(t, e, "paul.apps.test", map[string]string{
+ hopHeader: "1",
+ hopAuthHeader: signHop(key, "paul.apps.test", time.Now()),
+ })
+ if resp.StatusCode != http.StatusOK {
+ t.Fatalf("status = %d, want 200 — a catch-all route serves a hop like any other client", resp.StatusCode)
+ }
+}
+
+// The stale-map refusal: the sender's map named us for a host we do not
+// serve. 503 + Retry-After: 2, and structurally never a re-forward — the home
+// map here is POISONED to name another box for exactly that host, so anything
+// that consulted it would leave through a dial instead.
+func TestVerifiedHopForAnUnservedHostRefusesWithoutForwarding(t *testing.T) {
+ e, m, key := hopEdge(t, "annie", "app.test")
+ putJSON(t, m.s, "control/edges/betty.json",
+ `{"v":1,"box":"betty","eip":"2.2.2.2","hosts":{"moved.test":{"role":"home"}}}`)
+ refreshHomeMap(t, m)
+ if _, _, ok := m.Home("moved.test"); !ok {
+ t.Fatal("precondition: the map must name a remote home for moved.test")
+ }
+
+ resp := hopRequest(t, e, "moved.test", map[string]string{
+ hopHeader: "1",
+ hopAuthHeader: signHop(key, "moved.test", time.Now()),
+ })
+ if resp.StatusCode != http.StatusServiceUnavailable {
+ t.Fatalf("status = %d, want 503 for a verified hop naming a host we do not serve", resp.StatusCode)
+ }
+ if got := resp.Header.Get("Retry-After"); got != "2" {
+ t.Fatalf("Retry-After = %q, want 2", got)
+ }
+ if body := readBody(t, resp); strings.Contains(body, "moved.test") || strings.Contains(body, "betty") {
+ t.Fatalf("body = %q — the refusal must not name the host or where it went", body)
+ }
+}
+
+// Verification failure is not an error path: it is "you are an ordinary
+// client", with every forwarding claim you made discarded.
+func TestUnverifiedHopIsServedAsAnOrdinaryClientWithForwardedHeadersStripped(t *testing.T) {
+ e, _, key := hopEdge(t, "annie", "app.test")
+ tampered := []byte(signHop(key, "app.test", time.Now()))
+ tampered[len(tampered)-1] ^= 'f' ^ '0' // flip the last hex nibble
+
+ resp := hopRequest(t, e, "app.test", map[string]string{
+ hopHeader: "1",
+ hopAuthHeader: string(tampered),
+ "X-Forwarded-For": "203.0.113.9",
+ "X-Forwarded-Host": "evil.test",
+ "X-Forwarded-Proto": "https",
+ })
+ if resp.StatusCode != http.StatusOK {
+ t.Fatalf("status = %d, want 200 — an unverified hop is an ordinary client, not a rejection", resp.StatusCode)
+ }
+ body := readBody(t, resp)
+ if strings.Contains(body, "203.0.113.9") {
+ t.Fatalf("body = %q — an unverified sender's X-Forwarded-For must not be honoured", body)
+ }
+ if strings.Contains(body, "evil.test") {
+ t.Fatalf("body = %q — an unverified sender's X-Forwarded-Host must not be honoured", body)
+ }
+ if !strings.Contains(body, "|hop=|auth=") {
+ t.Fatalf("body = %q — the hop headers must be stripped either way", body)
+ }
+}
+
+// The strip is at the header level, not just a proxy side effect: every route
+// class behind the edge (static trees, the status host, the activator) reads
+// the same request.
+func TestAcceptHopStripsEveryForwardedHeaderOnFailedVerification(t *testing.T) {
+ e, _, _ := hopEdge(t, "annie", "app.test")
+ r := httptest.NewRequest("GET", "http://app.test/", nil)
+ r.Header.Set(hopHeader, "1")
+ r.Header.Set(hopAuthHeader, "0:deadbeef")
+ r.Header.Set("Forwarded", "for=203.0.113.9")
+ r.Header.Set("X-Forwarded-For", "203.0.113.9")
+ r.Header.Set("X-Forwarded-Host", "evil.test")
+ r.Header.Set("X-Forwarded-Proto", "https")
+
+ r, verified := e.acceptHop(r, "app.test")
+ if verified {
+ t.Fatal("a garbage auth header must not verify")
+ }
+ for _, h := range []string{"Forwarded", "X-Forwarded-For", "X-Forwarded-Host", "X-Forwarded-Proto", hopHeader, hopAuthHeader} {
+ if got := r.Header.Get(h); got != "" {
+ t.Fatalf("%s survived an unverified hop as %q", h, got)
+ }
+ }
+ if hopVerified(r.Context()) {
+ t.Fatal("an unverified request must not carry the hop mark")
+ }
+
+ // The half-formed case: a signature with no X-Carlos-Hop beside it. It is
+ // not a hop and must not be treated as one — but the header is still this
+ // edge's vocabulary, and leaving it on the request would let an app that
+ // reflects its request headers launder it back onto the wire (the
+ // scrubInboundPurgeHeader precedent, purge_header.go).
+ stray := httptest.NewRequest("GET", "http://app.test/", nil)
+ stray.Header.Set(hopAuthHeader, "0:deadbeef")
+ stray, verified = e.acceptHop(stray, "app.test")
+ if verified {
+ t.Fatal("an auth header with no hop marker is not a hop")
+ }
+ if got := stray.Header.Get(hopAuthHeader); got != "" {
+ t.Fatalf("%s survived without a hop marker as %q — hop vocabulary must never reach the app", hopAuthHeader, got)
+ }
+}
+
+// The MAC is over the host, so a signature minted for one hop cannot be
+// replayed against another — and the timestamp bounds it in time.
+func TestHopVerificationIsBoundToTheHostAndTheClock(t *testing.T) {
+ e, _, key := hopEdge(t, "annie", "app.test", "other.test")
+
+ wrongHost := hopRequest(t, e, "app.test", map[string]string{
+ hopHeader: "1",
+ hopAuthHeader: signHop(key, "other.test", time.Now()),
+ "X-Forwarded-For": "203.0.113.9",
+ })
+ if body := readBody(t, wrongHost); strings.Contains(body, "203.0.113.9") {
+ t.Fatalf("body = %q — a signature minted for another host must not verify", body)
+ }
+
+ stale := hopRequest(t, e, "app.test", map[string]string{
+ hopHeader: "1",
+ hopAuthHeader: signHop(key, "app.test", time.Now().Add(-2*hopSkew)),
+ "X-Forwarded-For": "203.0.113.9",
+ })
+ if body := readBody(t, stale); strings.Contains(body, "203.0.113.9") {
+ t.Fatalf("body = %q — a signature outside the skew bound must not verify", body)
+ }
+}
+
+// Fail closed: with no mesh key mirrored (a brand new deployment, or a box
+// whose bucket is unreachable at boot) nothing verifies.
+func TestHopWithoutAMirroredMeshKeyIsAnOrdinaryClient(t *testing.T) {
+ e, _, key := hopEdge(t, "annie", "app.test")
+ e.homes.Store(nil) // no mesh at all — the no-deployment-store case
+ resp := hopRequest(t, e, "app.test", map[string]string{
+ hopHeader: "1",
+ hopAuthHeader: signHop(key, "app.test", time.Now()),
+ "X-Forwarded-For": "203.0.113.9",
+ })
+ if resp.StatusCode != http.StatusOK {
+ t.Fatalf("status = %d, want 200", resp.StatusCode)
+ }
+ if body := readBody(t, resp); strings.Contains(body, "203.0.113.9") {
+ t.Fatalf("body = %q — with no key to verify against, the sender is an ordinary client", body)
+ }
+}
+
+// An unserved host that NO box homes keeps today's answer exactly: 404, with
+// no Retry-After, and nothing said about the rest of the deployment.
+//
+// The home map here is deliberately non-empty and deliberately about a
+// DIFFERENT host: the map is consulted (Task 4's backhaul) and misses, which
+// is the only route to this answer now that a homed host is dialled instead.
+// The stale-map 503 stays reachable only by a party holding the mesh key, so
+// the two answers still cannot be told apart by a stranger.
+func TestUnknownHostWithNoHomeIsStillThe404(t *testing.T) {
+ e, m, _ := hopEdge(t, "annie", "app.test")
+ putJSON(t, m.s, "control/edges/betty.json",
+ `{"v":1,"box":"betty","eip":"2.2.2.2","hosts":{"moved.test":{"role":"home"}}}`)
+ refreshHomeMap(t, m)
+ resp := hopRequest(t, e, "nowhere.test", nil)
+ if resp.StatusCode != http.StatusNotFound {
+ t.Fatalf("status = %d, want the unchanged 404 for a host nothing in the deployment homes", resp.StatusCode)
+ }
+ if got := resp.Header.Get("Retry-After"); got != "" {
+ t.Fatalf("Retry-After = %q — a stranger must not be told to come back", got)
+ }
+}
+
+// Regression pin: an ordinary client's forged X-Forwarded-For is no more
+// trusted after this change than before it.
+func TestOrdinaryClientForgedXFFIsStillDiscarded(t *testing.T) {
+ e, _, _ := hopEdge(t, "annie", "app.test")
+ resp := hopRequest(t, e, "app.test", map[string]string{"X-Forwarded-For": "203.0.113.9"})
+ if resp.StatusCode != http.StatusOK {
+ t.Fatalf("status = %d, want 200", resp.StatusCode)
+ }
+ if body := readBody(t, resp); strings.Contains(body, "203.0.113.9") {
+ t.Fatalf("body = %q — a client-supplied XFF is never honoured", body)
+ }
+}
+
+// ---- backhaul dialing (the sending half) ------------------------------------
+
+// homeEdgeSeen is what a fake home edge was handed on the wire.
+type homeEdgeSeen struct {
+ n int
+ sni, host, hop, auth string
+ xff []string
+}
+
+type homeEdgeRecord struct {
+ mu sync.Mutex
+ seen homeEdgeSeen
+}
+
+func (h *homeEdgeRecord) get() homeEdgeSeen {
+ h.mu.Lock()
+ defer h.mu.Unlock()
+ return h.seen
+}
+
+const homeEdgeBody = "served by the home edge"
+
+// fakeHomeEdge stands in for the box that homes a host: a TLS server that
+// records the ClientHello's SNI and the trust-bearing headers of whatever
+// arrives, then runs h (a plain body writer unless the test needs a protocol
+// switch).
+//
+// It also arms the two dial seams for the test's lifetime — backhaulPort, so
+// the dial reaches an ephemeral httptest port instead of 443, and
+// backhaulRootCAs, so this server's self-signed certificate is trusted the way
+// a real home edge's publicly-trusted one is. Nothing about verification is
+// weakened: httptest's certificate names example.com, so every test below asks
+// for exactly that host and a name mismatch would fail the handshake here just
+// as it would in production.
+func fakeHomeEdge(t *testing.T, h http.Handler) (*homeEdgeRecord, string) {
+ t.Helper()
+ rec := &homeEdgeRecord{}
+ if h == nil {
+ h = http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { fmt.Fprint(w, homeEdgeBody) })
+ }
+ ts := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ rec.mu.Lock()
+ rec.seen.n++
+ rec.seen.host = r.Host
+ rec.seen.hop = r.Header.Get(hopHeader)
+ rec.seen.auth = r.Header.Get(hopAuthHeader)
+ rec.seen.xff = r.Header.Values("X-Forwarded-For")
+ rec.mu.Unlock()
+ h.ServeHTTP(w, r)
+ }))
+ ts.TLS = &tls.Config{GetConfigForClient: func(chi *tls.ClientHelloInfo) (*tls.Config, error) {
+ rec.mu.Lock()
+ rec.seen.sni = chi.ServerName
+ rec.mu.Unlock()
+ return nil, nil // the server's own config, unchanged — this hook only watches
+ }}
+ ts.StartTLS()
+ t.Cleanup(ts.Close)
+
+ eip, port, err := net.SplitHostPort(ts.Listener.Addr().String())
+ if err != nil {
+ t.Fatalf("split home edge addr: %v", err)
+ }
+ pool := x509.NewCertPool()
+ pool.AddCert(ts.Certificate())
+ oldPort, oldCAs := backhaulPort, backhaulRootCAs
+ backhaulPort, backhaulRootCAs = port, pool
+ t.Cleanup(func() { backhaulPort, backhaulRootCAs = oldPort, oldCAs })
+ return rec, eip
+}
+
+// homedOn publishes one box's edges file naming hosts as its own, and lands it
+// in the map — the second box a single-box deployment never has.
+func homedOn(t *testing.T, m *homeMap, box, eip string, hosts ...string) {
+ t.Helper()
+ entries := make([]string, 0, len(hosts))
+ for _, h := range hosts {
+ entries = append(entries, fmt.Sprintf("%q:{\"role\":\"home\"}", h))
+ }
+ putJSON(t, m.s, "control/edges/"+box+".json", fmt.Sprintf(
+ `{"v":1,"box":%q,"eip":%q,"hosts":{%s}}`, box, eip, strings.Join(entries, ",")))
+ refreshHomeMap(t, m)
+}
+
+// The whole of the sending half in one request: a host this box does not serve
+// but another box homes reaches that box over real TLS, with the SNI it will
+// need to present a certificate for, the real Host header, a hop signature the
+// home edge can verify, and an X-Forwarded-For that names only the client.
+func TestBackhaulDialsTheHomeEdgeOverVerifiedTLS(t *testing.T) {
+ e, m, key := hopEdge(t, "annie", "local.test")
+ rec, eip := fakeHomeEdge(t, nil)
+ homedOn(t, m, "betty", eip, "example.com")
+
+ resp := hopRequest(t, e, "example.com", nil)
+ if resp.StatusCode != http.StatusOK {
+ t.Fatalf("status = %d, want 200 — a remote-homed host must be served through its home edge", resp.StatusCode)
+ }
+ if body := readBody(t, resp); body != homeEdgeBody {
+ t.Fatalf("body = %q, want %q", body, homeEdgeBody)
+ }
+ seen := rec.get()
+ if seen.n != 1 {
+ t.Fatalf("home edge saw %d request(s), want exactly one — one hop, ever", seen.n)
+ }
+ if seen.sni != "example.com" {
+ t.Fatalf("SNI = %q, want the request host — the home edge answers with the cert for the host being asked for", seen.sni)
+ }
+ if seen.host != "example.com" {
+ t.Fatalf("Host = %q, want the real hostname — the home edge routes on it", seen.host)
+ }
+ if seen.hop != "1" {
+ t.Fatalf("%s = %q, want 1", hopHeader, seen.hop)
+ }
+ if !verifyHopEitherKey(key, nil, "example.com", seen.auth, time.Now()) {
+ t.Fatalf("%s = %q did not verify under the deployment's mesh key", hopAuthHeader, seen.auth)
+ }
+ // One element, the client. Neither this edge's own address nor the home EIP
+ // may be appended: the home edge restores this chain verbatim on a verified
+ // hop, and every consumer takes the LAST element as the caller
+ // (setForwarded, homemap.go).
+ if len(seen.xff) != 1 || strings.Contains(seen.xff[0], ",") {
+ t.Fatalf("X-Forwarded-For = %q, want a single-element chain naming only the client", seen.xff)
+ }
+}
+
+// Precedence, first rule: a host served HERE is served here, whatever the home
+// map says about it. The map is poisoned to name a reachable home for exactly
+// that host, so a lookup in the wrong order would show up as a dial.
+func TestLocalHitIgnoresTheHomeMap(t *testing.T) {
+ e, m, _ := hopEdge(t, "annie", "example.com")
+ rec, eip := fakeHomeEdge(t, nil)
+ homedOn(t, m, "betty", eip, "example.com")
+
+ resp := hopRequest(t, e, "example.com", nil)
+ if resp.StatusCode != http.StatusOK {
+ t.Fatalf("status = %d, want 200", resp.StatusCode)
+ }
+ if body := readBody(t, resp); !strings.HasPrefix(body, "xff=") {
+ t.Fatalf("body = %q, want the LOCAL backend's answer — a local hit must never consult the home map", body)
+ }
+ if n := rec.get().n; n != 0 {
+ t.Fatalf("home edge saw %d request(s), want none", n)
+ }
+}
+
+// One hop, ever: a request that ARRIVED as a verified hop is refused, never
+// forwarded onward, even when the map offers a live target for it. The fake
+// home edge is the zero-call witness — a re-forward would answer 200 here.
+func TestArrivedHopNeverDialsTheHomeEdge(t *testing.T) {
+ e, m, key := hopEdge(t, "annie", "local.test")
+ rec, eip := fakeHomeEdge(t, nil)
+ homedOn(t, m, "betty", eip, "example.com")
+
+ resp := hopRequest(t, e, "example.com", map[string]string{
+ hopHeader: "1",
+ hopAuthHeader: signHop(key, "example.com", time.Now()),
+ })
+ if resp.StatusCode != http.StatusServiceUnavailable {
+ t.Fatalf("status = %d, want 503 — an arrived hop is refused, not re-forwarded", resp.StatusCode)
+ }
+ if got := resp.Header.Get("Retry-After"); got != "2" {
+ t.Fatalf("Retry-After = %q, want 2", got)
+ }
+ if n := rec.get().n; n != 0 {
+ t.Fatalf("home edge saw %d request(s), want none — a hop must never become a second hop", n)
+ }
+}
+
+// The backhaul's own guard, exercised directly rather than through ServeHTTP's
+// branch above it: if this call ever moves, an arrived hop must still fail
+// closed here instead of being forwarded.
+func TestBackhaulHelperRefusesAnArrivedHopOnItsOwn(t *testing.T) {
+ e, m, _ := hopEdge(t, "annie", "local.test")
+ rec, eip := fakeHomeEdge(t, nil)
+ homedOn(t, m, "betty", eip, "example.com")
+
+ r := httptest.NewRequest("GET", "http://example.com/", nil)
+ r = r.WithContext(context.WithValue(r.Context(), hopVerifiedKey{}, true))
+ w := httptest.NewRecorder()
+ if !e.backhaul(w, r, "example.com") {
+ t.Fatal("backhaul must ANSWER an arrived hop, not fall through to the unknown-host path")
+ }
+ if w.Code != http.StatusServiceUnavailable {
+ t.Fatalf("status = %d, want the stale-map refusal", w.Code)
+ }
+ if n := rec.get().n; n != 0 {
+ t.Fatalf("home edge saw %d request(s), want none", n)
+ }
+}
+
+// Fail closed on the SENDING side too: with no mesh key mirrored there is
+// nothing to prove the hop with, and signing under an empty key would cross a
+// region only to be served as an anonymous client. The request gets today's
+// unknown-host answer instead.
+func TestBackhaulRefusesToDialWithoutAMeshKey(t *testing.T) {
+ e, m, _ := hopEdge(t, "annie", "local.test")
+ rec, eip := fakeHomeEdge(t, nil)
+ homedOn(t, m, "betty", eip, "example.com")
+ if err := m.s.Delete(context.Background(), meshKeyKey); err != nil {
+ t.Fatalf("delete mesh key: %v", err)
+ }
+ refreshHomeMap(t, m)
+ if cur, next := m.meshKeys(); len(cur) != 0 || len(next) != 0 {
+ t.Fatal("precondition: the mirror must hold no key")
+ }
+
+ resp := hopRequest(t, e, "example.com", nil)
+ if resp.StatusCode != http.StatusNotFound {
+ t.Fatalf("status = %d, want the unchanged 404 — an unprovable hop is not dialled", resp.StatusCode)
+ }
+ if n := rec.get().n; n != 0 {
+ t.Fatalf("home edge saw %d request(s), want none — never sign with an empty key", n)
+ }
+}
+
+// A stranger forging the hop vocabulary for a remote-homed host is backhauled
+// like any other client, and gains exactly nothing by having tried.
+//
+// Two halves have to hold at once and they meet only here. acceptHop deletes
+// both hop headers and — verification having failed — strips every forwarding
+// claim the forger made, so the request reaching this point is an ordinary
+// one; backhaul then mints a FRESH signature under the deployment's own key,
+// so the home edge sees a hop it can verify and an X-Forwarded-For naming the
+// forger. What the forger must NOT get is the stale-map 503 (which would tell
+// them the host lives elsewhere), a chain of their own choosing, or their
+// garbage signature carried onward under this edge's name.
+func TestForgedHopForARemoteHostIsBackhauledAsAnOrdinaryClient(t *testing.T) {
+ e, m, key := hopEdge(t, "annie", "local.test")
+ rec, eip := fakeHomeEdge(t, nil)
+ homedOn(t, m, "betty", eip, "example.com")
+
+ const garbage = "0:deadbeef"
+ resp := hopRequest(t, e, "example.com", map[string]string{
+ hopHeader: "1",
+ hopAuthHeader: garbage,
+ "X-Forwarded-For": "203.0.113.9",
+ })
+ if resp.StatusCode != http.StatusOK {
+ t.Fatalf("status = %d, want 200 — a forged hop is an ordinary client, not a refusal", resp.StatusCode)
+ }
+ if got := resp.Header.Get("Retry-After"); got != "" {
+ t.Fatalf("Retry-After = %q — the stale-map answer must stay unreachable without the mesh key", got)
+ }
+ if body := readBody(t, resp); body != homeEdgeBody {
+ t.Fatalf("body = %q, want the home edge's answer %q", body, homeEdgeBody)
+ }
+ seen := rec.get()
+ if seen.n != 1 {
+ t.Fatalf("home edge saw %d request(s), want exactly one", seen.n)
+ }
+ if seen.auth == garbage {
+ t.Fatalf("%s = %q — the forger's signature must never be forwarded under this edge's name", hopAuthHeader, seen.auth)
+ }
+ if seen.hop != "1" || !verifyHopEitherKey(key, nil, "example.com", seen.auth, time.Now()) {
+ t.Fatalf("hop=%q auth=%q — the hop must carry a FRESH signature this edge minted", seen.hop, seen.auth)
+ }
+ // The forger's claimed chain is gone and the one that arrives names the
+ // forger's own address, once. Anything else would let a stranger choose
+ // what the app (and the console's rate limiter) believes about who called.
+ if len(seen.xff) != 1 || strings.Contains(seen.xff[0], ",") {
+ t.Fatalf("X-Forwarded-For = %q, want a single-element chain", seen.xff)
+ }
+ if strings.Contains(seen.xff[0], "203.0.113.9") {
+ t.Fatalf("X-Forwarded-For = %q — a forged chain must be discarded, not carried", seen.xff)
+ }
+ if ip := net.ParseIP(seen.xff[0]); ip == nil || !ip.IsLoopback() {
+ t.Fatalf("X-Forwarded-For = %q, want the forger's own (loopback) address", seen.xff)
+ }
+}
+
+// A catch-all row in another box's edges file resolves through the same
+// one-label wildcard rule lookupUpstream uses — but the signature must be over
+// the CONCRETE request host, because that is what the home edge verifies
+// against. Signing the map KEY would produce a MAC for "*.com", which is not a
+// request host under any canonicalization.
+func TestBackhaulSignsTheConcreteHostNotTheWildcardKey(t *testing.T) {
+ e, m, key := hopEdge(t, "annie", "local.test")
+ rec, eip := fakeHomeEdge(t, nil)
+ homedOn(t, m, "betty", eip, "*.com")
+ if _, _, ok := m.Home("example.com"); !ok {
+ t.Fatal("precondition: the wildcard row must offer a home for example.com")
+ }
+
+ if resp := hopRequest(t, e, "example.com", nil); resp.StatusCode != http.StatusOK {
+ t.Fatalf("status = %d, want 200", resp.StatusCode)
+ }
+ seen := rec.get()
+ if !verifyHop(key, "example.com", seen.auth, time.Now()) {
+ t.Fatalf("%s = %q must verify against the concrete request host", hopAuthHeader, seen.auth)
+ }
+ if verifyHop(key, "*.com", seen.auth, time.Now()) {
+ t.Fatalf("%s = %q verifies against the wildcard map key — the MAC must bind the concrete host", hopAuthHeader, seen.auth)
+ }
+}
+
+// A WebSocket must compose through the hop with no code of its own: the dial
+// branch sits above ServeHTTP's cache/compression wrapping, so the client's
+// hijackable writer reaches httputil.ReverseProxy's protocol-switch path, and
+// the transport speaks HTTP/1.1 (no ForceAttemptHTTP2) so that path works.
+// Bytes must then flow both ways with no whole-body deadline cutting them off.
+func TestBackhaulCarriesAWebSocketUpgrade(t *testing.T) {
+ e, m, _ := hopEdge(t, "annie", "local.test")
+ rec, eip := fakeHomeEdge(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ hj, ok := w.(http.Hijacker)
+ if !ok {
+ http.Error(w, "not hijackable", http.StatusInternalServerError)
+ return
+ }
+ conn, buf, err := hj.Hijack()
+ if err != nil {
+ return
+ }
+ defer conn.Close()
+ buf.WriteString("HTTP/1.1 101 Switching Protocols\r\nUpgrade: websocket\r\nConnection: Upgrade\r\n\r\n")
+ if err := buf.Flush(); err != nil {
+ return
+ }
+ line, err := buf.ReadString('\n')
+ if err != nil {
+ return
+ }
+ buf.WriteString("home:" + line)
+ buf.Flush()
+ }))
+ homedOn(t, m, "betty", eip, "example.com")
+
+ srv := httptest.NewServer(e)
+ t.Cleanup(srv.Close)
+ c, err := net.Dial("tcp", strings.TrimPrefix(srv.URL, "http://"))
+ if err != nil {
+ t.Fatalf("dial edge: %v", err)
+ }
+ defer c.Close()
+ if err := c.SetDeadline(time.Now().Add(10 * time.Second)); err != nil {
+ t.Fatalf("set deadline: %v", err)
+ }
+ if _, err := fmt.Fprint(c, "GET /socket HTTP/1.1\r\nHost: example.com\r\nConnection: Upgrade\r\nUpgrade: websocket\r\n\r\n"); err != nil {
+ t.Fatalf("write handshake: %v", err)
+ }
+ br := bufio.NewReader(c)
+ resp, err := http.ReadResponse(br, nil)
+ if err != nil {
+ t.Fatalf("read handshake response: %v", err)
+ }
+ if resp.StatusCode != http.StatusSwitchingProtocols {
+ t.Fatalf("handshake status = %d, want 101 — an upgrade must switch protocols through the hop", resp.StatusCode)
+ }
+ if _, err := fmt.Fprint(c, "ping\n"); err != nil {
+ t.Fatalf("write post-upgrade frame: %v", err)
+ }
+ line, err := br.ReadString('\n')
+ if err != nil {
+ t.Fatalf("read post-upgrade frame: %v", err)
+ }
+ if line != "home:ping\n" {
+ t.Fatalf("post-upgrade echo = %q, want %q — bytes must flow both ways across the backhaul", line, "home:ping\n")
+ }
+ if seen := rec.get(); seen.hop != "1" {
+ t.Fatalf("%s = %q — an upgrade is a hop like any other", hopHeader, seen.hop)
+ }
+}
+
+// A home whose edge cannot be reached is a bad gateway, not a 503: this box
+// believed a specific box was there. The unreachable target here is the
+// documented cert-less bootstrap window's shape too — a dial or handshake that
+// does not complete, healed by the next mirror poll.
+func TestBackhaulToAnUnreachableHomeIsA502(t *testing.T) {
+ e, m, _ := hopEdge(t, "annie", "local.test")
+ // A port nothing listens on, reached through the same seam the fake home
+ // edge uses. 127.0.0.1 refuses immediately rather than hanging for the dial
+ // budget.
+ ln, err := net.Listen("tcp", "127.0.0.1:0")
+ if err != nil {
+ t.Fatalf("listen: %v", err)
+ }
+ eip, port, err := net.SplitHostPort(ln.Addr().String())
+ if err != nil {
+ t.Fatalf("split: %v", err)
+ }
+ ln.Close()
+ old := backhaulPort
+ backhaulPort = port
+ t.Cleanup(func() { backhaulPort = old })
+ homedOn(t, m, "betty", eip, "example.com")
+
+ resp := hopRequest(t, e, "example.com", nil)
+ if resp.StatusCode != http.StatusBadGateway {
+ t.Fatalf("status = %d, want 502 for a home edge that could not be reached", resp.StatusCode)
+ }
+ if got := resp.Header.Get("Retry-After"); got != "" {
+ t.Fatalf("Retry-After = %q — 503 + Retry-After is reserved for knowing nobody is home", got)
+ }
+}
+
+// One transport per home BOX. The pool is what makes a backhaul cost one
+// handshake rather than one per request; keying it per box rather than per
+// host is what keeps a catch-all home — where every invented subdomain is a
+// backhaul target — from growing the map without limit. Per-host identity
+// still exists, one layer down: the outbound authority is the hostname, so SNI
+// and pooled connections are per host inside the box's transport (the SNI
+// assertion in TestBackhaulDialsTheHomeEdgeOverVerifiedTLS is that half).
+func TestBackhaulTransportsAreCachedPerHomeBox(t *testing.T) {
+ e, _, _ := hopEdge(t, "annie", "local.test")
+ a := e.backhaulTransport("1.1.1.1")
+ if b := e.backhaulTransport("1.1.1.1"); b != a {
+ t.Fatal("the same home box must reuse one transport, or every request pays a fresh TLS handshake")
+ }
+ if b := e.backhaulTransport("2.2.2.2"); b == a {
+ t.Fatal("a different home box must get its own transport — the dial address is baked into it")
+ }
+ if sn := a.TLSClientConfig.ServerName; sn != "" {
+ t.Fatalf("ServerName = %q, want it unset — pinning one host per transport is what a wildcard home would multiply", sn)
+ }
+ if a.TLSClientConfig.InsecureSkipVerify {
+ t.Fatal("the hop verifies the home edge's certificate — always")
+ }
+}
+
+// Two hosts homed on one box share that box's transport, and each still
+// reaches the home edge under its OWN name: the shared transport is the memory
+// property, the per-host SNI is the correctness one.
+func TestBackhaulSharesOneTransportAcrossAWildcardHome(t *testing.T) {
+ e, m, _ := hopEdge(t, "annie", "local.test")
+ rec, eip := fakeHomeEdge(t, nil)
+ homedOn(t, m, "betty", eip, "*.com")
+
+ if resp := hopRequest(t, e, "example.com", nil); resp.StatusCode != http.StatusOK {
+ t.Fatalf("status = %d, want 200", resp.StatusCode)
+ }
+ if sni := rec.get().sni; sni != "example.com" {
+ t.Fatalf("SNI = %q, want the concrete request host", sni)
+ }
+ n := 0
+ e.backhaulTransports.Range(func(_, _ any) bool { n++; return true })
+ if n != 1 {
+ t.Fatalf("%d transports cached for one home box, want 1 — a catch-all home must not mint one per hostname", n)
+ }
+ if _, ok := e.backhaulTransports.Load(eip); !ok {
+ t.Fatalf("the cached transport must be keyed by the home box's EIP (%s)", eip)
+ }
+}
+
+// The :80 half of the backhaul, carried from Task 4's review. A host homed on
+// another box is answered on 443 (the hop), so plain http:// must redirect it
+// there like any other routable host — otherwise a first-time visitor typing a
+// bare hostname in the wrong region gets a 404 for an app that is up.
+//
+// The redirect grants nothing that the 443 answer does not: it is a 302 to the
+// SAME host, never an ACME order (httpFallbackHandler's own doc records why
+// this gate is wider than Allowed's). And a verified hop can never arrive
+// here — the sending edge dials 443 — so there is no hop logic on this path.
+func TestHTTPFallbackRedirectsARemoteHomedHost(t *testing.T) {
+ e, m, _ := hopEdge(t, "annie") // serves nothing itself
+ homedOn(t, m, "betty", "2.2.2.2", "remote.test", "*.wild.test")
+ h := e.httpFallbackHandler()
+
+ for _, host := range []string{"remote.test", "paul.wild.test"} {
+ req := httptest.NewRequest(http.MethodGet, "http://"+host+"/deep?q=1", nil)
+ req.Host = host
+ rec := httptest.NewRecorder()
+ h.ServeHTTP(rec, req)
+ if rec.Code != http.StatusFound {
+ t.Fatalf("%s: status = %d, want 302 — the host is homed on another box and 443 backhauls it", host, rec.Code)
+ }
+ if loc := rec.Header().Get("Location"); loc != "https://"+host+"/deep?q=1" {
+ t.Fatalf("%s: Location = %q", host, loc)
+ }
+ }
+
+ // A host nothing in the deployment homes keeps today's answer exactly.
+ req := httptest.NewRequest(http.MethodGet, "http://nowhere.test/", nil)
+ req.Host = "nowhere.test"
+ rec := httptest.NewRecorder()
+ h.ServeHTTP(rec, req)
+ if rec.Code != http.StatusNotFound {
+ t.Fatalf("nowhere.test: status = %d, want the unchanged 404", rec.Code)
+ }
+ if loc := rec.Header().Get("Location"); loc != "" {
+ t.Fatalf("nowhere.test: Location = %q, want none", loc)
+ }
+}
+
+// oneKeyFailingStore fails Get for one named key — a single unreadable object
+// mid-poll, the case the carry-forward exists for.
+type oneKeyFailingStore struct {
+ release.Store
+ mu sync.Mutex
+ bad string
+}
+
+func (o *oneKeyFailingStore) failKey(key string) {
+ o.mu.Lock()
+ o.bad = key
+ o.mu.Unlock()
+}
+
+func (o *oneKeyFailingStore) Get(ctx context.Context, key string) ([]byte, string, error) {
+ o.mu.Lock()
+ bad := o.bad
+ o.mu.Unlock()
+ if key == bad {
+ return nil, "", errors.New("bucket down for this object")
+ }
+ return o.Store.Get(ctx, key)
+}
diff --git a/internal/edge/mesh.go b/internal/edge/mesh.go
new file mode 100644
index 000000000..25d14fe24
--- /dev/null
+++ b/internal/edge/mesh.go
@@ -0,0 +1,303 @@
+package edge
+
+// The mesh key — housekeeper-minted, edge-mirrored, HMAC hop vocabulary
+// (spec docs/superpowers/specs/2026-08-22-multi-region-backhaul-design.md,
+// Part 3, Task 2).
+//
+// A multi-region deployment has more than one edge; Tasks 3/4 have one edge
+// forward a request to another (a "hop") when it does not own the region a
+// request landed in. That hop has to prove it came from a box inside this
+// deployment's own mesh — the writer map's answer is the same shape the
+// housekeeper's other duties already use: exactly ONE box mints a shared
+// secret, every box mirrors it, and no second election is invented.
+//
+// - mintMeshKey is the housekeeper duty (wired as its own loop,
+// internal/host/housekeeper.go's mintMeshKeyLoop — see that func's
+// comment for why it is NOT folded into retentionLoop's tick): 32
+// random bytes, hex-encoded, written to control/edge-mesh-key with an
+// empty ifMatch — release.Store's create-only form. That IS the
+// compare-and-swap: two housekeepers racing the same tick (the
+// transient double-holder window housekeeper.go's poll cadence allows)
+// both attempt the Put, exactly one wins, and the loser's
+// release.ErrCASConflict is not a failure — it is proof a key already
+// exists, which is the only thing this func promises. Every later tick
+// calls it again and finds the same conflict, forever, for the
+// deployment's life: minting is a one-time event, not a duty that ever
+// "finishes" — and it stays SILENT after the first tick (see
+// mintMeshKey's own comment): the steady state of every tick but one,
+// forever, is not something worth a log line, exactly like the
+// housekeeper election's own contention.
+//
+// - signHop / verifyHop are the hop vocabulary Task 3/4 wire onto the
+// request path under the headers X-Carlos-Hop: 1 and X-Carlos-Hop-Auth:
+// <unix>:<hex>. The MAC binds the target host into the signature so one
+// hop's header cannot be replayed against another, and the timestamp
+// bounds replay in time (±hopSkew) independent of the host check. The
+// host is canonicalized (canonicalizeHopHost: lowercased, trailing dot
+// stripped, :port stripped) before it ever reaches the MAC, and the MAC
+// is always over the CONCRETE request host a hop is bound for — never a
+// wildcard map key (e.g. "*.example.com") — because Task 3 verifies
+// against the literal net/http request host and Task 4 signs the same;
+// canonicalizing in one place (hopMAC) is what keeps a signer's and a
+// verifier's idea of "host" from ever diverging on case, a trailing
+// root-label dot, or a stray port.
+//
+// - Rotation's READ side ships in this task, ahead of any code that
+// writes .next: control/edge-mesh-key.next mirrors a rotation
+// successor while one is in flight, loadMeshKeys reads both,
+// verifyHopEitherKey accepts a signature under either, and signingKey
+// is what a signer consults to prefer .next the instant it exists —
+// never waiting for a runbook step.
+//
+// What this removes is the PROMOTION window, not the PROPAGATION one.
+// A signer starts using .next as soon as its OWN box has picked it up;
+// a remote verifier accepts a .next-signed hop only once ITS OWN
+// mirror has too (Task 3/4's poll of the control/ prefix, up to ~30s
+// behind a write). Between those two moments a hop signed under .next
+// CAN be refused by a verifier that hasn't caught up — that is a real,
+// if short, window, and per Task 3's rules the cost of that refusal is
+// the request being served as an ordinary client (its X-Forwarded-For
+// stripped), not dropped: a degraded hop, not an outage. What
+// dual-accept actually buys is that the runbook's promotion step
+// itself (copy .next over control/edge-mesh-key, delete .next) needs
+// no synchronized cutover: because every verifier already accepts
+// EITHER key well before promotion happens, that write can land as an
+// ordinary, un-sequenced action with no box needing to be told "stop
+// signing with the old key" first.
+//
+// Single-box inertness: on a deployment with one platform box, this box is
+// always the housekeeper (housekeeper.go's housekeeperSole), so it mints its
+// own mesh key and nothing else ever reads it — Tasks 3/4 are the first
+// readers. Minting here is therefore inert today, by design: this task ships
+// the vocabulary and the mint, not a consumer.
+import (
+ "context"
+ "crypto/hmac"
+ "crypto/rand"
+ "crypto/sha256"
+ "encoding/hex"
+ "errors"
+ "fmt"
+ "log"
+ "net"
+ "os"
+ "strconv"
+ "strings"
+ "time"
+
+ "github.com/carlosframework/platform/internal/release"
+)
+
+// meshKeyKey is the control-plane object the housekeeper mints once per
+// deployment. Named beside the storedoor ACL tripwire
+// (TestDoorNeverServesCertMaterial), which refuses to serve it — a fleet
+// box authenticating through the store door must never read this key.
+const meshKeyKey = "control/edge-mesh-key"
+
+// meshKeyNextKey is the rotation successor, mirrored alongside meshKeyKey
+// only while a rotation is in flight. Promoting it (copy over meshKeyKey,
+// delete this object) is a runbook step, not code this task ships — see the
+// file header.
+const meshKeyNextKey = meshKeyKey + ".next"
+
+// meshKeyBytes is the random key length before hex encoding: 256 bits,
+// matched to the HMAC-SHA256 hop signature below.
+const meshKeyBytes = 32
+
+// hopSkew bounds how far a hop signature's timestamp may drift from the
+// verifier's clock, in either direction: generous enough for ordinary box
+// clock skew and request latency, tight enough that a captured header stops
+// working well within a minute.
+const hopSkew = 30 * time.Second
+
+// mintMeshKey mints the deployment's mesh key if, and only if, nothing has
+// minted one yet. crypto/rand is required — this key authenticates hops
+// between platform boxes, so it gets the same random source every other
+// credential in this codebase does, never math/rand.
+//
+// A lost race (release.ErrCASConflict — a key already exists, from this
+// box's own earlier tick or another housekeeper that won the same race) is
+// deliberately SILENT, not logged even at "already minted": that outcome is
+// the steady state of every tick but the very first, forever, for the
+// deployment's whole life — one line every tick would be ~264 lines/day of
+// nothing happening, exactly the noise housekeeper.go's own election
+// contention refuses to log for the same reason. Only an actual, novel mint
+// gets a line.
+func mintMeshKey(ctx context.Context, s release.Store) error {
+ buf := make([]byte, meshKeyBytes)
+ if _, err := rand.Read(buf); err != nil {
+ return fmt.Errorf("mesh key: read random bytes: %w", err)
+ }
+ body := []byte(hex.EncodeToString(buf))
+ if _, err := s.Put(ctx, meshKeyKey, body, ""); err != nil {
+ if errors.Is(err, release.ErrCASConflict) {
+ return nil
+ }
+ return fmt.Errorf("mesh key: put: %w", err)
+ }
+ log.Printf("edge: mesh key: minted")
+ return nil
+}
+
+// MintMeshKey is mintMeshKey's exported wrapper — the seam
+// internal/host's housekeeper duty (mintMeshKeyLoop, housekeeper.go) calls
+// through, so the duty lives beside the housekeeper while the key's own
+// shape (its object name, its random-byte discipline) stays edge's to own.
+func MintMeshKey(ctx context.Context, s release.Store) error {
+ return mintMeshKey(ctx, s)
+}
+
+// getMeshKeyHex reads and hex-decodes one mirrored mesh-key object, folding
+// "does not exist" (os.ErrNotExist) into (nil, nil) — the ordinary case for
+// meshKeyNextKey outside a rotation window, and for meshKeyKey itself before
+// the housekeeper's first tick on a brand new deployment.
+func getMeshKeyHex(ctx context.Context, s release.Store, key string) ([]byte, error) {
+ body, _, err := s.Get(ctx, key)
+ if errors.Is(err, os.ErrNotExist) {
+ return nil, nil
+ }
+ if err != nil {
+ return nil, fmt.Errorf("mesh key: get %s: %w", key, err)
+ }
+ dec, err := hex.DecodeString(string(body))
+ if err != nil {
+ return nil, fmt.Errorf("mesh key: %s: not valid hex: %w", key, err)
+ }
+ return dec, nil
+}
+
+// loadMeshKeys is the dual-accept read side's key material access: it reads
+// BOTH control/edge-mesh-key and control/edge-mesh-key.next. current is nil
+// only before the housekeeper has minted anything; next is nil the whole
+// rest of the time — nearly always, since a rotation is a deliberate,
+// short-lived operator action.
+//
+// A decode failure on meshKeyNextKey does NOT propagate: .next is written
+// by hand during a rotation (there is no code in this task that writes it),
+// so one bad upload is a real possibility, and the safe response to "I
+// can't read the rotation successor" is to act as though no rotation is in
+// flight — not to take every hop fleet-wide down by refusing the perfectly
+// good current key alongside it. The failure is loud (an operator mistake
+// worth paging on), just not fatal to this call. A decode failure on
+// meshKeyKey itself DOES propagate: that is the live, day-to-day signing
+// and verifying key, and there is no safe fallback for it being unreadable.
+func loadMeshKeys(ctx context.Context, s release.Store) (current, next []byte, err error) {
+ current, err = getMeshKeyHex(ctx, s, meshKeyKey)
+ if err != nil {
+ return nil, nil, err
+ }
+ next, nextErr := getMeshKeyHex(ctx, s, meshKeyNextKey)
+ if nextErr != nil {
+ log.Printf("edge: mesh key: %s is unreadable, treating as no rotation in flight until fixed: %v", meshKeyNextKey, nextErr)
+ next = nil
+ }
+ return current, next, nil
+}
+
+// signingKey is which of the two mirrored keys a signer should use: .next
+// the instant it exists, else current. A signer never waits for the
+// runbook's promotion step — verifyHopEitherKey's dual-accept on the reader
+// side is what makes that safe, so switching signers over early costs
+// nothing and shortens how long a rotation needs both keys mirrored.
+func signingKey(current, next []byte) []byte {
+ if len(next) > 0 {
+ return next
+ }
+ return current
+}
+
+// signHop produces the value of the X-Carlos-Hop-Auth header: an
+// HMAC-SHA256 over "host\nunix-timestamp", hex-encoded and prefixed with the
+// timestamp so a verifier does not have to guess it. host is bound into the
+// MAC so a signature minted for one hop cannot be replayed against another.
+// host must be the CONCRETE host a hop is bound for (a net/http request
+// host, e.g. from Task 4's forwarding decision) — never a wildcard home-map
+// key like "*.example.com" — because hopMAC canonicalizes it the same way a
+// verifier's literal request host is canonicalized, and a wildcard pattern
+// is not a request host under any canonicalization.
+func signHop(key []byte, host string, now time.Time) string {
+ ts := strconv.FormatInt(now.Unix(), 10)
+ return ts + ":" + hex.EncodeToString(hopMAC(key, host, ts))
+}
+
+// verifyHop checks an X-Carlos-Hop-Auth header against a single key: the
+// timestamp must parse and fall within ±hopSkew of now, and the MAC must
+// match under constant-time comparison (hmac.Equal) — a hop signature is
+// exactly the kind of secret-derived value a timing side channel could leak,
+// so no byte-wise or string-equality shortcut is taken anywhere in this
+// path.
+func verifyHop(key []byte, host, header string, now time.Time) bool {
+ ts, sig, ok := strings.Cut(header, ":")
+ if !ok || ts == "" || sig == "" {
+ return false
+ }
+ sec, err := strconv.ParseInt(ts, 10, 64)
+ if err != nil {
+ return false
+ }
+ skew := now.Sub(time.Unix(sec, 0))
+ if skew < 0 {
+ skew = -skew
+ }
+ if skew > hopSkew {
+ return false
+ }
+ got, err := hex.DecodeString(sig)
+ if err != nil {
+ return false
+ }
+ return hmac.Equal(got, hopMAC(key, host, ts))
+}
+
+// verifyHopEitherKey is the dual-accept read side of a rotation: it accepts
+// a signature made under either mirrored key, current or next, so neither a
+// straggling signer still on the old key nor one that has already switched
+// to .next (signingKey prefers it immediately) is ever refused while both
+// are mirrored. Task 3/4's request path calls this once it has loaded both
+// keys via loadMeshKeys.
+func verifyHopEitherKey(current, next []byte, host, header string, now time.Time) bool {
+ if len(next) > 0 && verifyHop(next, host, header, now) {
+ return true
+ }
+ if len(current) > 0 && verifyHop(current, host, header, now) {
+ return true
+ }
+ return false
+}
+
+// hopMAC is the MAC signHop and verifyHop both compute — one place so the
+// two can never drift out of sync on what bytes are actually signed, and
+// the one choke point that canonicalizes host before it enters the MAC.
+func hopMAC(key []byte, host, ts string) []byte {
+ mac := hmac.New(sha256.New, key)
+ mac.Write([]byte(canonicalizeHopHost(host)))
+ mac.Write([]byte("\n"))
+ mac.Write([]byte(ts))
+ return mac.Sum(nil)
+}
+
+// canonicalizeHopHost normalizes a request host into the exact form the MAC
+// is computed over, so a signer and a verifier can never disagree on what
+// "host" means for the same request:
+//
+// - a :port is stripped via net.SplitHostPort when present — Task 3
+// verifies against a literal net/http request Host, which carries a
+// port when the client set one (or the request came in over a
+// non-default port), while a signer's own idea of the host (e.g. a
+// home-map lookup key) never does;
+// - lowercased — HTTP host headers are case-insensitive, but bytes
+// entering an HMAC are not, so a client or intermediary that
+// capitalizes a label must not be able to produce a request this MAC
+// treats as a different host;
+// - a single trailing root-label dot is stripped — "example.com." and
+// "example.com" name the same host, and a client is free to send
+// either.
+//
+// host is returned unchanged (net.SplitHostPort's error path) when it
+// carries no port — the common case, and not a failure.
+func canonicalizeHopHost(host string) string {
+ if h, _, err := net.SplitHostPort(host); err == nil {
+ host = h
+ }
+ return strings.TrimSuffix(strings.ToLower(host), ".")
+}
diff --git a/internal/edge/mesh_test.go b/internal/edge/mesh_test.go
new file mode 100644
index 000000000..e873dedbf
--- /dev/null
+++ b/internal/edge/mesh_test.go
@@ -0,0 +1,389 @@
+package edge
+
+import (
+ "bytes"
+ "context"
+ "encoding/hex"
+ "log"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/carlosframework/platform/internal/release"
+)
+
+// captureLog redirects the standard logger for the duration of a test —
+// synchronous callers only (mintMeshKey, loadMeshKeys direct calls here are
+// never run concurrently with the read), same shape certsync_test.go uses.
+func captureLog(t *testing.T) *bytes.Buffer {
+ t.Helper()
+ var buf bytes.Buffer
+ prev := log.Writer()
+ log.SetOutput(&buf)
+ t.Cleanup(func() { log.SetOutput(prev) })
+ return &buf
+}
+
+// meshTestStore is a fresh directory-backed store, the same DirStore every
+// other edge test in this package uses in place of live S3.
+func meshTestStore(t *testing.T) release.Store {
+ t.Helper()
+ return release.NewDirStore(t.TempDir())
+}
+
+func TestMintMeshKeyIsCreateOnly(t *testing.T) {
+ s := meshTestStore(t)
+ if err := mintMeshKey(context.Background(), s); err != nil {
+ t.Fatal(err)
+ }
+ first, _, _ := s.Get(context.Background(), meshKeyKey)
+ if err := mintMeshKey(context.Background(), s); err != nil {
+ t.Fatalf("second mint must be success-by-other: %v", err)
+ }
+ second, _, _ := s.Get(context.Background(), meshKeyKey)
+ if !bytes.Equal(first, second) {
+ t.Fatal("second mint replaced the key")
+ }
+}
+
+// The exported wrapper is the seam internal/host's housekeeper duty calls
+// through — prove it actually reaches the same key, not a copy of the logic.
+func TestMintMeshKeyExportedWrapperMintsTheSameKey(t *testing.T) {
+ s := meshTestStore(t)
+ if err := MintMeshKey(context.Background(), s); err != nil {
+ t.Fatal(err)
+ }
+ body, _, err := s.Get(context.Background(), meshKeyKey)
+ if err != nil {
+ t.Fatalf("get %s: %v", meshKeyKey, err)
+ }
+ if len(body) != meshKeyBytes*2 { // hex-encoded
+ t.Fatalf("mesh key body length = %d, want %d (hex of %d random bytes)", len(body), meshKeyBytes*2, meshKeyBytes)
+ }
+}
+
+// A lost race is the steady state of every tick but the first, forever — it
+// must not log anything, matching housekeeper.go's own silence on election
+// contention. Only the first, real mint gets a line.
+func TestMintMeshKeySecondMintIsSilent(t *testing.T) {
+ buf := captureLog(t)
+ s := meshTestStore(t)
+ if err := mintMeshKey(context.Background(), s); err != nil {
+ t.Fatal(err)
+ }
+ first := buf.String()
+ if !strings.Contains(first, "minted") {
+ t.Fatalf("first mint log = %q, want it to mention minting", first)
+ }
+ buf.Reset()
+ if err := mintMeshKey(context.Background(), s); err != nil {
+ t.Fatalf("second mint must be success-by-other: %v", err)
+ }
+ if got := buf.String(); got != "" {
+ t.Fatalf("second mint (a lost CAS race) logged %q, want silence", got)
+ }
+}
+
+func TestHopSignatureRoundTripsAndBoundsSkew(t *testing.T) {
+ key := []byte("0123456789abcdef0123456789abcdef")
+ now := time.Unix(1_766_000_000, 0)
+ h := signHop(key, "app.example.com", now)
+ if !verifyHop(key, "app.example.com", h, now.Add(29*time.Second)) {
+ t.Fatal("within skew must verify")
+ }
+ if verifyHop(key, "app.example.com", h, now.Add(31*time.Second)) {
+ t.Fatal("past skew must refuse")
+ }
+ if verifyHop(key, "other.example.com", h, now) {
+ t.Fatal("host is bound into the mac")
+ }
+ if verifyHop([]byte("wrong-key-wrong-key-wrong-key-00"), "app.example.com", h, now) {
+ t.Fatal("wrong key must refuse")
+ }
+}
+
+// Skew is bounded in both directions — a clock that runs fast must be
+// refused exactly like one that runs slow.
+func TestHopSignatureRefusesFutureSkewToo(t *testing.T) {
+ key := []byte("0123456789abcdef0123456789abcdef")
+ now := time.Unix(1_766_000_000, 0)
+ h := signHop(key, "app.example.com", now)
+ if verifyHop(key, "app.example.com", h, now.Add(-31*time.Second)) {
+ t.Fatal("verifying 31s before the signature's timestamp must refuse")
+ }
+ if !verifyHop(key, "app.example.com", h, now.Add(-29*time.Second)) {
+ t.Fatal("verifying 29s before the signature's timestamp must still verify")
+ }
+}
+
+// A malformed header (no ":", a non-numeric timestamp, non-hex mac) must
+// refuse, never panic — it arrives off the wire from a hop this box does not
+// control.
+func TestVerifyHopRefusesMalformedHeaders(t *testing.T) {
+ key := []byte("0123456789abcdef0123456789abcdef")
+ now := time.Unix(1_766_000_000, 0)
+ for _, header := range []string{
+ "",
+ "nocolon",
+ "notanumber:deadbeef",
+ "1766000000:not-hex-zzzz",
+ "1766000000:",
+ } {
+ if verifyHop(key, "app.example.com", header, now) {
+ t.Fatalf("verifyHop(%q) = true, want false", header)
+ }
+ }
+}
+
+// The MAC is over the CANONICAL host, so a signer's and a verifier's
+// differing-but-equivalent forms of the same host must both verify against
+// the same signature: Task 4 signs one form (e.g. a home-map lookup key,
+// unlikely to carry a port), Task 3 verifies a literal net/http request Host
+// (may carry a port, whatever case and trailing dot the client sent) —
+// divergence here would be spurious refusals presenting as regional
+// flakiness, not a security property either side intends.
+func TestHopMACCanonicalizesHost(t *testing.T) {
+ key := []byte("0123456789abcdef0123456789abcdef")
+ now := time.Unix(1_766_000_000, 0)
+ canonical := signHop(key, "app.example.com", now)
+
+ // A signature made for the canonical host verifies against every
+ // equivalent form a verifier might see the request host in.
+ for _, host := range []string{
+ "app.example.com",
+ "APP.EXAMPLE.COM",
+ "App.Example.Com",
+ "app.example.com.",
+ "app.example.com:443",
+ "app.example.com.:8443",
+ "APP.EXAMPLE.COM.:443",
+ } {
+ if !verifyHop(key, host, canonical, now) {
+ t.Fatalf("verifyHop(%q) against a signature made for the canonical host = false, want true", host)
+ }
+ }
+
+ // And signing under any of those same equivalent forms produces a
+ // signature that verifies against the canonical form too — the
+ // canonicalization is symmetric, not a one-sided allowance.
+ for _, host := range []string{"APP.EXAMPLE.COM", "app.example.com.", "app.example.com:9443"} {
+ h := signHop(key, host, now)
+ if !verifyHop(key, "app.example.com", h, now) {
+ t.Fatalf("a signature made for host %q must verify against the canonical form", host)
+ }
+ }
+
+ // A genuinely different host must still refuse.
+ if verifyHop(key, "other.example.com", canonical, now) {
+ t.Fatal("a different host must still refuse — canonicalization must not collapse distinct hosts")
+ }
+}
+
+// ---- rotation dual-accept (plan's global constraint: the READ side ships
+// now — verifyHopEitherKey and signingKey are what Task 3/4's request path
+// calls once it has loaded both mirrored keys) --------------------------------
+
+func TestSignHopPrefersNextKeyWhenPresent(t *testing.T) {
+ current := []byte("current-key-current-key-current")
+ next := []byte("next-key-next-key-next-key-next")
+ if got := signingKey(current, next); !bytes.Equal(got, next) {
+ t.Fatal("signingKey must prefer .next when it exists")
+ }
+ if got := signingKey(current, nil); !bytes.Equal(got, current) {
+ t.Fatal("signingKey must fall back to current when there is no .next")
+ }
+}
+
+func TestVerifyHopEitherKeyAcceptsSignaturesUnderCurrentOrNext(t *testing.T) {
+ current := []byte("current-key-current-key-current")
+ next := []byte("next-key-next-key-next-key-next-")
+ wrong := []byte("wrong-key-wrong-key-wrong-key-00")
+ now := time.Unix(1_766_000_000, 0)
+ host := "app.example.com"
+
+ // sign with .next -> verifies
+ hNext := signHop(signingKey(current, next), host, now)
+ if !verifyHopEitherKey(current, next, host, hNext, now) {
+ t.Fatal("a signature made with .next must verify while .next exists")
+ }
+
+ // sign with old key while .next exists -> still verifies (dual-accept)
+ hOld := signHop(current, host, now)
+ if !verifyHopEitherKey(current, next, host, hOld, now) {
+ t.Fatal("a signature made with the pre-rotation key must still verify while .next exists")
+ }
+
+ // wrong key -> refuses
+ hWrong := signHop(wrong, host, now)
+ if verifyHopEitherKey(current, next, host, hWrong, now) {
+ t.Fatal("a signature made with neither key must refuse")
+ }
+
+ // no rotation in flight (next is nil): only current verifies.
+ if !verifyHopEitherKey(current, nil, host, hOld, now) {
+ t.Fatal("current key must verify with no rotation in flight")
+ }
+ if verifyHopEitherKey(current, nil, host, hNext, now) {
+ t.Fatal("a .next signature must not verify once .next is gone")
+ }
+}
+
+func TestLoadMeshKeysReadsBothMirroredObjects(t *testing.T) {
+ ctx := context.Background()
+ s := meshTestStore(t)
+
+ // Before the housekeeper's first tick: neither object exists, and that
+ // is quiet — not an error.
+ current, next, err := loadMeshKeys(ctx, s)
+ if err != nil {
+ t.Fatalf("loadMeshKeys before minting: %v", err)
+ }
+ if current != nil || next != nil {
+ t.Fatalf("loadMeshKeys before minting = (%x, %x), want (nil, nil)", current, next)
+ }
+
+ if err := mintMeshKey(ctx, s); err != nil {
+ t.Fatal(err)
+ }
+ current, next, err = loadMeshKeys(ctx, s)
+ if err != nil {
+ t.Fatalf("loadMeshKeys after minting: %v", err)
+ }
+ if len(current) != meshKeyBytes {
+ t.Fatalf("current key length = %d, want %d raw bytes", len(current), meshKeyBytes)
+ }
+ if next != nil {
+ t.Fatal("next key must be nil outside a rotation window")
+ }
+
+ // A rotation in flight: .next mirrored alongside the current key.
+ nextBody := []byte(hex.EncodeToString(bytes.Repeat([]byte{0xab}, meshKeyBytes)))
+ if _, err := s.Put(ctx, meshKeyNextKey, nextBody, ""); err != nil {
+ t.Fatal(err)
+ }
+ current, next, err = loadMeshKeys(ctx, s)
+ if err != nil {
+ t.Fatalf("loadMeshKeys during rotation: %v", err)
+ }
+ if len(current) != meshKeyBytes || len(next) != meshKeyBytes {
+ t.Fatalf("key lengths = (%d, %d), want both %d", len(current), len(next), meshKeyBytes)
+ }
+}
+
+// A malformed .next — hand-written during a rotation, so a bad upload is a
+// real possibility — must not take the live current key down with it: the
+// safe response is to treat it as no rotation in flight (loudly logged),
+// never to refuse every hop fleet-wide over one bad object.
+func TestLoadMeshKeysToleratesAnUnreadableNext(t *testing.T) {
+ ctx := context.Background()
+ s := meshTestStore(t)
+ if err := mintMeshKey(ctx, s); err != nil {
+ t.Fatal(err)
+ }
+ if _, err := s.Put(ctx, meshKeyNextKey, []byte("not-valid-hex-zzzz"), ""); err != nil {
+ t.Fatal(err)
+ }
+
+ buf := captureLog(t)
+ current, next, err := loadMeshKeys(ctx, s)
+ if err != nil {
+ t.Fatalf("loadMeshKeys with a malformed .next must not fail: %v", err)
+ }
+ if len(current) != meshKeyBytes {
+ t.Fatalf("current key length = %d, want %d — a malformed .next took the live key down with it", len(current), meshKeyBytes)
+ }
+ if next != nil {
+ t.Fatal("a malformed .next must be treated as absent (no rotation in flight), not surfaced")
+ }
+ if got := buf.String(); !strings.Contains(got, meshKeyNextKey) {
+ t.Fatalf("log = %q, want a loud mention of %s — this is an operator mistake worth paging on", got, meshKeyNextKey)
+ }
+
+ // And a signature made under the still-good current key verifies fine —
+ // the tolerance is not merely "does not error", it's "hop auth keeps
+ // working".
+ now := time.Unix(1_766_000_000, 0)
+ h := signHop(current, "app.example.com", now)
+ if !verifyHopEitherKey(current, next, "app.example.com", h, now) {
+ t.Fatal("hop auth must keep working off the current key despite the malformed .next")
+ }
+}
+
+// ---- the key half of the home-map mirror ------------------------------------
+//
+// Task 3 folds the mesh-key read into the home map's 30s poll rather than
+// giving it a loop of its own: one list-and-get pass keeps both halves of a
+// hop's answer (who homes this host, and what key proves the hop) from ever
+// being read from two different moments in the bucket's life.
+
+func TestHomeMapRefreshMirrorsTheMeshKeys(t *testing.T) {
+ ctx := context.Background()
+ s := meshTestStore(t)
+ m := newHomeMap(s, "annie")
+
+ // Before the housekeeper's first tick: no key, and every hop fails
+ // closed — refreshing is still not an error.
+ if err := m.Refresh(ctx); err != nil {
+ t.Fatalf("refresh before minting: %v", err)
+ }
+ if cur, next := m.meshKeys(); cur != nil || next != nil {
+ t.Fatalf("keys before minting = (%x, %x), want (nil, nil)", cur, next)
+ }
+
+ if err := mintMeshKey(ctx, s); err != nil {
+ t.Fatal(err)
+ }
+ if err := m.Refresh(ctx); err != nil {
+ t.Fatalf("refresh after minting: %v", err)
+ }
+ cur, next := m.meshKeys()
+ if len(cur) != meshKeyBytes || next != nil {
+ t.Fatalf("keys after minting = (%d bytes, %v), want the current key alone", len(cur), next)
+ }
+
+ // A rotation in flight: dual-accept means a signature under EITHER key is
+ // good, which is what the request path calls (verifyHopEitherKey, never
+ // the single-key verifyHop).
+ nextBody := []byte(hex.EncodeToString(bytes.Repeat([]byte{0xcd}, meshKeyBytes)))
+ if _, err := s.Put(ctx, meshKeyNextKey, nextBody, ""); err != nil {
+ t.Fatal(err)
+ }
+ if err := m.Refresh(ctx); err != nil {
+ t.Fatalf("refresh during rotation: %v", err)
+ }
+ cur, next = m.meshKeys()
+ if len(cur) != meshKeyBytes || len(next) != meshKeyBytes {
+ t.Fatalf("keys during rotation = (%d, %d), want both mirrored", len(cur), len(next))
+ }
+ now := time.Now()
+ for _, k := range [][]byte{cur, next} {
+ if !verifyHopEitherKey(cur, next, "app.example.com", signHop(k, "app.example.com", now), now) {
+ t.Fatal("a hop signed under either mirrored key must verify while both are mirrored")
+ }
+ }
+}
+
+// The keys ride the same last-good posture as the host fold: a bucket blip
+// must not disarm hop verification fleet-wide.
+func TestHomeMapKeepsLastGoodMeshKeysWhenTheBucketIsDown(t *testing.T) {
+ ctx := context.Background()
+ inner := meshTestStore(t)
+ s := &listFailingStore{Store: inner}
+ if err := mintMeshKey(ctx, inner); err != nil {
+ t.Fatal(err)
+ }
+ m := newHomeMap(s, "annie")
+ if err := m.Refresh(ctx); err != nil {
+ t.Fatalf("refresh: %v", err)
+ }
+ good, _ := m.meshKeys()
+
+ s.setFail(true)
+ if err := m.Refresh(ctx); err == nil {
+ t.Fatal("a bucket blip must be reported, not swallowed")
+ }
+ cur, _ := m.meshKeys()
+ if !bytes.Equal(cur, good) {
+ t.Fatal("the last good mesh key must stand through a bucket blip")
+ }
+}
diff --git a/internal/edge/storedoor_test.go b/internal/edge/storedoor_test.go
index 0d3f8d50c..c85458132 100644
--- a/internal/edge/storedoor_test.go
+++ b/internal/edge/storedoor_test.go
@@ -536,10 +536,13 @@ func TestDoorNeverServesCertMaterial(t *testing.T) {
// The real Task 4/5 key shapes (certsync.go's certKey naming: bare host,
// the autocert account key, an HTTP-01 token response, a tls-alpn-01
- // token cert, a wildcard account key) plus the two other control/
+ // token cert, a wildcard account key) plus the three other control/
// entries the multi-region design reserves for the housekeeper alone
- // (bucket-writer-audit.md, "Not yet built" table): the mesh pairing key
- // and the housekeeper's own election lock.
+ // (bucket-writer-audit.md, "Not yet built" table): the mesh key, its
+ // rotation successor (Task 2's dual-accept — control/edge-mesh-key.next,
+ // mirrored only while a rotation is in flight, but readable through this
+ // door at every other time too and refused just as hard), and the
+ // housekeeper's own election lock.
certKeys := []string{
"control/certs/example.com",
"control/certs/acme_account+key",
@@ -547,16 +550,20 @@ func TestDoorNeverServesCertMaterial(t *testing.T) {
"control/certs/tok+http-01",
"control/certs/example.com+token",
"control/edge-mesh-key",
+ "control/edge-mesh-key.next",
"control/housekeeper-lease/lock.json",
}
- // Seed two of the seven with plausible-looking secret bytes: if the ACL
+ // Seed three of the eight with plausible-looking secret bytes: if the ACL
// ever let a Get through, these are the ones that would leak real
// material instead of a "not found" — and mixing seeded with unseeded
// keys through the identical-shape check below is what proves the door
// isn't just refusing absent keys, it's refusing to distinguish "exists"
- // from "doesn't" at all for this whole prefix.
+ // from "doesn't" at all for this whole prefix. The mesh key and its
+ // rotation successor are seeded as a pair, same secret shape, since a
+ // leak of either is the same failure.
seedObject(t, store, "control/certs/example.com", "-----BEGIN CERTIFICATE-----\nfake\n-----END CERTIFICATE-----")
seedObject(t, store, "control/edge-mesh-key", "fake-mesh-shared-secret")
+ seedObject(t, store, "control/edge-mesh-key.next", "fake-mesh-shared-secret")
type shape struct {
code int
diff --git a/internal/host/agent.go b/internal/host/agent.go
index 59ac7428a..8a609ae33 100644
--- a/internal/host/agent.go
+++ b/internal/host/agent.go
@@ -321,6 +321,14 @@ func RunAgent(args []string) {
stopHousekeeper := startHousekeeper()
go retentionLoop(reg)
go registrySnapshotLoop(reg)
+ go edgesFileLoop(reg)
+ // The mesh key's own loop (multi-region backhaul spec, Part 3 Task 2) —
+ // deliberately NOT a duty riding retentionLoop's tick: that loop returns
+ // early, never arming at all, on a box missing CARLOS_RELEASE_PUBKEY,
+ // and such a box still WINS the housekeeper election. A mint duty living
+ // only inside retentionLoop would then silently never run fleet-wide.
+ // See mintMeshKeyLoop's own comment (housekeeper.go).
+ go mintMeshKeyLoop()
// The sweep reads the registry only through this closure — the stamped
// bespoke units to sample beside the carlos- ones, re-read each tick.
go metricsLoop(cgm.Root(), func() []string { return stampedUnitNames(reg) })
@@ -353,7 +361,14 @@ func RunAgent(args []string) {
// for seconds to minutes, and the context ends the instant this box
// stops being the housekeeper. Armed above, so this closure is live
// and correct by the time the edge reads it.
- edge.WithIssuerGate(housekeeperGate))
+ edge.WithIssuerGate(housekeeperGate),
+ // This box's own label, for the home map's own-box exclusion
+ // (internal/edge/homemap.go). metricsBoxLabel is the SAME call
+ // edgesFileLoop publishes under, deliberately: the map excludes our
+ // hosts by matching the box field of the file we ourselves wrote, so
+ // the two derivations must be one derivation. Without it the edge
+ // would offer nothing (fail-closed), never dial itself.
+ edge.WithBoxLabel(metricsBoxLabel(os.Getenv, readMachineID())))
// The edge has drained; take the children down too (they're this process's
// children — orphaning them would strand their leases until TTL expiry).
diff --git a/internal/host/edgesfile.go b/internal/host/edgesfile.go
new file mode 100644
index 000000000..4fa38a39d
--- /dev/null
+++ b/internal/host/edgesfile.go
@@ -0,0 +1,153 @@
+package host
+
+import (
+ "bytes"
+ "context"
+ "encoding/json"
+ "fmt"
+ "log"
+ "math/rand/v2"
+ "os"
+ "time"
+
+ "github.com/carlosframework/platform/internal/registry"
+ "github.com/carlosframework/platform/internal/release"
+)
+
+// The multi-region ingress spec (2026-08-20, Part 3, closes the design half
+// of #238): every platform box publishes what it homes so any edge can fold
+// the fleet's per-box files into a deployment-wide host→home map. v1 writes
+// only `role: "home"` — the anticipated read-local role values are Task 3's
+// concern, not this file's.
+func edgesFileKey(box string) string { return "control/edges/" + box + ".json" }
+
+// edgeHost is the per-host entry in edgesFile.Hosts. An object, not a bare
+// string, is deliberately load-bearing: it is read-local anticipation —
+// Task 3's mirror and later roles (e.g. a "read" role) need somewhere to
+// grow into without a wire-shape break.
+type edgeHost struct {
+ Role string `json:"role"`
+}
+
+// edgesFile is the wire shape at edgesFileKey(box), schema v1 (spec Part 3):
+//
+// {"v":1,"box":"annie-welding","eip":"99.81.104.219","generation":"2026-08-22T09:00:00Z","hosts":{"app.example.com":{"role":"home"}}}
+type edgesFile struct {
+ V int `json:"v"`
+ Box string `json:"box"`
+ EIP string `json:"eip"`
+ Generation string `json:"generation"`
+ Hosts map[string]edgeHost `json:"hosts"`
+}
+
+// edgesFilePass builds this box's edges file from the routes currently in
+// reg (every SERVED row on this box, ALL kinds — the serving-class split
+// belongs to the reader, Task 3, not to this writer) and publishes it at
+// edgesFileKey(box) when it changed.
+//
+// The change gate is the marshaled body WITH Generation cleared: Generation
+// is a fresh RFC3339 stamp every pass, so comparing the full body would
+// write on every tick even when the box's homed hosts and EIP are
+// identical — a timestamp-only refresh is not a change worth a bucket
+// write. *last holds the previous pass's gate body (also Generation-
+// cleared) and is updated only after a successful write, mirroring
+// registrySnapshotPass's *lastFP contract.
+//
+// eip may be "" (readPublicIP's "unknown" case, e.g. not on EC2, IMDS
+// disabled, or a transient metadata-endpoint failure). A blank reading is
+// NOT taken at face value: this box's public IP does not change tick to
+// tick, so "" almost always means the probe hiccuped, not that the address
+// vanished — publishing it verbatim would blank a known-good EIP for
+// every remote edge's dial address for up to one loop cadence. When eip is
+// "" and *last already holds a published file, this pass reuses that
+// file's EIP instead. "" only ever reaches the wire for a box that has
+// NEVER had a discoverable IP — the DR runbook's hosts list still gets
+// published in that case, just without a dial address.
+func edgesFilePass(ctx context.Context, s release.Store, reg *registry.Registry, box, eip string, last *[]byte) error {
+ routes, err := reg.List()
+ if err != nil {
+ return fmt.Errorf("edges: list: %w", err)
+ }
+ hosts := make(map[string]edgeHost, len(routes))
+ for _, r := range routes {
+ // edge.Reload (internal/edge/edge.go) is the deployment's own
+ // definition of "serves": it skips any row whose Status != "ready"
+ // because a pending/creating/failed row has no working backend and
+ // is never dialed by any edge, home or remote. Mirroring that gate
+ // here means this file never claims role:"home" for a host nothing
+ // actually serves. Hibernating routes stay Status "ready" (that is
+ // what makes wake-on-request possible), so nothing reachable is
+ // excluded by this filter.
+ if r.Status != "ready" {
+ continue
+ }
+ // Keys are registry row Host values verbatim, INCLUDING *.parent
+ // catch-all wildcard rows (Task 3's reconcile pass derives these
+ // from a claim's catch-all row, same as edge.go's upstream map) —
+ // this writer performs no expansion. A reader that needs exact-host
+ // answers must apply edge.go's lookupUpstream one-label wildcard
+ // rule itself: an exact match wins, and on a miss "*.<parent>"
+ // (the leading dot of the first label only) is tried next.
+ hosts[r.Host] = edgeHost{Role: "home"}
+ }
+
+ if eip == "" && *last != nil {
+ var prior edgesFile
+ if err := json.Unmarshal(*last, &prior); err == nil {
+ eip = prior.EIP
+ }
+ }
+
+ f := edgesFile{V: 1, Box: box, EIP: eip, Hosts: hosts}
+ gate, err := json.Marshal(f)
+ if err != nil {
+ return fmt.Errorf("edges: marshal: %w", err)
+ }
+ if bytes.Equal(gate, *last) {
+ return nil
+ }
+
+ f.Generation = time.Now().UTC().Format(time.RFC3339)
+ body, err := json.Marshal(f)
+ if err != nil {
+ return fmt.Errorf("edges: marshal: %w", err)
+ }
+ if err := putReplacing(ctx, s, edgesFileKey(box), body); err != nil {
+ return err
+ }
+ *last = gate
+ return nil
+}
+
+// edgesFileLoop publishes this box's edges file every 5 minutes + jitter —
+// the same cadence and arming site as registrySnapshotLoop (home moves are
+// placement events, not per-request events; the backhaul hop's stale-map
+// 503 + converge heals faster than this loop could anyway). Platform-box
+// job, retentionLoop's rules: skipped on fleet boxes, disabled with a log
+// line when there is no deployment bucket. Single-box deployments: the file
+// is published and nothing reads it yet — inert until Task 3's mirror
+// ships.
+func edgesFileLoop(reg *registry.Registry) {
+ ctx := context.Background()
+ if shouldSkipPublisherElection(os.Getenv) {
+ log.Printf("agent: edges file loop is a platform-box job — skipped on a fleet box")
+ return
+ }
+ store, err := openDeployStore(ctx)
+ if err != nil {
+ log.Printf("agent: edges file loop disabled: %v", err)
+ return
+ }
+ box := metricsBoxLabel(os.Getenv, readMachineID())
+ log.Printf("agent: edges file loop armed (5min + jitter, %s)", edgesFileKey(box))
+ var last []byte
+ delay := time.Minute
+ for {
+ time.Sleep(delay)
+ delay = 5*time.Minute + rand.N(time.Minute)
+ eip := readPublicIP()
+ if err := edgesFilePass(ctx, store, reg, box, eip, &last); err != nil {
+ log.Printf("agent: edges file pass: %v", err)
+ }
+ }
+}
diff --git a/internal/host/edgesfile_test.go b/internal/host/edgesfile_test.go
new file mode 100644
index 000000000..162db2870
--- /dev/null
+++ b/internal/host/edgesfile_test.go
@@ -0,0 +1,270 @@
+package host
+
+import (
+ "context"
+ "encoding/json"
+ "path/filepath"
+ "testing"
+ "time"
+
+ "github.com/carlosframework/platform/internal/registry"
+ "github.com/carlosframework/platform/internal/release"
+)
+
+func edgesTestRegistry(t *testing.T) *registry.Registry {
+ t.Helper()
+ reg, err := registry.Open(filepath.Join(t.TempDir(), "registry.db"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() { reg.Close() })
+ return reg
+}
+
+// seedRoute upserts a minimal ready route under the given host — any served
+// kind does for these tests, since edgesFilePass folds every kind in
+// (the serving-class split is the reader's business, Task 3).
+func seedRoute(t *testing.T, reg *registry.Registry, host string) {
+ t.Helper()
+ if err := reg.Upsert(registry.Route{Host: host, Kind: "service", Addr: "10.0.0.1:3000", Status: "ready"}); err != nil {
+ t.Fatal(err)
+ }
+}
+
+// putCountingStore wraps a release.Store and counts Put calls — the same
+// wrap-and-count idiom agent_test.go's countingStore uses for Get, applied
+// to Put so the change-gate test can pin "second identical pass writes
+// nothing" without inspecting store internals.
+type putCountingStore struct {
+ release.Store
+ puts int
+}
+
+func (p *putCountingStore) Put(ctx context.Context, key string, body []byte, ifMatch string) (string, error) {
+ p.puts++
+ return p.Store.Put(ctx, key, body, ifMatch)
+}
+
+func TestEdgesFilePublishesHomedHosts(t *testing.T) {
+ s, reg := release.NewDirStore(t.TempDir()), edgesTestRegistry(t)
+ seedRoute(t, reg, "app.example.com")
+ var lastBody []byte
+ if err := edgesFilePass(context.Background(), s, reg, "annie", "1.2.3.4", &lastBody); err != nil {
+ t.Fatal(err)
+ }
+ got, _, err := s.Get(context.Background(), "control/edges/annie.json")
+ if err != nil {
+ t.Fatal(err)
+ }
+ var f edgesFile
+ if err := json.Unmarshal(got, &f); err != nil {
+ t.Fatal(err)
+ }
+ if f.V != 1 || f.Box != "annie" || f.EIP != "1.2.3.4" {
+ t.Fatalf("file = %+v", f)
+ }
+ if r, ok := f.Hosts["app.example.com"]; !ok || r.Role != "home" {
+ t.Fatalf("hosts = %+v", f.Hosts)
+ }
+ if f.Generation == "" {
+ t.Fatal("generation must be stamped")
+ }
+ if got, want := edgesFileKey("annie"), "control/edges/annie.json"; got != want {
+ t.Fatalf("edgesFileKey = %q, want %q", got, want)
+ }
+}
+
+// A box with no discoverable public IP still publishes the file, with
+// "eip":"" — the DR runbook's hosts list must not depend on IMDS working.
+func TestEdgesFilePublishesWithEmptyEIP(t *testing.T) {
+ s, reg := release.NewDirStore(t.TempDir()), edgesTestRegistry(t)
+ seedRoute(t, reg, "app.example.com")
+ var lastBody []byte
+ if err := edgesFilePass(context.Background(), s, reg, "annie", "", &lastBody); err != nil {
+ t.Fatal(err)
+ }
+ got, _, err := s.Get(context.Background(), "control/edges/annie.json")
+ if err != nil {
+ t.Fatal(err)
+ }
+ var f edgesFile
+ if err := json.Unmarshal(got, &f); err != nil {
+ t.Fatal(err)
+ }
+ if f.EIP != "" {
+ t.Fatalf("eip = %q, want empty", f.EIP)
+ }
+ if _, ok := f.Hosts["app.example.com"]; !ok {
+ t.Fatalf("hosts = %+v, want app.example.com present despite empty eip", f.Hosts)
+ }
+}
+
+func TestEdgesFileSkipsWriteWhenUnchanged(t *testing.T) {
+ real := release.NewDirStore(t.TempDir())
+ s := &putCountingStore{Store: real}
+ reg := edgesTestRegistry(t)
+ seedRoute(t, reg, "app.example.com")
+
+ var lastBody []byte
+ if err := edgesFilePass(context.Background(), s, reg, "annie", "1.2.3.4", &lastBody); err != nil {
+ t.Fatal(err)
+ }
+ if s.puts != 1 {
+ t.Fatalf("first pass puts = %d, want 1", s.puts)
+ }
+
+ // Second pass, identical registry state: a fresh generation timestamp
+ // would differ from the first pass's, but the change-gate excludes
+ // Generation, so this must not write.
+ if err := edgesFilePass(context.Background(), s, reg, "annie", "1.2.3.4", &lastBody); err != nil {
+ t.Fatal(err)
+ }
+ if s.puts != 1 {
+ t.Fatalf("unchanged second pass puts = %d, want still 1 (no write)", s.puts)
+ }
+
+ // A registry change makes the next pass write again.
+ seedRoute(t, reg, "second.example.com")
+ if err := edgesFilePass(context.Background(), s, reg, "annie", "1.2.3.4", &lastBody); err != nil {
+ t.Fatal(err)
+ }
+ if s.puts != 2 {
+ t.Fatalf("changed third pass puts = %d, want 2", s.puts)
+ }
+}
+
+// FINDING 1 (review, 2026-08-22): readPublicIP is a single 2s best-effort
+// IMDS attempt, re-run every tick — a transient throttle or blip returns ""
+// exactly like a box that never had a public IP at all. Without a hold, a
+// single hiccuped tick would blank a known-good EIP in the published file,
+// costing every remote edge its dial address for this box's home-boxed
+// hosts until the next successful tick (up to 5min + jitter). A pass
+// reporting "" after a pass that reported "1.2.3.4" must keep "1.2.3.4".
+func TestEdgesFileHoldsLastKnownGoodEIPOnTransientBlank(t *testing.T) {
+ s, reg := release.NewDirStore(t.TempDir()), edgesTestRegistry(t)
+ seedRoute(t, reg, "app.example.com")
+
+ var last []byte
+ if err := edgesFilePass(context.Background(), s, reg, "annie", "1.2.3.4", &last); err != nil {
+ t.Fatal(err)
+ }
+
+ // Simulate a transient IMDS blip: this tick's probe returns "".
+ if err := edgesFilePass(context.Background(), s, reg, "annie", "", &last); err != nil {
+ t.Fatal(err)
+ }
+
+ got, _, err := s.Get(context.Background(), "control/edges/annie.json")
+ if err != nil {
+ t.Fatal(err)
+ }
+ var f edgesFile
+ if err := json.Unmarshal(got, &f); err != nil {
+ t.Fatal(err)
+ }
+ if f.EIP != "1.2.3.4" {
+ t.Fatalf("eip after a blank probe = %q, want held value %q", f.EIP, "1.2.3.4")
+ }
+}
+
+// FINDING 2 (review, 2026-08-22, ruling): edge.Reload (internal/edge/edge.go)
+// skips any route whose Status != "ready" — that gate is the deployment's
+// own definition of "serves", so a pending/creating/failed row must not
+// claim role:"home" here either. Hibernating routes stay Status "ready",
+// so this filter excludes nothing a remote backhaul could actually reach.
+func TestEdgesFileExcludesNonReadyRows(t *testing.T) {
+ s, reg := release.NewDirStore(t.TempDir()), edgesTestRegistry(t)
+ seedRoute(t, reg, "ready.example.com")
+ if err := reg.Upsert(registry.Route{Host: "creating.example.com", Kind: "service", Addr: "10.0.0.2:3000", Status: "creating"}); err != nil {
+ t.Fatal(err)
+ }
+
+ var last []byte
+ if err := edgesFilePass(context.Background(), s, reg, "annie", "1.2.3.4", &last); err != nil {
+ t.Fatal(err)
+ }
+ got, _, err := s.Get(context.Background(), "control/edges/annie.json")
+ if err != nil {
+ t.Fatal(err)
+ }
+ var f edgesFile
+ if err := json.Unmarshal(got, &f); err != nil {
+ t.Fatal(err)
+ }
+ if _, ok := f.Hosts["ready.example.com"]; !ok {
+ t.Fatalf("hosts = %+v, want ready.example.com present", f.Hosts)
+ }
+ if _, ok := f.Hosts["creating.example.com"]; ok {
+ t.Fatalf("hosts = %+v, want creating.example.com ABSENT (not ready, nothing serves it yet)", f.Hosts)
+ }
+}
+
+// FINDING 3 (review, 2026-08-22): a catch-all claim's registry row is a real
+// "*.parent" host (Task 3's reconcile pass derives it the same way
+// edge.go's upstream map does). This writer performs no wildcard expansion
+// — the row's Host lands in Hosts verbatim, exactly like every other row —
+// so an exact-match reader must apply edge.go's lookupUpstream one-label
+// rule itself; that contract is documented at the call site and in the
+// spec, not enforced here.
+func TestEdgesFileWildcardRowPublishesVerbatim(t *testing.T) {
+ s, reg := release.NewDirStore(t.TempDir()), edgesTestRegistry(t)
+ if err := reg.Upsert(registry.Route{Host: "*.example.com", Kind: "service", Addr: "10.0.0.3:3000", Status: "ready"}); err != nil {
+ t.Fatal(err)
+ }
+
+ var last []byte
+ if err := edgesFilePass(context.Background(), s, reg, "annie", "1.2.3.4", &last); err != nil {
+ t.Fatal(err)
+ }
+ got, _, err := s.Get(context.Background(), "control/edges/annie.json")
+ if err != nil {
+ t.Fatal(err)
+ }
+ var f edgesFile
+ if err := json.Unmarshal(got, &f); err != nil {
+ t.Fatal(err)
+ }
+ if r, ok := f.Hosts["*.example.com"]; !ok || r.Role != "home" {
+ t.Fatalf("hosts = %+v, want \"*.example.com\" present verbatim with role home", f.Hosts)
+ }
+}
+
+// FINDING 4 (review, 2026-08-22): every other test in this file round-trips
+// through the edgesFile struct on both the write and the assert side, so a
+// json-tag rename (e.g. "eip" -> "publicIP") would stay green everywhere
+// while breaking Task 3's parser, which reads the wire bytes independently
+// of this package's Go types. This test pins the RAW body against a
+// hand-written literal matching the brief's exact schema string — every
+// byte is fixed except the generation timestamp, which this fixed
+// signature has no clock seam to inject (edgesFilePass takes no `now`
+// parameter, per the brief's pinned interface); that one field is read
+// back and re-inserted into the literal rather than relaxed by a regex.
+func TestEdgesFileRawWireBodyMatchesSchema(t *testing.T) {
+ s, reg := release.NewDirStore(t.TempDir()), edgesTestRegistry(t)
+ seedRoute(t, reg, "app.example.com")
+
+ var last []byte
+ if err := edgesFilePass(context.Background(), s, reg, "annie-welding", "99.81.104.219", &last); err != nil {
+ t.Fatal(err)
+ }
+ got, _, err := s.Get(context.Background(), "control/edges/annie-welding.json")
+ if err != nil {
+ t.Fatal(err)
+ }
+
+ var probe struct {
+ Generation string `json:"generation"`
+ }
+ if err := json.Unmarshal(got, &probe); err != nil {
+ t.Fatal(err)
+ }
+ if _, err := time.Parse(time.RFC3339, probe.Generation); err != nil {
+ t.Fatalf("generation %q is not RFC3339: %v", probe.Generation, err)
+ }
+
+ want := `{"v":1,"box":"annie-welding","eip":"99.81.104.219","generation":"` +
+ probe.Generation + `","hosts":{"app.example.com":{"role":"home"}}}`
+ if string(got) != want {
+ t.Fatalf("raw wire body =\n%s\nwant\n%s", got, want)
+ }
+}
diff --git a/internal/host/housekeeper.go b/internal/host/housekeeper.go
index ab7386fa0..baf4442c4 100644
--- a/internal/host/housekeeper.go
+++ b/internal/host/housekeeper.go
@@ -58,10 +58,12 @@ import (
"context"
"errors"
"log"
+ "math/rand/v2"
"os"
"sync/atomic"
"time"
+ "github.com/carlosframework/platform/internal/edge"
"github.com/carlosframework/platform/internal/lease"
)
@@ -422,3 +424,49 @@ func runHousekeeperDuties(gate func() (context.Context, bool), duties ...houseke
}
}
}
+
+// mintMeshKeyLoop mints the deployment's mesh key (multi-region backhaul
+// spec, Part 3 Task 2) — a housekeeper duty, but deliberately its OWN loop
+// rather than a duty folded into retentionLoop's runHousekeeperDuties call
+// (agent.go's retentionLoop, gated on prune's cadence).
+//
+// The plan first drafted the mint as riding retentionLoop's tick; review
+// caught the trap: retentionLoop returns early — never arming its loop at
+// all, so runHousekeeperDuties is never even called — on a box missing
+// CARLOS_RELEASE_PUBKEY (release-signature verification, something wholly
+// unrelated to the mesh key). Critically, a box in that state still WINS
+// the housekeeper election (newHousekeeper/hk.run don't consult
+// CARLOS_RELEASE_PUBKEY at all), so the one box holding the lease would
+// simply never mint, and no other box would ever try either — the mesh key
+// would silently never exist, and Tasks 3/4's hop auth would never arm,
+// fleet-wide, with no error anywhere. This loop is gated on nothing but
+// deployment-store presence (shouldSkipPublisherElection, openDeployStore),
+// mirroring edgesfile.go's edgesFileLoop shape exactly, so minting is
+// available on every housekeeper independent of every other loop's
+// preconditions.
+//
+// mintMeshKey itself is create-only and idempotent (mesh.go's own comment):
+// every tick after the deployment's first successful mint finds the key
+// already there and does nothing, silently.
+func mintMeshKeyLoop() {
+ ctx := context.Background()
+ if shouldSkipPublisherElection(os.Getenv) {
+ log.Printf("agent: mesh key mint loop is a platform-box job — skipped on a fleet box")
+ return
+ }
+ store, err := openDeployStore(ctx)
+ if err != nil {
+ log.Printf("agent: mesh key mint loop disabled: %v", err)
+ return
+ }
+ log.Printf("agent: mesh key mint loop armed (5min + jitter) — mints only while this box holds the housekeeper lease")
+ mint := housekeeperDuty{"mesh key mint", func(ctx context.Context) error {
+ return edge.MintMeshKey(ctx, store)
+ }}
+ delay := time.Minute
+ for {
+ time.Sleep(delay)
+ delay = 5*time.Minute + rand.N(time.Minute)
+ runHousekeeperDuties(housekeeperGate, mint)
+ }
+}
diff --git a/internal/host/housekeeper_test.go b/internal/host/housekeeper_test.go
index 13ec86c3e..31d19e64f 100644
--- a/internal/host/housekeeper_test.go
+++ b/internal/host/housekeeper_test.go
@@ -511,3 +511,52 @@ func TestHousekeeperHeldIsFalseBeforeAnyElectionIsArmed(t *testing.T) {
t.Fatal("housekeeperGate() reports held with no housekeeper armed")
}
}
+
+// ---- mintMeshKeyLoop (multi-region backhaul spec, Part 3 Task 2's review
+// fix): the mesh key's own arming loop, independent of retentionLoop -------
+
+// Mirrors TestRetentionLoopIsAPlatformBoxJob's shape (agent_test.go): a
+// fleet box must skip before ever opening a store.
+func TestMintMeshKeyLoopIsAPlatformBoxJob(t *testing.T) {
+ const skipLine = "agent: mesh key mint loop is a platform-box job — skipped on a fleet box"
+
+ logs := captureLog(t)
+ door := newDoorFake(t)
+ for k, v := range fleetDoorEnv(door.url) {
+ t.Setenv(k, v)
+ }
+ t.Setenv("CARLOS_DEPLOYMENT_BUCKET", "")
+ t.Setenv("CARLOS_DEPLOYMENT_DIR", "")
+
+ mintMeshKeyLoop() // must return; a fleet box never reaches the loop
+
+ if got := logs(); !strings.Contains(got, skipLine) {
+ t.Fatalf("log = %q, want the skip line verbatim", got)
+ }
+ if hits := door.seen(); len(hits) != 0 {
+ t.Fatalf("door saw %+v, want zero — the skip must land BEFORE the store opens", hits)
+ }
+}
+
+// The regression guard for the Critical review finding: the mint must NOT be
+// gated on CARLOS_RELEASE_PUBKEY (retentionLoop's own gate, unrelated to the
+// mesh key) — a box missing that var still wins the housekeeper election, so
+// a mint duty tied to retentionLoop's arming would silently never run
+// fleet-wide. Proven by explicitly leaving CARLOS_RELEASE_PUBKEY unset and
+// confirming the loop still arms.
+func TestMintMeshKeyLoopArmsWithoutReleasePubkey(t *testing.T) {
+ t.Setenv("CARLOS_FLEET", "")
+ t.Setenv("CARLOS_DEPLOYMENT_BUCKET", "")
+ t.Setenv("CARLOS_DEPLOYMENT_DIR", t.TempDir())
+ t.Setenv("CARLOS_RELEASE_PUBKEY", "") // deliberately unset — the whole point
+
+ buf := captureRacyLog(t)
+ // mintMeshKeyLoop never returns once armed: run it in its own goroutine
+ // (it parks in a time.Sleep for a minute, long past this test's end) and
+ // wait for the arming log line against a synchronized sink.
+ go mintMeshKeyLoop()
+
+ waitFor(t, "the mint loop to arm despite no CARLOS_RELEASE_PUBKEY", func() bool {
+ return strings.Contains(buf.String(), "agent: mesh key mint loop armed")
+ })
+}
diff --git a/internal/provision/reconcile.go b/internal/provision/reconcile.go
index 71ee5a3b7..8b532196a 100644
--- a/internal/provision/reconcile.go
+++ b/internal/provision/reconcile.go
@@ -1,7 +1,9 @@
package provision
// Reconcile turns bucket instance records into registry routes, for THIS
-// agent's own region and pool set. Spec:
+// agent's own region and pool set — plus, since the multi-region ingress
+// design's Part 3, the serving classes every platform box materializes
+// wherever they are homed (see "serving classes" below). Spec:
// docs/superpowers/specs/2026-08-04-instance-provisioning-design.md §4.
//
// DNS seam (spec §10): multi-region deployments write the per-slug CNAME at
@@ -14,6 +16,7 @@ import (
"errors"
"fmt"
"log"
+ "net"
"os"
"slices"
"strings"
@@ -52,6 +55,174 @@ type ReconcileConfig struct {
// only within an account, never globally.
type acctApp struct{ acct, app string }
+// ---- serving classes -------------------------------------------------------
+//
+// Which boxes materialize a record. The multi-region ingress design's Part 3
+// table, verbatim, and the ONE place this pass states it:
+//
+// Everywhere static (bake-served): the edge resolves the app's
+// channel pointer and serves the shipped tar out of the
+// deployment store every box already reads, so no box
+// needs the home box to serve it.
+// Everywhere, probed addr routes whose upstreams are ALL publicly dialable:
+// each edge runs its own prober and dials the backends
+// directly, so per-edge live sets are per-edge by design.
+// Home-boxed exec, unit-owned, socket-activated, tunnel- and
+// fleet-backed: the process exists on exactly ONE box,
+// and a request that lands elsewhere backhauls to it
+// (internal/edge's hop). Unchanged invariant.
+//
+// Everything here decides SCOPE only. Nothing about the route a record mints
+// changes with the box that mints it — routeFor stays the one renderer, so a
+// far box's copy and the home box's copy are the same route, column for
+// column, including the record's own Region.
+//
+// Single-box inertness: with one box there is no far box, every record is in
+// scope for the same reason it was before, and the widened half of the gate
+// never fires.
+
+// everywhereClass reports whether rec's serving CLASS is one every platform
+// box materializes. It answers the class question only — the tenancy
+// questions (pool, fleet placement) are inScope's, below.
+//
+// Conservative by construction: anything this function cannot positively
+// classify as servable from a box that holds nothing but the bucket is
+// home-boxed. That is the direction a wrong answer must fail in — a route that
+// stays home-boxed is served exactly as it is today (and backhauled from
+// elsewhere), while one wrongly promoted to everywhere is a box proxying to an
+// upstream it cannot reach.
+func everywhereClass(rec Record) bool {
+ // Unit names the systemd unit that owns this route's process (issue
+ // #110); adoption CYCLES it. That unit exists on the home box, so a far
+ // box adopting the row would be stamping authority over a unit name that
+ // means something else there, or nothing at all.
+ if rec.Unit != "" {
+ return false
+ }
+ // Belt and braces: ValidateShape already refuses hibernation on both
+ // classes below (a static route has no process to wake, an addr-backed one
+ // has no socket), and this pass validates every record before the gate. A
+ // hibernating route needs the activator and the socket only its home box
+ // has, so if that door ever moved, this must not be the hole it opens.
+ if rec.HibernateOrDefault() {
+ return false
+ }
+ if rec.KindOrDefault() == "static" {
+ return true
+ }
+ ups := rec.AddrsOrDefault()
+ if len(ups) == 0 {
+ return false // socket-backed: the exec child lives on one box
+ }
+ for _, a := range ups {
+ if !publiclyDialable(a) {
+ return false
+ }
+ }
+ return true
+}
+
+// everywhereRoute is everywhereClass read off a registry ROW instead of a
+// record — the removal phase's half of the same question, asked when the
+// record is gone and the row is all that is left to ask.
+//
+// The two must agree, or the widened mint gate leaves immortal routes: a far
+// box that mints an everywhere-class route and then cannot see it as one is a
+// box serving a retired site forever. They agree by construction today because
+// routeFor is the one renderer between them, and
+// TestEverywhereClassAgreesAcrossRecordAndRow locks that: a class rule added to
+// one of these functions alone recreates exactly the immortality bug this pair
+// was written to close, with every other test still green.
+//
+// They are separate functions rather than one over a shared shape because the
+// row carries columns no record can state (Backing, Tunnel, Caps), and a row
+// that has GROWN one of those since it was minted is deliberately read as
+// home-boxed here. State that cost honestly, because it is bigger than a stale
+// row: an operator who runs `carlos route -grant` (or -backing/-tunnel) against
+// the FAR box's copy takes that copy out of removal scope permanently. Deleting
+// the record then retires the route on its home box and leaves the far box
+// still SERVING the retired site, grants intact, with no record left anywhere
+// to explain it — so `carlos remove` stops being fleet-wide the moment anyone
+// touches a far copy. The alternative (removing rows carrying operator columns)
+// hands an unattended bucket-record deletion the grant purge registry.Delete
+// documents as human-gated, which is worse. The conservative gate stands; the
+// gap is tracked in the bookkeeping issue, not papered over here.
+func everywhereRoute(rt registry.Route) bool {
+ if rt.Unit != "" || rt.Hibernate || rt.Backing != "" || rt.Tunnel != "" || rt.Caps != "" {
+ return false
+ }
+ if rt.Kind == "static" {
+ return true
+ }
+ ups := rt.AddrList()
+ if len(ups) == 0 {
+ return false
+ }
+ for _, a := range ups {
+ if !publiclyDialable(a) {
+ return false
+ }
+ }
+ return true
+}
+
+// publiclyDialable reports whether one upstream is an address every edge in
+// the deployment can reach — the "publicly-reachable upstreams" half of the
+// spec's table.
+//
+// A NAME is not dialable for this purpose even if it resolves to a public
+// address today: validAddrsElement's doc comment already records why a name
+// cannot be judged at all here (it is free to move after the check), and the
+// legacy Addr column is full of names pointing at box-local backends. It stays
+// home-boxed, which is where those routes are serving from today.
+//
+// Loopback and link-local are the spec's own examples (another box's
+// 127.0.0.1 is not this box's). RFC1918 and ULA are refused for exactly the
+// same reason one layer out: a private address is private to ONE network, and
+// two boxes in two regions are two networks. IsGlobalUnicast covers the rest
+// (unspecified, multicast, loopback, link-local).
+func publiclyDialable(addr string) bool {
+ h, _, err := net.SplitHostPort(addr)
+ if err != nil {
+ return false
+ }
+ ip := net.ParseIP(h)
+ if ip == nil {
+ return false
+ }
+ return ip.IsGlobalUnicast() && !ip.IsPrivate()
+}
+
+// inScope answers "is this region/pool pair this pass's business" — the one
+// gate the record scan and the removal phase both ask, so a record this box
+// mints is a record this box can retire.
+//
+// Pool is compared FIRST and never widened. It is a TENANCY boundary, not a
+// geography: a pool names the boxes a tenant's routes are allowed to land on
+// (customer fleets, dedicated boxes), and "everywhere" in the spec's table
+// means every box that already serves this tenant, not every box in the world.
+//
+// Region equality is the unchanged home-box test. It is widened for the
+// everywhere classes, and only between PLATFORM regions: a fleet placement
+// ("fleet/<name>", registry.FleetPlacement) is the other tenancy boundary and
+// is refused from both sides — the platform's boxes do not pick up a
+// customer fleet's routes, and a customer's box does not pick up the
+// platform's.
+func (cfg ReconcileConfig) inScope(region, pool string, everywhere bool) bool {
+ if !poolServed(cfg.Pools, pool) {
+ return false
+ }
+ if region == cfg.Region {
+ return true
+ }
+ if !everywhere {
+ return false
+ }
+ _, recordOnFleet := registry.FleetPlacement(region)
+ _, boxOnFleet := registry.FleetPlacement(cfg.Region)
+ return !recordOnFleet && !boxOnFleet
+}
+
// routeFor is the one answer to "what route does this record describe". The
// mint path builds from it and the adoption match test (design §4.2)
// compares against it — one function, so a shape the reconciler would mint
@@ -418,7 +589,10 @@ func Reconcile(ctx context.Context, s release.Store, reg *registry.Registry, cfg
protect[host] = true
continue
}
- if rec.Region != cfg.Region || !poolServed(cfg.Pools, rec.Pool) {
+ // Scope. For a home-boxed record this is the region/pool test it
+ // always was; for a static or public-addr one the region half widens
+ // to every platform box in the deployment (everywhereClass, above).
+ if !cfg.inScope(rec.Region, rec.Pool, everywhereClass(rec)) {
continue // another agent's jurisdiction, not this pass's business
}
// Two accounts writing the same host is a collision this pass
@@ -640,6 +814,12 @@ func Reconcile(ctx context.Context, s release.Store, reg *registry.Registry, cfg
// absence from the record set this pass just read (the bucket's own
// decision to retire the instance, not a guess this pass made).
//
+ // Scope is asked through the SAME inScope as the record scan above, with
+ // the class read off the row (everywhereRoute) because the record whose
+ // class it was is exactly what is gone. Both halves must widen together:
+ // a far box that mints a static route under the widened mint gate and then
+ // judges it out of scope here would serve a retired site forever.
+ //
// What reaches reg.Delete here may carry Backing, Tunnel or Caps even
// though shapeMismatch refuses to ADOPT a row carrying any of them: those
// refusals cover the adoption instant only. `carlos route -backing/-unit/
@@ -650,7 +830,7 @@ func Reconcile(ctx context.Context, s release.Store, reg *registry.Registry, cfg
// the adoption guard is not mistaken for a standing guarantee.
if !readFailed {
for _, rt := range routes {
- if !rt.Provisioned || rt.Region != cfg.Region || !poolServed(cfg.Pools, rt.Pool) {
+ if !rt.Provisioned || !cfg.inScope(rt.Region, rt.Pool, everywhereRoute(rt)) {
continue
}
if _, ok := recs[rt.Host]; ok {
diff --git a/internal/provision/reconcile_test.go b/internal/provision/reconcile_test.go
index f9193a80d..039ba4b19 100644
--- a/internal/provision/reconcile_test.go
+++ b/internal/provision/reconcile_test.go
@@ -1826,6 +1826,292 @@ func TestReconcileRefusesAdoptInstanceRouteForStaticRecord(t *testing.T) {
}
}
+// ---- serving classes (multi-region ingress design, Part 3's table) ---------
+//
+// The same deployment seen from two PLATFORM boxes: the record's home region
+// ("ie") and another one ("us"), both serving the tito pool. Everything below
+// asks the same question of one record — "does the far box materialize it?" —
+// and the answer is the record's serving class.
+
+func homeBoxCfg() ReconcileConfig {
+ return ReconcileConfig{Region: "ie", Pools: []string{"tito"}, DataDir: "/data"}
+}
+
+func farBoxCfg() ReconcileConfig {
+ return ReconcileConfig{Region: "us", Pools: []string{"tito"}, DataDir: "/data"}
+}
+
+// titoRecord is the tito-app fixture every serving-class case below mutates:
+// homed in "ie", in the "tito" pool, on the stable channel.
+func titoRecord(sqid, host string) Record {
+ return Record{Schema: 1, Host: host + ".go.tito.io", Slug: host, App: "tito",
+ Account: sqid, Pool: "tito", Region: "ie", Channel: "stable", CreatedAt: time.Now()}
+}
+
+// A static (bake-served) record is materialized by EVERY platform box in the
+// deployment, not only its home: the edge resolves the app's channel pointer
+// out of the same bucket every box already reads, so no box needs the home box
+// to serve it. Spec Part 3, class "Everywhere".
+func TestReconcileStaticIsMaterializedOnEveryPlatformBox(t *testing.T) {
+ s, sqid := setup(t)
+ rec := titoRecord(sqid, "site")
+ rec.Kind = "static"
+ if err := CreateRecord(ctx, s, rec); err != nil {
+ t.Fatal(err)
+ }
+ reg := regWith(t)
+ if err := Reconcile(ctx, s, reg, farBoxCfg()); err != nil {
+ t.Fatal(err)
+ }
+ got := mustGet(t, reg, "site.go.tito.io")
+ if !got.Provisioned || got.Kind != "static" {
+ t.Fatalf("far box did not materialize the static record: %+v", *got)
+ }
+ // The row still carries the RECORD's region — the record is the desired
+ // state and routeFor is its one renderer, so a far box's copy and the home
+ // box's copy are the same route, byte for byte.
+ if got.Region != "ie" {
+ t.Errorf("Region = %q, want the record's own %q", got.Region, "ie")
+ }
+ if got.Socket != "" || got.DBPath != "" || got.Hibernate {
+ t.Errorf("a static route grew a process shape on the far box: %+v", *got)
+ }
+}
+
+// An addr route whose upstreams are all publicly dialable is materialized
+// everywhere too — every edge probes and dials the same public backends. Spec
+// Part 3, class "Everywhere, probed".
+func TestReconcilePublicAddrIsMaterializedOnEveryPlatformBox(t *testing.T) {
+ s, sqid := setup(t)
+ one := titoRecord(sqid, "pub")
+ one.Kind = "service"
+ one.Addr = "203.0.113.10:443"
+ many := titoRecord(sqid, "pubs")
+ many.Kind = "service"
+ many.Addrs = []string{"203.0.113.11:443", "198.51.100.7:443"}
+ for _, rec := range []Record{one, many} {
+ if err := CreateRecord(ctx, s, rec); err != nil {
+ t.Fatal(err)
+ }
+ }
+ reg := regWith(t)
+ if err := Reconcile(ctx, s, reg, farBoxCfg()); err != nil {
+ t.Fatal(err)
+ }
+ for _, host := range []string{"pub.go.tito.io", "pubs.go.tito.io"} {
+ if got := mustGet(t, reg, host); !got.Provisioned {
+ t.Errorf("%s: far box did not materialize a public-upstream record: %+v", host, *got)
+ }
+ }
+}
+
+// The home-boxed classes, unchanged: the process backing them exists on
+// exactly ONE box, so a far box must still refuse every one of them. This is
+// the refusal test the class change is only safe with.
+func TestReconcileHomeBoxedClassesStayHomeBoxed(t *testing.T) {
+ s, sqid := setup(t)
+ socketed := titoRecord(sqid, "exec") // no kind, no addr: the socket/exec shape
+ loopback := titoRecord(sqid, "loop")
+ loopback.Kind = "service"
+ loopback.Addr = "127.0.0.1:8090"
+ private := titoRecord(sqid, "priv")
+ private.Kind = "service"
+ private.Addrs = []string{"10.0.0.5:3000"}
+ unitOwned := titoRecord(sqid, "unit")
+ unitOwned.Addr = "203.0.113.20:8090"
+ unitOwned.Unit = "console.service"
+ named := titoRecord(sqid, "named")
+ named.Kind = "service"
+ named.Addr = "backend.example.com:443"
+ for _, rec := range []Record{socketed, loopback, private, unitOwned, named} {
+ if err := CreateRecord(ctx, s, rec); err != nil {
+ t.Fatal(err)
+ }
+ }
+ reg := regWith(t)
+ if err := Reconcile(ctx, s, reg, farBoxCfg()); err != nil {
+ t.Fatal(err)
+ }
+ for _, host := range []string{"exec.go.tito.io", "loop.go.tito.io", "priv.go.tito.io",
+ "unit.go.tito.io", "named.go.tito.io"} {
+ if rt, err := reg.Get(host); err != nil || rt != nil {
+ t.Errorf("%s: a home-boxed record materialized on a foreign box: %+v %v", host, rt, err)
+ }
+ }
+ // ...and every one of them still materializes on its OWN box.
+ if err := Reconcile(ctx, s, reg, homeBoxCfg()); err != nil {
+ t.Fatal(err)
+ }
+ for _, host := range []string{"exec.go.tito.io", "loop.go.tito.io", "priv.go.tito.io",
+ "unit.go.tito.io", "named.go.tito.io"} {
+ mustGet(t, reg, host)
+ }
+}
+
+// Pool is a TENANCY boundary, not a geography: widening the region half of the
+// gate must not widen the pool half. A static record in the tito pool stays
+// off a box that does not serve that pool, however far it travels.
+func TestReconcileEverywhereClassStillRespectsThePool(t *testing.T) {
+ s, sqid := setup(t)
+ rec := titoRecord(sqid, "site")
+ rec.Kind = "static"
+ if err := CreateRecord(ctx, s, rec); err != nil {
+ t.Fatal(err)
+ }
+ reg := regWith(t)
+ // A far box serving only the shared "" pool (nil Pools), and one serving
+ // somebody else's pool.
+ for _, cfg := range []ReconcileConfig{
+ {Region: "us", DataDir: "/data"},
+ {Region: "us", Pools: []string{"eleven"}, DataDir: "/data"},
+ } {
+ if err := Reconcile(ctx, s, reg, cfg); err != nil {
+ t.Fatal(err)
+ }
+ if rt, err := reg.Get("site.go.tito.io"); err != nil || rt != nil {
+ t.Fatalf("pools %v: a static record crossed a pool boundary: %+v %v", cfg.Pools, rt, err)
+ }
+ }
+}
+
+// Fleet placement is the other tenancy boundary, and it is refused from BOTH
+// sides: a customer fleet's static site is not served by the platform's boxes,
+// and a customer's fleet box does not pick up the platform's static records.
+// (registry.FleetPlacement is the grammar — "fleet/<name>" in the Region.)
+func TestReconcileFleetPlacementIsNeverEverywhere(t *testing.T) {
+ s, sqid := setup(t)
+ onFleet := titoRecord(sqid, "pisite")
+ onFleet.Kind = "static"
+ onFleet.Region = "fleet/pi"
+ platform := titoRecord(sqid, "site")
+ platform.Kind = "static"
+ for _, rec := range []Record{onFleet, platform} {
+ if err := CreateRecord(ctx, s, rec); err != nil {
+ t.Fatal(err)
+ }
+ }
+ reg := regWith(t)
+
+ // A platform box must not serve the fleet's site...
+ if err := Reconcile(ctx, s, reg, farBoxCfg()); err != nil {
+ t.Fatal(err)
+ }
+ if rt, err := reg.Get("pisite.go.tito.io"); err != nil || rt != nil {
+ t.Errorf("a fleet-placed static record materialized on a platform box: %+v %v", rt, err)
+ }
+ // ...and the fleet's own box must not pick up the platform's.
+ fleetReg := regWith(t)
+ if err := Reconcile(ctx, s, fleetReg, ReconcileConfig{Region: "fleet/pi", Pools: []string{"tito"}, DataDir: "/data"}); err != nil {
+ t.Fatal(err)
+ }
+ if rt, err := fleetReg.Get("site.go.tito.io"); err != nil || rt != nil {
+ t.Errorf("a platform static record materialized on a customer fleet box: %+v %v", rt, err)
+ }
+ mustGet(t, fleetReg, "pisite.go.tito.io") // its own placement, unchanged
+}
+
+// The other half of adopting everywhere: retiring everywhere. A far box that
+// mints an everywhere-class route from a record must remove it when that
+// record goes, or the widened mint gate leaves an immortal route serving a
+// retired site on every box but its home.
+func TestReconcileRemovesEverywhereRouteWhenItsRecordGoes(t *testing.T) {
+ s, sqid := setup(t)
+ rec := titoRecord(sqid, "site")
+ rec.Kind = "static"
+ if err := CreateRecord(ctx, s, rec); err != nil {
+ t.Fatal(err)
+ }
+ reg := regWith(t)
+ cfg := farBoxCfg()
+ if err := Reconcile(ctx, s, reg, cfg); err != nil {
+ t.Fatal(err)
+ }
+ mustGet(t, reg, "site.go.tito.io")
+
+ if err := DeleteRecord(ctx, s, sqid, "tito", "site.go.tito.io"); err != nil {
+ t.Fatal(err)
+ }
+ if err := Reconcile(ctx, s, reg, cfg); err != nil {
+ t.Fatal(err)
+ }
+ if rt, err := reg.Get("site.go.tito.io"); err != nil || rt != nil {
+ t.Errorf("a far box's everywhere-class route outlived its record: %+v %v", rt, err)
+ }
+}
+
+// TestEverywhereClassAgreesAcrossRecordAndRow is the drift pin between the two
+// halves of the class rule: everywhereClass reads a RECORD (the mint gate) and
+// everywhereRoute reads a ROW (the removal gate, asked when the record is gone
+// and the row is all that is left). routeFor is the one renderer between them,
+// so the property holds by construction today — and it is exactly what a class
+// rule added to one function alone would break, silently, with every other test
+// in this file still green. That divergence is the immortality bug the pair was
+// written to close: a far box that mints an everywhere-class route and then
+// cannot see it as one serves a retired site forever.
+//
+// The fixture set is every shape the file already exercises, plus the two
+// directions a new rule would most plausibly be written into one side only (a
+// probe config, an explicitly non-hibernating addr record).
+func TestEverywhereClassAgreesAcrossRecordAndRow(t *testing.T) {
+ off := false
+ on := true
+ base := func(host string) Record {
+ return Record{Schema: 1, Host: host, Slug: "s", App: "tito", Account: "bac",
+ Pool: "tito", Region: "ie", Channel: "stable"}
+ }
+ with := func(host string, mut func(*Record)) Record {
+ rec := base(host)
+ mut(&rec)
+ return rec
+ }
+ fixtures := []struct {
+ name string
+ rec Record
+ }{
+ {"socket instance (no kind, no addr)", base("exec.go.tito.io")},
+ {"static", with("site.go.tito.io", func(r *Record) { r.Kind = "static" })},
+ {"static, status-named", with("named-site.go.tito.io", func(r *Record) { r.Kind, r.StatusNamed = "static", &on })},
+ {"public single addr", with("pub.go.tito.io", func(r *Record) { r.Kind, r.Addr = "service", "203.0.113.10:443" })},
+ {"public addrs set", with("pubs.go.tito.io", func(r *Record) {
+ r.Kind, r.Addrs = "service", []string{"203.0.113.11:443", "198.51.100.7:443"}
+ })},
+ {"public addrs with an explicit probe", with("probed.go.tito.io", func(r *Record) {
+ r.Kind, r.Addrs, r.Health = "service", []string{"203.0.113.12:443", "198.51.100.8:443"}, &Health{Path: "/healthz"}
+ })},
+ {"public addr, explicitly never hibernating", with("awake.go.tito.io", func(r *Record) {
+ r.Kind, r.Addr, r.Hibernate = "service", "203.0.113.13:443", &off
+ })},
+ {"loopback addr", with("loop.go.tito.io", func(r *Record) { r.Kind, r.Addr = "service", "127.0.0.1:8090" })},
+ {"private addrs", with("priv.go.tito.io", func(r *Record) { r.Kind, r.Addrs = "service", []string{"10.0.0.5:3000"} })},
+ {"mixed public and private addrs", with("mixed.go.tito.io", func(r *Record) {
+ r.Kind, r.Addrs = "service", []string{"203.0.113.14:443", "192.168.1.9:3000"}
+ })},
+ {"name-based addr", with("named.go.tito.io", func(r *Record) { r.Kind, r.Addr = "service", "backend.example.com:443" })},
+ {"unit-owned public addr", with("unit.go.tito.io", func(r *Record) {
+ r.Addr, r.Unit = "203.0.113.20:8090", "console.service"
+ })},
+ {"environment-bound static", with("env-site.go.tito.io", func(r *Record) { r.Kind, r.Environment = "static", "staging" })},
+ {"fleet-placed static", with("pisite.go.tito.io", func(r *Record) { r.Kind, r.Region = "static", "fleet/pi" })},
+ }
+ for _, f := range fixtures {
+ t.Run(f.name, func(t *testing.T) {
+ // Every fixture must be one the pass would actually materialize —
+ // a shape ValidateShape refuses never reaches either gate, so a
+ // property proved on one would prove nothing.
+ if err := ValidateShape(f.rec); err != nil {
+ t.Fatalf("fixture is not a shape this pass would ever see: %v", err)
+ }
+ fromRecord := everywhereClass(f.rec)
+ fromRow := everywhereRoute(routeFor(f.rec, "/data", `["seed"]`))
+ if fromRecord != fromRow {
+ t.Fatalf("class disagrees across the mint and removal gates: everywhereClass = %v, "+
+ "everywhereRoute(routeFor(...)) = %v — a record this box mints must be a row it can retire",
+ fromRecord, fromRow)
+ }
+ })
+ }
+}
+
// TestRouteForCarriesAddrsAndHealth is Task 3's mint-side proof: a record
// with an explicit multi-upstream Addrs set mints a route whose Addrs is the
// comma-joined string and whose Health carries a fully-materialized default