amadan

carlos / platform Public

Merge pull request #283 from carlosframework/egress-and-minting-rulings

multi-region: backhaul egress ruling (cap + hop metering) and the #277 cert-gate fix
Paul Campbell pushed by paul@keymail.dev 6a861ce174b59a52292b499545ccb87b002eea18 parent 1010be2parent 5d59179
15 files changed, +1348 −31
  • docs/superpowers/plans/2026-08-22-multi-region-backhaul.md +25 −0
  • docs/superpowers/plans/2026-08-23-multi-region-steering.md +37 −0
  • docs/superpowers/specs/2026-08-20-multi-region-ingress-design.md +148 −1
  • internal/edge/backhaul_e2e_test.go +50 −0
  • internal/edge/certsync.go +33 −13
  • internal/edge/certsync_e2e_test.go +141 −0
  • internal/edge/certsync_test.go +192 −10
  • internal/edge/edge.go +154 −1
  • internal/edge/homemap.go +39 −0
  • internal/edge/homemap_test.go +388 −0
  • internal/edge/usagemeter.go +12 −2
  • internal/edge/usagemeter_test.go +30 −4
  • internal/usage/edge.go +16 −0
  • internal/usage/edge_test.go +76 −0
  • internal/usage/usage.go +7 −0
diff --git a/docs/superpowers/plans/2026-08-22-multi-region-backhaul.md b/docs/superpowers/plans/2026-08-22-multi-region-backhaul.md
index 79e338ea9..dbff81597 100644
--- a/docs/superpowers/plans/2026-08-22-multi-region-backhaul.md
+++ b/docs/superpowers/plans/2026-08-22-multi-region-backhaul.md
@@ -625,6 +625,31 @@ b. **Visibility for the two stuck-state paths before steering sends
ruling should be recorded as a numbered issue once Paul rules on the
containment approach.
+ **Resolved (2026-08-24) — the ruling landed; this item is closed.**
+ Both halves are implemented and dormant on a single box; the
+ authoritative statement is the spec's **Amendment (2026-08-24)** in
+ Part 3 of `docs/superpowers/specs/2026-08-20-multi-region-ingress-design.md`.
+ *Containment:* the semaphore option above was taken — a per-home-box
+ in-flight cap on the SENDING edge (`CARLOS_BACKHAUL_MAX_INFLIGHT`,
+ default 64), answering 503 + `Retry-After: 2` at the limit with no
+ dial, logged on the throttled per-box key `"backhaul-saturated "+box`.
+ A slot is held for the life of the hop's response, streams and
+ upgrades included. *Attribution:* the missing half above is closed
+ too, on the HOME box rather than by relocating the meter — the one
+ meter site keeps counting each hop-served exchange exactly once and
+ additionally annotates it into two additive, `omitempty`, SUBSET
+ fields on `usage.EdgeRow` (`hop_requests`/`hop_bytes_out`), so every
+ backhauled byte is attributed to the route, account and app that
+ caused it. The territorial cross-ref is honored: the shapes live in
+ `internal/usage`, `internal/edge` only gained a flag parameter.
+ *Pricing stays Paul's*: `backhaul_gb` is reserved in the meter
+ vocabulary and wired into nothing — bills are byte-identical to
+ before, absorbed by default. The numbered issue this item asks for is
+ still owed, to be filed carrying the ruling's text when the PR opens.
+ Rollout: must be on every box of a deployment before that deployment
+ arms a second edge (ahead of the steering plan's arming checklist
+ step 0).
+
### Constraint for Part 4 / plan 3 (steering) to honor
6. **Pool narrowing: steered non-shared-pool routes must not get
diff --git a/docs/superpowers/plans/2026-08-23-multi-region-steering.md b/docs/superpowers/plans/2026-08-23-multi-region-steering.md
index 4f7d48737..074da8dd6 100644
--- a/docs/superpowers/plans/2026-08-23-multi-region-steering.md
+++ b/docs/superpowers/plans/2026-08-23-multi-region-steering.md
@@ -350,6 +350,15 @@ func desiredState(routes []registry.Route, edges []edgesFileView, appsDomain str
commands ready-to-run); edges-file `role` over-claim note (steering
reads the ARMED-EDGE set, not roles — unaffected, but read-local
still wants the replica role first, restated).
+ **RULED (2026-08-24), superseding this item's "egress ruling request
+ stays open (#-ref)":** the T7 ruling landed — sender-side per-home-box
+ in-flight cap plus home-box hop annotation, pricing deferred to Paul
+ and absorbed by default. Authoritative text: the **Amendment
+ (2026-08-24)** in Part 3 of
+ `docs/superpowers/specs/2026-08-20-multi-region-ingress-design.md`.
+ The arming checklist below is unchanged except for its ordering: the
+ ruling's code must be rolled to every box of a deployment BEFORE that
+ deployment arms a second edge, i.e. ahead of step 0.
- [x] Full gate; commit `docs: steering rollout notes`.
## Rollout notes (filled by Task 8 at execution)
@@ -605,6 +614,21 @@ rollout notes and constraints where they name Part 4 directly.
window the backhaul plan called out ("today it is dormant/low-volume
by construction... which is the window in which to decide, not
after") ends the moment stage 5 begins.
+ **RULED (2026-08-24), inside that window:** the ruling landed and both
+ halves are implemented — containment is the semaphore option named
+ above (a per-home-box in-flight cap on the sending edge,
+ `CARLOS_BACKHAUL_MAX_INFLIGHT`, default 64, 503 + `Retry-After: 2` at
+ the limit with no dial), and attribution is a home-box hop annotation
+ (`usage.EdgeRow`'s additive, `omitempty`, SUBSET `hop_requests` /
+ `hop_bytes_out`), so backhauled bytes are attributed to the route,
+ account and app that caused them. Pricing them stays Paul's call and
+ ships absorbed: `backhaul_gb` is reserved and wired into nothing, so
+ bills are byte-identical to before. Authoritative text: the
+ **Amendment (2026-08-24)** in Part 3 of
+ `docs/superpowers/specs/2026-08-20-multi-region-ingress-design.md`.
+ Still owed: the numbered issue, to be filed with the ruling's text
+ when the PR opens. Stage-5 consequence: this must be rolled to every
+ box of a deployment before that deployment arms its second edge.
### Deferred-minor triage (every task's ledger entry, included or excluded with a reason)
@@ -721,6 +745,19 @@ the task's own two fix rounds. Nothing left to triage.
later") is needed before or alongside the stage-5 arming decision —
raised here rather than left buried in plan 2's own notes, since this
is the plan whose stage-5 work makes the ruling time-sensitive.
+ **RULED (2026-08-24) — no longer blocking, one question left.** The
+ containment approach is decided and built (per-home-box in-flight cap
+ on the sending edge, default 64, 503 + `Retry-After: 2`), as is
+ attribution (home-box `hop_requests`/`hop_bytes_out` on
+ `usage.EdgeRow`); see the **Amendment (2026-08-24)** in Part 3 of
+ `docs/superpowers/specs/2026-08-20-multi-region-ingress-design.md`.
+ What is still Paul's: **whether backhauled bytes should be PRICED** (a
+ `backhaul_gb` rate-card line or an egress multiplier) or absorbed as a
+ platform cost of any-edge serving. Shipped default: absorbed — the
+ counters accumulate dormant, so the decision can be made on real data
+ after stage 5, and nothing in the engineering depends on the answer.
+ Also owed: file the numbered issue carrying the ruling's text, and
+ roll this to every box of a deployment before arming its second edge.
- **PR routing**: this branch merges to `main` directly — no stack, no
base branch other than `main` (see the PR body, drafted in the task
report).
diff --git a/docs/superpowers/specs/2026-08-20-multi-region-ingress-design.md b/docs/superpowers/specs/2026-08-20-multi-region-ingress-design.md
index a3e830387..246e84bf6 100644
--- a/docs/superpowers/specs/2026-08-20-multi-region-ingress-design.md
+++ b/docs/superpowers/specs/2026-08-20-multi-region-ingress-design.md
@@ -126,7 +126,7 @@ the one box always holds the housekeeper lease, so behaviour is identical
to today. Residual: tls-alpn-01 (autocert's PREFERRED challenge type) is
NOT end-to-end proven — only http-01 is; tls-alpn-01 has real unit
coverage (`TestMirrorAnswersAnotherEdgesALPNValidation`,
-`TestMirrorALPNReadThroughOnlyForServedHosts`,
+`TestMirrorALPNReadThroughOnlyForServedOrRemoteHomedHosts`,
`TestMirrorNeverServesATokenCertToAnOrdinaryHello`,
`TestALPNTokenCertIsReadLiveNotFromTheSnapshot` in
`internal/edge/certsync_test.go`) but not the e2e proof — tracked as a
@@ -168,6 +168,108 @@ prefix already listed and read on every mirror poll — and is left for a
later sweep rather than solved here; tracked as issue #243 (filed). The
issuer remains the only box that talks to the CA.
+**Amendment (2026-08-24, #277): the three cert-adjacent host gates admit
+exact home-map names, not only this box's own routes.** As shipped, the
+HTTP-01 challenge read-through, the tls-alpn-01 read-through and
+`issuerHostPolicy`'s order gate all bounded on `Edge.Allowed` — this
+box's registry. That was the right anti-stranger bound for a world where
+every box's registry named every host it could be validated for, and
+home-boxed routes falsified it the moment a second edge armed: DNS for
+such a route may point at ANY edge, that edge backhauls to the home box
+with the real SNI, the home box (as issuer) orders — and the CA then
+validates against the box DNS names, whose registry has never heard of
+the host. Every authorization failed, each client retry re-triggered
+another through the backhaul, and five failures burned the name's LE
+per-hostname budget in minutes while the route was down (#277, observed
+in the dev topology).
+
+All three gates now admit a host when it is in this box's registry
+(`Allowed`, unchanged) **or** is an **EXACT** key of the mirrored home
+map (`homeMap.KnownHost`, folded from the same `control/edges/**` objects
+Part 3 already mirrors, read on the reader's 30s poll) — bridged in one
+place, `Edge.deploymentHost`. Both halves stay registry-backed and
+deployment-controlled: edges files are written by each box's own loop
+from its own registry, so a stranger cannot add a name without a control-
+bucket write, which Part 2's trust statement already treats as
+game-over. The widened set is exactly-named registry **routes**:
+`edgesFilePass` publishes `reg.List()`, which is the routes table and
+nothing else.
+
+**The one-label wildcard rule stays excluded, and that is load-bearing.**
+The home map carries `*.<parent>` catch-all rows verbatim, and `Home()`'s
+one-label fallback makes labels under them dialable. Admitting the
+fallback here would let a stranger's SNI or Host of
+`<anything>.<remote-parent>` drive fresh per-host ACME orders against the
+deployment's single shared account and rate-limit bucket, and spend the
+read-through budgets on invented names — exactly what PR #180's
+exact-registry `Allowed` rule exists to refuse. That rule therefore
+survives verbatim, merely widened from "this box's routes" to "this
+deployment's exactly-named routes"; coverage for labels under a catch-all
+remains the wildcard cert's job, as on a single box. Exact-only loses
+nothing: an order only ever exists for a name that passed the exact gate,
+so a challenge only ever needs answering for an exact name. Refusals are
+pinned by test on all three surfaces
+(`TestHostPolicyRefusesWildcardLabelUnderRemoteCatchAll`,
+`TestALPNReadThroughRefusesWildcardLabelUnderRemoteCatchAll`,
+`TestChallengeReadThroughStillRefusesStrangers`).
+
+*Residuals, filed rather than fixed.* Three, none of them blocking.
+
+**The order trigger stays on the issuer.** autocert orders lazily out of
+`GetCertificate`, so in a 3-plus-edge deployment where DNS points at
+neither the home box nor the issuer, no hello ever reaches the issuer and
+the name simply stays certless — **failing closed, with zero LE spend** (a
+non-issuer answers `errNotIssuer` before autocert's cache lookup, so
+backhauled hellos there order nothing). Steering heals the steered case
+(its latency records include the issuer edge) and the custom-domain CNAME
+guidance covers the rest; a proactive issuer-side order nudge for exact
+home-map names is the natural fix and is a follow-up, not an overnight
+change.
+
+**Attached custom domains are NOT covered.** A custom domain attached to
+an app that already has a ready route lands in `route_aliases`
+(`internal/console/domainattach.go`'s alias branch,
+`internal/domains/aliases.go`), never in the routes table — so it appears
+in no edges file and `KnownHost` is false for it on every box. Such a name
+is admissible only where it is an alias in that box's OWN registry
+(`Allowed` already resolves alias rows), which is unchanged from today.
+Publishing aliases into edges files is the obvious extension and is a
+follow-up issue, with one constraint stated in advance: an alias whose
+host is a `*.<parent>` catch-all must stay wildcard-cert-only if it is
+ever published — the same #180 bound this amendment refuses to widen.
+
+**Order amplification, accepted.** The set of names a stranger can induce
+a fresh ACME order for at the issuer grows from one box's routes to every
+box's routes. Each such order for a name whose DNS is dead burns that
+hostname's LE failed-validation budget and a slot of the shared account's
+new-order rate. This is a pre-existing class — any routed name has always
+been inducible on the box that homes it — amplified by deployment size
+rather than introduced here, and it is inside this ruling's accepted
+envelope: the alternative is leaving home-boxed routes unable to mint at
+all.
+
+Issue #277 carries this ruling and all three residuals.
+
+*Rollout.* Read-side only — no wire or storage shape, no new writer, no
+knob — and dormant on single-box deployments, where the fold is empty and
+every gate reduces to `Allowed`. Must be rolled to EVERY edge of a
+deployment before its second edge arms: a not-yet-rolled edge is exactly
+the refusing validator #277 describes. Until then, do not point real DNS
+at a non-home edge for a home-boxed name; that hour-long rate-limit has
+already been paid once.
+
+*A new route is not admissible instantly, and that is safe.* The 30s
+figure is the READER's poll; the publisher is `edgesFileLoop`'s 5min +
+up-to-1min jitter (`internal/host/edgesfile.go`), so a freshly created
+route can take ≈6.5 minutes worst case to become admissible on remote
+edges. During that window `Home()` misses the name too — the two read the
+same fold — so a non-home edge does not backhaul it: it answers the
+ordinary `unknownHost` 302 (or 404), no hello reaches the home box, and
+no order starts. The cost is a routing wait, not an LE burn. The one
+exception is a hello that reaches the issuer another way (a steered
+client, an operator's `--resolve`) while the issuer's own registry
+already has the row — which is simply today's single-box behaviour.
+
**Issuer death**: the lease expires and the next box's housekeeper takes
over with identical machinery. Distributed certs keep every edge serving
through the gap; a dead issuer costs nothing until the earliest renewal
@@ -216,6 +318,51 @@ rollout notes). Egress attribution for backhauled bytes is an open
ruling, not a defect: see the plan's rollout notes, T7 egress ruling
request.
+**Amendment (2026-08-24): the T7 egress ruling landed — backhaul is
+contained and hops are attributed.** The open ruling above is closed in
+two halves, both dormant on a single-box deployment.
+
+*Containment, on the SENDING edge.* At most
+`CARLOS_BACKHAUL_MAX_INFLIGHT` (default **64**; unset, unparseable or
+≤ 0 all mean the default, and there is deliberately no "off" value)
+backhauled requests may be in flight toward any ONE home box. At the
+limit the sender answers **503 + `Retry-After: 2`** itself — no dial, no
+cross-region bytes — and logs on the throttled per-home-box key
+`"backhaul-saturated "+box`. A concurrency bound rather than a rate,
+because cross-region egress is (concurrent streams × per-stream
+bandwidth): short requests flow at full rate forever and only sustained
+streams pile into the cap. A slot is held for the LIFE of the hop's
+response, streams and upgrades included — that IS the sustained egress
+being bounded. Sender-side because every backhauled byte crosses both
+legs, so one cap bounds both boxes at once; the home box needs no twin.
+
+*Attribution, on the HOME box.* The single existing meter site keeps
+counting every hop-served exchange exactly once, and on a verified hop it
+additionally records the same request and the same wire bytes into two
+new additive, `omitempty` fields on `usage.EdgeRow` — `hop_requests` /
+`hop_bytes_out`. They are a **subset annotation inside** `requests` /
+`bytes_out`, never an addition to them: a consumer that sums `bytes_out`
+has already counted every backhauled byte and must never add the hop
+fields on top. `EdgeDay.V` stays 1 (old objects parse, old binaries
+ignore the new keys, single-region objects stay byte-identical). The
+sending edge stays meterless by design — it has no route row for the
+host, so no account or app to attribute to.
+
+*Pricing stays with Paul.* The counters ship dormant: `backhaul_gb` is
+reserved in the meter vocabulary and wired into nothing — no rate-card
+line, no statement — so bills are byte-identical to before and the
+decision (price backhauled bytes, or absorb them as a platform cost of
+any-edge serving) can be made later on real data. Default taken tonight:
+**absorbed**.
+
+*Rollout.* Must be rolled to every box of a deployment BEFORE that
+deployment arms a second edge — the home box needs the new binary to
+annotate hops, the senders need the cap. It slots ahead of the steering
+plan's arming checklist step 0. Mixed versions are safe in both
+directions: an old-binary home box writes rows without hop fields
+(under-annotated, never wrong), and an old-binary sender is simply
+uncapped until rolled, which is today's behaviour.
+
| Class | Kinds | Behaviour on a non-home edge |
|---|---|---|
| Everywhere | static, bake-served | adopt and serve locally from converged artifacts — zero backhaul |
diff --git a/internal/edge/backhaul_e2e_test.go b/internal/edge/backhaul_e2e_test.go
index 98eca40ba..5716803e0 100644
--- a/internal/edge/backhaul_e2e_test.go
+++ b/internal/edge/backhaul_e2e_test.go
@@ -760,6 +760,56 @@ func TestBackhaulE2EAcrossTwoEdges(t *testing.T) {
})
}
+// ---- attribution: the hop annotation, end to end -----------------------------
+
+// The attribution half of the egress ruling, on the real two-edge scene: a
+// backhauled request is metered ONCE, on the box that holds the route's
+// authoritative row, and that one row also carries the hop annotation. The
+// sending edge records nothing at all for the host — it has no route row and
+// therefore no account or app to bill, which is exactly why the annotation
+// lives here and not there.
+func TestHopServedRequestMetersOnceWithHopCounters(t *testing.T) {
+ s := newBackhaulScene(t)
+ s.converge(t)
+
+ home, sender := NewUsageMeter(), NewUsageMeter()
+ s.a.edge.EnableUsage(home)
+ s.b.edge.EnableUsage(sender)
+
+ resp, body := s.get(t, s.b, "https://"+e2eHomedHost+"/deep?q=1", nil, true)
+ if resp.StatusCode != http.StatusOK {
+ t.Fatalf("status = %d, want 200", resp.StatusCode)
+ }
+ if seen := e2eParseApp(t, body); seen.marker != e2eHomeMarker {
+ t.Fatalf("marker = %q, want the home edge's app", seen.marker)
+ }
+
+ rows := home.Drain()
+ row, ok := rows[e2eHomedHost]
+ if !ok {
+ t.Fatalf("the home edge metered nothing for %s: %+v", e2eHomedHost, rows)
+ }
+ if row.Requests != 1 {
+ t.Fatalf("row = %+v, want exactly one request — a hop is ONE metered exchange, not two", row)
+ }
+ if row.BytesOut == 0 {
+ t.Fatalf("row = %+v, want the wire bytes the home edge sent", row)
+ }
+ if row.HopRequests != 1 || row.HopBytesOut != row.BytesOut {
+ t.Fatalf("row = %+v, want the whole exchange annotated as hopped (hop_bytes_out == bytes_out)", row)
+ }
+ if row.HopBytesOut > row.BytesOut || row.HopRequests > row.Requests {
+ t.Fatalf("row = %+v — the hop counters are a subset of the totals", row)
+ }
+
+ // The sending edge is meterless for this host BY DESIGN: no route row, no
+ // identity, so anything it wrote would be unattributed — and a second
+ // Record for one exchange is the double count this arrangement refuses.
+ if got, ok := sender.Drain()[e2eHomedHost]; ok {
+ t.Fatalf("the sending edge metered %+v for a host it has no route for", got)
+ }
+}
+
// ---- the single-box regression pin -------------------------------------------
// Everything above is dormant on a deployment with one platform box, and this
diff --git a/internal/edge/certsync.go b/internal/edge/certsync.go
index a1c0c23d7..06eaeabc3 100644
--- a/internal/edge/certsync.go
+++ b/internal/edge/certsync.go
@@ -141,7 +141,15 @@ func (e *Edge) armCertIssuance(inner autocert.Cache) autocert.Cache {
}
// issuerHostPolicy is the autocert HostPolicy: the registry check this edge
-// has always applied (Allowed), plus the issuance gate.
+// has always applied, plus the issuance gate.
+//
+// The registry check is deploymentHost (edge.go) rather than Allowed: a
+// deployment's issuer must be able to mint for any EXACTLY-named route in the
+// deployment, not only the ones this box happens to home, or a home-boxed
+// route whose hellos reach the issuer through the backhaul can never get a
+// certificate at all (#277). Exact-only, so PR #180's rule is untouched: a
+// label under a catch-all still drives no order. The errNotIssuer gate below
+// is unaffected — non-issuers still order nothing, for any name.
//
// Two autocert call sites reach this, and both are deliberate:
// GetCertificate consults the policy BEFORE its cache lookup
@@ -176,7 +184,7 @@ func (e *Edge) armCertIssuance(inner autocert.Cache) autocert.Cache {
// order for a deployment that can stop issuing altogether.
func (e *Edge) issuerHostPolicy() autocert.HostPolicy {
return func(_ context.Context, host string) error {
- if !e.Allowed(host) {
+ if !e.deploymentHost(host) {
return fmt.Errorf("acme: host %q not in registry", host)
}
if _, ok := e.issuerGate(); !ok {
@@ -420,9 +428,12 @@ type certMirror struct {
// instead of on the manager's own 12h tick. nil when wildcards are off.
wake func()
- // allowed reports whether a hostname is one this box serves (Edge.Allowed)
- // — the bound on the tls-alpn-01 read-through, whose key is
- // attacker-chosen SNI. nil = unbounded, which is tests only.
+ // allowed reports whether a hostname is one this box serves, or one
+ // another box in this deployment exactly homes (Edge.deploymentHost) —
+ // the bound on BOTH read-throughs, whose keys are attacker-chosen SNI and
+ // an attacker-chosen Host header. Exact names only, either side: a label
+ // under a catch-all spends neither budget (#180, and KnownHost's doc).
+ // nil = unbounded, which is tests only.
allowed func(string) bool
// gated reports whether issuance is currently BARRED on this box, which
@@ -485,11 +496,15 @@ func (e *Edge) newCertMirror(local autocert.Cache) *certMirror {
_, held := e.issuerGate()
return !held
}
- // The tls-alpn-01 read-through is keyed by attacker-chosen SNI, so it is
- // bounded to the hosts this box actually serves. Allowed is the same
- // exact-match registry check autocert's HostPolicy uses, which is the
- // right bound: a per-host order only ever exists for a name that passed it.
- m.allowed = e.Allowed
+ // Both read-throughs are keyed by remote input (SNI on 443, the Host
+ // header on :80), so they are bounded to the names this DEPLOYMENT serves:
+ // this box's own routes, plus the exact names another box's edges file
+ // homes. deploymentHost is the same exact-match, registry-backed check
+ // autocert's HostPolicy uses (issuerHostPolicy, above), which is the right
+ // bound for the same reason it always was — a per-host order only ever
+ // exists for a name that passed it — now that the policy admits the
+ // deployment's remote-homed routes too (#277).
+ m.allowed = e.deploymentHost
return m
}
@@ -1160,7 +1175,10 @@ func (m *certMirror) challengeResponse(ctx context.Context, host, token string)
// r.Host first, autocert.go:405). A real CA always sends the domain it is
// validating, so the bound costs a genuine validation nothing — and
// without it an unauthenticated :80 flood sustains a bucket GET per
- // distinct path, forever, from anywhere.
+ // distinct path, forever, from anywhere. The bound is the DEPLOYMENT's
+ // exact names, not this box's (m.allowed, Edge.deploymentHost): DNS for a
+ // home-boxed route can point at any edge, and the one it points at is the
+ // one the CA arrives at (#277).
if m.allowed != nil && !m.allowed(host) {
return nil, false
}
@@ -1205,8 +1223,10 @@ func validChallengeToken(token string) bool {
// validation with probability (N-1)/N before it ever tries the HTTP path.
//
// Bounded harder than the HTTP-01 path, because the name here is
-// attacker-chosen SNI rather than a high-entropy token: only hosts this box
-// actually serves are ever looked up.
+// attacker-chosen SNI rather than a high-entropy token: only names this
+// deployment serves — this box's own routes, or another box's exactly-named
+// ones (m.allowed, Edge.deploymentHost) — are ever looked up. A stranger's
+// invented label under a mirrored catch-all is not one of them.
func (m *certMirror) alpnTokenCert(ctx context.Context, host string) *tls.Certificate {
if host == "" {
return nil
diff --git a/internal/edge/certsync_e2e_test.go b/internal/edge/certsync_e2e_test.go
index e7f538956..03de19a88 100644
--- a/internal/edge/certsync_e2e_test.go
+++ b/internal/edge/certsync_e2e_test.go
@@ -1155,3 +1155,144 @@ func TestCertSyncE2EConcurrentRefreshIsRaceFree(t *testing.T) {
t.Fatalf("account key seeding raced: %q, %v", got, err)
}
}
+
+// ---- #277: the non-home edge answers, and the home-boxed name mints ---------
+
+// e2eRemoteHomedHost is homed on edge A and absent from edge B's registry —
+// the shape the original scenario deliberately did not have (there, B carried
+// every host in its own route table).
+const e2eRemoteHomedHost = "remote.bab.example.test"
+
+// The automated shadow of the dev proof that failed as #277: DNS for an
+// exactly-named registry ROUTE points at edge B, which homes nothing for that
+// name. B backhauls to A, A (the issuer) sees the hello and orders, and the CA
+// then validates against B — the box whose registry has never heard of the
+// host. Before the widening, B refused the challenge, every authorization
+// failed, and the name's Let's Encrypt budget went in minutes. After it, B
+// answers out of the bucket, A mints, and the certificate reaches B's mirror.
+//
+// A route, deliberately, and not an attached custom domain: an alias attach
+// lands in route_aliases (internal/domains/aliases.go), which edgesFilePass
+// never publishes — so no alias is in any home map, on any box. The spec
+// amendment records that as an open follow-up rather than a claim this test
+// could make.
+//
+// The backhaul leg itself is not re-proved here (homemap_test.go owns it);
+// what this pins is that everything CERT-side works with the host in ONE
+// box's registry and the other box's home map.
+func TestNonHomeEdgeAnswersValidationForRemoteHomedHost(t *testing.T) {
+ e2eQuietLogs(t)
+ bucket := t.TempDir()
+ ca := newFakeACME(t)
+
+ // Edge B: no routes at all for the name — this is the box DNS points at.
+ edgeB := newE2EEdge(t, bucket, false, ca.directoryURL())
+ ca.validateAgainst(edgeB.serveHTTP(t))
+ if edgeB.edge.Allowed(e2eRemoteHomedHost) {
+ t.Fatal("test bug: edge B must not carry the host in its own registry")
+ }
+
+ // ...but it mirrors the deployment's edges files, in which edge A homes
+ // the name exactly. That, and only that, is what admits the validation.
+ homes := newHomeMap(release.NewDirStore(bucket), "edge-b")
+ homedOn(t, homes, "edge-a", "203.0.113.10", e2eRemoteHomedHost, "*.other.example.test")
+ edgeB.edge.homes.Store(homes)
+
+ // Edge A: the housekeeper, and the box that homes the route. Its own :80
+ // chain is served purely so "the CA never reached the issuer" is an
+ // assertion rather than an assumption.
+ edgeA := newE2EEdge(t, bucket, true, ca.directoryURL(), e2eRemoteHomedHost)
+ edgeA.serveHTTP(t)
+
+ lnB := edgeB.serveTLS(t)
+ if _, err := e2eDial(lnB, e2eRemoteHomedHost, ca.rootPool()); err == nil {
+ t.Fatal("edge B answered a handshake for a certificate that does not exist yet")
+ }
+
+ // The mint, on edge A — exactly what a backhauled hello triggers there.
+ minted, err := edgeA.getCert(ecdsaHello(e2eRemoteHomedHost))
+ if err != nil {
+ t.Fatalf("the issuer failed to mint a host it homes: %v", err)
+ }
+ ca.checkFaults(t)
+ if minted.Leaf == nil || minted.Leaf.VerifyHostname(e2eRemoteHomedHost) != nil {
+ t.Fatalf("minted cert does not cover %s", e2eRemoteHomedHost)
+ }
+
+ t.Run("the validation was answered by the box that homes nothing", func(t *testing.T) {
+ v, ok := ca.lastValidation()
+ if !ok {
+ t.Fatal("the CA never validated anything")
+ }
+ if !v.ok || v.host != e2eRemoteHomedHost {
+ t.Fatalf("validation of %q ok=%v — the body B served was not the key authorization", v.host, v.ok)
+ }
+ if n := edgeB.hits.Load(); n == 0 {
+ t.Fatal("edge B's :80 chain was never reached")
+ }
+ if n := edgeA.hits.Load(); n != 0 {
+ t.Fatalf("edge A's :80 chain served %d request(s) — the CA must never have reached the issuer", n)
+ }
+ // It came out of the bucket, live: B never polled, and its own
+ // autocert handler is shut for this name by the issuer gate.
+ if n := len(edgeB.mirror.snapshot().entries); n != 0 {
+ t.Fatalf("edge B's mirror holds %d entry/entries — the read-through is not what was tested", n)
+ }
+ if err := edgeB.edge.issuerHostPolicy()(context.Background(), e2eRemoteHomedHost); !errors.Is(err, errNotIssuer) {
+ t.Fatalf("edge B's HostPolicy = %v, want errNotIssuer — a non-issuer orders nothing, however it is named", err)
+ }
+ want := certKey(v.token + httpTokenSuffix)
+ reads, _ := edgeB.store.snapshot()
+ if !slices.Contains(reads, want) {
+ t.Fatalf("edge B read %v, want a live read of %q", reads, want)
+ }
+ })
+
+ t.Run("the tls-alpn-01 twin is admitted for the same name", func(t *testing.T) {
+ // The fake CA offers http-01 only (its doc says why), so the ALPN half
+ // is proved at the same seam the unit tests use: a token cert in the
+ // bucket, an acme-tls/1 hello at edge B for a name only edge A homes.
+ // Written through a THIRD handle on the bucket, standing in for the
+ // issuer's publishingCache: edge B's own handle must stay write-free,
+ // which the last subtest asserts.
+ issuerHandle := release.NewDirStore(bucket)
+ putBucket(t, issuerHandle, e2eRemoteHomedHost+alpnTokenSuffix,
+ hostCertPEM(t, e2eRemoteHomedHost, time.Now().Add(24*time.Hour)))
+ if c := edgeB.mirror.alpnTokenCert(context.Background(), e2eRemoteHomedHost); c == nil {
+ t.Fatal("edge B refused a tls-alpn-01 validation for a host edge A homes")
+ }
+ // And the label under the mirrored catch-all is still refused (#180).
+ // The token cert for it is PUT FIRST and the label is asserted
+ // dialable: without both, a nil answer here would prove only that the
+ // bucket is empty, and the refusal would pass even if KnownHost leaked
+ // through Home()'s one-label rule.
+ const invented = "invented.other.example.test"
+ putBucket(t, issuerHandle, invented+alpnTokenSuffix,
+ hostCertPEM(t, invented, time.Now().Add(24*time.Hour)))
+ if _, _, ok := edgeB.edge.homes.Load().Home(invented); !ok {
+ t.Fatal("test bug: the invented label is supposed to be DIALABLE through the one-label rule")
+ }
+ if c := edgeB.mirror.alpnTokenCert(context.Background(), invented); c != nil {
+ t.Fatal("a label under a mirrored catch-all was served a token cert")
+ }
+ })
+
+ t.Run("the minted cert reaches the non-home edge", func(t *testing.T) {
+ if err := edgeB.mirror.Refresh(context.Background()); err != nil {
+ t.Fatalf("one refresh must be enough: %v", err)
+ }
+ state, err := e2eDial(lnB, e2eRemoteHomedHost, ca.rootPool())
+ if err != nil {
+ t.Fatalf("edge B must serve the certificate edge A minted: %v", err)
+ }
+ if len(state.PeerCertificates) == 0 || !bytes.Equal(state.PeerCertificates[0].Raw, minted.Leaf.Raw) {
+ t.Fatal("edge B served a certificate that is not the one edge A minted")
+ }
+ _, wrote := edgeB.store.snapshot()
+ if len(wrote) != 0 {
+ t.Fatalf("a gated edge wrote %v to the control bucket — only the issuer may", wrote)
+ }
+ })
+
+ ca.checkFaults(t)
+}
diff --git a/internal/edge/certsync_test.go b/internal/edge/certsync_test.go
index 5934656b5..7e830e3c1 100644
--- a/internal/edge/certsync_test.go
+++ b/internal/edge/certsync_test.go
@@ -890,7 +890,14 @@ const mirrorHost = "app.bab.oncarlos.com"
// pass on the other.
func gatedChain(t *testing.T, m *certMirror, local autocert.Cache, hosts ...string) (func(*tls.ClientHelloInfo) (*tls.Certificate, error), *countingCache) {
t.Helper()
- e := testIssuerEdge(hosts...)
+ return gatedChainFor(t, testIssuerEdge(hosts...), m, local)
+}
+
+// gatedChainFor is gatedChain over an edge the caller built — the form a test
+// needs when the edge carries something beyond its route table, such as the
+// mirrored home map the cert gates now consult (deploymentHost, #277).
+func gatedChainFor(t *testing.T, e *Edge, m *certMirror, local autocert.Cache) (func(*tls.ClientHelloInfo) (*tls.Certificate, error), *countingCache) {
+ t.Helper()
e.issuer = func() (context.Context, bool) { return nil, false } // never the issuer
counting := &countingCache{Cache: local, gets: map[string]int{}}
mgr := offlineManager(t, counting, e.issuerHostPolicy())
@@ -1059,15 +1066,17 @@ func TestMirrorAnswersAnotherEdgesALPNValidation(t *testing.T) {
}
}
-// The SNI here is attacker-chosen, so the live lookup is bounded to hosts
-// this box actually serves.
-func TestMirrorALPNReadThroughOnlyForServedHosts(t *testing.T) {
+// The SNI here is attacker-chosen, so the live lookup is bounded to the names
+// this DEPLOYMENT serves — this box's own routes, or another box's
+// exactly-named ones (#277). A stranger is refused before any bucket read, as
+// it always was.
+func TestMirrorALPNReadThroughOnlyForServedOrRemoteHomedHosts(t *testing.T) {
s := &bucketGetCounter{Store: release.NewDirStore(t.TempDir())}
m := newCertMirror(s, nil)
get, _ := gatedChain(t, m, autocert.DirCache(t.TempDir()), mirrorHost)
m.allowed = func(string) bool { return false }
if _, err := get(alpnHello("stranger.example")); err == nil {
- t.Fatal("there is no token cert for a host this box does not serve")
+ t.Fatal("there is no token cert for a host this deployment does not serve")
}
if n := s.gets(); n != 0 {
t.Fatalf("bucket gets = %d, want 0 for an unserved host", n)
@@ -1124,17 +1133,20 @@ const mirrorTestToken = "0123456789abcdefghij0123456789abcdefghijkl"
func challengeRequest(t *testing.T, h http.Handler, token string) *httptest.ResponseRecorder {
t.Helper()
- r := httptest.NewRequest("GET", "http://"+mirrorHost+"/.well-known/acme-challenge/"+token, nil)
- w := httptest.NewRecorder()
- h.ServeHTTP(w, r)
- return w
+ return challengeRequestHost(t, h, mirrorHost, token)
}
// gatedHTTPHandler is the :80 chain of a non-issuer: the mirror's handler in
// front of autocert's, which 403s every challenge on this box.
func gatedHTTPHandler(t *testing.T, m *certMirror) http.Handler {
t.Helper()
- e := testIssuerEdge(mirrorHost)
+ return gatedHTTPHandlerFor(t, testIssuerEdge(mirrorHost), m)
+}
+
+// gatedHTTPHandlerFor is gatedHTTPHandler over an edge the caller built, for
+// the same reason gatedChainFor exists.
+func gatedHTTPHandlerFor(t *testing.T, e *Edge, m *certMirror) http.Handler {
+ t.Helper()
e.issuer = func() (context.Context, bool) { return nil, false }
mgr := offlineManager(t, autocert.DirCache(t.TempDir()), e.issuerHostPolicy())
fallback := http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
@@ -1669,3 +1681,173 @@ func TestChallengeFloodDoesNotStarveALPNRescue(t *testing.T) {
t.Fatalf("a :80 flood starved the 443 validation path: %v", err)
}
}
+
+// ---- #277: the three gates admit the deployment's exactly-named hosts -------
+//
+// DNS for a home-boxed route can point at ANY edge. The edge it points at
+// backhauls to the home box with the real SNI, the home box (as issuer)
+// orders — and the CA then validates against the box DNS names, which is the
+// one whose registry does not carry the host. Bounding the gates by Allowed
+// meant every such validation failed, five failures burned the name's Let's
+// Encrypt per-hostname budget in minutes, and the route was down throughout.
+//
+// The gates now ask deploymentHost: this box's routes, OR an exact key of the
+// mirrored home map. Exact is the whole discipline — every test below that
+// says "wildcard label" is PR #180's rule standing on the new surface.
+
+// remoteHomedEdge is one box of a two-box deployment: it serves served from
+// its own registry, and mirrors an edges file in which box "betty" homes
+// homed (pass "*.<parent>" to mirror a catch-all row, which Task 1 publishes
+// verbatim). The handle it issues and mirrors through COUNTS bucket Gets, so
+// "did this gate spend a read-through?" is observable; the home map polls the
+// same directory through its own handle, so the map's own reads never show up
+// in that count.
+func remoteHomedEdge(t *testing.T, served []string, homed ...string) (*Edge, *bucketGetCounter, release.Store) {
+ t.Helper()
+ base := release.NewDirStore(t.TempDir())
+ counting := &bucketGetCounter{Store: base}
+ e := testIssuerEdge(served...)
+ e.staticStore = counting
+ m := newHomeMap(base, "annie")
+ homedOn(t, m, "betty", "2.2.2.2", homed...)
+ e.homes.Store(m)
+ return e, counting, base
+}
+
+const (
+ // remoteHomedHost is homed on the OTHER box: absent from this box's
+ // registry, exactly named in its home map.
+ remoteHomedHost = "app.remote.test"
+ // remoteCatchAll is the catch-all row that box publishes, and
+ // remoteWildcardLabel is a label under it — dialable through Home()'s
+ // one-label rule, and admissible by none of the three cert gates.
+ remoteCatchAll = "*.remote.test"
+ remoteWildcardLabel = "invented.remote.test"
+)
+
+// HTTP-01, the failing half of #277: the CA arrives at a box that does not
+// serve the name, and the token is in the bucket. It is answered.
+func TestChallengeReadThroughAdmitsRemoteHomedHost(t *testing.T) {
+ e, counting, base := remoteHomedEdge(t, []string{mirrorHost}, remoteHomedHost)
+ putBucket(t, base, mirrorTestToken+"+http-01", []byte("keyauth"))
+ m := e.newCertMirror(nil) // production wiring: m.allowed = e.deploymentHost
+ if m == nil {
+ t.Fatal("a store-backed edge must have a mirror")
+ }
+ if e.Allowed(remoteHomedHost) {
+ t.Fatal("test bug: the host is supposed to be absent from THIS box's registry")
+ }
+ got := challengeRequestHost(t, gatedHTTPHandlerFor(t, e, m), remoteHomedHost, mirrorTestToken)
+ if got.Code != 200 || got.Body.String() != "keyauth" {
+ t.Fatalf("challenge for a remote-homed host = %d %q, want 200 \"keyauth\"", got.Code, got.Body.String())
+ }
+ if n := counting.gets(); n != 1 {
+ t.Fatalf("bucket gets = %d, want exactly 1 — the live read-through", n)
+ }
+}
+
+// ...and the anti-stranger bound the widening had to survive: a name in
+// neither the registry nor the map spends nothing at all, and neither does an
+// invented label under a MIRRORED catch-all (#180).
+func TestChallengeReadThroughStillRefusesStrangers(t *testing.T) {
+ for _, host := range []string{"stranger.example", remoteWildcardLabel, remoteCatchAll} {
+ t.Run(host, func(t *testing.T) {
+ e, counting, base := remoteHomedEdge(t, []string{mirrorHost}, remoteHomedHost, remoteCatchAll)
+ putBucket(t, base, mirrorTestToken+"+http-01", []byte("keyauth"))
+ m := e.newCertMirror(nil)
+ if got := challengeRequestHost(t, gatedHTTPHandlerFor(t, e, m), host, mirrorTestToken); got.Code == 200 {
+ t.Fatalf("%s was answered a live token body", host)
+ }
+ if n := counting.gets(); n != 0 {
+ t.Fatalf("bucket gets = %d, want 0 — %s must never drive a bucket read", n, host)
+ }
+ })
+ }
+}
+
+// tls-alpn-01, the half autocert tries FIRST: the validation cert for another
+// edge's order is served for a remote-homed name.
+func TestMirrorALPNReadThroughAdmitsRemoteHomedHost(t *testing.T) {
+ e, _, base := remoteHomedEdge(t, []string{mirrorHost}, remoteHomedHost)
+ putBucket(t, base, remoteHomedHost+"+token", hostCertPEM(t, remoteHomedHost, time.Now().Add(24*time.Hour)))
+ m := e.newCertMirror(nil)
+ get, _ := gatedChainFor(t, e, m, autocert.DirCache(t.TempDir()))
+ // Deliberately NOT refreshed: a validation is live, and the poll is 30s.
+ if _, err := get(alpnHello(remoteHomedHost)); err != nil {
+ t.Fatalf("an edge DNS points at must answer the home box's tls-alpn-01 validation: %v", err)
+ }
+}
+
+// The #180 pin on the 443 surface: SNI is attacker-chosen, and a mirrored
+// catch-all must not turn an invented label into a bucket read.
+func TestALPNReadThroughRefusesWildcardLabelUnderRemoteCatchAll(t *testing.T) {
+ e, counting, base := remoteHomedEdge(t, []string{mirrorHost}, remoteCatchAll)
+ putBucket(t, base, remoteWildcardLabel+"+token", hostCertPEM(t, remoteWildcardLabel, time.Now().Add(24*time.Hour)))
+ if _, _, ok := e.homes.Load().Home(remoteWildcardLabel); !ok {
+ t.Fatal("test bug: the label is supposed to be DIALABLE through the one-label rule")
+ }
+ m := e.newCertMirror(nil)
+ get, _ := gatedChainFor(t, e, m, autocert.DirCache(t.TempDir()))
+ if _, err := get(alpnHello(remoteWildcardLabel)); err == nil {
+ t.Fatal("a label under a mirrored catch-all must not be served a token cert")
+ }
+ if n := counting.gets(); n != 0 {
+ t.Fatalf("bucket gets = %d, want 0 — the label must not spend the ALPN budget", n)
+ }
+}
+
+// The order gate: the issuer mints for any exactly-named route in the
+// deployment, whichever box homes it. Without this the home-boxed route can
+// only ever mint while the housekeeper lease sits on its own box.
+func TestIssuerOrdersForRemoteHomedHost(t *testing.T) {
+ e, _, _ := remoteHomedEdge(t, []string{mirrorHost}, remoteHomedHost)
+ e.issuer = func() (context.Context, bool) { return context.Background(), true }
+ if err := e.issuerHostPolicy()(context.Background(), remoteHomedHost); err != nil {
+ t.Fatalf("HostPolicy for a remote-homed host = %v, want nil", err)
+ }
+ if err := e.issuerHostPolicy()(context.Background(), mirrorHost); err != nil {
+ t.Fatalf("HostPolicy for this box's own route = %v, want nil", err)
+ }
+}
+
+// The #180 pin on the order gate, and the one that matters most: an
+// attacker-chosen label must never drive a fresh per-host ACME order against
+// the deployment's single shared account and rate-limit bucket.
+func TestHostPolicyRefusesWildcardLabelUnderRemoteCatchAll(t *testing.T) {
+ e, _, _ := remoteHomedEdge(t, []string{mirrorHost}, remoteCatchAll)
+ e.issuer = func() (context.Context, bool) { return context.Background(), true }
+ for _, host := range []string{remoteWildcardLabel, remoteCatchAll, "stranger.example"} {
+ err := e.issuerHostPolicy()(context.Background(), host)
+ if err == nil {
+ t.Fatalf("%s: HostPolicy allowed an order", host)
+ }
+ if !strings.Contains(err.Error(), "not in registry") {
+ t.Fatalf("%s: HostPolicy = %v, want the unchanged not-in-registry refusal", host, err)
+ }
+ }
+}
+
+// Widening the NAMES a policy admits does not widen the BOXES that may order:
+// a non-issuer still refuses every name, remote-homed ones included.
+func TestHostPolicyStillGatesNonIssuer(t *testing.T) {
+ e, _, _ := remoteHomedEdge(t, []string{mirrorHost}, remoteHomedHost)
+ e.issuer = func() (context.Context, bool) { return nil, false }
+ for _, host := range []string{remoteHomedHost, mirrorHost} {
+ if err := e.issuerHostPolicy()(context.Background(), host); !errors.Is(err, errNotIssuer) {
+ t.Fatalf("%s on a non-issuer = %v, want errNotIssuer", host, err)
+ }
+ }
+}
+
+// Single-box deployments are byte-for-byte unchanged: no home map armed (or an
+// empty one), so deploymentHost reduces to Allowed.
+func TestDeploymentHostIsAllowedOnASingleBoxDeployment(t *testing.T) {
+ e := testIssuerEdge(mirrorHost) // e.homes never stored: nil pointer
+ if !e.deploymentHost(mirrorHost) || e.deploymentHost("stranger.example") {
+ t.Fatal("with no home map at all, deploymentHost must be exactly Allowed")
+ }
+ e2, _, _ := remoteHomedEdge(t, []string{mirrorHost}) // map armed, folds to nothing
+ if !e2.deploymentHost(mirrorHost) || e2.deploymentHost("stranger.example") {
+ t.Fatal("with an empty home map, deploymentHost must be exactly Allowed")
+ }
+}
diff --git a/internal/edge/edge.go b/internal/edge/edge.go
index 19a565b38..7c712f448 100644
--- a/internal/edge/edge.go
+++ b/internal/edge/edge.go
@@ -194,6 +194,13 @@ type Edge struct {
// backhaul target, and keying per host would let a stranger's stream of
// invented hostnames grow this map without limit.
backhaulTransports sync.Map
+ // backhaulInflight counts the hops currently in flight toward each home
+ // box — the containment half of the egress ruling (backhaulAcquire).
+ // Keyed by BOX for the same reason backhaulTransports is keyed by EIP: a
+ // catch-all row in another box's edges file makes every invented
+ // subdomain a backhaul target, so a per-host key would let a stranger
+ // grow this map without limit. Bounded by the deployment's box count.
+ backhaulInflight sync.Map // box → *atomic.Int64
}
// RunOption configures the Edge that Run/RunWithActivator* builds, before it
@@ -760,6 +767,37 @@ func (e *Edge) Allowed(host string) bool {
return ok
}
+// deploymentHost reports whether host is served HERE (Allowed) or is exactly
+// named by another box's edges file (homeMap.KnownHost) — the admission bound
+// for the three cert-adjacent gates in certsync.go: issuerHostPolicy's order
+// gate and the HTTP-01 and tls-alpn-01 read-throughs.
+//
+// The widening exists because Allowed alone was the wrong bound the moment a
+// deployment grew a second edge (#277): DNS for a home-boxed route can point
+// at any edge, the backhaul then dials the home box with the real SNI, the
+// home box orders — and the CA validates against whichever edge DNS names,
+// which refused every challenge for a host outside its OWN registry. Five
+// such failures burn the name's Let's Encrypt per-hostname budget in minutes
+// while the route is down. Both halves are registry-backed and
+// deployment-controlled: the home map is folded from control/edges/<box>.json,
+// which each box writes from its own registry (internal/host's edgesFilePass),
+// so a stranger cannot add a name to it without a control-bucket write.
+//
+// EXACT only, on both halves: Allowed refuses the wildcard key and consults
+// the exact route map, KnownHost refuses "*." and skips Home()'s one-label
+// fallback. That is PR #180's rule, unchanged and merely widened from "this
+// box's routes" to "this deployment's exactly-named routes" — an
+// attacker-chosen label under a catch-all must never drive an ACME order or a
+// bucket read.
+//
+// Both halves take the same normalized form: serverName/requestHost lowercase
+// and punycode into the shape registry rows, and therefore edges files, are
+// written in. e.homes may hold nil before the map arms; KnownHost is
+// nil-receiver-safe, matching Home's pattern.
+func (e *Edge) deploymentHost(host string) bool {
+ return e.Allowed(host) || e.homes.Load().KnownHost(host)
+}
+
// lookupUpstream resolves host to its upstream entry — the request-routing
// lookup ServeHTTP uses (NOT Allowed's autocert HostPolicy — see the comment
// there). An exact match (a real route, or a plain #151 alias landed in the
@@ -871,10 +909,21 @@ func (e *Edge) ServeHTTP(w http.ResponseWriter, r *http.Request) {
// returns above (edge.go:531-544) — that hostname isn't a registry
// route, so up.route isn't defined yet there, and neither one is
// billable traffic.
+ //
+ // hopped is the ATTRIBUTION half of the backhaul egress ruling
+ // (2026-08-24): a request that arrived over a verified hop is metered
+ // here exactly once, as it always was, and additionally annotated into
+ // the row's subset hop counters — so every backhauled byte is attributed
+ // to the route, account and app that caused it, on the box that holds
+ // the authoritative row. The sending edge stays meterless by design: it
+ // has no route row for the host and so no identity to bill. Nothing
+ // about the wrapper stack moves for this — recorder above the
+ // compressor, this counter below it — because the annotation changes
+ // WHAT is written at drain, never WHERE the bytes are counted.
if m := e.usageMeter.Load(); m != nil && up.route.Kind != "status" {
var bytes int64
w = &countingWriter{ResponseWriter: w, n: &bytes}
- defer func() { m.Record(up.route, 1, bytes) }()
+ defer func() { m.Record(up.route, 1, bytes, hopped) }()
}
// A Connection: Upgrade request (a WebSocket handshake, typically GET)
@@ -1785,6 +1834,86 @@ func fleetProxy(fleet, boxLabel string, hub *TunnelHub) *httputil.ReverseProxy {
// every request and this whole path is dead code until a second box publishes
// its edges file.
+// backhaulMaxInflight is the containment ruling of 2026-08-24: at most this
+// many backhauled requests may be in flight toward any ONE home box at a
+// time, counted on the SENDING edge. At the limit the sender answers 503 +
+// Retry-After: 2 (backhaul, below) rather than dialling.
+//
+// A concurrency bound and not a rate: cross-region egress is (concurrent
+// streams × per-stream bandwidth), so a request-rate cap would neither bound
+// a handful of long-lived streams — a hop carries WebSockets and has no
+// whole-body deadline by design — nor tolerate a legitimate burst. This
+// degrades the other way round: short requests flow at full rate forever, and
+// only slow, sustained streams pile into the cap.
+//
+// Sender-side because every backhauled byte traverses BOTH legs (home →
+// sender, sender → client), so one cap here bounds both boxes' egress at
+// once; the home box needs no twin, its inbound hops being bounded by the sum
+// of its siblings' caps.
+//
+// A var, not a const, so a test can shorten it — the same reason backhaulPort
+// is one. There is deliberately no "off" value: an unbounded cross-region
+// path is the thing this closes.
+//
+// Atomic rather than a plain int because it is READ on the request path by
+// every backhauling goroutine: it is written once today, before any listener
+// binds, but a future env re-read (or a second Edge composed in one process)
+// would turn a plain int into a real data race rather than a stale read.
+var backhaulMaxInflight atomic.Int64
+
+const defaultBackhaulMaxInflight = 64
+
+func init() { backhaulMaxInflight.Store(defaultBackhaulMaxInflight) }
+
+// backhaulInflightFromEnv reads CARLOS_BACKHAUL_MAX_INFLIGHT. Unset,
+// unparseable or ≤ 0 all mean the default: a typo'd knob must not silently
+// uncap (or, worse, zero-cap) a deployment's backhaul.
+func backhaulInflightFromEnv() int {
+ raw := strings.TrimSpace(os.Getenv("CARLOS_BACKHAUL_MAX_INFLIGHT"))
+ if raw == "" {
+ return defaultBackhaulMaxInflight
+ }
+ n, err := strconv.Atoi(raw)
+ if err != nil || n <= 0 {
+ log.Printf("edge: CARLOS_BACKHAUL_MAX_INFLIGHT=%q is not a positive integer — using the default of %d concurrent backhauled requests per home box", raw, defaultBackhaulMaxInflight)
+ return defaultBackhaulMaxInflight
+ }
+ return n
+}
+
+// backhaulAcquire reserves one in-flight slot toward box, or reports
+// saturation. releaseSlot must be called exactly once when the hop's
+// ServeHTTP returns — a streaming or upgraded response holds its slot for the
+// life of the stream, deliberately: a long-lived stream IS sustained
+// cross-region egress, which is the thing being bounded.
+//
+// The counter is per home BOX and the map is never keyed by anything a
+// stranger controls (backhaulInflight's own doc). The count is incremented
+// FIRST and rolled back on refusal, so the cap can never be exceeded: two
+// requests racing at the limit both see the excess and both refuse, and a
+// request arriving inside another's rollback window can be refused while a
+// slot is in fact free. Both errors are in the safe direction — a refusal is
+// a 503 the client retries in two seconds, where an over-admission would be
+// cross-region bytes the cap exists to prevent.
+func (e *Edge) backhaulAcquire(box string) (releaseSlot func(), ok bool) {
+ // Load first, like backhaulTransport: LoadOrStore would allocate a
+ // counter per backhauled request just to throw it away on the hit path.
+ v, hit := e.backhaulInflight.Load(box)
+ if !hit {
+ v, _ = e.backhaulInflight.LoadOrStore(box, new(atomic.Int64))
+ }
+ n := v.(*atomic.Int64)
+ if n.Add(1) > backhaulMaxInflight.Load() {
+ n.Add(-1)
+ return nil, false
+ }
+ // Once-guarded because a slot released twice would let the cap drift
+ // upwards for the life of the process — the one failure mode of a
+ // hand-released semaphore, and cheaper to make impossible than to audit.
+ var once sync.Once
+ return func() { once.Do(func() { n.Add(-1) }) }, true
+}
+
// backhaulPort is the port a home edge answers on. Always 443 in production —
// an edges file publishes a bare EIP (Task 1) and every edge's front door is
// 443 — and a var only so a test can point the dial at an httptest server, the
@@ -1852,6 +1981,25 @@ func (e *Edge) backhaul(w http.ResponseWriter, r *http.Request, host string) boo
e.logThrottled("backhaul-no-mesh-key", fmt.Sprintf("%s is homed on %s but no mesh key is mirrored — not dialling (the request gets the unknown-host answer until the key lands)", host, box))
return false
}
+ // Containment (the egress ruling, 2026-08-24), between the last guard and
+ // the dial: a saturated home box is answered here, with no dial and no
+ // bytes crossing a region. 503 + Retry-After: 2 is this package's
+ // established "come back shortly" (staleHopRefusal, fleetProxy) and is
+ // deliberately NOT the unknown-host answer — that is a 302 on a fallback
+ // edge, which would lie to a legitimate client about where its app lives.
+ // The log key is per home BOX, like "backhaul-error "+box below and for
+ // the same bounded-key reason.
+ releaseSlot, ok := e.backhaulAcquire(box)
+ if !ok {
+ e.logThrottled("backhaul-saturated "+box,
+ fmt.Sprintf("backhaul to %s (%s) is at its in-flight cap (%d) — answering 503 + Retry-After: 2", box, eip, backhaulMaxInflight.Load()))
+ w.Header().Set("Retry-After", "2")
+ http.Error(w, "temporarily busy", http.StatusServiceUnavailable)
+ return true
+ }
+ // Held until the hop's ServeHTTP returns, which for a stream or an
+ // upgraded connection is the life of the stream (backhaulAcquire).
+ defer releaseSlot()
// The MAC is over the CONCRETE request host — never the wildcard map key
// that may have matched it — because the home edge verifies against its own
// literal request host (hopMAC canonicalizes both, mesh.go).
@@ -2095,6 +2243,11 @@ func RunWithActivatorReady(args []string, act Activator, cache *Cache, ready fun
log.Fatalf("edge: tunnels: %v", err)
}
+ // The backhaul in-flight cap, read from the environment once here rather
+ // than per request (backhaulMaxInflight). Inert on a single-box
+ // deployment, where nothing is ever backhauled at all.
+ backhaulMaxInflight.Store(int64(backhaulInflightFromEnv()))
+
go e.Watch(make(chan struct{})) // process-lifetime; never stopped
// Reload routes on SIGHUP too (an operator's explicit "now").
diff --git a/internal/edge/homemap.go b/internal/edge/homemap.go
index 044709a88..5b115d9bd 100644
--- a/internal/edge/homemap.go
+++ b/internal/edge/homemap.go
@@ -413,6 +413,45 @@ func (m *homeMap) Home(host string) (eip, box string, ok bool) {
return "", "", false
}
+// KnownHost reports whether host is an EXACT key in the mirrored home map — a
+// name some OTHER box in this deployment homes, spelled precisely.
+//
+// Deliberately NOT Home()'s one-label wildcard fallback, and refusing "*."
+// outright: this predicate widens cert issuance and challenge admission
+// (certsync.go's three gates, through Edge.deploymentHost), where PR #180's
+// rule stands — attacker-chosen labels under a catch-all must never drive an
+// ACME order or a bucket read, and their TLS coverage is the wildcard cert's
+// job (Allowed's own doc says the same thing for this box's own rows). The
+// widening loses nothing by staying exact: an order only ever exists for a
+// name that passed the exact gate, so a challenge only ever needs answering
+// for an exact name.
+//
+// Unlike Home it needs no EIP and no dial target — membership is the whole
+// question — but it reads the same fold, so it inherits both of the fold's
+// exclusions. Own-box hosts are excluded and that is harmless: they pass
+// Allowed on the other half of deploymentHost. Boxes with EIP=="" are
+// excluded and that is accepted: a box with no public IP offers no backhaul
+// target and appears in no DNS answer, so a validation for its hosts has
+// nowhere legitimate to land anyway.
+//
+// There is deliberately no m.self gate, unlike Home: Home fail-closes without
+// a self label because it could otherwise offer this box itself as a dial
+// target, and membership carries no such hazard — an unlabelled box's fold
+// keeps its own file, whose hosts are its own registry rows and so pass
+// Allowed anyway, modulo edges-file staleness (a route deleted from the
+// registry lingers in the published file until the next publish, minutes;
+// inherent to the design, and the widened set is registry rows either way).
+//
+// Nil-receiver-safe, like Home: the map may not be armed yet (or at all, on a
+// box with no deployment store).
+func (m *homeMap) KnownHost(host string) bool {
+ if m == nil || strings.HasPrefix(host, "*.") {
+ return false
+ }
+ _, ok := m.snapshot().hosts[host]
+ return ok
+}
+
// refreshKeys re-reads the mesh key pair. loadMeshKeys is already tolerant of
// a malformed .next and fail-closed on the current key (mesh.go); this only
// adds the mirror's own posture on top — the pair is replaced ONLY on a clean
diff --git a/internal/edge/homemap_test.go b/internal/edge/homemap_test.go
index 8854054b2..ce605928c 100644
--- a/internal/edge/homemap_test.go
+++ b/internal/edge/homemap_test.go
@@ -235,11 +235,71 @@ func TestNilHomeMapIsInert(t *testing.T) {
if _, _, ok := m.Home("b.test"); ok {
t.Fatal("a nil home map must offer nothing")
}
+ if m.KnownHost("b.test") {
+ t.Fatal("a nil home map knows no host")
+ }
if cur, next := m.meshKeys(); cur != nil || next != nil {
t.Fatal("a nil home map holds no mesh keys")
}
}
+// KnownHost is the membership question the three cert gates ask through
+// Edge.deploymentHost (#277), and it is EXACT-only where Home is not. Home's
+// one-label fallback is what makes an invented subdomain under a remote
+// catch-all DIALABLE; letting the same fallback into a cert gate would let
+// that invented label drive an ACME order and spend a read-through budget,
+// which is precisely what PR #180's Allowed rule refuses. The two answers
+// diverging on the same map, in the same test, is the pin.
+func TestKnownHostExactOnly(t *testing.T) {
+ s := meshTestStore(t)
+ putJSON(t, s, "control/edges/annie.json",
+ `{"v":1,"box":"annie","eip":"1.1.1.1","hosts":{"own.test":{"role":"home"}}}`)
+ putJSON(t, s, "control/edges/betty.json",
+ `{"v":1,"box":"betty","eip":"2.2.2.2","hosts":{"*.apps.test":{"role":"home"},"exact.apps.test":{"role":"home"}}}`)
+ m := newHomeMap(s, "annie")
+ refreshHomeMap(t, m)
+
+ if !m.KnownHost("exact.apps.test") {
+ t.Fatal("an exactly-named remote-homed host is the whole point of the predicate")
+ }
+ // Precondition: the fallback Home applies is live on this very map — so
+ // the refusal below is KnownHost's own, not an empty map's.
+ if _, _, ok := m.Home("invented.apps.test"); !ok {
+ t.Fatal("test bug: the one-label wildcard rule did not apply")
+ }
+ if m.KnownHost("invented.apps.test") {
+ t.Fatal("a label under a mirrored catch-all must never be admitted — that is #180's rule")
+ }
+ if m.KnownHost("*.apps.test") {
+ t.Fatal("the catch-all key itself is not a certifiable name")
+ }
+ if m.KnownHost("stranger.example") {
+ t.Fatal("a host no box in the deployment homes is a stranger")
+ }
+ // Own-box rows are excluded by the fold and stay excluded here. Harmless:
+ // deploymentHost's other half (Allowed) is what admits this box's routes.
+ if m.KnownHost("own.test") {
+ t.Fatal("the fold excludes this box's own file, so KnownHost inherits that exclusion")
+ }
+ // Nothing polled yet: no membership, and no panic on the gate's path.
+ if newHomeMap(s, "annie").KnownHost("exact.apps.test") {
+ t.Fatal("a map that has never polled knows no host")
+ }
+}
+
+// A box publishing eip:"" is in the fold's no-dial exclusion, and KnownHost
+// inherits it: there is no DNS answer pointing at such a box, so a validation
+// for its hosts has nowhere legitimate to land.
+func TestKnownHostSkipsBoxesWithNoEIP(t *testing.T) {
+ s := meshTestStore(t)
+ putJSON(t, s, "control/edges/betty.json", `{"v":1,"box":"betty","eip":"","hosts":{"b.test":{"role":"home"}}}`)
+ m := newHomeMap(s, "annie")
+ refreshHomeMap(t, m)
+ if m.KnownHost("b.test") {
+ t.Fatal("a box with no public IP offers no validation target")
+ }
+}
+
// ---- hop acceptance ---------------------------------------------------------
// forwardEcho is an upstream that reports back exactly the trust-bearing
@@ -1138,6 +1198,334 @@ func TestHTTPFallbackRedirectsARemoteHomedHost(t *testing.T) {
}
}
+// ---- containment: the per-home-box in-flight cap --------------------------
+
+// backhaulCap lowers the in-flight cap for one test and restores it after —
+// the package-seam pattern backhaulPort and homeMapInterval already follow.
+func backhaulCap(t *testing.T, n int64) {
+ t.Helper()
+ old := backhaulMaxInflight.Load()
+ backhaulMaxInflight.Store(n)
+ t.Cleanup(func() { backhaulMaxInflight.Store(old) })
+}
+
+// backhaulGetter drives requests for host at one edge over a single reused
+// server, so several can be in flight at once (hopRequest stands a server per
+// call, which is fine for one-shot tests and useless for a concurrency one).
+func backhaulGetter(t *testing.T, e *Edge) func(host string) (*http.Response, error) {
+ t.Helper()
+ srv := httptest.NewServer(e)
+ t.Cleanup(srv.Close)
+ return func(host string) (*http.Response, error) {
+ req, err := http.NewRequest(http.MethodGet, srv.URL+"/", nil)
+ if err != nil {
+ return nil, err
+ }
+ req.Host = host
+ return http.DefaultClient.Do(req)
+ }
+}
+
+// The cap itself: with two hops held open toward one home box, a third is
+// answered 503 + Retry-After: 2 HERE — no dial, no cross-region bytes — and
+// the slot a finished hop gives back is immediately usable again.
+//
+// This is what bounds a scanner's forged-Host flood: egress is (concurrent
+// streams × per-stream bandwidth), and this caps the first factor per box.
+func TestBackhaulRefusesAtInflightCap(t *testing.T) {
+ e, m, _ := hopEdge(t, "annie", "local.test")
+ backhaulCap(t, 2)
+
+ started, hold := make(chan struct{}, 4), make(chan struct{})
+ rec, eip := fakeHomeEdge(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ started <- struct{}{}
+ <-hold
+ fmt.Fprint(w, homeEdgeBody)
+ }))
+ homedOn(t, m, "betty", eip, "example.com")
+ get := backhaulGetter(t, e)
+
+ done := make(chan error, 2)
+ for i := 0; i < 2; i++ {
+ go func() {
+ resp, err := get("example.com")
+ if err != nil {
+ done <- err
+ return
+ }
+ defer resp.Body.Close()
+ if resp.StatusCode != http.StatusOK {
+ done <- fmt.Errorf("held hop: status = %d, want 200", resp.StatusCode)
+ return
+ }
+ body, err := io.ReadAll(resp.Body)
+ if err != nil {
+ done <- err
+ return
+ }
+ if string(body) != homeEdgeBody {
+ done <- fmt.Errorf("held hop: body = %q", body)
+ return
+ }
+ done <- nil
+ }()
+ }
+ for i := 0; i < 2; i++ {
+ select {
+ case <-started:
+ case err := <-done:
+ t.Fatalf("a hop finished before the cap was full: %v", err)
+ case <-time.After(15 * time.Second):
+ t.Fatal("the two hops that fill the cap never reached the home edge")
+ }
+ }
+
+ // Both slots are held. The third request must be refused by THIS box.
+ dialled := rec.get().n
+ resp, err := get("example.com")
+ if err != nil {
+ t.Fatalf("third request: %v", err)
+ }
+ defer resp.Body.Close()
+ if resp.StatusCode != http.StatusServiceUnavailable {
+ t.Fatalf("status = %d, want 503 at the in-flight cap", resp.StatusCode)
+ }
+ if got := resp.Header.Get("Retry-After"); got != "2" {
+ t.Fatalf("Retry-After = %q, want 2 — the cap clears as hops finish", got)
+ }
+ if got := rec.get().n; got != dialled {
+ t.Fatalf("the home edge saw %d request(s), want %d — a refusal must cost no dial and no cross-region bytes", got, dialled)
+ }
+
+ // Released slots are usable again: the cap is a bound on CONCURRENCY, not
+ // a quota that a box burns through and never gets back.
+ close(hold)
+ for i := 0; i < 2; i++ {
+ select {
+ case err := <-done:
+ if err != nil {
+ t.Fatal(err)
+ }
+ case <-time.After(15 * time.Second):
+ t.Fatal("a held hop never completed after its home edge was released")
+ }
+ }
+ // Retried for a bounded window rather than asserted on the first attempt:
+ // a held hop's client-side goroutine reports completion when the BODY is
+ // read, which can be a hair before the edge's own deferred release runs.
+ // The property under test is that the slots come back, not that they come
+ // back within one scheduling quantum.
+ var (
+ after *http.Response
+ lastErr error
+ )
+ deadline := time.Now().Add(10 * time.Second)
+ for {
+ after, lastErr = get("example.com")
+ if lastErr != nil {
+ t.Fatalf("fourth request: %v", lastErr)
+ }
+ if after.StatusCode == http.StatusOK {
+ break
+ }
+ after.Body.Close()
+ if after.StatusCode != http.StatusServiceUnavailable {
+ t.Fatalf("status = %d after the held hops finished, want 200", after.StatusCode)
+ }
+ if time.Now().After(deadline) {
+ t.Fatal("the cap never cleared after both held hops finished — a released slot was not given back")
+ }
+ time.Sleep(10 * time.Millisecond)
+ }
+ defer after.Body.Close()
+ if got := rec.get().n; got < dialled+1 {
+ t.Fatalf("the home edge saw %d request(s), want at least %d — the fourth request must dial normally", got, dialled+1)
+ }
+}
+
+// The cap is per HOME BOX: one saturated box must not stop this edge serving
+// hosts homed on any other. A deployment-wide counter would let one busy (or
+// one attacked) box take the whole mesh down.
+func TestBackhaulInflightCapIsPerHomeBox(t *testing.T) {
+ e, m, _ := hopEdge(t, "annie", "local.test")
+ backhaulCap(t, 1)
+ rec, eip := fakeHomeEdge(t, nil)
+ homedOn(t, m, "carol", eip, "example.com")
+ // Betty's EIP is never dialled in this test — her slot is filled directly
+ // below, so the refusal happens before any dial. (TEST-NET-1: a dial
+ // would hang for the whole 5s budget and be visible as a failure.)
+ homedOn(t, m, "betty", "192.0.2.5", "saturated.test")
+
+ release, ok := e.backhaulAcquire("betty")
+ if !ok {
+ t.Fatal("the first slot toward a box must always be available")
+ }
+ t.Cleanup(release)
+
+ if resp := hopRequest(t, e, "saturated.test", nil); resp.StatusCode != http.StatusServiceUnavailable {
+ t.Fatalf("status = %d, want 503 — betty's only slot is held", resp.StatusCode)
+ }
+ resp := hopRequest(t, e, "example.com", nil)
+ if resp.StatusCode != http.StatusOK {
+ t.Fatalf("status = %d, want 200 — carol's cap is her own", resp.StatusCode)
+ }
+ if body := readBody(t, resp); body != homeEdgeBody {
+ t.Fatalf("body = %q, want the home edge's own answer", body)
+ }
+ if n := rec.get().n; n != 1 {
+ t.Fatalf("carol saw %d request(s), want exactly one", n)
+ }
+}
+
+// A slot is held for the LIFE of the stream, not until the response headers
+// arrive: a long-lived stream (a WebSocket, an SSE feed, a large download) is
+// precisely the sustained cross-region egress the cap exists to bound, so it
+// must count against the cap for as long as it flows.
+func TestBackhaulSlotHeldForStreamLife(t *testing.T) {
+ e, m, _ := hopEdge(t, "annie", "local.test")
+ backhaulCap(t, 1)
+
+ streaming, finish := make(chan struct{}), make(chan struct{})
+ _, eip := fakeHomeEdge(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ fmt.Fprint(w, "first ")
+ w.(http.Flusher).Flush() // headers and a first chunk are on the wire
+ close(streaming)
+ <-finish
+ fmt.Fprint(w, "last")
+ }))
+ homedOn(t, m, "betty", eip, "example.com")
+ get := backhaulGetter(t, e)
+
+ stream, err := get("example.com")
+ if err != nil {
+ t.Fatalf("stream request: %v", err)
+ }
+ defer stream.Body.Close()
+ if stream.StatusCode != http.StatusOK {
+ t.Fatalf("stream status = %d, want 200", stream.StatusCode)
+ }
+ select {
+ case <-streaming:
+ case <-time.After(15 * time.Second):
+ t.Fatal("the streaming hop never reached the home edge")
+ }
+
+ // Headers have arrived and the body is still flowing: the slot is held.
+ busy, err := get("example.com")
+ if err != nil {
+ t.Fatalf("second request: %v", err)
+ }
+ defer busy.Body.Close()
+ if busy.StatusCode != http.StatusServiceUnavailable {
+ t.Fatalf("status = %d while a hop is mid-stream, want 503 — the slot is held for the stream's life", busy.StatusCode)
+ }
+
+ close(finish)
+ body, err := io.ReadAll(stream.Body)
+ if err != nil {
+ t.Fatalf("read stream: %v", err)
+ }
+ if string(body) != "first last" {
+ t.Fatalf("stream body = %q, want both chunks", body)
+ }
+ stream.Body.Close()
+
+ // And the moment it ends, the slot is back.
+ after, err := get("example.com")
+ if err != nil {
+ t.Fatalf("third request: %v", err)
+ }
+ defer after.Body.Close()
+ if after.StatusCode != http.StatusOK {
+ t.Fatalf("status = %d after the stream ended, want 200 — the slot must have been released", after.StatusCode)
+ }
+}
+
+// Saturation is one throttled journal line per home box, never silence and
+// never one line per refused request: a flood is a STANDING condition, and
+// the key is the box (bounded by the deployment) and never the host (which a
+// stranger picks, and which a catch-all home would multiply without limit).
+func TestBackhaulSaturationLogsThrottledPerBox(t *testing.T) {
+ e, m, _ := hopEdge(t, "annie", "local.test")
+ backhaulCap(t, 1)
+ // Neither EIP is ever dialled: both boxes' only slot is held below, so
+ // every request in this test is refused before the dial.
+ homedOn(t, m, "betty", "192.0.2.5", "betty.test")
+ homedOn(t, m, "carol", "192.0.2.6", "carol.test")
+ for _, box := range []string{"betty", "carol"} {
+ release, ok := e.backhaulAcquire(box)
+ if !ok {
+ t.Fatalf("%s: the first slot toward a box must always be available", box)
+ }
+ t.Cleanup(release)
+ }
+
+ buf := captureLog(t)
+ for _, host := range []string{"betty.test", "betty.test", "carol.test"} {
+ if resp := hopRequest(t, e, host, nil); resp.StatusCode != http.StatusServiceUnavailable {
+ t.Fatalf("%s: status = %d, want 503", host, resp.StatusCode)
+ }
+ }
+
+ got := buf.String()
+ // logThrottled prints no key of its own, so key isolation is observed as
+ // TWO first occurrences (one bucket per box) with betty's second refusal
+ // throttled inside her own bucket.
+ if n := strings.Count(got, "occurrence #1"); n != 2 {
+ t.Fatalf("%d first-occurrence line(s), want one per home box (two buckets): %s", n, got)
+ }
+ if strings.Contains(got, "occurrence #2") {
+ t.Fatalf("log output = %q, want the second refusal against the same box throttled", got)
+ }
+ if n := strings.Count(got, "in-flight cap"); n != 2 {
+ t.Fatalf("%d saturation line(s), want exactly two: %s", n, got)
+ }
+ if !strings.Contains(got, "betty") || !strings.Contains(got, "carol") {
+ t.Fatalf("log output = %q, want each saturated home box named", got)
+ }
+}
+
+// The knob: unset, unparseable and non-positive all mean the default. There
+// is deliberately no "off" — an unbounded cross-region path is what the cap
+// closes — and a typo must not silently zero-cap a deployment either.
+func TestBackhaulMaxInflightFromEnv(t *testing.T) {
+ // The ambient value is captured and restored by hand: t.Setenv restores
+ // what IT set, but the unset case below has to clear the variable, and an
+ // operator's own CARLOS_BACKHAUL_MAX_INFLIGHT must survive this test.
+ ambient, had := os.LookupEnv("CARLOS_BACKHAUL_MAX_INFLIGHT")
+ t.Cleanup(func() {
+ if had {
+ os.Setenv("CARLOS_BACKHAUL_MAX_INFLIGHT", ambient)
+ return
+ }
+ os.Unsetenv("CARLOS_BACKHAUL_MAX_INFLIGHT")
+ })
+
+ for _, tc := range []struct {
+ set bool
+ raw string
+ want int
+ }{
+ {set: false, want: defaultBackhaulMaxInflight},
+ {set: true, raw: "", want: defaultBackhaulMaxInflight},
+ {set: true, raw: " ", want: defaultBackhaulMaxInflight},
+ {set: true, raw: "sixty-four", want: defaultBackhaulMaxInflight},
+ {set: true, raw: "0", want: defaultBackhaulMaxInflight},
+ {set: true, raw: "-5", want: defaultBackhaulMaxInflight},
+ {set: true, raw: "128", want: 128},
+ {set: true, raw: " 8 ", want: 8},
+ } {
+ if tc.set {
+ t.Setenv("CARLOS_BACKHAUL_MAX_INFLIGHT", tc.raw)
+ } else {
+ os.Unsetenv("CARLOS_BACKHAUL_MAX_INFLIGHT")
+ }
+ if got := backhaulInflightFromEnv(); got != tc.want {
+ t.Fatalf("CARLOS_BACKHAUL_MAX_INFLIGHT=%q (set=%v) → %d, want %d", tc.raw, tc.set, got, tc.want)
+ }
+ }
+}
+
// oneKeyFailingStore fails Get for one named key — a single unreadable object
// mid-poll, the case the carry-forward exists for.
type oneKeyFailingStore struct {
diff --git a/internal/edge/usagemeter.go b/internal/edge/usagemeter.go
index b82caa55a..1cbd7db65 100644
--- a/internal/edge/usagemeter.go
+++ b/internal/edge/usagemeter.go
@@ -26,14 +26,24 @@ func NewUsageMeter() *UsageMeter {
return &UsageMeter{m: map[string]usage.EdgeRow{}}
}
-// Record adds one served exchange for rt.
-func (u *UsageMeter) Record(rt registry.Route, requests, bytes int64) {
+// Record adds one served exchange for rt. hopped says the request arrived
+// over a verified backhaul hop from a sibling edge (homemap.go) — the same
+// exchange is then ALSO added to the row's hop counters, which are a subset
+// annotation inside Requests/BytesOut and never a second exchange (see
+// usage.EdgeRow). One method rather than a separate RecordHop: two methods
+// for one exchange invite calling both, which is exactly the double count
+// this arrangement exists to make impossible.
+func (u *UsageMeter) Record(rt registry.Route, requests, bytes int64, hopped bool) {
u.mu.Lock()
row := u.m[rt.Host]
row.Account = release.RouteAccount(rt)
row.App = rt.App
row.Requests += requests
row.BytesOut += bytes
+ if hopped {
+ row.HopRequests += requests
+ row.HopBytesOut += bytes
+ }
u.m[rt.Host] = row
u.mu.Unlock()
}
diff --git a/internal/edge/usagemeter_test.go b/internal/edge/usagemeter_test.go
index 983174211..abe4e46dd 100644
--- a/internal/edge/usagemeter_test.go
+++ b/internal/edge/usagemeter_test.go
@@ -13,9 +13,9 @@ import (
func TestUsageMeterRecordAndDrain(t *testing.T) {
m := NewUsageMeter()
rt := registry.Route{Host: "a.example.com", App: "hello", Account: "bab"}
- m.Record(rt, 1, 100)
- m.Record(rt, 1, 50)
- m.Record(registry.Route{Host: "b.example.com", App: "site"}, 1, 10) // Account "" -> release.OpsAccount
+ m.Record(rt, 1, 100, false)
+ m.Record(rt, 1, 50, false)
+ m.Record(registry.Route{Host: "b.example.com", App: "site"}, 1, 10, false) // Account "" -> release.OpsAccount
got := m.Drain()
a := got["a.example.com"]
@@ -30,6 +30,32 @@ func TestUsageMeterRecordAndDrain(t *testing.T) {
}
}
+// A hopped exchange is recorded ONCE — into the totals, as any exchange is —
+// and additionally annotated into the hop counters, which are therefore a
+// subset of the totals and never an addition to them. A consumer that sums
+// BytesOut has already counted every backhauled byte; one that added
+// HopBytesOut on top would bill the tenant twice for the same wire bytes.
+func TestRecordHopAnnotatesSubset(t *testing.T) {
+ m := NewUsageMeter()
+ rt := registry.Route{Host: "a.example.com", App: "hello", Account: "bab"}
+ m.Record(rt, 1, 100, true)
+ m.Record(rt, 1, 50, false)
+
+ row := m.Drain()["a.example.com"]
+ if row.Requests != 2 || row.BytesOut != 150 {
+ t.Fatalf("row = %+v, want both exchanges counted once in the totals", row)
+ }
+ if row.HopRequests != 1 || row.HopBytesOut != 100 {
+ t.Fatalf("row = %+v, want only the hopped exchange annotated", row)
+ }
+ if row.HopRequests > row.Requests || row.HopBytesOut > row.BytesOut {
+ t.Fatalf("row = %+v — the hop counters must stay inside the totals", row)
+ }
+ if row.Account != "bab" || row.App != "hello" {
+ t.Fatalf("row = %+v, want the hop attributed to the route's own account and app", row)
+ }
+}
+
func TestUsageMeterConcurrent(t *testing.T) {
m := NewUsageMeter()
rt := registry.Route{Host: "a.example.com", App: "hello", Account: "bab"}
@@ -39,7 +65,7 @@ func TestUsageMeterConcurrent(t *testing.T) {
go func() {
defer wg.Done()
for j := 0; j < 100; j++ {
- m.Record(rt, 1, 10)
+ m.Record(rt, 1, 10, false)
}
}()
}
diff --git a/internal/usage/edge.go b/internal/usage/edge.go
index af4ba8f5f..04f7bef1f 100644
--- a/internal/usage/edge.go
+++ b/internal/usage/edge.go
@@ -17,6 +17,20 @@ type EdgeRow struct {
App string `json:"app"`
Requests int64 `json:"requests"`
BytesOut int64 `json:"bytes_out"`
+ // HopRequests/HopBytesOut are the SUBSET of Requests/BytesOut that
+ // arrived over a verified backhaul hop (multi-region spec Part 3) — an
+ // annotation inside the totals, never an addition to them: a consumer
+ // that sums BytesOut already includes every hop byte, and must never add
+ // HopBytesOut on top. They exist so the platform can see which traffic
+ // crossed a region (its own egress is doubled by a hop) and so backhaul
+ // can be priced later without a shape change; nothing bills them today.
+ //
+ // Additive and omitempty: a day object written before these existed
+ // parses with both zero, one written after them is ignored field-for-
+ // field by an older binary, and a single-region deployment's objects
+ // stay byte-identical to what it wrote yesterday.
+ HopRequests int64 `json:"hop_requests,omitempty"`
+ HopBytesOut int64 `json:"hop_bytes_out,omitempty"`
}
// EdgeDay is usage/edge/<box>/days/<date>.json — a box's edge traffic for
@@ -59,6 +73,8 @@ func (d *EdgeDay) AddDeltas(hour string, deltas map[string]EdgeRow) {
row.Account, row.App = delta.Account, delta.App
row.Requests += delta.Requests
row.BytesOut += delta.BytesOut
+ row.HopRequests += delta.HopRequests
+ row.HopBytesOut += delta.HopBytesOut
rows[host] = row
}
}
diff --git a/internal/usage/edge_test.go b/internal/usage/edge_test.go
index 424f0c9e7..97f1a382f 100644
--- a/internal/usage/edge_test.go
+++ b/internal/usage/edge_test.go
@@ -69,6 +69,82 @@ func TestUpsertEdgeDayMergesAcrossFlushes(t *testing.T) {
}
}
+// The hop counters are an ADDITIVE shape change (the paid-for wire rule):
+// every day object already in a production bucket predates them, so one
+// written without them must still parse, fold and round-trip — and a
+// single-region box, which never sees a hop, must keep writing objects that
+// are byte-identical to what it wrote before the fields existed.
+func TestEdgeDayHopFieldsAdditive(t *testing.T) {
+ // A day object as a pre-ruling binary wrote it: no hop keys at all.
+ const old = `{"v":1,"box":"box1","date":"2026-08-24","hours":{"09":{"a.example.com":` +
+ `{"account":"bab","app":"hello","requests":10,"bytes_out":1000}}}}`
+ var day EdgeDay
+ if err := json.Unmarshal([]byte(old), &day); err != nil {
+ t.Fatalf("a day object written before the hop fields existed must still parse: %v", err)
+ }
+ row := day.Hours["09"]["a.example.com"]
+ if row.Requests != 10 || row.BytesOut != 1000 || row.HopRequests != 0 || row.HopBytesOut != 0 {
+ t.Fatalf("row = %+v, want the old counters and zero hop counters", row)
+ }
+ // And it folds a hop-bearing delta on top without losing the old counts.
+ day.AddDeltas("09", map[string]EdgeRow{
+ "a.example.com": {Account: "bab", App: "hello", Requests: 3, BytesOut: 300, HopRequests: 2, HopBytesOut: 250},
+ })
+ row = day.Hours["09"]["a.example.com"]
+ if row.Requests != 13 || row.BytesOut != 1300 || row.HopRequests != 2 || row.HopBytesOut != 250 {
+ t.Fatalf("folded row = %+v", row)
+ }
+
+ // Round-trip: what a hop-annotating box writes, a hop-annotating box reads.
+ raw, err := json.Marshal(day)
+ if err != nil {
+ t.Fatal(err)
+ }
+ var back EdgeDay
+ if err := json.Unmarshal(raw, &back); err != nil {
+ t.Fatal(err)
+ }
+ if got := back.Hours["09"]["a.example.com"]; got != row {
+ t.Fatalf("round-tripped row = %+v, want %+v", got, row)
+ }
+
+ // omitempty, the single-region promise: a row with no hop traffic writes
+ // no hop keys, so a one-box deployment's objects do not change shape.
+ plain, err := json.Marshal(EdgeRow{Account: "bab", App: "hello", Requests: 1, BytesOut: 10})
+ if err != nil {
+ t.Fatal(err)
+ }
+ if want := `{"account":"bab","app":"hello","requests":1,"bytes_out":10}`; string(plain) != want {
+ t.Fatalf("hop-free row marshals as %s, want %s — omitempty keeps single-region objects unchanged", plain, want)
+ }
+}
+
+// The fold accumulates the hop counters across flushes exactly as it does the
+// totals — they are counters in the same hour bucket, not a snapshot.
+func TestEdgeDayHourFoldSumsHopFields(t *testing.T) {
+ var day EdgeDay
+ delta := map[string]EdgeRow{
+ "a.example.com": {Account: "bab", App: "hello", Requests: 4, BytesOut: 400, HopRequests: 1, HopBytesOut: 100},
+ }
+ day.AddDeltas("09", delta)
+ day.AddDeltas("09", delta)
+ day.AddDeltas("10", delta)
+
+ if got := day.Hours["09"]["a.example.com"]; got.Requests != 8 || got.BytesOut != 800 ||
+ got.HopRequests != 2 || got.HopBytesOut != 200 {
+ t.Fatalf("hour 09 row = %+v, want two flushes summed", got)
+ }
+ if got := day.Hours["10"]["a.example.com"]; got.HopRequests != 1 || got.HopBytesOut != 100 {
+ t.Fatalf("hour 10 row = %+v, want the third flush in its own bucket", got)
+ }
+ // The subset invariant, stated where a future reader will meet it: hop
+ // bytes live INSIDE bytes_out, so a consumer summing BytesOut has already
+ // counted them.
+ if got := day.Hours["09"]["a.example.com"]; got.HopBytesOut > got.BytesOut || got.HopRequests > got.Requests {
+ t.Fatalf("row = %+v — the hop counters are a subset of the totals, never an addition", got)
+ }
+}
+
func TestUpsertEdgeDayEmptyDeltasIsNoop(t *testing.T) {
s := release.NewDirStore(t.TempDir())
if err := UpsertEdgeDay(context.Background(), s, "box1", time.Now().UTC(), nil); err != nil {
diff --git a/internal/usage/usage.go b/internal/usage/usage.go
index 65c586505..437949e54 100644
--- a/internal/usage/usage.go
+++ b/internal/usage/usage.go
@@ -23,6 +23,13 @@ const (
MeterInstanceHours = "instance_hours"
MeterFleetBoxHoursServer = "fleet_box_hours_server"
MeterFleetBoxHoursDevice = "fleet_box_hours_device"
+ // MeterBackhaulGB is RESERVED, not yet billed: the egress a request
+ // spent crossing a region on a backhaul hop (EdgeRow.HopBytesOut). The
+ // ruling that added the counters (2026-08-24) deliberately left the
+ // pricing to Paul and shipped them dormant — no statement, no rate-card
+ // line, bills byte-identical to before. The name is reserved here
+ // because meter names are frozen once merged: reserving beats renaming.
+ MeterBackhaulGB = "backhaul_gb"
)
// DayOf is the UTC date label used in every usage key.