amadan

carlos / platform Public

edge: backhaul — remote-homed hosts dial the home edge, one hop, real TLS

The sending half of edge-to-edge backhaul (spec Part 3, Task 4). A request
that lands here for a host homed on ANOTHER box no longer dead-ends in the
unknown-host answer: it is forwarded to that box's edge over ordinary,
publicly-verified TLS, with SNI set to the host being asked for and the two
hop headers Task 3 already accepts.

Precedence, exactly: a local lookupUpstream hit is served here, always,
whatever the home map says; a miss with a home dials it; a miss with no home
keeps today's 404 byte for byte; and a request that ARRIVED as a verified hop
never consults the map at all — ServeHTTP refuses it above the dial, and the
forwarding helper opens with the same refusal so a reordering cannot open a
loop.

Fail-closed on the sending side too: with no mesh key mirrored there is
nothing to prove the hop with, so the miss falls through to the unknown-host
answer rather than signing under an empty key. The MAC binds the CONCRETE
request host, never the wildcard map row that matched it.

One transport per home BOX, not per host: the outbound URL's authority is the
hostname (SNI, verification and connection pooling all key on it), so a
catch-all home shares one transport instead of minting one per invented
subdomain. Dial 5s, response headers 30s, no whole-body deadline — a
backhauled stream or WebSocket must not be cut.

No metering here: the one meter sits below a successful lookupUpstream, so
the home edge counts a backhauled request exactly once. Egress attribution
for the second internet leg is a unit-economics ruling, not a proxy's guess.

Inert on a single-box deployment: Home() never offers a target, so the whole
path is dead code until a second box publishes its edges file.
Paul Campbell pushed by paul@keymail.dev bdcf6f68cc88730571f3f7ff29611a53df07e45c parent 67c2034
2 files changed, +663 −8
  • internal/edge/edge.go +261 −2
  • internal/edge/homemap_test.go +402 −6
diff --git a/internal/edge/edge.go b/internal/edge/edge.go
index 2b57d2d1a..9879ded98 100644
--- a/internal/edge/edge.go
+++ b/internal/edge/edge.go
@@ -20,6 +20,8 @@ import (
"bufio"
"context"
"crypto/ed25519"
+ "crypto/tls"
+ "crypto/x509"
"encoding/json"
"errors"
"flag"
@@ -179,6 +181,13 @@ type Edge struct {
// an inbound hop. Atomic because it is armed in RunWithActivatorReady after
// construction, and read on the request path while that is happening.
homes atomic.Pointer[homeMap]
+ // backhaulTransports caches the outbound TLS transport of each backhaul
+ // target, keyed by the home box's EIP — see backhaulTransport. Bounded by
+ // the number of BOXES in the deployment, never by hostnames: a catch-all
+ // row in another box's edges file makes every subdomain under it a
+ // backhaul target, and keying per host would let a stranger's stream of
+ // invented hostnames grow this map without limit.
+ backhaulTransports sync.Map
}
// RunOption configures the Edge that Run/RunWithActivator* builds, before it
@@ -819,11 +828,20 @@ func (e *Edge) ServeHTTP(w http.ResponseWriter, r *http.Request) {
// A PROVEN hop from a sibling edge, for a host that has moved (or
// never lived) here: its map is stale. 503 + Retry-After: 2, and
// this return is the structural half of "one hop, ever" — the
- // backhaul dial Task 4 adds lives further down this same branch,
- // so an accepted hop physically cannot reach it.
+ // backhaul dial below lives further down this same branch, so an
+ // accepted hop physically cannot reach it. (e.backhaul opens with
+ // the same refusal anyway, so moving this branch cannot open a
+ // forwarding loop either — see its doc.)
staleHopRefusal(w)
return
}
+ // Nothing here serves this host — but another box in the deployment
+ // may (backhaul, below). It answers false when no box homes the host,
+ // or when nothing could prove the hop, and today's unknown-host answer
+ // then follows exactly as it always has.
+ if e.backhaul(w, r, host) {
+ return
+ }
e.unknownHost(w, r)
return
}
@@ -1683,6 +1701,247 @@ func fleetProxy(fleet, boxLabel string, hub *TunnelHub) *httputil.ReverseProxy {
}
}
+// ---- backhaul: the SENDING half of one-hop edge-to-edge forwarding ----------
+//
+// homemap.go holds the map (which OTHER box homes a host) and the RECEIVING
+// half (what proves an inbound hop, and what a stale one is answered with).
+// This is the other end: a request that landed on this edge for a host homed
+// elsewhere is forwarded to that box's edge — over ordinary, publicly-verified
+// TLS, exactly once — instead of dead-ending in the unknown-host answer.
+//
+// Three things are deliberately NOT done here, and each is a decision rather
+// than an omission:
+//
+// - NO METERING. The one usage meter (ServeHTTP, above) sits BELOW a
+// successful lookupUpstream, so a backhauled request misses it here by
+// construction while the HOME edge — which does serve the route — records
+// the request and its bytes exactly once. A second Record at this site
+// would double-count every backhauled request against the tenant. What
+// that leaves genuinely open is EGRESS ATTRIBUTION: the bytes cross the
+// internet twice and the second leg is billed to this box's account while
+// the meter runs on the home box's route. That is a unit-economics
+// question — the plan's Task 7 ruling — and not one to guess at from
+// inside a proxy.
+//
+// - NO CACHING AND NO COMPRESSION. This branch returns before ServeHTTP's
+// cache and gzip wrapping, so a backhauled response is neither stored nor
+// re-encoded here. That is load-bearing, not lazy: a tenant's Carlos-Purge
+// is honoured and stripped by the HOME edge's purgeModifier
+// (purge_header.go), so a copy cached on THIS box could never be
+// invalidated by the app that owns it. The home edge's own cache and
+// compression still apply and reach the client through the hop — as does
+// X-Carlos-Version, which the home edge stamps from the route's own row
+// (versionHeader is per-route, and this box has no row for the host), so
+// `carlos deploy`'s verification works through a backhaul with nothing
+// added here.
+//
+// - NO RETRY, AND NO SECOND TARGET. Home() answers with one box or none.
+// A failed dial is a 502 (see backhaulProxy's ErrorHandler), not a
+// scan for somewhere else to try: the process backing a home-boxed route
+// exists on exactly ONE box, so there is no second copy a retry could
+// reach.
+//
+// Single-box inertness: with one platform box Home() never offers a target
+// (own-box exclusion, homemap.go's foldHomes), so e.backhaul returns false on
+// every request and this whole path is dead code until a second box publishes
+// its edges file.
+
+// backhaulPort is the port a home edge answers on. Always 443 in production —
+// an edges file publishes a bare EIP (Task 1) and every edge's front door is
+// 443 — and a var only so a test can point the dial at an httptest server, the
+// same reason homeMapInterval is one.
+var backhaulPort = "443"
+
+// backhaulRootCAs is the trust anchor set the hop's TLS handshake verifies the
+// home edge's certificate against. nil — the system trust store, standard
+// verification — is the production answer and the only one: the home edge
+// presents the SAME publicly-trusted certificate it presents to any browser
+// (Part 2's bucket-distributed certs), so a hop needs no private CA, no
+// pinning, and above all no skipped verification. A var only so a test can
+// trust a fake home edge's self-signed cert.
+//
+// The bootstrap window this implies is stated rather than papered over: a box
+// that has just joined and whose cert for a host has not yet been distributed
+// presents something the dialer will not verify, so the hop fails the
+// handshake and the client gets a 502 until the cert mirror catches up. That
+// is the correct failure — the alternative, dialing without verification,
+// would make the hop's TLS decorative — and it heals on its own within a
+// mirror poll.
+var backhaulRootCAs *x509.CertPool
+
+// The hop's timeouts, per the plan. Deliberately no whole-body deadline: a
+// backhauled response may be a stream or an upgraded WebSocket that is
+// supposed to stay open for hours (the fleetapi client's ruling, same
+// reasoning), so the only bounds are on getting connected and on the home edge
+// beginning to answer.
+const (
+ backhaulDialTimeout = 5 * time.Second
+ backhaulHeaderTimeout = 30 * time.Second
+)
+
+// backhaul forwards a request for a host this box does not serve to the box
+// that does, and reports whether it answered. false means "not mine to
+// forward" and the caller falls through to the unchanged unknown-host answer:
+// either no box homes the host, or nothing could prove the hop.
+//
+// The hopVerified guard is FIRST, before the map is even consulted. ServeHTTP
+// already refuses an arrived hop above this call (that is the structural half
+// of "one hop, ever"), so this line is unreachable today — which is exactly
+// why it is here: if this call is ever moved, or that branch reordered, a hop
+// that arrived from a sibling edge fails closed with the stale-map refusal
+// instead of being forwarded onward. A forwarding loop must take TWO mistakes,
+// not one.
+func (e *Edge) backhaul(w http.ResponseWriter, r *http.Request, host string) bool {
+ if hopVerified(r.Context()) {
+ staleHopRefusal(w)
+ return true
+ }
+ m := e.homes.Load()
+ eip, box, ok := m.Home(host)
+ if !ok {
+ return false
+ }
+ // Sign with the mirror's cached pair, .next preferred (signingKey, mesh.go)
+ // — never a store read on the request path. NO KEY MEANS NO DIAL: signHop
+ // would happily HMAC under an empty key, the home edge would refuse the
+ // result, and the request would have crossed a region to be served as an
+ // ordinary client with its forwarding chain stripped. Falling through to
+ // the unknown-host answer is both cheaper and honest about the deployment
+ // being mid-bootstrap.
+ key := signingKey(m.meshKeys())
+ if len(key) == 0 {
+ e.logThrottled("backhaul-no-mesh-key", fmt.Sprintf("%s is homed on %s but no mesh key is mirrored — not dialling (the request gets the unknown-host answer until the key lands)", host, box))
+ return false
+ }
+ // The MAC is over the CONCRETE request host — never the wildcard map key
+ // that may have matched it — because the home edge verifies against its own
+ // literal request host (hopMAC canonicalizes both, mesh.go).
+ auth := signHop(key, host, time.Now())
+ backhaulProxy(host, eip, box, auth, e.backhaulTransport(eip)).ServeHTTP(w, r)
+ return true
+}
+
+// backhaulTransport is the cached outbound transport for one home box. The
+// TRANSPORT is what is cached and the ReverseProxy above is what is rebuilt
+// per request: a proxy value is two closures, while a transport owns the
+// connection pool, and a cross-region TCP+TLS handshake on every backhauled
+// page is a real, visible cost.
+//
+// Keyed by EIP and NOT by host, which is both the cheaper and the safer half
+// of that trade:
+//
+// - Nothing about the connection is per-host. The dial goes to the box, and
+// the TLS ServerName is derived per request from the outbound URL's
+// authority — backhaulProxy sets it to the real hostname — so one
+// transport serves every host homed on one box while each still gets its
+// own SNI, its own certificate verification, and (the stdlib pools by that
+// same authority) its own pooled connections.
+// - Per host, this map would be unbounded. A catch-all row in another box's
+// edges file homes every subdomain under it, so a stranger asking for a
+// stream of invented hostnames would mint a transport apiece. Per box it
+// is bounded by the deployment, and the pooled connections inside it are
+// bounded by MaxIdleConns and IdleConnTimeout as usual.
+//
+// A host whose home MOVES therefore lands on the new box's transport, with the
+// old box's pooled connections left to idle out — no equivalent of the pinning
+// multiProxy pays DisableKeepAlives to avoid, because the authority a
+// connection is pooled under names the host and the transport it lives in
+// names the box.
+//
+// HTTP/2 is not attempted, and that is on purpose: ForceAttemptHTTP2 stays
+// false (the stdlib disables the automatic upgrade anyway once TLSClientConfig
+// or a custom dialer is set), so the hop speaks HTTP/1.1 and
+// httputil.ReverseProxy's protocol-switch path — the one that carries a
+// WebSocket through — works exactly as it does for every other proxy in this
+// file.
+func (e *Edge) backhaulTransport(eip string) *http.Transport {
+ if tr, ok := e.backhaulTransports.Load(eip); ok {
+ return tr.(*http.Transport)
+ }
+ dialer := &net.Dialer{Timeout: backhaulDialTimeout}
+ tr := &http.Transport{
+ // The address is ignored, multiProxy-style: it is the outbound URL's
+ // authority, which is the HOST (for SNI, verification and pooling) and
+ // never where the bytes go. The home edge is at its EIP.
+ DialContext: func(ctx context.Context, network, _ string) (net.Conn, error) {
+ return dialer.DialContext(ctx, network, net.JoinHostPort(eip, backhaulPort))
+ },
+ // No ServerName: the stdlib fills it from the outbound URL's authority,
+ // which is exactly the host being asked for. RootCAs nil is the system
+ // trust store — standard verification, see backhaulRootCAs.
+ TLSClientConfig: &tls.Config{
+ RootCAs: backhaulRootCAs,
+ MinVersion: tls.VersionTLS12,
+ },
+ TLSHandshakeTimeout: backhaulDialTimeout,
+ ResponseHeaderTimeout: backhaulHeaderTimeout,
+ MaxIdleConns: 32,
+ IdleConnTimeout: 90 * time.Second,
+ }
+ // Two requests for a box nobody has dialled yet can both build one; the
+ // loser's transport is simply dropped, having opened nothing and started no
+ // goroutine.
+ if existing, loaded := e.backhaulTransports.LoadOrStore(eip, tr); loaded {
+ return existing.(*http.Transport)
+ }
+ return tr
+}
+
+// backhaulProxy is the hop itself: an ordinary reverse proxy to the home box,
+// carrying the client's Host through unchanged so the home edge routes on the
+// real hostname, and carrying the two hop headers that make it a hop rather
+// than an anonymous request from a stranger with a data-centre IP.
+//
+// The outbound URL names the HOST, not the EIP, and the transport's dialer is
+// what turns that into bytes at the home box (backhaulTransport). The
+// authority is therefore doing three jobs at once and all three want the
+// hostname: the TLS ServerName the home edge selects a certificate by, the
+// name that certificate is verified against, and the key the connection is
+// pooled under.
+//
+// It takes the signature ready-made instead of minting one inside Rewrite, so
+// that the "no key, no dial" decision (backhaul, above) and the signing happen
+// at the SAME read of the mirror's key pair — a Rewrite that signed for itself
+// could still reach for a key that had gone away since the guard.
+//
+// X-Forwarded-For is setForwarded's single-element chain and nothing more: the
+// client that dialled THIS edge, with neither this edge's own address nor the
+// home EIP appended. That is the contract homemap.go's setForwarded documents
+// from the receiving end — the home edge restores this chain verbatim on a
+// verified hop, so the app's last XFF element is the real client, exactly as
+// it would be if the request had landed on the home box directly. (The restore
+// branch cannot fire on THIS side: backhaul only runs when the request did not
+// arrive as a verified hop.)
+func backhaulProxy(host, eip, box, auth string, tr http.RoundTripper) *httputil.ReverseProxy {
+ return &httputil.ReverseProxy{
+ Rewrite: func(pr *httputil.ProxyRequest) {
+ setForwarded(pr)
+ pr.Out.URL.Scheme = "https"
+ pr.Out.URL.Host = host // SNI + verification + pool key; the dial is the transport's
+ pr.Out.Host = pr.In.Host // the home edge routes on the real hostname
+ pr.Out.Header.Set(hopHeader, "1")
+ pr.Out.Header.Set(hopAuthHeader, auth)
+ // The same discipline every other Rewrite here keeps, and needed for
+ // the same reason one hop further along: the home edge scrubs it
+ // again before the app, but a header this edge forwards is a header
+ // this edge vouched for.
+ scrubInboundPurgeHeader(pr)
+ },
+ Transport: tr,
+ ErrorHandler: func(w http.ResponseWriter, r *http.Request, err error) {
+ // 502, not the 503 + Retry-After that multiProxy and fleetProxy
+ // answer with: those two KNOW nobody is home (an empty live set, no
+ // connected channel), while this one believed a specific box was
+ // there and could not reach it. A failed dial, a TLS handshake
+ // against a cert the mirror has not distributed yet, or a home edge
+ // that hung up mid-response are all genuinely bad gateways — and the
+ // cert case heals itself on the next mirror poll.
+ log.Printf("edge: backhaul %s → %s (%s): %v", host, box, eip, err)
+ w.WriteHeader(http.StatusBadGateway)
+ },
+ }
+}
+
// dialUnixWithRetry masks the brief gap during an instance restart: while the
// old process has exited and the new one hasn't rebound the socket, dials fail
// with ENOENT/ECONNREFUSED. Retry those for a short window so no in-flight
diff --git a/internal/edge/homemap_test.go b/internal/edge/homemap_test.go
index 3fe510c9c..2bfd2edb1 100644
--- a/internal/edge/homemap_test.go
+++ b/internal/edge/homemap_test.go
@@ -1,10 +1,14 @@
package edge
import (
+ "bufio"
"context"
+ "crypto/tls"
+ "crypto/x509"
"errors"
"fmt"
"io"
+ "net"
"net/http"
"net/http/httptest"
"os"
@@ -495,17 +499,22 @@ func TestHopWithoutAMirroredMeshKeyIsAnOrdinaryClient(t *testing.T) {
}
}
-// An unserved host with NO hop header keeps today's answer exactly: the
-// stale-map 503 is reachable only by a party holding the mesh key, so it
-// never tells a stranger whether a host exists elsewhere in the deployment.
-func TestUnknownHostWithoutAHopIsStillThe404(t *testing.T) {
+// An unserved host that NO box homes keeps today's answer exactly: 404, with
+// no Retry-After, and nothing said about the rest of the deployment.
+//
+// The home map here is deliberately non-empty and deliberately about a
+// DIFFERENT host: the map is consulted (Task 4's backhaul) and misses, which
+// is the only route to this answer now that a homed host is dialled instead.
+// The stale-map 503 stays reachable only by a party holding the mesh key, so
+// the two answers still cannot be told apart by a stranger.
+func TestUnknownHostWithNoHomeIsStillThe404(t *testing.T) {
e, m, _ := hopEdge(t, "annie", "app.test")
putJSON(t, m.s, "control/edges/betty.json",
`{"v":1,"box":"betty","eip":"2.2.2.2","hosts":{"moved.test":{"role":"home"}}}`)
refreshHomeMap(t, m)
- resp := hopRequest(t, e, "moved.test", nil)
+ resp := hopRequest(t, e, "nowhere.test", nil)
if resp.StatusCode != http.StatusNotFound {
- t.Fatalf("status = %d, want the unchanged 404 for an unauthenticated stranger", resp.StatusCode)
+ t.Fatalf("status = %d, want the unchanged 404 for a host nothing in the deployment homes", resp.StatusCode)
}
if got := resp.Header.Get("Retry-After"); got != "" {
t.Fatalf("Retry-After = %q — a stranger must not be told to come back", got)
@@ -525,6 +534,393 @@ func TestOrdinaryClientForgedXFFIsStillDiscarded(t *testing.T) {
}
}
+// ---- backhaul dialing (the sending half) ------------------------------------
+
+// homeEdgeSeen is what a fake home edge was handed on the wire.
+type homeEdgeSeen struct {
+ n int
+ sni, host, hop, auth string
+ xff []string
+}
+
+type homeEdgeRecord struct {
+ mu sync.Mutex
+ seen homeEdgeSeen
+}
+
+func (h *homeEdgeRecord) get() homeEdgeSeen {
+ h.mu.Lock()
+ defer h.mu.Unlock()
+ return h.seen
+}
+
+const homeEdgeBody = "served by the home edge"
+
+// fakeHomeEdge stands in for the box that homes a host: a TLS server that
+// records the ClientHello's SNI and the trust-bearing headers of whatever
+// arrives, then runs h (a plain body writer unless the test needs a protocol
+// switch).
+//
+// It also arms the two dial seams for the test's lifetime — backhaulPort, so
+// the dial reaches an ephemeral httptest port instead of 443, and
+// backhaulRootCAs, so this server's self-signed certificate is trusted the way
+// a real home edge's publicly-trusted one is. Nothing about verification is
+// weakened: httptest's certificate names example.com, so every test below asks
+// for exactly that host and a name mismatch would fail the handshake here just
+// as it would in production.
+func fakeHomeEdge(t *testing.T, h http.Handler) (*homeEdgeRecord, string) {
+ t.Helper()
+ rec := &homeEdgeRecord{}
+ if h == nil {
+ h = http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { fmt.Fprint(w, homeEdgeBody) })
+ }
+ ts := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ rec.mu.Lock()
+ rec.seen.n++
+ rec.seen.host = r.Host
+ rec.seen.hop = r.Header.Get(hopHeader)
+ rec.seen.auth = r.Header.Get(hopAuthHeader)
+ rec.seen.xff = r.Header.Values("X-Forwarded-For")
+ rec.mu.Unlock()
+ h.ServeHTTP(w, r)
+ }))
+ ts.TLS = &tls.Config{GetConfigForClient: func(chi *tls.ClientHelloInfo) (*tls.Config, error) {
+ rec.mu.Lock()
+ rec.seen.sni = chi.ServerName
+ rec.mu.Unlock()
+ return nil, nil // the server's own config, unchanged — this hook only watches
+ }}
+ ts.StartTLS()
+ t.Cleanup(ts.Close)
+
+ eip, port, err := net.SplitHostPort(ts.Listener.Addr().String())
+ if err != nil {
+ t.Fatalf("split home edge addr: %v", err)
+ }
+ pool := x509.NewCertPool()
+ pool.AddCert(ts.Certificate())
+ oldPort, oldCAs := backhaulPort, backhaulRootCAs
+ backhaulPort, backhaulRootCAs = port, pool
+ t.Cleanup(func() { backhaulPort, backhaulRootCAs = oldPort, oldCAs })
+ return rec, eip
+}
+
+// homedOn publishes one box's edges file naming hosts as its own, and lands it
+// in the map — the second box a single-box deployment never has.
+func homedOn(t *testing.T, m *homeMap, box, eip string, hosts ...string) {
+ t.Helper()
+ entries := make([]string, 0, len(hosts))
+ for _, h := range hosts {
+ entries = append(entries, fmt.Sprintf("%q:{\"role\":\"home\"}", h))
+ }
+ putJSON(t, m.s, "control/edges/"+box+".json", fmt.Sprintf(
+ `{"v":1,"box":%q,"eip":%q,"hosts":{%s}}`, box, eip, strings.Join(entries, ",")))
+ refreshHomeMap(t, m)
+}
+
+// The whole of the sending half in one request: a host this box does not serve
+// but another box homes reaches that box over real TLS, with the SNI it will
+// need to present a certificate for, the real Host header, a hop signature the
+// home edge can verify, and an X-Forwarded-For that names only the client.
+func TestBackhaulDialsTheHomeEdgeOverVerifiedTLS(t *testing.T) {
+ e, m, key := hopEdge(t, "annie", "local.test")
+ rec, eip := fakeHomeEdge(t, nil)
+ homedOn(t, m, "betty", eip, "example.com")
+
+ resp := hopRequest(t, e, "example.com", nil)
+ if resp.StatusCode != http.StatusOK {
+ t.Fatalf("status = %d, want 200 — a remote-homed host must be served through its home edge", resp.StatusCode)
+ }
+ if body := readBody(t, resp); body != homeEdgeBody {
+ t.Fatalf("body = %q, want %q", body, homeEdgeBody)
+ }
+ seen := rec.get()
+ if seen.n != 1 {
+ t.Fatalf("home edge saw %d request(s), want exactly one — one hop, ever", seen.n)
+ }
+ if seen.sni != "example.com" {
+ t.Fatalf("SNI = %q, want the request host — the home edge answers with the cert for the host being asked for", seen.sni)
+ }
+ if seen.host != "example.com" {
+ t.Fatalf("Host = %q, want the real hostname — the home edge routes on it", seen.host)
+ }
+ if seen.hop != "1" {
+ t.Fatalf("%s = %q, want 1", hopHeader, seen.hop)
+ }
+ if !verifyHopEitherKey(key, nil, "example.com", seen.auth, time.Now()) {
+ t.Fatalf("%s = %q did not verify under the deployment's mesh key", hopAuthHeader, seen.auth)
+ }
+ // One element, the client. Neither this edge's own address nor the home EIP
+ // may be appended: the home edge restores this chain verbatim on a verified
+ // hop, and every consumer takes the LAST element as the caller
+ // (setForwarded, homemap.go).
+ if len(seen.xff) != 1 || strings.Contains(seen.xff[0], ",") {
+ t.Fatalf("X-Forwarded-For = %q, want a single-element chain naming only the client", seen.xff)
+ }
+}
+
+// Precedence, first rule: a host served HERE is served here, whatever the home
+// map says about it. The map is poisoned to name a reachable home for exactly
+// that host, so a lookup in the wrong order would show up as a dial.
+func TestLocalHitIgnoresTheHomeMap(t *testing.T) {
+ e, m, _ := hopEdge(t, "annie", "example.com")
+ rec, eip := fakeHomeEdge(t, nil)
+ homedOn(t, m, "betty", eip, "example.com")
+
+ resp := hopRequest(t, e, "example.com", nil)
+ if resp.StatusCode != http.StatusOK {
+ t.Fatalf("status = %d, want 200", resp.StatusCode)
+ }
+ if body := readBody(t, resp); !strings.HasPrefix(body, "xff=") {
+ t.Fatalf("body = %q, want the LOCAL backend's answer — a local hit must never consult the home map", body)
+ }
+ if n := rec.get().n; n != 0 {
+ t.Fatalf("home edge saw %d request(s), want none", n)
+ }
+}
+
+// One hop, ever: a request that ARRIVED as a verified hop is refused, never
+// forwarded onward, even when the map offers a live target for it. The fake
+// home edge is the zero-call witness — a re-forward would answer 200 here.
+func TestArrivedHopNeverDialsTheHomeEdge(t *testing.T) {
+ e, m, key := hopEdge(t, "annie", "local.test")
+ rec, eip := fakeHomeEdge(t, nil)
+ homedOn(t, m, "betty", eip, "example.com")
+
+ resp := hopRequest(t, e, "example.com", map[string]string{
+ hopHeader: "1",
+ hopAuthHeader: signHop(key, "example.com", time.Now()),
+ })
+ if resp.StatusCode != http.StatusServiceUnavailable {
+ t.Fatalf("status = %d, want 503 — an arrived hop is refused, not re-forwarded", resp.StatusCode)
+ }
+ if got := resp.Header.Get("Retry-After"); got != "2" {
+ t.Fatalf("Retry-After = %q, want 2", got)
+ }
+ if n := rec.get().n; n != 0 {
+ t.Fatalf("home edge saw %d request(s), want none — a hop must never become a second hop", n)
+ }
+}
+
+// The backhaul's own guard, exercised directly rather than through ServeHTTP's
+// branch above it: if this call ever moves, an arrived hop must still fail
+// closed here instead of being forwarded.
+func TestBackhaulHelperRefusesAnArrivedHopOnItsOwn(t *testing.T) {
+ e, m, _ := hopEdge(t, "annie", "local.test")
+ rec, eip := fakeHomeEdge(t, nil)
+ homedOn(t, m, "betty", eip, "example.com")
+
+ r := httptest.NewRequest("GET", "http://example.com/", nil)
+ r = r.WithContext(context.WithValue(r.Context(), hopVerifiedKey{}, true))
+ w := httptest.NewRecorder()
+ if !e.backhaul(w, r, "example.com") {
+ t.Fatal("backhaul must ANSWER an arrived hop, not fall through to the unknown-host path")
+ }
+ if w.Code != http.StatusServiceUnavailable {
+ t.Fatalf("status = %d, want the stale-map refusal", w.Code)
+ }
+ if n := rec.get().n; n != 0 {
+ t.Fatalf("home edge saw %d request(s), want none", n)
+ }
+}
+
+// Fail closed on the SENDING side too: with no mesh key mirrored there is
+// nothing to prove the hop with, and signing under an empty key would cross a
+// region only to be served as an anonymous client. The request gets today's
+// unknown-host answer instead.
+func TestBackhaulRefusesToDialWithoutAMeshKey(t *testing.T) {
+ e, m, _ := hopEdge(t, "annie", "local.test")
+ rec, eip := fakeHomeEdge(t, nil)
+ homedOn(t, m, "betty", eip, "example.com")
+ if err := m.s.Delete(context.Background(), meshKeyKey); err != nil {
+ t.Fatalf("delete mesh key: %v", err)
+ }
+ refreshHomeMap(t, m)
+ if cur, next := m.meshKeys(); len(cur) != 0 || len(next) != 0 {
+ t.Fatal("precondition: the mirror must hold no key")
+ }
+
+ resp := hopRequest(t, e, "example.com", nil)
+ if resp.StatusCode != http.StatusNotFound {
+ t.Fatalf("status = %d, want the unchanged 404 — an unprovable hop is not dialled", resp.StatusCode)
+ }
+ if n := rec.get().n; n != 0 {
+ t.Fatalf("home edge saw %d request(s), want none — never sign with an empty key", n)
+ }
+}
+
+// A catch-all row in another box's edges file resolves through the same
+// one-label wildcard rule lookupUpstream uses — but the signature must be over
+// the CONCRETE request host, because that is what the home edge verifies
+// against. Signing the map KEY would produce a MAC for "*.com", which is not a
+// request host under any canonicalization.
+func TestBackhaulSignsTheConcreteHostNotTheWildcardKey(t *testing.T) {
+ e, m, key := hopEdge(t, "annie", "local.test")
+ rec, eip := fakeHomeEdge(t, nil)
+ homedOn(t, m, "betty", eip, "*.com")
+ if _, _, ok := m.Home("example.com"); !ok {
+ t.Fatal("precondition: the wildcard row must offer a home for example.com")
+ }
+
+ if resp := hopRequest(t, e, "example.com", nil); resp.StatusCode != http.StatusOK {
+ t.Fatalf("status = %d, want 200", resp.StatusCode)
+ }
+ seen := rec.get()
+ if !verifyHop(key, "example.com", seen.auth, time.Now()) {
+ t.Fatalf("%s = %q must verify against the concrete request host", hopAuthHeader, seen.auth)
+ }
+ if verifyHop(key, "*.com", seen.auth, time.Now()) {
+ t.Fatalf("%s = %q verifies against the wildcard map key — the MAC must bind the concrete host", hopAuthHeader, seen.auth)
+ }
+}
+
+// A WebSocket must compose through the hop with no code of its own: the dial
+// branch sits above ServeHTTP's cache/compression wrapping, so the client's
+// hijackable writer reaches httputil.ReverseProxy's protocol-switch path, and
+// the transport speaks HTTP/1.1 (no ForceAttemptHTTP2) so that path works.
+// Bytes must then flow both ways with no whole-body deadline cutting them off.
+func TestBackhaulCarriesAWebSocketUpgrade(t *testing.T) {
+ e, m, _ := hopEdge(t, "annie", "local.test")
+ rec, eip := fakeHomeEdge(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ hj, ok := w.(http.Hijacker)
+ if !ok {
+ http.Error(w, "not hijackable", http.StatusInternalServerError)
+ return
+ }
+ conn, buf, err := hj.Hijack()
+ if err != nil {
+ return
+ }
+ defer conn.Close()
+ buf.WriteString("HTTP/1.1 101 Switching Protocols\r\nUpgrade: websocket\r\nConnection: Upgrade\r\n\r\n")
+ if err := buf.Flush(); err != nil {
+ return
+ }
+ line, err := buf.ReadString('\n')
+ if err != nil {
+ return
+ }
+ buf.WriteString("home:" + line)
+ buf.Flush()
+ }))
+ homedOn(t, m, "betty", eip, "example.com")
+
+ srv := httptest.NewServer(e)
+ t.Cleanup(srv.Close)
+ c, err := net.Dial("tcp", strings.TrimPrefix(srv.URL, "http://"))
+ if err != nil {
+ t.Fatalf("dial edge: %v", err)
+ }
+ defer c.Close()
+ if err := c.SetDeadline(time.Now().Add(10 * time.Second)); err != nil {
+ t.Fatalf("set deadline: %v", err)
+ }
+ if _, err := fmt.Fprint(c, "GET /socket HTTP/1.1\r\nHost: example.com\r\nConnection: Upgrade\r\nUpgrade: websocket\r\n\r\n"); err != nil {
+ t.Fatalf("write handshake: %v", err)
+ }
+ br := bufio.NewReader(c)
+ resp, err := http.ReadResponse(br, nil)
+ if err != nil {
+ t.Fatalf("read handshake response: %v", err)
+ }
+ if resp.StatusCode != http.StatusSwitchingProtocols {
+ t.Fatalf("handshake status = %d, want 101 — an upgrade must switch protocols through the hop", resp.StatusCode)
+ }
+ if _, err := fmt.Fprint(c, "ping\n"); err != nil {
+ t.Fatalf("write post-upgrade frame: %v", err)
+ }
+ line, err := br.ReadString('\n')
+ if err != nil {
+ t.Fatalf("read post-upgrade frame: %v", err)
+ }
+ if line != "home:ping\n" {
+ t.Fatalf("post-upgrade echo = %q, want %q — bytes must flow both ways across the backhaul", line, "home:ping\n")
+ }
+ if seen := rec.get(); seen.hop != "1" {
+ t.Fatalf("%s = %q — an upgrade is a hop like any other", hopHeader, seen.hop)
+ }
+}
+
+// A home whose edge cannot be reached is a bad gateway, not a 503: this box
+// believed a specific box was there. The unreachable target here is the
+// documented cert-less bootstrap window's shape too — a dial or handshake that
+// does not complete, healed by the next mirror poll.
+func TestBackhaulToAnUnreachableHomeIsA502(t *testing.T) {
+ e, m, _ := hopEdge(t, "annie", "local.test")
+ // A port nothing listens on, reached through the same seam the fake home
+ // edge uses. 127.0.0.1 refuses immediately rather than hanging for the dial
+ // budget.
+ ln, err := net.Listen("tcp", "127.0.0.1:0")
+ if err != nil {
+ t.Fatalf("listen: %v", err)
+ }
+ eip, port, err := net.SplitHostPort(ln.Addr().String())
+ if err != nil {
+ t.Fatalf("split: %v", err)
+ }
+ ln.Close()
+ old := backhaulPort
+ backhaulPort = port
+ t.Cleanup(func() { backhaulPort = old })
+ homedOn(t, m, "betty", eip, "example.com")
+
+ resp := hopRequest(t, e, "example.com", nil)
+ if resp.StatusCode != http.StatusBadGateway {
+ t.Fatalf("status = %d, want 502 for a home edge that could not be reached", resp.StatusCode)
+ }
+ if got := resp.Header.Get("Retry-After"); got != "" {
+ t.Fatalf("Retry-After = %q — 503 + Retry-After is reserved for knowing nobody is home", got)
+ }
+}
+
+// One transport per home BOX. The pool is what makes a backhaul cost one
+// handshake rather than one per request; keying it per box rather than per
+// host is what keeps a catch-all home — where every invented subdomain is a
+// backhaul target — from growing the map without limit. Per-host identity
+// still exists, one layer down: the outbound authority is the hostname, so SNI
+// and pooled connections are per host inside the box's transport (the SNI
+// assertion in TestBackhaulDialsTheHomeEdgeOverVerifiedTLS is that half).
+func TestBackhaulTransportsAreCachedPerHomeBox(t *testing.T) {
+ e, _, _ := hopEdge(t, "annie", "local.test")
+ a := e.backhaulTransport("1.1.1.1")
+ if b := e.backhaulTransport("1.1.1.1"); b != a {
+ t.Fatal("the same home box must reuse one transport, or every request pays a fresh TLS handshake")
+ }
+ if b := e.backhaulTransport("2.2.2.2"); b == a {
+ t.Fatal("a different home box must get its own transport — the dial address is baked into it")
+ }
+ if sn := a.TLSClientConfig.ServerName; sn != "" {
+ t.Fatalf("ServerName = %q, want it unset — pinning one host per transport is what a wildcard home would multiply", sn)
+ }
+ if a.TLSClientConfig.InsecureSkipVerify {
+ t.Fatal("the hop verifies the home edge's certificate — always")
+ }
+}
+
+// Two hosts homed on one box share that box's transport, and each still
+// reaches the home edge under its OWN name: the shared transport is the memory
+// property, the per-host SNI is the correctness one.
+func TestBackhaulSharesOneTransportAcrossAWildcardHome(t *testing.T) {
+ e, m, _ := hopEdge(t, "annie", "local.test")
+ rec, eip := fakeHomeEdge(t, nil)
+ homedOn(t, m, "betty", eip, "*.com")
+
+ if resp := hopRequest(t, e, "example.com", nil); resp.StatusCode != http.StatusOK {
+ t.Fatalf("status = %d, want 200", resp.StatusCode)
+ }
+ if sni := rec.get().sni; sni != "example.com" {
+ t.Fatalf("SNI = %q, want the concrete request host", sni)
+ }
+ n := 0
+ e.backhaulTransports.Range(func(_, _ any) bool { n++; return true })
+ if n != 1 {
+ t.Fatalf("%d transports cached for one home box, want 1 — a catch-all home must not mint one per hostname", n)
+ }
+ if _, ok := e.backhaulTransports.Load(eip); !ok {
+ t.Fatalf("the cached transport must be keyed by the home box's EIP (%s)", eip)
+ }
+}
+
// oneKeyFailingStore fails Get for one named key — a single unreadable object
// mid-poll, the case the carry-forward exists for.
type oneKeyFailingStore struct {