amadan

carlos / platform Public

edge: home map mirror + one-hop acceptance

Task 3 of the multi-region backhaul plan (spec 2026-08-20, Part 3): the
RECEIVING half of edge-to-edge backhaul, on the production request path.

- internal/edge/homemap.go: homeMap mirrors control/edges/ on the cert
  mirror's 30s list-then-get, carry-forward, last-good idiom (copied,
  not reinvented) and folds every box's file into one host -> home
  answer. The fold applies lookupUpstream's one-label wildcard rule
  (Task 1 publishes registry hosts verbatim, "*.parent" rows included),
  reads only role:"home" and ignores roles this build does not know,
  and excludes OWN-BOX entries, blank-EIP boxes and duplicate claims
  (first box in sorted order wins, identically on every edge). The mesh
  keys ride the same refresh — verified per request from cache, never a
  store read — so who homes a host and what proves a hop come from one
  view of the bucket.
- Hop acceptance in ServeHTTP: X-Carlos-Hop + a signature that passes
  verifyHopEitherKey (dual-accept, never the single-key verifyHop) is
  served locally with the inbound X-Forwarded-For believed and this
  edge appended to it; the same headers without a valid signature are
  an ordinary client with every X-Forwarded-* stripped. A verified hop
  naming a host this box does not serve gets 503 + Retry-After: 2 from
  inside the lookupUpstream-miss branch, ahead of where Task 4's dial
  goes — one hop, ever, is structural, not a policy check. The refusal
  is reachable only by a holder of the mesh key, so a stranger asking
  the same question still gets the unchanged 404.
- setForwarded replaces pr.SetXForwarded() in all five proxy Rewrite
  funcs: ReverseProxy deletes inbound X-Forwarded-* before Rewrite
  runs, so a verified hop's chain has to be restored onto the outbound
  request for SetXForwarded to append to it. Gated on a context mark
  nothing outside the package can set.
- WithBoxLabel hands the edge its own box label from the agent, the
  same metricsBoxLabel call the edges publisher uses; without it the
  map offers nothing rather than risk dialing itself.

Single-box inertness: the only edges file is this box's own, own-box
exclusion drops it, Home() never offers a target, and nothing signs
hops — the verified branch is unreachable until a second box publishes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Paul Campbell pushed by paul@keymail.dev 42844ae29637bb6b703c5c03f17c4624842777f8 parent 0c07490
5 files changed, +1241 −6
  • internal/edge/edge.go +60 −5
  • internal/edge/homemap.go +564 −0
  • internal/edge/homemap_test.go +530 −0
  • internal/edge/mesh_test.go +79 −0
  • internal/host/agent.go +8 −1
diff --git a/internal/edge/edge.go b/internal/edge/edge.go
index 29c0a6568..2b57d2d1a 100644
--- a/internal/edge/edge.go
+++ b/internal/edge/edge.go
@@ -168,6 +168,17 @@ type Edge struct {
// every single-box deployment's behaviour to date: mint as always.
// Serving is never gated by it.
issuer IssuerGate
+ // box is this box's own label — the SAME derivation its edges file is
+ // published under (internal/host's metricsBoxLabel), handed down through
+ // WithBoxLabel for the reason the issuer gate is: that package imports
+ // this one and never the reverse. It is what the home map excludes its own
+ // hosts by; "" (bare `carlos edge`) leaves the map offering nothing.
+ box string
+ // homes, when non-nil, is the mirror of control/edges/ plus the mesh keys
+ // (homemap.go): who homes a host this box does not serve, and what proves
+ // an inbound hop. Atomic because it is armed in RunWithActivatorReady after
+ // construction, and read on the request path while that is happening.
+ homes atomic.Pointer[homeMap]
}
// RunOption configures the Edge that Run/RunWithActivator* builds, before it
@@ -181,6 +192,14 @@ type RunOption func(*Edge)
// as it always has. See IssuerGate (certsync.go).
func WithIssuerGate(g IssuerGate) RunOption { return func(e *Edge) { e.issuer = g } }
+// WithBoxLabel hands the edge this box's own label, the one its edges file is
+// published under (internal/host's metricsBoxLabel — the same call the edges
+// publisher makes, so the two can never disagree about which file is ours).
+// Without it the home map excludes nothing and therefore offers nothing: an
+// edge that cannot recognise its own hosts must not hand out dial targets.
+// See homemap.go.
+func WithBoxLabel(label string) RunOption { return func(e *Edge) { e.box = label } }
+
// upstream is one served host: the handler for its backend, plus — when the
// activator manages it — the route itself, so a request can trigger the wake.
type upstream struct {
@@ -782,8 +801,29 @@ func (e *Edge) ServeHTTP(w http.ResponseWriter, r *http.Request) {
e.tunnels.ServeHTTP(w, r)
return
}
+ // Backhaul hop acceptance (homemap.go). Deliberately placed HERE — after
+ // the reserved tunnel hostname's branch above, which stays byte-for-byte
+ // what it was (that hostname is not a registry route, so it can never
+ // appear in any box's edges file and can never be a hop's target), and
+ // before the route lookup, so that a verified hop naming a host we do not
+ // serve is answered by the stale-map refusal rather than the ordinary
+ // unknown-host answer.
+ //
+ // On a request carrying no X-Carlos-Hop this is one Header.Get and
+ // nothing else.
+ r, hopped := e.acceptHop(r, host)
+
up, ok := e.lookupUpstream(host)
if !ok {
+ if hopped {
+ // A PROVEN hop from a sibling edge, for a host that has moved (or
+ // never lived) here: its map is stale. 503 + Retry-After: 2, and
+ // this return is the structural half of "one hop, ever" — the
+ // backhaul dial Task 4 adds lives further down this same branch,
+ // so an accepted hop physically cannot reach it.
+ staleHopRefusal(w)
+ return
+ }
e.unknownHost(w, r)
return
}
@@ -1364,7 +1404,7 @@ func unixProxy(socketPath string) *httputil.ReverseProxy {
dialer := &net.Dialer{}
return &httputil.ReverseProxy{
Rewrite: func(pr *httputil.ProxyRequest) {
- pr.SetXForwarded()
+ setForwarded(pr)
pr.Out.URL.Scheme = "http"
pr.Out.URL.Host = "carlos" // placeholder; the unix dialer ignores it
pr.Out.Host = pr.In.Host // backend sees the real hostname
@@ -1389,7 +1429,7 @@ func unixProxy(socketPath string) *httputil.ReverseProxy {
func tcpProxy(addr string) *httputil.ReverseProxy {
return &httputil.ReverseProxy{
Rewrite: func(pr *httputil.ProxyRequest) {
- pr.SetXForwarded()
+ setForwarded(pr)
pr.Out.URL.Scheme = "http"
pr.Out.URL.Host = addr
pr.Out.Host = pr.In.Host // backend sees the real hostname
@@ -1476,7 +1516,7 @@ func multiProxy(host string, p *proberSet) *httputil.ReverseProxy {
dialer := &net.Dialer{Timeout: multiDialTimeout}
return &httputil.ReverseProxy{
Rewrite: func(pr *httputil.ProxyRequest) {
- pr.SetXForwarded()
+ setForwarded(pr)
pr.Out.URL.Scheme = "http"
pr.Out.URL.Host = multiDialHost // placeholder; the dialer below ignores it
pr.Out.Host = pr.In.Host // backend sees the real hostname
@@ -1561,7 +1601,7 @@ func multiProxy(host string, p *proberSet) *httputil.ReverseProxy {
func tunnelProxy(name string, hub *TunnelHub) *httputil.ReverseProxy {
return &httputil.ReverseProxy{
Rewrite: func(pr *httputil.ProxyRequest) {
- pr.SetXForwarded()
+ setForwarded(pr)
pr.Out.URL.Scheme = "https"
pr.Out.URL.Host = pr.In.Host // h2 :authority = the real hostname
pr.Out.Host = pr.In.Host
@@ -1615,7 +1655,7 @@ func fleetChannelFor(fleet, boxLabel string, hub *TunnelHub) string {
func fleetProxy(fleet, boxLabel string, hub *TunnelHub) *httputil.ReverseProxy {
return &httputil.ReverseProxy{
Rewrite: func(pr *httputil.ProxyRequest) {
- pr.SetXForwarded()
+ setForwarded(pr)
pr.Out.URL.Scheme = "https"
pr.Out.URL.Host = pr.In.Host // h2 :authority = the real hostname
pr.Out.Host = pr.In.Host
@@ -1818,6 +1858,21 @@ func RunWithActivatorReady(args []string, act Activator, cache *Cache, ready fun
cancel()
}
+ // The home map and the mesh keys (homemap.go), armed beside the cert
+ // mirror and on the same terms: nil when there is no deployment store to
+ // mirror from, one synchronous refresh before anything listens so the
+ // first request does not meet an empty map, then the 30s loop. Stored
+ // before either listener binds, so ServeHTTP never sees a half-armed edge.
+ if hm := e.newHomeMap(); hm != nil {
+ ctx, cancel := context.WithTimeout(context.Background(), homeMapBootTimeout)
+ if err := hm.Refresh(ctx); err != nil {
+ log.Printf("edge: first home map refresh failed: %v — hosts homed on another box are unreachable from here until a later poll lands", err)
+ }
+ cancel()
+ e.homes.Store(hm)
+ go hm.run(make(chan struct{})) // process-lifetime; never stopped, like the cert mirror's
+ }
+
m := &autocert.Manager{
Prompt: autocert.AcceptTOS,
Cache: certCache,
diff --git a/internal/edge/homemap.go b/internal/edge/homemap.go
new file mode 100644
index 000000000..7ea68def7
--- /dev/null
+++ b/internal/edge/homemap.go
@@ -0,0 +1,564 @@
+package edge
+
+// The home map, and the RECEIVING half of edge-to-edge backhaul (spec
+// docs/superpowers/specs/2026-08-20-multi-region-ingress-design.md, Part 3,
+// Task 3).
+//
+// Two things live here because they are read on the same poll and used in the
+// same breath:
+//
+// - homeMap mirrors control/edges/ — the per-box files internal/host's
+// edgesFilePass publishes (its edgesFile shape, parsed here as
+// edgesFileView) — and folds them into one deployment-wide host→home
+// answer, Home(host). It is the same 30s list-then-get, carry-forward,
+// last-good mirror certsync.go's certMirror is; that shape survived heavy
+// review and is copied rather than reinvented. It NEVER writes: the
+// writer map gives control/edges/<box>.json to that box's own loop, and
+// nothing in this package bumps control/serial.
+//
+// - the mesh keys ride the SAME refresh (loadMeshKeys, mesh.go), so a
+// request path never reads the store: who homes a host and what key
+// proves a hop are answered from one poll's view of the bucket rather
+// than two.
+//
+// acceptHop is where a hop lands on the request path. The trust reasoning it
+// rests on — why the MAC binds the concrete host, why ±hopSkew is the replay
+// bound, and why dual-accept rotation means the request path must call
+// verifyHopEitherKey and never the single-key verifyHop — is written out in
+// mesh.go's file header; read that, this code does not re-derive it. Two
+// consequences of it are worth naming HERE because they are this file's
+// decisions rather than mesh.go's:
+//
+// - The replay envelope is deliberately loose and that is affordable. A
+// captured X-Carlos-Hop-Auth is good for at most 2×hopSkew against ONE
+// host, and what it buys a replayer is a request to a route the edge
+// already serves to the whole internet — no new reachability, no
+// privilege. What it must NOT buy is the ability to state who the client
+// was, which is why verification gates exactly one thing: whether the
+// inbound X-Forwarded-For is believed.
+//
+// - XFF is trusted as an APPENDED element, never as a replacement. On a
+// verified hop the inbound chain is carried onto the outbound request and
+// the sending edge's own address is appended to it (setForwarded, below),
+// so the app sees "<client>, <sending edge>" — the same shape a
+// single-region app already sees, with one more hop in it. On anything
+// else the chain is discarded outright: an unverified sender is an
+// ordinary client, and an ordinary client has never been allowed to state
+// its own address here.
+//
+// One hop, ever, is STRUCTURAL and not a policy check: a verified hop that
+// names a host this box does not serve leaves through staleHopRefusal (503 +
+// Retry-After: 2) from inside the lookupUpstream-miss branch, BEFORE the point
+// where Task 4 adds the backhaul dial. There is no path from an accepted hop
+// to Home() at all, so a re-forward cannot be introduced by adding a branch —
+// only by deleting one.
+//
+// Single-box inertness: with one platform box the only edges file is this
+// box's own, own-box exclusion drops it, and Home() therefore never offers a
+// target. Nothing signs hops either (Task 4 is the only signer, and it needs a
+// target), so acceptHop's verified branch is unreachable on a single-box
+// deployment — this task ships the receiving half, dormant.
+
+import (
+ "context"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "log"
+ "net/http"
+ "net/http/httputil"
+ "os"
+ "slices"
+ "strings"
+ "sync"
+ "time"
+
+ "github.com/carlosframework/platform/internal/release"
+)
+
+// edgesPrefix is where every box publishes what it homes — one flat object
+// per box, control/edges/<box>.json (internal/host's edgesFileKey).
+const edgesPrefix = "control/edges/"
+
+// The hop's two headers, verbatim from the spec's Part 3. Task 2 left them
+// prose-only; this file is their first consumer, and Task 4's signer imports
+// these constants rather than spelling the strings a second time.
+const (
+ hopHeader = "X-Carlos-Hop"
+ hopAuthHeader = "X-Carlos-Hop-Auth"
+)
+
+// edgeRoleHome is the only role v1 writes, and the only one that routes.
+// Unknown roles (the spec's read-local anticipation) are IGNORED rather than
+// refused: a newer box publishing a role this build has never heard of must
+// not take its whole file — or the poll — down with it.
+const edgeRoleHome = "home"
+
+// homeMapInterval / homeMapTimeout / homeMapBootTimeout mirror the cert
+// mirror's cadence and budgets exactly (certsync.go), for the same reasons: a
+// home move is a placement event that converge heals, the poll runs on its own
+// goroutine and blocks nothing, and the ONE boot refresh must delay the front
+// door by seconds rather than minutes. Vars so tests can shorten them.
+var (
+ homeMapInterval = 30 * time.Second
+ homeMapTimeout = 2 * time.Minute
+ homeMapBootTimeout = 20 * time.Second
+)
+
+// edgesFileView is the READER's half of internal/host's edgesFile wire shape
+// (edgesfile.go). Deliberately a separate declaration rather than a shared
+// type: internal/host imports this package and never the reverse, and the
+// fields this side needs are the stable three (box, eip, hosts) plus the
+// version. Additive growth on the writer's side lands here as fields this
+// decoder ignores, which is the point.
+type edgesFileView struct {
+ V int `json:"v"`
+ Box string `json:"box"`
+ EIP string `json:"eip"`
+ Hosts map[string]edgesHostView `json:"hosts"`
+}
+
+// edgesHostView is the per-host object. An object rather than a bare string is
+// what makes the role field addable without a wire break — see edgeRoleHome.
+type edgesHostView struct {
+ Role string `json:"role"`
+}
+
+// homeEntry is one folded answer: where to dial, and which box that is.
+type homeEntry struct {
+ eip string
+ box string
+}
+
+// homeSnapshot is one poll's result, swapped in whole and immutable once
+// published, so a reader holds no lock past the pointer read.
+type homeSnapshot struct {
+ // files is the parsed per-box file, keyed by its BUCKET key. This is the
+ // carry-forward unit: an object that could not be read (or could not be
+ // parsed) this poll keeps its previous parse, per box, so one bad file
+ // never costs the deployment its whole map.
+ files map[string]edgesFileView
+ // hosts is the fold: host → home, own box already excluded.
+ hosts map[string]homeEntry
+}
+
+var emptyHomeSnapshot = &homeSnapshot{files: map[string]edgesFileView{}, hosts: map[string]homeEntry{}}
+
+type homeMap struct {
+ s release.Store
+ // self is THIS box's label — the same derivation the box's own edges file
+ // is published under (internal/host's metricsBoxLabel), handed down at
+ // edge construction through WithBoxLabel. "" means this process does not
+ // know which box it is, and Home then offers nothing at all: see its doc.
+ self string
+
+ // refreshMu serializes Refresh's whole body, for the reason certMirror's
+ // does — two overlapping polls could otherwise publish out of order, the
+ // slower one clobbering the fresher.
+ refreshMu sync.Mutex
+
+ mu sync.RWMutex
+ snap *homeSnapshot
+ // partialPolls counts polls that could not read (or parse) every object,
+ // for the first-and-every-tenth log cadence.
+ partialPolls int
+
+ // The mirrored mesh keys, read on the same refresh. Replaced only on a
+ // SUCCESSFUL load: a bucket blip leaves the last good key verifying hops,
+ // exactly as a blip leaves the last good host map routing them.
+ keyMu sync.RWMutex
+ keyCurrent []byte
+ keyNext []byte
+}
+
+func newHomeMap(s release.Store, self string) *homeMap {
+ return &homeMap{s: s, self: self, snap: emptyHomeSnapshot}
+}
+
+// newHomeMap is the Edge's map, or nil when there is nothing to mirror FROM —
+// the same no-deployment-store case that leaves the cert mirror nil. A box
+// with no deployment bucket is not part of any mesh: it offers no backhaul
+// target AND verifies no inbound hop (no key to verify against), which is the
+// fail-closed answer either way.
+func (e *Edge) newHomeMap() *homeMap {
+ if e.staticStore == nil {
+ return nil
+ }
+ return newHomeMap(e.staticStore, e.box)
+}
+
+// Refresh rebuilds the host fold and re-reads the mesh keys. The two halves
+// fail INDEPENDENTLY — a bucket that answered the edges list but not the key
+// object should still land the fold — so both run and their errors join.
+// Whatever fails leaves its own last-good in place; the caller logs.
+func (m *homeMap) Refresh(ctx context.Context) error {
+ m.refreshMu.Lock()
+ defer m.refreshMu.Unlock()
+ return errors.Join(m.refreshHosts(ctx), m.refreshKeys(ctx))
+}
+
+// refreshHosts rebuilds the whole map from one list plus a get per key — no
+// delta cleverness, for certMirror.Refresh's reason: a full rebuild is what
+// makes a RETIRED box's hosts actually leave the map.
+//
+// An error leaves the last-good snapshot in place. That matters as much here
+// as it does for certs: an empty map on a bucket blip would stop every
+// backhaul in the deployment at once, turning a blip into a regional outage
+// for every host homed elsewhere.
+func (m *homeMap) refreshHosts(ctx context.Context) error {
+ keys, err := m.s.List(ctx, edgesPrefix)
+ if err != nil {
+ return fmt.Errorf("list %s: %w", edgesPrefix, err)
+ }
+ prev := m.snapshot()
+ next := make(map[string]edgesFileView, len(keys))
+ // failed counts both fault classes; unread counts only the objects the
+ // BUCKET would not hand over, which is the one that says "this is not an
+ // object fault at all" when it covers everything (see below).
+ var tried, failed, unread int
+ var lastErr error
+ for _, key := range keys {
+ // The prefix is flat and every member is <box>.json. Anything else
+ // under it was not written by this deployment's edges loop.
+ name, ok := strings.CutPrefix(key, edgesPrefix)
+ if !ok || !strings.HasSuffix(name, ".json") || strings.Contains(name, "/") {
+ continue
+ }
+ tried++
+ body, _, err := m.s.Get(ctx, key)
+ if errors.Is(err, os.ErrNotExist) {
+ // Raced a delete between the list and the get — the box was
+ // retired mid-poll. Not a fault; it is simply gone.
+ continue
+ }
+ // ONE bad object is not a bad poll, whichever way it went bad.
+ // Aborting would discard every other box's file that landed this tick,
+ // and a permanently unreadable (or permanently corrupt) key would
+ // freeze the map forever. Carry this box's previous parse forward and
+ // keep going.
+ if err != nil {
+ unread++
+ } else if f, perr := parseEdgesFile(key, body); perr != nil {
+ err = perr
+ } else {
+ next[key] = f
+ continue
+ }
+ failed++
+ lastErr = err
+ if old, ok := prev.files[key]; ok {
+ next[key] = old
+ }
+ }
+ if tried > 0 && unread == tried {
+ // Nothing at all could be READ: a bucket fault, not an object fault,
+ // reported like a failed List — last-good stands. Note the bound is
+ // unread and not failed: a deployment whose only edges file is a
+ // corrupt one has a bucket that answered perfectly well, and calling
+ // that a bucket fault would report a blip that is not happening.
+ return fmt.Errorf("get %s*: every one of %d object(s) was unreadable, last: %w", edgesPrefix, tried, lastErr)
+ }
+ if failed > 0 {
+ m.logPartialPoll(failed, tried, lastErr)
+ }
+ snap := &homeSnapshot{files: next, hosts: foldHomes(next, m.self)}
+ m.mu.Lock()
+ m.snap = snap
+ m.mu.Unlock()
+ return nil
+}
+
+// parseEdgesFile decodes one box's file. A parse failure gets the same
+// treatment an unreadable object gets — carry the last good parse forward —
+// because a half-written body is no more usable than a body that never
+// arrived, and the fix for both is the publisher's next pass.
+func parseEdgesFile(key string, body []byte) (edgesFileView, error) {
+ var f edgesFileView
+ if err := json.Unmarshal(body, &f); err != nil {
+ return edgesFileView{}, fmt.Errorf("%s: %w", key, err)
+ }
+ if f.V < 1 {
+ // No version at all: not a file this reader wrote a decoder for. A
+ // HIGHER version is accepted on purpose — the schema grows additively
+ // (the plan's global constraint), so a newer box's file still carries
+ // the box/eip/hosts this reader needs, and refusing it would black out
+ // backhaul to every box that upgraded first.
+ return edgesFileView{}, fmt.Errorf("%s: unversioned edges file", key)
+ }
+ return f, nil
+}
+
+// logPartialPoll reports unreadable or unparseable files on the loop's own
+// first-and-every-tenth cadence: a permanently bad object would otherwise say
+// the same thing every 30 seconds forever.
+func (m *homeMap) logPartialPoll(failed, tried int, err error) {
+ m.mu.Lock()
+ m.partialPolls++
+ n := m.partialPolls
+ m.mu.Unlock()
+ if n == 1 || n%10 == 0 {
+ log.Printf("edge: %d of %d object(s) under %s unreadable this poll (#%d, last: %v) — the previous copy of each is carried forward", failed, tried, edgesPrefix, n, err)
+ }
+}
+
+// foldHomes turns the per-box files into one host→home map.
+//
+// Three exclusions, each of which is a request this box must never make:
+//
+// - OWN BOX. A host homed here is SERVED here; offering it as a backhaul
+// target would have the edge dial itself. Excluded by the file's own box
+// field rather than by its object name, because the field is what the
+// publisher stamps and what self is derived to match.
+// - NO EIP. Task 1 publishes a box with no discoverable public IP as
+// eip:"" so the DR runbook still gets its hosts list. There is nothing to
+// dial, so it is not a target.
+// - A ROLE THIS BUILD DOES NOT KNOW. Only "home" routes. The role field
+// exists so read-local roles can be added later; a reader that treated
+// every role as a home would send traffic to a box that had declared it
+// serves only reads.
+//
+// Iteration is over SORTED file keys so a duplicate claim — two boxes both
+// saying they home one host, which is a placement bug mid-move — resolves the
+// same way on every edge in the deployment. Convergent-but-wrong beats
+// split-and-wrong: the whole fleet aiming at one box is a condition that heals
+// on the next publish, where half the fleet aiming at each is a condition that
+// looks fine from any single box.
+func foldHomes(files map[string]edgesFileView, self string) map[string]homeEntry {
+ keys := make([]string, 0, len(files))
+ for k := range files {
+ keys = append(keys, k)
+ }
+ slices.Sort(keys)
+ hosts := make(map[string]homeEntry)
+ for _, k := range keys {
+ f := files[k]
+ if f.Box == "" || f.Box == self || f.EIP == "" {
+ continue
+ }
+ for host, h := range f.Hosts {
+ if host == "" || h.Role != edgeRoleHome {
+ continue
+ }
+ if _, dup := hosts[host]; dup {
+ continue // first box in sorted order wins, everywhere
+ }
+ hosts[host] = homeEntry{eip: f.EIP, box: f.Box}
+ }
+ }
+ return hosts
+}
+
+func (m *homeMap) snapshot() *homeSnapshot {
+ m.mu.RLock()
+ defer m.mu.RUnlock()
+ return m.snap
+}
+
+// Home answers "which OTHER box homes this host, and where do I dial it?".
+//
+// The lookup is lookupUpstream's, rule for rule, and that is load-bearing
+// rather than tidy: Task 1 publishes registry row hosts VERBATIM, catch-all
+// "*.<parent>" rows included, so an exact match wins and a miss retries with
+// the leading dot of the FIRST label swapped for "*" — one label, never two,
+// and never the parent matching its own wildcard. Deliberately NOT
+// case-folded, for the same reason: a host lookupUpstream would miss on case
+// must miss here too, or this edge would backhaul a request the home edge
+// would then refuse with a 503.
+//
+// A nil map (no deployment store) and a map with no self label both answer
+// "nothing". The self case is fail-closed on purpose: without knowing which
+// box it is, this edge cannot exclude its own hosts, and an edge that cannot
+// exclude itself must not be handing out dial targets at all.
+func (m *homeMap) Home(host string) (eip, box string, ok bool) {
+ if m == nil || m.self == "" {
+ return "", "", false
+ }
+ hosts := m.snapshot().hosts
+ if h, ok := hosts[host]; ok {
+ return h.eip, h.box, true
+ }
+ if i := strings.IndexByte(host, '.'); i > 0 {
+ if h, ok := hosts["*"+host[i:]]; ok {
+ return h.eip, h.box, true
+ }
+ }
+ return "", "", false
+}
+
+// refreshKeys re-reads the mesh key pair. loadMeshKeys is already tolerant of
+// a malformed .next and fail-closed on the current key (mesh.go); this only
+// adds the mirror's own posture on top — the pair is replaced ONLY on a clean
+// read, so a bucket blip leaves the last good key verifying hops.
+//
+// A clean read that finds NOTHING does replace the pair with nil, and that is
+// not a blip: an absent control/edge-mesh-key means no hop can be proven, and
+// refusing every hop (they degrade to ordinary clients) is the only safe
+// reading of "the deployment has no mesh key".
+func (m *homeMap) refreshKeys(ctx context.Context) error {
+ current, next, err := loadMeshKeys(ctx, m.s)
+ if err != nil {
+ return err
+ }
+ m.keyMu.Lock()
+ m.keyCurrent, m.keyNext = current, next
+ m.keyMu.Unlock()
+ return nil
+}
+
+// meshKeys is the request path's cached view of the key pair — never a store
+// read per request. Both may be nil (nothing minted yet, or nothing mirrored
+// yet), which verifyHopEitherKey answers as "no".
+func (m *homeMap) meshKeys() (current, next []byte) {
+ if m == nil {
+ return nil, nil
+ }
+ m.keyMu.RLock()
+ defer m.keyMu.RUnlock()
+ return m.keyCurrent, m.keyNext
+}
+
+// run polls until stop closes — the edge's standing background-loop shape,
+// the same one certMirror.run uses: fixed ticker, errors counted and logged at
+// the first and then every tenth, recovery logged once.
+func (m *homeMap) run(stop <-chan struct{}) {
+ log.Printf("edge: home map armed (%s poll of %s*, this box is %q) — hosts homed on another box are reachable through it", homeMapInterval, edgesPrefix, m.self)
+ tick := time.NewTicker(homeMapInterval)
+ defer tick.Stop()
+ errs := 0
+ for {
+ select {
+ case <-stop:
+ return
+ case <-tick.C:
+ }
+ ctx, cancel := context.WithTimeout(context.Background(), homeMapTimeout)
+ err := m.Refresh(ctx)
+ cancel()
+ if err != nil {
+ errs++
+ if errs == 1 || errs%10 == 0 {
+ log.Printf("edge: home map refresh failed (#%d): %v — serving the last good map and key", errs, err)
+ }
+ continue
+ }
+ if errs > 0 {
+ log.Printf("edge: home map refresh recovered after %d failure(s)", errs)
+ errs = 0
+ }
+ }
+}
+
+// ---- hop acceptance on the request path -------------------------------------
+
+// hopVerifiedKey marks a request that arrived as a PROVEN hop from another
+// edge in this deployment's mesh. A context value rather than a header,
+// because a header is something a client can send: nothing outside this file
+// can put this mark on a request, which is exactly the property the XFF trust
+// below rests on.
+type hopVerifiedKey struct{}
+
+// hopVerified reports whether this request arrived as a verified hop. Task 4's
+// forwarding decision reads it too — a request that ARRIVED as a hop never
+// consults the home map.
+func hopVerified(ctx context.Context) bool {
+ v, _ := ctx.Value(hopVerifiedKey{}).(bool)
+ return v
+}
+
+// acceptHop is the whole of hop handling on the request path, and it runs
+// before the route lookup so that BOTH of its outcomes are decided ahead of
+// anything that serves:
+//
+// no hop header → nothing happens at all (the overwhelming case: one
+// Header.Get on a request that is not a hop).
+// verified hop → the mark goes on the context, the inbound
+// X-Forwarded-For is believed, and the caller serves
+// the route locally or refuses (staleHopRefusal).
+// unverified hop → every forwarded header is stripped and the sender is
+// an ordinary client from here on.
+//
+// The hop headers themselves are deleted in ALL hop cases, verified included:
+// they are the edge's vocabulary between boxes and have no business reaching a
+// tenant's app, and a stripped header cannot be laundered back out by an app
+// that reflects what it was sent (scrubInboundPurgeHeader, purge_header.go,
+// records what that class of bug already cost this package).
+//
+// host is the caller's already-port-stripped request host; canonicalization
+// into the exact bytes the MAC covers is hopMAC's job, in one place, so a
+// signer and a verifier can never disagree (mesh.go).
+func (e *Edge) acceptHop(r *http.Request, host string) (*http.Request, bool) {
+ if r.Header.Get(hopHeader) == "" {
+ return r, false
+ }
+ auth := r.Header.Get(hopAuthHeader)
+ r.Header.Del(hopHeader)
+ r.Header.Del(hopAuthHeader)
+
+ // verifyHopEitherKey, never verifyHop: the single-key form cannot accept a
+ // signer that has already moved to .next, and a rotation is precisely when
+ // refusing hops would hurt. Keys come from the mirror's cache — a hop must
+ // never cost a bucket read.
+ current, next := e.homes.Load().meshKeys()
+ if auth == "" || !verifyHopEitherKey(current, next, host, auth, time.Now()) {
+ stripForwardedHeaders(r)
+ return r, false
+ }
+ return r.WithContext(context.WithValue(r.Context(), hopVerifiedKey{}, true)), true
+}
+
+// stripForwardedHeaders discards every claim a request makes about who sent
+// it. httputil.ReverseProxy's Rewrite path already deletes these from the
+// OUTBOUND request, so for a proxied route this is belt and braces — but the
+// edge serves plenty that is not a ReverseProxy (static trees, the status
+// host, cached replays, the activator's dispatch), and all of them read this
+// same inbound request.
+func stripForwardedHeaders(r *http.Request) {
+ r.Header.Del("Forwarded")
+ r.Header.Del("X-Forwarded-For")
+ r.Header.Del("X-Forwarded-Host")
+ r.Header.Del("X-Forwarded-Proto")
+}
+
+// staleHopRefusal answers a VERIFIED hop that named a host this box does not
+// serve: the sending edge's home map is stale, mid-move. 503 + Retry-After: 2
+// is the same "come back shortly" answer a dark fleet gets (fleetProxy), and
+// converge heals it within a publish cycle.
+//
+// The body says nothing. It is reachable only by a party holding the mesh key
+// — an unauthenticated stranger asking for the same host gets the ordinary,
+// unchanged unknownHost answer — so it cannot tell an outsider that a host
+// exists somewhere else in the deployment. Keeping it contentless holds that
+// line for the ONE case where it could still leak: a peer whose key is valid
+// but whose map is wrong learns only "not here", never "here is where it
+// went".
+func staleHopRefusal(w http.ResponseWriter) {
+ w.Header().Set("Retry-After", "2")
+ http.Error(w, "not served here", http.StatusServiceUnavailable)
+}
+
+// setForwarded is what every proxy Rewrite in this package calls in place of
+// pr.SetXForwarded(), and it exists for one narrow case.
+//
+// ReverseProxy's Rewrite path deletes Forwarded and X-Forwarded-* from the
+// outbound request before Rewrite runs, so SetXForwarded's documented "append
+// to an existing value" behaviour never fires: an app behind this edge sees
+// exactly one address, the peer that dialled us. That is right for a client
+// and wrong for a verified hop, where the peer that dialled us is a sibling
+// edge and the address the app actually wants is the one that edge recorded.
+//
+// So on a verified hop — and only there — the inbound chain is copied onto the
+// outbound request first, and SetXForwarded then appends the sending edge to
+// it. Multiple inbound X-Forwarded-For lines are joined, which is what
+// list-valued header semantics already say they mean (RFC 9110 §5.3).
+//
+// X-Forwarded-Host and X-Forwarded-Proto need no such restore: the hop carries
+// the original Host header and is itself TLS, so SetXForwarded recomputes both
+// correctly at this edge.
+func setForwarded(pr *httputil.ProxyRequest) {
+ if hopVerified(pr.In.Context()) {
+ if chain := pr.In.Header.Values("X-Forwarded-For"); len(chain) > 0 {
+ pr.Out.Header.Set("X-Forwarded-For", strings.Join(chain, ", "))
+ }
+ }
+ pr.SetXForwarded()
+}
diff --git a/internal/edge/homemap_test.go b/internal/edge/homemap_test.go
new file mode 100644
index 000000000..b490a4522
--- /dev/null
+++ b/internal/edge/homemap_test.go
@@ -0,0 +1,530 @@
+package edge
+
+import (
+ "context"
+ "errors"
+ "fmt"
+ "io"
+ "net/http"
+ "net/http/httptest"
+ "os"
+ "strings"
+ "sync"
+ "testing"
+ "time"
+
+ "github.com/carlosframework/platform/internal/registry"
+ "github.com/carlosframework/platform/internal/release"
+)
+
+// putJSON writes a control-plane object body verbatim, replacing whatever is
+// there. release.Store.Put with an empty ifMatch is create-only, so a test
+// that rewrites a key (the last-good and stale-file cases below) has to carry
+// the etag — the same putReplacing shape certsync.go uses in production.
+func putJSON(t *testing.T, s release.Store, key, body string) {
+ t.Helper()
+ _, etag, err := s.Get(context.Background(), key)
+ switch {
+ case errors.Is(err, os.ErrNotExist):
+ etag = ""
+ case err != nil:
+ t.Fatalf("get %s: %v", key, err)
+ }
+ if _, err := s.Put(context.Background(), key, []byte(body), etag); err != nil {
+ t.Fatalf("put %s: %v", key, err)
+ }
+}
+
+func refreshHomeMap(t *testing.T, m *homeMap) {
+ t.Helper()
+ if err := m.Refresh(context.Background()); err != nil {
+ t.Fatalf("refresh: %v", err)
+ }
+}
+
+func TestHomeMapFoldsAndExcludesOwnBox(t *testing.T) {
+ s := meshTestStore(t)
+ putJSON(t, s, "control/edges/annie.json", `{"v":1,"box":"annie","eip":"1.1.1.1","hosts":{"a.test":{"role":"home"}}}`)
+ putJSON(t, s, "control/edges/betty.json", `{"v":1,"box":"betty","eip":"2.2.2.2","hosts":{"b.test":{"role":"home"}}}`)
+ m := newHomeMap(s, "annie")
+ refreshHomeMap(t, m)
+ if eip, box, ok := m.Home("b.test"); !ok || eip != "2.2.2.2" || box != "betty" {
+ t.Fatalf("b.test home = %s/%s/%v", eip, box, ok)
+ }
+ if _, _, ok := m.Home("a.test"); ok {
+ t.Fatal("own-box hosts must not offer a backhaul target")
+ }
+}
+
+// Task 1's writer publishes registry row hosts VERBATIM, catch-all rows
+// included, so the reader owes lookupUpstream's one-label wildcard rule: an
+// exact match wins, then "*.<parent>" — and only ONE label, so a two-label
+// host misses exactly as it does locally.
+func TestHomeMapAppliesOneLabelWildcardRule(t *testing.T) {
+ s := meshTestStore(t)
+ putJSON(t, s, "control/edges/betty.json",
+ `{"v":1,"box":"betty","eip":"2.2.2.2","hosts":{"*.apps.test":{"role":"home"},"exact.apps.test":{"role":"home"}}}`)
+ m := newHomeMap(s, "annie")
+ refreshHomeMap(t, m)
+
+ // Exact form.
+ if eip, _, ok := m.Home("exact.apps.test"); !ok || eip != "2.2.2.2" {
+ t.Fatalf("exact host = %s/%v, want the home edge", eip, ok)
+ }
+ // Wildcard form: one label under the catch-all parent.
+ if eip, box, ok := m.Home("paul.apps.test"); !ok || eip != "2.2.2.2" || box != "betty" {
+ t.Fatalf("one-label wildcard host = %s/%s/%v", eip, box, ok)
+ }
+ // Two labels: a miss here exactly as in lookupUpstream, which composes
+ // "*.b.apps.test" and finds nothing.
+ if _, _, ok := m.Home("a.b.apps.test"); ok {
+ t.Fatal("a two-label host must not match a one-label wildcard")
+ }
+ // The wildcard key itself is not a request host and must not resolve as
+ // one via the fallback ("*.apps.test" composes "*.apps.test" again).
+ if _, _, ok := m.Home("apps.test"); ok {
+ t.Fatal("the catch-all parent must not match its own wildcard")
+ }
+}
+
+// The role object is read-local anticipation (Task 1). Only "home" routes;
+// anything else is a role this build does not implement, and an unknown role
+// must neither route nor fail the poll.
+func TestHomeMapIgnoresUnknownRoles(t *testing.T) {
+ s := meshTestStore(t)
+ putJSON(t, s, "control/edges/betty.json",
+ `{"v":1,"box":"betty","eip":"2.2.2.2","hosts":{"read.test":{"role":"read"},"home.test":{"role":"home"},"blank.test":{}}}`)
+ m := newHomeMap(s, "annie")
+ refreshHomeMap(t, m) // an unknown role is not an error
+ if _, _, ok := m.Home("read.test"); ok {
+ t.Fatal("role:\"read\" is not a home — it must not offer a backhaul target")
+ }
+ if _, _, ok := m.Home("blank.test"); ok {
+ t.Fatal("an absent role is not a home")
+ }
+ if _, _, ok := m.Home("home.test"); !ok {
+ t.Fatal("role:\"home\" must still route alongside roles this build ignores")
+ }
+}
+
+// A box with no discoverable public IP publishes its hosts list with eip:""
+// (Task 1's DR-runbook case). There is nothing to dial, so it must never be
+// offered as a target — a "" EIP would otherwise become a dial to port 443 of
+// nothing.
+func TestHomeMapSkipsBoxesWithNoEIP(t *testing.T) {
+ s := meshTestStore(t)
+ putJSON(t, s, "control/edges/betty.json", `{"v":1,"box":"betty","eip":"","hosts":{"b.test":{"role":"home"}}}`)
+ m := newHomeMap(s, "annie")
+ refreshHomeMap(t, m)
+ if _, _, ok := m.Home("b.test"); ok {
+ t.Fatal("a box with no EIP has no dial address and must not be a backhaul target")
+ }
+}
+
+// Two boxes claiming the same host is a placement bug, but every edge must
+// still pick the SAME one — a coin flip per box would split traffic across a
+// move instead of converging on it.
+func TestHomeMapPicksTheSameHomeOnADuplicateClaim(t *testing.T) {
+ s := meshTestStore(t)
+ putJSON(t, s, "control/edges/betty.json", `{"v":1,"box":"betty","eip":"2.2.2.2","hosts":{"b.test":{"role":"home"}}}`)
+ putJSON(t, s, "control/edges/carol.json", `{"v":1,"box":"carol","eip":"3.3.3.3","hosts":{"b.test":{"role":"home"}}}`)
+ for i := 0; i < 5; i++ {
+ m := newHomeMap(s, "annie")
+ refreshHomeMap(t, m)
+ if _, box, _ := m.Home("b.test"); box != "betty" {
+ t.Fatalf("duplicate claim resolved to %q, want the deterministic first (betty)", box)
+ }
+ }
+}
+
+// A box that is retired (its file deleted) must LEAVE the map — the fold is a
+// full rebuild from each list, never a merge into the previous one.
+func TestHomeMapForgetsADeletedEdgesFile(t *testing.T) {
+ s := meshTestStore(t)
+ putJSON(t, s, "control/edges/betty.json", `{"v":1,"box":"betty","eip":"2.2.2.2","hosts":{"b.test":{"role":"home"}}}`)
+ m := newHomeMap(s, "annie")
+ refreshHomeMap(t, m)
+ if _, _, ok := m.Home("b.test"); !ok {
+ t.Fatal("precondition: b.test should be homed on betty")
+ }
+ if err := s.Delete(context.Background(), "control/edges/betty.json"); err != nil {
+ t.Fatalf("delete: %v", err)
+ }
+ refreshHomeMap(t, m)
+ if _, _, ok := m.Home("b.test"); ok {
+ t.Fatal("a retired box's hosts must leave the map on the next poll")
+ }
+}
+
+// One unreadable object is not a bad poll: the previous copy of that box's
+// file carries forward and every other box's file still lands, exactly as the
+// cert mirror does it.
+func TestHomeMapCarriesForwardOneUnreadableFile(t *testing.T) {
+ inner := meshTestStore(t)
+ s := &oneKeyFailingStore{Store: inner}
+ putJSON(t, inner, "control/edges/betty.json", `{"v":1,"box":"betty","eip":"2.2.2.2","hosts":{"b.test":{"role":"home"}}}`)
+ m := newHomeMap(s, "annie")
+ refreshHomeMap(t, m)
+
+ // betty's file goes unreadable; carol appears in the same poll.
+ putJSON(t, inner, "control/edges/carol.json", `{"v":1,"box":"carol","eip":"3.3.3.3","hosts":{"c.test":{"role":"home"}}}`)
+ s.failKey("control/edges/betty.json")
+ refreshHomeMap(t, m)
+
+ if eip, _, ok := m.Home("b.test"); !ok || eip != "2.2.2.2" {
+ t.Fatalf("b.test = %s/%v — the previous copy of an unreadable file must carry forward", eip, ok)
+ }
+ if _, _, ok := m.Home("c.test"); !ok {
+ t.Fatal("one bad object must not discard the rest of the rebuild")
+ }
+}
+
+// A corrupt body is the same class of fault as an unreadable one: carry the
+// last good parse forward rather than dropping the box off the map.
+func TestHomeMapCarriesForwardACorruptFile(t *testing.T) {
+ s := meshTestStore(t)
+ putJSON(t, s, "control/edges/betty.json", `{"v":1,"box":"betty","eip":"2.2.2.2","hosts":{"b.test":{"role":"home"}}}`)
+ m := newHomeMap(s, "annie")
+ refreshHomeMap(t, m)
+ putJSON(t, s, "control/edges/betty.json", `{"v":1,"box":"betty",`)
+ refreshHomeMap(t, m)
+ if eip, _, ok := m.Home("b.test"); !ok || eip != "2.2.2.2" {
+ t.Fatalf("b.test = %s/%v — a corrupt file must leave the last good parse standing", eip, ok)
+ }
+}
+
+// A failed List is a bucket fault: last-good stands and the error is returned
+// for the loop to log. An empty map here would silently stop every backhaul
+// in the deployment on one blip.
+func TestHomeMapKeepsLastGoodWhenTheBucketIsDown(t *testing.T) {
+ inner := meshTestStore(t)
+ s := &listFailingStore{Store: inner}
+ putJSON(t, inner, "control/edges/betty.json", `{"v":1,"box":"betty","eip":"2.2.2.2","hosts":{"b.test":{"role":"home"}}}`)
+ m := newHomeMap(s, "annie")
+ refreshHomeMap(t, m)
+
+ s.setFail(true)
+ if err := m.Refresh(context.Background()); err == nil {
+ t.Fatal("a failed List must be reported, not swallowed")
+ }
+ if _, _, ok := m.Home("b.test"); !ok {
+ t.Fatal("last-good must stand through a bucket blip")
+ }
+}
+
+// An edge with no box identity of its own cannot exclude itself, so it must
+// never offer a target at all — fail closed rather than backhaul to itself.
+func TestHomeMapWithoutASelfLabelNeverOffersATarget(t *testing.T) {
+ s := meshTestStore(t)
+ putJSON(t, s, "control/edges/betty.json", `{"v":1,"box":"betty","eip":"2.2.2.2","hosts":{"b.test":{"role":"home"}}}`)
+ m := newHomeMap(s, "")
+ refreshHomeMap(t, m)
+ if _, _, ok := m.Home("b.test"); ok {
+ t.Fatal("an edge that does not know its own box label must not backhaul")
+ }
+}
+
+// A nil map (no deployment store, so no mesh at all) answers like an empty
+// one rather than panicking on the request path.
+func TestNilHomeMapIsInert(t *testing.T) {
+ var m *homeMap
+ if _, _, ok := m.Home("b.test"); ok {
+ t.Fatal("a nil home map must offer nothing")
+ }
+ if cur, next := m.meshKeys(); cur != nil || next != nil {
+ t.Fatal("a nil home map holds no mesh keys")
+ }
+}
+
+// ---- hop acceptance ---------------------------------------------------------
+
+// forwardEcho is an upstream that reports back exactly the trust-bearing
+// headers it was handed, so a test can assert both what survived and what was
+// stripped on the way in.
+func forwardEcho(t *testing.T) *httptest.Server {
+ t.Helper()
+ ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ fmt.Fprintf(w, "xff=%s|xfh=%s|hop=%s|auth=%s",
+ r.Header.Get("X-Forwarded-For"), r.Header.Get("X-Forwarded-Host"),
+ r.Header.Get(hopHeader), r.Header.Get(hopAuthHeader))
+ }))
+ t.Cleanup(ts.Close)
+ return ts
+}
+
+// hopEdge builds an edge serving hosts from one echo backend, with a mesh key
+// minted and a home map armed over the same store — the receiving box of a
+// backhaul hop.
+func hopEdge(t *testing.T, self string, hosts ...string) (*Edge, *homeMap, []byte) {
+ t.Helper()
+ s := meshTestStore(t)
+ if err := mintMeshKey(context.Background(), s); err != nil {
+ t.Fatalf("mint: %v", err)
+ }
+ backend := forwardEcho(t)
+ reg := openReg(t)
+ for _, h := range hosts {
+ if err := reg.Upsert(registry.Route{Host: h, Kind: "service", Addr: backend.Listener.Addr().String(), Status: "ready"}); err != nil {
+ t.Fatalf("upsert %s: %v", h, err)
+ }
+ }
+ e, err := New(reg, nil)
+ if err != nil {
+ t.Fatalf("New: %v", err)
+ }
+ t.Cleanup(e.probers.Stop)
+ m := newHomeMap(s, self)
+ refreshHomeMap(t, m)
+ e.homes.Store(m)
+ key, _ := m.meshKeys()
+ if len(key) == 0 {
+ t.Fatal("precondition: the mesh key must be mirrored")
+ }
+ return e, m, key
+}
+
+// hopRequest drives one request at the edge with the given headers.
+func hopRequest(t *testing.T, e *Edge, host string, headers map[string]string) *http.Response {
+ t.Helper()
+ srv := httptest.NewServer(e)
+ t.Cleanup(srv.Close)
+ req, err := http.NewRequest("GET", srv.URL+"/", nil)
+ if err != nil {
+ t.Fatalf("new request: %v", err)
+ }
+ req.Host = host
+ for k, v := range headers {
+ req.Header.Set(k, v)
+ }
+ resp, err := http.DefaultClient.Do(req)
+ if err != nil {
+ t.Fatalf("do: %v", err)
+ }
+ t.Cleanup(func() { resp.Body.Close() })
+ return resp
+}
+
+func readBody(t *testing.T, resp *http.Response) string {
+ t.Helper()
+ b, err := io.ReadAll(resp.Body)
+ if err != nil {
+ t.Fatalf("read body: %v", err)
+ }
+ return string(b)
+}
+
+// A verified hop is served locally, and the chain the sending edge appended
+// is the one thing an ordinary client is never trusted for.
+func TestVerifiedHopIsServedLocallyAndHonoursTheInboundXFF(t *testing.T) {
+ e, _, key := hopEdge(t, "annie", "app.test")
+ resp := hopRequest(t, e, "app.test", map[string]string{
+ hopHeader: "1",
+ hopAuthHeader: signHop(key, "app.test", time.Now()),
+ "X-Forwarded-For": "203.0.113.9",
+ })
+ if resp.StatusCode != http.StatusOK {
+ t.Fatalf("status = %d, want 200", resp.StatusCode)
+ }
+ body := readBody(t, resp)
+ if !strings.HasPrefix(body, "xff=203.0.113.9, ") {
+ t.Fatalf("body = %q — a verified hop's inbound XFF must be kept and the sending edge appended to it", body)
+ }
+ // The hop's own trust headers are the edge's vocabulary, never the app's.
+ if !strings.Contains(body, "|hop=|auth=") {
+ t.Fatalf("body = %q — the hop headers must not reach the app", body)
+ }
+}
+
+// The MAC binds the CONCRETE request host, and a catch-all route serves it
+// through lookupUpstream's wildcard fallback — the two halves of carry C1 on
+// the receiving side.
+func TestVerifiedHopServesAWildcardRoutedHost(t *testing.T) {
+ e, _, key := hopEdge(t, "annie", "*.apps.test")
+ resp := hopRequest(t, e, "paul.apps.test", map[string]string{
+ hopHeader: "1",
+ hopAuthHeader: signHop(key, "paul.apps.test", time.Now()),
+ })
+ if resp.StatusCode != http.StatusOK {
+ t.Fatalf("status = %d, want 200 — a catch-all route serves a hop like any other client", resp.StatusCode)
+ }
+}
+
+// The stale-map refusal: the sender's map named us for a host we do not
+// serve. 503 + Retry-After: 2, and structurally never a re-forward — the home
+// map here is POISONED to name another box for exactly that host, so anything
+// that consulted it would leave through a dial instead.
+func TestVerifiedHopForAnUnservedHostRefusesWithoutForwarding(t *testing.T) {
+ e, m, key := hopEdge(t, "annie", "app.test")
+ putJSON(t, m.s, "control/edges/betty.json",
+ `{"v":1,"box":"betty","eip":"2.2.2.2","hosts":{"moved.test":{"role":"home"}}}`)
+ refreshHomeMap(t, m)
+ if _, _, ok := m.Home("moved.test"); !ok {
+ t.Fatal("precondition: the map must name a remote home for moved.test")
+ }
+
+ resp := hopRequest(t, e, "moved.test", map[string]string{
+ hopHeader: "1",
+ hopAuthHeader: signHop(key, "moved.test", time.Now()),
+ })
+ if resp.StatusCode != http.StatusServiceUnavailable {
+ t.Fatalf("status = %d, want 503 for a verified hop naming a host we do not serve", resp.StatusCode)
+ }
+ if got := resp.Header.Get("Retry-After"); got != "2" {
+ t.Fatalf("Retry-After = %q, want 2", got)
+ }
+ if body := readBody(t, resp); strings.Contains(body, "moved.test") || strings.Contains(body, "betty") {
+ t.Fatalf("body = %q — the refusal must not name the host or where it went", body)
+ }
+}
+
+// Verification failure is not an error path: it is "you are an ordinary
+// client", with every forwarding claim you made discarded.
+func TestUnverifiedHopIsServedAsAnOrdinaryClientWithForwardedHeadersStripped(t *testing.T) {
+ e, _, key := hopEdge(t, "annie", "app.test")
+ tampered := []byte(signHop(key, "app.test", time.Now()))
+ tampered[len(tampered)-1] ^= 'f' ^ '0' // flip the last hex nibble
+
+ resp := hopRequest(t, e, "app.test", map[string]string{
+ hopHeader: "1",
+ hopAuthHeader: string(tampered),
+ "X-Forwarded-For": "203.0.113.9",
+ "X-Forwarded-Host": "evil.test",
+ "X-Forwarded-Proto": "https",
+ })
+ if resp.StatusCode != http.StatusOK {
+ t.Fatalf("status = %d, want 200 — an unverified hop is an ordinary client, not a rejection", resp.StatusCode)
+ }
+ body := readBody(t, resp)
+ if strings.Contains(body, "203.0.113.9") {
+ t.Fatalf("body = %q — an unverified sender's X-Forwarded-For must not be honoured", body)
+ }
+ if strings.Contains(body, "evil.test") {
+ t.Fatalf("body = %q — an unverified sender's X-Forwarded-Host must not be honoured", body)
+ }
+ if !strings.Contains(body, "|hop=|auth=") {
+ t.Fatalf("body = %q — the hop headers must be stripped either way", body)
+ }
+}
+
+// The strip is at the header level, not just a proxy side effect: every route
+// class behind the edge (static trees, the status host, the activator) reads
+// the same request.
+func TestAcceptHopStripsEveryForwardedHeaderOnFailedVerification(t *testing.T) {
+ e, _, _ := hopEdge(t, "annie", "app.test")
+ r := httptest.NewRequest("GET", "http://app.test/", nil)
+ r.Header.Set(hopHeader, "1")
+ r.Header.Set(hopAuthHeader, "0:deadbeef")
+ r.Header.Set("Forwarded", "for=203.0.113.9")
+ r.Header.Set("X-Forwarded-For", "203.0.113.9")
+ r.Header.Set("X-Forwarded-Host", "evil.test")
+ r.Header.Set("X-Forwarded-Proto", "https")
+
+ r, verified := e.acceptHop(r, "app.test")
+ if verified {
+ t.Fatal("a garbage auth header must not verify")
+ }
+ for _, h := range []string{"Forwarded", "X-Forwarded-For", "X-Forwarded-Host", "X-Forwarded-Proto", hopHeader, hopAuthHeader} {
+ if got := r.Header.Get(h); got != "" {
+ t.Fatalf("%s survived an unverified hop as %q", h, got)
+ }
+ }
+ if hopVerified(r.Context()) {
+ t.Fatal("an unverified request must not carry the hop mark")
+ }
+}
+
+// The MAC is over the host, so a signature minted for one hop cannot be
+// replayed against another — and the timestamp bounds it in time.
+func TestHopVerificationIsBoundToTheHostAndTheClock(t *testing.T) {
+ e, _, key := hopEdge(t, "annie", "app.test", "other.test")
+
+ wrongHost := hopRequest(t, e, "app.test", map[string]string{
+ hopHeader: "1",
+ hopAuthHeader: signHop(key, "other.test", time.Now()),
+ "X-Forwarded-For": "203.0.113.9",
+ })
+ if body := readBody(t, wrongHost); strings.Contains(body, "203.0.113.9") {
+ t.Fatalf("body = %q — a signature minted for another host must not verify", body)
+ }
+
+ stale := hopRequest(t, e, "app.test", map[string]string{
+ hopHeader: "1",
+ hopAuthHeader: signHop(key, "app.test", time.Now().Add(-2*hopSkew)),
+ "X-Forwarded-For": "203.0.113.9",
+ })
+ if body := readBody(t, stale); strings.Contains(body, "203.0.113.9") {
+ t.Fatalf("body = %q — a signature outside the skew bound must not verify", body)
+ }
+}
+
+// Fail closed: with no mesh key mirrored (a brand new deployment, or a box
+// whose bucket is unreachable at boot) nothing verifies.
+func TestHopWithoutAMirroredMeshKeyIsAnOrdinaryClient(t *testing.T) {
+ e, _, key := hopEdge(t, "annie", "app.test")
+ e.homes.Store(nil) // no mesh at all — the no-deployment-store case
+ resp := hopRequest(t, e, "app.test", map[string]string{
+ hopHeader: "1",
+ hopAuthHeader: signHop(key, "app.test", time.Now()),
+ "X-Forwarded-For": "203.0.113.9",
+ })
+ if resp.StatusCode != http.StatusOK {
+ t.Fatalf("status = %d, want 200", resp.StatusCode)
+ }
+ if body := readBody(t, resp); strings.Contains(body, "203.0.113.9") {
+ t.Fatalf("body = %q — with no key to verify against, the sender is an ordinary client", body)
+ }
+}
+
+// An unserved host with NO hop header keeps today's answer exactly: the
+// stale-map 503 is reachable only by a party holding the mesh key, so it
+// never tells a stranger whether a host exists elsewhere in the deployment.
+func TestUnknownHostWithoutAHopIsStillThe404(t *testing.T) {
+ e, m, _ := hopEdge(t, "annie", "app.test")
+ putJSON(t, m.s, "control/edges/betty.json",
+ `{"v":1,"box":"betty","eip":"2.2.2.2","hosts":{"moved.test":{"role":"home"}}}`)
+ refreshHomeMap(t, m)
+ resp := hopRequest(t, e, "moved.test", nil)
+ if resp.StatusCode != http.StatusNotFound {
+ t.Fatalf("status = %d, want the unchanged 404 for an unauthenticated stranger", resp.StatusCode)
+ }
+ if got := resp.Header.Get("Retry-After"); got != "" {
+ t.Fatalf("Retry-After = %q — a stranger must not be told to come back", got)
+ }
+}
+
+// Regression pin: an ordinary client's forged X-Forwarded-For is no more
+// trusted after this change than before it.
+func TestOrdinaryClientForgedXFFIsStillDiscarded(t *testing.T) {
+ e, _, _ := hopEdge(t, "annie", "app.test")
+ resp := hopRequest(t, e, "app.test", map[string]string{"X-Forwarded-For": "203.0.113.9"})
+ if resp.StatusCode != http.StatusOK {
+ t.Fatalf("status = %d, want 200", resp.StatusCode)
+ }
+ if body := readBody(t, resp); strings.Contains(body, "203.0.113.9") {
+ t.Fatalf("body = %q — a client-supplied XFF is never honoured", body)
+ }
+}
+
+// oneKeyFailingStore fails Get for one named key — a single unreadable object
+// mid-poll, the case the carry-forward exists for.
+type oneKeyFailingStore struct {
+ release.Store
+ mu sync.Mutex
+ bad string
+}
+
+func (o *oneKeyFailingStore) failKey(key string) {
+ o.mu.Lock()
+ o.bad = key
+ o.mu.Unlock()
+}
+
+func (o *oneKeyFailingStore) Get(ctx context.Context, key string) ([]byte, string, error) {
+ o.mu.Lock()
+ bad := o.bad
+ o.mu.Unlock()
+ if key == bad {
+ return nil, "", errors.New("bucket down for this object")
+ }
+ return o.Store.Get(ctx, key)
+}
diff --git a/internal/edge/mesh_test.go b/internal/edge/mesh_test.go
index 59be378da..e873dedbf 100644
--- a/internal/edge/mesh_test.go
+++ b/internal/edge/mesh_test.go
@@ -308,3 +308,82 @@ func TestLoadMeshKeysToleratesAnUnreadableNext(t *testing.T) {
t.Fatal("hop auth must keep working off the current key despite the malformed .next")
}
}
+
+// ---- the key half of the home-map mirror ------------------------------------
+//
+// Task 3 folds the mesh-key read into the home map's 30s poll rather than
+// giving it a loop of its own: one list-and-get pass keeps both halves of a
+// hop's answer (who homes this host, and what key proves the hop) from ever
+// being read from two different moments in the bucket's life.
+
+func TestHomeMapRefreshMirrorsTheMeshKeys(t *testing.T) {
+ ctx := context.Background()
+ s := meshTestStore(t)
+ m := newHomeMap(s, "annie")
+
+ // Before the housekeeper's first tick: no key, and every hop fails
+ // closed — refreshing is still not an error.
+ if err := m.Refresh(ctx); err != nil {
+ t.Fatalf("refresh before minting: %v", err)
+ }
+ if cur, next := m.meshKeys(); cur != nil || next != nil {
+ t.Fatalf("keys before minting = (%x, %x), want (nil, nil)", cur, next)
+ }
+
+ if err := mintMeshKey(ctx, s); err != nil {
+ t.Fatal(err)
+ }
+ if err := m.Refresh(ctx); err != nil {
+ t.Fatalf("refresh after minting: %v", err)
+ }
+ cur, next := m.meshKeys()
+ if len(cur) != meshKeyBytes || next != nil {
+ t.Fatalf("keys after minting = (%d bytes, %v), want the current key alone", len(cur), next)
+ }
+
+ // A rotation in flight: dual-accept means a signature under EITHER key is
+ // good, which is what the request path calls (verifyHopEitherKey, never
+ // the single-key verifyHop).
+ nextBody := []byte(hex.EncodeToString(bytes.Repeat([]byte{0xcd}, meshKeyBytes)))
+ if _, err := s.Put(ctx, meshKeyNextKey, nextBody, ""); err != nil {
+ t.Fatal(err)
+ }
+ if err := m.Refresh(ctx); err != nil {
+ t.Fatalf("refresh during rotation: %v", err)
+ }
+ cur, next = m.meshKeys()
+ if len(cur) != meshKeyBytes || len(next) != meshKeyBytes {
+ t.Fatalf("keys during rotation = (%d, %d), want both mirrored", len(cur), len(next))
+ }
+ now := time.Now()
+ for _, k := range [][]byte{cur, next} {
+ if !verifyHopEitherKey(cur, next, "app.example.com", signHop(k, "app.example.com", now), now) {
+ t.Fatal("a hop signed under either mirrored key must verify while both are mirrored")
+ }
+ }
+}
+
+// The keys ride the same last-good posture as the host fold: a bucket blip
+// must not disarm hop verification fleet-wide.
+func TestHomeMapKeepsLastGoodMeshKeysWhenTheBucketIsDown(t *testing.T) {
+ ctx := context.Background()
+ inner := meshTestStore(t)
+ s := &listFailingStore{Store: inner}
+ if err := mintMeshKey(ctx, inner); err != nil {
+ t.Fatal(err)
+ }
+ m := newHomeMap(s, "annie")
+ if err := m.Refresh(ctx); err != nil {
+ t.Fatalf("refresh: %v", err)
+ }
+ good, _ := m.meshKeys()
+
+ s.setFail(true)
+ if err := m.Refresh(ctx); err == nil {
+ t.Fatal("a bucket blip must be reported, not swallowed")
+ }
+ cur, _ := m.meshKeys()
+ if !bytes.Equal(cur, good) {
+ t.Fatal("the last good mesh key must stand through a bucket blip")
+ }
+}
diff --git a/internal/host/agent.go b/internal/host/agent.go
index 4438131c2..8a609ae33 100644
--- a/internal/host/agent.go
+++ b/internal/host/agent.go
@@ -361,7 +361,14 @@ func RunAgent(args []string) {
// for seconds to minutes, and the context ends the instant this box
// stops being the housekeeper. Armed above, so this closure is live
// and correct by the time the edge reads it.
- edge.WithIssuerGate(housekeeperGate))
+ edge.WithIssuerGate(housekeeperGate),
+ // This box's own label, for the home map's own-box exclusion
+ // (internal/edge/homemap.go). metricsBoxLabel is the SAME call
+ // edgesFileLoop publishes under, deliberately: the map excludes our
+ // hosts by matching the box field of the file we ourselves wrote, so
+ // the two derivations must be one derivation. Without it the edge
+ // would offer nothing (fail-closed), never dial itself.
+ edge.WithBoxLabel(metricsBoxLabel(os.Getenv, readMachineID())))
// The edge has drained; take the children down too (they're this process's
// children — orphaning them would strand their leases until TTL expiry).