amadan

carlos / platform Public

edge+host: mesh key — housekeeper mints create-only, HMAC hop vocabulary

mintMeshKey (internal/edge/mesh.go) writes 32 random bytes hex-encoded to
control/edge-mesh-key with an empty ifMatch — release.Store's create-only
form is the CAS itself; a lost race (release.ErrCASConflict) is
success-by-other, not a failure. Wired as a second housekeeper duty beside
the releases retention prune in internal/host/agent.go's retentionLoop,
sharing its store and its 5min+jitter tick.

signHop/verifyHop are the HMAC-SHA256 hop vocabulary Task 3/4 wire onto the
request path (X-Carlos-Hop / X-Carlos-Hop-Auth): host and a unix timestamp
are bound into the MAC, verified with hmac.Equal (constant-time) and a
±30s skew bound.

Rotation's read side ships now per the plan's global constraint:
control/edge-mesh-key.next mirrors a rotation successor; loadMeshKeys reads
both, verifyHopEitherKey accepts a signature under either, and signingKey
prefers .next the instant it exists. Promotion (copy .next over the current
key, delete .next) stays a runbook step.

storedoor_test.go's TestDoorNeverServesCertMaterial gains
control/edge-mesh-key.next in its key list, seeded the same as the current
key — the door's allowlist already refuses anything not explicitly listed,
so no storedoor.go change was needed.

Single-box inertness: on a one-box deployment this box is always the
housekeeper (housekeeperSole), so it mints its own key and nothing reads it
until Tasks 3/4.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Paul Campbell pushed by paul@keymail.dev fc2bad7774c624bd8df5542b463fbbd16318a463 parent e964df3
4 files changed, +443 −6
  • internal/edge/mesh.go +230 −0
  • internal/edge/mesh_test.go +191 −0
  • internal/edge/storedoor_test.go +12 −5
  • internal/host/agent.go +10 −1
diff --git a/internal/edge/mesh.go b/internal/edge/mesh.go
new file mode 100644
index 000000000..9f056ea5d
--- /dev/null
+++ b/internal/edge/mesh.go
@@ -0,0 +1,230 @@
+package edge
+
+// The mesh key — housekeeper-minted, edge-mirrored, HMAC hop vocabulary
+// (spec docs/superpowers/specs/2026-08-22-multi-region-backhaul-design.md,
+// Part 3, Task 2).
+//
+// A multi-region deployment has more than one edge; Tasks 3/4 have one edge
+// forward a request to another (a "hop") when it does not own the region a
+// request landed in. That hop has to prove it came from a box inside this
+// deployment's own mesh — the writer map's answer is the same shape the
+// housekeeper's other duties already use: exactly ONE box mints a shared
+// secret, every box mirrors it, and no second election is invented.
+//
+// - mintMeshKey is the housekeeper duty (wired beside the releases
+// retention prune, internal/host/agent.go): 32 random bytes,
+// hex-encoded, written to control/edge-mesh-key with an empty ifMatch —
+// release.Store's create-only form. That IS the compare-and-swap: two
+// housekeepers racing the same tick (the transient double-holder window
+// housekeeper.go's poll cadence allows) both attempt the Put, exactly
+// one wins, and the loser's release.ErrCASConflict is not a failure —
+// it is proof a key already exists, which is the only thing this func
+// promises. Every later tick calls it again and finds the same
+// conflict, forever, for the deployment's life: minting is a one-time
+// event, not a duty that ever "finishes".
+//
+// - signHop / verifyHop are the hop vocabulary Task 3/4 wire onto the
+// request path under the headers X-Carlos-Hop: 1 and X-Carlos-Hop-Auth:
+// <unix>:<hex>. The MAC binds the target host into the signature so one
+// hop's header cannot be replayed against another, and the timestamp
+// bounds replay in time (±hopSkew) independent of the host check.
+//
+// - Rotation's READ side ships in this task, ahead of any code that
+// writes .next: control/edge-mesh-key.next mirrors a rotation
+// successor while one is in flight, loadMeshKeys reads both,
+// verifyHopEitherKey accepts a signature under either, and signingKey
+// is what a signer consults to prefer .next the instant it exists —
+// never waiting for a runbook step. That is what makes the promotion
+// itself (copy .next over control/edge-mesh-key, delete .next) safe as
+// a plain runbook action: every verifier already accepts both keys
+// before promotion happens, so there is no window where a box that
+// picked up .next early gets refused by a box that hasn't promoted yet.
+//
+// Single-box inertness: on a deployment with one platform box, this box is
+// always the housekeeper (housekeeper.go's housekeeperSole), so it mints its
+// own mesh key and nothing else ever reads it — Tasks 3/4 are the first
+// readers. Minting here is therefore inert today, by design: this task ships
+// the vocabulary and the mint, not a consumer.
+import (
+ "context"
+ "crypto/hmac"
+ "crypto/rand"
+ "crypto/sha256"
+ "encoding/hex"
+ "errors"
+ "fmt"
+ "log"
+ "os"
+ "strconv"
+ "strings"
+ "time"
+
+ "github.com/carlosframework/platform/internal/release"
+)
+
+// meshKeyKey is the control-plane object the housekeeper mints once per
+// deployment. Named beside the storedoor ACL tripwire
+// (TestDoorNeverServesCertMaterial), which refuses to serve it — a fleet
+// box authenticating through the store door must never read this key.
+const meshKeyKey = "control/edge-mesh-key"
+
+// meshKeyNextKey is the rotation successor, mirrored alongside meshKeyKey
+// only while a rotation is in flight. Promoting it (copy over meshKeyKey,
+// delete this object) is a runbook step, not code this task ships — see the
+// file header.
+const meshKeyNextKey = meshKeyKey + ".next"
+
+// meshKeyBytes is the random key length before hex encoding: 256 bits,
+// matched to the HMAC-SHA256 hop signature below.
+const meshKeyBytes = 32
+
+// hopSkew bounds how far a hop signature's timestamp may drift from the
+// verifier's clock, in either direction: generous enough for ordinary box
+// clock skew and request latency, tight enough that a captured header stops
+// working well within a minute.
+const hopSkew = 30 * time.Second
+
+// mintMeshKey mints the deployment's mesh key if, and only if, nothing has
+// minted one yet. crypto/rand is required — this key authenticates hops
+// between platform boxes, so it gets the same random source every other
+// credential in this codebase does, never math/rand.
+func mintMeshKey(ctx context.Context, s release.Store) error {
+ buf := make([]byte, meshKeyBytes)
+ if _, err := rand.Read(buf); err != nil {
+ return fmt.Errorf("mesh key: read random bytes: %w", err)
+ }
+ body := []byte(hex.EncodeToString(buf))
+ if _, err := s.Put(ctx, meshKeyKey, body, ""); err != nil {
+ if errors.Is(err, release.ErrCASConflict) {
+ // Success-by-other: a key already exists (this box's earlier
+ // tick, or another housekeeper that won the same race). Debug,
+ // not error — this is the steady state for every tick after the
+ // first, forever.
+ log.Printf("edge: mesh key: already minted")
+ return nil
+ }
+ return fmt.Errorf("mesh key: put: %w", err)
+ }
+ log.Printf("edge: mesh key: minted")
+ return nil
+}
+
+// MintMeshKey is mintMeshKey's exported wrapper — the seam
+// internal/host's housekeeper duty calls through, so the duty lives beside
+// the releases-retention prune in internal/host/agent.go while the key's own
+// shape (its object name, its random-byte discipline) stays edge's to own.
+func MintMeshKey(ctx context.Context, s release.Store) error {
+ return mintMeshKey(ctx, s)
+}
+
+// getMeshKeyHex reads and hex-decodes one mirrored mesh-key object, folding
+// "does not exist" (os.ErrNotExist) into (nil, nil) — the ordinary case for
+// meshKeyNextKey outside a rotation window, and for meshKeyKey itself before
+// the housekeeper's first tick on a brand new deployment.
+func getMeshKeyHex(ctx context.Context, s release.Store, key string) ([]byte, error) {
+ body, _, err := s.Get(ctx, key)
+ if errors.Is(err, os.ErrNotExist) {
+ return nil, nil
+ }
+ if err != nil {
+ return nil, fmt.Errorf("mesh key: get %s: %w", key, err)
+ }
+ dec, err := hex.DecodeString(string(body))
+ if err != nil {
+ return nil, fmt.Errorf("mesh key: %s: not valid hex: %w", key, err)
+ }
+ return dec, nil
+}
+
+// loadMeshKeys is the dual-accept read side's key material access: it reads
+// BOTH control/edge-mesh-key and control/edge-mesh-key.next. current is nil
+// only before the housekeeper has minted anything; next is nil the whole
+// rest of the time — nearly always, since a rotation is a deliberate,
+// short-lived operator action.
+func loadMeshKeys(ctx context.Context, s release.Store) (current, next []byte, err error) {
+ current, err = getMeshKeyHex(ctx, s, meshKeyKey)
+ if err != nil {
+ return nil, nil, err
+ }
+ next, err = getMeshKeyHex(ctx, s, meshKeyNextKey)
+ if err != nil {
+ return nil, nil, err
+ }
+ return current, next, nil
+}
+
+// signingKey is which of the two mirrored keys a signer should use: .next
+// the instant it exists, else current. A signer never waits for the
+// runbook's promotion step — verifyHopEitherKey's dual-accept on the reader
+// side is what makes that safe, so switching signers over early costs
+// nothing and shortens how long a rotation needs both keys mirrored.
+func signingKey(current, next []byte) []byte {
+ if len(next) > 0 {
+ return next
+ }
+ return current
+}
+
+// signHop produces the value of the X-Carlos-Hop-Auth header: an
+// HMAC-SHA256 over "host\nunix-timestamp", hex-encoded and prefixed with the
+// timestamp so a verifier does not have to guess it. host is bound into the
+// MAC so a signature minted for one hop cannot be replayed against another.
+func signHop(key []byte, host string, now time.Time) string {
+ ts := strconv.FormatInt(now.Unix(), 10)
+ return ts + ":" + hex.EncodeToString(hopMAC(key, host, ts))
+}
+
+// verifyHop checks an X-Carlos-Hop-Auth header against a single key: the
+// timestamp must parse and fall within ±hopSkew of now, and the MAC must
+// match under constant-time comparison (hmac.Equal) — a hop signature is
+// exactly the kind of secret-derived value a timing side channel could leak,
+// so no byte-wise or string-equality shortcut is taken anywhere in this
+// path.
+func verifyHop(key []byte, host, header string, now time.Time) bool {
+ ts, sig, ok := strings.Cut(header, ":")
+ if !ok || ts == "" || sig == "" {
+ return false
+ }
+ sec, err := strconv.ParseInt(ts, 10, 64)
+ if err != nil {
+ return false
+ }
+ skew := now.Sub(time.Unix(sec, 0))
+ if skew < 0 {
+ skew = -skew
+ }
+ if skew > hopSkew {
+ return false
+ }
+ got, err := hex.DecodeString(sig)
+ if err != nil {
+ return false
+ }
+ return hmac.Equal(got, hopMAC(key, host, ts))
+}
+
+// verifyHopEitherKey is the dual-accept read side of a rotation: it accepts
+// a signature made under either mirrored key, current or next, so neither a
+// straggling signer still on the old key nor one that has already switched
+// to .next (signingKey prefers it immediately) is ever refused while both
+// are mirrored. Task 3/4's request path calls this once it has loaded both
+// keys via loadMeshKeys.
+func verifyHopEitherKey(current, next []byte, host, header string, now time.Time) bool {
+ if len(next) > 0 && verifyHop(next, host, header, now) {
+ return true
+ }
+ if len(current) > 0 && verifyHop(current, host, header, now) {
+ return true
+ }
+ return false
+}
+
+// hopMAC is the MAC signHop and verifyHop both compute — one place so the
+// two can never drift out of sync on what bytes are actually signed.
+func hopMAC(key []byte, host, ts string) []byte {
+ mac := hmac.New(sha256.New, key)
+ mac.Write([]byte(host))
+ mac.Write([]byte("\n"))
+ mac.Write([]byte(ts))
+ return mac.Sum(nil)
+}
diff --git a/internal/edge/mesh_test.go b/internal/edge/mesh_test.go
new file mode 100644
index 000000000..ebadd551d
--- /dev/null
+++ b/internal/edge/mesh_test.go
@@ -0,0 +1,191 @@
+package edge
+
+import (
+ "bytes"
+ "context"
+ "encoding/hex"
+ "testing"
+ "time"
+
+ "github.com/carlosframework/platform/internal/release"
+)
+
+// meshTestStore is a fresh directory-backed store, the same DirStore every
+// other edge test in this package uses in place of live S3.
+func meshTestStore(t *testing.T) release.Store {
+ t.Helper()
+ return release.NewDirStore(t.TempDir())
+}
+
+func TestMintMeshKeyIsCreateOnly(t *testing.T) {
+ s := meshTestStore(t)
+ if err := mintMeshKey(context.Background(), s); err != nil {
+ t.Fatal(err)
+ }
+ first, _, _ := s.Get(context.Background(), meshKeyKey)
+ if err := mintMeshKey(context.Background(), s); err != nil {
+ t.Fatalf("second mint must be success-by-other: %v", err)
+ }
+ second, _, _ := s.Get(context.Background(), meshKeyKey)
+ if !bytes.Equal(first, second) {
+ t.Fatal("second mint replaced the key")
+ }
+}
+
+// The exported wrapper is the seam internal/host's housekeeper duty calls
+// through — prove it actually reaches the same key, not a copy of the logic.
+func TestMintMeshKeyExportedWrapperMintsTheSameKey(t *testing.T) {
+ s := meshTestStore(t)
+ if err := MintMeshKey(context.Background(), s); err != nil {
+ t.Fatal(err)
+ }
+ body, _, err := s.Get(context.Background(), meshKeyKey)
+ if err != nil {
+ t.Fatalf("get %s: %v", meshKeyKey, err)
+ }
+ if len(body) != meshKeyBytes*2 { // hex-encoded
+ t.Fatalf("mesh key body length = %d, want %d (hex of %d random bytes)", len(body), meshKeyBytes*2, meshKeyBytes)
+ }
+}
+
+func TestHopSignatureRoundTripsAndBoundsSkew(t *testing.T) {
+ key := []byte("0123456789abcdef0123456789abcdef")
+ now := time.Unix(1_766_000_000, 0)
+ h := signHop(key, "app.example.com", now)
+ if !verifyHop(key, "app.example.com", h, now.Add(29*time.Second)) {
+ t.Fatal("within skew must verify")
+ }
+ if verifyHop(key, "app.example.com", h, now.Add(31*time.Second)) {
+ t.Fatal("past skew must refuse")
+ }
+ if verifyHop(key, "other.example.com", h, now) {
+ t.Fatal("host is bound into the mac")
+ }
+ if verifyHop([]byte("wrong-key-wrong-key-wrong-key-00"), "app.example.com", h, now) {
+ t.Fatal("wrong key must refuse")
+ }
+}
+
+// Skew is bounded in both directions — a clock that runs fast must be
+// refused exactly like one that runs slow.
+func TestHopSignatureRefusesFutureSkewToo(t *testing.T) {
+ key := []byte("0123456789abcdef0123456789abcdef")
+ now := time.Unix(1_766_000_000, 0)
+ h := signHop(key, "app.example.com", now)
+ if verifyHop(key, "app.example.com", h, now.Add(-31*time.Second)) {
+ t.Fatal("verifying 31s before the signature's timestamp must refuse")
+ }
+ if !verifyHop(key, "app.example.com", h, now.Add(-29*time.Second)) {
+ t.Fatal("verifying 29s before the signature's timestamp must still verify")
+ }
+}
+
+// A malformed header (no ":", a non-numeric timestamp, non-hex mac) must
+// refuse, never panic — it arrives off the wire from a hop this box does not
+// control.
+func TestVerifyHopRefusesMalformedHeaders(t *testing.T) {
+ key := []byte("0123456789abcdef0123456789abcdef")
+ now := time.Unix(1_766_000_000, 0)
+ for _, header := range []string{
+ "",
+ "nocolon",
+ "notanumber:deadbeef",
+ "1766000000:not-hex-zzzz",
+ "1766000000:",
+ } {
+ if verifyHop(key, "app.example.com", header, now) {
+ t.Fatalf("verifyHop(%q) = true, want false", header)
+ }
+ }
+}
+
+// ---- rotation dual-accept (plan's global constraint: the READ side ships
+// now — verifyHopEitherKey and signingKey are what Task 3/4's request path
+// calls once it has loaded both mirrored keys) --------------------------------
+
+func TestSignHopPrefersNextKeyWhenPresent(t *testing.T) {
+ current := []byte("current-key-current-key-current")
+ next := []byte("next-key-next-key-next-key-next")
+ if got := signingKey(current, next); !bytes.Equal(got, next) {
+ t.Fatal("signingKey must prefer .next when it exists")
+ }
+ if got := signingKey(current, nil); !bytes.Equal(got, current) {
+ t.Fatal("signingKey must fall back to current when there is no .next")
+ }
+}
+
+func TestVerifyHopEitherKeyAcceptsSignaturesUnderCurrentOrNext(t *testing.T) {
+ current := []byte("current-key-current-key-current")
+ next := []byte("next-key-next-key-next-key-next-")
+ wrong := []byte("wrong-key-wrong-key-wrong-key-00")
+ now := time.Unix(1_766_000_000, 0)
+ host := "app.example.com"
+
+ // sign with .next -> verifies
+ hNext := signHop(signingKey(current, next), host, now)
+ if !verifyHopEitherKey(current, next, host, hNext, now) {
+ t.Fatal("a signature made with .next must verify while .next exists")
+ }
+
+ // sign with old key while .next exists -> still verifies (dual-accept)
+ hOld := signHop(current, host, now)
+ if !verifyHopEitherKey(current, next, host, hOld, now) {
+ t.Fatal("a signature made with the pre-rotation key must still verify while .next exists")
+ }
+
+ // wrong key -> refuses
+ hWrong := signHop(wrong, host, now)
+ if verifyHopEitherKey(current, next, host, hWrong, now) {
+ t.Fatal("a signature made with neither key must refuse")
+ }
+
+ // no rotation in flight (next is nil): only current verifies.
+ if !verifyHopEitherKey(current, nil, host, hOld, now) {
+ t.Fatal("current key must verify with no rotation in flight")
+ }
+ if verifyHopEitherKey(current, nil, host, hNext, now) {
+ t.Fatal("a .next signature must not verify once .next is gone")
+ }
+}
+
+func TestLoadMeshKeysReadsBothMirroredObjects(t *testing.T) {
+ ctx := context.Background()
+ s := meshTestStore(t)
+
+ // Before the housekeeper's first tick: neither object exists, and that
+ // is quiet — not an error.
+ current, next, err := loadMeshKeys(ctx, s)
+ if err != nil {
+ t.Fatalf("loadMeshKeys before minting: %v", err)
+ }
+ if current != nil || next != nil {
+ t.Fatalf("loadMeshKeys before minting = (%x, %x), want (nil, nil)", current, next)
+ }
+
+ if err := mintMeshKey(ctx, s); err != nil {
+ t.Fatal(err)
+ }
+ current, next, err = loadMeshKeys(ctx, s)
+ if err != nil {
+ t.Fatalf("loadMeshKeys after minting: %v", err)
+ }
+ if len(current) != meshKeyBytes {
+ t.Fatalf("current key length = %d, want %d raw bytes", len(current), meshKeyBytes)
+ }
+ if next != nil {
+ t.Fatal("next key must be nil outside a rotation window")
+ }
+
+ // A rotation in flight: .next mirrored alongside the current key.
+ nextBody := []byte(hex.EncodeToString(bytes.Repeat([]byte{0xab}, meshKeyBytes)))
+ if _, err := s.Put(ctx, meshKeyNextKey, nextBody, ""); err != nil {
+ t.Fatal(err)
+ }
+ current, next, err = loadMeshKeys(ctx, s)
+ if err != nil {
+ t.Fatalf("loadMeshKeys during rotation: %v", err)
+ }
+ if len(current) != meshKeyBytes || len(next) != meshKeyBytes {
+ t.Fatalf("key lengths = (%d, %d), want both %d", len(current), len(next), meshKeyBytes)
+ }
+}
diff --git a/internal/edge/storedoor_test.go b/internal/edge/storedoor_test.go
index 0d3f8d50c..c85458132 100644
--- a/internal/edge/storedoor_test.go
+++ b/internal/edge/storedoor_test.go
@@ -536,10 +536,13 @@ func TestDoorNeverServesCertMaterial(t *testing.T) {
// The real Task 4/5 key shapes (certsync.go's certKey naming: bare host,
// the autocert account key, an HTTP-01 token response, a tls-alpn-01
- // token cert, a wildcard account key) plus the two other control/
+ // token cert, a wildcard account key) plus the three other control/
// entries the multi-region design reserves for the housekeeper alone
- // (bucket-writer-audit.md, "Not yet built" table): the mesh pairing key
- // and the housekeeper's own election lock.
+ // (bucket-writer-audit.md, "Not yet built" table): the mesh key, its
+ // rotation successor (Task 2's dual-accept — control/edge-mesh-key.next,
+ // mirrored only while a rotation is in flight, but readable through this
+ // door at every other time too and refused just as hard), and the
+ // housekeeper's own election lock.
certKeys := []string{
"control/certs/example.com",
"control/certs/acme_account+key",
@@ -547,16 +550,20 @@ func TestDoorNeverServesCertMaterial(t *testing.T) {
"control/certs/tok+http-01",
"control/certs/example.com+token",
"control/edge-mesh-key",
+ "control/edge-mesh-key.next",
"control/housekeeper-lease/lock.json",
}
- // Seed two of the seven with plausible-looking secret bytes: if the ACL
+ // Seed three of the eight with plausible-looking secret bytes: if the ACL
// ever let a Get through, these are the ones that would leak real
// material instead of a "not found" — and mixing seeded with unseeded
// keys through the identical-shape check below is what proves the door
// isn't just refusing absent keys, it's refusing to distinguish "exists"
- // from "doesn't" at all for this whole prefix.
+ // from "doesn't" at all for this whole prefix. The mesh key and its
+ // rotation successor are seeded as a pair, same secret shape, since a
+ // leak of either is the same failure.
seedObject(t, store, "control/certs/example.com", "-----BEGIN CERTIFICATE-----\nfake\n-----END CERTIFICATE-----")
seedObject(t, store, "control/edge-mesh-key", "fake-mesh-shared-secret")
+ seedObject(t, store, "control/edge-mesh-key.next", "fake-mesh-shared-secret")
type shape struct {
code int
diff --git a/internal/host/agent.go b/internal/host/agent.go
index 06f3b0fb7..34d64e6a0 100644
--- a/internal/host/agent.go
+++ b/internal/host/agent.go
@@ -1626,11 +1626,20 @@ func retentionLoop(reg *registry.Registry) {
prune := housekeeperDuty{"retention pass", func(ctx context.Context) error {
return retentionPass(ctx, store, pub, reg)
}}
+ // The mesh key mint (multi-region backhaul spec, Part 3 Task 2) rides
+ // the same tick and the same store: it is a second deployment-global
+ // duty, so it belongs behind the same one election this loop already
+ // gates prune on, not a second loop contending for a second lock.
+ // mintMeshKey is create-only and idempotent — every tick after the
+ // first finds the key already minted and does nothing, quietly.
+ mintMesh := housekeeperDuty{"mesh key mint", func(ctx context.Context) error {
+ return edge.MintMeshKey(ctx, store)
+ }}
delay := time.Minute
for {
time.Sleep(delay)
delay = 5*time.Minute + rand.N(time.Minute)
- runHousekeeperDuties(housekeeperGate, prune)
+ runHousekeeperDuties(housekeeperGate, prune, mintMesh)
}
}