amadan

carlos / platform Public

edge: cert mirror — every edge serves every cert and challenge

The serving half of Task 4's issuance gate, and what makes that gate safe
to shut: a 30s poll of control/certs/ that any edge serves from.

Handshakes (wrapGetCertificate): wildcard → mirror → autocert → the local
cache read DIRECTLY. autocert runs HostPolicy BEFORE its cache lookup, so a
gated manager serves nothing at all — the mirror is the PRIMARY path on a
non-issuer, not a fallback, and the direct cache read is what carries a box
through a bucket outage or an undecided election. The short-circuit ahead of
autocert applies while gated and only then: autocert arms its renewal timers
lazily out of GetCertificate, so a mirror that answered every hello would
leave the ISSUER with no renewal timers at all.

Challenges (challengeHandler): in FRONT of autocert's HTTPHandler, which
403s a non-issuer before it looks a token up. A miss reads THROUGH to the
bucket, because a validation's whole life is shorter than a poll — bounded
by token shape, a memo of hits and misses, single flight per name, a 2s
budget and a global rate limit. The same bounded path answers tls-alpn-01
for another edge's order, which autocert PREFERS over http-01 and would
otherwise fail on (N-1)/N of every deployment's orders.

Every refresh also copies fresh material into this box's own autocert cache:
renewal timers bypass HostPolicy and check the cache first, so a stood-down
box's stale timers find a cert there instead of ordering. Account keys are
seeded when absent (one deployment, one ACME account) but never overwritten
and never servable. A changed wildcard PEM wakes the wildcard manager, so a
takeover propagates in one poll rather than twelve hours.

Refresh errors keep the last-good map: an empty map on a bucket blip would
be a TLS outage on every non-issuer. Nothing here writes to the bucket.
Paul Campbell pushed by paul@keymail.dev eea3f60828e2010252e56726531890c33e56af1c parent 7187b4d
4 files changed, +1874 −4
  • internal/edge/certsync.go +916 −0
  • internal/edge/certsync_test.go +899 −0
  • internal/edge/edge.go +50 −2
  • internal/edge/wildcard_test.go +9 −2
diff --git a/internal/edge/certsync.go b/internal/edge/certsync.go
index 40a3e835b..a12c68c58 100644
--- a/internal/edge/certsync.go
+++ b/internal/edge/certsync.go
@@ -28,14 +28,21 @@ package edge
// in seconds is never on a deadline.
import (
+ "bytes"
"context"
+ "crypto/tls"
+ "crypto/x509"
"errors"
"fmt"
"log"
+ "net/http"
"os"
+ "path"
"strings"
+ "sync"
"time"
+ "golang.org/x/crypto/acme"
"golang.org/x/crypto/acme/autocert"
"github.com/carlosframework/platform/internal/release"
@@ -322,3 +329,912 @@ func putCert(ctx context.Context, s release.Store, name string, body []byte) err
}
return fmt.Errorf("%s: lost CAS race three times", key)
}
+
+// ---- the mirror: every edge serves every cert --------------------------------
+//
+// certMirror is the serving half of the pair at the top of this file, and the
+// reason the issuance gate above is safe to shut. It POLLS control/certs/ (it
+// never writes there — the writer map gives that prefix to the housekeeper
+// alone) and keeps the whole prefix in memory, so any edge can answer a
+// handshake for a name some OTHER edge issued, and can answer an HTTP-01
+// challenge for an order some OTHER edge is running.
+//
+// Three consumers, in the order a request meets them:
+//
+// - GetCertificate (wrapGetCertificate, wildcard.go): the mirror is the
+// PRIMARY serving path on a non-issuer, ahead of autocert, because
+// autocert's HostPolicy runs BEFORE its cache lookup — a gated manager
+// serves nothing at all, cached or not.
+// - the :80 challenge handler (challengeHandler): in FRONT of autocert's
+// HTTPHandler for the same reason — that handler 403s a non-issuer before
+// it ever looks a token up.
+// - the LOCAL autocert cache: every refresh copies fresh cert material into
+// it. autocert's renewal timers never consult HostPolicy (see
+// issuerHostPolicy), and renewal.go's do() checks the cache FIRST, so a
+// mirror-refreshed local cache is what stops a stood-down box's stale
+// timers from ordering — and what carries a box through the boot and
+// crash windows where the election has not yet been decided.
+//
+// Account keys are the one thing that is mirrored but never served: they are
+// issuer material and have no business in a map a handshake can reach.
+
+// certMirrorInterval is the poll cadence. Renewals run ~30 days before
+// expiry, so nothing on this path is on a deadline — except live HTTP-01
+// validations, which the poll deliberately does NOT try to catch (see
+// challengeResponse's read-through).
+var certMirrorInterval = 30 * time.Second
+
+// certMirrorTimeout budgets one whole poll: list plus the gets. Generous,
+// because it runs on its own goroutine and blocks nothing; bounded, because a
+// stalled bucket must not wedge the loop forever.
+var certMirrorTimeout = 2 * time.Minute
+
+// certMirrorBootTimeout budgets the ONE refresh that runs before the
+// listeners bind. Much shorter than a poll's: a stalled bucket must delay the
+// front door by seconds, not minutes, and the box still has its own cache to
+// serve from while the loop catches up.
+var certMirrorBootTimeout = 20 * time.Second
+
+// certSnapshot is one poll's result, swapped in whole. Immutable once
+// published, so readers hold no lock beyond the pointer read.
+type certSnapshot struct {
+ // entries is every mirrored name → body, account keys included (the local
+ // sync needs them; Cert refuses to serve them).
+ entries map[string][]byte
+ // certs is the parsed form of every cert-shaped entry, keyed by the same
+ // name. Parsing happens ONCE per poll per changed entry, never per
+ // handshake. A name whose body does not parse is absent, not nil.
+ certs map[string]*tls.Certificate
+}
+
+var emptyCertSnapshot = &certSnapshot{entries: map[string][]byte{}, certs: map[string]*tls.Certificate{}}
+
+type certMirror struct {
+ s release.Store
+ // local is this box's OWN autocert cache — the raw one, NOT the
+ // publishing wrapper: a mirror write is a copy inwards, and pushing it
+ // back out to the bucket would have the issuer republishing its own
+ // material on every poll.
+ local autocert.Cache
+
+ // wake is called after a poll that changed wildcard material — the
+ // wildcard manager's refresh (R4), so a takeover propagates on the poll
+ // instead of on the manager's own 12h tick. nil when wildcards are off.
+ wake func()
+
+ // allowed reports whether a hostname is one this box serves (Edge.Allowed)
+ // — the bound on the tls-alpn-01 read-through, whose key is
+ // attacker-chosen SNI. nil = unbounded, which is tests only.
+ allowed func(string) bool
+
+ // gated reports whether issuance is currently BARRED on this box, which
+ // is what decides where the mirror sits in the handshake chain (see
+ // wrapGetCertificate). Read per hello, because the election moves under a
+ // running process. nil = treat as gated, the safe default: it serves the
+ // same certificates either way, it only costs an issuer its lazily-armed
+ // renewal timers, and a mirror built without a gate (tests) has no
+ // issuance to arm.
+ gated func() bool
+
+ mu sync.RWMutex
+ snap *certSnapshot
+
+ // synced remembers, per name, the mirrored bytes last reconciled into the
+ // local cache, so a steady state costs no local I/O at all.
+ syncMu sync.Mutex
+ synced map[string][]byte
+
+ // rt is the read-through memo (hits and misses), its in-flight set, and
+ // the global token bucket that bounds live bucket reads.
+ rtMu sync.Mutex
+ rt map[string]readThroughEntry
+ rtInflight map[string]chan struct{}
+ rtTokens float64
+ rtLast time.Time
+ lastRTLog time.Time
+
+ // localCerts memoes the last-resort reads of this box's own cache;
+ // lastLocalLog throttles the line that says the mirror had nothing.
+ localMu sync.Mutex
+ localCerts map[string]localCertEntry
+ lastLocalLog time.Time
+}
+
+// newCertMirror is the Edge's mirror, or nil when there is nothing to mirror
+// FROM. The nil case is exactly armCertIssuance's no-store case — the two
+// decisions are the same decision, and a box that lands there is ungated and
+// serves its own cache through autocert as it always has.
+func (e *Edge) newCertMirror(local autocert.Cache) *certMirror {
+ if e.staticStore == nil {
+ return nil
+ }
+ m := newCertMirror(e.staticStore, local)
+ m.gated = func() bool {
+ _, held := e.issuerGate()
+ return !held
+ }
+ // The tls-alpn-01 read-through is keyed by attacker-chosen SNI, so it is
+ // bounded to the hosts this box actually serves. Allowed is the same
+ // exact-match registry check autocert's HostPolicy uses, which is the
+ // right bound: a per-host order only ever exists for a name that passed it.
+ m.allowed = e.Allowed
+ return m
+}
+
+func newCertMirror(s release.Store, local autocert.Cache) *certMirror {
+ return &certMirror{
+ s: s,
+ local: local,
+ snap: emptyCertSnapshot,
+ synced: map[string][]byte{},
+ rt: map[string]readThroughEntry{},
+ rtInflight: map[string]chan struct{}{},
+ localCerts: map[string]localCertEntry{},
+ }
+}
+
+// Refresh rebuilds the whole map from one list plus a get per key. No delta
+// cleverness: the prefix holds one small object per host and renewals are
+// rare, and a full rebuild is what makes a withdrawn entry (a spent challenge
+// token) actually leave.
+//
+// An error leaves the last-good snapshot in place and is returned for the
+// caller to log. That is the edge's standing degrade-not-die posture, and it
+// matters more here than usual: an empty map on a bucket blip would be a TLS
+// outage on every non-issuer in the deployment.
+func (m *certMirror) Refresh(ctx context.Context) error {
+ keys, err := m.s.List(ctx, certPrefix)
+ if err != nil {
+ return fmt.Errorf("list %s: %w", certPrefix, err)
+ }
+ next := make(map[string][]byte, len(keys))
+ for _, key := range keys {
+ name, ok := strings.CutPrefix(key, certPrefix)
+ // publishableCertName is the same filter the publish side applies:
+ // anything nested under the prefix was not written by this deployment's
+ // issuer and has no cache name to map back onto.
+ if !ok || !publishableCertName(name) {
+ continue
+ }
+ body, _, err := m.s.Get(ctx, key)
+ if errors.Is(err, os.ErrNotExist) {
+ // Raced a delete between the list and the get — a spent token. Not
+ // a fault; the entry is simply gone.
+ continue
+ }
+ if err != nil {
+ return fmt.Errorf("get %s: %w", key, err)
+ }
+ next[name] = body
+ }
+ prev := m.snapshot()
+ snap := buildCertSnapshot(prev, next)
+ m.mu.Lock()
+ m.snap = snap
+ m.mu.Unlock()
+ m.syncLocal(ctx, snap)
+ if m.wake != nil && wildcardMaterialChanged(prev, snap) {
+ // A takeover (or a first wildcard for a new account) reaches the
+ // serving path in one poll instead of up to twelve hours.
+ m.wake()
+ }
+ return nil
+}
+
+// buildCertSnapshot parses what changed and carries the rest of the previous
+// parse forward — a handshake never pays for x509 parsing, and a quiet
+// deployment pays for it once.
+func buildCertSnapshot(prev *certSnapshot, entries map[string][]byte) *certSnapshot {
+ snap := &certSnapshot{entries: entries, certs: make(map[string]*tls.Certificate, len(entries))}
+ for name, body := range entries {
+ if !certShapedName(name) {
+ continue
+ }
+ if _, seen := prev.entries[name]; seen && bytes.Equal(prev.entries[name], body) {
+ // Unchanged. Carry the previous RESULT forward, failure included:
+ // re-parsing identical bytes cannot answer differently, and
+ // re-logging a corrupt entry every 30s forever is not "loud", it is
+ // noise.
+ if old, ok := prev.certs[name]; ok {
+ snap.certs[name] = old
+ }
+ continue
+ }
+ cert, err := parseCertEntry(body)
+ if err != nil {
+ // Loud, and skipped rather than served: a body that does not parse
+ // is a body no handshake can use, and the next poll re-reads it.
+ log.Printf("certsync: mirrored entry %q does not parse as a certificate (%v) — skipping it", name, err)
+ continue
+ }
+ snap.certs[name] = cert
+ }
+ return snap
+}
+
+func (m *certMirror) snapshot() *certSnapshot {
+ m.mu.RLock()
+ defer m.mu.RUnlock()
+ return m.snap
+}
+
+// Cert returns the mirrored body for a cache name. Account keys are refused
+// here, not filtered at poll time, because the local sync legitimately needs
+// them: the refusal belongs at the SERVING door.
+func (m *certMirror) Cert(name string) ([]byte, bool) {
+ if accountKeyName(name) {
+ return nil, false
+ }
+ b, ok := m.snapshot().entries[name]
+ return b, ok
+}
+
+// Challenge returns the HTTP-01 response body for token, if the mirror holds
+// it. The cache name is autocert's own (autocert.go:896,
+// httpTokenCacheKey): path.Base(<challenge path>) + "+http-01" — flat, with
+// no /.well-known/ prefix.
+func (m *certMirror) Challenge(token string) ([]byte, bool) {
+ b, ok := m.snapshot().entries[token+httpTokenSuffix]
+ return b, ok
+}
+
+const (
+ httpTokenSuffix = "+http-01" // autocert's HTTP-01 response entries
+ alpnTokenSuffix = "+token" // autocert's tls-alpn-01 challenge certs
+ // autocertAccountKeyName is autocert's own ACME account key entry
+ // (autocert.go's accountKey cache name). wildcardAccountKeyName
+ // (wildcard.go) is the DNS-01 manager's.
+ autocertAccountKeyName = "acme_account+key"
+)
+
+// accountKeyName reports whether name is ACME ACCOUNT material — the issuer's
+// identity at the CA, never a serving artifact.
+func accountKeyName(name string) bool {
+ return name == autocertAccountKeyName || name == wildcardAccountKeyName
+}
+
+// certShapedName reports whether an entry is expected to parse as a
+// certificate: everything except account keys and HTTP-01 response bodies.
+// That leaves per-host certs (<host>, <host>+rsa), wildcard PEMs
+// (wildcard-<parent>.pem) and tls-alpn-01 token certs (<host>+token).
+func certShapedName(name string) bool {
+ return !accountKeyName(name) && !strings.HasSuffix(name, httpTokenSuffix)
+}
+
+// parseCertEntry parses an autocert cache entry — the private key PEM
+// followed by the chain, which is also wildcard.go's encodeWildcardPEM shape —
+// into a usable tls.Certificate. Both halves come out of the one blob, and
+// X509KeyPair is what checks that the key actually matches the leaf.
+func parseCertEntry(raw []byte) (*tls.Certificate, error) {
+ cert, err := tls.X509KeyPair(raw, raw)
+ if err != nil {
+ return nil, err
+ }
+ if cert.Leaf == nil {
+ // X509KeyPair has populated Leaf since Go 1.23; belt and braces, since
+ // every serving decision below reads it.
+ leaf, err := x509.ParseCertificate(cert.Certificate[0])
+ if err != nil {
+ return nil, err
+ }
+ cert.Leaf = leaf
+ }
+ return &cert, nil
+}
+
+// run polls until stop closes — the edge's standing background-loop shape
+// (Watch): fixed ticker, errors counted and logged at the first and then
+// every tenth, recovery logged once.
+func (m *certMirror) run(stop <-chan struct{}) {
+ log.Printf("edge: cert mirror armed (%s poll of %s*) — this box serves every cert and challenge the issuer publishes", certMirrorInterval, certPrefix)
+ tick := time.NewTicker(certMirrorInterval)
+ defer tick.Stop()
+ errs := 0
+ for {
+ select {
+ case <-stop:
+ return
+ case <-tick.C:
+ }
+ ctx, cancel := context.WithTimeout(context.Background(), certMirrorTimeout)
+ err := m.Refresh(ctx)
+ cancel()
+ if err != nil {
+ errs++
+ if errs == 1 || errs%10 == 0 {
+ log.Printf("edge: cert mirror refresh failed (#%d): %v — serving the last good copy", errs, err)
+ }
+ continue
+ }
+ if errs > 0 {
+ log.Printf("edge: cert mirror refresh recovered after %d failure(s)", errs)
+ errs = 0
+ }
+ }
+}
+
+// syncLocal copies mirrored material into this box's OWN autocert cache. It
+// is the quiet half of the mirror, and the one that matters when nothing is
+// looking:
+//
+// - autocert's renewal timers bypass HostPolicy and check the cache FIRST
+// (renewal.go's do()), so a stood-down box whose timers are still armed
+// finds a fresh cert here and never reaches the CA;
+// - a box that boots, or crashes and restarts, has no election result for
+// the first ~20 minutes; a warm local cache serves through that window
+// even before the mirror map is consulted;
+// - the wildcard manager loads parents from this same cache (loadCached),
+// which is how a wildcard the issuer minted reaches a non-issuer's
+// handshakes at all.
+//
+// Never written locally: challenge tokens of either kind. They are seconds
+// long, the issuer owns their deletion, and a copy on every box's disk is
+// litter nothing sweeps.
+//
+// Writes are best-effort and loud: a full disk must not stop the mirror from
+// SERVING what it already holds in memory.
+func (m *certMirror) syncLocal(ctx context.Context, snap *certSnapshot) {
+ if m.local == nil {
+ return
+ }
+ m.syncMu.Lock()
+ defer m.syncMu.Unlock()
+ var failed int
+ var lastErr error
+ for name, body := range snap.entries {
+ if strings.HasSuffix(name, httpTokenSuffix) || strings.HasSuffix(name, alpnTokenSuffix) {
+ continue
+ }
+ // Unchanged since the last reconcile: the local copy was already
+ // settled then, and nothing but autocert itself (which only ever writes
+ // fresher material) can have moved it since. Steady state costs no
+ // local I/O at all.
+ if prev, ok := m.synced[name]; ok && bytes.Equal(prev, body) {
+ continue
+ }
+ switch err := m.reconcileLocal(ctx, name, body, snap.certs[name]); {
+ case err == nil:
+ m.synced[name] = body
+ case errors.Is(err, errUnparseableEntry):
+ // Already reported by the snapshot build; nothing to say twice.
+ default:
+ failed++
+ lastErr = err
+ }
+ }
+ if failed > 0 {
+ // One line per poll, not one per name: a local cache that has stopped
+ // accepting writes fails for EVERY name at once, and the fault is the
+ // cache, not the certificate.
+ log.Printf("certsync: %d mirrored entry/entries could not be reconciled into this box's local cert cache (last: %v) — they are still served from memory", failed, lastErr)
+ }
+ // Forget names the bucket no longer holds, so the map tracks the prefix
+ // rather than growing with every token this box has ever seen.
+ for name := range m.synced {
+ if _, ok := snap.entries[name]; !ok {
+ delete(m.synced, name)
+ }
+ }
+}
+
+// errUnparseableEntry marks the one reconcile failure that is the ENTRY's
+// fault rather than the local cache's — already reported once by the snapshot
+// build, and not worth a second line.
+var errUnparseableEntry = errors.New("certsync: mirrored entry does not parse")
+
+// reconcileLocal writes one entry into the local cache when the local copy is
+// worse. A nil error means the local copy can be considered settled.
+func (m *certMirror) reconcileLocal(ctx context.Context, name string, body []byte, cert *tls.Certificate) error {
+ local, err := m.local.Get(ctx, name)
+ if err != nil && !errors.Is(err, autocert.ErrCacheMiss) {
+ // An unreadable local cache is a real fault (permissions, disk).
+ return fmt.Errorf("read %q: %w", name, err)
+ }
+ have := err == nil
+ if accountKeyName(name) {
+ // ACME ACCOUNT material. Seeded when absent, so that a box winning the
+ // election later takes over the DEPLOYMENT's account — one account, one
+ // rate-limit bucket, which is the envelope the issuance gate's
+ // no-fencing decision is priced against. Never overwritten: a box that
+ // already registered its own account has orders and authorizations tied
+ // to that key, and replacing it under them buys nothing.
+ if have {
+ return nil
+ }
+ if err := m.local.Put(ctx, name, body); err != nil {
+ return fmt.Errorf("seed the deployment's ACME account key %q: %w", name, err)
+ }
+ log.Printf("certsync: seeded the deployment's ACME account key (%s) into the local cache", name)
+ return nil
+ }
+ if cert == nil {
+ // Did not parse (already reported by buildCertSnapshot). Writing an
+ // unparseable body over a working local cert would be a downgrade.
+ return errUnparseableEntry
+ }
+ if have {
+ if cur, err := parseCertEntry(local); err == nil && !cur.Leaf.NotAfter.Before(cert.Leaf.NotAfter) {
+ // The local copy is at least as fresh — this box may BE the issuer,
+ // one write ahead of what it has published. Leave it alone.
+ return nil
+ }
+ }
+ if err := m.local.Put(ctx, name, body); err != nil {
+ return fmt.Errorf("write %q: %w", name, err)
+ }
+ return nil
+}
+
+// wildcardPEMName reports whether an entry is a wildcard cert PEM
+// (wildcard.go's wildcardPEMKey shape).
+func wildcardPEMName(name string) bool {
+ return strings.HasPrefix(name, "wildcard-") && strings.HasSuffix(name, ".pem")
+}
+
+// wildcardMaterialChanged reports whether this poll brought a wildcard PEM
+// that is new or different. Only then is the wildcard manager nudged: its own
+// loop sleeps 12h between passes, so a takeover — or a first wildcard for a
+// new account, minted on the issuer — would otherwise wait that long to reach
+// this box's handshakes. A withdrawal is not a wake: there is nothing new to
+// load, and the manager keeps serving what it has until it expires.
+func wildcardMaterialChanged(prev, next *certSnapshot) bool {
+ for name, body := range next.entries {
+ if !wildcardPEMName(name) {
+ continue
+ }
+ if !bytes.Equal(prev.entries[name], body) {
+ return true
+ }
+ }
+ return false
+}
+
+// ---- serving: the four-layer chain ------------------------------------------
+//
+// wrapGetCertificate is the mirror's half of the handshake path. The FULL
+// chain a hello meets, outermost first (RunWithActivatorReady composes it):
+//
+// wildcard (wildcard.go) → mirror → local autocert → local cache, direct
+//
+// The order is forced by autocert's own shape, not by preference. autocert
+// runs HostPolicy — which carries the issuance gate — BEFORE its cache
+// lookup (autocert.go:292), so on a non-issuer autocert serves NOTHING: not
+// what the mirror handed it, not what it holds on disk itself. Putting the
+// mirror behind it would therefore serve nothing at all on every box but one.
+//
+// The last layer is the local cache read DIRECTLY, bypassing autocert, and it
+// is not redundant: it is what a box serves when the bucket is unreachable at
+// boot (empty mirror) but its own disk still holds yesterday's certs, and
+// what covers the ~20 minute window after a boot or a crash while the
+// housekeeper election is undecided.
+//
+// The short-circuit ahead of autocert applies while this box is GATED, and
+// only then. autocert arms its renewal timers LAZILY, out of GetCertificate
+// (m.cert -> startRenew, autocert.go:464) — a mirror that answered every
+// hello would leave an ISSUER with no renewal timers at all, and its certs
+// would expire ninety days later with nothing in the journal. An ungated box
+// loses nothing by going through autocert first: syncLocal has already put
+// every mirrored cert in the cache autocert reads, and the mirror stays
+// behind it as a fallback for anything autocert cannot serve.
+func (m *certMirror) wrapGetCertificate(inner func(*tls.ClientHelloInfo) (*tls.Certificate, error)) func(*tls.ClientHelloInfo) (*tls.Certificate, error) {
+ return func(hello *tls.ClientHelloInfo) (*tls.Certificate, error) {
+ name := strings.ToLower(strings.TrimSuffix(hello.ServerName, "."))
+ if len(hello.SupportedProtos) == 1 && hello.SupportedProtos[0] == acme.ALPNProto {
+ // A tls-alpn-01 validation. autocert answers its OWN order from
+ // memory; anything else is another edge's order, and the CA reached
+ // this box because every edge answers the same names.
+ cert, err := inner(hello)
+ if err == nil {
+ return cert, nil
+ }
+ if c := m.alpnTokenCert(helloContext(hello), name); c != nil {
+ return c, nil
+ }
+ return nil, err
+ }
+ wantRSA := !helloSupportsECDSA(hello)
+ if m.isGated() {
+ if c := m.certificate(name, wantRSA); c != nil {
+ return c, nil
+ }
+ }
+ cert, err := inner(hello)
+ if err == nil {
+ return cert, nil
+ }
+ // Ungated and autocert could not answer (an order that failed, a name
+ // it will not order for): the mirror is still the better answer than a
+ // dropped handshake. Gated, this repeat is a cheap map miss.
+ if c := m.certificate(name, wantRSA); c != nil {
+ return c, nil
+ }
+ if c := m.localCertificate(helloContext(hello), name, wantRSA); c != nil {
+ return c, nil
+ }
+ return nil, err
+ }
+}
+
+// isGated reads the issuance gate, defaulting to "gated" (see the field).
+func (m *certMirror) isGated() bool {
+ return m.gated == nil || m.gated()
+}
+
+// helloContext is the handshake's context, or a background one for the
+// hand-built hellos tests use.
+func helloContext(hello *tls.ClientHelloInfo) context.Context {
+ if ctx := hello.Context(); ctx != nil {
+ return ctx
+ }
+ return context.Background()
+}
+
+// certificate returns a mirrored cert for name, or nil. The two cache names
+// are autocert's own ("<host>" for ECDSA, "<host>+rsa" for legacy clients);
+// the preferred one is tried first and the other accepted rather than
+// dropping the handshake, since on a non-issuer no order can produce the
+// missing type.
+func (m *certMirror) certificate(name string, wantRSA bool) *tls.Certificate {
+ if name == "" {
+ return nil
+ }
+ snap := m.snapshot()
+ for _, key := range certEntryNames(name, wantRSA) {
+ if c := snap.certs[key]; usableFor(c, name) {
+ return c
+ }
+ }
+ return nil
+}
+
+// certEntryNames lists the cache names that can hold a cert for host, most
+// preferred first.
+func certEntryNames(host string, wantRSA bool) [2]string {
+ if wantRSA {
+ return [2]string{host + "+rsa", host}
+ }
+ return [2]string{host, host + "+rsa"}
+}
+
+// usableFor reports whether cert can be served for name right now. An expired
+// or wrong-named entry is treated as absent so the chain keeps falling
+// through — a dead cert fails the handshake just as surely as no cert, and
+// the next layer may have a live one.
+func usableFor(cert *tls.Certificate, name string) bool {
+ return cert != nil && cert.Leaf != nil &&
+ time.Now().Before(cert.Leaf.NotAfter) &&
+ cert.Leaf.VerifyHostname(name) == nil
+}
+
+// helloSupportsECDSA is autocert's own supportsECDSA (autocert.go), ported
+// because it is unexported there and the mirror must look under the SAME
+// cache name autocert would have chosen for this client.
+func helloSupportsECDSA(hello *tls.ClientHelloInfo) bool {
+ if hello.SignatureSchemes != nil {
+ ecdsaOK := false
+ schemeLoop:
+ for _, scheme := range hello.SignatureSchemes {
+ const tlsECDSAWithSHA1 tls.SignatureScheme = 0x0203
+ switch scheme {
+ case tlsECDSAWithSHA1, tls.ECDSAWithP256AndSHA256,
+ tls.ECDSAWithP384AndSHA384, tls.ECDSAWithP521AndSHA512:
+ ecdsaOK = true
+ break schemeLoop
+ }
+ }
+ if !ecdsaOK {
+ return false
+ }
+ }
+ if hello.SupportedCurves != nil {
+ ecdsaOK := false
+ for _, curve := range hello.SupportedCurves {
+ if curve == tls.CurveP256 {
+ ecdsaOK = true
+ break
+ }
+ }
+ if !ecdsaOK {
+ return false
+ }
+ }
+ for _, suite := range hello.CipherSuites {
+ switch suite {
+ case tls.TLS_ECDHE_ECDSA_WITH_RC4_128_SHA,
+ tls.TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,
+ tls.TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,
+ tls.TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,
+ tls.TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,
+ tls.TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,
+ tls.TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305:
+ return true
+ }
+ }
+ return false
+}
+
+// localCertTTL bounds how long a last-resort read is remembered. Short
+// enough that a renewal reaching the local cache is picked up promptly, long
+// enough that a flood of handshakes for one name is not a file read each.
+const localCertTTL = 30 * time.Second
+
+type localCertEntry struct {
+ cert *tls.Certificate
+ at time.Time
+}
+
+// localCertificate is the last layer: this box's own autocert cache, read
+// DIRECTLY, because a gated autocert.Manager refuses the name before it ever
+// looks there. Both outcomes are remembered briefly so a busy name costs one
+// file read per TTL rather than one per handshake.
+func (m *certMirror) localCertificate(ctx context.Context, name string, wantRSA bool) *tls.Certificate {
+ if m.local == nil || name == "" {
+ return nil
+ }
+ for _, key := range certEntryNames(name, wantRSA) {
+ m.localMu.Lock()
+ e, ok := m.localCerts[key]
+ fresh := ok && time.Since(e.at) < localCertTTL
+ m.localMu.Unlock()
+ if fresh {
+ if usableFor(e.cert, name) {
+ return e.cert
+ }
+ continue
+ }
+ var cert *tls.Certificate
+ if raw, err := m.local.Get(ctx, key); err == nil {
+ if c, err := parseCertEntry(raw); err == nil {
+ cert = c
+ }
+ }
+ m.localMu.Lock()
+ if len(m.localCerts) > readThroughCacheMax {
+ // Crude but bounded: the map is a cache, and every entry is
+ // re-readable from disk.
+ clear(m.localCerts)
+ }
+ m.localCerts[key] = localCertEntry{cert: cert, at: time.Now()}
+ m.localMu.Unlock()
+ if usableFor(cert, name) {
+ // Worth saying: the mirror had nothing for a name this box is
+ // serving anyway. Once a minute for the whole box, not once per
+ // name — every host on a box whose bucket is unreachable lands here
+ // at once, and one line is enough to know it.
+ m.localMu.Lock()
+ quiet := time.Since(m.lastLocalLog) < time.Minute
+ if !quiet {
+ m.lastLocalLog = time.Now()
+ }
+ m.localMu.Unlock()
+ if !quiet {
+ log.Printf("edge: serving %s from this box's local cert cache — the mirror has no copy and issuance is gated here", name)
+ }
+ return cert
+ }
+ }
+ return nil
+}
+
+// ---- the :80 challenge path -------------------------------------------------
+
+const acmeChallengePrefix = "/.well-known/acme-challenge/"
+
+// challengeHandler answers HTTP-01 challenges from the mirror BEFORE
+// autocert's own handler sees them, and delegates everything else. The order
+// is forced, again, by autocert: HTTPHandler runs HostPolicy first
+// (autocert.go:405), so on a non-issuer it 403s a challenge whose answer is
+// sitting in the mirror. Behind this, autocert still answers its OWN live
+// orders from memory, which is what the issuer needs.
+func (m *certMirror) challengeHandler(next http.Handler) http.Handler {
+ return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ if !strings.HasPrefix(r.URL.Path, acmeChallengePrefix) {
+ next.ServeHTTP(w, r)
+ return
+ }
+ // path.Base is the token autocert itself would derive
+ // (httpTokenCacheKey), so both handlers agree on the name.
+ if body, ok := m.challengeResponse(r.Context(), path.Base(r.URL.Path)); ok {
+ w.Header().Set("Content-Type", "text/plain; charset=utf-8")
+ _, _ = w.Write(body)
+ return
+ }
+ next.ServeHTTP(w, r)
+ })
+}
+
+// challengeResponse answers from the polled map, then — and this is the part
+// the 30s poll cannot do — reads through to the bucket. A validation's whole
+// life is a few seconds: the issuer writes the token, sends Accept, and the
+// CA arrives at whichever edge DNS handed it. A poll that runs every 30s
+// misses that window every time.
+func (m *certMirror) challengeResponse(ctx context.Context, token string) ([]byte, bool) {
+ if body, ok := m.Challenge(token); ok {
+ return body, true
+ }
+ if !validChallengeToken(token) {
+ return nil, false
+ }
+ body, _, ok := m.readThrough(ctx, token+httpTokenSuffix)
+ return body, ok
+}
+
+// validChallengeToken bounds what an unauthenticated URL path may turn into a
+// bucket read: ACME tokens are base64url of at least 128 bits of entropy, so
+// nothing shorter or otherwise shaped can be one. It also keeps the key
+// inside the flat prefix (publishableCertName's concern, from the other
+// side).
+func validChallengeToken(token string) bool {
+ if len(token) < 16 || len(token) > 128 {
+ return false
+ }
+ for _, r := range token {
+ switch {
+ case r >= 'a' && r <= 'z', r >= 'A' && r <= 'Z', r >= '0' && r <= '9', r == '-', r == '_':
+ default:
+ return false
+ }
+ }
+ return true
+}
+
+// alpnTokenCert answers a tls-alpn-01 validation for an order some OTHER edge
+// is running. autocert prefers tls-alpn-01 over http-01 and only falls back
+// to it after a whole failed order (autocert.go's AuthorizeOrderLoop), so
+// without this every issuance in an N-edge deployment burns a failed
+// validation with probability (N-1)/N before it ever tries the HTTP path.
+//
+// Bounded harder than the HTTP-01 path, because the name here is
+// attacker-chosen SNI rather than a high-entropy token: only hosts this box
+// actually serves are ever looked up.
+func (m *certMirror) alpnTokenCert(ctx context.Context, host string) *tls.Certificate {
+ if host == "" {
+ return nil
+ }
+ key := host + alpnTokenSuffix
+ if c := m.snapshot().certs[key]; usableFor(c, host) {
+ return c
+ }
+ if m.allowed != nil && !m.allowed(host) {
+ return nil
+ }
+ _, cert, _ := m.readThrough(ctx, key)
+ if usableFor(cert, host) {
+ return cert
+ }
+ return nil
+}
+
+// ---- read-through: the bounded live lookup ----------------------------------
+
+const (
+ // readThroughTimeout budgets one live bucket read. It runs inside a
+ // handshake or a :80 request, so it is short by construction.
+ readThroughTimeout = 2 * time.Second
+ // readThroughHitTTL is about one poll: after that the poll itself holds
+ // the entry (or it has been withdrawn, which is also the answer).
+ readThroughHitTTL = 30 * time.Second
+ // readThroughMissTTL keeps a scanner walking the challenge path from
+ // turning into a bucket GET per request.
+ readThroughMissTTL = 5 * time.Second
+ // readThroughCacheMax bounds the memo; over it, the map is dropped whole.
+ readThroughCacheMax = 512
+ // readThroughBurst / readThroughRefill are the global budget: distinct
+ // names defeat the negative cache, so the last bound is a rate. Under a
+ // flood the read-through degrades to "off" — the 30s poll still covers
+ // everything that is not a live validation.
+ readThroughBurst = 20
+ readThroughRefill = 5 // per second
+)
+
+type readThroughEntry struct {
+ body []byte
+ cert *tls.Certificate
+ ok bool
+ at time.Time
+}
+
+func (e readThroughEntry) fresh() bool {
+ ttl := readThroughMissTTL
+ if e.ok {
+ ttl = readThroughHitTTL
+ }
+ return time.Since(e.at) < ttl
+}
+
+// readThrough fetches one control/certs/ entry outside the poll cycle: memoed
+// (hits and misses), single-flighted per name, rate-limited globally, and
+// budgeted per call. Errors are answers here, not faults — a bucket that is
+// down simply means this box cannot help with someone else's live
+// validation, and the CA retries.
+func (m *certMirror) readThrough(ctx context.Context, name string) ([]byte, *tls.Certificate, bool) {
+ m.rtMu.Lock()
+ if e, ok := m.rt[name]; ok && e.fresh() {
+ m.rtMu.Unlock()
+ return e.body, e.cert, e.ok
+ }
+ if done, ok := m.rtInflight[name]; ok {
+ // Somebody is already asking. The CA opens several connections for one
+ // validation; they share the one bucket read.
+ m.rtMu.Unlock()
+ select {
+ case <-done:
+ case <-ctx.Done():
+ return nil, nil, false
+ }
+ m.rtMu.Lock()
+ e := m.rt[name]
+ m.rtMu.Unlock()
+ return e.body, e.cert, e.ok
+ }
+ if !m.allowFetchLocked() {
+ m.rtMu.Unlock()
+ return nil, nil, false
+ }
+ done := make(chan struct{})
+ m.rtInflight[name] = done
+ m.rtMu.Unlock()
+
+ // Detached from the caller: a client that hangs up mid-validation must not
+ // cancel the read every other in-flight waiter is sharing.
+ fetchCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), readThroughTimeout)
+ body, _, err := m.s.Get(fetchCtx, certKey(name))
+ cancel()
+ e := readThroughEntry{at: time.Now()}
+ switch {
+ case err == nil:
+ e.body, e.ok = body, true
+ if certShapedName(name) {
+ if cert, perr := parseCertEntry(body); perr == nil {
+ e.cert = cert
+ } else {
+ log.Printf("certsync: live read of %q does not parse as a certificate: %v", name, perr)
+ }
+ }
+ case errors.Is(err, os.ErrNotExist):
+ // The ordinary case: no such token. Negative-cached, not logged.
+ default:
+ m.logReadThroughError(name, err)
+ }
+ m.rtMu.Lock()
+ if len(m.rt) > readThroughCacheMax {
+ clear(m.rt)
+ }
+ m.rt[name] = e
+ delete(m.rtInflight, name)
+ m.rtMu.Unlock()
+ close(done)
+ return e.body, e.cert, e.ok
+}
+
+// logReadThroughError says once a minute that live lookups are failing. Not
+// per call: the rate limiter still allows several a second, and a bucket
+// outage is one fault however many validations hit it.
+func (m *certMirror) logReadThroughError(name string, err error) {
+ m.rtMu.Lock()
+ quiet := time.Since(m.lastRTLog) < time.Minute
+ if !quiet {
+ m.lastRTLog = time.Now()
+ }
+ m.rtMu.Unlock()
+ if !quiet {
+ log.Printf("certsync: live read of %q failed: %v — the %s poll still covers everything but live validations", name, err, certMirrorInterval)
+ }
+}
+
+// allowFetchLocked spends one token from the read-through budget. Caller
+// holds rtMu.
+func (m *certMirror) allowFetchLocked() bool {
+ now := time.Now()
+ if m.rtLast.IsZero() {
+ m.rtTokens = readThroughBurst
+ } else {
+ m.rtTokens = min(readThroughBurst, m.rtTokens+now.Sub(m.rtLast).Seconds()*readThroughRefill)
+ }
+ m.rtLast = now
+ if m.rtTokens < 1 {
+ return false
+ }
+ m.rtTokens--
+ return true
+}
diff --git a/internal/edge/certsync_test.go b/internal/edge/certsync_test.go
index 42cbff42b..34f0d8946 100644
--- a/internal/edge/certsync_test.go
+++ b/internal/edge/certsync_test.go
@@ -5,8 +5,10 @@ import (
"context"
"crypto/tls"
"errors"
+ "fmt"
"io"
"log"
+ "net/http"
"net/http/httptest"
"os"
"strings"
@@ -527,3 +529,900 @@ func TestWildcardUngatedOrdersAsToday(t *testing.T) {
t.Fatal("with no gate configured (carlos edge, single box) issuance must run exactly as today")
}
}
+
+// ---- the mirror (Task 5) ----------------------------------------------------
+
+// hostCertPEM mints a cert for host in autocert's own cache-entry shape (the
+// private key PEM, then the chain) — byte-for-byte what the issuer publishes
+// to control/certs/<host>.
+func hostCertPEM(t *testing.T, host string, notAfter time.Time) []byte {
+ t.Helper()
+ cert, key := selfSignedLeaf(t, host, notAfter)
+ raw, err := encodeWildcardPEM(cert, key)
+ if err != nil {
+ t.Fatal(err)
+ }
+ return raw
+}
+
+// putBucket writes one control/certs/ entry the way the issuer's
+// publishingCache would.
+func putBucket(t *testing.T, s release.Store, name string, body []byte) {
+ t.Helper()
+ if err := putCert(context.Background(), s, name, body); err != nil {
+ t.Fatal(err)
+ }
+}
+
+func TestCertMirrorServesBucketCert(t *testing.T) {
+ s := release.NewDirStore(t.TempDir())
+ putBucket(t, s, "example.com", []byte("PEM"))
+ m := newCertMirror(s, nil)
+ if err := m.Refresh(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ if b, ok := m.Cert("example.com"); !ok || string(b) != "PEM" {
+ t.Fatalf("mirror cert = %q, %v", b, ok)
+ }
+ if _, ok := m.Cert("nobody.example"); ok {
+ t.Fatal("a name the bucket does not hold must miss")
+ }
+}
+
+// The HTTP-01 cache key is autocert's, read from the vendored source
+// (autocert.go:896): path.Base(tokenPath) + "+http-01" — a FLAT name, no
+// /.well-known/ prefix (publishableCertName would refuse one anyway). Task 7
+// fails loudly if this drifts.
+func TestCertMirrorServesChallengeTokens(t *testing.T) {
+ const token = "0123456789abcdefghij0123456789abcdefghijkl"
+ s := release.NewDirStore(t.TempDir())
+ putBucket(t, s, token+"+http-01", []byte("resp"))
+ m := newCertMirror(s, nil)
+ if err := m.Refresh(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ if b, ok := m.Challenge(token); !ok || string(b) != "resp" {
+ t.Fatalf("challenge = %q, %v", b, ok)
+ }
+}
+
+// Account keys are ISSUER material: they must never enter the serving map,
+// where a name lookup could hand one to a handshake.
+func TestCertMirrorKeepsAccountKeysOutOfTheServingMap(t *testing.T) {
+ s := release.NewDirStore(t.TempDir())
+ putBucket(t, s, "acme_account+key", []byte("KEY"))
+ putBucket(t, s, wildcardAccountKeyName, []byte("KEY"))
+ putBucket(t, s, "example.com", []byte("PEM"))
+ m := newCertMirror(s, nil)
+ if err := m.Refresh(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ for _, name := range []string{"acme_account+key", wildcardAccountKeyName} {
+ if _, ok := m.Cert(name); ok {
+ t.Fatalf("%s is issuer material and must not be servable", name)
+ }
+ }
+ if _, ok := m.Cert("example.com"); !ok {
+ t.Fatal("ordinary cert material must still be served")
+ }
+}
+
+// listFailingStore fails List (and optionally Get) on demand — a bucket blip
+// mid-poll.
+type listFailingStore struct {
+ release.Store
+ mu sync.Mutex
+ fail bool
+}
+
+func (l *listFailingStore) setFail(v bool) {
+ l.mu.Lock()
+ l.fail = v
+ l.mu.Unlock()
+}
+
+func (l *listFailingStore) failing() bool {
+ l.mu.Lock()
+ defer l.mu.Unlock()
+ return l.fail
+}
+
+func (l *listFailingStore) List(ctx context.Context, prefix string) ([]string, error) {
+ if l.failing() {
+ return nil, errors.New("bucket down")
+ }
+ return l.Store.List(ctx, prefix)
+}
+
+func (l *listFailingStore) Get(ctx context.Context, key string) ([]byte, string, error) {
+ if l.failing() {
+ return nil, "", errors.New("bucket down")
+ }
+ return l.Store.Get(ctx, key)
+}
+
+// Degrade, don't die: a failed poll keeps the last-good map. The alternative
+// (an empty map) turns a bucket blip into a TLS outage on every non-issuer.
+func TestCertMirrorRefreshErrorKeepsLastGood(t *testing.T) {
+ s := &listFailingStore{Store: release.NewDirStore(t.TempDir())}
+ putBucket(t, s.Store, "example.com", []byte("PEM"))
+ m := newCertMirror(s, nil)
+ if err := m.Refresh(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ s.setFail(true)
+ if err := m.Refresh(context.Background()); err == nil {
+ t.Fatal("a failed poll must be reported to the loop, which logs it")
+ }
+ if b, ok := m.Cert("example.com"); !ok || string(b) != "PEM" {
+ t.Fatalf("after a failed refresh the mirror = %q, %v — want the last-good copy", b, ok)
+ }
+}
+
+// A withdrawn entry (a spent challenge token the issuer deleted) leaves the
+// map on the next good poll.
+func TestCertMirrorRefreshDropsWithdrawnEntries(t *testing.T) {
+ s := release.NewDirStore(t.TempDir())
+ putBucket(t, s, "example.com", []byte("PEM"))
+ m := newCertMirror(s, nil)
+ if err := m.Refresh(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ if err := s.Delete(context.Background(), certKey("example.com")); err != nil {
+ t.Fatal(err)
+ }
+ if err := m.Refresh(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ if _, ok := m.Cert("example.com"); ok {
+ t.Fatal("a deleted bucket entry must leave the mirror")
+ }
+}
+
+// ---- R3: the mirror feeds this box's own autocert cache ---------------------
+
+// localCert reads a name back out of a local autocert cache.
+func localCert(t *testing.T, c autocert.Cache, name string) ([]byte, error) {
+ t.Helper()
+ return c.Get(context.Background(), name)
+}
+
+// autocert's renewal timers never consult HostPolicy, and renewal.go's do()
+// checks the CACHE first — so a mirror-fed local cache is what stops a
+// stood-down box's stale timers from ordering. It also carries a box through
+// the boot window before the election has been decided.
+func TestCertMirrorWritesFreshCertsIntoTheLocalCache(t *testing.T) {
+ const host = "app.bab.oncarlos.com"
+ s := release.NewDirStore(t.TempDir())
+ fresh := hostCertPEM(t, host, time.Now().Add(80*24*time.Hour))
+ putBucket(t, s, host, fresh)
+ putBucket(t, s, "wildcard-bab.oncarlos.com.pem", hostCertPEM(t, "*.bab.oncarlos.com", time.Now().Add(80*24*time.Hour)))
+
+ local := autocert.DirCache(t.TempDir())
+ m := newCertMirror(s, local)
+ if err := m.Refresh(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ got, err := localCert(t, local, host)
+ if err != nil || !bytes.Equal(got, fresh) {
+ t.Fatalf("local cache after refresh = %d byte(s), %v — want the mirrored cert", len(got), err)
+ }
+ // The wildcard PEM lands under wildcard.go's own cache name, which is what
+ // the wildcard manager's loadCached reads.
+ if _, err := localCert(t, local, "wildcard-bab.oncarlos.com.pem"); err != nil {
+ t.Fatalf("wildcard PEM did not reach the local cache: %v", err)
+ }
+}
+
+// A renewal in the bucket replaces an older local copy; a local copy that is
+// already newer is left alone (this box may be the issuer, one poll ahead of
+// what it published).
+func TestCertMirrorLocalWriteRespectsFreshness(t *testing.T) {
+ const host = "app.bab.oncarlos.com"
+ s := release.NewDirStore(t.TempDir())
+ local := autocert.DirCache(t.TempDir())
+
+ old := hostCertPEM(t, host, time.Now().Add(10*24*time.Hour))
+ newer := hostCertPEM(t, host, time.Now().Add(80*24*time.Hour))
+
+ // Local holds the OLD one; the bucket holds a renewal.
+ if err := local.Put(context.Background(), host, old); err != nil {
+ t.Fatal(err)
+ }
+ putBucket(t, s, host, newer)
+ m := newCertMirror(s, local)
+ if err := m.Refresh(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ if got, _ := localCert(t, local, host); !bytes.Equal(got, newer) {
+ t.Fatal("a renewal in the bucket must replace the older local copy")
+ }
+
+ // Now the other way round: local is fresher than the bucket.
+ local2 := autocert.DirCache(t.TempDir())
+ if err := local2.Put(context.Background(), host, newer); err != nil {
+ t.Fatal(err)
+ }
+ s2 := release.NewDirStore(t.TempDir())
+ putBucket(t, s2, host, old)
+ m2 := newCertMirror(s2, local2)
+ if err := m2.Refresh(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ if got, _ := localCert(t, local2, host); !bytes.Equal(got, newer) {
+ t.Fatal("a local cert that is already fresher must not be downgraded by the mirror")
+ }
+}
+
+// Account keys reach the local cache for issuer-takeover warmth (one ACME
+// account per deployment, one rate-limit bucket) — but only when this box has
+// none: clobbering a registered account key is never an improvement.
+func TestCertMirrorSeedsAccountKeysOnlyWhenAbsent(t *testing.T) {
+ s := release.NewDirStore(t.TempDir())
+ putBucket(t, s, "acme_account+key", []byte("BUCKET-KEY"))
+ putBucket(t, s, wildcardAccountKeyName, []byte("BUCKET-WILDCARD-KEY"))
+
+ local := autocert.DirCache(t.TempDir())
+ m := newCertMirror(s, local)
+ if err := m.Refresh(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ if got, err := localCert(t, local, "acme_account+key"); err != nil || string(got) != "BUCKET-KEY" {
+ t.Fatalf("account key not seeded: %q, %v", got, err)
+ }
+
+ local2 := autocert.DirCache(t.TempDir())
+ if err := local2.Put(context.Background(), "acme_account+key", []byte("MINE")); err != nil {
+ t.Fatal(err)
+ }
+ m2 := newCertMirror(s, local2)
+ if err := m2.Refresh(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ if got, _ := localCert(t, local2, "acme_account+key"); string(got) != "MINE" {
+ t.Fatalf("this box's own account key was clobbered: %q", got)
+ }
+}
+
+// Challenge tokens are ephemeral and the issuer owns their deletion: copying
+// them onto every box's disk would litter caches nothing ever sweeps.
+func TestCertMirrorNeverWritesChallengeTokensLocally(t *testing.T) {
+ const token = "0123456789abcdefghij0123456789abcdefghijkl"
+ s := release.NewDirStore(t.TempDir())
+ putBucket(t, s, token+"+http-01", []byte("resp"))
+ local := autocert.DirCache(t.TempDir())
+ m := newCertMirror(s, local)
+ if err := m.Refresh(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ if _, err := localCert(t, local, token+"+http-01"); !errors.Is(err, autocert.ErrCacheMiss) {
+ t.Fatalf("a challenge token must not land in the local cache: %v", err)
+ }
+}
+
+// A steady state costs no local I/O: an unchanged entry is not rewritten.
+func TestCertMirrorSkipsUnchangedLocalWrites(t *testing.T) {
+ const host = "app.bab.oncarlos.com"
+ s := release.NewDirStore(t.TempDir())
+ putBucket(t, s, host, hostCertPEM(t, host, time.Now().Add(80*24*time.Hour)))
+ local := &countingWriteCache{Cache: autocert.DirCache(t.TempDir())}
+ m := newCertMirror(s, local)
+ for range 3 {
+ if err := m.Refresh(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ }
+ if n := local.puts(); n != 1 {
+ t.Fatalf("local writes = %d, want exactly 1 — an unchanged mirror must not rewrite the cache", n)
+ }
+}
+
+// countingWriteCache counts Puts.
+type countingWriteCache struct {
+ autocert.Cache
+ mu sync.Mutex
+ n int
+}
+
+func (c *countingWriteCache) Put(ctx context.Context, key string, data []byte) error {
+ c.mu.Lock()
+ c.n++
+ c.mu.Unlock()
+ return c.Cache.Put(ctx, key, data)
+}
+
+func (c *countingWriteCache) puts() int {
+ c.mu.Lock()
+ defer c.mu.Unlock()
+ return c.n
+}
+
+// ---- R4: a wildcard takeover propagates on the poll -------------------------
+
+func TestCertMirrorWakesTheWildcardManagerOnChangedWildcardMaterial(t *testing.T) {
+ s := release.NewDirStore(t.TempDir())
+ putBucket(t, s, "app.bab.oncarlos.com", hostCertPEM(t, "app.bab.oncarlos.com", time.Now().Add(80*24*time.Hour)))
+ local := autocert.DirCache(t.TempDir())
+ m := newCertMirror(s, local)
+ var wakes int
+ m.wake = func() { wakes++ }
+
+ if err := m.Refresh(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ if wakes != 0 {
+ t.Fatalf("a per-host cert is not wildcard material: %d wake(s)", wakes)
+ }
+
+ putBucket(t, s, "wildcard-bab.oncarlos.com.pem", hostCertPEM(t, "*.bab.oncarlos.com", time.Now().Add(80*24*time.Hour)))
+ if err := m.Refresh(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ if wakes != 1 {
+ t.Fatalf("a new wildcard must wake the manager: %d wake(s)", wakes)
+ }
+ if err := m.Refresh(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ if wakes != 1 {
+ t.Fatalf("an unchanged poll must not wake the manager: %d wake(s)", wakes)
+ }
+ putBucket(t, s, "wildcard-bab.oncarlos.com.pem", hostCertPEM(t, "*.bab.oncarlos.com", time.Now().Add(85*24*time.Hour)))
+ if err := m.Refresh(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ if wakes != 2 {
+ t.Fatalf("a renewed wildcard must wake the manager: %d wake(s)", wakes)
+ }
+}
+
+// ---- R1: the serving chain --------------------------------------------------
+
+const mirrorHost = "app.bab.oncarlos.com"
+
+// gatedChain builds what RunWithActivatorReady builds for a NON-ISSUER: a
+// gated autocert manager with m in front of it and autocert's own cache
+// (counted, so "did autocert serve this?" is observable) behind it.
+//
+// The mirror is passed in rather than built here so that a test can give it
+// NO local cache: the mirror layer and the last-resort local-cache layer both
+// serve the same bytes, and a test that means to pin one must not be able to
+// pass on the other.
+func gatedChain(t *testing.T, m *certMirror, local autocert.Cache, hosts ...string) (func(*tls.ClientHelloInfo) (*tls.Certificate, error), *countingCache) {
+ t.Helper()
+ e := testIssuerEdge(hosts...)
+ e.issuer = func() (context.Context, bool) { return nil, false } // never the issuer
+ counting := &countingCache{Cache: local, gets: map[string]int{}}
+ mgr := offlineManager(t, counting, e.issuerHostPolicy())
+ return m.wrapGetCertificate(mgr.GetCertificate), counting
+}
+
+// The whole point of Task 5: a gated box serves the issuer's cert, and does
+// so BEFORE autocert — which would refuse the name outright.
+func TestMirrorServesTheGatedNonIssuersHandshake(t *testing.T) {
+ s := release.NewDirStore(t.TempDir())
+ putBucket(t, s, mirrorHost, hostCertPEM(t, mirrorHost, time.Now().Add(80*24*time.Hour)))
+ // No local cache on the mirror: only the mirror layer itself can answer.
+ m := newCertMirror(s, nil)
+ get, counting := gatedChain(t, m, autocert.DirCache(t.TempDir()), mirrorHost)
+ if err := m.Refresh(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ got, err := get(ecdsaHello(mirrorHost))
+ if err != nil {
+ t.Fatalf("a non-issuer must serve the mirrored cert: %v", err)
+ }
+ if got.Leaf == nil || got.Leaf.VerifyHostname(mirrorHost) != nil {
+ t.Fatalf("served cert does not cover %s", mirrorHost)
+ }
+ if n := counting.count(mirrorHost); n != 0 {
+ t.Fatalf("autocert was consulted %d time(s) — a valid mirror cert must short-circuit before it", n)
+ }
+}
+
+// Each layer's miss falls through cleanly. With no gate (a single-box
+// deployment) and an empty mirror, autocert serves from its own cache exactly
+// as it does today.
+func TestMirrorFallsThroughToAutocert(t *testing.T) {
+ local := autocert.DirCache(t.TempDir())
+ if err := local.Put(context.Background(), mirrorHost, hostCertPEM(t, mirrorHost, time.Now().Add(80*24*time.Hour))); err != nil {
+ t.Fatal(err)
+ }
+ e := testIssuerEdge(mirrorHost) // ungated
+ m := newCertMirror(release.NewDirStore(t.TempDir()), local)
+ get := m.wrapGetCertificate(offlineManager(t, local, e.issuerHostPolicy()).GetCertificate)
+ if _, err := get(ecdsaHello(mirrorHost)); err != nil {
+ t.Fatalf("an empty mirror must fall through to autocert: %v", err)
+ }
+}
+
+// Last resort: gated autocert AND an empty mirror (a bucket outage across a
+// restart), but the cert is on this box's disk from before. autocert refuses
+// it — HostPolicy runs before its cache lookup — so the chain reads the
+// DirCache directly rather than dropping the handshake.
+func TestMirrorLastResortServesTheLocalCacheEvenWhenGated(t *testing.T) {
+ local := autocert.DirCache(t.TempDir())
+ if err := local.Put(context.Background(), mirrorHost, hostCertPEM(t, mirrorHost, time.Now().Add(80*24*time.Hour))); err != nil {
+ t.Fatal(err)
+ }
+ m := newCertMirror(release.NewDirStore(t.TempDir()), local)
+ get, _ := gatedChain(t, m, local, mirrorHost)
+ got, err := get(ecdsaHello(mirrorHost))
+ if err != nil {
+ t.Fatalf("the local cache is the last resort, and it has the cert: %v", err)
+ }
+ if got.Leaf.VerifyHostname(mirrorHost) != nil {
+ t.Fatal("served cert does not cover the host")
+ }
+}
+
+// Nothing anywhere: the error the caller sees is autocert's own, so the
+// journal still says WHY (not-issuer, or unknown host) rather than blaming
+// the mirror.
+func TestMirrorChainKeepsAutocertsError(t *testing.T) {
+ get, _ := gatedChain(t, newCertMirror(release.NewDirStore(t.TempDir()), nil), autocert.DirCache(t.TempDir()), mirrorHost)
+ _, err := get(ecdsaHello(mirrorHost))
+ if !errors.Is(err, errNotIssuer) {
+ t.Fatalf("chain error = %v, want errNotIssuer", err)
+ }
+}
+
+// A mirrored entry that is expired, or does not cover the name asked for, is
+// not served: the chain falls through instead. Serving a dead cert is worse
+// than serving none — the handshake fails either way, but a fall-through can
+// still find a good one.
+func TestMirrorRefusesUnusableEntries(t *testing.T) {
+ for _, tc := range []struct {
+ name string
+ key string
+ pem []byte
+ }{
+ {"expired", mirrorHost, hostCertPEM(t, mirrorHost, time.Now().Add(-time.Hour))},
+ {"wrong host", mirrorHost, hostCertPEM(t, "other.bab.oncarlos.com", time.Now().Add(80*24*time.Hour))},
+ } {
+ t.Run(tc.name, func(t *testing.T) {
+ s := release.NewDirStore(t.TempDir())
+ putBucket(t, s, tc.key, tc.pem)
+ m := newCertMirror(s, nil)
+ get, _ := gatedChain(t, m, autocert.DirCache(t.TempDir()), mirrorHost)
+ if err := m.Refresh(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ if _, err := get(ecdsaHello(mirrorHost)); !errors.Is(err, errNotIssuer) {
+ t.Fatalf("an unusable mirrored entry must fall through, got %v", err)
+ }
+ })
+ }
+}
+
+// autocert keys legacy clients' certs under "<host>+rsa", so the mirror must
+// look under the SAME name for the SAME client — with both entries present,
+// the choice is the one autocert would have made.
+func TestMirrorPicksTheEntryAutocertWouldHave(t *testing.T) {
+ s := release.NewDirStore(t.TempDir())
+ ecdsaPEM := hostCertPEM(t, mirrorHost, time.Now().Add(80*24*time.Hour))
+ rsaPEM := hostCertPEM(t, mirrorHost, time.Now().Add(70*24*time.Hour))
+ putBucket(t, s, mirrorHost, ecdsaPEM)
+ putBucket(t, s, mirrorHost+"+rsa", rsaPEM)
+ m := newCertMirror(s, nil)
+ get, _ := gatedChain(t, m, autocert.DirCache(t.TempDir()), mirrorHost)
+ if err := m.Refresh(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ rsaOnly := &tls.ClientHelloInfo{
+ ServerName: mirrorHost,
+ SignatureSchemes: []tls.SignatureScheme{tls.PKCS1WithSHA256},
+ CipherSuites: []uint16{tls.TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256},
+ }
+ for _, tc := range []struct {
+ name string
+ hello *tls.ClientHelloInfo
+ want []byte
+ }{
+ {"ecdsa client", ecdsaHello(mirrorHost), ecdsaPEM},
+ {"rsa-only client", rsaOnly, rsaPEM},
+ } {
+ got, err := get(tc.hello)
+ if err != nil {
+ t.Fatalf("%s: %v", tc.name, err)
+ }
+ want, err := parseCertEntry(tc.want)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if !bytes.Equal(got.Certificate[0], want.Certificate[0]) {
+ t.Fatalf("%s was served the other entry", tc.name)
+ }
+ }
+}
+
+// ---- tls-alpn-01: the challenge autocert PREFERS ----------------------------
+
+func alpnHello(host string) *tls.ClientHelloInfo {
+ h := ecdsaHello(host)
+ h.SupportedProtos = []string{acme.ALPNProto}
+ return h
+}
+
+// autocert tries tls-alpn-01 first and only falls back to http-01 after a
+// whole failed order, so an N-edge deployment would burn a failed validation
+// on (N-1)/N of its orders if only the issuer could answer these.
+func TestMirrorAnswersAnotherEdgesALPNValidation(t *testing.T) {
+ s := release.NewDirStore(t.TempDir())
+ putBucket(t, s, mirrorHost+"+token", hostCertPEM(t, mirrorHost, time.Now().Add(24*time.Hour)))
+ m := newCertMirror(s, nil)
+ get, _ := gatedChain(t, m, autocert.DirCache(t.TempDir()), mirrorHost)
+ m.allowed = func(h string) bool { return h == mirrorHost }
+ // Deliberately NOT refreshed: a validation is live, and the poll is 30s.
+ if _, err := get(alpnHello(mirrorHost)); err != nil {
+ t.Fatalf("a non-issuer must answer another edge's tls-alpn-01 validation: %v", err)
+ }
+}
+
+// The SNI here is attacker-chosen, so the live lookup is bounded to hosts
+// this box actually serves.
+func TestMirrorALPNReadThroughOnlyForServedHosts(t *testing.T) {
+ s := &bucketGetCounter{Store: release.NewDirStore(t.TempDir())}
+ m := newCertMirror(s, nil)
+ get, _ := gatedChain(t, m, autocert.DirCache(t.TempDir()), mirrorHost)
+ m.allowed = func(string) bool { return false }
+ if _, err := get(alpnHello("stranger.example")); err == nil {
+ t.Fatal("there is no token cert for a host this box does not serve")
+ }
+ if n := s.gets(); n != 0 {
+ t.Fatalf("bucket gets = %d, want 0 for an unserved host", n)
+ }
+}
+
+// A token cert is NOT a certificate for the host: it is self-signed and only
+// meaningful to a validating CA. It must never leak into an ordinary
+// handshake.
+func TestMirrorNeverServesATokenCertToAnOrdinaryHello(t *testing.T) {
+ s := release.NewDirStore(t.TempDir())
+ putBucket(t, s, mirrorHost+"+token", hostCertPEM(t, mirrorHost, time.Now().Add(24*time.Hour)))
+ m := newCertMirror(s, nil)
+ get, _ := gatedChain(t, m, autocert.DirCache(t.TempDir()), mirrorHost)
+ m.allowed = func(string) bool { return true }
+ if err := m.Refresh(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ if _, err := get(ecdsaHello(mirrorHost)); !errors.Is(err, errNotIssuer) {
+ t.Fatalf("an ordinary hello must not be served the tls-alpn-01 token cert: %v", err)
+ }
+}
+
+// The wildcard stays in front of everything: a covered name never reaches the
+// mirror at all (this is the composition RunWithActivatorReady builds).
+func TestWildcardWinsAheadOfTheMirror(t *testing.T) {
+ const parent = "bab.oncarlos.com"
+ s := release.NewDirStore(t.TempDir())
+ putBucket(t, s, mirrorHost, hostCertPEM(t, mirrorHost, time.Now().Add(80*24*time.Hour)))
+ m := newCertMirror(s, nil)
+ inner, _ := gatedChain(t, m, autocert.DirCache(t.TempDir()), mirrorHost)
+ if err := m.Refresh(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ w := newWildcardManager([]string{parent}, "", func() []string { return nil }, nil, "", autocert.DirCache(t.TempDir()), &fakeDNS{})
+ wildcard, _ := selfSignedWildcard(t, parent, time.Now().Add(80*24*time.Hour))
+ w.refresh()
+ w.mu.Lock()
+ w.certs[parent] = wildcard
+ w.mu.Unlock()
+
+ got, err := wrapGetCertificate(w, inner)(ecdsaHello(mirrorHost))
+ if err != nil {
+ t.Fatal(err)
+ }
+ if got != wildcard {
+ t.Fatal("a wildcard-covered name must be served the wildcard, not the mirror's per-host cert")
+ }
+}
+
+// ---- R2: challenges, in front of autocert's own handler ---------------------
+
+const mirrorTestToken = "0123456789abcdefghij0123456789abcdefghijkl"
+
+func challengeRequest(t *testing.T, h http.Handler, token string) *httptest.ResponseRecorder {
+ t.Helper()
+ r := httptest.NewRequest("GET", "http://"+mirrorHost+"/.well-known/acme-challenge/"+token, nil)
+ w := httptest.NewRecorder()
+ h.ServeHTTP(w, r)
+ return w
+}
+
+// gatedHTTPHandler is the :80 chain of a non-issuer: the mirror's handler in
+// front of autocert's, which 403s every challenge on this box.
+func gatedHTTPHandler(t *testing.T, m *certMirror) http.Handler {
+ t.Helper()
+ e := testIssuerEdge(mirrorHost)
+ e.issuer = func() (context.Context, bool) { return nil, false }
+ mgr := offlineManager(t, autocert.DirCache(t.TempDir()), e.issuerHostPolicy())
+ fallback := http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+ w.Header().Set("X-Fallback", "yes")
+ w.WriteHeader(http.StatusTeapot)
+ })
+ return m.challengeHandler(mgr.HTTPHandler(fallback))
+}
+
+// autocert's HTTPHandler runs its HostPolicy — the issuance gate — before it
+// ever looks a token up, so a non-issuer 403s a challenge it is holding the
+// answer to. The mirror answers first.
+func TestChallengeHandlerAnswersInFrontOfAutocert(t *testing.T) {
+ s := release.NewDirStore(t.TempDir())
+ putBucket(t, s, mirrorTestToken+"+http-01", []byte("keyauth"))
+ m := newCertMirror(s, nil)
+ if err := m.Refresh(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ got := challengeRequest(t, gatedHTTPHandler(t, m), mirrorTestToken)
+ if got.Code != 200 || got.Body.String() != "keyauth" {
+ t.Fatalf("challenge = %d %q, want 200 \"keyauth\"", got.Code, got.Body.String())
+ }
+}
+
+// Everything that is not a challenge path is somebody else's request.
+func TestChallengeHandlerDelegatesEverythingElse(t *testing.T) {
+ m := newCertMirror(release.NewDirStore(t.TempDir()), nil)
+ r := httptest.NewRequest("GET", "http://"+mirrorHost+"/whatever", nil)
+ w := httptest.NewRecorder()
+ gatedHTTPHandler(t, m).ServeHTTP(w, r)
+ if w.Header().Get("X-Fallback") != "yes" {
+ t.Fatalf("non-challenge request = %d, want the fallback handler", w.Code)
+ }
+}
+
+// A live validation is faster than a 30s poll: the token was written to the
+// bucket a second ago and Accept() has already been sent. A miss reads
+// through — once, then from a short-lived cache.
+func TestChallengeReadsThroughForLiveTokens(t *testing.T) {
+ s := release.NewDirStore(t.TempDir())
+ putBucket(t, s, mirrorTestToken+"+http-01", []byte("keyauth"))
+ counting := &bucketGetCounter{Store: s}
+ m := newCertMirror(counting, nil) // deliberately never refreshed
+ h := gatedHTTPHandler(t, m)
+ for range 3 {
+ got := challengeRequest(t, h, mirrorTestToken)
+ if got.Code != 200 || got.Body.String() != "keyauth" {
+ t.Fatalf("read-through challenge = %d %q", got.Code, got.Body.String())
+ }
+ }
+ if n := counting.gets(); n != 1 {
+ t.Fatalf("bucket gets = %d, want 1 — a live token is cached briefly after the first read", n)
+ }
+}
+
+// A miss is negative-cached: a scanner walking /.well-known/acme-challenge/
+// must not turn into one bucket GET per request.
+func TestChallengeReadThroughNegativeCachesMisses(t *testing.T) {
+ counting := &bucketGetCounter{Store: release.NewDirStore(t.TempDir())}
+ m := newCertMirror(counting, nil)
+ h := gatedHTTPHandler(t, m)
+ for range 5 {
+ if got := challengeRequest(t, h, mirrorTestToken); got.Code == 200 {
+ t.Fatal("there is no such token; the handler must not answer 200")
+ }
+ }
+ if n := counting.gets(); n != 1 {
+ t.Fatalf("bucket gets = %d, want 1 — misses are negative-cached", n)
+ }
+}
+
+// Only names shaped like an ACME token are ever fetched: the read-through is
+// a bucket read driven by an unauthenticated URL path.
+func TestChallengeReadThroughRefusesNonTokenPaths(t *testing.T) {
+ counting := &bucketGetCounter{Store: release.NewDirStore(t.TempDir())}
+ m := newCertMirror(counting, nil)
+ h := gatedHTTPHandler(t, m)
+ for _, token := range []string{"short", "../../control/serial", "wildcard-account.key", strings.Repeat("a", 300)} {
+ if got := challengeRequest(t, h, token); got.Code == 200 {
+ t.Fatalf("%q answered 200", token)
+ }
+ }
+ if n := counting.gets(); n != 0 {
+ t.Fatalf("bucket gets = %d, want 0 — a name that cannot be an ACME token is never fetched", n)
+ }
+}
+
+// Distinct token names defeat the negative cache, so the read-through carries
+// its own budget: a flood degrades to "no read-through", never to a bucket
+// GET per request.
+func TestChallengeReadThroughIsRateLimited(t *testing.T) {
+ counting := &bucketGetCounter{Store: release.NewDirStore(t.TempDir())}
+ m := newCertMirror(counting, nil)
+ h := gatedHTTPHandler(t, m)
+ for i := range 200 {
+ challengeRequest(t, h, fmt.Sprintf("%s%03d", strings.Repeat("z", 39), i))
+ }
+ if n := counting.gets(); n > readThroughBurst {
+ t.Fatalf("bucket gets = %d under a flood, want <= %d", n, readThroughBurst)
+ }
+ if counting.gets() == 0 {
+ t.Fatal("test bug: the read-through never ran at all")
+ }
+}
+
+// bucketGetCounter counts Gets.
+type bucketGetCounter struct {
+ release.Store
+ mu sync.Mutex
+ n int
+}
+
+func (c *bucketGetCounter) Get(ctx context.Context, key string) ([]byte, string, error) {
+ c.mu.Lock()
+ c.n++
+ c.mu.Unlock()
+ return c.Store.Get(ctx, key)
+}
+
+func (c *bucketGetCounter) gets() int {
+ c.mu.Lock()
+ defer c.mu.Unlock()
+ return c.n
+}
+
+// slowGetStore blocks every Get until release is closed.
+type slowGetStore struct {
+ release.Store
+ release chan struct{}
+ mu sync.Mutex
+ n int
+}
+
+func (s *slowGetStore) Get(ctx context.Context, key string) ([]byte, string, error) {
+ s.mu.Lock()
+ s.n++
+ s.mu.Unlock()
+ <-s.release
+ return s.Store.Get(ctx, key)
+}
+
+func (s *slowGetStore) gets() int {
+ s.mu.Lock()
+ defer s.mu.Unlock()
+ return s.n
+}
+
+// A CA opens several connections for one validation. They share one bucket
+// read: the slow one is in flight, the rest wait on it.
+func TestChallengeReadThroughIsSingleFlighted(t *testing.T) {
+ inner := release.NewDirStore(t.TempDir())
+ putBucket(t, inner, mirrorTestToken+"+http-01", []byte("keyauth"))
+ s := &slowGetStore{Store: inner, release: make(chan struct{})}
+ m := newCertMirror(s, nil)
+ h := gatedHTTPHandler(t, m)
+
+ var wg sync.WaitGroup
+ got := make([]int, 6)
+ for i := range got {
+ wg.Add(1)
+ go func() {
+ defer wg.Done()
+ got[i] = challengeRequest(t, h, mirrorTestToken).Code
+ }()
+ }
+ // Let the waiters pile up behind the leader, then let it through.
+ time.Sleep(50 * time.Millisecond)
+ close(s.release)
+ wg.Wait()
+
+ for i, code := range got {
+ if code != 200 {
+ t.Fatalf("request %d = %d, want 200", i, code)
+ }
+ }
+ if n := s.gets(); n != 1 {
+ t.Fatalf("bucket gets = %d, want 1 — concurrent lookups share one read", n)
+ }
+}
+
+// The loop is the whole delivery mechanism: it must pick material up without
+// anybody asking, keep going after a failure, and stop when told.
+func TestCertMirrorLoopPollsAndStops(t *testing.T) {
+ prev := certMirrorInterval
+ certMirrorInterval = 5 * time.Millisecond
+ t.Cleanup(func() { certMirrorInterval = prev })
+ log.SetOutput(io.Discard)
+ t.Cleanup(func() { log.SetOutput(os.Stderr) })
+
+ s := &listFailingStore{Store: release.NewDirStore(t.TempDir())}
+ s.setFail(true) // the loop starts against a broken bucket
+ m := newCertMirror(s, nil)
+ stop := make(chan struct{})
+ done := make(chan struct{})
+ go func() {
+ m.run(stop)
+ close(done)
+ }()
+
+ putBucket(t, s.Store, "example.com", []byte("PEM"))
+ s.setFail(false)
+ deadline := time.Now().Add(2 * time.Second)
+ for {
+ if _, ok := m.Cert("example.com"); ok {
+ break
+ }
+ if time.Now().After(deadline) {
+ t.Fatal("the loop never picked the cert up")
+ }
+ time.Sleep(2 * time.Millisecond)
+ }
+ close(stop)
+ select {
+ case <-done:
+ case <-time.After(2 * time.Second):
+ t.Fatal("the loop did not stop")
+ }
+}
+
+// autocert arms its renewal timers LAZILY, from GetCertificate: m.cert()
+// loads the cache and calls startRenew (autocert.go:464), and nothing else
+// ever does. A mirror that short-circuited every hello would therefore
+// silently stop an ISSUER from renewing anything it had not ordered in this
+// process — the certs would simply expire in 90 days.
+//
+// So the short-circuit is for the GATED case only, which is the case it
+// exists for: an ungated box's autocert can serve everything the mirror holds
+// anyway, because syncLocal has already put it in its cache.
+func TestMirrorLetsAnIssuersAutocertSeeTheHello(t *testing.T) {
+ s := release.NewDirStore(t.TempDir())
+ putBucket(t, s, mirrorHost, hostCertPEM(t, mirrorHost, time.Now().Add(80*24*time.Hour)))
+
+ local := autocert.DirCache(t.TempDir())
+ counting := &countingCache{Cache: local, gets: map[string]int{}}
+ m := newCertMirror(s, local)
+ m.gated = func() bool { return false } // this box IS the issuer
+ if err := m.Refresh(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ e := testIssuerEdge(mirrorHost) // ungated host policy
+ get := m.wrapGetCertificate(offlineManager(t, counting, e.issuerHostPolicy()).GetCertificate)
+ if _, err := get(ecdsaHello(mirrorHost)); err != nil {
+ t.Fatalf("the issuer must serve the mirrored cert (through autocert): %v", err)
+ }
+ if n := counting.count(mirrorHost); n == 0 {
+ t.Fatal("autocert never saw the hello — its renewal timer for this host is never armed")
+ }
+}
+
+// The mirror is still there behind autocert on an ungated box: a name
+// autocert cannot serve (and, being offline, cannot order) is rescued.
+func TestMirrorStillRescuesAnUngatedBox(t *testing.T) {
+ s := release.NewDirStore(t.TempDir())
+ putBucket(t, s, mirrorHost, hostCertPEM(t, mirrorHost, time.Now().Add(80*24*time.Hour)))
+ m := newCertMirror(s, nil)
+ m.gated = func() bool { return false }
+ if err := m.Refresh(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ e := testIssuerEdge(mirrorHost)
+ get := m.wrapGetCertificate(offlineManager(t, autocert.DirCache(t.TempDir()), e.issuerHostPolicy()).GetCertificate)
+ if _, err := get(ecdsaHello(mirrorHost)); err != nil {
+ t.Fatalf("an ungated box whose order fails must still fall back to the mirror: %v", err)
+ }
+}
+
+// A mirror with no gate wired (every test, and any future caller that
+// forgets) behaves as though this box were gated: it serves, which is always
+// safe, rather than assuming an issuance authority it was never given.
+func TestMirrorWithNoGateDefaultsToServingFirst(t *testing.T) {
+ s := release.NewDirStore(t.TempDir())
+ putBucket(t, s, mirrorHost, hostCertPEM(t, mirrorHost, time.Now().Add(80*24*time.Hour)))
+ m := newCertMirror(s, nil) // no gate
+ if err := m.Refresh(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ local := autocert.DirCache(t.TempDir())
+ if err := local.Put(context.Background(), mirrorHost, hostCertPEM(t, mirrorHost, time.Now().Add(80*24*time.Hour))); err != nil {
+ t.Fatal(err)
+ }
+ counting := &countingCache{Cache: local, gets: map[string]int{}}
+ e := testIssuerEdge(mirrorHost) // ungated host policy: autocert WOULD serve
+ get := m.wrapGetCertificate(offlineManager(t, counting, e.issuerHostPolicy()).GetCertificate)
+ if _, err := get(ecdsaHello(mirrorHost)); err != nil {
+ t.Fatal(err)
+ }
+ if n := counting.count(mirrorHost); n != 0 {
+ t.Fatalf("autocert was consulted %d time(s) — with no gate the mirror answers first", n)
+ }
+}
diff --git a/internal/edge/edge.go b/internal/edge/edge.go
index 5574f1701..29c0a6568 100644
--- a/internal/edge/edge.go
+++ b/internal/edge/edge.go
@@ -1788,6 +1788,11 @@ func RunWithActivatorReady(args []string, act Activator, cache *Cache, ready fun
if err != nil {
log.Fatalf("edge: %v", err)
}
+ // The RAW local cache, kept before the publishing wrapper goes on. The
+ // mirror writes INWARDS through this handle and reads it as its last
+ // serving resort; going through the wrapper instead would have the issuer
+ // re-publishing its own material to the bucket on every poll.
+ localCerts := certCache
// Publish every issuance artifact to the control bucket, and gate
// issuance on being the box that may write there (certsync.go): per-host
// certs, the ACME account keys, HTTP-01 tokens, and — through the same
@@ -1796,6 +1801,23 @@ func RunWithActivatorReady(args []string, act Activator, cache *Cache, ready fun
// serve from, so it stays ungated and keeps issuing for itself.
certCache = e.armCertIssuance(certCache)
+ // The other side of that decision (certsync.go): the mirror of
+ // control/certs/, which is what makes the gate safe to shut. nil when
+ // there is no deployment store — armCertIssuance has then cleared the
+ // gate too, and this box issues and serves entirely for itself.
+ mirror := e.newCertMirror(localCerts)
+ if mirror != nil {
+ // One synchronous pass before anything listens: a box that has just
+ // booted must not spend its first 30 seconds handshaking with nothing,
+ // and the wildcard manager's first pass (below) reads what this lands
+ // in the local cache.
+ ctx, cancel := context.WithTimeout(context.Background(), certMirrorBootTimeout)
+ if err := mirror.Refresh(ctx); err != nil {
+ log.Printf("edge: first cert mirror refresh failed: %v — serving from this box's own cache until a later poll lands", err)
+ }
+ cancel()
+ }
+
m := &autocert.Manager{
Prompt: autocert.AcceptTOS,
Cache: certCache,
@@ -1805,28 +1827,54 @@ func RunWithActivatorReady(args []string, act Activator, cache *Cache, ready fun
HostPolicy: e.issuerHostPolicy(),
}
tlsCfg := m.TLSConfig()
+ // The handshake chain, built inside out: autocert, then the mirror (which
+ // also carries the direct local-cache last resort behind it), then the
+ // wildcard. See certMirror.wrapGetCertificate for why the mirror sits in
+ // FRONT of autocert rather than behind it.
+ getCert := tlsCfg.GetCertificate
+ if mirror != nil {
+ getCert = mirror.wrapGetCertificate(getCert)
+ }
// Wildcard certs (wildcard.go, ACME DNS-01): covered names — one label
// under a per-account or configured parent — are served the wildcard;
// everything else (custom domains, canary hosts, parents not yet
// obtained) stays on autocert's per-host HTTP-01 above. nil = not
// configured = exactly the pre-wildcard edge.
if w := newWildcardFromEnv(e, certCache); w != nil {
- tlsCfg.GetCertificate = wrapGetCertificate(w, tlsCfg.GetCertificate)
+ getCert = wrapGetCertificate(w, getCert)
e.OnReload(w.refresh)
+ if mirror != nil {
+ // A wildcard the ISSUER minted reaches this box's handshakes on the
+ // next poll rather than on this manager's 12h tick.
+ mirror.wake = w.refresh
+ }
w.refresh()
go w.run()
log.Printf("edge: wildcard certs on (static %v, apps domain %q, delegation zone %q)",
splitDomains(os.Getenv("CARLOS_WILDCARD_DOMAINS")), os.Getenv("PLATFORM_APPS_DOMAIN"), w.delegationZone)
}
+ tlsCfg.GetCertificate = getCert
+ if mirror != nil {
+ // Started last: every field the loop reads (wake above) is in place.
+ go mirror.run(make(chan struct{})) // process-lifetime; never stopped
+ }
// The non-ACME :80 handler. See httpFallbackHandler's doc comment for the
// Allowed-vs-lookupUpstream split.
httpFallback := e.httpFallbackHandler()
+ // :80's ACME path, outermost first: the mirror answers any edge's HTTP-01
+ // challenge, autocert answers its own live orders, and everything else
+ // falls through to the redirect handler.
+ acmeAndFallback := m.HTTPHandler(httpFallback)
+ if mirror != nil {
+ acmeAndFallback = mirror.challengeHandler(acmeAndFallback)
+ }
+
// ReadHeaderTimeout guards the public front door against Slowloris-style
// slow-header connection exhaustion. No ReadTimeout/WriteTimeout — those
// would break long-lived proxied WebSockets.
- httpSrv := &http.Server{Addr: *httpAddr, Handler: m.HTTPHandler(httpFallback), ReadHeaderTimeout: 10 * time.Second}
+ httpSrv := &http.Server{Addr: *httpAddr, Handler: acmeAndFallback, ReadHeaderTimeout: 10 * time.Second}
httpsSrv := &http.Server{Addr: *httpsAddr, Handler: e, TLSConfig: tlsCfg, ReadHeaderTimeout: 10 * time.Second}
// :80 serves the ACME HTTP-01 challenge and redirects everything else.
diff --git a/internal/edge/wildcard_test.go b/internal/edge/wildcard_test.go
index eead55f42..17b90d41d 100644
--- a/internal/edge/wildcard_test.go
+++ b/internal/edge/wildcard_test.go
@@ -24,6 +24,13 @@ import (
// selfSignedWildcard mints a throwaway *.parent cert valid for the given
// window — enough for matching/renewal/PEM tests, no ACME involved.
func selfSignedWildcard(t *testing.T, parent string, notAfter time.Time) (*tls.Certificate, *ecdsa.PrivateKey) {
+ t.Helper()
+ return selfSignedLeaf(t, "*."+parent, notAfter)
+}
+
+// selfSignedLeaf mints a throwaway cert for one DNS name (a host, or a
+// "*.parent" wildcard) valid until notAfter.
+func selfSignedLeaf(t *testing.T, dnsName string, notAfter time.Time) (*tls.Certificate, *ecdsa.PrivateKey) {
t.Helper()
key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
if err != nil {
@@ -31,8 +38,8 @@ func selfSignedWildcard(t *testing.T, parent string, notAfter time.Time) (*tls.C
}
tmpl := &x509.Certificate{
SerialNumber: big.NewInt(1),
- Subject: pkix.Name{CommonName: "*." + parent},
- DNSNames: []string{"*." + parent},
+ Subject: pkix.Name{CommonName: dnsName},
+ DNSNames: []string{dnsName},
NotBefore: time.Now().Add(-time.Hour),
NotAfter: notAfter,
}