amadan
edge: the platform orders and renews custom-domain certs (#299 stage 3)
Per-host certs were minted lazily on the first TLS handshake. That is fine for a name people visit from the box that serves it, and wrong for a custom domain on a steering deployment, for two compounding reasons. The first visitor pays a multi-second ACME order — and if the customer's DNS only just landed, they are also the one who discovers it failed. Renewal is the worse half. autocert renews through per-host domainRenewal timers that live in process memory. They die on every fleet roll and re-arm only when a handshake reaches m.cert again. Under latency steering a custom domain's visitors may never land on the issuer, so after a roll NOTHING re-arms the timer and the cert quietly expires at day 90 — on a host that was serving perfectly all along. So the warm calls GetCertificate for every host in its set on EVERY pass, unconditionally. For a healthy cert that is one cache read plus the startRenew that re-arms the timer, and the re-arm is the entire point: a skip-if-fresh check would defeat it, because m.cert returns any unexpired cert and a cert 29 days from expiry is a cache hit. The set is "exact hosts we serve that are not under the apps domain and not wildcard-covered". That predicate is deliberately simpler than "aliases": first-route claims mint a real ROUTE, so an alias-only rule would never warm "carlos deploy -kind static -host <site>", the shipped headline flow. Excluding the apps domain is what keeps a first run from being a burst — canaries are three labels under it and no wildcard covers them. Wildcard coverage is tested BY SHAPE, not by asking certFor, which returns nil for a parent whose cert has not landed yet and would warm every app of a new account per-host in exactly that window. New orders are DNS-gated, and ALL resolved answers must be ours: a name mid-migration resolves both old and new, an any-match gate passes it, and validation then fails from the CA's vantage anyway — burning the shared account's failed-validation budget, which is the thing the gate exists to protect. Renewal re-arms are NOT gated: they cost the CA nothing, and gating them would strand the domain whose resolver is briefly sick. The synthetic hello advertises an ECDSA cipher suite, and that is load-bearing rather than decoration. autocert picks the key type from the hello; an empty one reads as RSA-only, so the warm would mint <host>+rsa while every browser asks for <host> — order spent, name uncovered, first visitor triggering a second issuance. Pinned against the repo's own port of autocert's chooser so the two cannot drift. Armed only with a deployment store, so an agent-less "carlos edge" never starts placing proactive orders on a hobbyist's box.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
4 files changed,
+716
−0
internal/edge/certwarm.go+376 −0internal/edge/certwarm_test.go+269 −0internal/edge/edge.go+39 −0internal/edge/homemap.go+32 −0
diff --git a/internal/edge/certwarm.go b/internal/edge/certwarm.go| new file mode 100644 |
| index 000000000..018cbab10 |
| --- /dev/null |
| +++ b/internal/edge/certwarm.go |
| @@ -0,0 +1,376 @@ |
| +// Cert warm (#299 stage 3): the platform orders and RENEWS certificates for |
| +// the custom domains it serves, instead of waiting for a visitor's handshake |
| +// to trigger the first issuance. |
| +// |
| +// Why this exists at all. Per-host certs are minted lazily by autocert on the |
| +// first TLS handshake for a name. That is fine for a name people visit from |
| +// the box that serves it, and wrong for a custom domain on a steering |
| +// deployment, for two compounding reasons: |
| +// |
| +// - The FIRST visitor pays a multi-second ACME order, and if the customer's |
| +// DNS only just landed, they are also the one who discovers it failed. |
| +// - RENEWAL is worse. autocert renews through per-host `domainRenewal` |
| +// timers that live in process memory. They die on every fleet roll and |
| +// re-arm only when a handshake reaches m.cert again. Under latency |
| +// steering a custom domain's visitors may never land on the issuer, so |
| +// after a roll NOTHING re-arms the timer and the cert quietly expires at |
| +// day 90 — on a host that was serving perfectly the whole time. |
| +// |
| +// So the warm calls GetCertificate for every host in its set on EVERY pass, |
| +// unconditionally. That is not wasteful and it is not a retry loop: for a |
| +// healthy cert it is one cache read plus the startRenew that re-arms the |
| +// timer, and the re-arm is the entire point. A skip-if-fresh check would |
| +// defeat it — autocert's m.cert returns any unexpired cached cert, so a cert |
| +// 29 days from expiry is a cache hit and no renewal is ever scheduled. |
| +package edge |
| + |
| +import ( |
| + "context" |
| + "crypto/tls" |
| + "errors" |
| + "log" |
| + "net" |
| + "sort" |
| + "strings" |
| + "sync" |
| + "time" |
| + |
| + "golang.org/x/crypto/acme" |
| +) |
| + |
| +const ( |
| + // warmInterval is the quiet-path cadence. Renewal is the job with the |
| + // real deadline (autocert renews at ~30 days remaining), so hourly is |
| + // far more often than strictly needed — the cost of a pass over a |
| + // healthy set is a cache read per host, and being early is free. |
| + warmInterval = time.Hour |
| + // warmBackoff is the per-host floor after a failed order. Matches the |
| + // wildcard manager's hourly retry. |
| + warmBackoff = time.Hour |
| + // warmRateLimitBackoff is the floor after the CA says rateLimited. |
| + // Deliberately much harder than warmBackoff: hourly retries INTO a rate |
| + // limit are what turn a transient limit into a sustained one, and every |
| + // host in the deployment shares the one ACME account's budget. |
| + warmRateLimitBackoff = 12 * time.Hour |
| + // warmOrdersPerPass bounds NEW orders per pass. Renewal re-arms are not |
| + // orders and are never capped — they are the reason the pass exists. |
| + // When the cap bites, pass() returns a short delay so a backlog drains |
| + // in minutes rather than trickling out over days (the wildcard pass() |
| + // precedent). |
| + warmOrdersPerPass = 10 |
| + // warmCappedDelay is how soon to come back with capped work remaining. |
| + warmCappedDelay = time.Minute |
| +) |
| + |
| +// certWarm proactively orders and renews per-host certificates. |
| +// |
| +// It is a sibling of wildcardManager, with three deliberate departures from |
| +// it, each of which is a bug if copied across: |
| +// |
| +// - No holding-context claim. The wildcard manager runs its own ACME |
| +// client, so it can hand the holding's context to the order. The warm's |
| +// trigger is autocert's GetCertificate, which takes NO context and builds |
| +// its own background one with a 5-minute timeout. So the gate is |
| +// re-checked BETWEEN orders instead, and a ≤5-minute overhang past a lost |
| +// holding is accepted — inside the double-holder envelope issuerHostPolicy |
| +// already documents. If the holding ends mid-order, the finishing |
| +// cachePut re-checks the gate and the cert lands local-only; the new |
| +// issuer re-orders it. Bounded waste, not a hazard. |
| +// - No cache-load-before-gate step. The wildcard manager loads cached certs |
| +// before standing down because it SERVES from its own map. The warm |
| +// serves nothing — per-host serving is the mirror plus autocert — so a |
| +// non-issuer scanning caches would be pure waste. It stands down at the |
| +// gate. |
| +// - The unconditional GetCertificate described in the package comment. |
| +type certWarm struct { |
| + // hosts is the warm set: exact hostnames this deployment serves that are |
| + // not under the apps domain and not covered by a wildcard. See |
| + // Edge.warmSet. |
| + hosts func() []string |
| + // getCert is autocert's per-host path (NOT the wildcard-wrapped one — a |
| + // wrapped call would be answered by the wildcard and order nothing). |
| + getCert func(*tls.ClientHelloInfo) (*tls.Certificate, error) |
| + // pointsHere reports whether host's DNS resolves to this deployment. |
| + // Ordering for a half-pointed name burns the shared account's |
| + // failed-validation budget; see dnsGate. |
| + pointsHere func(host string) bool |
| + // issuer gates ORDERING, exactly as it gates the wildcard manager's. |
| + issuer IssuerGate |
| + |
| + mu sync.Mutex |
| + // nextAttempt is the per-host backoff floor. |
| + nextAttempt map[string]time.Time |
| + // warmed is the set of hosts a cert has been obtained for at least once, |
| + // so a first order can be told from a renewal re-arm for the order cap |
| + // and for logging. |
| + warmed map[string]bool |
| + |
| + lastStandDownLog time.Time |
| +} |
| + |
| +func newCertWarm(hosts func() []string, getCert func(*tls.ClientHelloInfo) (*tls.Certificate, error), pointsHere func(string) bool, issuer IssuerGate) *certWarm { |
| + return &certWarm{ |
| + hosts: hosts, |
| + getCert: getCert, |
| + pointsHere: pointsHere, |
| + issuer: issuer, |
| + nextAttempt: map[string]time.Time{}, |
| + warmed: map[string]bool{}, |
| + } |
| +} |
| + |
| +// issuerGate is this warm's read of the deployment's gate, ungated default |
| +// applied (see IssuerGate, certsync.go). |
| +func (c *certWarm) issuerGate() (context.Context, bool) { |
| + if c.issuer == nil { |
| + return context.Background(), true |
| + } |
| + return c.issuer() |
| +} |
| + |
| +func (c *certWarm) run() { |
| + for { |
| + time.Sleep(c.pass()) |
| + } |
| +} |
| + |
| +// pass warms every host in the set once and returns how long to wait. |
| +func (c *certWarm) pass() time.Duration { |
| + hosts := c.hosts() |
| + if len(hosts) == 0 { |
| + return warmInterval |
| + } |
| + // Stand down before ANY work: unlike the wildcard manager there is no |
| + // cache to load first, so a non-issuer has nothing useful to do here. |
| + if _, ok := c.issuerGate(); !ok { |
| + c.logStandDown(len(hosts)) |
| + return warmInterval |
| + } |
| + orders, capped := 0, false |
| + for _, h := range hosts { |
| + now := time.Now() |
| + c.mu.Lock() |
| + until, backing := c.nextAttempt[h] |
| + seen := c.warmed[h] |
| + c.mu.Unlock() |
| + if backing && now.Before(until) { |
| + continue |
| + } |
| + // The gate is re-checked BETWEEN orders, not once per pass: a pass |
| + // over a large set can outlive a holding, and every order after the |
| + // holding ends is an order this box has no authority to place. |
| + if _, ok := c.issuerGate(); !ok { |
| + c.logStandDown(len(hosts) - orders) |
| + return warmInterval |
| + } |
| + // A host with no cert yet is a NEW order and is both capped and |
| + // DNS-gated. A host already warmed is a renewal re-arm: uncapped and |
| + // ungated, because it is a cache read that costs the CA nothing and |
| + // re-arms the timer that renewal depends on. Gating the re-arm on DNS |
| + // would strand exactly the domain whose resolver is briefly sick. |
| + if !seen { |
| + if orders >= warmOrdersPerPass { |
| + capped = true |
| + continue |
| + } |
| + if !c.pointsHere(h) { |
| + continue |
| + } |
| + orders++ |
| + } |
| + if err := c.warmOne(h); err != nil { |
| + c.fail(h, err) |
| + continue |
| + } |
| + c.mu.Lock() |
| + delete(c.nextAttempt, h) |
| + first := !c.warmed[h] |
| + c.warmed[h] = true |
| + c.mu.Unlock() |
| + if first { |
| + log.Printf("edge: cert warmed for %s", h) |
| + } |
| + } |
| + if capped { |
| + return warmCappedDelay |
| + } |
| + return warmInterval |
| +} |
| + |
| +// warmOne asks autocert for host's certificate, which mints it when absent |
| +// and re-arms its renewal timer when present. |
| +func (c *certWarm) warmOne(host string) error { |
| + _, err := c.getCert(warmHello(host)) |
| + return err |
| +} |
| + |
| +// fail arms the per-host backoff, harder when the CA is rate-limiting. |
| +func (c *certWarm) fail(host string, err error) { |
| + backoff := warmBackoff |
| + var ae *acme.Error |
| + if errors.As(err, &ae) && ae.ProblemType == "urn:ietf:params:acme:error:rateLimited" { |
| + backoff = warmRateLimitBackoff |
| + log.Printf("edge: cert warm for %s rate-limited by the CA — backing off %s (the whole deployment shares one ACME account)", host, backoff) |
| + } else { |
| + log.Printf("edge: cert warm for %s: %v — retrying in %s", host, err, backoff) |
| + } |
| + c.mu.Lock() |
| + c.nextAttempt[host] = time.Now().Add(backoff) |
| + c.mu.Unlock() |
| +} |
| + |
| +func (c *certWarm) logStandDown(hosts int) { |
| + c.mu.Lock() |
| + defer c.mu.Unlock() |
| + if time.Since(c.lastStandDownLog) < time.Hour { |
| + return |
| + } |
| + c.lastStandDownLog = time.Now() |
| + log.Printf("edge: cert warm standing down for %d host(s) — another box holds the issuer lease", hosts) |
| +} |
| + |
| +// warmHello builds the synthetic ClientHelloInfo the warm hands autocert. |
| +// |
| +// The CipherSuites entry is LOAD-BEARING and must not be dropped as |
| +// decoration. autocert picks the key type from the hello via supportsECDSA |
| +// (ported here as helloSupportsECDSA): with a hello carrying no signature |
| +// schemes, no curves and no cipher suites, all three checks fall through to |
| +// false, autocert chooses certKey{isRSA: true}, and the warm mints an RSA |
| +// cert under "<host>+rsa" — while every real browser asks for "<host>" and |
| +// gets nothing. The order would be spent, the name still uncovered, and the |
| +// first visitor would trigger a SECOND issuance. |
| +// |
| +// One ECDSA suite is enough: helloSupportsECDSA's first two checks skip on |
| +// nil, and its cipher-suite loop returns true on this suite. |
| +func warmHello(host string) *tls.ClientHelloInfo { |
| + return &tls.ClientHelloInfo{ |
| + ServerName: host, |
| + CipherSuites: []uint16{tls.TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256}, |
| + } |
| +} |
| + |
| +// warmSet is the set of hostnames the cert warm orders and renews for: |
| +// EXACT names this deployment serves that are not under the apps domain and |
| +// not covered by a wildcard. |
| +// |
| +// The predicate is deliberately simpler than "aliases" or "custom domains". |
| +// A custom domain is a foreign domain by definition — the attach door refuses |
| +// apps-domain hosts outright — so "not under the apps domain" covers attached |
| +// aliases and first-route claims alike, with no need to reason about which |
| +// registry table a name arrived through. That matters: first-route claims |
| +// (`carlos deploy -kind static -host <site>`) mint a real ROUTE, not an |
| +// alias, so an alias-only rule would silently never warm the shipped headline |
| +// flow. |
| +// |
| +// What it excludes, and why each one: |
| +// |
| +// - "*." rows — covered by a wildcard PEM, and autocert refuses the |
| +// wildcard label anyway. |
| +// - anything under the apps domain — canary hosts and platform hosts. This |
| +// is the exclusion that keeps a first run from being a burst: canaries are |
| +// three labels under the apps domain, no wildcard parent covers them, and |
| +// a naive "everything we serve" set would order a cert AND a CT-log entry |
| +// for every one of them. |
| +// - anything one label under a CONFIGURED wildcard parent, tested BY SHAPE |
| +// against currentParents() rather than by asking certFor. certFor returns |
| +// nil for a parent whose cert has not been obtained yet, so a shape-blind |
| +// test would warm every app of a new account per-host during exactly the |
| +// window before their wildcard lands. |
| +func (e *Edge) warmSet(appsDomain string, parents func() []string) []string { |
| + appsDomain = strings.ToLower(strings.TrimSpace(appsDomain)) |
| + covered := wildcardCoverer(parents) |
| + seen := map[string]bool{} |
| + var out []string |
| + consider := func(h string) { |
| + h = strings.ToLower(strings.TrimSpace(h)) |
| + if h == "" || seen[h] || strings.HasPrefix(h, "*.") { |
| + return |
| + } |
| + if appsDomain != "" && (h == appsDomain || strings.HasSuffix(h, "."+appsDomain)) { |
| + return |
| + } |
| + if covered(h) { |
| + return |
| + } |
| + seen[h] = true |
| + out = append(out, h) |
| + } |
| + for h := range e.current() { |
| + consider(h) |
| + } |
| + for _, h := range e.homes.Load().ServedHosts() { |
| + consider(h) |
| + } |
| + sort.Strings(out) |
| + return out |
| +} |
| + |
| +// wildcardCoverer reports whether a host is exactly one label under a |
| +// wildcard parent this box has configured or derived — by SHAPE, never by |
| +// whether the parent's cert exists yet (see warmSet). A nil parents func |
| +// covers nothing, which is the correct answer for a deployment with no |
| +// wildcard manager at all. |
| +func wildcardCoverer(parents func() []string) func(string) bool { |
| + return func(host string) bool { |
| + if parents == nil { |
| + return false |
| + } |
| + for _, p := range parents() { |
| + rest, ok := strings.CutSuffix(host, "."+p) |
| + if ok && rest != "" && !strings.Contains(rest, ".") { |
| + return true |
| + } |
| + } |
| + return false |
| + } |
| +} |
| + |
| +// dnsGate builds the "does this name point at us?" test the warm applies |
| +// before placing a NEW order. |
| +// |
| +// Let's Encrypt caps failed validations at 5 per account per hostname per |
| +// hour, and the ACME account is shared by every name this issuer mints. So a |
| +// half-configured customer domain that we order for anyway does not just fail |
| +// its own order — it spends budget every other tenant needs. The gate is what |
| +// keeps one customer's unfinished DNS from being everyone's problem. |
| +// |
| +// ALL resolved answers must be in the edge set, not merely one. A name in |
| +// mid-migration resolves to both the old host and ours; an any-match test |
| +// passes it, the order goes out, and validation fails from the CA's vantage |
| +// anyway — burning exactly the budget this gate exists to protect. Requiring |
| +// all of them means the gate only opens once the customer has actually |
| +// finished. |
| +// |
| +// An empty edge set fails CLOSED (no host points anywhere), matching the |
| +// domains pass's stand-down: a set we could not read is not evidence about |
| +// anybody's DNS, and the cost of waiting a pass is one hour of delay against |
| +// the cost of burning a shared rate limit. |
| +func dnsGate(resolve func(string) ([]string, error), edgeIPs func() []string) func(string) bool { |
| + return func(host string) bool { |
| + set := map[string]bool{} |
| + for _, ip := range edgeIPs() { |
| + set[ip] = true |
| + } |
| + if len(set) == 0 { |
| + return false |
| + } |
| + ips, err := resolve(host) |
| + if err != nil || len(ips) == 0 { |
| + return false |
| + } |
| + for _, ip := range ips { |
| + if !set[ip] { |
| + return false |
| + } |
| + } |
| + return true |
| + } |
| +} |
| + |
| +// warmResolver is the DNS gate's lookup: the system resolver, bounded so a |
| +// hung one costs this host its turn in the pass and not the pass itself. |
| +func warmResolver(host string) ([]string, error) { |
| + ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) |
| + defer cancel() |
| + return net.DefaultResolver.LookupHost(ctx, host) |
| +} |
diff --git a/internal/edge/certwarm_test.go b/internal/edge/certwarm_test.go| new file mode 100644 |
| index 000000000..95aa55b75 |
| --- /dev/null |
| +++ b/internal/edge/certwarm_test.go |
| @@ -0,0 +1,269 @@ |
| +package edge |
| + |
| +import ( |
| + "context" |
| + "crypto/tls" |
| + "errors" |
| + "strings" |
| + "sync" |
| + "testing" |
| + "time" |
| + |
| + "golang.org/x/crypto/acme" |
| +) |
| + |
| +// THE trap this whole file exists to avoid. autocert picks the key type from |
| +// the hello. A hello with no signature schemes, no curves and no cipher |
| +// suites reads as "no ECDSA", so autocert mints an RSA cert under |
| +// "<host>+rsa" while every browser asks for "<host>" — the order spent, the |
| +// name still uncovered, and the first real visitor triggering a second |
| +// issuance. |
| +// |
| +// This pins the warm's hello against the repo's own port of autocert's |
| +// chooser, so the two cannot drift. |
| +func TestWarmHelloAsksForAnECDSACert(t *testing.T) { |
| + if !helloSupportsECDSA(warmHello("app.example.com")) { |
| + t.Fatal("the warm's hello reads as RSA-only — autocert would mint <host>+rsa and no browser would ever be served it") |
| + } |
| + // The bare hello is the bug, kept here so the test documents WHY the |
| + // cipher-suite entry above is load-bearing rather than decorative. |
| + if helloSupportsECDSA(&tls.ClientHelloInfo{ServerName: "app.example.com"}) { |
| + t.Fatal("an empty hello now reads as ECDSA-capable — warmHello's cipher suite may no longer be needed, re-derive the trap before deleting it") |
| + } |
| +} |
| + |
| +func TestWarmHelloCarriesTheServerName(t *testing.T) { |
| + if got := warmHello("App.Example.com").ServerName; got != "App.Example.com" { |
| + t.Fatalf("ServerName = %q", got) |
| + } |
| +} |
| + |
| +// The warm set is "exact hosts we serve that are not under the apps domain |
| +// and not wildcard-covered". Each exclusion below is a separate bug if it |
| +// regresses. |
| +func TestWarmSetSelectsCustomDomainsOnly(t *testing.T) { |
| + e := &Edge{} |
| + e.upstreams.Store(&map[string]upstream{ |
| + "www.customer.test": {}, |
| + "shop.customer.test": {}, |
| + "app.acct.oncarlos.test": {}, // platform host — apps domain |
| + "c1.app.acct.oncarlos.test": {}, // canary — apps domain, 3 labels |
| + "*.catchall.test": {}, // catch-all — wildcard covers it |
| + "one.go.tito.test": {}, // one label under a configured parent |
| + "deep.one.go.tito.test": {}, // TWO labels under it: NOT covered |
| + }) |
| + parents := func() []string { return []string{"go.tito.test"} } |
| + |
| + got := e.warmSet("oncarlos.test", parents) |
| + want := []string{"deep.one.go.tito.test", "shop.customer.test", "www.customer.test"} |
| + if strings.Join(got, ",") != strings.Join(want, ",") { |
| + t.Fatalf("warm set = %v, want %v", got, want) |
| + } |
| +} |
| + |
| +// A parent whose cert has not been obtained yet must STILL exclude its |
| +// children. Testing coverage by asking certFor (which returns nil until the |
| +// wildcard lands) would warm every app of a new account per-host during |
| +// exactly the window before their wildcard arrives — the burst the shape |
| +// test exists to prevent. |
| +func TestWarmSetExcludesChildrenOfAnUnobtainedParent(t *testing.T) { |
| + e := &Edge{} |
| + e.upstreams.Store(&map[string]upstream{"new.acct.test": {}}) |
| + // currentParents lists the parent; no cert exists for it anywhere. |
| + if got := e.warmSet("oncarlos.test", func() []string { return []string{"acct.test"} }); len(got) != 0 { |
| + t.Fatalf("warm set = %v; a configured parent covers its children whether or not its cert exists yet", got) |
| + } |
| +} |
| + |
| +func TestWarmSetWithNoWildcardManagerCoversNothing(t *testing.T) { |
| + e := &Edge{} |
| + e.upstreams.Store(&map[string]upstream{"www.customer.test": {}}) |
| + if got := e.warmSet("oncarlos.test", nil); len(got) != 1 || got[0] != "www.customer.test" { |
| + t.Fatalf("warm set = %v, want the custom domain", got) |
| + } |
| +} |
| + |
| +// ALL answers must be in the edge set. A name mid-migration resolves to the |
| +// old host AND ours; an any-match gate passes it, the order goes out, and |
| +// validation fails from the CA's vantage anyway — burning the shared |
| +// account's failed-validation budget, which is exactly what the gate is for. |
| +func TestDNSGateRequiresEveryAnswerToBeOurs(t *testing.T) { |
| + edges := func() []string { return []string{"203.0.113.7", "198.51.100.9"} } |
| + for name, tc := range map[string]struct { |
| + ips []string |
| + err error |
| + want bool |
| + }{ |
| + "all ours": {ips: []string{"203.0.113.7", "198.51.100.9"}, want: true}, |
| + "one of ours": {ips: []string{"203.0.113.7"}, want: true}, |
| + "partly elsewhere": {ips: []string{"203.0.113.7", "192.0.2.1"}, want: false}, |
| + "wholly elsewhere": {ips: []string{"192.0.2.1"}, want: false}, |
| + "no answer": {ips: nil, want: false}, |
| + "resolver failed": {err: errors.New("SERVFAIL"), want: false}, |
| + } { |
| + gate := dnsGate(func(string) ([]string, error) { return tc.ips, tc.err }, edges) |
| + if got := gate("www.customer.test"); got != tc.want { |
| + t.Errorf("%s: gate = %v, want %v", name, got, tc.want) |
| + } |
| + } |
| +} |
| + |
| +// An unreadable edge set is not evidence about anybody's DNS. Failing closed |
| +// costs one pass of delay; failing open spends a shared rate limit. |
| +func TestDNSGateFailsClosedOnAnEmptyEdgeSet(t *testing.T) { |
| + gate := dnsGate( |
| + func(string) ([]string, error) { return []string{"203.0.113.7"}, nil }, |
| + func() []string { return nil }, |
| + ) |
| + if gate("www.customer.test") { |
| + t.Fatal("an empty edge set opened the gate") |
| + } |
| +} |
| + |
| +// The heart of the design: a host that already has a cert is asked for AGAIN |
| +// every pass. autocert renews through in-memory timers that die on every |
| +// fleet roll and re-arm only via a GetCertificate that reaches m.cert — so a |
| +// skip-if-fresh check would silently stop renewing every custom domain whose |
| +// visitors never land on the issuer. |
| +func TestWarmReAsksForAlreadyWarmHostsEveryPass(t *testing.T) { |
| + var mu sync.Mutex |
| + calls := map[string]int{} |
| + w := newCertWarm( |
| + func() []string { return []string{"www.customer.test"} }, |
| + func(h *tls.ClientHelloInfo) (*tls.Certificate, error) { |
| + mu.Lock() |
| + defer mu.Unlock() |
| + calls[h.ServerName]++ |
| + return &tls.Certificate{}, nil |
| + }, |
| + func(string) bool { return true }, |
| + nil, |
| + ) |
| + for i := 0; i < 3; i++ { |
| + w.pass() |
| + } |
| + if calls["www.customer.test"] != 3 { |
| + t.Fatalf("GetCertificate calls = %d, want one per pass — the renewal re-arm depends on it", calls["www.customer.test"]) |
| + } |
| +} |
| + |
| +// A renewal re-arm is NOT gated on DNS. It is a cache read that costs the CA |
| +// nothing, and gating it would strand exactly the domain whose resolver is |
| +// briefly sick — dropping its renewal on the floor for as long as the |
| +// resolver stays unhappy. |
| +func TestWarmRenewalReArmIgnoresTheDNSGate(t *testing.T) { |
| + calls := 0 |
| + pointing := true |
| + w := newCertWarm( |
| + func() []string { return []string{"www.customer.test"} }, |
| + func(*tls.ClientHelloInfo) (*tls.Certificate, error) { calls++; return &tls.Certificate{}, nil }, |
| + func(string) bool { return pointing }, |
| + nil, |
| + ) |
| + w.pass() // first order: gate open, host becomes warmed |
| + pointing = false |
| + w.pass() // DNS now fails; the re-arm must still happen |
| + if calls != 2 { |
| + t.Fatalf("calls = %d, want 2 — a warmed host's re-arm is not DNS-gated", calls) |
| + } |
| +} |
| + |
| +// A never-warmed host IS DNS-gated: ordering for a half-pointed name spends |
| +// the shared account's failed-validation budget. |
| +func TestWarmDoesNotOrderForAHostThatDoesNotPointHere(t *testing.T) { |
| + calls := 0 |
| + w := newCertWarm( |
| + func() []string { return []string{"www.customer.test"} }, |
| + func(*tls.ClientHelloInfo) (*tls.Certificate, error) { calls++; return &tls.Certificate{}, nil }, |
| + func(string) bool { return false }, |
| + nil, |
| + ) |
| + w.pass() |
| + if calls != 0 { |
| + t.Fatalf("calls = %d, want 0 — the DNS gate must hold the first order", calls) |
| + } |
| +} |
| + |
| +// A non-issuer does no work at all: unlike the wildcard manager there is no |
| +// cache to load first, because the warm serves nothing. |
| +func TestWarmStandsDownWhenNotTheIssuer(t *testing.T) { |
| + calls := 0 |
| + w := newCertWarm( |
| + func() []string { return []string{"www.customer.test"} }, |
| + func(*tls.ClientHelloInfo) (*tls.Certificate, error) { calls++; return &tls.Certificate{}, nil }, |
| + func(string) bool { return true }, |
| + func() (context.Context, bool) { return nil, false }, |
| + ) |
| + if d := w.pass(); d != warmInterval { |
| + t.Fatalf("delay = %s", d) |
| + } |
| + if calls != 0 { |
| + t.Fatalf("a non-issuer placed %d call(s)", calls) |
| + } |
| +} |
| + |
| +// A rate limit must back off much harder than an ordinary failure: hourly |
| +// retries INTO a rate limit are what turn a transient limit into a sustained |
| +// one, and every host shares the one ACME account's budget. |
| +func TestWarmBacksOffHarderOnARateLimit(t *testing.T) { |
| + rateLimited := &acme.Error{ProblemType: "urn:ietf:params:acme:error:rateLimited"} |
| + w := newCertWarm(nil, nil, nil, nil) |
| + w.fail("a.test", rateLimited) |
| + w.fail("b.test", errors.New("connection refused")) |
| + w.mu.Lock() |
| + defer w.mu.Unlock() |
| + limited, ordinary := time.Until(w.nextAttempt["a.test"]), time.Until(w.nextAttempt["b.test"]) |
| + if limited <= ordinary { |
| + t.Fatalf("rate-limited backoff %s is not harder than the ordinary %s", limited, ordinary) |
| + } |
| +} |
| + |
| +// A failing host is not retried until its backoff expires — and a healthy |
| +// host in the same set is not held up by it. |
| +func TestWarmHonoursPerHostBackoff(t *testing.T) { |
| + calls := map[string]int{} |
| + w := newCertWarm( |
| + func() []string { return []string{"bad.test", "good.test"} }, |
| + func(h *tls.ClientHelloInfo) (*tls.Certificate, error) { |
| + calls[h.ServerName]++ |
| + if h.ServerName == "bad.test" { |
| + return nil, errors.New("order failed") |
| + } |
| + return &tls.Certificate{}, nil |
| + }, |
| + func(string) bool { return true }, |
| + nil, |
| + ) |
| + w.pass() |
| + w.pass() |
| + if calls["bad.test"] != 1 { |
| + t.Errorf("bad.test tried %d times, want 1 — backoff not honoured", calls["bad.test"]) |
| + } |
| + if calls["good.test"] != 2 { |
| + t.Errorf("good.test tried %d times, want 2 — a failing sibling must not stall it", calls["good.test"]) |
| + } |
| +} |
| + |
| +// New orders are capped per pass, and a capped pass comes back SOON so a |
| +// backlog drains in minutes rather than trickling out over days. Renewal |
| +// re-arms are never capped. |
| +func TestWarmCapsNewOrdersAndReturnsSoon(t *testing.T) { |
| + var hosts []string |
| + for i := 0; i < warmOrdersPerPass+5; i++ { |
| + hosts = append(hosts, string(rune('a'+i))+".customer.test") |
| + } |
| + calls := 0 |
| + w := newCertWarm( |
| + func() []string { return hosts }, |
| + func(*tls.ClientHelloInfo) (*tls.Certificate, error) { calls++; return &tls.Certificate{}, nil }, |
| + func(string) bool { return true }, |
| + nil, |
| + ) |
| + d := w.pass() |
| + if calls != warmOrdersPerPass { |
| + t.Fatalf("placed %d orders, want the cap of %d", calls, warmOrdersPerPass) |
| + } |
| + if d != warmCappedDelay { |
| + t.Fatalf("capped pass returned %s, want the short %s so the backlog drains", d, warmCappedDelay) |
| + } |
| +} |
diff --git a/internal/edge/edge.go b/internal/edge/edge.go| index c08c3f022..bb5cbf6bc 100644 |
| --- a/internal/edge/edge.go |
| +++ b/internal/edge/edge.go |
| @@ -47,6 +47,7 @@ import ( |
| "golang.org/x/crypto/acme/autocert" |
| + "github.com/carlosframework/platform/internal/edgesfile" |
| "github.com/carlosframework/platform/internal/provision" |
| "github.com/carlosframework/platform/internal/registry" |
| "github.com/carlosframework/platform/internal/release" |
| @@ -2415,12 +2416,18 @@ func RunWithActivatorReady(args []string, act Activator, cache *Cache, ready fun |
| if mirror != nil { |
| getCert = mirror.wrapGetCertificate(getCert) |
| } |
| + // The per-host path, captured before the wildcard wrap goes on top of it: |
| + // this is what the cert warm below calls. Calling the WRAPPED chain would |
| + // let the wildcard answer for a covered name and order nothing. |
| + perHostCert := getCert |
| // Wildcard certs (wildcard.go, ACME DNS-01): covered names — one label |
| // under a per-account or configured parent — are served the wildcard; |
| // everything else (custom domains, canary hosts, parents not yet |
| // obtained) stays on autocert's per-host HTTP-01 above. nil = not |
| // configured = exactly the pre-wildcard edge. |
| + var wm *wildcardManager |
| if w := newWildcardFromEnv(e, certCache); w != nil { |
| + wm = w |
| getCert = wrapGetCertificate(w, getCert) |
| e.OnReload(w.refresh) |
| attachWildcardToMirror(mirror, w) |
| @@ -2432,6 +2439,38 @@ func RunWithActivatorReady(args []string, act Activator, cache *Cache, ready fun |
| log.Printf("edge: wildcard certs on (static %v, apps domain %q, delegation zone %q)", |
| splitDomains(os.Getenv("CARLOS_WILDCARD_DOMAINS")), os.Getenv("PLATFORM_APPS_DOMAIN"), w.delegationZone) |
| } |
| + // Cert warm (#299 stage 3, certwarm.go): the platform ORDERS and RENEWS |
| + // custom domains' certs instead of waiting for a visitor's handshake. |
| + // |
| + // It calls perHostCert — the chain as it stood BEFORE the wildcard wrap — |
| + // because the warm's set already excludes every wildcard-covered name, |
| + // and a wrapped call would be answered by the wildcard and order nothing. |
| + // |
| + // Armed only with a deployment store: an agent-less `carlos edge` is |
| + // ungated (issuerGate nil means yes), and would otherwise start placing |
| + // proactive ACME orders on a hobbyist's box. |
| + if e.staticStore != nil { |
| + appsDomain := strings.ToLower(strings.TrimSpace(os.Getenv("PLATFORM_APPS_DOMAIN"))) |
| + parents := func() []string { return nil } |
| + if wm != nil { |
| + parents = wm.currentParents |
| + } |
| + envIPs := splitDomains(os.Getenv("PLATFORM_EDGE_IPS")) |
| + store := e.staticStore |
| + edgeIPs := func() []string { |
| + ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) |
| + defer cancel() |
| + return edgesfile.EdgeIPs(ctx, store, envIPs, time.Now()) |
| + } |
| + warm := newCertWarm( |
| + func() []string { return e.warmSet(appsDomain, parents) }, |
| + perHostCert, |
| + dnsGate(warmResolver, edgeIPs), |
| + e.issuerGate, |
| + ) |
| + go warm.run() |
| + log.Printf("edge: cert warm on (%s pass) — custom domains are ordered and renewed proactively, not on first handshake", warmInterval) |
| + } |
| tlsCfg.GetCertificate = getCert |
| if mirror != nil { |
| // Started last: every field the loop reads (wake above) is in place. |
diff --git a/internal/edge/homemap.go b/internal/edge/homemap.go| index aa675bfd8..4057747db 100644 |
| --- a/internal/edge/homemap.go |
| +++ b/internal/edge/homemap.go |
| @@ -427,6 +427,38 @@ func (m *homeMap) Hosts() []string { |
| return out |
| } |
| +// ServedHosts lists every EXACT hostname any box in this deployment homes — |
| +// routes and attached custom domains alike, with no class filter. |
| +// |
| +// Deliberately NOT Hosts(): that one applies parentInput, which keeps only |
| +// platform-class rows because its consumer derives wildcard PARENTS from |
| +// them. The cert warm's question is the opposite one — "what exact names does |
| +// this deployment answer for?" — and a custom domain is precisely the |
| +// non-platform row Hosts() drops. Filtering by class here would warm every |
| +// canary and no customer domain, which is backwards. |
| +// |
| +// Catch-all "*." keys are excluded: they are covered by a wildcard PEM, and |
| +// autocert refuses the wildcard label anyway (Allowed). Only dial-able |
| +// entries count — a cert-only wildcard row homes nothing. |
| +// |
| +// An unlabelled box (self == "") reports nothing, as Home does: an edge that |
| +// cannot tell its own rows from another box's must not answer at all. |
| +func (m *homeMap) ServedHosts() []string { |
| + if m == nil || m.self == "" { |
| + return nil |
| + } |
| + snap := m.snapshot() |
| + out := make([]string, 0, len(snap.hosts)) |
| + for h, e := range snap.hosts { |
| + if !e.dial || strings.HasPrefix(h, "*.") { |
| + continue |
| + } |
| + out = append(out, h) |
| + } |
| + sort.Strings(out) |
| + return out |
| +} |
| + |
| // setHostsWake installs the callback a changed host set fires (Refresh). |
| // Under the map's own lock: run() is already polling when the wildcard |
| // manager is wired in after it. |