Custom-domain certs wait up to an hour after DNS points: order on the event, and say what's happening open
🤖 Filed from the patrix.cc setup on 2026-10-18, at Paul's request: we want certificates to issue as quickly as possible.
What happened
- 05:37 UTC:
carlos domains attach --app patrix patrix.cc(account bfb). DNS not yet published. - ~07:45: Paul publishes a root CNAME (Cloudflare-flattened) to
patrix.bfb.oncarlos.com, which resolves only to the Dublin edge, 99.81.104.219. - 07:47 to 07:58:
carlos domains listsaysissuing-cert/pointing, with the detail "usually a minute or two, and no action is needed". HTTPS fails with a TLS internal-error alert on every edge. - 07:58:59: Let's Encrypt cert issued (notBefore is backdated an hour, to 06:58:59). It served within minutes on three of the four edges. 16.60.78.128 still failed TLS a few minutes later.
- After the cert was serving,
domains liststill saidissuing-certuntil the next edge report.
Nothing failed. It was simply the hourly cadence, but it looked broken, and from the member side there was no way to tell the difference.
Why it takes up to an hour
internal/edge/certwarm.go:pass()returnswarmInterval(1h) in every path except the capped one, so the loop wakes hourly.- A host that isn't seen yet and doesn't
pointsHerewhen a pass runs is skipped without being recorded. So a domain whose DNS lands one minute after a pass waits about 59 minutes. - Only the issuer orders. Dublin answers
/.well-known/acme-challenge/withacme: not available here; Sydney (54.252.186.195) holds the lease. A hello reaching a non-issuer never triggers an order (#286 fixed this with the warm, at the warm's cadence). - The
issuing-certdetail ininternal/console/domainstate.gopromises "a minute or two", which the design can't currently deliver.
Asks, roughly in order of payoff
- Order on the event, not the clock. When a host enters the warm set, or its DNS reading flips to
pointing, wake the warm for that host now. For example: a channel from the domains reconcile and alias readings tocertWarm, the same waywarmCappedDelayalready short-circuits the sleep. Hosts that are seen but have no cert could also use a short poll (say 1 minute) for their first hour, falling back to hourly once they have a cert or a failure record. Rate-limit maths stays the same: still one order per host, gated bypointsHere, andwarmOrdersPerPassstill applies. - Let a non-issuer nudge the issuer. A hello for an exact deployment host with no cert, arriving at a non-issuer, could post a "please warm
<host>" to the issuer (backhaul or a control-bucket marker) instead of only failing closed. - Say what's actually happening. Have
domains listand the console show the warm's real state for anissuing-certhost: last attempt time, next attempt time, and the last error class from the warm record. Mark the reading's age too, soissuing-certdoesn't linger after the cert is live. Members can't readcontrol/cert-warm/themselves, so today the only answer to "is Let's Encrypt erroring?" is to wait three hours forfailed. - Check the stragglers. 16.60.78.128 wasn't serving the mirrored cert minutes after the other three were. That may just be mirror-poll cadence, but worth a look while in here.
Target: a cert within about a minute of DNS pointing, and an honest status line while waiting.