carlos / platform Public

Custom-domain certs wait up to an hour after DNS points: order on the event, and say what's happening open

🤖 Filed from the patrix.cc setup on 2026-10-18, at Paul's request: we want certificates to issue as quickly as possible.

What happened

  • 05:37 UTC: carlos domains attach --app patrix patrix.cc (account bfb). DNS not yet published.
  • ~07:45: Paul publishes a root CNAME (Cloudflare-flattened) to patrix.bfb.oncarlos.com, which resolves only to the Dublin edge, 99.81.104.219.
  • 07:47 to 07:58: carlos domains list says issuing-cert / pointing, with the detail "usually a minute or two, and no action is needed". HTTPS fails with a TLS internal-error alert on every edge.
  • 07:58:59: Let's Encrypt cert issued (notBefore is backdated an hour, to 06:58:59). It served within minutes on three of the four edges. 16.60.78.128 still failed TLS a few minutes later.
  • After the cert was serving, domains list still said issuing-cert until the next edge report.

Nothing failed. It was simply the hourly cadence, but it looked broken, and from the member side there was no way to tell the difference.

Why it takes up to an hour

  • internal/edge/certwarm.go: pass() returns warmInterval (1h) in every path except the capped one, so the loop wakes hourly.
  • A host that isn't seen yet and doesn't pointsHere when a pass runs is skipped without being recorded. So a domain whose DNS lands one minute after a pass waits about 59 minutes.
  • Only the issuer orders. Dublin answers /.well-known/acme-challenge/ with acme: not available here; Sydney (54.252.186.195) holds the lease. A hello reaching a non-issuer never triggers an order (#286 fixed this with the warm, at the warm's cadence).
  • The issuing-cert detail in internal/console/domainstate.go promises "a minute or two", which the design can't currently deliver.

Asks, roughly in order of payoff

  1. Order on the event, not the clock. When a host enters the warm set, or its DNS reading flips to pointing, wake the warm for that host now. For example: a channel from the domains reconcile and alias readings to certWarm, the same way warmCappedDelay already short-circuits the sleep. Hosts that are seen but have no cert could also use a short poll (say 1 minute) for their first hour, falling back to hourly once they have a cert or a failure record. Rate-limit maths stays the same: still one order per host, gated by pointsHere, and warmOrdersPerPass still applies.
  2. Let a non-issuer nudge the issuer. A hello for an exact deployment host with no cert, arriving at a non-issuer, could post a "please warm <host>" to the issuer (backhaul or a control-bucket marker) instead of only failing closed.
  3. Say what's actually happening. Have domains list and the console show the warm's real state for an issuing-cert host: last attempt time, next attempt time, and the last error class from the warm record. Mark the reading's age too, so issuing-cert doesn't linger after the cert is live. Members can't read control/cert-warm/ themselves, so today the only answer to "is Let's Encrypt erroring?" is to wait three hours for failed.
  4. Check the stragglers. 16.60.78.128 wasn't serving the mirrored cert minutes after the other three were. That may just be mirror-poll cadence, but worth a look while in here.

Target: a cert within about a minute of DNS pointing, and an honest status line while waiting.