amadan
The cert mirror does not scale: every box holds and re-fetches every cert in the deployment Idea open
Came out of Paul's question on the 1b48ed5 fix: what happens at thousands or millions of certs. The honest answer is that 1b48ed5 buys a constant factor and the design still has a hard ceiling.
Where it stands
certMirror.Refresh lists control/certs/*, fetches every object, and holds every body in memory. It runs on a 30s poll for the process lifetime, and once synchronously before either listener binds.
After 1b48ed5 the fetches are concurrent at 10, so the pass costs ceil(count/10) x latency instead of count x latency. At today's 101 certs on the AU box (~250ms to eu-west-1) that is ~4s instead of ~25s.
The ceilings
Time, at ~250ms per round trip:
101 certs (today) ~2s ~1,300 certs ~20s <- back over the update gate (see the boot-budget thread) 10,000 certs ~2.6 min 1,000,000 certs ~4.3 hours
Memory, at a few KB per PEM chain: ~4MB at 1,000, ~400MB at 100,000, ~4GB at 1,000,000 -- on boxes that are t4g.small with 2GB. It breaks on memory before it breaks on time.
Both ceilings are per box, and every box pays them independently for the whole deployment's certs, whether or not it serves those hosts.
Three fixes, in order of value
1. Fetch only what changed. release.Store.List returns []string and throws the metadata away, but S3's ListObjectsV2 already returns ETag and Size in the same response -- free, no extra round trip. Add a ListMeta, diff against the previous snapshot, and Get only what moved. Steady state becomes one round trip; boot with a warm disk cache becomes one round trip. Biggest win for the smallest change, and it removes the update-gate cliff for any realistic count.
2. Do not block listener bind on the pass. The synchronous boot refresh exists so the first handshake is not cold, but a cold handshake falls back to the local cache and autocert, while a stalled boot means not serving at all. Filed separately as the boot-budget thread; noting it here because (1) and (2) together are what retire the near-term risk.
3. Fetch per-SNI on demand, with an LRU. The endgame. A box needs the certs for the hosts it serves, not for the deployment. This is the only one that survives millions: nothing that holds every cert in memory can, and nothing that walks every cert on a timer can. It is a design change, not a patch -- read-through on handshake, negative caching, a bound on the resident set, and something sensible for the issuer box which legitimately wants breadth.
Note on what 1b48ed5 did and did not do
It made the fetches concurrent and bounded the worker pool, which fixed the live incident. It did not change what is fetched, how often, or what is retained. syncLocal still writes every cert to local disk on every pass too, which is its own N-per-poll cost nobody has measured.