amadan
cert sweep: remove what is provably dead, and nothing else (#299 §3f, #243)
Two kinds of litter accumulate under control/certs/, for two different
reasons, and they need two different predicates.
Spent challenge tokens (#243): the issuer writes an HTTP-01 response or a
tls-alpn-01 token cert, validates, and deletes it — but the delete is
housekeeper-gated, so a stand-down or a crash mid-order leaves the token
behind permanently. A validation's whole life is seconds, so AGE is a
complete predicate. One hour, generous by two orders of magnitude against
autocert's five-minute order context, because deleting a token out from
under a live validation burns the order.
Certs for hosts nobody serves any more: a detached custom domain's cert is
never republished (ownsCertName refuses) and never deleted, so the object
lingers, the mirror serves it to matching SNI until it expires, and the
expired object then lingers forever.
THE PREDICATE FOR THE SECOND IS EXPIRY, NEVER DETACHMENT. The tempting
version of this sweep is catastrophic: "no claim exists for this host"
would delete every platform host's cert, every wildcard PEM and the
account keys — and a registry lag or a stale edges read would take a LIVE
host down fleet-wide through the mirror. Inference never destroys owner
property (reconcile.go §1c), and a certificate is the member's property.
So an entry goes only when its own leaf expired over a month ago. That
dissolves "the mirror serves it until expiry" by construction and makes a
wrong deletion harmless — the entry was already dead, and anything still
wanted is re-minted by the warm.
Three bounds the spec did not name:
- A parse floor. Nothing younger than sixty days is read at all.
autocert renews at ~30 days remaining on a 90-day cert, so a healthy
entry's stamp never gets that old — steady state is one listing and
zero reads instead of the per-object N+1 this would otherwise be. An
entry that IS that old is one nothing is renewing, which is exactly
the population worth reading.
- Anything unreadable is left alone. A sweep that deletes what it
cannot parse is a sweep that deletes tomorrow's format.
- Fifty deletions per pass. A bug in the predicate removes at most
fifty objects before somebody reads the log, and a real backlog
drains over days rather than in one burst against the bucket.
Account keys are excluded by name before anything else is considered, and
again by shape. They are the issuer's identity at the CA; deleting one
costs the deployment its ACME registration.
The gate check moves above the empty-warm-set return in pass(). Token
litter and dead certs exist in a deployment with no custom domains at all,
and the old ordering meant such a deployment never swept. A non-issuer
still sweeps nothing.
A store with no last-modified stamps does nothing rather than something
approximate: guessing an object's age from its contents is exactly the
inference this file refuses to make.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
4 files changed,
+514
−8
docs/superpowers/specs/2026-08-27-managed-custom-domains-design.md+32 −5internal/edge/certsweep.go+180 −0internal/edge/certsweep_test.go+286 −0internal/edge/certwarm.go+16 −3
CI cancelled — superseded by a newer push — run details
diff --git a/docs/superpowers/specs/2026-08-27-managed-custom-domains-design.md b/docs/superpowers/specs/2026-08-27-managed-custom-domains-design.md| index 0870d9ffa..9d1020a2e 100644 |
| --- a/docs/superpowers/specs/2026-08-27-managed-custom-domains-design.md |
| +++ b/docs/superpowers/specs/2026-08-27-managed-custom-domains-design.md |
| @@ -695,11 +695,38 @@ alternative to `issuing-cert`. The vocabulary honesty test now produces it |
| from an input the platform can actually produce, which is the same bar every |
| other state clears. |
| -§3f remains open. Consequence stated plainly: `control/certs/` entries for |
| -detached hosts linger until something else removes them. That is not a |
| -correctness or safety problem — §3f's whole ruling was that the sweep must |
| -key on EXPIRY rather than detachment, so nothing is being deleted wrongly by |
| -not running it. |
| +**§3f is built (2026-08-29)**, in `internal/edge/certsweep.go`, and it took |
| +#243's spent-token litter with it — one issuer-side pass over |
| +`control/certs/`, two predicates, because the two kinds of litter are not the |
| +same kind of thing: |
| + |
| +- **Spent challenge tokens** (#243) go on AGE alone. A validation's whole |
| + life is seconds and autocert's order context is five minutes, so the |
| + one-hour TTL is generous by two orders of magnitude — deliberately, since |
| + deleting a token out from under a live validation burns the order. |
| +- **Certificates** go only on their own leaf's expiry, plus a month. §3f's |
| + ruling is unchanged and is the reason the file opens with it: a sweep keyed |
| + "no claim exists for this host" would delete every platform host's cert, |
| + every wildcard PEM and the account keys, and a registry lag could take a |
| + live host down fleet-wide through the mirror. |
| + |
| +Three bounds the spec did not name: |
| + |
| +- **A parse floor.** Nothing younger than sixty days is read at all. autocert |
| + renews at ~30 days remaining on a 90-day certificate, so a healthy entry's |
| + object stamp never gets that old — which makes the steady-state cost one |
| + listing and zero reads, instead of the per-object N+1 this would otherwise |
| + be. An entry that IS that old is one nothing is renewing, which is exactly |
| + the population worth reading. |
| +- **Anything unreadable is left alone.** A sweep that deletes what it cannot |
| + parse is a sweep that deletes tomorrow's format. |
| +- **Fifty deletions per pass.** A bug in the predicate removes at most fifty |
| + objects before somebody reads the log. |
| + |
| +The gate check moved above the empty-warm-set return in `pass()`: token |
| +litter and dead certs exist in a deployment with no custom domains at all, |
| +and the old ordering would have meant such a deployment never swept. A |
| +non-issuer still sweeps nothing. |
| ### The shared reader is a READER, not the wire-shape owner |
diff --git a/internal/edge/certsweep.go b/internal/edge/certsweep.go| new file mode 100644 |
| index 000000000..858af2883 |
| --- /dev/null |
| +++ b/internal/edge/certsweep.go |
| @@ -0,0 +1,180 @@ |
| +// Cert sweep (#299 §3f, #243): the issuer removes cert-prefix entries that |
| +// are provably dead — and nothing else. |
| +// |
| +// Two kinds of litter accumulate under control/certs/, for two different |
| +// reasons, and they need two different predicates: |
| +// |
| +// - SPENT CHALLENGE TOKENS (#243). The issuer writes an HTTP-01 response |
| +// or a tls-alpn-01 token cert, validates, and deletes it — but the |
| +// delete is housekeeper-gated, so a stand-down or a crash mid-order |
| +// leaves the token behind permanently. A validation's whole life is |
| +// seconds, so AGE is a complete predicate here. |
| +// - CERTS FOR HOSTS NOBODY SERVES ANY MORE. A detached custom domain's |
| +// cert is never republished (ownsCertName refuses) and never deleted, |
| +// so the object lingers, the mirror serves it to matching SNI until it |
| +// expires, and the expired object then lingers forever. |
| +// |
| +// THE PREDICATE FOR THE SECOND IS EXPIRY, NEVER DETACHMENT, and the reason |
| +// is worth stating at length because the tempting version of this sweep is |
| +// catastrophic. A sweep keyed "no claim exists for this host" would delete |
| +// every non-claim-backed cert in the deployment — every platform host, every |
| +// wildcard PEM, the account keys — and worse, a registry lag or a stale |
| +// edges read would take a LIVE host's certs down fleet-wide through the |
| +// mirror. This repo's standing rule is that inference never destroys owner |
| +// property (internal/domains/reconcile.go §1c), and a certificate is the |
| +// member's property. |
| +// |
| +// So: an entry is deleted only when its own leaf expired long ago. That |
| +// dissolves the "the mirror serves it until expiry" concern by |
| +// construction, and it makes a wrong deletion harmless — the entry was |
| +// already dead, and anything still wanted is re-minted by the warm. |
| +// |
| +// Account keys are never touched, by name and again by shape: they are the |
| +// issuer's identity at the CA, and deleting one costs the deployment its |
| +// ACME registration. |
| +package edge |
| + |
| +import ( |
| + "context" |
| + "log" |
| + "strings" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/release" |
| +) |
| + |
| +const ( |
| + // tokenLitterTTL is how long a spent challenge token may linger before |
| + // the sweep takes it (#243). A validation completes in seconds and |
| + // autocert's own order context is five minutes, so an hour is generous |
| + // by two orders of magnitude — deliberately, because deleting a token |
| + // out from under a live validation burns the order. |
| + tokenLitterTTL = time.Hour |
| + // certSweepParseAge is the pre-filter: an object younger than this is |
| + // never even read, let alone deleted. |
| + // |
| + // It is what keeps the sweep from being the per-object N+1 it would |
| + // otherwise be. autocert renews at ~30 days remaining on a 90-day |
| + // certificate, so a HEALTHY entry is rewritten roughly every 60 days and |
| + // its object stamp never gets this old. An entry that does is one |
| + // nothing is renewing — which is exactly the population worth reading. |
| + // Steady-state cost: one listing, zero reads. |
| + certSweepParseAge = 60 * 24 * time.Hour |
| + // certSweepGrace is how long a leaf must have been EXPIRED before its |
| + // entry is removed. A month, because there is no hurry whatsoever: the |
| + // cost of keeping a dead certificate is bytes, and the cost of removing |
| + // a live one is an outage. |
| + certSweepGrace = 30 * 24 * time.Hour |
| + // certSweepInterval is how often the expiry half runs. The token half |
| + // runs every pass, because it needs no reads at all. |
| + certSweepInterval = 24 * time.Hour |
| + // certSweepDeletesPerPass bounds the damage any single pass can do, in |
| + // either direction: a bug here deletes at most this many objects before |
| + // somebody sees the log, and a genuine backlog drains over days rather |
| + // than in one burst against the bucket. |
| + certSweepDeletesPerPass = 50 |
| +) |
| + |
| +// sweep removes provably dead entries under the cert prefix. Issuer-gated by |
| +// its caller; see certWarm.pass. |
| +// |
| +// Never returns an error. Every failure is logged and skipped: a sweep is |
| +// housekeeping, and housekeeping that can fail a pass would take the warm |
| +// down with it. |
| +func (c *certWarm) sweep(ctx context.Context, now time.Time) { |
| + if c.store == nil { |
| + return |
| + } |
| + lister, ok := c.store.(release.ModLister) |
| + if !ok { |
| + // No stamps, no age, no sweep. Guessing at an object's age from its |
| + // contents is exactly the inference this file refuses to make. |
| + return |
| + } |
| + ctx, cancel := context.WithTimeout(ctx, certSweepTimeout) |
| + defer cancel() |
| + entries, err := lister.ListMod(ctx, certPrefix) |
| + if err != nil { |
| + log.Printf("edge: cert sweep: could not list %s: %v", certPrefix, err) |
| + return |
| + } |
| + expiryDue := now.Sub(c.lastExpirySweep) >= certSweepInterval |
| + deletes := 0 |
| + for _, e := range entries { |
| + if deletes >= certSweepDeletesPerPass { |
| + log.Printf("edge: cert sweep: stopping at %d deletions this pass — the rest go next pass", deletes) |
| + break |
| + } |
| + name, ok := strings.CutPrefix(e.Key, certPrefix) |
| + if !ok || name == "" { |
| + continue |
| + } |
| + // Account material, first and unconditionally. Never by age, never |
| + // by expiry, never at all. |
| + if accountKeyName(name) { |
| + continue |
| + } |
| + switch { |
| + case isChallengeToken(name): |
| + if now.Sub(e.Mod) < tokenLitterTTL { |
| + continue |
| + } |
| + if c.deleteEntry(ctx, e.Key, "spent challenge token") { |
| + deletes++ |
| + } |
| + case expiryDue && now.Sub(e.Mod) >= certSweepParseAge: |
| + if c.sweepExpired(ctx, e.Key, name, now) { |
| + deletes++ |
| + } |
| + } |
| + } |
| + if expiryDue { |
| + c.lastExpirySweep = now |
| + } |
| +} |
| + |
| +// certSweepTimeout budgets one sweep. It runs on the warm's own goroutine, |
| +// so a hung bucket must not hold the next warm pass hostage. |
| +const certSweepTimeout = 2 * time.Minute |
| + |
| +// isChallengeToken reports whether name is validation scratch rather than a |
| +// serving artifact — an HTTP-01 response body or a tls-alpn-01 token cert. |
| +func isChallengeToken(name string) bool { |
| + return strings.HasSuffix(name, httpTokenSuffix) || strings.HasSuffix(name, alpnTokenSuffix) |
| +} |
| + |
| +// sweepExpired reads one long-untouched entry and removes it only if its own |
| +// leaf says it is long dead. |
| +// |
| +// Everything that is not a confident yes is a no. An entry that will not |
| +// read, will not parse, or carries no leaf is LEFT ALONE — it may be a shape |
| +// a newer build publishes, and a sweep that deletes what it cannot |
| +// understand is a sweep that deletes tomorrow's format. |
| +func (c *certWarm) sweepExpired(ctx context.Context, key, name string, now time.Time) bool { |
| + raw, _, err := c.store.Get(ctx, key) |
| + if err != nil { |
| + log.Printf("edge: cert sweep: could not read %s: %v", key, err) |
| + return false |
| + } |
| + cert, err := parseCertEntry(raw) |
| + if err != nil || cert.Leaf == nil { |
| + log.Printf("edge: cert sweep: %s does not parse as a certificate, leaving it alone", key) |
| + return false |
| + } |
| + if now.Sub(cert.Leaf.NotAfter) < certSweepGrace { |
| + return false |
| + } |
| + return c.deleteEntry(ctx, key, "certificate expired "+cert.Leaf.NotAfter.UTC().Format("2006-01-02")+" for "+name) |
| +} |
| + |
| +// deleteEntry removes one object and says so. Every deletion under this |
| +// prefix is logged with its reason — there is no such thing as a quiet |
| +// delete of cert material. |
| +func (c *certWarm) deleteEntry(ctx context.Context, key, why string) bool { |
| + if err := c.store.Delete(ctx, key); err != nil { |
| + log.Printf("edge: cert sweep: could not delete %s: %v", key, err) |
| + return false |
| + } |
| + log.Printf("edge: cert sweep: removed %s — %s", key, why) |
| + return true |
| +} |
diff --git a/internal/edge/certsweep_test.go b/internal/edge/certsweep_test.go| new file mode 100644 |
| index 000000000..217487d94 |
| --- /dev/null |
| +++ b/internal/edge/certsweep_test.go |
| @@ -0,0 +1,286 @@ |
| +package edge |
| + |
| +import ( |
| + "context" |
| + "errors" |
| + "os" |
| + "testing" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/release" |
| +) |
| + |
| +func sweepWarm(t *testing.T, s release.Store) *certWarm { |
| + t.Helper() |
| + return newCertWarm(func() []string { return nil }, nil, nil, nil, s) |
| +} |
| + |
| +// seedCertEntry writes one cert-prefix object and backdates its stamp, which |
| +// is how an object that has been sitting in the bucket for months looks. |
| +func seedCertEntry(t *testing.T, dir string, s release.Store, name string, body []byte, age time.Duration) { |
| + t.Helper() |
| + if _, err := s.Put(context.Background(), certKey(name), body, ""); err != nil { |
| + t.Fatal(err) |
| + } |
| + when := time.Now().Add(-age) |
| + if err := os.Chtimes(dir+"/"+certKey(name), when, when); err != nil { |
| + t.Fatal(err) |
| + } |
| +} |
| + |
| +func certGone(t *testing.T, s release.Store, name string) bool { |
| + t.Helper() |
| + _, _, err := s.Get(context.Background(), certKey(name)) |
| + return errors.Is(err, os.ErrNotExist) |
| +} |
| + |
| +func expiredPEM(t *testing.T, host string, expiredAgo time.Duration) []byte { |
| + t.Helper() |
| + cert, key := selfSignedLeaf(t, host, time.Now().Add(-expiredAgo)) |
| + pem, err := encodeWildcardPEM(cert, key) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + return pem |
| +} |
| + |
| +// #243: a spent token whose gated delete never ran. Age is a complete |
| +// predicate — a validation's whole life is seconds. |
| +func TestSweepTakesSpentChallengeTokens(t *testing.T) { |
| + dir := t.TempDir() |
| + s := release.NewDirStore(dir) |
| + w := sweepWarm(t, s) |
| + seedCertEntry(t, dir, s, "abc123"+httpTokenSuffix, []byte("key-authorization"), 2*time.Hour) |
| + seedCertEntry(t, dir, s, "www.customer.test"+alpnTokenSuffix, []byte("whatever"), 2*time.Hour) |
| + // A token from a validation that may still be running must survive. |
| + seedCertEntry(t, dir, s, "fresh"+httpTokenSuffix, []byte("key-authorization"), time.Minute) |
| + |
| + w.sweep(context.Background(), time.Now()) |
| + |
| + if !certGone(t, s, "abc123"+httpTokenSuffix) { |
| + t.Error("a spent HTTP-01 token survived the sweep") |
| + } |
| + if !certGone(t, s, "www.customer.test"+alpnTokenSuffix) { |
| + t.Error("a spent tls-alpn-01 token cert survived the sweep") |
| + } |
| + if certGone(t, s, "fresh"+httpTokenSuffix) { |
| + t.Error("a token from a possibly-live validation was deleted — that burns the order") |
| + } |
| +} |
| + |
| +// §3f's whole ruling. The predicate is the leaf's own expiry and nothing |
| +// else: a cert for a host nobody serves any more goes ONLY once it is long |
| +// dead, and a current cert is untouchable however old its object is. |
| +func TestSweepDeletesOnlyLongExpiredCertificates(t *testing.T) { |
| + dir := t.TempDir() |
| + s := release.NewDirStore(dir) |
| + w := sweepWarm(t, s) |
| + old := certSweepParseAge + 24*time.Hour |
| + |
| + seedCertEntry(t, dir, s, "dead.customer.test", expiredPEM(t, "dead.customer.test", certSweepGrace+24*time.Hour), old) |
| + // Expired, but not long enough ago. Patience costs bytes; haste costs |
| + // an outage. |
| + seedCertEntry(t, dir, s, "recent.customer.test", expiredPEM(t, "recent.customer.test", time.Hour), old) |
| + // A CURRENT cert whose object happens to be ancient — a CA with long |
| + // certs, or a renewal that has simply not been needed. |
| + cert, key := selfSignedLeaf(t, "live.customer.test", time.Now().Add(365*24*time.Hour)) |
| + livePEM, err := encodeWildcardPEM(cert, key) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + seedCertEntry(t, dir, s, "live.customer.test", livePEM, old) |
| + |
| + w.sweep(context.Background(), time.Now()) |
| + |
| + if !certGone(t, s, "dead.customer.test") { |
| + t.Error("a certificate that expired a month ago survived") |
| + } |
| + if certGone(t, s, "recent.customer.test") { |
| + t.Error("a certificate that expired an hour ago was deleted — the grace period is not being applied") |
| + } |
| + if certGone(t, s, "live.customer.test") { |
| + t.Fatal("A CURRENT CERTIFICATE WAS DELETED. The predicate is expiry, never object age.") |
| + } |
| +} |
| + |
| +// The account keys are the issuer's identity at the CA. They are written |
| +// once and never rewritten, so they are ALWAYS older than the parse floor — |
| +// which is exactly why they are excluded by name before anything else is |
| +// considered. Deleting one costs the deployment its ACME registration. |
| +func TestSweepNeverTouchesAccountKeys(t *testing.T) { |
| + dir := t.TempDir() |
| + s := release.NewDirStore(dir) |
| + w := sweepWarm(t, s) |
| + ancient := 5 * 365 * 24 * time.Hour |
| + seedCertEntry(t, dir, s, autocertAccountKeyName, []byte("account key material"), ancient) |
| + seedCertEntry(t, dir, s, wildcardAccountKeyName, []byte("account key material"), ancient) |
| + |
| + w.sweep(context.Background(), time.Now()) |
| + |
| + for _, name := range []string{autocertAccountKeyName, wildcardAccountKeyName} { |
| + if certGone(t, s, name) { |
| + t.Fatalf("ACCOUNT KEY %s WAS DELETED — the deployment has lost its ACME registration", name) |
| + } |
| + } |
| +} |
| + |
| +// A sweep that deletes what it cannot understand is a sweep that deletes |
| +// tomorrow's format. Anything that will not read, will not parse, or has no |
| +// leaf is left exactly where it is. |
| +func TestSweepLeavesWhatItCannotUnderstand(t *testing.T) { |
| + dir := t.TempDir() |
| + s := release.NewDirStore(dir) |
| + w := sweepWarm(t, s) |
| + old := certSweepParseAge + 24*time.Hour |
| + seedCertEntry(t, dir, s, "future-shape.customer.test", []byte("a format this build has never seen"), old) |
| + |
| + w.sweep(context.Background(), time.Now()) |
| + |
| + if certGone(t, s, "future-shape.customer.test") { |
| + t.Error("an entry this build could not parse was deleted") |
| + } |
| +} |
| + |
| +// The pre-filter is what keeps this from being an N+1: nothing younger than |
| +// the parse floor is even read. A healthy deployment renews every ~60 days, |
| +// so the steady-state cost is one listing and zero reads. |
| +func TestSweepReadsNothingYoungerThanTheParseFloor(t *testing.T) { |
| + dir := t.TempDir() |
| + s := &bucketGetCounter{Store: release.NewDirStore(dir)} |
| + w := sweepWarm(t, s) |
| + for _, host := range []string{"a.customer.test", "b.customer.test", "c.customer.test"} { |
| + seedCertEntry(t, dir, s, host, expiredPEM(t, host, time.Hour), 10*24*time.Hour) |
| + } |
| + before := s.gets() |
| + w.sweep(context.Background(), time.Now()) |
| + if n := s.gets() - before; n != 0 { |
| + t.Errorf("sweep spent %d get(s) on entries younger than the parse floor, want 0", n) |
| + } |
| +} |
| + |
| +// The expiry half runs daily, not hourly: it is the half that costs reads, |
| +// and a certificate that has been dead for a month can wait a day. The token |
| +// half runs every pass, because it needs no reads at all. |
| +func TestSweepRunsTheExpiryHalfDailyAndTheTokenHalfEveryPass(t *testing.T) { |
| + dir := t.TempDir() |
| + s := release.NewDirStore(dir) |
| + w := sweepWarm(t, s) |
| + old := certSweepParseAge + 24*time.Hour |
| + now := time.Now() |
| + |
| + seedCertEntry(t, dir, s, "dead-one.customer.test", expiredPEM(t, "dead-one.customer.test", certSweepGrace+24*time.Hour), old) |
| + w.sweep(context.Background(), now) |
| + if !certGone(t, s, "dead-one.customer.test") { |
| + t.Fatal("the first sweep did not run the expiry half") |
| + } |
| + |
| + // An hour later, a second dead cert appears and a token goes stale. |
| + seedCertEntry(t, dir, s, "dead-two.customer.test", expiredPEM(t, "dead-two.customer.test", certSweepGrace+24*time.Hour), old) |
| + seedCertEntry(t, dir, s, "tok"+httpTokenSuffix, []byte("key-authorization"), 2*time.Hour) |
| + w.sweep(context.Background(), now.Add(time.Hour)) |
| + if certGone(t, s, "dead-two.customer.test") { |
| + t.Error("the expiry half ran again an hour later — it is meant to be daily") |
| + } |
| + if !certGone(t, s, "tok"+httpTokenSuffix) { |
| + t.Error("the token half did not run on the second pass — it is meant to run every pass") |
| + } |
| + |
| + // A day later it comes back round. |
| + w.sweep(context.Background(), now.Add(certSweepInterval+time.Minute)) |
| + if !certGone(t, s, "dead-two.customer.test") { |
| + t.Error("the expiry half did not run again after a day") |
| + } |
| +} |
| + |
| +// A store with no last-modified stamps gives the sweep no age to judge by, |
| +// and guessing an object's age from its contents is exactly the inference |
| +// this file refuses to make. It does nothing at all rather than something |
| +// approximate. |
| +func TestSweepStandsDownWithoutStamps(t *testing.T) { |
| + dir := t.TempDir() |
| + inner := release.NewDirStore(dir) |
| + w := sweepWarm(t, stamplessEdgeStore{inner}) |
| + seedCertEntry(t, dir, inner, "tok"+httpTokenSuffix, []byte("key-authorization"), 30*24*time.Hour) |
| + |
| + w.sweep(context.Background(), time.Now()) |
| + |
| + if certGone(t, inner, "tok"+httpTokenSuffix) { |
| + t.Error("a store with no stamps still had entries deleted") |
| + } |
| +} |
| + |
| +type stamplessEdgeStore struct{ inner release.Store } |
| + |
| +func (s stamplessEdgeStore) Get(ctx context.Context, key string) ([]byte, string, error) { |
| + return s.inner.Get(ctx, key) |
| +} |
| +func (s stamplessEdgeStore) Put(ctx context.Context, key string, body []byte, ifMatch string) (string, error) { |
| + return s.inner.Put(ctx, key, body, ifMatch) |
| +} |
| +func (s stamplessEdgeStore) List(ctx context.Context, prefix string) ([]string, error) { |
| + return s.inner.List(ctx, prefix) |
| +} |
| +func (s stamplessEdgeStore) ListDirs(ctx context.Context, prefix string) ([]string, error) { |
| + return s.inner.ListDirs(ctx, prefix) |
| +} |
| +func (s stamplessEdgeStore) Delete(ctx context.Context, key string) error { |
| + return s.inner.Delete(ctx, key) |
| +} |
| + |
| +// A bug in the predicate deletes at most this many objects before somebody |
| +// reads the log, and a genuine backlog drains over days rather than in one |
| +// burst against the bucket. |
| +func TestSweepBoundsHowMuchOnePassCanDelete(t *testing.T) { |
| + dir := t.TempDir() |
| + s := release.NewDirStore(dir) |
| + w := sweepWarm(t, s) |
| + total := certSweepDeletesPerPass + 5 |
| + for i := 0; i < total; i++ { |
| + seedCertEntry(t, dir, s, string(rune('a'+i%26))+string(rune('a'+i/26))+".tok"+httpTokenSuffix, |
| + []byte("key-authorization"), 2*time.Hour) |
| + } |
| + w.sweep(context.Background(), time.Now()) |
| + left, err := s.List(context.Background(), certPrefix) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if len(left) != total-certSweepDeletesPerPass { |
| + t.Errorf("%d entries left after one pass, want %d — the per-pass bound is not holding", |
| + len(left), total-certSweepDeletesPerPass) |
| + } |
| +} |
| + |
| +// The sweep's work exists whether or not this deployment has custom |
| +// domains: spent challenge tokens and long-expired certificates are not the |
| +// warm SET's business. A pass over an empty set still sweeps — the gate check |
| +// deliberately comes before the empty-set return in pass(). |
| +func TestPassWithNoWarmSetStillSweeps(t *testing.T) { |
| + dir := t.TempDir() |
| + s := release.NewDirStore(dir) |
| + w := sweepWarm(t, s) |
| + seedCertEntry(t, dir, s, "orphan"+httpTokenSuffix, []byte("key-authorization"), 2*time.Hour) |
| + |
| + if d := w.pass(); d != warmInterval { |
| + t.Fatalf("pass over an empty set returned %s", d) |
| + } |
| + if !certGone(t, s, "orphan"+httpTokenSuffix) { |
| + t.Error("a deployment with no custom domains never sweeps its own token litter") |
| + } |
| +} |
| + |
| +// A non-issuer sweeps nothing. Deleting cert material is the sharpest thing |
| +// this package does, and a box that is not the issuer has no authority to |
| +// do it. |
| +func TestSweepNeverRunsOnANonIssuer(t *testing.T) { |
| + dir := t.TempDir() |
| + s := release.NewDirStore(dir) |
| + w := newCertWarm(func() []string { return nil }, nil, nil, |
| + func() (context.Context, bool) { return nil, false }, s) |
| + seedCertEntry(t, dir, s, "orphan"+httpTokenSuffix, []byte("key-authorization"), 2*time.Hour) |
| + |
| + w.pass() |
| + |
| + if certGone(t, s, "orphan"+httpTokenSuffix) { |
| + t.Error("a stood-down box deleted cert material") |
| + } |
| +} |
diff --git a/internal/edge/certwarm.go b/internal/edge/certwarm.go| index de905e723..28b4d6c34 100644 |
| --- a/internal/edge/certwarm.go |
| +++ b/internal/edge/certwarm.go |
| @@ -115,6 +115,10 @@ type certWarm struct { |
| warmed map[string]bool |
| lastStandDownLog time.Time |
| + // lastExpirySweep is when the expiry half of the cert sweep last ran |
| + // (certsweep.go). Guarded by mu like everything else here, though the |
| + // sweep is only ever reached from pass(). |
| + lastExpirySweep time.Time |
| } |
| func newCertWarm(hosts func() []string, getCert func(*tls.ClientHelloInfo) (*tls.Certificate, error), pointsHere func(string) bool, issuer IssuerGate, store release.Store) *certWarm { |
| @@ -147,16 +151,21 @@ func (c *certWarm) run() { |
| // pass warms every host in the set once and returns how long to wait. |
| func (c *certWarm) pass() time.Duration { |
| hosts := c.hosts() |
| - if len(hosts) == 0 { |
| - return warmInterval |
| - } |
| // Stand down before ANY work: unlike the wildcard manager there is no |
| // cache to load first, so a non-issuer has nothing useful to do here. |
| + // The gate comes BEFORE the empty-set check, not after, because the |
| + // sweep at the bottom of this pass has work to do in a deployment with |
| + // no custom domains at all — spent challenge tokens and long-expired |
| + // certs are not the warm set's business. |
| ctx, ok := c.issuerGate() |
| if !ok { |
| c.logStandDown(len(hosts)) |
| return warmInterval |
| } |
| + if len(hosts) == 0 { |
| + c.sweep(ctx, time.Now()) |
| + return warmInterval |
| + } |
| // The durable outcome records (#299 §3e), read ONCE per pass. Two things |
| // come out of this read and neither is available in memory: the attempt |
| // count a previous issuer accumulated (so a failover does not restart the |
| @@ -223,6 +232,10 @@ func (c *certWarm) pass() time.Duration { |
| } |
| } |
| c.prune(ctx, records, inSet) |
| + // Housekeeping last: it is the least urgent thing this pass does, and |
| + // running it after the orders means a slow bucket delays litter removal |
| + // rather than certificate issuance (certsweep.go). |
| + c.sweep(ctx, time.Now()) |
| if capped { |
| return warmCappedDelay |
| } |