amadan

carlos / platform Public

edge: fetch the cert mirror's boot refresh concurrently

The Sydney box has failed every self-update since 2026-09-07 and taken no
build for eleven days. It was not the build and not the box: it was
distance.

certMirror.Refresh does one S3 get per published cert, one after another,
and it runs as a synchronous pass BEFORE either listener binds, so its cost
is the edge's time-to-serve. The deployment bucket is in eu-west-1 and the
AU box is ~250ms away from it, so at 101 published certs the pass took about
25 seconds. certMirrorBootTimeout is 20s and the self-update gate waits 20s
for the edge to answer on loopback, so the edge bound its listeners about
two seconds after the gate had already given up. The apply rolled back, the
rollback restarted the edge and failed the same probe for the same reason,
and the box stayed where it was. Every roll, identically.

The gets are independent, so they now run concurrently, bounded at 16: the
pass costs ceil(count/concurrency) x latency instead of count x latency,
which is about 2s instead of 25s at today's count. Error handling is
unchanged and is applied in key order rather than completion order, so a
partial poll reports the same lastErr however the fetches interleave.

The count is why this is a fleet problem and not an AU one. Every box pays
count x latency, so each newly published cert moves every box closer to the
same cliff; us-east-1 at ~75ms is about a third of the way there today.

homeMap.refreshHosts has the same serial shape over control/edges/*. It is
bounded by box count rather than cert count, so it is nowhere near the
cliff, and it is left alone here rather than changed without a reason.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Paul Campbell pushed by paul@keymail.dev 6e8c2992191e590e41a6afe57fa0b85ece53dee3 parent 203f67a
2 files changed, +135 −5
  • internal/edge/certsync.go +62 −5
  • internal/edge/certsync_test.go +73 −0

CI passed — run details

diff --git a/internal/edge/certsync.go b/internal/edge/certsync.go
index 46c318f3a..05a56970b 100644
--- a/internal/edge/certsync.go
+++ b/internal/edge/certsync.go
@@ -420,6 +420,14 @@ var certMirrorTimeout = 2 * time.Minute
// serve from while the loop catches up.
var certMirrorBootTimeout = 20 * time.Second
+// certRefreshConcurrency bounds the in-flight gets in Refresh. It is a
+// latency fix, not a throughput one: the pass costs
+// ceil(count/concurrency) x latency instead of count x latency, which at
+// 101 certs and a 250ms round trip is about 2s instead of 25s. Bounded so a
+// deployment with thousands of published certs opens a predictable number of
+// connections rather than one per object.
+const certRefreshConcurrency = 16
+
// certSnapshot is one poll's result, swapped in whole. Immutable once
// published, so readers hold no lock beyond the pointer read.
type certSnapshot struct {
@@ -616,6 +624,30 @@ func (m *certMirror) Refresh(ctx context.Context) error {
next := make(map[string][]byte, len(keys))
var tried, failed int
var lastErr error
+
+ // The eligible names first, then their bodies CONCURRENTLY. The fetch is
+ // one round trip per object and the objects are independent, so doing
+ // them one after another makes the whole pass cost
+ // count x latency — which is fine beside the bucket and is not fine
+ // across the planet.
+ //
+ // That is not hypothetical (amadan #118 thread, diagnosed 2026-09-19).
+ // The Sydney box reads this deployment's bucket in eu-west-1 at ~250ms a
+ // round trip. At 101 published certs the serial pass took ~25s, blowing
+ // certMirrorBootTimeout, and since this refresh runs BEFORE either
+ // listener binds, the edge was not serving until it finished. The
+ // self-update gate waits 20s for the edge to answer on loopback, so that
+ // box failed its gate, rolled back, failed the rollback's gate too, and
+ // took no update at all from 2026-09-07 to 2026-09-19.
+ //
+ // The count is what makes it a fleet problem rather than an AU one: every
+ // box pays count x latency, so each new published cert moves every box
+ // closer to the same cliff. us-east-1 at ~75ms is about a third of the
+ // way there at today's count.
+ type fetch struct {
+ key, name string
+ }
+ var items []fetch
for _, key := range keys {
name, ok := strings.CutPrefix(key, certPrefix)
// publishableCertName is the same filter the publish side applies:
@@ -624,8 +656,33 @@ func (m *certMirror) Refresh(ctx context.Context) error {
if !ok || !publishableCertName(name) {
continue
}
- tried++
- body, _, err := m.s.Get(ctx, key)
+ items = append(items, fetch{key: key, name: name})
+ }
+ tried = len(items)
+
+ type result struct {
+ body []byte
+ err error
+ }
+ results := make([]result, len(items))
+ sem := make(chan struct{}, certRefreshConcurrency)
+ var wg sync.WaitGroup
+ for i, it := range items {
+ wg.Add(1)
+ go func(i int, key string) {
+ defer wg.Done()
+ sem <- struct{}{}
+ defer func() { <-sem }()
+ body, _, err := m.s.Get(ctx, key)
+ results[i] = result{body: body, err: err}
+ }(i, it.key)
+ }
+ wg.Wait()
+
+ // Applied in key order, not completion order, so a partial poll reports
+ // the same lastErr whichever way the fetches happened to interleave.
+ for i, it := range items {
+ body, err := results[i].body, results[i].err
switch {
case errors.Is(err, os.ErrNotExist):
// Raced a delete between the list and the get — a spent token. Not
@@ -639,12 +696,12 @@ func (m *certMirror) Refresh(ctx context.Context) error {
// the previous copy of this name forward and keep going.
failed++
lastErr = err
- if body, ok := prev.entries[name]; ok {
- next[name] = body
+ if body, ok := prev.entries[it.name]; ok {
+ next[it.name] = body
}
continue
}
- next[name] = body
+ next[it.name] = body
}
if tried > 0 && failed == tried {
// Nothing at all could be read: that is a bucket fault, not an object
diff --git a/internal/edge/certsync_test.go b/internal/edge/certsync_test.go
index 7e830e3c1..df84756a3 100644
--- a/internal/edge/certsync_test.go
+++ b/internal/edge/certsync_test.go
@@ -1851,3 +1851,76 @@ func TestDeploymentHostIsAllowedOnASingleBoxDeployment(t *testing.T) {
t.Fatal("with an empty home map, deploymentHost must be exactly Allowed")
}
}
+
+// slowStore adds a fixed delay to every Get, standing in for a box that
+// reads the deployment bucket from the other side of the planet.
+type slowStore struct {
+ release.Store
+ delay time.Duration
+ mu sync.Mutex
+ peak int // greatest number of gets in flight at once
+ cur int
+}
+
+func (s *slowStore) Get(ctx context.Context, key string) ([]byte, string, error) {
+ s.mu.Lock()
+ s.cur++
+ if s.cur > s.peak {
+ s.peak = s.cur
+ }
+ s.mu.Unlock()
+ time.Sleep(s.delay)
+ s.mu.Lock()
+ s.cur--
+ s.mu.Unlock()
+ return s.Store.Get(ctx, key)
+}
+
+// TestCertMirrorRefreshFetchesConcurrently is amadan #118's second half as a
+// test. The mirror's boot refresh runs BEFORE either listener binds, so its
+// cost is the edge's time-to-serve; the self-update gate allows 20s for the
+// edge to answer on loopback. Serially, the pass costs count x latency, and
+// the Sydney box (101 certs, ~250ms to eu-west-1) therefore took ~25s to
+// start serving and failed every gate from 2026-09-07 to 2026-09-19.
+//
+// The property is that the pass does NOT cost count x latency. Asserted as
+// wall-clock against a deliberately slow store, and by the peak number of
+// gets in flight, so a change that quietly re-serialises it is caught by
+// both.
+func TestCertMirrorRefreshFetchesConcurrently(t *testing.T) {
+ const (
+ certs = 40
+ delay = 20 * time.Millisecond
+ )
+ inner := release.NewDirStore(t.TempDir())
+ for i := range certs {
+ putBucket(t, inner, fmt.Sprintf("host%02d.example.com", i), []byte("PEM"))
+ }
+ s := &slowStore{Store: inner, delay: delay}
+ m := newCertMirror(s, nil)
+
+ start := time.Now()
+ if err := m.Refresh(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ took := time.Since(start)
+
+ // Serial would be at least certs*delay (800ms). Concurrency 16 puts it
+ // near ceil(40/16)*20ms = 60ms; half the serial cost is a wide margin
+ // that still fails loudly if the loop goes back to one at a time.
+ if serial := certs * delay; took > serial/2 {
+ t.Errorf("refresh took %s, serial would be ~%s — the gets are not concurrent", took, serial)
+ }
+ if s.peak < 2 {
+ t.Errorf("peak gets in flight = %d, want more than one", s.peak)
+ }
+ if s.peak > certRefreshConcurrency {
+ t.Errorf("peak gets in flight = %d, above the %d bound", s.peak, certRefreshConcurrency)
+ }
+ // And it still mirrors everything it was asked for.
+ for i := range certs {
+ if _, ok := m.Cert(fmt.Sprintf("host%02d.example.com", i)); !ok {
+ t.Fatalf("host%02d.example.com missing from the mirror", i)
+ }
+ }
+}