| new file mode 100644 |
| index 000000000..046849599 |
| --- /dev/null |
| +++ b/docs/superpowers/plans/2026-08-20-multi-region-writer-map-and-certs.md |
| @@ -0,0 +1,593 @@ |
| +# Multi-region ingress, plan 1 of 2: writer map + cert distribution |
| + |
| +> **For agentic workers:** REQUIRED SUB-SKILL: Use |
| +> superpowers:subagent-driven-development (recommended) or |
| +> superpowers:executing-plans to implement this plan task-by-task. Steps |
| +> use checkbox (`- [ ]`) syntax for tracking. |
| + |
| +**Goal:** Ship spec stages 1–2 (writer map fixes + elected-issuer cert |
| +distribution), fully dormant on today's single-box deployments. |
| + |
| +**Architecture:** Per-box snapshot keys close #158; the releases prune |
| +moves behind a new housekeeper lease (same election pattern as the status |
| +publisher); cert issuance stays exactly today's autocert machinery but |
| +only the housekeeper mints, publishing cert material and HTTP-01 tokens |
| +through the control bucket so every edge can serve both. Plan 2 |
| +(backhaul + steering, spec Parts 3–4) starts only after this plan merges. |
| + |
| +**Tech Stack:** Go (platform repo), existing `release.Store` bucket |
| +interface, `golang.org/x/crypto/acme/autocert` (already vendored via |
| +edge), existing fake-store test harness. |
| + |
| +## Global Constraints |
| + |
| +- Spec: `docs/superpowers/specs/2026-08-20-multi-region-ingress-design.md`. |
| + Key shapes verbatim from it: `control/registry-snapshot/<box>.db`, |
| + `control/registry-snapshot/<box>.json`, |
| + `control/registry-snapshots/<box>/<stamp>.db`, `control/certs/**`, |
| + `control/housekeeper.json`. |
| +- **Wire-shape changes are additive.** Old fixed snapshot keys are NOT |
| + deleted by this plan; deletion happens after both deployments roll |
| + (recorded in the rollout notes, not code). |
| +- **`control/serial` stays console-written.** Nothing in this plan may |
| + write it. The certs mirror refreshes on its own 30s list-poll. |
| +- **One election pattern.** The housekeeper reuses the status publisher's |
| + election approach (CAS-lease in the bucket, renew at half-TTL); no new |
| + election mechanism. |
| +- **The fleet store door ACL never includes `control/certs/**` or |
| + `control/edge-mesh-key`** — Task 6 pins this with a permanent test. |
| +- Gate before every commit: `go build ./... && go vet ./... && |
| + go test ./... -count=1` plus `gofmt -l` clean on touched files, with |
| + `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off` |
| + (sandbox may need disabling — "read-only file system" is the tell). |
| +- Tests that shell out to the carlos binary build env through |
| + `carlosTestEnv` and set `cmd.Dir` to a scratch dir (standing rule; no |
| + task here should need it, but fix loops sometimes reach for the CLI). |
| +- Production channel names are pinned in |
| + `internal/release/laddercheck_test.go` — nothing here may alter ladder |
| + behaviour. |
| + |
| +## File Structure |
| + |
| +- `internal/host/snapshot_loop.go` — per-box key derivation (Task 1) |
| +- `internal/host/snapshot_loop_test.go` — updated + two-box test (Task 1) |
| +- `docs/superpowers/specs/2026-08-20-bucket-writer-audit.md` — audit |
| + deliverable (Task 2) |
| +- `internal/host/housekeeper.go` + `_test.go` — the lease (Task 3) |
| +- `internal/host/agent.go` — retention call gated by housekeeper (Task 3) |
| +- `internal/edge/certsync.go` + `_test.go` — publishing cache wrapper + |
| + edge mirror (Tasks 4–5) |
| +- `internal/edge/edge.go` — issuer gating + mirror wiring + challenge |
| + route (Tasks 4–5) |
| +- `internal/edge/store_door_test.go` (or the door's existing test file) — |
| + ACL guard (Task 6) |
| +- `internal/edge/certsync_e2e_test.go` — two-edge proof (Task 7) |
| + |
| +--- |
| + |
| +### Task 1: Per-box registry snapshot keys (closes #158) |
| + |
| +**Files:** |
| +- Modify: `internal/host/snapshot_loop.go` (consts at lines 24–26; every |
| + use of them in `registrySnapshotPass`, `seedSnapshotFingerprint`, |
| + `readSnapshotMeta`) |
| +- Test: `internal/host/snapshot_loop_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: existing `registrySnapshotPass(ctx, s, reg, box, now, lastFP)` |
| + — the `box` parameter already exists and becomes the key discriminator. |
| +- Produces: `snapshotDBKey(box string) string`, |
| + `snapshotMetaKey(box string) string`, |
| + `snapshotGenPrefix(box string) string` — Task 7's e2e and the DR |
| + runbook update rely on these exact shapes. |
| + |
| +- [ ] **Step 1: Write the failing tests** — replace fixed-key |
| + expectations and add the two-box isolation test: |
| + |
| +```go |
| +func TestSnapshotKeysArePerBox(t *testing.T) { |
| + if got, want := snapshotDBKey("annie"), "control/registry-snapshot/annie.db"; got != want { |
| + t.Fatalf("db key = %q, want %q", got, want) |
| + } |
| + if got, want := snapshotMetaKey("annie"), "control/registry-snapshot/annie.json"; got != want { |
| + t.Fatalf("meta key = %q, want %q", got, want) |
| + } |
| + if got, want := snapshotGenPrefix("annie"), "control/registry-snapshots/annie/"; got != want { |
| + t.Fatalf("gen prefix = %q, want %q", got, want) |
| + } |
| +} |
| + |
| +func TestTwoBoxesSnapshotWithoutColliding(t *testing.T) { |
| + s := newFakeStore(t) // the fake used by the existing pass tests |
| + regA, regB := scratchRegistry(t), scratchRegistry(t) |
| + var fpA, fpB string |
| + now := time.Date(2026, 8, 20, 12, 0, 0, 0, time.UTC) |
| + if err := registrySnapshotPass(context.Background(), s, regA, "annie", now, &fpA); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := registrySnapshotPass(context.Background(), s, regB, "betty", now, &fpB); err != nil { |
| + t.Fatal(err) |
| + } |
| + for _, key := range []string{ |
| + "control/registry-snapshot/annie.db", "control/registry-snapshot/annie.json", |
| + "control/registry-snapshot/betty.db", "control/registry-snapshot/betty.json", |
| + } { |
| + if _, err := s.Get(context.Background(), key); err != nil { |
| + t.Fatalf("missing %s: %v", key, err) |
| + } |
| + } |
| + // betty's pass must not have disturbed annie's generations. |
| + gens, err := s.List(context.Background(), "control/registry-snapshots/annie/") |
| + if err != nil || len(gens) != 1 { |
| + t.Fatalf("annie generations = %v, %v; want exactly 1", gens, err) |
| + } |
| +} |
| +``` |
| + |
| +(Adopt the existing test file's fake-store and scratch-registry helpers — |
| +they exist for the current pass tests; do not invent new ones.) |
| + |
| +- [ ] **Step 2: Run to verify failure** |
| + |
| +Run: `go test ./internal/host/ -run 'TestSnapshotKeys|TestTwoBoxes' -count=1` |
| +Expected: FAIL — `snapshotDBKey` undefined. |
| + |
| +- [ ] **Step 3: Implement.** Replace the three consts with: |
| + |
| +```go |
| +// Per-box keys (spec 2026-08-20 Part 1, closes #158): every platform box |
| +// snapshots ITS OWN registry under its own name; boxes never contend. |
| +// The pre-multi-region fixed keys ("control/registry-snapshot.db" etc.) |
| +// are still present in production buckets until both deployments roll — |
| +// they are abandoned, not rewritten; deletion is a rollout step, not code. |
| +func snapshotDBKey(box string) string { return "control/registry-snapshot/" + box + ".db" } |
| +func snapshotMetaKey(box string) string { return "control/registry-snapshot/" + box + ".json" } |
| +func snapshotGenPrefix(box string) string { |
| + return "control/registry-snapshots/" + box + "/" |
| +} |
| +``` |
| + |
| +Thread `box` to every call site inside the pass, the generation prune |
| +(which now lists `snapshotGenPrefix(box)`), `seedSnapshotFingerprint` |
| +(reads own box's sidecar — takes `box string` as a new parameter), and |
| +`readSnapshotMeta` (same). The foreign-box "OVERWRITING a snapshot |
| +written by box …" guard becomes unreachable for differently-named boxes; |
| +keep it — same-named boxes (a rebuilt standby reusing the dead box's |
| +name per the runbook) still hit it, and that is exactly when it should |
| +be loud. |
| + |
| +- [ ] **Step 4: Run the focused tests, then the package** |
| + |
| +Run: `go test ./internal/host/ -count=1` |
| +Expected: PASS, including all pre-existing snapshot tests updated to the |
| +per-box shapes. |
| + |
| +- [ ] **Step 5: Commit** |
| + |
| +```bash |
| +git add internal/host/snapshot_loop.go internal/host/snapshot_loop_test.go |
| +git commit -m "host: per-box registry snapshot keys — boxes never contend (#158)" |
| +``` |
| + |
| +**Cross-repo deliverable (report, don't do):** the DR runbook |
| +(`teamtito/carlos-infrastructure/RUNBOOK-edge-failover.md`) parses these |
| +keys; its update is a separate PR listed in this plan's rollout notes. |
| +The controller opens it after this plan's platform PR merges. Flagship |
| +has no DR runbook (spec records the gap). |
| + |
| +--- |
| + |
| +### Task 2: Bucket-writer audit (stage-1 deliverable) |
| + |
| +**Files:** |
| +- Create: `docs/superpowers/specs/2026-08-20-bucket-writer-audit.md` |
| + |
| +**Interfaces:** |
| +- Consumes: nothing from other tasks (pure survey; do it against the |
| + Task-1 tree). |
| +- Produces: the audit table later tasks and plan 2 cite; any writer |
| + discovered outside the spec's table gets a ruling row here. |
| + |
| +- [ ] **Step 1: Enumerate every bucket write.** Survey, verbatim recipe: |
| + |
| +```bash |
| +grep -rn "\.Put(\|\.Delete(\|putReplacing(" internal/ --include='*.go' | grep -v _test |
| +``` |
| + |
| +For each call site record: file:line, the key/prefix written, which |
| +process runs it (console / every box / housekeeper / edge), and the |
| +spec ruling that covers it. Known-expected rows (verify, don't assume): |
| +`usage/edge/<box>/days/` (already box-scoped — `internal/usage/edge.go:40`), |
| +`usage/census/` + `usage/storage/` + statements (console's |
| +`economicspass.go` — console-only, fine), status publisher (elected), |
| +snapshots (Task 1), retention deletes (Task 3), records/domains/serial |
| +(console). |
| + |
| +- [ ] **Step 2: Write the audit doc.** One table, same columns as the |
| + spec's writer map, one row per call site, each with a ruling of: |
| + `console-only` | `per-box keys` | `housekeeper` | `elected already` | |
| + `NEEDS RULING`. A `NEEDS RULING` row is a finding: stop and surface it |
| + to the controller rather than inventing a ruling. |
| + |
| +- [ ] **Step 3: Commit** |
| + |
| +```bash |
| +git add docs/superpowers/specs/2026-08-20-bucket-writer-audit.md |
| +git commit -m "docs: bucket-writer audit — every write call site ruled for N boxes" |
| +``` |
| + |
| +--- |
| + |
| +### Task 3: Housekeeper lease; releases prune moves behind it |
| + |
| +**Files:** |
| +- Create: `internal/host/housekeeper.go` |
| +- Test: `internal/host/housekeeper_test.go` |
| +- Modify: `internal/host/agent.go` (the tick that calls |
| + `retentionPass` — find it with `grep -n retentionPass internal/host/agent.go`) |
| + |
| +**Interfaces:** |
| +- Consumes: `retentionPass(ctx, store, pub, reg)` from |
| + `internal/host/retention.go:26` (unchanged signature); the status |
| + publisher's election in `agent.go`'s statusLoop as the pattern to |
| + mirror (read it first — reuse its store client and clock idioms, and |
| + if a helper can be shared without contortion, share it). |
| +- Produces: `acquireHousekeeper(ctx context.Context, s release.Store, |
| + box string, now time.Time) (held bool, err error)` — Task 4 gates |
| + issuance on it. |
| + |
| +- [ ] **Step 1: Failing tests** — CAS contention and expiry: |
| + |
| +```go |
| +func TestHousekeeperSingleHolder(t *testing.T) { |
| + s := newFakeStore(t) |
| + now := time.Date(2026, 8, 20, 12, 0, 0, 0, time.UTC) |
| + heldA, err := acquireHousekeeper(context.Background(), s, "annie", now) |
| + if err != nil || !heldA { |
| + t.Fatalf("annie should acquire: held=%v err=%v", heldA, err) |
| + } |
| + heldB, err := acquireHousekeeper(context.Background(), s, "betty", now.Add(10*time.Second)) |
| + if err != nil || heldB { |
| + t.Fatalf("betty must NOT hold while annie's lease is live: held=%v err=%v", heldB, err) |
| + } |
| + // After expiry betty takes over. |
| + heldB, err = acquireHousekeeper(context.Background(), s, "betty", now.Add(housekeeperTTL+time.Second)) |
| + if err != nil || !heldB { |
| + t.Fatalf("betty should acquire after expiry: held=%v err=%v", heldB, err) |
| + } |
| +} |
| + |
| +func TestHousekeeperRenewsOwnLease(t *testing.T) { |
| + s := newFakeStore(t) |
| + now := time.Date(2026, 8, 20, 12, 0, 0, 0, time.UTC) |
| + if held, _ := acquireHousekeeper(context.Background(), s, "annie", now); !held { |
| + t.Fatal("first acquire") |
| + } |
| + // Same box re-acquiring mid-TTL renews rather than being locked out. |
| + if held, _ := acquireHousekeeper(context.Background(), s, "annie", now.Add(housekeeperTTL/2)); !held { |
| + t.Fatal("holder must renew its own live lease") |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Verify failure** — `go test ./internal/host/ -run |
| + Housekeeper -count=1` → FAIL, undefined. |
| + |
| +- [ ] **Step 3: Implement `housekeeper.go`:** |
| + |
| +```go |
| +// Package-internal single election for deployment-global duties (spec |
| +// 2026-08-20 Part 1): releases retention prune (this plan) and cert |
| +// issuance (Task 4). ONE lease, several duties — never a second election. |
| +const ( |
| + housekeeperKey = "control/housekeeper.json" |
| + housekeeperTTL = 90 * time.Second |
| +) |
| + |
| +type housekeeperLease struct { |
| + Box string `json:"box"` |
| + Expires time.Time `json:"expires"` |
| +} |
| + |
| +// acquireHousekeeper CAS-writes the lease if absent, expired, or already |
| +// ours (renewal). It never blocks; a false return means another box |
| +// holds a live lease and the caller simply skips housekeeper duties this |
| +// tick — duties are idempotent and re-run within a TTL of any handover. |
| +func acquireHousekeeper(ctx context.Context, s release.Store, box string, now time.Time) (bool, error) |
| +``` |
| + |
| +Implementation mirrors the status publisher election: `Get` the key; if |
| +present, unmarshal, and return true early only when `Box == box || |
| +now.After(Expires)` fails the other way — i.e. contend only when absent, |
| +expired, or self-owned; write with the store's conditional-PUT (the same |
| +CAS the publisher election and `putReplacing` use — reuse, don't |
| +duplicate; if extraction of a shared helper from the statusLoop election |
| +is clean, do it and note it in the report). Losing the CAS race returns |
| +`(false, nil)`. |
| + |
| +- [ ] **Step 4: Gate the prune.** In `agent.go`, wrap the existing |
| + `retentionPass` call: |
| + |
| +```go |
| +held, err := acquireHousekeeper(ctx, store, boxName, time.Now()) |
| +if err != nil { |
| + log.Printf("housekeeper: election error (prune skipped this tick): %v", err) |
| +} else if held { |
| + if err := retentionPass(ctx, store, pub, reg); err != nil { /* existing handling */ } |
| +} |
| +``` |
| + |
| +(Exact surrounding names come from the call site — keep its logging |
| +style. On a single-box deployment the box always holds: provably inert.) |
| + |
| +- [ ] **Step 5: Package gate + commit** |
| + |
| +```bash |
| +go test ./internal/host/ -count=1 |
| +git add internal/host/housekeeper.go internal/host/housekeeper_test.go internal/host/agent.go |
| +git commit -m "host: housekeeper lease; releases prune single-holder (spec Part 1)" |
| +``` |
| + |
| +--- |
| + |
| +### Task 4: Issuer-side cert publishing (housekeeper mints, bucket carries) |
| + |
| +**Files:** |
| +- Create: `internal/edge/certsync.go` |
| +- Test: `internal/edge/certsync_test.go` |
| +- Modify: `internal/edge/edge.go` (autocert manager construction at |
| + ~line 1765; `RunWithActivatorReady`'s HTTP-01 handler wiring ~1222) |
| + |
| +**Interfaces:** |
| +- Consumes: `acquireHousekeeper` (Task 3) via a |
| + `held func() bool` closure the host layer already knows how to |
| + provide (edge receives it as a field, mirroring how existing |
| + host-owned knobs reach edge construction — follow the pattern of the |
| + edge's existing config fields); the edge's bucket store handle. |
| +- Produces: `publishingCache{inner autocert.Cache, s release.Store}` |
| + satisfying `autocert.Cache`; bucket key shape |
| + `certKey(name string) = "control/certs/" + name` — Task 5's mirror |
| + and Task 6's ACL test rely on it. |
| + |
| +Key facts for the implementer (verified against the vendored autocert): |
| +`autocert.Manager` writes EVERYTHING through its `Cache` — account key, |
| +per-host cert+key PEM under the host name, and HTTP-01 challenge |
| +responses under `<token-path>+http-01` (cluster support is the |
| +documented purpose). Wrapping the cache captures certs AND tokens with |
| +zero autocert forks. |
| + |
| +- [ ] **Step 1: Failing tests:** |
| + |
| +```go |
| +func TestPublishingCacheMirrorsPutsToBucket(t *testing.T) { |
| + s := newFakeStore(t) |
| + pc := &publishingCache{inner: autocert.DirCache(t.TempDir()), s: s} |
| + if err := pc.Put(context.Background(), "example.com", []byte("PEM")); err != nil { |
| + t.Fatal(err) |
| + } |
| + got, err := s.Get(context.Background(), "control/certs/example.com") |
| + if err != nil || string(got) != "PEM" { |
| + t.Fatalf("bucket copy = %q, %v", got, err) |
| + } |
| + // Local cache also holds it (autocert's own read path keeps working). |
| + if b, err := pc.Get(context.Background(), "example.com"); err != nil || string(b) != "PEM" { |
| + t.Fatalf("local copy = %q, %v", b, err) |
| + } |
| +} |
| + |
| +func TestPublishingCacheDeleteRemovesBucketCopy(t *testing.T) { |
| + s := newFakeStore(t) |
| + pc := &publishingCache{inner: autocert.DirCache(t.TempDir()), s: s} |
| + _ = pc.Put(context.Background(), "tok+http-01", []byte("challenge")) |
| + if err := pc.Delete(context.Background(), "tok+http-01"); err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, err := s.Get(context.Background(), "control/certs/tok+http-01"); err == nil { |
| + t.Fatal("token must be deleted from bucket after validation") |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Verify failure**, then **Step 3: implement** — |
| + `publishingCache` methods delegate to `inner` first (local durability), |
| + then best-effort mirror to the bucket with `putReplacing`/`Delete`; a |
| + bucket error logs loudly (`certsync: bucket publish failed for %q`) |
| + and does NOT fail the autocert operation (local issuance must survive |
| + bucket blips; the 30s mirror poll self-heals). The account key entry |
| + (`acme_account+key`) is mirrored like everything else — a failed-over |
| + housekeeper reuses the account instead of registering a new one. |
| + |
| +- [ ] **Step 4: Gate issuance on the housekeeper.** In `edge.go` where |
| + the manager's `HostPolicy` is set, wrap: when `e.housekeeperHeld()` is |
| + false, return a sentinel `errNotIssuer` from HostPolicy — autocert |
| + then refuses new orders on non-issuers, while `GetCertificate` still |
| + serves anything already in cache/mirror (Task 5). Single-box |
| + deployments always hold → behaviour identical to today. Renewals run |
| + only where orders can start, so only on the issuer. |
| + |
| +- [ ] **Step 5: Package gate + commit** |
| + |
| +```bash |
| +go test ./internal/edge/ -count=1 |
| +git add internal/edge/certsync.go internal/edge/certsync_test.go internal/edge/edge.go |
| +git commit -m "edge: housekeeper-gated issuance, certs+tokens published to bucket" |
| +``` |
| + |
| +--- |
| + |
| +### Task 5: Edge mirror — serve certs and challenges everywhere |
| + |
| +**Files:** |
| +- Modify: `internal/edge/certsync.go` (mirror type), `internal/edge/edge.go` |
| + (GetCertificate layering at `wrapGetCertificate` — `wildcard.go:665` — |
| + and the HTTP-01 path) |
| +- Test: `internal/edge/certsync_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: `certKey` shape from Task 4. |
| +- Produces: `certMirror` with `Refresh(ctx) error` (30s loop), |
| + `Cert(name string) ([]byte, bool)`, `Challenge(token string) ([]byte, |
| + bool)` — Task 7 exercises all three. |
| + |
| +- [ ] **Step 1: Failing tests:** |
| + |
| +```go |
| +func TestCertMirrorServesBucketCert(t *testing.T) { |
| + s := newFakeStore(t) |
| + _ = putReplacing(context.Background(), s, "control/certs/example.com", []byte("PEM")) |
| + m := newCertMirror(s) |
| + if err := m.Refresh(context.Background()); err != nil { |
| + t.Fatal(err) |
| + } |
| + if b, ok := m.Cert("example.com"); !ok || string(b) != "PEM" { |
| + t.Fatalf("mirror cert = %q, %v", b, ok) |
| + } |
| +} |
| + |
| +func TestCertMirrorServesChallengeTokens(t *testing.T) { |
| + s := newFakeStore(t) |
| + _ = putReplacing(context.Background(), s, "control/certs/.well-known/acme-challenge/tok+http-01", []byte("resp")) |
| + m := newCertMirror(s) |
| + _ = m.Refresh(context.Background()) |
| + if b, ok := m.Challenge("tok"); !ok || string(b) != "resp" { |
| + t.Fatalf("challenge = %q, %v", b, ok) |
| + } |
| +} |
| +``` |
| + |
| +(The exact cache key autocert uses for HTTP-01 entries must be read from |
| +the vendored source in Step 3 — the `+http-01` suffix is stable; the |
| +path prefix in the key mirrors the request path. Pin the REAL observed |
| +key shape in these tests during implementation, not this plan's guess — |
| +then update `Challenge()` to match. This is the one place the plan |
| +defers to the vendored code on purpose; getting it wrong fails Task 7 |
| +loudly.) |
| + |
| +- [ ] **Step 2: Verify failure. Step 3: implement** — `certMirror` holds |
| + a `map[string][]byte` behind a `sync.RWMutex`, rebuilt by `Refresh` |
| + from `s.List("control/certs/")` + gets (list-then-get; sizes are small |
| + and renewals rare — no delta cleverness). A refresh error keeps the |
| + last-good map (degrade-not-die, the edge's standing pattern). Start a |
| + 30s refresh loop next to the edge's existing background loops. |
| + |
| +- [ ] **Step 4: Wire serving.** `GetCertificate` layering becomes: |
| + wildcard → local autocert → mirror (parse PEM with |
| + `tls.X509KeyPair`, cache the parsed `tls.Certificate` keyed by |
| + mirror-map generation so parsing isn't per-handshake). HTTP-01: in the |
| + handler where autocert's `HTTPHandler` is consulted (`edge.go` ~1222), |
| + fall back to `mirror.Challenge(token)` when local autocert has no |
| + answer. Non-issuer edges thereby answer any edge's challenges. |
| + |
| +- [ ] **Step 5: Package gate + commit** |
| + |
| +```bash |
| +go test ./internal/edge/ -count=1 |
| +git add internal/edge/certsync.go internal/edge/certsync_test.go internal/edge/edge.go internal/edge/wildcard.go |
| +git commit -m "edge: cert mirror — every edge serves every cert and challenge" |
| +``` |
| + |
| +--- |
| + |
| +### Task 6: Door ACL guard — keys never leave through the fleet door |
| + |
| +**Files:** |
| +- Test: add to the store door's existing ACL test file (find with |
| + `grep -rln "store/o\|door" internal/edge/*_test.go`) |
| + |
| +**Interfaces:** consumes the door's existing test harness; produces the |
| +permanent guard the spec promises. |
| + |
| +- [ ] **Step 1: Write the test** (goes straight to green if the ACL is |
| + the allowlist the spec says it is — that's fine; it's a tripwire, and |
| + the review gate checks it FAILS if the allowlist is loosened): |
| + |
| +```go |
| +func TestDoorNeverServesCertMaterial(t *testing.T) { |
| + // Use the door test harness's authenticated fleet client. |
| + for _, key := range []string{ |
| + "control/certs/example.com", |
| + "control/certs/acme_account+key", |
| + "control/edge-mesh-key", |
| + "control/housekeeper.json", |
| + } { |
| + resp := doorGet(t, key) // harness's existing helper for authed reads |
| + if resp.StatusCode != http.StatusForbidden && resp.StatusCode != http.StatusNotFound { |
| + t.Fatalf("door served %s: %d — cert material must never transit the door", key, resp.StatusCode) |
| + } |
| + } |
| +} |
| +``` |
| + |
| +Also assert the `dirs`/`list` verbs refuse `prefix=control/certs/`. |
| +Adapt helper names to the harness's actual ones; do not weaken the |
| +assertion to "not 200". |
| + |
| +- [ ] **Step 2: Run** `go test ./internal/edge/ -run DoorNever -count=1` |
| + → PASS (or a real finding — surface it, don't paper it). |
| + |
| +- [ ] **Step 3: Commit** |
| + |
| +```bash |
| +git add internal/edge/ |
| +git commit -m "edge: door ACL tripwire — cert material never transits the fleet door" |
| +``` |
| + |
| +--- |
| + |
| +### Task 7: Two-edge e2e — mint on one, serve on the other |
| + |
| +**Files:** |
| +- Create: `internal/edge/certsync_e2e_test.go` |
| + |
| +**Interfaces:** consumes everything above; pattern-match the existing |
| +in-process e2e harnesses (`tunnel_e2e_test.go`, `fleet_e2e_test.go`) for |
| +fake-ACME and dual-edge scaffolding. |
| + |
| +- [ ] **Step 1: Write the test:** one fake store; edge A with |
| + `housekeeperHeld = () => true` and a fake ACME CA (the wildcard tests |
| + already carry one — reuse it); edge B with `housekeeperHeld = () => |
| + false` and the same store. Drive: A mints `app.example.test` (fake CA |
| + validates via HTTP-01 against **B's** handler — the challenge-anywhere |
| + proof); then a TLS handshake against **B** with SNI `app.example.test` |
| + succeeds using the mirrored cert after one `Refresh`. Assert B started |
| + zero ACME orders (fake CA order counter), and that A's |
| + `errNotIssuer`-gated twin refuses to mint when `held` is false. |
| + |
| +- [ ] **Step 2: Run** `go test ./internal/edge/ -run CertSyncE2E -count=1 -v` |
| + → PASS, output pristine. |
| + |
| +- [ ] **Step 3: Full gate + commit** |
| + |
| +```bash |
| +go build ./... && go vet ./... && go test ./... -count=1 |
| +git add internal/edge/certsync_e2e_test.go |
| +git commit -m "edge: e2e — issuer mints, every edge serves cert and challenge" |
| +``` |
| + |
| +--- |
| + |
| +### Task 8: Spec/rollout bookkeeping |
| + |
| +**Files:** |
| +- Modify: `docs/superpowers/specs/2026-08-20-multi-region-ingress-design.md` |
| + (mark stage 1–2 rows as implemented-dormant; the 30s-poll amendment is |
| + already in) |
| +- Modify: `docs/superpowers/plans/2026-08-20-multi-region-writer-map-and-certs.md` |
| + (rollout notes section below filled with actuals) |
| + |
| +- [ ] **Step 1:** Record in the plan's rollout notes: (a) old fixed |
| + snapshot keys still live in both buckets — delete after both |
| + deployments run this build (one `aws s3 rm` pair per bucket, listed |
| + verbatim); (b) the teamtito runbook PR (key-shape update §Step 0/3) |
| + is owed at that same moment; (c) #158 closes on merge — comment |
| + linking the audit doc; pinfra #15's blocker note gets updated. |
| +- [ ] **Step 2:** Full gate, commit `docs: stage 1-2 rollout notes`. |
| + |
| +--- |
| + |
| +## Rollout notes (filled by Task 8 at execution) |
| + |
| +- Old-key deletion commands: (recorded at execution) |
| +- Runbook PR: (recorded at execution) |
| +- Issue comments: #158 close, pinfra #15 unblock, #207 interaction — |
| + per-box keys change the generation-window arithmetic note. |
amadan