| index 4298c70a1..ce724a806 100644 |
| --- a/docs/superpowers/plans/2026-08-20-multi-region-writer-map-and-certs.md |
| +++ b/docs/superpowers/plans/2026-08-20-multi-region-writer-map-and-certs.md |
| @@ -546,7 +546,163 @@ git commit -m "edge: e2e — issuer mints, every edge serves cert and challenge" |
| |
| ## Rollout notes (filled by Task 8 at execution) |
| |
| -- Old-key deletion commands: (recorded at execution) |
| -- Runbook PR: (recorded at execution) |
| -- Issue comments: #158 close, pinfra #15 unblock, #207 interaction — |
| - per-box keys change the generation-window arithmetic note. |
| +Synthesized from the SDD ledger (`progress.md`) across all seven |
| +implementation tasks and their reviews. This plan ships stages 1-2 |
| +dormant; nothing below is armed by merging this branch — it is the |
| +checklist for the rollout that follows. |
| + |
| +1. **Delete the old fixed snapshot keys — only after BOTH deployments run |
| + this build.** Task 1 (`b466af4`) replaced the fixed |
| + `control/registry-snapshot.{db,json}` + `control/registry-snapshots/` |
| + keys with per-box ones; the old keys are abandoned in place, not |
| + migrated, and stay live in both deployment buckets until every box in |
| + both deployments has converged onto a release containing this branch |
| + (verify each box has written its own `control/registry-snapshot/<box>.db` |
| + before deleting the fixed keys — there is no CLI/console surface for |
| + this today, so it's a direct-bucket check, consistent with CLAUDE.md's |
| + carve-out for platform-internal infra work). Exact commands, one pair |
| + of buckets (flagship profile `keymail`, tito profile `titogo` — |
| + coordinates per CLAUDE.md/session memory): |
| + |
| + ```bash |
| + # flagship (carlos-flagship-271376211898) |
| + aws --profile keymail s3 rm s3://carlos-flagship-271376211898/control/registry-snapshot.db |
| + aws --profile keymail s3 rm s3://carlos-flagship-271376211898/control/registry-snapshot.json |
| + aws --profile keymail s3 rm s3://carlos-flagship-271376211898/control/registry-snapshots/ --recursive |
| + |
| + # tito (carlos-tito-platform-665106592142) |
| + aws --profile titogo s3 rm s3://carlos-tito-platform-665106592142/control/registry-snapshot.db |
| + aws --profile titogo s3 rm s3://carlos-tito-platform-665106592142/control/registry-snapshot.json |
| + aws --profile titogo s3 rm s3://carlos-tito-platform-665106592142/control/registry-snapshots/ --recursive |
| + ``` |
| + |
| +2. **teamtito runbook PR is owed at that same moment** (Task 1 ledger). |
| + `teamtito/carlos-infrastructure/RUNBOOK-edge-failover.md` parses the |
| + snapshot keys and must land the key-shape update in the same window as |
| + the deletion above, not before (old runbook still needs to name the old |
| + keys until the deletion happens) and not long after (a runbook naming |
| + dead keys is a live footgun during a restore). Two content requirements |
| + beyond the key rename, both from the ledger: |
| + - **Step 0 ordering rule**: restore-before-agent-start / |
| + machine-id-reuse — a rebuilt standby must be restored from the dead |
| + box's snapshot BEFORE the agent starts and mints a fresh machine id, |
| + or it seeds its own empty registry under a new box label instead of |
| + inheriting the dead box's key. |
| + - **The agent-log "OVERWRITING a snapshot written by box …" line is NO |
| + LONGER a clobber signal** for a runbook reader once keys are per-box: |
| + it now only fires for a same-named box reusing a dead box's label |
| + (the intended restore path), not for two different boxes colliding — |
| + that collision is structurally impossible post-Task-1. The runbook |
| + should say so explicitly so an operator doesn't panic at a line that |
| + is now expected, not alarming. |
| + - Flagship has no DR runbook today — this gap is recorded in the spec, |
| + not hidden; the teamtito PR only touches tito's runbook. |
| + |
| +3. **Never deploy Task 4's issuer gate without Task 5's mirror.** The |
| + HostPolicy gate (`errNotIssuer` on non-issuers) and the cert/challenge |
| + mirror are two halves of one mechanism — a box running only the gate |
| + with no mirror to serve from has a no-TLS window for exactly as long as |
| + it isn't the issuer. Roll both in the same build, same deploy. (Task 4 |
| + T8 rollout rule, promoted from review.) |
| + |
| +4. **Drop `CARLOS_DEPLOYMENT_BUCKET` on grandfathered fleet boxes (the |
| + Pi) before this rolls.** `internal/edge/certsync.go`'s `armCertIssuance` |
| + comment: a fleet box (`CARLOS_FLEET` set) that still carries a |
| + deployment-bucket env from before the fleet-placement cutover reads |
| + `control/certs/**` directly over its own IAM and is gated + mirrored |
| + like any other non-issuer edge — "a transition state, resolved by the |
| + rollout dropping that env." Until that env is dropped, the Pi is a real |
| + (if inert) reader of cert material outside the intended fleet-door |
| + boundary. Task 4 fix-round-1 residual. |
| + |
| +5. **Don't let a pubkey-less bucket-backed box become housekeeper.** The |
| + host-level housekeeper election (`internal/host/housekeeper.go`) has no |
| + visibility into `CARLOS_RELEASE_PUBKEY` — a box with a deployment bucket |
| + but no release pubkey CAN win the housekeeper lease at the host layer, |
| + while its edge layer (`enableStaticFromEnv`) leaves `staticStore` nil for |
| + want of a pubkey, which makes `armCertIssuance` treat it as "no |
| + deployment store": it issues and serves its OWN certs unconditionally, |
| + outside the gate entirely, orphaned from the shared mechanism — a second |
| + uncoordinated ACME order stream running alongside whichever box the |
| + deployment actually intends as issuer, defeating the one-elected-issuer |
| + invariant. No code fix landed this round (Task 4 fix-round-1 residual, |
| + tagged T8); the rollout mitigation is operational: audit every box's env |
| + before arming a second housekeeper-eligible box, and exclude (or hand a |
| + pubkey to) any bucket-backed box that lacks one. |
| + |
| +6. **Check `carlos-agent`'s systemd `TimeoutStopSec` covers the shutdown |
| + chain (pinfra).** Housekeeper stand-down now runs AFTER `svc.Shutdown` |
| + in `agent.go`'s post-drain block (`stopHousekeeper()` called after the |
| + 60s-bounded `svc.Shutdown`), so a full stop can take up to ~80s |
| + (60s child-shutdown budget + the housekeeper's own stand-down/lease |
| + handling). If the unit's `TimeoutStopSec` is shorter, systemd SIGKILLs |
| + mid-stand-down, stranding the housekeeper lease for up to its 15-minute |
| + TTL instead of releasing it cleanly. Verify the pinfra unit file before |
| + rolling to a second box. (Task 3 review, ledgered.) |
| + |
| +7. **Retention coverage narrows to the housekeeper's app set the moment a |
| + second box arms.** `retentionPass` protects whatever apps are pinned in |
| + the CALLING box's own registry (`internal/host/retention.go`) — today, |
| + with one box, that registry is the deployment's whole app set. The |
| + instant a second box exists and the housekeeper lease can move to it, |
| + prune coverage narrows to whichever box currently holds the lease and |
| + its own registry's view, not the deployment-wide union. Protection |
| + never gets WEAKER (idempotent, keep-newest-N, fails toward keeping |
| + data) but coverage can miss apps the current holder's registry doesn't |
| + know about. Future fix, ordering matters: **union the pins across every |
| + box's registry BEFORE enumerating the bucket for deletion candidates**, |
| + not after — this is a real fix owed before arming a second box's |
| + housekeeper eligibility in production, not merely a documentation note. |
| + (Task 3 review, "concern-1" restated for rollout.) |
| + |
| +8. **#241 (restore-verify per-box keys) must land before a second platform |
| + box arms `carlos-hibernate-verify.timer`.** The bucket-writer audit |
| + found `control/restore-verify.json` is a single fixed key whose |
| + `Merge` step deterministically drops any host not in the CALLING box's |
| + own fleet view on every publish — not a race, a guaranteed clobber the |
| + moment two boxes with different hibernating-instance sets both publish. |
| + Ruled per-box keys (`control/restore-verify/<box>.json`, snapshots' |
| + shape); implementation deferred to #241, gate: land it before stage 5 |
| + (arming a second box). Note the timer being INSTALLED on every box |
| + (`internal/host/units/carlos-hibernate-verify.timer`) is not the same as |
| + it being ARMED/enabled — it's not in the bootstrap `enabledUnits` set |
| + today, which is the margin that makes "land #241 before a second box |
| + arms it" an operational gate rather than an already-blown deadline. |
| + |
| +9. **tls-alpn-01 is not end-to-end proven — residual, not a blocker.** |
| + Task 7's two-edge e2e (`f7e67fe`) proves challenge-anywhere over |
| + http-01 only; tls-alpn-01 is autocert's PREFERRED challenge type, so |
| + the headline "challenge-anywhere e2e-proven" claim is true only of the |
| + second-choice path in production. Real unit coverage exists — four |
| + named tests in `internal/edge/certsync_test.go`: |
| + `TestMirrorAnswersAnotherEdgesALPNValidation`, |
| + `TestMirrorALPNReadThroughOnlyForServedHosts`, |
| + `TestMirrorNeverServesATokenCertToAnOrdinaryHello`, |
| + `TestALPNTokenCertIsReadLiveNotFromTheSnapshot` — but not an e2e path. |
| + Carried forward rather than looped on; worth an e2e pass before or |
| + during stage 5 (arming flagship AU). |
| + |
| +10. **Issuer observability gap: which box holds the housekeeper lease has |
| + no readout beyond one agent log line.** `internal/host/housekeeper.go` |
| + logs `"agent: housekeeper: lease acquired — this box runs the |
| + deployment's global duties"` on acquisition and nothing else durable — |
| + no status-heartbeat field, no CLI/console surface. An operator asking |
| + "which box is minting certs right now" today has to grep agent logs on |
| + a specific box, which is exactly the box-local-only gap CLAUDE.md's |
| + operate-via-API-and-CLI rule calls out. Task 4's review flagged three |
| + issue-worthy items together, all tagged (T8), none filed as numbered |
| + issues yet — the controller should decide whether to file them before |
| + or alongside the stage-2 rollout: |
| + - which-box-is-issuer observability (a `held` bool on the status |
| + heartbeat) — the one described above; |
| + - `domains` package's `hostCertExpiry` reads blank on non-issuer boxes |
| + (it reads the local `DirCache` directly — `internal/domains/reconcile.go:446,452,724` |
| + — which is empty on a box that never issues locally; it needs to |
| + consult the mirror too); |
| + - spent ACME-challenge token litter has no TTL sweep on the mirror |
| + side — tokens are deleted from the bucket after validation by the |
| + issuer, but a crashed order's token has no independent expiry check |
| + from the mirror's perspective. |
| + |
| +**Issue comments** (drafted here; the controller posts them — see the |
| +task-8 report for the exact text of each). |
amadan