carlos / platform Public

docs: final review fix wave — real token key shapes, TTL walkback, runbook-flip ordering fix, None guard, housekeeper.json cleanup, bare-edge ruling

Six corrections from the whole-branch final review, all doc-only:

- writer-map row and Part 2's HTTP-01 section quoted a token prefix
  (control/acme-challenges/<token>) that was never implemented; the code
  publishes tokens under control/certs/ via autocert's own cache-key
  naming (<token>+http-01, <host>+token for ALPN) — corrected both sites
  and merged the acme-challenges writer-map row into control/certs/.
- dropped the promised TTL stamp on tokens (none exists); spent-token
  litter is now described as bounded and left for a later sweep, tracked
  as issue #243.
- rollout note 2 had the runbook flipping to new key shapes in the same
  window as the old-key DELETION, contradicting spec :91-93 (the runbook
  flips per-deployment, the moment THAT deployment rolls this build — an
  old-runbook restore in the gap would restore a stale snapshot). Only
  the deletion waits for both deployments; rewritten to match the spec.
- the rollout delete loops didn't guard against "None", the literal
  string the AWS CLI prints for an empty Contents[].Key — added the
  guard with the one-line why.
- plan's Global Constraints still named the voided control/housekeeper.json
  sketch; replaced with Task 3's RULED control/housekeeper-lease/lock.json.
- added a line beside the pubkey-less-box residual: a standalone
  `carlos edge` (nil IssuerGate) on a bucket-backed box is also ungated
  and also publishes — the same orphaned-second-issuer failure by a
  different door. Don't run bare edges on multi-edge deployments; ruling
  tracked as issue #244.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Paul Campbell pushed by paul@keymail.dev 2 months ago 896f2a5892c6b340c60c6552ce830cce9825dbbc parent 7db3f3e

2 files changed, +40 −17

docs/superpowers/plans/2026-08-20-multi-region-writer-map-and-certs.md +30 −11
docs/superpowers/specs/2026-08-20-multi-region-ingress-design.md +10 −6

diff --git a/docs/superpowers/plans/2026-08-20-multi-region-writer-map-and-certs.md b/docs/superpowers/plans/2026-08-20-multi-region-writer-map-and-certs.md

index a2bc2c735..2c874f80f 100644

--- a/docs/superpowers/plans/2026-08-20-multi-region-writer-map-and-certs.md

+++ b/docs/superpowers/plans/2026-08-20-multi-region-writer-map-and-certs.md

@@ -25,7 +25,8 @@ edge), existing fake-store test harness.

Key shapes verbatim from it: `control/registry-snapshot/<box>.db`,

`control/registry-snapshot/<box>.json`,

`control/registry-snapshots/<box>/<stamp>.db`, `control/certs/**`,

- `control/housekeeper.json`.

+ `control/housekeeper-lease/lock.json` (Task 3's RULED replacement for the

+ original `control/housekeeper.json` sketch — see Task 3).

- **Wire-shape changes are additive.** Old fixed snapshot keys are NOT

deleted by this plan; deletion happens after both deployments roll

(recorded in the rollout notes, not code).

@@ -603,7 +604,10 @@ checklist for the rollout that follows.

--prefix control/registry-snapshots/ --delimiter / \

--query 'Contents[].Key' --output text | tr '\t' '\n' | \

while read -r key; do

- [ -n "$key" ] && aws --profile keymail s3 rm "s3://$BUCKET/$key"

+ # "None" is what the AWS CLI's --output text prints for an empty

+ # Contents[].Key result (not an empty string) — without this guard

+ # an already-empty prefix runs `aws s3 rm s3://$BUCKET/None`.

+ [ -n "$key" ] && [ "$key" != "None" ] && aws --profile keymail s3 rm "s3://$BUCKET/$key"

done

aws --profile keymail s3 rm "s3://$BUCKET/control/registry-snapshot.db"

aws --profile keymail s3 rm "s3://$BUCKET/control/registry-snapshot.json"

@@ -620,19 +624,28 @@ checklist for the rollout that follows.

--prefix control/registry-snapshots/ --delimiter / \

--query 'Contents[].Key' --output text | tr '\t' '\n' | \

while read -r key; do

- [ -n "$key" ] && aws --profile titogo s3 rm "s3://$BUCKET/$key"

+ # Same "None" guard as the flagship loop above — the AWS CLI prints

+ # the literal string "None" for an empty Contents[].Key, not "".

+ [ -n "$key" ] && [ "$key" != "None" ] && aws --profile titogo s3 rm "s3://$BUCKET/$key"

done

aws --profile titogo s3 rm "s3://$BUCKET/control/registry-snapshot.db"

aws --profile titogo s3 rm "s3://$BUCKET/control/registry-snapshot.json"

```

-2. **teamtito runbook PR is owed at that same moment** (Task 1 ledger).

- `teamtito/carlos-infrastructure/RUNBOOK-edge-failover.md` parses the

- snapshot keys and must land the key-shape update in the same window as

- the deletion above, not before (old runbook still needs to name the old

- keys until the deletion happens) and not long after (a runbook naming

- dead keys is a live footgun during a restore). Two content requirements

- beyond the key rename, both from the ledger:

+2. **teamtito runbook flips to the new key shapes when TITO rolls this

+ build — not when both deployments have** (spec :91-93 is the ruling;

+ this note previously said "same window as the deletion", which

+ contradicted it and is corrected here). New keys and the runbook update

+ land in the same PR per the spec: the moment tito's deployment runs a

+ binary containing Task 1, its boxes stop writing the old fixed keys —

+ an operator following a runbook that still names the old keys during a

+ restore in that window would restore a stale (frozen, no-longer-updated)

+ snapshot instead of the live per-box one. The runbook must therefore

+ name the new keys as of tito's own rollout, independent of flagship's.

+ Only the old-key DELETION (note 1 above) waits for BOTH deployments —

+ deleting is destructive and irreversible, flipping a runbook's key names

+ is not. Two content requirements beyond the key rename, both from the

+ ledger:

- **Step 0 ordering rule**: restore-before-agent-start /

machine-id-reuse — a rebuilt standby must be restored from the dead

box's snapshot BEFORE the agent starts and mints a fresh machine id,

@@ -678,7 +691,13 @@ checklist for the rollout that follows.

invariant. No code fix landed this round (Task 4 fix-round-1 residual,

tagged T8); the rollout mitigation is operational: audit every box's env

before arming a second housekeeper-eligible box, and exclude (or hand a

- pubkey to) any bucket-backed box that lacks one.

+ pubkey to) any bucket-backed box that lacks one. Related: a standalone

+ `carlos edge` (run outside the agent — `IssuerGate` is nil, which

+ `issuerGate()` reads as "ungated") on a bucket-backed box is ALSO

+ ungated and ALSO publishes to `control/certs/**` unconditionally, the

+ same orphaned-second-issuer failure mode by a different door — don't run

+ a bare `carlos edge` against a deployment bucket on a multi-edge

+ deployment; ruling tracked in issue #244.

6. **Check `carlos-agent`'s systemd `TimeoutStopSec` covers the shutdown

chain (pinfra).** Housekeeper stand-down now runs AFTER `svc.Shutdown`

diff --git a/docs/superpowers/specs/2026-08-20-multi-region-ingress-design.md b/docs/superpowers/specs/2026-08-20-multi-region-ingress-design.md

index 20ce37f69..0cd27bb3a 100644

--- a/docs/superpowers/specs/2026-08-20-multi-region-ingress-design.md

+++ b/docs/superpowers/specs/2026-08-20-multi-region-ingress-design.md

@@ -73,7 +73,7 @@ in the plan's rollout notes, not code.

| `status/**` | lease-elected publisher | unchanged; verify the lease is deployment-scoped (it is bucket-scoped, which is the same thing). **Audited**: `internal/lease`'s `status-lease` election confirmed deployment-scoped (bucket-writer audit §1) |

| `releases/**` retention prune | every box | moves behind the **housekeeper lease** — five concurrent pruners racing a promote is a bug class we refuse to have. **Implemented, dormant** (`45a2ea6`..`aadd18e`) |

| usage metering day objects | the (single) edge | **audit**: if day-object keys are not box-scoped, they become `…/<box>/…` in the same shape as snapshots. **Confirmed already box-scoped** (`usage/edge/<box>/days/`, bucket-writer audit) |

-| `control/certs/**`, `control/acme-challenges/**` (new) | housekeeper | Part 2 — **implemented, dormant** (`7eff14b`..`f7e67fe`) |

+| `control/certs/**` (new; carries HTTP-01 tokens as `<token>+http-01` and tls-alpn-01 token certs as `<host>+token` — no separate acme-challenges prefix) | housekeeper | Part 2 — **implemented, dormant** (`7eff14b`..`f7e67fe`) |

| `control/edges/<box>.json` (new) | each box, own key only | Part 3 |

| `control/edge-mesh-key` (new) | housekeeper mints if absent (create-only CAS) | Part 3 |

@@ -158,11 +158,15 @@ certs never replicate to data buckets.

**HTTP-01 under latency DNS** — the one new mechanism. LE's validators

hit whichever edge is nearest to them, not the issuer. Before answering

an order, the issuer writes each active token to

-`control/acme-challenges/<token>`; EVERY edge serves

-`/.well-known/acme-challenge/*` from its mirror of that prefix. Tokens

-are deleted after validation and carry a TTL stamp so a crashed order

-never leaves permanent litter. The issuer remains the only box that

-talks to the CA.

+`control/certs/<token>+http-01` (the autocert cache-key naming — no

+separate acme-challenges prefix); EVERY edge serves

+`/.well-known/acme-challenge/*` from its mirror of that key space. Tokens

+are deleted from the bucket by the issuer after validation. There is no

+TTL stamp on them: a crashed order can leave a spent token behind, but

+the litter is bounded — one small object per abandoned order, under a

+prefix already listed and read on every mirror poll — and is left for a

+later sweep rather than solved here; tracked as issue #243 (filed). The

+issuer remains the only box that talks to the CA.

**Issuer death**: the lease expires and the next box's housekeeper takes

over with identical machinery. Distributed certs keep every edge serving