| 1 | # Managed custom-domain connections — platform-owned verification and cert status |
| 2 | |
| 3 | **Date:** 2026-08-27 · **Status:** approved (design approved in conversation |
| 4 | 2026-08-27) · **Requested by:** the Tito Go team, via Paul |
| 5 | |
| 6 | **Closes:** #299 (managed custom-domain connections). **Folds in:** #287 |
| 7 | (attached custom domains are not published to edges files). |
| 8 | |
| 9 | **Builds on:** #151 (the global claim + `route_aliases` + the `domains` |
| 10 | door), #82 / 2026-08-12-domain-attach-design.md (first-route attach, the |
| 11 | DNS-pointing-as-evidence ruling this spec keeps), 2026-08-20-multi-region-ingress-design.md |
| 12 | (the edges files, the elected issuer, `control/certs/` and the mirror), |
| 13 | #300 / PR #301 (the per-host class mark this spec's alias mark follows), |
| 14 | #291 / PR #297 (the cert backfill that makes `control/certs/` complete), |
| 15 | #289 (the housekeeper gating this spec narrows), #207, #180 and #277 (the |
| 16 | exact-only cert admission bound this spec must not widen). |
| 17 | |
| 18 | ## 0. How this spec was arrived at |
| 19 | |
| 20 | Four adversarial review rounds (one per stage, then a whole-plan pass) ran |
| 21 | against earlier revisions. Five blockers were found and folded in; the |
| 22 | findings that changed the design are kept inline as rationale rather than |
| 23 | stripped, because in each case the wrong answer is the one a reader would |
| 24 | otherwise reach again — the `deploymentHosts`/`deploymentHost` confusion and |
| 25 | the `Generation` staleness bound especially. |
| 26 | |
| 27 | Scope rulings (Paul, 2026-08-26/27): |
| 28 | - Status + structured records + capability discovery. NOT TXT ownership |
| 29 | proof — claim authz stays first-writer-wins + DNS-pointing-as-evidence |
| 30 | (2026-08-12-domain-attach-design.md §2). |
| 31 | - #287 folded in as stage 1. |
| 32 | - Cert provisioning becomes proactive on the elected issuer. |
| 33 | |
| 34 | --- |
| 35 | |
| 36 | ## Stage 1 — make the inputs true |
| 37 | |
| 38 | ### 1a. Aliases reach the edges files, with a class mark (#287) |
| 39 | |
| 40 | `edgesFilePass` (internal/host/edgesfile.go:101) folds `reg.ListAliases()` |
| 41 | alongside `reg.List()`. |
| 42 | |
| 43 | Wire: `edgeHost` gains `Alias bool \`json:"alias"\`` — additive, ALWAYS |
| 44 | emitted, the convention #300's `Platform` mark set (a file with no `alias` |
| 45 | key is a pre-mark publisher, not a box publishing no aliases). |
| 46 | |
| 47 | Per alias: join to the primary via `RouteAlias.Host` against the `reg.List()` |
| 48 | map already in hand; skip unless the primary is `Status == "ready"`; emit |
| 49 | `edgeHost{Role:"home", Platform:false, Alias:true}`. `Platform:false` keeps |
| 50 | customer domains out of wildcard-parent derivation (`parentInput`, |
| 51 | homemap.go:343). Routes are written first; an alias never overwrites a route |
| 52 | key on the same box. |
| 53 | |
| 54 | **The fold rule — the security fix.** `foldHomes` (homemap.go:446-465) today |
| 55 | resolves a duplicate host by "first box in sorted order wins". New rule: **a |
| 56 | non-alias entry beats an alias entry; among equals, sorted order still |
| 57 | decides.** |
| 58 | |
| 59 | Why: `ReconcileAliases`' "a real route owns that host" refusal is per-box |
| 60 | (`byHost` is this box's routes — aliases.go:41,66), and the attach door |
| 61 | deliberately does not verify a route exists, because operator-minted routes |
| 62 | have no instance record (apidomains.go:106-113). So an account can claim a |
| 63 | hostname that is a real, UNCLAIMED route on another box. Today inert. Once |
| 64 | both boxes publish `role:"home"`, sorted-order-wins hands the victim's |
| 65 | hostname to the claimant on every other edge — under steering, most traffic. |
| 66 | The vulnerable set is exactly the claimless routes: operator `carlos add` |
| 67 | rows, `go.tito.io`, every route on a self-hosted box. |
| 68 | |
| 69 | **The mark must reach the reader, not just the writer.** The fold |
| 70 | rule alone is not enough: `edgesHostView` (homemap.go:134-141) and |
| 71 | `homeEntry` (homemap.go:143-149) carry only `Role`/`Platform`. Stage 1 must |
| 72 | carry `alias` into both, plus an accessor enumerating fold entries — |
| 73 | **all** of them, not only alias-marked ones, since 3a's corrected warm set is |
| 74 | "not under the apps domain" rather than "alias". Without this accessor stage |
| 75 | 3 is unimplementable. |
| 76 | |
| 77 | Note: the always-emit choice (never `omitempty`) matches what #300 |
| 78 | actually shipped — `Platform bool \`json:"platform"\`` with the comment "always |
| 79 | emitted (never omitempty), so a file with no key at all is definitely a |
| 80 | pre-mark publisher" (edgesfile.go:37-46). A review claimed #301 used |
| 81 | `omitempty`; the code says otherwise, and the always-emit reasoning is what |
| 82 | makes "old publisher" distinguishable from "nothing to declare". |
| 83 | |
| 84 | **The fold rule alone is NOT the whole fix.** It corrects what REMOTE |
| 85 | edges answer. It does nothing on the claimant's own box: `ReconcileAliases` |
| 86 | materializes the alias whenever the claim's primary exists locally and the |
| 87 | victim's host is not a LOCAL route, and request routing consults the local |
| 88 | upstream map before `Home()` — an exact local alias match wins regardless of |
| 89 | the fold. So for a pre-existing or door-race claim, any traffic landing on the |
| 90 | claimant's box serves the claimant's content over valid TLS (the mirror hands |
| 91 | every box the victim's cert). That is reachable: an operator route outside the |
| 92 | apps domain (the `go.tito.io` class — apps-domain hosts are door-denied) whose |
| 93 | owner followed our own steering guidance (CNAME to `edge.<apps-domain>`) |
| 94 | steers a share of traffic to every edge, the claimant's included. |
| 95 | |
| 96 | The authoritative refusal therefore belongs in **`ReconcileAliases`**, using |
| 97 | the same shared reader: refuse to materialize an alias whose host any live box |
| 98 | publishes as a non-alias `home` entry. That also retro-cleans pre-existing |
| 99 | claims on the next pass, which neither the fold rule nor a door check can do. |
| 100 | |
| 101 | **Defence in depth at the door.** The attach door additionally refuses such a |
| 102 | host at attach time. Best-effort (it races a concurrent mint), so it shortens |
| 103 | the window but is not the fix. Its refusal must be byte-identical |
| 104 | to the existing 409 "hostname unavailable" (apidomains.go:26-31), or it |
| 105 | becomes an existence oracle for operator routes. |
| 106 | |
| 107 | **Roll order: readers before publishers — the INVERSE of #301's rule.** |
| 108 | #301's mark failed CLOSED in an old reader (no mark → derive no parent), so |
| 109 | publishers could lead. This one fails OPEN: an old reader sees no `alias` key |
| 110 | and falls back to sorted-order-wins. Both halves are in one binary, so a |
| 111 | box-by-box roll would have an early-rolled box publishing into un-rolled |
| 112 | readers. Stage 1 therefore ships as TWO releases: |
| 113 | |
| 114 | 1. **Reader release** — fold preference + `alias` carried into |
| 115 | `edgesHostView`/`homeEntry`. Inert while nothing publishes the mark. |
| 116 | 2. **Writer release** — alias publication, plus 1b–1e below. |
| 117 | |
| 118 | ### 1b. #289 fixed the wrong thing — return the pass to every box |
| 119 | |
| 120 | The per-box divergence #289 saw was ONE INPUT (`cfg.EdgeIPs`, judged |
| 121 | per-box), not the pass. Gating the whole pass silenced N-1 writers instead of |
| 122 | fixing the input. Every section except the auto-purge writes only the box's |
| 123 | own registry, derived deterministically from shared bucket state: |
| 124 | |
| 125 | - §1a account materialization (`reg.PutAccount`/`DeleteAccount`, |
| 126 | reconcile.go:127-137) — gated, non-housekeeper account tables are frozen, |
| 127 | so `carlos accounts list` on those boxes lies. |
| 128 | - §1b route-account stamping (`reg.SetAccount`) — local. |
| 129 | - §1c trashed-route removal (`reg.Delete`, reconcile.go:244-258) — gated, |
| 130 | **a trashed app's routes on non-housekeeper boxes keep serving** until the |
| 131 | purge cooloff deletes the records, against the code's own "Trashed apps |
| 132 | stop routing NOW" and its "the worse betrayal" rationale. |
| 133 | - §2/§3 minting — mints into the LOCAL registry from `firstReadyUpstream` |
| 134 | over LOCAL routes. Per-box is CORRECT (it is how an everywhere-class app |
| 135 | gets its platform host on every box that serves it); housekeeper-only means |
| 136 | an app served only on non-housekeeper boxes never gets a platform domain. |
| 137 | - §3 canary-route removal — local, deterministic from the bucket pointer. |
| 138 | - §4/§5 DNS + cert stamps — local; the original Finding A. |
| 139 | |
| 140 | **Only §1d auto-purge is deployment-global** — it DESTROYS bucket objects, |
| 141 | so it stays a housekeeper duty, lifted into `domains.PurgeTrashed`. |
| 142 | |
| 143 | ### 1c. `pointing` judged against the deployment, not the box |
| 144 | |
| 145 | - `domains.Config.EdgeIPs []string` becomes `EdgeIPs func() []string`. |
| 146 | `domainsTickConfig` is read ONCE at arm (agent.go:1619) while the edge set |
| 147 | moves at runtime; a boot-frozen list is the bug being fixed. |
| 148 | - **The parser needs a home both the agent and the console can |
| 149 | import.** 1c originally put the fold in `internal/host` — but the console |
| 150 | is a separate process (`cmd/console`) that does not import `internal/host` |
| 151 | at all, and stage 2's records/`edge_ips` need the same fold. Putting a |
| 152 | reader in the console would make it the FOURTH parser of the shape, which |
| 153 | is the rule 1c invokes against itself. So: extract one exported reader |
| 154 | (package `internal/edgesfile`, or exported from `internal/steering`) and |
| 155 | have `internal/host`, the console, and `internal/edge` share it. |
| 156 | Also: `internal/edge` needs the EIP set too (stage 3's DNS gate), and |
| 157 | host imports edge, never the reverse — a shared low-level package is the |
| 158 | only shape that serves all three. |
| 159 | - Semantics: **UNION** the fold with `PLATFORM_EDGE_IPS`. The fold can only |
| 160 | widen, so a short or failed fold degrades to today's behaviour, never |
| 161 | below. (Contrast `internal/steering`, which fails CLOSED on a short fold |
| 162 | because a short fold there DELETES records; nothing is deleted here.) |
| 163 | - **Liveness bound — NOT `Generation`.** An earlier revision proposed ignoring an |
| 164 | edges file whose `Generation` is older than a few publish cadences. That |
| 165 | cannot work: `Generation` is stamped ONLY when a write happens |
| 166 | (edgesfile.go:145), the change gate suppresses writes for unchanged content |
| 167 | (`bytes.Equal(gate, *last)`), and `seedEdgesFileGate` exists precisely so an |
| 168 | unchanged file is never republished across restarts. On a stable fleet every |
| 169 | HEALTHY box's `Generation` is arbitrarily old, so that bound would ignore |
| 170 | every healthy box and silently degrade the fold to env-only — defeating its |
| 171 | own purpose, in all three stages that share it. |
| 172 | |
| 173 | Use box-snapshot freshness instead: `statusTick`'s `WriteBox` is |
| 174 | UNCONDITIONAL every tick (agent.go:1863) and `BoxStale` compares a moving |
| 175 | `GeneratedAt` (status/fleet.go:137-139) — a real liveness signal. A box |
| 176 | whose status snapshot is stale drops out of the EIP fold. This reuses the |
| 177 | loader and the staleness rule the console already applies |
| 178 | (apidomains.go:650-656), so there is one definition of "a box that still |
| 179 | counts", not two. |
| 180 | |
| 181 | Why it matters: nothing deletes a retired box's `control/edges/<box>.json`, |
| 182 | so without a liveness bound a retired box's file keeps a DEAD customer |
| 183 | domain reading `pointing` — a lying green. |
| 184 | - Guidance consistency: the console's `dnsTargets` (server.go:666) must read |
| 185 | the same fold, or the platform tells a customer to point at an address its |
| 186 | own checker will not credit. |
| 187 | |
| 188 | ### 1d. Carry the stamp's age, and give claim-backed ROUTES a reading |
| 189 | |
| 190 | **This is a stage-2 blocker discovered here.** `Metrics.Aliases` is |
| 191 | built exclusively from `Probes.Aliases` = `route_aliases` rows |
| 192 | (snapshot.go:435-441), and `RouteVersions` carries `Version` only |
| 193 | (snapshot.go:166). A **first-route claim** (`Claim.Route == ""`) mints a real |
| 194 | registry ROUTE, not an alias — so it never gets a reading, and |
| 195 | `carlos domains list` shows it `pending` forever. That is live today for |
| 196 | `carlos deploy -kind static -host <site>`, a shipped headline flow. |
| 197 | |
| 198 | The fix is nearly free: `CollectBox(routes []registry.Route, ...)` |
| 199 | (snapshot.go:376) already holds the routes, and `registry.Route` already |
| 200 | carries the `DNSStatus`/`CertExpiresAt` columns §4/§5 stamp. Fold claim- |
| 201 | backed route stamps into the same readings map in the loop already running, |
| 202 | with `Route: r.Host` — which is exactly what `claims.go` means by "an empty |
| 203 | `route` means this host is its own primary". |
| 204 | |
| 205 | Also: |
| 206 | - Add `DNSCheckedAt int64 \`json:"dns_checked_at,omitempty"\`` to |
| 207 | `status.AliasReading` (additive). `registry` has `dns_checked_at` |
| 208 | (registry.go:198) and nothing downstream reads it. |
| 209 | - Make the console fold prefer the FRESHEST reading. Today it is |
| 210 | last-box-wins by map iteration with no tie-break (apidomains.go:656-658), |
| 211 | so a frozen stamp can deterministically overwrite a fresh one. |
| 212 | |
| 213 | ### 1e. `pending-dns` honesty |
| 214 | |
| 215 | `dnsStatus` returns `""` for ANY resolver error (reconcile.go:465-476) — |
| 216 | NXDOMAIN and a box-side SERVFAIL are indistinguishable. Deriving |
| 217 | `pending-dns` from that blames the customer's record for our outage, the |
| 218 | failure the `unknown` state exists to prevent. Either add a third stamp value |
| 219 | separating "authoritative no answer" from "lookup failed" (an additive |
| 220 | registry + snapshot field, belonging in this release), or weaken the |
| 221 | `pending-dns` description to what is actually derivable. **Recommend the |
| 222 | third value** — it is small here and unpickable later. |
| 223 | |
| 224 | ### 1f. Repins |
| 225 | |
| 226 | - `internal/host/domains_duty_test.go`, the `domainsDuty` doc comment, and |
| 227 | the arm log (agent.go:1632) pin "only the lease holder may reconcile" for |
| 228 | the WHOLE pass. Invert to the new boundary, or a future reader re-gates it |
| 229 | citing the old test. |
| 230 | - New pins: an alias derives no wildcard parent; the fold prefers a route |
| 231 | over an alias; a not-ready primary publishes nothing; the stamping half |
| 232 | runs ungated on every box; a first-route claim gets a reading. |
| 233 | |
| 234 | ### What stage 1 does NOT do |
| 235 | |
| 236 | It does not make a latency-steered custom domain work end to end. The |
| 237 | client's TLS terminates at the NEAREST edge, before any backhaul; that edge |
| 238 | serves the alias's cert only from the mirror, and the mirror holds one only |
| 239 | after the issuer has ordered — which today needs a hello AT the issuer. Stage |
| 240 | 1 removes the structural blockers (challenge admission, challenge serving, |
| 241 | backhaul, and the fact that an alias homed off the issuer cannot be ordered |
| 242 | at all). The cert EXISTING is stage 3. |
| 243 | |
| 244 | --- |
| 245 | |
| 246 | ## Stage 2 — one status, structured records, capability |
| 247 | |
| 248 | **Rollout: stage 2's console roll TRAILS stage 3's fleet roll.** On |
| 249 | the original order, every pointing-but-certless domain would report |
| 250 | `issuing-cert` during the gap — naming platform activity that is |
| 251 | structurally impossible until the warm ships — and would advertise a |
| 252 | `failed` state nothing can produce. That is the same honesty test used to |
| 253 | rename `verifying`, applied to my own state name and failed. |
| 254 | |
| 255 | ### 2a. Vocabulary |
| 256 | |
| 257 | One `state` per domain, DERIVED on read, never stored: |
| 258 | |
| 259 | | state | means | derived from | |
| 260 | |---|---|---| |
| 261 | | `pending-dns` | no record found for this host yet | fresh reading, authoritative no-answer | |
| 262 | | `misdirected` | resolves, but not to this deployment | DNSStatus "elsewhere" | |
| 263 | | `issuing-cert` | points; no usable cert yet | pointing + no cert covering the host | |
| 264 | | `live` | points AND a current cert covers it | pointing + cert | |
| 265 | | `failed` | the platform tried and stopped | warm attempts exhausted (stage 3) | |
| 266 | | `unknown` | nothing fresh reports on this host | no non-stale reading, or lookup failed | |
| 267 | |
| 268 | `misdirected` rather than the issue's `verifying`: `verifying` named titogo's |
| 269 | ownership-proof phase, which Paul cut from scope, so the phase the word names |
| 270 | does not exist here — and reusing it for "your record resolves elsewhere" |
| 271 | would label a customer-side condition as platform activity. **This must be |
| 272 | said on #299** so titogo does not build against the proposed vocabulary. |
| 273 | |
| 274 | Plus `reason` (stable machine slug) and `detail` (human sentence). |
| 275 | |
| 276 | **`status` becomes legacy-frozen.** Today `status:"live"` means "some |
| 277 | non-stale box reported ANY reading" — it reads `live` while `dns_status` is |
| 278 | `elsewhere` and no cert exists. Post-change the same row can read |
| 279 | `status:"live", state:"misdirected"`. Keeping both is right (the wire rule; |
| 280 | `status`'s values are frozen), but the trap needs fencing: (a) a test |
| 281 | freezing `status`'s exact current semantics so nobody "fixes" it toward |
| 282 | `state`; (b) every first-party surface — CLI STATUS column, console Settings, |
| 283 | `carlos deploy`'s exit — moves to `state` in the same change; (c) the door |
| 284 | comment and the issue reply mark `status` legacy. |
| 285 | |
| 286 | ### 2b. Structured records |
| 287 | |
| 288 | `records: [...]` — what the customer must create. **Corrections:** |
| 289 | - `alias_target` is emitted ONLY when steering is armed. `edge.<apps-domain>` |
| 290 | exists only where the steering converger maintains it (server.go:525-534); |
| 291 | on a non-steering deployment the draft's unconditional field named a |
| 292 | hostname with no records. |
| 293 | - The array needs an **alternative-set marker**: with steering, the English |
| 294 | guidance offers CNAME *or* A records (view.go:163-165), and a flat array |
| 295 | cannot say "one of these two sets" — the records would contradict the |
| 296 | sentence beside them. |
| 297 | - A **wildcard-enabled claim needs its `_acme-challenge` CNAME** in the array |
| 298 | (apidomains.go:409-410) — the platform already instructs it and stamps |
| 299 | `delegation_status`; omitting it makes "what you must create" incomplete |
| 300 | against our own surface. |
| 301 | - **TTL 300**, with the reason stated: steering withdraws a dead edge in |
| 302 | ~90s, and a customer A-record TTL of 3600 nullifies that. |
| 303 | - The English `guidance` sentence changes in LOCKSTEP (apex guidance never |
| 304 | mentions ALIAS today), or 2d's one-story rule breaks on day one. |
| 305 | |
| 306 | Source: the same shared fold as 1c, with the same Generation staleness bound. |
| 307 | |
| 308 | ### 2c. Capability discovery — folded into the list answer |
| 309 | |
| 310 | **No new endpoint.** The original justification ("facts an app needs |
| 311 | before it holds a claim") conflated *no claim* with *no app binding*: every |
| 312 | caller is already app-bound, and `GET .../domains` answers fine with zero |
| 313 | claims. Folding a capability block into the list answer's top level gives |
| 314 | present-or-absent for free (an old console omits the keys; a console with no |
| 315 | door at all 404s — the exact signal `carlos deploy` already uses, |
| 316 | deploy.go:915-923), and needs no new role decision and no new limiter. |
| 317 | |
| 318 | ``` |
| 319 | {"domains": [...], |
| 320 | "deployment": {"managed_domains": true, "apps_domain": "...", |
| 321 | "steering_host": "...", "edge_ips": [...], |
| 322 | "apex_alias_target": "...", "delegation_zone": "...", |
| 323 | "states": [...]}} |
| 324 | ``` |
| 325 | |
| 326 | This also retires `domainsDoorMissing`'s 404-sniff (deploy.go:915-931) — our |
| 327 | own version of titogo's `TITO_EDGE_ATTACH_URL` guessing. |
| 328 | |
| 329 | ### 2d. Cert currency — an issuer-written index, not request-path reads |
| 330 | |
| 331 | **+The original 3d was wrong twice.** `control/certs/` entries |
| 332 | are `<private key PEM><chain>` (certsync.go:741-744), so "read the cert from |
| 333 | the bucket" means a Get and an x509 parse of a **key-bearing blob per host |
| 334 | per list call** — strictly more expensive than the box stamp it replaces, on |
| 335 | a door whose existing precedent caps *eight DNS lookups* per call |
| 336 | (domainListCNAMELookupCap). And a custom domain served by a **delegated |
| 337 | wildcard parent** has no per-host entry at all — its cert is |
| 338 | `wildcard-<parent>.pem` — so presence-by-key would read `issuing-cert` |
| 339 | forever for exactly the wildcard customers. |
| 340 | |
| 341 | Instead the console reads a small **index** — host → NotAfter — with no |
| 342 | private keys in it. |
| 343 | |
| 344 | **The index is WRITTEN BY STAGE 3, read by stage 2.** An earlier revision described |
| 345 | it inside stage 2, whose rollout entry is a console roll only — leaving the |
| 346 | writer assigned to no release at all. It is issuer-side fleet code: it ships |
| 347 | and rolls in stage 3 (§3h), and stage 2 merely consumes it. Stage 2 cannot |
| 348 | land before stage 3 for this reason as well as the `issuing-cert` one. |
| 349 | |
| 350 | Storage shape differs from 3e's warm records, deliberately: NotAfter is |
| 351 | **fully derivable** from the mirrored cache on every pass, so a single |
| 352 | rebuilt-every-pass object is safe under last-writer-wins — a second issuer |
| 353 | recomputes the same answer. Attempt counts are NOT derivable, which is why |
| 354 | 3e needs per-host CAS keys. Two shapes, two reasons, stated so neither is |
| 355 | copied to the other. |
| 356 | |
| 357 | The index must replicate `hostCertExpiry`'s coverage logic |
| 358 | (reconcile.go:744-755): per-host entry OR covering wildcard PEM — **and** |
| 359 | check `<host>+rsa`, a gap `hostCertExpiry` itself has. Fix that gap at |
| 360 | the source in stage 1, where the stamps are already being touched, so the |
| 361 | box-stamped path and the index cannot disagree. |
| 362 | |
| 363 | ### 2e. Scale and the poller contract |
| 364 | |
| 365 | **The list door cannot carry the poller stage 2 invites.** |
| 366 | - `GET .../domains` sits behind `instanceGate` — `cliInstanceBurst` 60/min, |
| 367 | keyed on the token hash and **shared with the instance verbs**. A status |
| 368 | poller would eat the budget its signups and instance-creates run on. |
| 369 | - Every list call is `ListClaims` = one List plus **one Get per claim in the |
| 370 | whole bucket**, filtered in memory afterwards (claims.go:175-199). |
| 371 | - Required: rebuild the list on the #165 fast-path claims index (already |
| 372 | referenced at apidomains.go:436-439) **or** add a per-host |
| 373 | `GET .../domains/{host}` reading exactly one claim object; state a poll |
| 374 | cadence; move the poller's budget off the shared instance ceiling. |
| 375 | |
| 376 | ### 2f. Detach |
| 377 | |
| 378 | The issue names "detach-failed-still-remove". After a 200 the claim is gone |
| 379 | and the host leaves the list while edges still serve it until reconcile and |
| 380 | the cert mirror catch up — a poller cannot tell "draining" from "never |
| 381 | existed". Ruling: detach stays fire-and-forget and the answer says so; the |
| 382 | alternative (a draining state) would require a tombstone the claim shape |
| 383 | deliberately does not keep. |
| 384 | |
| 385 | ### 2g. Surfaces |
| 386 | |
| 387 | `carlos domains list` gains a State column; console Settings shows the same |
| 388 | state and records; `carlos deploy`'s DNS exit branches on `state` instead of |
| 389 | `DNSStatus == "pointing"` (deploy.go:958-975), which currently cannot tell |
| 390 | "your DNS is wrong" from "your DNS is right, the cert is coming". All read |
| 391 | ONE derivation function. |
| 392 | |
| 393 | --- |
| 394 | |
| 395 | ## Stage 3 — the platform provisions the cert |
| 396 | |
| 397 | ### 3a. The warm set — alias-derived |
| 398 | |
| 399 | **The draft cited the wrong symbol.** `deploymentHosts()` (plural, |
| 400 | edge.go:738-756) excludes aliases by design (`!up.alias`) and its doc |
| 401 | comment says "Not to be confused with deploymentHost (singular)". The |
| 402 | singular (edge.go:831-833) is the set meant — but it is a PREDICATE with no |
| 403 | enumerator. Built naively it would also warm **every canary host** (three |
| 404 | labels under the apps domain, no wildcard parent covers them, DNS gate |
| 405 | passes) — a per-host cert and CT-log entry for names that are today lazily |
| 406 | issued and rarely visited. That, not custom domains, is where a first-run |
| 407 | burst would come from. |
| 408 | |
| 409 | **An earlier revision said "aliases only". That is also wrong** — it drops |
| 410 | first-route claims, which mint a real ROUTE, not an alias (1d). So |
| 411 | `carlos deploy -kind static -host <site>`, the shipped headline flow this |
| 412 | design itself cites, would never have its cert proactively ordered OR |
| 413 | renewed, and stage 2's `failed` would be unreachable for that whole class. |
| 414 | |
| 415 | The right predicate is simpler than either: **exact hosts this deployment |
| 416 | serves that are NOT under the apps domain.** A custom domain is a foreign |
| 417 | domain by definition — the attach door refuses apps-domain hosts outright |
| 418 | (apidomains.go:63-69) — so this one test covers aliases and first-route |
| 419 | claims alike, and excludes canary hosts, platform hosts and everything a |
| 420 | wildcard covers without needing to reason about label depth. |
| 421 | |
| 422 | - own box: keys of `e.current()` (routes AND aliases), |
| 423 | - remote: entries from the home-map fold (1a's accessor, which must |
| 424 | therefore expose ALL entries, not only alias-marked ones), |
| 425 | - minus `*.` rows (`Allowed` refuses them; a keys-of-map walk includes them), |
| 426 | - minus anything under the apps domain, |
| 427 | - minus names covered by a CONFIGURED wildcard parent |
| 428 | (`CARLOS_WILDCARD_DOMAINS`, e.g. `go.tito.io`) tested **by SHAPE** against |
| 429 | `currentParents()`, NOT `certFor`: `certFor` returns nil for a parent whose |
| 430 | cert is not yet obtained, so a new account's apps would all be warmed |
| 431 | per-host in the window before their wildcard lands. |
| 432 | |
| 433 | ### 3b. The pass |
| 434 | |
| 435 | A per-host sibling of `wildcardManager` (wildcard.go:416-495), with three |
| 436 | deliberate departures: |
| 437 | |
| 438 | - **No holding-context claim.** 3a originally copied "the order runs |
| 439 | under the holding's context" — impossible here: the trigger is |
| 440 | `m.GetCertificate`, which takes no context and builds its own |
| 441 | `context.Background()` with a 5-minute timeout (autocert.go:273-275). The |
| 442 | wildcard manager can honour the holding because it drives the ACME client |
| 443 | itself. Instead: re-check the gate BETWEEN orders (the backfill's |
| 444 | per-entry precedent, certsync.go:886) and accept a ≤5-minute overhang, |
| 445 | inside the double-holder envelope `issuerHostPolicy` already documents. |
| 446 | Consequence stated honestly: a finishing order's `cachePut` re-checks the |
| 447 | gate, so the cert lands local-only and the new issuer re-orders it — |
| 448 | bounded waste, not a hazard. |
| 449 | - **The warm calls `GetCertificate` for every warm-set host EVERY |
| 450 | pass, unconditionally** — this is what owns renewal. autocert cannot renew |
| 451 | through a skip-if-fresh check: `m.cert` returns any unexpired cached cert, |
| 452 | so a cert 29 days from expiry is a cache hit and no order happens. Per-host |
| 453 | renewal is exclusively `domainRenewal` timers, which die on every fleet |
| 454 | roll and re-arm only via a `GetCertificate` that reaches `m.cert`. On a |
| 455 | steering deployment a custom domain's visitors may NEVER land on the |
| 456 | issuer, so after a roll nothing re-arms the timer and the cert expires at |
| 457 | day 90 silently. An unconditional call is one `cacheGet` plus `startRenew` |
| 458 | — cheap, and exactly the re-arm. Pin it with a test. |
| 459 | - **No cache-load-before-gate step.** Unlike the wildcard manager, the warm |
| 460 | SERVES nothing (per-host serving is the mirror plus autocert), so a |
| 461 | non-issuer scanning caches is pure waste. It stands down at the gate. |
| 462 | |
| 463 | Retained: per-host hourly backoff on failure; **a harder backoff on an |
| 464 | ACME `rateLimited` error** than on a validation failure, since hourly retries |
| 465 | into a rate limit compound it; a per-pass order cap that returns a SHORT |
| 466 | delay while capped work remains (the wildcard `pass()` precedent) so a |
| 467 | backlog does not trickle out over days. |
| 468 | |
| 469 | **Arm only when `e.staticStore != nil`.** An agent-less `carlos edge` |
| 470 | is ungated (`issuerGate` nil → yes), and would otherwise start proactively |
| 471 | ordering on hobbyist boxes. |
| 472 | |
| 473 | ### 3c. The DNS gate |
| 474 | |
| 475 | Resolve each host against the deployment edge set and order only when it |
| 476 | points. LE's failed-validation limit is 5 per account per hostname per hour |
| 477 | and the ACME account is shared per issuer — ordering for a half-finished DNS |
| 478 | is how one customer burns everyone's budget. |
| 479 | |
| 480 | - **Require ALL resolved answers in the set**, not any: partial |
| 481 | pointing passes an any-match gate and then fails validation from LE's |
| 482 | vantage, burning exactly the budget the gate protects. |
| 483 | - **The input does not exist in `internal/edge` yet.** 1c's fold lands |
| 484 | behind `internal/host`, and host imports edge, never the reverse. 1c's |
| 485 | shared reader package is what makes this reachable; the edge also already |
| 486 | parses `EIP` in `edgesFileView` (homemap.go:125-130), so the accessor can |
| 487 | live on the home map with 1c's Generation bound and env union. |
| 488 | |
| 489 | ### 3d. The ECDSA trap — verified |
| 490 | |
| 491 | `m.GetCertificate(&tls.ClientHelloInfo{ServerName: host})` with an EMPTY |
| 492 | hello: `supportsECDSA` returns **false** (nil `SignatureSchemes` and nil |
| 493 | `SupportedCurves` skip their checks; the `CipherSuites` loop finds nothing) |
| 494 | → `certKey{isRSA:true}` → the warm mints an **RSA** cert under `<host>+rsa` |
| 495 | while browsers ask for `<host>`. The order is spent, nothing is satisfied, |
| 496 | and the first visitor still triggers a second issuance. |
| 497 | |
| 498 | Fix: advertise an ECDSA suite in the synthetic hello. Independently confirmed |
| 499 | against x/crypto v0.52.0 (go.mod:17). Everything else checks out: `ServerName` |
| 500 | must be non-empty and contain a dot; idna/lowercasing applied internally; nil |
| 501 | `SupportedProtos` → `wantsTokenCert` false; no `Conn` dereference. The repo |
| 502 | already ports `supportsECDSA` as `helloSupportsECDSA` (certsync.go:1285-1327) |
| 503 | — pin PARITY between the two copies, plus the non-`+rsa`-key assertion. |
| 504 | |
| 505 | ### 3e. Where warm outcomes go |
| 506 | |
| 507 | **Per-host CAS keys, not one file.** `control/cert-warm/<host>.json`, |
| 508 | written with the `putCert` CAS idiom. A whole-file map has no per-host |
| 509 | last-writer-wins safety: two transient holders, or a failover, overwrite each |
| 510 | other wholesale, and a takeover issuer starting from empty in-memory state |
| 511 | erases every attempt count. Each record carries a **written-at stamp**, and |
| 512 | the console applies an age bound — a wedged issuer must not read as "warm |
| 513 | quiet". The writer prunes records for detached hosts. |
| 514 | |
| 515 | Verified: the fleet door is an **allowlist** (`aclRead`, |
| 516 | storedoor.go:383-397), so any new `control/` key is denied by default — but |
| 517 | `TestDoorNeverServesCertMaterial` (storedoor_test.go:515-567) must gain the |
| 518 | new keys. Cert writes bump no serial (certsync.go:25-28) — correct here too. |
| 519 | The bucket-writer audit and the ingress spec's writer table need a new row |
| 520 | ruled "issuer" before the write exists. |
| 521 | |
| 522 | ### 3f. Removal — expiry-bounded, never inference-driven |
| 523 | |
| 524 | Nothing ever deletes `control/certs/<host>` for a detached domain (`Delete` |
| 525 | fires only for spent tokens). The backfill will not re-publish it |
| 526 | (`ownsCertName` refuses), but the object lingers, the mirror serves it to |
| 527 | matching SNI until expiry, and the expired object then lingers forever. |
| 528 | |
| 529 | **The sweep's predicate must be EXPIRY, not detachment.** A sweep |
| 530 | keyed "no claim exists for this host" would delete every non-claim-backed |
| 531 | cert in the deployment — every platform host, every wildcard PEM, the account |
| 532 | keys. Worse, registry lag or a stale edges read could take down a live host's |
| 533 | certs fleet-wide via the mirror. This repo's own rule is that inference never |
| 534 | destroys owner property (reconcile.go §1c). |
| 535 | |
| 536 | So: delete only entries whose LEAF IS ALREADY EXPIRED plus a margin. That |
| 537 | also dissolves the "mirror serves it until expiry" concern by construction, |
| 538 | and makes a wrong deletion harmless — the entry was already dead. Never |
| 539 | "detached implies delete". Prune cert-warm records on the same pass, on the |
| 540 | same terms. Issuer-side; #243's token-litter ruling is the precedent for |
| 541 | assigning it, not for its predicate. |
| 542 | |
| 543 | ### 3h. The cert index (read by stage 2 §2d) |
| 544 | |
| 545 | On every warm pass the issuer rebuilds `control/cert-index.json`: |
| 546 | host → NotAfter, for every host in the warm set, resolved through |
| 547 | `hostCertExpiry`'s coverage rules (per-host entry, `<host>+rsa`, or a |
| 548 | covering wildcard PEM). Whole-file and last-writer-wins is safe here because |
| 549 | the content is fully derived — a second issuer recomputes the same answer |
| 550 | (contrast 3e, whose attempt counts are not derivable and need per-host CAS). |
| 551 | Carries a written-at stamp; the console applies an age bound so a wedged |
| 552 | issuer does not read as "everything current". Added to |
| 553 | `TestDoorNeverServesCertMaterial`'s pinned list and to the bucket-writer |
| 554 | audit's table, ruled "issuer", before the write exists. |
| 555 | |
| 556 | ### 3g. Observability |
| 557 | |
| 558 | Failures are 3e. Add the success line (one per obtained host) and a per-pass |
| 559 | summary, matching the wildcard manager's logging discipline. |
| 560 | |
| 561 | --- |
| 562 | |
| 563 | ## The shared edges-file reader — the plan's load-bearing artifact |
| 564 | |
| 565 | Six things stand on it: 1a's `ReconcileAliases` refusal and door check, 1c's |
| 566 | EIP fold, 2b's records, 2c's `edge_ips`, and 3c's order gate. Earlier revisions gave the |
| 567 | riskiest new surface the least design, so it gets its own contract: |
| 568 | |
| 569 | - **Where:** one small exported package (`internal/edgesfile`) that owns the |
| 570 | wire struct. `internal/host` (writer), `internal/edge`, the console and |
| 571 | `internal/domains` all import it. `internal/steering` keeps its own |
| 572 | deliberately narrow view (it reads no `Hosts` — steering.go:57-60) and is |
| 573 | named here as a knowing exception rather than left unlisted, per the |
| 574 | consumer-list rule. |
| 575 | - **What it answers:** the live edge-IP set (union with `PLATFORM_EDGE_IPS`), |
| 576 | and per-host "is this published as a non-alias home by a live box". |
| 577 | - **Liveness:** the box-snapshot join above, never `Generation`. |
| 578 | - **Failure:** degrades to env-only, never to empty — nothing here deletes, |
| 579 | so widening is always the safe direction. |
| 580 | - **Cost:** cached with a refresh cadence on the console side; the list door |
| 581 | must not pay a per-request fold. |
| 582 | |
| 583 | ## Rollout order |
| 584 | |
| 585 | 1. **Stage 1 reader** — fold preference + `alias`/full-entry accessor in the |
| 586 | reader types. Inert no-op roll. |
| 587 | 2. **Stage 1 writer** — split into independently green PRs rather than |
| 588 | one: (i) alias publication + fold + `ReconcileAliases` refusal + door |
| 589 | check; (ii) the #289 inversion + `PurgeTrashed`; (iii) the shared reader + |
| 590 | live EIP set + `dnsTargets`; (iv) route readings + stamp age + |
| 591 | resolver-error value + the `+rsa` fix. Fleet roll. |
| 592 | 3. **Stage 3** — the warm and the cert index. Fleet roll. |
| 593 | 4. **Stage 2** — the API, CLI and console. Console roll plus a CLI release |
| 594 | (tap/apt/website, per the standing sync rule), LAST, so no state the |
| 595 | deployment cannot yet produce is ever advertised. |
| 596 | |
| 597 | Stage 1 ships real value alone (a live security gap, a live regression, a |
| 598 | live false-negative). Stages 3 and 2 together are what make `live` mean what |
| 599 | #299 asks it to mean. |
| 600 | |
| 601 | ## Live checks owed before the first armed roll |
| 602 | |
| 603 | - Count `control/certs/` keys against the enumerated warm set on the flagship |
| 604 | issuer — confirm the first pass orders nothing for the existing estate. |
| 605 | - Confirm which box currently holds the housekeeper lease on flagship and |
| 606 | tito, to size 1b's live blast radius. |
| 607 | |
| 608 | --- |
| 609 | |
| 610 | ## Addendum — what was built, and where it diverged (2026-08-27) |
| 611 | |
| 612 | The spec above was written against `main` at `da9a7ee`. By the time it was |
| 613 | implemented `main` had moved to `4235e9f` and had independently built most of |
| 614 | stage 1 (#287's alias publication, #303's per-box stamps, the |
| 615 | `Reconcile`/`StampHealth` split). Stage 1 was therefore re-derived as a |
| 616 | salvage — see `.superpowers/sdd/salvage-onto-main/progress.md` — and stages 2 |
| 617 | and 3 were built against the newer base. The divergences below are decisions, |
| 618 | not omissions, and each names what it would cost to be wrong. |
| 619 | |
| 620 | ### The vocabulary gained `checking` and never gained `failed` |
| 621 | |
| 622 | `checking` (attached; no edge has reported yet) was split out of `unknown`. |
| 623 | Both mean "no verdict", but they are opposite news: one is the ordinary first |
| 624 | few minutes after an attach, the other is a fault on our side. Collapsed, |
| 625 | every successful attach opened with the word "unknown" — which an e2e test |
| 626 | caught by asserting on the freshly-attached row. |
| 627 | |
| 628 | `failed` is absent. It would mean "the platform tried and stopped", which |
| 629 | requires §3e's per-host warm-outcome records — not built (below). The spec's |
| 630 | own honesty test, the one that renamed `verifying`, applies to it directly: |
| 631 | a state nothing can produce must not be advertised. A test now pins that the |
| 632 | advertised set and the producible set are exactly equal, so `failed` returns |
| 633 | with §3e and not before. |
| 634 | |
| 635 | ### §2d's cert index (§3h) was NOT built; stamps are folded instead |
| 636 | |
| 637 | The spec argued for an issuer-written host→NotAfter index because reading |
| 638 | `control/certs/` on the request path means a Get and an x509 parse of a |
| 639 | key-bearing blob per host, and because a wildcard-covered domain has no |
| 640 | per-host entry at all. |
| 641 | |
| 642 | Both objections stand — against reading the BUCKET. They do not apply to the |
| 643 | path actually taken: the boxes already stamp cert expiry into the registry, |
| 644 | `status.LoadBoxes` already carries those stamps to the console, and the |
| 645 | console already folds them across boxes (the freshest-stamp fold). Stage 2 |
| 646 | reads that fold. The wildcard case is covered because `hostCertExpiry` |
| 647 | already checks the covering wildcard PEM — and the `<host>+rsa` gap the spec |
| 648 | flagged was fixed at the source, so the box-stamped path and any future index |
| 649 | cannot disagree. |
| 650 | |
| 651 | Cost if wrong: the fold is bounded by box staleness, so a deployment whose |
| 652 | boxes all go stale reads `checking` rather than `live`. That is the correct |
| 653 | failure direction, and §3h remains available if the fold proves too coarse. |
| 654 | |
| 655 | Not building it also avoided a new bucket object, a new writer-table row, a |
| 656 | new `aclRead` allowlist entry and an update to |
| 657 | `TestDoorNeverServesCertMaterial` — none of which buy anything the existing |
| 658 | stamps do not already carry. |
| 659 | |
| 660 | ### §3e and §3f were follow-ups; §3e is now built |
| 661 | |
| 662 | §3e (per-host `control/cert-warm/<host>.json` CAS records) and §3f (the |
| 663 | expiry-bounded cert sweep) shipped after the first change, not with it. The |
| 664 | warm worked without either: it ordered, it renewed, it backed off per host, |
| 665 | and it logged. What was missing was durable observability of warm outcomes |
| 666 | across a failover, and the pruning of certs for long-detached hosts. |
| 667 | |
| 668 | **§3e is built (2026-08-29).** `internal/certwarm` owns the wire shape for |
| 669 | both sides, the issuer writes a record through the same CAS loop `putCert` |
| 670 | uses (extracted as `putCAS`), and the console folds it into `domainState`. |
| 671 | Three decisions the spec did not settle, decided here: |
| 672 | |
| 673 | - **A record exists only while a host is FAILING.** Success deletes it. That |
| 674 | makes absence the healthy case, so the console's fold costs one object per |
| 675 | BROKEN hostname rather than one per domain — which is what makes it safe |
| 676 | to add to a list door that already pays one Get per claim (#173). The |
| 677 | per-host door reads one key instead of listing, preserving the property |
| 678 | #328 exists for. |
| 679 | - **Freshness is judged against `next_attempt_at`, not `written_at`.** A |
| 680 | rate-limited host is deliberately not retried for twelve hours, and a bound |
| 681 | on write time alone would call that healthy silence stale every time. The |
| 682 | accepted limit: an issuer that wedges immediately after writing a |
| 683 | rate-limited record is believed for up to fourteen hours. Closing that |
| 684 | needs a per-pass heartbeat object — a second key and a second writer-table |
| 685 | row for a sharper edge on an already rare case. |
| 686 | - **`failed` fires at three consecutive failures, or immediately on a rate |
| 687 | limit.** Three is about three hours of trying. The rate-limit exception is |
| 688 | because the retry is twelve hours away, and a member watching a "usually a |
| 689 | minute or two" message for half a day is worse served than one told |
| 690 | plainly what is happening. |
| 691 | |
| 692 | `failed` never overrides a customer-side condition (`misdirected`, |
| 693 | `pending-dns`) and never overrides a live certificate: it is only ever the |
| 694 | alternative to `issuing-cert`. The vocabulary honesty test now produces it |
| 695 | from an input the platform can actually produce, which is the same bar every |
| 696 | other state clears. |
| 697 | |
| 698 | **§3f is built (2026-08-29)**, in `internal/edge/certsweep.go`, and it took |
| 699 | #243's spent-token litter with it — one issuer-side pass over |
| 700 | `control/certs/`, two predicates, because the two kinds of litter are not the |
| 701 | same kind of thing: |
| 702 | |
| 703 | - **Spent challenge tokens** (#243) go on AGE alone. A validation's whole |
| 704 | life is seconds and autocert's order context is five minutes, so the |
| 705 | one-hour TTL is generous by two orders of magnitude — deliberately, since |
| 706 | deleting a token out from under a live validation burns the order. |
| 707 | - **Certificates** go only on their own leaf's expiry, plus a month. §3f's |
| 708 | ruling is unchanged and is the reason the file opens with it: a sweep keyed |
| 709 | "no claim exists for this host" would delete every platform host's cert, |
| 710 | every wildcard PEM and the account keys, and a registry lag could take a |
| 711 | live host down fleet-wide through the mirror. |
| 712 | |
| 713 | Three bounds the spec did not name: |
| 714 | |
| 715 | - **A parse floor.** Nothing younger than sixty days is read at all. autocert |
| 716 | renews at ~30 days remaining on a 90-day certificate, so a healthy entry's |
| 717 | object stamp never gets that old — which makes the steady-state cost one |
| 718 | listing and zero reads, instead of the per-object N+1 this would otherwise |
| 719 | be. An entry that IS that old is one nothing is renewing, which is exactly |
| 720 | the population worth reading. |
| 721 | - **Anything unreadable is left alone.** A sweep that deletes what it cannot |
| 722 | parse is a sweep that deletes tomorrow's format. |
| 723 | - **Fifty deletions per pass.** A bug in the predicate removes at most fifty |
| 724 | objects before somebody reads the log. |
| 725 | |
| 726 | The gate check moved above the empty-warm-set return in `pass()`: token |
| 727 | litter and dead certs exist in a deployment with no custom domains at all, |
| 728 | and the old ordering would have meant such a deployment never swept. A |
| 729 | non-issuer still sweeps nothing. |
| 730 | |
| 731 | ### The shared reader is a READER, not the wire-shape owner |
| 732 | |
| 733 | `internal/edgesfile` reads the deployment-wide edges picture. It does not own |
| 734 | the writer's struct: `internal/host` had developed that file substantially |
| 735 | (wildcard parents, withheld-alias throttling, EIP-reuse-on-blank), and |
| 736 | re-plumbing it onto a shared type would have fought a file under active |
| 737 | development for no gain to #299. The two shapes are joined by a test instead, |
| 738 | which also preserves the additive-wire property — the writer can add a field |
| 739 | and roll before every reader understands it. |
| 740 | |
| 741 | ### Rollout |
| 742 | |
| 743 | The spec required stage 3's fleet roll to precede stage 2's console roll, so |
| 744 | that `issuing-cert` never names activity that cannot happen. Both ship in one |
| 745 | branch, so the constraint applies to the ROLL and not the merge: **roll the |
| 746 | fleet before the consoles.** Rolled the other way, every pointing-but-certless |
| 747 | domain reports `issuing-cert` while nothing is issuing. |
| 748 | |
| 749 | ### Still owed |
| 750 | |
| 751 | - §2e's poller contract: `GET .../domains` remains behind `instanceGate`'s |
| 752 | shared 60/min budget and still costs one Get per claim in the bucket. A |
| 753 | status poller must not be pointed at it until the #165 claims index or a |
| 754 | per-host door lands. **This must be said on #299** — it is the one part of |
| 755 | the issue's ask this change does not make safe. |
| 756 | - `deploy.go`'s `domainsDoorMissing` 404-sniff could now be retired in favour |
| 757 | of the capability block. Left in place: it still works, and retiring it is |
| 758 | not free of risk against older consoles. |
| 759 | - The console Settings page still renders the pre-state wording. The CLI and |
| 760 | the deploy exit moved; the page has not. |
| 761 | |
amadan