amadan
docs: HA posture ladder (#153) + multi-upstream addr routes (#152) design
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 file changed,
+247
−0
docs/superpowers/specs/2026-08-12-ha-posture-and-multi-upstream-routes-design.md+247 −0
diff --git a/docs/superpowers/specs/2026-08-12-ha-posture-and-multi-upstream-routes-design.md b/docs/superpowers/specs/2026-08-12-ha-posture-and-multi-upstream-routes-design.md| new file mode 100644 |
| index 000000000..9a79e0c4a |
| --- /dev/null |
| +++ b/docs/superpowers/specs/2026-08-12-ha-posture-and-multi-upstream-routes-design.md |
| @@ -0,0 +1,247 @@ |
| +# HA posture and multi-upstream routes |
| + |
| +**Date:** 2026-08-12 |
| +**Status:** Approved design |
| +**Prompted by:** #152 (multi-upstream, health-checked addr routes) and #153 |
| +(the edge box is a per-deployment SPOF), both filed out of the Rails-behind- |
| +the-edge PoC's §7 HA fallout (`2026-08-11-rails-behind-carlos-edge-poc-design.md`, |
| +PR #150). This doc answers both at once because they are two altitudes of |
| +one problem: #152 raises the availability of the *upstream* layer, #153 is |
| +the availability ceiling above it — the edge box that terminates TLS for |
| +every route on a deployment. |
| + |
| +## Context |
| + |
| +- Every deployment (flagship, tito, dev) routes all traffic through one |
| + edge box. On tito that box (`carlos-tito-ie`) fronts go.tito.io, every |
| + instance, the console alias, and — since the Rails PoC — potentially |
| + Tito Classic/Pro staging. New tenants widen the blast radius of an |
| + existing, deliberately recorded risk; they don't create it. |
| +- An `addr` route today is one SQLite row with one `Addr` |
| + (`internal/registry/registry.go`). The edge's `tcpProxy` |
| + (`internal/edge/edge.go`) sets no dial timeout and no `ErrorHandler`: a |
| + dead upstream surfaces as the stdlib's bare 502, still stamped with the |
| + route-level `X-Carlos-Version`. The only "resolve a live target per |
| + request" precedent is `fleetProxy`'s tunnel-channel resolution; the only |
| + retry machinery is unix-socket-specific (`dialUnixWithRetry`). |
| +- The control bucket already holds releases, channels, pointers, and — |
| + once PR #137 (route records) merges — provisioned-route desired state. |
| + Genuinely box-local state: the registry DB (`/data/registry.db`), minted |
| + TLS certs, instance data dirs, and the running processes. #137's |
| + `shapeMismatch` deliberately refuses record authority over box-scoped |
| + rows (`Backing`, `Tunnel`, `Caps`, `Platform`, `Canary`), so records |
| + alone cannot reconstruct a full registry today. |
| +- Addr routes are forbidden from `Hibernate` |
| + (`provision.ValidateShape`, on the #137 branch), so probing an addr |
| + upstream can never trigger a wake. That invariant is load-bearing for |
| + this design; relaxing it later would reintroduce the wake hazard. |
| + |
| +## Decisions taken during design (Paul, 2026-08-12) |
| + |
| +- **One doc for both issues** — the #153 ceiling decided first, #152 |
| + designed within it; implementation still lands separately. |
| +- **#153 is a ladder, not a point choice**: accepted-risk now, cold |
| + standby documented and built, warm standby sketched behind explicit |
| + triggers. |
| +- **#152 designed implementation-ready**, riding #137's record shape |
| + additively — not roadmap-priced shape-only. |
| +- **#152 mechanism: active probe + in-request dial retry** (option C). |
| + Active probing owns fail-away/fail-back and dashboard truth; a dial |
| + failure additionally ejects the upstream and retries in-request so box |
| + death never surfaces a 502 even between probe ticks. Probe-only and |
| + passive-only were considered and rejected (the first 502s during the |
| + probe gap; the second has dial-level health only and no clean |
| + rolling-deploy story). |
| +- **Cold-standby enabler: registry snapshot to the bucket now**; |
| + records-complete stays the long-term trajectory and retires the |
| + snapshot when it gets there. Records-complete-first was rejected as a |
| + large scope increase on an unmerged PR; runbook-only was rejected as |
| + rot-prone. |
| + |
| +--- |
| + |
| +## Part I — the availability ladder (#153) |
| + |
| +### Rung 0: today, written down as accepted risk |
| + |
| +One edge box per deployment terminates TLS for everything on it. Box |
| +death = every route on that deployment dark until recovery. Mitigations |
| +that are already real: EC2 auto-recovery (action item: verify it is |
| +actually enabled in tofu for both production boxes — it is not believed |
| +to be explicit today), the control bucket, and 2s serial convergence. |
| +This is not an oversight; the single-box posture is the recorded |
| +"multi-node failover is where an orchestrator earns its keep" decision. |
| +This rung exists so the risk is written where the migration evaluation |
| +can cite it. |
| + |
| +### Rung 1: cold standby (built by this design) |
| + |
| +**1. Registry snapshot to the bucket.** The box's converge loop gains a |
| +snapshot pass: every 5 minutes, if `registry.Fingerprint()` changed since |
| +the last snapshot, `VACUUM INTO` a temp copy and PUT it to the control |
| +bucket: current at `control/registry-snapshot.db`, with a sidecar JSON |
| +(`control/registry-snapshot.json`: created-at, fingerprint, box |
| +identity); prior generations at |
| +`control/registry-snapshots/<created-at>.db`, pruned beyond the newest |
| +8. This |
| +covers every row, including the box-scoped ones records cannot rebuild. |
| +Worst-case staleness is one interval; after restore, the reconcile pass |
| +corrects any record-backed drift. The snapshot is additive bucket |
| +content — nothing existing parses it, so no wire-shape risk. |
| + |
| +**2. TLS: re-mint, don't replicate.** Private keys never enter the |
| +bucket. A replacement box re-mints via HTTP-01 the moment it holds the |
| +EIP; wildcards go via the existing DNS-01/DNSimple path (token in SSM). |
| +Cost: a brief per-host TLS brownout on first hits after failover — |
| +accepted at this rung. Let's Encrypt duplicate-certificate limits are |
| +generous enough for disaster recovery. |
| + |
| +**3. Runbook + rehearsal.** A pre-written standby `.tf` (count = 0) lands |
| +in platform-infrastructure per deployment. The runbook, kept next to it: |
| + |
| +1. `tofu apply` the standby box. |
| +2. Remap the EIP to it. |
| +3. Restore `control/registry-snapshot.db` to `/data/registry.db`. |
| +4. Arm the path units (`update-platform-binary.sh` does this). |
| +5. Verify: `X-Carlos-Version` over the remapped EIP via |
| + `curl -sI --resolve`, then a console login. Never a cached answer. |
| + |
| +The runbook explicitly separates platform recovery from tenant data: |
| +instance DBs come back only via their own `Replication` setting; |
| +unreplicated instance state on the dead box is lost, and the runbook says |
| +so plainly. The rung is not "done" until rehearsed once on the dev |
| +deployment (Kass account) with the measured RTO recorded in this doc's |
| +follow-up. Target: under 60 minutes; the rehearsal replaces this guess |
| +with a number. |
| + |
| +### Rung 2: warm standby (trigger-gated, sketched only) |
| + |
| +A second box converging the same channels, promoted by EIP remap for a |
| +~1–2 minute RTO. Known open problems, recorded but deliberately not |
| +solved here: |
| + |
| +- A standby cannot answer HTTP-01 without the EIP — pre-minted certs need |
| + DNS-01-everything or a cert-sync channel. |
| +- The console must stay single-writer; a standby console is read-only or |
| + off. |
| +- Exec/unit-backed tenants cannot run on two boxes at once, so an honest |
| + warm standby is closer to a continuously-rehearsed cold one for |
| + everything except addr/static/tunnel routes. |
| + |
| +**Triggers to design rung 2 for real:** the first paying or SLA-bearing |
| +traffic on a deployment (the Rails production cutover is the named |
| +candidate), or the first time measured rung-1 RTO exceeds what a real |
| +tenant conversation can bear. |
| + |
| +--- |
| + |
| +## Part II — multi-upstream, health-checked addr routes (#152) |
| + |
| +### Record and registry shape (additive, rides #137) |
| + |
| +The route record gains two optional fields: |
| + |
| +```json |
| +{ |
| + "addrs": ["10.0.1.10:3000", "10.0.2.11:3000"], |
| + "health": { |
| + "path": "/up", |
| + "status": 200, |
| + "host_header": "", |
| + "interval_ms": 2000, |
| + "fail_after": 3, |
| + "rise_after": 2 |
| + } |
| +} |
| +``` |
| + |
| +Defaults: path `/`, status `200`, host header = route host, 2000 ms, 3, |
| +2. The legacy singular `addr` stays; `AddrsOrDefault()` folds it into a |
| +one-element list, so every record written before this change parses and |
| +behaves identically (the wire-shape rule). `health` on a single-addr |
| +route is legal and useful: it buys dashboard state and a 503 with |
| +`Retry-After` instead of today's bare 502. |
| + |
| +Registry side: two new TEXT columns — `addrs` (JSON array) and `health` |
| +(JSON blob) — both folded into `Fingerprint()` so the 2s watch notices |
| +changes, and both compared in `shapeMismatch` with canonicalized |
| +ordering. `routeFor`, the single shared mint/adopt builder on the #137 |
| +branch, is the one place the widening happens. |
| + |
| +### The prober |
| + |
| +For each route with `health` set (or `len(addrs) > 1`, which implies the |
| +default health config), the edge runs one background prober: direct |
| +`GET http://<addr><path>` per upstream with the route host as `Host` |
| +header and a 2 s client timeout. `fail_after` consecutive failures → |
| +upstream leaves the live set; `rise_after` consecutive passes → rejoins. |
| +Transitions are logged with the reason (dial error vs wrong status). |
| +Probes dial upstreams directly, never the edge's public host — combined |
| +with the no-hibernate invariant for addr routes, there is no wake hazard. |
| +Probers start and stop at `Reload()` when the fingerprint moves. |
| + |
| +### The request path |
| + |
| +Live-upstream resolution happens at RoundTrip time (the `fleetProxy` |
| +precedent), round-robin across the live set — these upstreams are |
| +replicas, so the edge does the balancing an NLB would have. On a **dial** |
| +failure (no bytes sent, so safe for any method) the upstream is ejected |
| +immediately — not waiting for the probe to notice — and the request |
| +retries once against another live upstream. Never a retry after bytes |
| +have flowed. Empty live set → 503 + `Retry-After: 2`, mirroring |
| +`errFleetDark`; the bare-502 path disappears for health-configured |
| +routes. `X-Carlos-Version` stays route-level as today: addr upstreams are |
| +processes CARLOS didn't build, and per-upstream version attribution is |
| +out of scope. |
| + |
| +### Surfaces |
| + |
| +Per-upstream state (up/down, last transition, consecutive-failure count) |
| +flows into the existing readings/sweep path, so the dashboard's instance |
| +page lists each upstream with its state. The instances API carries the |
| +same, and `carlos instances` shows `2/3 healthy`-style summaries. Route |
| +mutation goes through the console API / route-records door like |
| +everything else post-#137 — no box-local-only verb (the CLI no-infra |
| +rule). |
| + |
| +### Rolling deploys |
| + |
| +V1 needs no drain verb: the deploy stops the app on box 1 → the probe |
| +fails it away → deploy → probe passes → fail-back → next box. |
| +`fail_after × interval_ms` (~6 s at defaults) is the window in which |
| +requests can still land on a stopping box; the in-request dial retry |
| +covers them. An explicit per-upstream drain/maintenance flag is named as |
| +a future nicety, not built. |
| + |
| +--- |
| + |
| +## Testing |
| + |
| +- Unit tests for prober state transitions (fail-away at `fail_after`, |
| + fail-back at `rise_after`, dial-error vs wrong-status reasons) against |
| + `httptest` upstreams. |
| +- An edge e2e with two backends where killing one mid-request-stream |
| + produces zero 5xx responses. |
| +- Golden compatibility tests proving pre-change records and registry rows |
| + still parse and produce today's behavior (the rule this repo has paid |
| + for three times). |
| +- Snapshot pass: a test that a restored snapshot plus a reconcile pass |
| + reproduces the source registry's routes. |
| + |
| +## Sequencing |
| + |
| +1. PR #137 (route records) merges first — #152's shape rides it. |
| +2. Part I's snapshot + runbook are independent of #137 and #152 and can |
| + proceed in parallel; the rehearsal on the dev deployment closes rung 1. |
| +3. #152 edge work follows #137; the Rails staging boxes are the first |
| + live consumer, replacing their internal-NLB rung when it lands. |
| + |
| +## Out of scope |
| + |
| +- An orchestrator, in any form (#153 says so explicitly). |
| +- Data HA: Postgres/Redis/instance-DB replication have their own stories |
| + (RDS/Elasticache multi-AZ; the route `Replication` column). |
| +- Warm-standby design (rung 2) until a trigger fires. |
| +- Per-upstream version attribution in `X-Carlos-Version`. |
| +- A drain/maintenance verb for upstreams (recorded as future nicety). |
| +- Relaxing the addr-routes-never-hibernate invariant. |