amadan
Merge pull request #154 from carlosframework/ha-posture-spec
docs: HA posture ladder + multi-upstream addr routes — design record
3 files changed,
+1408
−0
docs/superpowers/plans/2026-08-12-multi-upstream-routes.md+439 −0docs/superpowers/plans/2026-08-12-registry-snapshot-cold-standby.md+718 −0docs/superpowers/specs/2026-08-12-ha-posture-and-multi-upstream-routes-design.md+251 −0
diff --git a/docs/superpowers/plans/2026-08-12-multi-upstream-routes.md b/docs/superpowers/plans/2026-08-12-multi-upstream-routes.md| new file mode 100644 |
| index 000000000..0b2b6f49f |
| --- /dev/null |
| +++ b/docs/superpowers/plans/2026-08-12-multi-upstream-routes.md |
| @@ -0,0 +1,439 @@ |
| +# Multi-Upstream Health-Checked Addr Routes Implementation Plan |
| + |
| +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| + |
| +**Goal:** #152 / Part II of the HA spec (`2026-08-12-ha-posture-and-multi-upstream-routes-design.md`): an addr route carries N upstreams with an HTTP health probe; the edge fails away/back, retries dial failures in-request, answers 503+Retry-After when all are down, and per-upstream state reaches dashboard/API/CLI. |
| + |
| +**Architecture:** Additive `addrs`+`health` on #137's record shape → registry columns (comma-joined addrs TEXT, health JSON TEXT) → a new `internal/edge/prober.go` owning per-route probe goroutines and a live set with a `RetireExcept`-shaped lifecycle → a multi-addr `tcpProxy` whose `DialContext` picks a live upstream round-robin and retries other live upstreams on dial failure → probe state into the box status snapshot's `Metrics` → console API (create gains shape fields, new PATCH for addrs/health) → CLI. |
| + |
| +**Tech Stack:** Go stdlib (`httputil.ReverseProxy`, custom `DialContext`), modernc sqlite, `httptest` harness in `internal/edge/edge_test.go`. |
| + |
| +## Global Constraints |
| + |
| +- **Branch composition (Task 0):** `multi-upstream-routes` off `origin/main`, then `git merge origin/worktree-route-records` (the #137 branch — it is BEHIND main; both sides are needed). The PR will contain #137's commits until #137 merges; the PR body must say "stacked on #137; diff collapses when it merges". NO merging of #137 itself into main — that is Paul's. |
| +- Before every commit: `go build ./... && go vet ./... && go test ./... -count=1` clean, `GOOS=darwin go build ./...` compiles, `gofmt -l` clean on touched files. Sandbox: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off`; disable the sandbox if the toolchain reports "read-only file system". |
| +- **Wire shapes are additive.** Every new record field is `omitempty` with a zero value reproducing today's behavior; every registry column addition follows the FOUR-place rule (`ALTER TABLE` list, `routeCols`, `scanRoute`, `Upsert` placeholders+args — `registry.go:482-492`'s comment; `TestUpsertRoundTripsAllColumns` catches miscounts). |
| +- Addr routes never hibernate (`ValidateShape` already enforces; extend to addrs). This invariant is what makes probing wake-safe — do not relax it. |
| +- NO merges to main, NO deploys. PR only. |
| + |
| +--- |
| + |
| +### Task 0: branch composition |
| + |
| +- [ ] **Step 1:** `git worktree add <scratch>/multi-upstream -b multi-upstream-routes origin/main`, then in it `git merge origin/worktree-route-records`. |
| +- [ ] **Step 2:** Resolve conflicts. Expected hot spots: `internal/registry/registry.go` (main gained fleet columns — keep BOTH sides' columns/fields; `routeCols` count grows), `internal/provision/*` (fleet-api touched it), `internal/edge/edge.go` (main gained `fleetProxy`/tunnel changes; #137 changed none of the proxy builders — prefer main's edge, #137's provision/reconcile). Resolve by understanding, not by side-picking. |
| +- [ ] **Step 3:** Full gate (build/vet/test/darwin). The merged tree must be green BEFORE any feature work — a red merge base poisons every later bisect. |
| +- [ ] **Step 4:** Commit the merge; message `merge: compose main + route-records (#137) as the multi-upstream base`. |
| + |
| +--- |
| + |
| +### Task 1: record shape — `Addrs`, `Health`, validation |
| + |
| +**Files:** |
| +- Modify: `internal/provision/provision.go` |
| +- Test: `internal/provision/provision_test.go` (extend) |
| + |
| +**Interfaces (produced):** |
| + |
| +```go |
| +// Health is the per-route probe config (HA spec Part II). Zero value = unset. |
| +type Health struct { |
| + Path string `json:"path,omitempty"` // default "/" |
| + Status int `json:"status,omitempty"` // default 200 |
| + HostHeader string `json:"host_header,omitempty"` // default: the route host |
| + IntervalMS int `json:"interval_ms,omitempty"` // default 2000 |
| + FailAfter int `json:"fail_after,omitempty"` // default 3 |
| + RiseAfter int `json:"rise_after,omitempty"` // default 2 |
| +} |
| +func (h Health) WithDefaults() Health // fills the five defaults above (HostHeader stays "" — edge substitutes route host) |
| + |
| +// On Record (both omitempty, beside Addr): |
| +// Addrs []string `json:"addrs,omitempty"` |
| +// Health *Health `json:"health,omitempty"` |
| +func (r Record) AddrsOrDefault() []string // Addrs if set; else [Addr] if Addr != ""; else nil |
| +func (r Record) HealthOrDefault() *Health // Health if set; else defaults if len(AddrsOrDefault()) > 1; else nil |
| +``` |
| + |
| +- [ ] **Step 1: Failing tests** — in `provision_test.go` add: |
| + |
| +```go |
| +func TestAddrsOrDefaultFoldsLegacyAddr(t *testing.T) { |
| + one := Record{Addr: "10.0.0.1:3000"} |
| + if got := one.AddrsOrDefault(); len(got) != 1 || got[0] != "10.0.0.1:3000" { |
| + t.Fatalf("legacy fold = %v", got) |
| + } |
| + many := Record{Addrs: []string{"10.0.0.1:3000", "10.0.0.2:3000"}} |
| + if got := many.AddrsOrDefault(); len(got) != 2 { |
| + t.Fatalf("addrs = %v", got) |
| + } |
| + if got := (Record{}).AddrsOrDefault(); got != nil { |
| + t.Fatalf("empty = %v, want nil", got) |
| + } |
| +} |
| + |
| +func TestHealthDefaults(t *testing.T) { |
| + h := Health{}.WithDefaults() |
| + if h.Path != "/" || h.Status != 200 || h.IntervalMS != 2000 || h.FailAfter != 3 || h.RiseAfter != 2 { |
| + t.Fatalf("defaults = %+v", h) |
| + } |
| + // Multi-addr with no explicit health implies default probing. |
| + if (Record{Addrs: []string{"a:1", "b:1"}}).HealthOrDefault() == nil { |
| + t.Fatal("multi-addr record must default to probed") |
| + } |
| + // Single legacy addr with no health stays unprobed (today's behavior). |
| + if (Record{Addr: "a:1"}).HealthOrDefault() != nil { |
| + t.Fatal("legacy single-addr record must stay unprobed") |
| + } |
| +} |
| + |
| +func TestValidateShapeMultiAddr(t *testing.T) { |
| + base := Record{Host: "x", Kind: "service"} |
| + ok := base |
| + ok.Addrs = []string{"10.0.0.1:3000", "10.0.0.2:3000"} |
| + if err := ValidateShape(ok); err != nil { |
| + t.Fatalf("valid multi-addr rejected: %v", err) |
| + } |
| + bad := base |
| + bad.Addrs = []string{"10.0.0.1:3000", "not-an-addr"} |
| + if err := ValidateShape(bad); err == nil { |
| + t.Fatal("bad element accepted") |
| + } |
| + both := base |
| + both.Addr, both.Addrs = "10.0.0.1:3000", []string{"10.0.0.2:3000"} |
| + if err := ValidateShape(both); err == nil { |
| + t.Fatal("addr AND addrs together must be rejected — one shape per record") |
| + } |
| + hib := true |
| + sleepy := base |
| + sleepy.Addrs, sleepy.Hibernate = []string{"10.0.0.1:3000"}, &hib |
| + if err := ValidateShape(sleepy); err == nil { |
| + t.Fatal("addrs with hibernate must be rejected (wake-safety invariant)") |
| + } |
| + dup := base |
| + dup.Addrs = []string{"10.0.0.1:3000", "10.0.0.1:3000"} |
| + if err := ValidateShape(dup); err == nil { |
| + t.Fatal("duplicate upstream must be rejected") |
| + } |
| +} |
| + |
| +func TestLegacyRecordStillParses(t *testing.T) { |
| + // A record written before addrs/health existed — byte-for-byte old wire shape. |
| + raw := []byte(`{"schema":1,"host":"a.example.com","slug":"a","app":"web","account":"bab","channel":"stable","created_at":"2026-08-01T00:00:00Z","kind":"service","addr":"10.0.0.1:3000"}`) |
| + var rec Record |
| + if err := json.Unmarshal(raw, &rec); err != nil { |
| + t.Fatal(err) |
| + } |
| + if rec.Addrs != nil || rec.Health != nil { |
| + t.Fatalf("legacy record grew fields: %+v", rec) |
| + } |
| + if got := rec.AddrsOrDefault(); len(got) != 1 || got[0] != "10.0.0.1:3000" { |
| + t.Fatalf("legacy fold = %v", got) |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2:** Run `go test ./internal/provision/ -run 'TestAddrs|TestHealth|TestValidateShapeMultiAddr|TestLegacyRecord' -count=1` — FAIL (undefined fields/methods). |
| +- [ ] **Step 3:** Implement: add the two fields + `Health` type + the three helpers; extend `ValidateShape` (`provision.go:247-282`): factor the existing inline `host:port` check into `func validAddr(a string) error` and run it over every `AddrsOrDefault()` element; reject `Addr != "" && len(Addrs) > 0`; reject duplicates; extend the existing static-kind, service-kind-needs-upstream, and addr+hibernate cases to use `len(AddrsOrDefault()) > 0` instead of `Addr != ""` (keep error message wording style). Keep `DerivesPaths` coherent: its `r.Addr == ""` becomes `len(r.AddrsOrDefault()) == 0`. |
| +- [ ] **Step 4:** Tests pass; run the whole package: `go test ./internal/provision/ -count=1`. |
| +- [ ] **Step 5:** Commit: `provision: addrs + health on the record shape (additive, #152)`. |
| + |
| +--- |
| + |
| +### Task 2: registry columns |
| + |
| +**Files:** |
| +- Modify: `internal/registry/registry.go` |
| +- Test: `internal/registry/registry_test.go` (extend `TestUpsertRoundTripsAllColumns` and Fingerprint tests) |
| + |
| +**Interfaces (produced):** |
| +- `Route.Addrs string` — comma-joined `host:port` list, `""` = none (comma-join is safe: `validAddr` elements cannot contain commas). Doc comment must say the join rule and that `Addr` stays the legacy single-upstream field. |
| +- `Route.Health string` — health JSON verbatim from the record (canonical `json.Marshal` of `provision.Health`), `""` = none. |
| +- `func (t Route) AddrList() []string` — split `Addrs`; if empty, fold `Addr` into a one-element list; else nil. |
| +- `func (r *Registry) SetUpstreams(host, addrs, health string) error` — narrow UPDATE, error on unmatched host (the `SetChannel`/`SetUnit` pattern at registry.go:649-668). |
| + |
| +- [ ] **Step 1: Failing tests** — extend `TestUpsertRoundTripsAllColumns` with the two fields; add: |
| + |
| +```go |
| +func TestFingerprintSeesAddrsAndHealthChanges(t *testing.T) { |
| + reg := ... // the package's existing temp-registry helper |
| + reg.Upsert(Route{Host: "m.example.com", Kind: "service", Addrs: "10.0.0.1:3000,10.0.0.2:3000", Status: "ready"}) |
| + fp1, _ := reg.Fingerprint() |
| + reg.SetUpstreams("m.example.com", "10.0.0.1:3000,10.0.0.3:3000", "") |
| + fp2, _ := reg.Fingerprint() |
| + if fp1 == fp2 { |
| + t.Fatal("addrs change invisible to Fingerprint — edge would serve a stale upstream set forever") |
| + } |
| + reg.SetUpstreams("m.example.com", "10.0.0.1:3000,10.0.0.3:3000", `{"path":"/up","status":200}`) |
| + fp3, _ := reg.Fingerprint() |
| + if fp2 == fp3 { |
| + t.Fatal("health change invisible to Fingerprint") |
| + } |
| +} |
| + |
| +func TestAddrListFoldsLegacyAddr(t *testing.T) { |
| + if got := (Route{Addr: "a:1"}).AddrList(); len(got) != 1 || got[0] != "a:1" { |
| + t.Fatalf("fold = %v", got) |
| + } |
| + if got := (Route{Addrs: "a:1,b:2"}).AddrList(); len(got) != 2 || got[1] != "b:2" { |
| + t.Fatalf("split = %v", got) |
| + } |
| + if got := (Route{}).AddrList(); got != nil { |
| + t.Fatalf("empty = %v", got) |
| + } |
| +} |
| +``` |
| + |
| +(Adapt the temp-registry helper name to what `registry_test.go` actually uses — read it first.) |
| + |
| +- [ ] **Step 2:** FAIL as expected. |
| +- [ ] **Step 3:** Implement, touching ALL FOUR column places for each of `addrs`, `health` (`ALTER TABLE ... TEXT NOT NULL DEFAULT ''`; `routeCols`; `scanRoute`; `Upsert` — plain overwrite like `addr=excluded.addr`, NOT the preserve-when-empty CASE), plus: `Fingerprint()`'s SELECT/Scan/Fprintf triple gains both columns (registry.go:1152-1207 — this is reload-relevant routing state), and `SetUpstreams` as a narrow UPDATE with rows-affected check. |
| +- [ ] **Step 4:** `go test ./internal/registry/ -count=1` — pass, including the pre-existing round-trip test. |
| +- [ ] **Step 5:** Commit: `registry: addrs + health columns, Fingerprint coverage, SetUpstreams`. |
| + |
| +--- |
| + |
| +### Task 3: reconcile — `routeFor` + `shapeMismatch` |
| + |
| +**Files:** |
| +- Modify: `internal/provision/reconcile.go` |
| +- Test: `internal/provision/reconcile_test.go` (extend) |
| + |
| +**Interfaces:** |
| +- Consumes: Task 1 helpers, Task 2 `Route.Addrs`/`Route.Health`. |
| +- Produces: records with `addrs`/`health` mint and adopt routes carrying them. |
| + |
| +- [ ] **Step 1: Failing tests:** |
| + |
| +```go |
| +func TestRouteForCarriesAddrsAndHealth(t *testing.T) { |
| + rec := Record{Host: "m.example.com", App: "web", Account: "bab", Channel: "stable", |
| + Kind: "service", Addrs: []string{"10.0.0.1:3000", "10.0.0.2:3000"}} |
| + rt := routeFor(rec, "/data", "seed") |
| + if rt.Addrs != "10.0.0.1:3000,10.0.0.2:3000" { |
| + t.Fatalf("Addrs = %q", rt.Addrs) |
| + } |
| + if rt.Health == "" { |
| + t.Fatal("multi-addr record must mint default health JSON") |
| + } |
| + var h Health |
| + if err := json.Unmarshal([]byte(rt.Health), &h); err != nil || h.Path != "/" || h.FailAfter != 3 { |
| + t.Fatalf("health JSON = %q (%v)", rt.Health, err) |
| + } |
| + // Legacy single-addr record: byte-identical route to before this change. |
| + old := Record{Host: "s.example.com", App: "web", Account: "bab", Channel: "stable", Kind: "service", Addr: "10.0.0.9:3000"} |
| + ort := routeFor(old, "/data", "seed") |
| + if ort.Addrs != "" || ort.Health != "" || ort.Addr != "10.0.0.9:3000" { |
| + t.Fatalf("legacy record's route changed shape: %+v", ort) |
| + } |
| +} |
| + |
| +func TestShapeMismatchComparesAddrsAndHealth(t *testing.T) { |
| + a := registry.Route{Kind: "service", Addrs: "x:1,y:2"} |
| + b := registry.Route{Kind: "service", Addrs: "x:1,z:3"} |
| + if shapeMismatch(a, b) == "" { |
| + t.Fatal("differing addrs adopted silently") |
| + } |
| + c := registry.Route{Kind: "service", Addrs: "x:1,y:2", Health: `{"path":"/up"}`} |
| + if shapeMismatch(a, c) == "" { |
| + t.Fatal("differing health adopted silently") |
| + } |
| + if shapeMismatch(a, registry.Route{Kind: "service", Addrs: "x:1,y:2"}) != "" { |
| + t.Fatal("equal shapes must match") |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2:** FAIL. |
| +- [ ] **Step 3:** Implement: in `routeFor` (reconcile.go:68-82) set `Addrs: strings.Join(rec.Addrs, ",")` (the RECORD's explicit Addrs only — a legacy `Addr` record keeps `Addr`, so its minted route is byte-identical) and `Health:` = marshaled `rec.HealthOrDefault()` when non-nil (multi-addr default probing materializes here, once, at mint — the edge then treats route JSON as authoritative). In `shapeMismatch` add `case existing.Addrs != want.Addrs` and `case existing.Health != want.Health` beside the `Addr` case (string compare is enough: `routeFor` is the single writer, so ordering/format are canonical by construction). |
| +- [ ] **Step 4:** `go test ./internal/provision/ -count=1` — pass. |
| +- [ ] **Step 5:** Commit: `provision: mint/adopt multi-upstream shapes through routeFor + shapeMismatch`. |
| + |
| +--- |
| + |
| +### Task 4: the prober (`internal/edge/prober.go`) |
| + |
| +**Files:** |
| +- Create: `internal/edge/prober.go` |
| +- Test: `internal/edge/prober_test.go` |
| + |
| +**Interfaces (produced — Task 5 consumes all of it):** |
| + |
| +```go |
| +// probeSpec is one route's probing contract, parsed from registry JSON once |
| +// per Reload. |
| +type probeSpec struct { |
| + Host string // route host — the probe's Host header default |
| + Addrs []string |
| + Health provision.Health // already WithDefaults()ed |
| +} |
| + |
| +// proberSet owns one goroutine per (host) probing its addrs, and the shared |
| +// live-state table. Lifecycle mirrors Activator.RetireExcept: Reload computes |
| +// the desired specs and calls Sync; goroutines for vanished/changed hosts stop. |
| +type proberSet struct{ ... } |
| +func newProberSet() *proberSet |
| +func (p *proberSet) Sync(want map[string]probeSpec) // start/stop/replace probers |
| +func (p *proberSet) Live(host string) []string // currently-live addrs (probe view) |
| +func (p *proberSet) Eject(host, addr string) // dial-failure fast path; probe pass rejoins it |
| +func (p *proberSet) State() map[string][]UpstreamState // for the status snapshot (Task 6) |
| +func (p *proberSet) Stop() // stop everything (tests, shutdown) |
| + |
| +// UpstreamState is the dashboard-facing view of one upstream. |
| +type UpstreamState struct { |
| + Addr string `json:"addr"` |
| + Up bool `json:"up"` |
| + Since time.Time `json:"since"` // last transition |
| + Fails int `json:"fails"` // consecutive probe failures |
| + LastError string `json:"last_error,omitempty"` // dial vs status, for the log/dashboard |
| +} |
| +``` |
| + |
| +Probing rules (per spec): `GET http://<addr><path>`, `Host:` = `Health.HostHeader` or route host, client timeout 2 s, every `IntervalMS`; `FailAfter` consecutive failures → down; `RiseAfter` consecutive passes → up; transitions logged `edge: probe <host> <addr> down (dial: ...)` / `... up`. All upstreams START live (optimistic — a fresh Reload must not 503 a healthy fleet while first probes run). `Eject` marks down immediately with `Fails` set to `FailAfter`. |
| + |
| +- [ ] **Step 1: Failing tests** — with `httptest` backends and a fast interval: |
| + |
| +```go |
| +func TestProberFailsAwayAndBack(t *testing.T) { |
| + healthy := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(200) })) |
| + defer healthy.Close() |
| + flaky := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(500) })) |
| + spec := probeSpec{Host: "m.example.com", |
| + Addrs: []string{healthy.Listener.Addr().String(), flaky.Listener.Addr().String()}, |
| + Health: provision.Health{Path: "/", Status: 200, IntervalMS: 20, FailAfter: 3, RiseAfter: 2}.WithDefaults()} |
| + p := newProberSet() |
| + defer p.Stop() |
| + p.Sync(map[string]probeSpec{"m.example.com": spec}) |
| + |
| + waitFor(t, time.Second, func() bool { return len(p.Live("m.example.com")) == 1 }) |
| + if live := p.Live("m.example.com"); live[0] != healthy.Listener.Addr().String() { |
| + t.Fatalf("live = %v", live) |
| + } |
| + // flaky recovers → rejoins after RiseAfter passes |
| + flaky.Config.Handler = http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(200) }) |
| + waitFor(t, time.Second, func() bool { return len(p.Live("m.example.com")) == 2 }) |
| +} |
| + |
| +func TestProberSendsConfiguredHostHeader(t *testing.T) { ... assert r.Host == "m.example.com" inside the backend handler ... } |
| + |
| +func TestSyncStopsVanishedHosts(t *testing.T) { ... Sync with the spec, then Sync(map...{}), assert goroutine stopped via State() empty and no further backend hits after a settle ... } |
| + |
| +func TestEjectIsImmediateAndProbeRejoins(t *testing.T) { ... Eject a healthy addr, assert Live drops it instantly, then waitFor it to rejoin ... } |
| +``` |
| + |
| +(`waitFor` = small poll helper with deadline; write it in the test file. Flesh the elided tests fully when writing — each is the assertion its name states, in the harness style of `edge_test.go`.) |
| + |
| +- [ ] **Step 2:** FAIL. **Step 3:** Implement `prober.go` (~250 lines; one goroutine per host looping over its addrs each tick — sequential within a host is fine at these counts; mutex-guarded state map; `Sync` diffs by host + deep-equal spec, stopping via per-host `chan struct{}`). **Step 4:** `go test ./internal/edge/ -run TestProber -count=1 -race` — pass, race-clean. **Step 5:** Commit: `edge: upstream prober — live set, fail-away/fail-back, RetireExcept-shaped lifecycle`. |
| + |
| +--- |
| + |
| +### Task 5: multi-addr proxy + Reload wiring |
| + |
| +**Files:** |
| +- Modify: `internal/edge/edge.go` (Reload switch `edge.go:219-362`, proxy builders near `tcpProxy` at 938) |
| +- Test: `internal/edge/edge_test.go` (extend) |
| + |
| +**Interfaces:** |
| +- Consumes: `proberSet` (Task 4), `Route.AddrList()`/`Route.Health` (Task 2), `provision.Health`. |
| +- Produces: `multiProxy(host string, addrs []string, p *proberSet) *httputil.ReverseProxy`; `Edge` gains a `probers *proberSet` field, created in `New`, synced in `Reload`, stopped nowhere in prod (edge lives for the process) but `Stop`ped in tests. |
| + |
| +Mechanism (spec Part II): `Rewrite` sets a fixed sentinel `pr.Out.URL.Host = "multi.invalid"`; the Transport's `DialContext` ignores the sentinel and picks from `p.Live(host)` round-robin (atomic counter), dials with a 3 s per-attempt timeout, and on dial error calls `p.Eject(host, addr)` and tries the next live addr — up to `len(addrs)` attempts, all inside one request, all before any bytes flow. All dials failed (or live set empty) → return a sentinel error; the proxy's `ErrorHandler` maps it to `503` + `Retry-After: 2` (and any other proxy error to the current default 502 path). Wire in `Reload`: the `case t.Addr != ""` arm becomes `case len(t.AddrList()) > 0`, building `tcpProxy(t.Addr)` exactly as today when there is no health config, and `multiProxy` when `t.Health != ""` (single- or multi-addr — a single probed addr gets the 503 behavior and dashboard state per spec). Collect `probeSpecs[t.Host]` in the same loop (parse `t.Health` JSON once here; a parse failure logs and falls back to unprobed `tcpProxy` on `AddrList()[0]` — never a dead route) and call `e.probers.Sync(probeSpecs)` right after `e.upstreams.Store(&m)`, beside `e.act.RetireExcept(managed)`. |
| + |
| +- [ ] **Step 1: Failing tests** (harness style of `TestRoutesHostToTCPUpstream` / `TestReloadPicksUpNewRoute`): |
| + |
| +```go |
| +func TestMultiUpstreamBalancesAcrossLiveBackends(t *testing.T) { |
| + // two counting backends, route with Addrs both + default health JSON, |
| + // 20 requests → both counters > 0 |
| +} |
| + |
| +func TestDialFailureRetriesOtherUpstreamNoUserError(t *testing.T) { |
| + // backend A = real server; backend B = a LISTENER THAT IS CLOSED before |
| + // the test runs (reserve a port with net.Listen, note the addr, Close it). |
| + // Route carries both. 10 sequential requests → ALL 200, zero 5xx. |
| + // Then p.Live must show B ejected (the dial failure fast path). |
| +} |
| + |
| +func TestAllUpstreamsDownAnswers503WithRetryAfter(t *testing.T) { |
| + // both addrs closed listeners → request gets 503 AND Retry-After: 2, |
| + // not the stdlib bare 502. |
| +} |
| + |
| +func TestSingleAddrWithHealthGets503Semantics(t *testing.T) { |
| + // one closed addr + health JSON → 503+Retry-After (not 502). |
| +} |
| + |
| +func TestLegacySingleAddrRouteUnchanged(t *testing.T) { |
| + // Addr only, no health: dead upstream still produces the OLD behavior |
| + // (502) — proving the unprobed path is byte-compatible. |
| +} |
| + |
| +func TestReloadSwapsUpstreamSet(t *testing.T) { |
| + // route Addrs=[A], serve; SetUpstreams to [B] via second registry handle; |
| + // e.Reload(); next request hits B. Prober specs must follow (State() shows B). |
| +} |
| +``` |
| + |
| +Write each fully in the file — the comments above are the behavior contract; the harness helpers (`openReg`, `hostEchoBackend`, `getHost`) already exist at `edge_test.go:27-68`. |
| + |
| +- [ ] **Step 2:** FAIL. **Step 3:** Implement. **Step 4:** `go test ./internal/edge/ -count=1 -race` — all pass including every pre-existing test (the dispatch-order comment at edge.go:260-278 must stay true: tunnel > socket > addrs > static > status). **Step 5:** Commit: `edge: multi-upstream addr proxy — round-robin live set, in-request dial retry, 503+Retry-After when dark`. |
| + |
| +--- |
| + |
| +### Task 6: probe state into the box status snapshot |
| + |
| +**Files:** |
| +- Read FIRST: `internal/status/snapshot.go`, `internal/host/metrics_loop.go`, `internal/host/agent.go:309-321` (the `RunWithActivatorReady` callback — the usage-meter wiring precedent `e.EnableUsage(m)`). |
| +- Modify: `internal/status` (Metrics struct), `internal/host/metrics_loop.go` or the snapshot writer, `internal/edge/edge.go` (export `func (e *Edge) UpstreamStates() map[string][]UpstreamState`). |
| +- Test: alongside the existing metrics/snapshot tests (find and extend them; pattern-match, do not invent a new harness). |
| + |
| +**Interfaces:** |
| +- Produces: box snapshot `Metrics` gains `Upstreams map[string][]edge.UpstreamState \`json:"upstreams,omitempty"\`` — keyed by route host, ABSENT (omitempty) when the box has no probed routes, so every existing snapshot consumer parses unchanged. |
| + |
| +- [ ] **Step 1:** Read the three files; write the failing test in the existing snapshot/metrics test style asserting a probed route's state round-trips into the written `status/boxes/<label>.json`. |
| +- [ ] **Step 2:** FAIL. **Step 3:** Implement: `Edge.UpstreamStates()` delegates to `probers.State()`; plumb it into the snapshot the same way route versions already flow (`Metrics.RouteVersions` — `stagedVersions` at `internal/console/apiinstances.go:178-221` is the read side proving the pattern). Wire the edge handle via the `RunWithActivatorReady` callback exactly as the usage meter does. **Step 4:** full-package tests pass. **Step 5:** Commit: `status: per-upstream health in the box snapshot`. |
| + |
| +--- |
| + |
| +### Task 7: console API — create gains shape fields; PATCH for addrs/health; list join |
| + |
| +**Files:** |
| +- Modify: `internal/console/apiinstances.go`, `internal/console/authcli.go:267-272` (route table), `internal/provision/provision.go` (one CAS update helper) |
| +- Test: the package's existing API-handler tests (extend in their style) |
| + |
| +**Interfaces (produced):** |
| +- `POST .../instances` request body gains `Kind string`, `Addrs []string`, `Health *provision.Health` (all optional; `ValidateShape` gates; the handler copies them onto the `provision.Record` it builds at apiinstances.go:286-400). Response unchanged in shape (the record echoes new fields via omitempty). |
| +- New: `PATCH /api/cli/apps/{account}/{app}/instances/{host}` — body `{"addrs": [...], "health": {...}}`, both optional but not both absent. Flow: `GetRecord` (returns record + etag — confirm signature by reading provision.go), apply fields, `ValidateShape`, `provision.UpdateRecord(ctx, store, rec, etag)` (new helper: `Put` with `ifMatch=etag`, `ErrCASConflict` → 409), then bump `control/serial` the same way the create handler does (read the create handler's serial-bump call and mirror it exactly). Gate order is the console's fixed one: session → CSRF → owner (byte-identical 404) — copy `instanceGate` usage from the create handler; never invent a new gate order. |
| +- `GET .../instances` list records gain `"upstreams"` joined from box snapshots exactly as `staged_version` is (extend `instanceListRecord` at apiinstances.go:163-176 with `Upstreams []edge.UpstreamState \`json:"upstreams,omitempty"\`` — if importing `internal/edge` into console is unwanted (check existing imports), move `UpstreamState` to `internal/status` in Task 6 and have edge return status types; decide by following where `RouteVersions`' type lives). |
| + |
| +- [ ] **Step 1:** Failing handler tests: create-with-addrs round-trips the record; create with bad addr → 400 with the ValidateShape message; PATCH happy path rewrites the record + bumps serial; PATCH by non-member → byte-identical 404; PATCH etag race → 409; list join shows upstreams when the snapshot has them. |
| +- [ ] **Step 2:** FAIL. **Step 3:** Implement. **Step 4:** `go test ./internal/console/ ./internal/provision/ -count=1` — pass. **Step 5:** Commit: `console: instance create carries route shape; PATCH updates addrs/health; list joins upstream health`. |
| + |
| +--- |
| + |
| +### Task 8: CLI |
| + |
| +**Files:** |
| +- Modify: `instances_cli.go` (repo root, package main), `apimode.go` (the `instanceCreate` body struct at apimode.go:1045-1061 + a new `instanceSetUpstreams`) |
| +- Test: the CLI's existing test conventions (MUST build env through `carlosTestEnv` and set `cmd.Dir` to a scratch dir if any test shells out — see CLAUDE.md's minted-40-accounts incident). |
| + |
| +**Interfaces (produced):** |
| +- `carlos instances create` gains `-addr` (repeatable or comma-separated — match the repo's existing repeated-flag convention if one exists; else comma-separated single flag) and `-health-path`/`-health-status` (optional). |
| +- New verb: `carlos instances set-upstreams -app X -host H -addrs a:1,b:2 [-health-path /up -health-status 200]` → the Task 7 PATCH. |
| +- `carlos instances list` (`instances_cli.go:255-277`) gains an `UPSTREAMS` column rendering `2/3` (live/total) from the joined field, `-` when absent. |
| + |
| +- [ ] **Step 1:** Failing tests in the existing CLI test style. **Step 2:** FAIL. **Step 3:** Implement. **Step 4:** package tests pass. **Step 5:** Commit: `cli: multi-upstream create/set-upstreams/list`. |
| + |
| +--- |
| + |
| +### Task 9: full gate + PR |
| + |
| +- [ ] **Step 1:** Full gate: `go build ./... && go vet ./... && go test ./... -count=1` + `GOOS=darwin go build ./...` + `gofmt -l` on every touched file. ALL packages, not just the touched ones. |
| +- [ ] **Step 2:** Push `multi-upstream-routes`; open the PR with base `main`, body stating: implements #152 per the HA spec (PR #154); **stacked on #137** (`worktree-route-records` is merged in — the diff collapses to the feature once #137 lands); closes nothing until deployed; wake-safety invariant note; rolling-deploy semantics summary; the Rails staging boxes as first consumer replacing their NLB. |
| + |
| +--- |
| + |
| +## Self-review notes (already applied) |
| + |
| +- Spec coverage: record shape (T1), registry (T2), mint/adopt (T3), prober (T4), request path incl. 503/retry/round-robin + legacy-unchanged proof (T5), dashboard state (T6), API door incl. the PATCH the zero-downtime story needs (T7), CLI (T8). Rolling deploys need no code beyond T4/T5 (probe-driven fail-away) — the spec's drain verb stays unbuilt. |
| +- Type consistency: `probeSpec`/`UpstreamState` (T4) consumed by T5/T6; `SetUpstreams(host, addrs, health string)` (T2) used by T5's reload test; `Health.WithDefaults()` (T1) used by T3-T5. T7 note resolves where `UpstreamState` finally lives — follow `RouteVersions`' home. |
| +- Known softness, deliberate: T6/T7/T8 point at files whose exact current shape post-merge (Task 0) can't be quoted tonight — each such task's Step 1 is "read the named file first", with the precedent pattern named precisely. |
diff --git a/docs/superpowers/plans/2026-08-12-registry-snapshot-cold-standby.md b/docs/superpowers/plans/2026-08-12-registry-snapshot-cold-standby.md| new file mode 100644 |
| index 000000000..b8a6c2af5 |
| --- /dev/null |
| +++ b/docs/superpowers/plans/2026-08-12-registry-snapshot-cold-standby.md |
| @@ -0,0 +1,718 @@ |
| +# Registry Snapshot + Cold-Standby Runbook Implementation Plan |
| + |
| +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| + |
| +**Goal:** Rung 1 of the HA ladder (spec `2026-08-12-ha-posture-and-multi-upstream-routes-design.md`, Part I): the edge box snapshots its registry to the control bucket every 5 minutes when it changed, EC2 auto-recovery is armed in tofu, and a count=0 standby box + rehearsable runbook land in platform-infrastructure. |
| + |
| +**Architecture:** A new `registrySnapshotLoop` in `internal/host/agent.go`'s loop block (the `retentionLoop`/`usageLoop` pattern): a pure pass function snapshots via a new `registry.SnapshotTo` (`VACUUM INTO` on a fresh connection — the modernc WAL-visibility recipe `Fingerprint()` already uses) and writes `control/registry-snapshot.db` + sidecar + pruned generations with the CAS-merge pattern from `usage.UpsertEdgeDay`. Infra: an additive CloudWatch recover alarm in the shared `region-host` module; a standby module instance (count=0) and runbook in platform-infrastructure. |
| + |
| +**Tech Stack:** Go, modernc.org/sqlite (pure Go — no cgo anywhere), `release.Store` bucket interface, OpenTofu. |
| + |
| +## Global Constraints |
| + |
| +- Branch: `ha-snapshot` off `origin/main` in the platform repo; separate branch `edge-standby` in `/home/paulca/github.com/carlosframework/platform-infrastructure`. |
| +- Before every commit: `go build ./... && go vet ./... && go test ./... -count=1` all clean, `GOOS=darwin go build ./...` compiles, `gofmt -l` clean on touched files. Sandbox: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off`; go commands may need the sandbox disabled ("read-only file system" is the tell). |
| +- Wire-shape changes are additive; new bucket keys are new content, never a change to existing keys. |
| +- NO merges, NO deploys, NO `tofu apply` — PRs only; applies are queued for Paul. |
| +- Tofu files: run `tofu fmt` on touched files; `tofu validate` needs providers and is NOT expected to run offline — do not fight it. |
| + |
| +--- |
| + |
| +### Task 1: `registry.SnapshotTo` |
| + |
| +**Files:** |
| +- Modify: `internal/registry/registry.go` (add one method after `Fingerprint`, ~line 1297) |
| +- Test: `internal/registry/snapshot_test.go` (create) |
| + |
| +**Interfaces:** |
| +- Produces: `func (r *Registry) SnapshotTo(path string) error` — writes a consistent single-file copy of the registry DB to `path` (which must not exist). |
| + |
| +- [ ] **Step 1: Write the failing test** |
| + |
| +```go |
| +package registry |
| + |
| +import ( |
| + "os" |
| + "path/filepath" |
| + "testing" |
| +) |
| + |
| +func TestSnapshotToProducesOpenableCopy(t *testing.T) { |
| + dir := t.TempDir() |
| + reg, err := Open(filepath.Join(dir, "registry.db")) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + defer reg.Close() |
| + if err := reg.Upsert(Route{Host: "a.example.com", Kind: "service", Addr: "10.0.0.1:3000", Status: "ready"}); err != nil { |
| + t.Fatal(err) |
| + } |
| + |
| + out := filepath.Join(dir, "snap.db") |
| + if err := reg.SnapshotTo(out); err != nil { |
| + t.Fatal(err) |
| + } |
| + |
| + copyReg, err := Open(out) |
| + if err != nil { |
| + t.Fatalf("open snapshot: %v", err) |
| + } |
| + defer copyReg.Close() |
| + rt, err := copyReg.Get("a.example.com") |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if rt.Addr != "10.0.0.1:3000" { |
| + t.Fatalf("snapshot route addr = %q, want 10.0.0.1:3000", rt.Addr) |
| + } |
| +} |
| + |
| +func TestSnapshotToRefusesExistingTarget(t *testing.T) { |
| + dir := t.TempDir() |
| + reg, err := Open(filepath.Join(dir, "registry.db")) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + defer reg.Close() |
| + out := filepath.Join(dir, "snap.db") |
| + if err := os.WriteFile(out, []byte("x"), 0o644); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := reg.SnapshotTo(out); err == nil { |
| + t.Fatal("SnapshotTo over an existing file: want error, got nil") |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Run to verify failure** |
| + |
| +Run: `go test ./internal/registry/ -run TestSnapshotTo -count=1` |
| +Expected: FAIL — `reg.SnapshotTo undefined`. |
| + |
| +- [ ] **Step 3: Implement** |
| + |
| +Add after `Fingerprint` in `internal/registry/registry.go`: |
| + |
| +```go |
| +// SnapshotTo writes a consistent single-file copy of the registry to path |
| +// via VACUUM INTO on a fresh connection — same fresh-connection recipe as |
| +// Fingerprint (modernc WAL visibility). VACUUM INTO refuses an existing |
| +// target, which is the safety we want: callers stage into a temp dir. |
| +func (r *Registry) SnapshotTo(path string) error { |
| + db, err := sql.Open("sqlite", r.path) |
| + if err != nil { |
| + return err |
| + } |
| + defer db.Close() |
| + _, err = db.Exec(`VACUUM INTO ?`, path) |
| + return err |
| +} |
| +``` |
| + |
| +(`database/sql` is already imported. If modernc rejects the bound parameter for VACUUM INTO — possible, some pragmas refuse placeholders — fall back to `fmt.Sprintf("VACUUM INTO %q", path)`; `fmt` is already imported and the path is process-internal, never user input. The test proves whichever form works.) |
| + |
| +- [ ] **Step 4: Run to verify pass** |
| + |
| +Run: `go test ./internal/registry/ -run TestSnapshotTo -count=1` |
| +Expected: PASS (both tests). |
| + |
| +- [ ] **Step 5: Commit** |
| + |
| +```bash |
| +git add internal/registry/registry.go internal/registry/snapshot_test.go |
| +git commit -m "registry: SnapshotTo — consistent VACUUM INTO copy for the snapshot pass" |
| +``` |
| + |
| +--- |
| + |
| +### Task 2: snapshot pass (pure function) |
| + |
| +**Files:** |
| +- Create: `internal/host/snapshot_loop.go` |
| +- Test: `internal/host/snapshot_loop_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: `registry.SnapshotTo` (Task 1); `release.Store` (`internal/release/store.go:42` — `Get` returns `(body, etag, error)` with `os.ErrNotExist`-wrapped not-found; `Put(ctx, key, body, ifMatch)`: `ifMatch==""` is create-only, non-empty is replace-only, races are `release.ErrCASConflict`; `List(ctx, prefix)`; `Delete` idempotent). |
| +- Produces: `func registrySnapshotPass(ctx context.Context, s release.Store, reg *registry.Registry, box string, now time.Time, lastFP *string) error` and the key constants below — Task 3 wires the loop; the pinfra runbook (Task 6) names the keys. |
| + |
| +- [ ] **Step 1: Write the failing test** |
| + |
| +```go |
| +package host |
| + |
| +import ( |
| + "context" |
| + "encoding/json" |
| + "path/filepath" |
| + "testing" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/registry" |
| + "github.com/carlosframework/platform/internal/release" |
| +) |
| + |
| +func snapTestRegistry(t *testing.T) *registry.Registry { |
| + t.Helper() |
| + reg, err := registry.Open(filepath.Join(t.TempDir(), "registry.db")) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + t.Cleanup(func() { reg.Close() }) |
| + if err := reg.Upsert(registry.Route{Host: "a.example.com", Kind: "service", Addr: "10.0.0.1:3000", Status: "ready"}); err != nil { |
| + t.Fatal(err) |
| + } |
| + return reg |
| +} |
| + |
| +func TestSnapshotPassWritesCurrentSidecarAndGeneration(t *testing.T) { |
| + s := release.NewDirStore(t.TempDir()) |
| + ctx := context.Background() |
| + reg := snapTestRegistry(t) |
| + at := time.Date(2026, 8, 12, 3, 0, 0, 0, time.UTC) |
| + |
| + var lastFP string |
| + if err := registrySnapshotPass(ctx, s, reg, "box1", at, &lastFP); err != nil { |
| + t.Fatal(err) |
| + } |
| + |
| + raw, _, err := s.Get(ctx, snapshotKey) |
| + if err != nil { |
| + t.Fatalf("current snapshot missing: %v", err) |
| + } |
| + if len(raw) == 0 { |
| + t.Fatal("current snapshot is empty") |
| + } |
| + |
| + metaRaw, _, err := s.Get(ctx, snapshotMetaKey) |
| + if err != nil { |
| + t.Fatalf("sidecar missing: %v", err) |
| + } |
| + var meta snapshotMeta |
| + if err := json.Unmarshal(metaRaw, &meta); err != nil { |
| + t.Fatal(err) |
| + } |
| + if meta.V != 1 || meta.Box != "box1" || meta.Fingerprint == "" || meta.Bytes != len(raw) { |
| + t.Fatalf("sidecar = %+v (snapshot %d bytes)", meta, len(raw)) |
| + } |
| + if lastFP != meta.Fingerprint { |
| + t.Fatalf("lastFP %q not updated to %q", lastFP, meta.Fingerprint) |
| + } |
| + |
| + gens, err := s.List(ctx, snapshotGenPrefix) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if len(gens) != 1 || gens[0] != snapshotGenPrefix+"20260812T030000Z.db" { |
| + t.Fatalf("generations = %v", gens) |
| + } |
| +} |
| + |
| +func TestSnapshotPassSkipsWhenFingerprintUnchanged(t *testing.T) { |
| + s := release.NewDirStore(t.TempDir()) |
| + ctx := context.Background() |
| + reg := snapTestRegistry(t) |
| + at := time.Date(2026, 8, 12, 3, 0, 0, 0, time.UTC) |
| + |
| + var lastFP string |
| + if err := registrySnapshotPass(ctx, s, reg, "box1", at, &lastFP); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := registrySnapshotPass(ctx, s, reg, "box1", at.Add(5*time.Minute), &lastFP); err != nil { |
| + t.Fatal(err) |
| + } |
| + gens, err := s.List(ctx, snapshotGenPrefix) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if len(gens) != 1 { |
| + t.Fatalf("unchanged registry grew generations: %v", gens) |
| + } |
| + |
| + // A registry change makes the next pass write again. |
| + if err := reg.Upsert(registry.Route{Host: "b.example.com", Kind: "service", Addr: "10.0.0.2:3000", Status: "ready"}); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := registrySnapshotPass(ctx, s, reg, "box1", at.Add(10*time.Minute), &lastFP); err != nil { |
| + t.Fatal(err) |
| + } |
| + gens, err = s.List(ctx, snapshotGenPrefix) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if len(gens) != 2 { |
| + t.Fatalf("changed registry did not write a new generation: %v", gens) |
| + } |
| +} |
| + |
| +func TestSnapshotPassPrunesGenerationsBeyondKeep(t *testing.T) { |
| + s := release.NewDirStore(t.TempDir()) |
| + ctx := context.Background() |
| + reg := snapTestRegistry(t) |
| + at := time.Date(2026, 8, 12, 0, 0, 0, 0, time.UTC) |
| + |
| + var lastFP string |
| + for i := 0; i < snapshotKeep+3; i++ { |
| + // Change the registry each round so no pass skips. |
| + if err := reg.Upsert(registry.Route{Host: "a.example.com", Kind: "service", Addr: "10.0.0.1:3000", Status: "ready", Version: string(rune('a' + i))}); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := registrySnapshotPass(ctx, s, reg, "box1", at.Add(time.Duration(i)*5*time.Minute), &lastFP); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + gens, err := s.List(ctx, snapshotGenPrefix) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if len(gens) != snapshotKeep { |
| + t.Fatalf("kept %d generations, want %d: %v", len(gens), snapshotKeep, gens) |
| + } |
| + // The survivors must be the newest ones: the earliest kept stamp is round 3's. |
| + if gens[0] != snapshotGenPrefix+"20260812T001500Z.db" { |
| + t.Fatalf("oldest kept generation = %q, want the round-3 stamp", gens[0]) |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Run to verify failure** |
| + |
| +Run: `go test ./internal/host/ -run TestSnapshotPass -count=1` |
| +Expected: FAIL — `registrySnapshotPass`, `snapshotKey` etc. undefined. |
| + |
| +- [ ] **Step 3: Implement `internal/host/snapshot_loop.go`** |
| + |
| +```go |
| +package host |
| + |
| +import ( |
| + "context" |
| + "encoding/json" |
| + "errors" |
| + "fmt" |
| + "log" |
| + "os" |
| + "path/filepath" |
| + "sort" |
| + "time" |
| + |
| + "math/rand/v2" |
| + |
| + "github.com/carlosframework/platform/internal/registry" |
| + "github.com/carlosframework/platform/internal/release" |
| +) |
| + |
| +// The cold-standby spec (2026-08-12 HA posture, Part I rung 1): the current |
| +// snapshot at a fixed key a runbook can name, a sidecar describing it, and a |
| +// short tail of prior generations. All under control/ beside control/serial; |
| +// platform boxes write direct S3, so no store-door ACL is involved. |
| +const ( |
| + snapshotKey = "control/registry-snapshot.db" |
| + snapshotMetaKey = "control/registry-snapshot.json" |
| + snapshotGenPrefix = "control/registry-snapshots/" |
| + snapshotKeep = 8 |
| + snapshotStamp = "20060102T150405Z" |
| +) |
| + |
| +type snapshotMeta struct { |
| + V int `json:"v"` |
| + Box string `json:"box"` |
| + Fingerprint string `json:"fingerprint"` |
| + CreatedAt time.Time `json:"created_at"` |
| + Bytes int `json:"bytes"` |
| +} |
| + |
| +// registrySnapshotPass snapshots the registry into the bucket when its |
| +// fingerprint moved since *lastFP. One pass; the loop owns cadence. *lastFP |
| +// is updated only after every write (including pruning) succeeded, so a |
| +// partial failure is retried whole next tick. |
| +func registrySnapshotPass(ctx context.Context, s release.Store, reg *registry.Registry, box string, now time.Time, lastFP *string) error { |
| + fp, err := reg.Fingerprint() |
| + if err != nil { |
| + return fmt.Errorf("snapshot: fingerprint: %w", err) |
| + } |
| + if fp == *lastFP { |
| + return nil |
| + } |
| + |
| + dir, err := os.MkdirTemp("", "carlos-registry-snapshot") |
| + if err != nil { |
| + return err |
| + } |
| + defer os.RemoveAll(dir) |
| + file := filepath.Join(dir, "registry.db") |
| + if err := reg.SnapshotTo(file); err != nil { |
| + return fmt.Errorf("snapshot: vacuum: %w", err) |
| + } |
| + body, err := os.ReadFile(file) |
| + if err != nil { |
| + return err |
| + } |
| + |
| + // Generation first (create-only — a stamp collision is a real conflict), |
| + // then current + sidecar (replace with the CAS-retry recipe |
| + // usage.UpsertEdgeDay uses), then prune. |
| + genKey := snapshotGenPrefix + now.UTC().Format(snapshotStamp) + ".db" |
| + if _, err := s.Put(ctx, genKey, body, ""); err != nil && !errors.Is(err, release.ErrCASConflict) { |
| + return fmt.Errorf("snapshot: put %s: %w", genKey, err) |
| + } |
| + if err := putReplacing(ctx, s, snapshotKey, body); err != nil { |
| + return err |
| + } |
| + meta := snapshotMeta{V: 1, Box: box, Fingerprint: fp, CreatedAt: now.UTC(), Bytes: len(body)} |
| + metaRaw, err := json.Marshal(meta) |
| + if err != nil { |
| + return err |
| + } |
| + if err := putReplacing(ctx, s, snapshotMetaKey, metaRaw); err != nil { |
| + return err |
| + } |
| + |
| + gens, err := s.List(ctx, snapshotGenPrefix) |
| + if err != nil { |
| + return fmt.Errorf("snapshot: list generations: %w", err) |
| + } |
| + sort.Strings(gens) // stamps are lexically chronological |
| + for len(gens) > snapshotKeep { |
| + if err := s.Delete(ctx, gens[0]); err != nil { |
| + return fmt.Errorf("snapshot: prune %s: %w", gens[0], err) |
| + } |
| + gens = gens[1:] |
| + } |
| + |
| + *lastFP = fp |
| + return nil |
| +} |
| + |
| +// putReplacing writes key regardless of prior state: create-only when |
| +// absent, etag-replace when present, re-reading on a lost race. Three |
| +// attempts, like usage.UpsertEdgeDay. |
| +func putReplacing(ctx context.Context, s release.Store, key string, body []byte) error { |
| + for attempt := 0; attempt < 3; attempt++ { |
| + _, etag, err := s.Get(ctx, key) |
| + switch { |
| + case errors.Is(err, os.ErrNotExist): |
| + etag = "" |
| + case err != nil: |
| + return fmt.Errorf("snapshot: get %s: %w", key, err) |
| + } |
| + if _, err := s.Put(ctx, key, body, etag); err == nil { |
| + return nil |
| + } else if !errors.Is(err, release.ErrCASConflict) { |
| + return fmt.Errorf("snapshot: put %s: %w", key, err) |
| + } |
| + } |
| + return fmt.Errorf("snapshot: %s: lost CAS race three times", key) |
| +} |
| + |
| +// registrySnapshotLoop runs the pass every 5 minutes + jitter, forever. |
| +// Platform-box-only, like retentionLoop: a fleet box's registry is not the |
| +// deployment's control plane. Wired in Task 3; rand is used there. |
| +var _ = rand.N[time.Duration] // placeholder reference removed in Task 3 |
| +var _ = log.Printf // placeholder reference removed in Task 3 |
| +``` |
| + |
| +Note: the two `var _ =` lines exist only if the compiler complains about unused imports before Task 3 lands the loop; if you implement Task 3 in the same session immediately after, omit them and the imports they justify until then. Prefer omitting: keep `log`, `math/rand/v2` out of the import block in this task and add them in Task 3. |
| + |
| +- [ ] **Step 4: Run to verify pass** |
| + |
| +Run: `go test ./internal/host/ -run TestSnapshotPass -count=1` |
| +Expected: PASS (all three tests). |
| + |
| +- [ ] **Step 5: Commit** |
| + |
| +```bash |
| +git add internal/host/snapshot_loop.go internal/host/snapshot_loop_test.go |
| +git commit -m "host: registry snapshot pass — current + sidecar + pruned generations under control/" |
| +``` |
| + |
| +--- |
| + |
| +### Task 3: wire `registrySnapshotLoop` into the agent |
| + |
| +**Files:** |
| +- Modify: `internal/host/snapshot_loop.go` (add the loop func) |
| +- Modify: `internal/host/agent.go:294-308` (one `go` line in the loop block) |
| + |
| +**Interfaces:** |
| +- Consumes: `registrySnapshotPass` (Task 2), `openDeployStore` (`internal/host/agent.go:2020`), `shouldSkipPublisherElection` (used by `retentionLoop`, agent.go:1489), `metricsBoxLabel(os.Getenv, readMachineID())` (the box label `usageLoop` uses, `internal/host/usage_loop.go:47`). |
| + |
| +- [ ] **Step 1: Add the loop to `snapshot_loop.go`** |
| + |
| +```go |
| +// registrySnapshotLoop snapshots the registry to the deployment bucket every |
| +// 5 minutes + jitter when it changed — rung 1 of the HA ladder (cold |
| +// standby restores this file). Platform-box job, retentionLoop's rules: |
| +// skipped on fleet boxes, disabled with a log line when there is no bucket. |
| +func registrySnapshotLoop(reg *registry.Registry) { |
| + ctx := context.Background() |
| + if shouldSkipPublisherElection(os.Getenv) { |
| + log.Printf("agent: registry snapshot loop is a platform-box job — skipped on a fleet box") |
| + return |
| + } |
| + store, err := openDeployStore(ctx) |
| + if err != nil { |
| + log.Printf("agent: registry snapshot loop disabled: %v", err) |
| + return |
| + } |
| + box := metricsBoxLabel(os.Getenv, readMachineID()) |
| + log.Printf("agent: registry snapshot loop armed (5min + jitter, %s)", snapshotKey) |
| + var lastFP string |
| + delay := time.Minute |
| + for { |
| + time.Sleep(delay) |
| + delay = 5*time.Minute + rand.N(time.Minute) |
| + if err := registrySnapshotPass(ctx, store, reg, box, time.Now(), &lastFP); err != nil { |
| + log.Printf("agent: registry snapshot pass: %v", err) |
| + } |
| + } |
| +} |
| +``` |
| + |
| +Add `log`, `math/rand/v2` (import as used by `rand.N` — match `retentionLoop`'s import spelling in agent.go) to the file's imports. |
| + |
| +- [ ] **Step 2: Wire into `RunAgent`** |
| + |
| +In `internal/host/agent.go`, in the loop block at lines 294-308, directly after `go retentionLoop(reg)`: |
| + |
| +```go |
| +go registrySnapshotLoop(reg) |
| +``` |
| + |
| +- [ ] **Step 3: Full gate** |
| + |
| +Run (sandbox off if the toolchain complains): |
| +```bash |
| +GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go build ./... && \ |
| +GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go vet ./... && \ |
| +GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go test ./... -count=1 && \ |
| +GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off GOOS=darwin go build ./... && \ |
| +gofmt -l internal/host internal/registry |
| +``` |
| +Expected: all clean; `gofmt -l` prints nothing for touched files. |
| + |
| +- [ ] **Step 4: Commit** |
| + |
| +```bash |
| +git add internal/host/snapshot_loop.go internal/host/agent.go |
| +git commit -m "host: arm registry snapshot loop (5min, platform boxes only)" |
| +``` |
| + |
| +--- |
| + |
| +### Task 4: EC2 auto-recovery alarm in the region-host module |
| + |
| +**Files:** |
| +- Modify: `infra/modules/region-host/main.tf` (append one resource; the instance is `resource "aws_instance" "host"` at main.tf:154, EIP at :178) |
| +- Modify: `infra/modules/region-host/variables.tf` if the module keeps variables there (inspect; otherwise the top of main.tf) — add `enable_auto_recovery`. |
| + |
| +**Interfaces:** |
| +- Produces: every `region-host` module instance (both production boxes, dev, and the Task 6 standby) gets a CloudWatch recover alarm by default; `enable_auto_recovery = false` opts out. |
| + |
| +- [ ] **Step 1: Add the variable** (default keeps every existing caller unchanged-but-armed after Paul's next apply) |
| + |
| +```hcl |
| +variable "enable_auto_recovery" { |
| + description = "Arm an EC2 auto-recovery alarm on the host (HA ladder rung 0)." |
| + type = bool |
| + default = true |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Add the alarm resource** to `infra/modules/region-host/main.tf`: |
| + |
| +```hcl |
| +# HA ladder rung 0 (2026-08-12 spec): system-status failure recovers the |
| +# instance onto healthy hardware, keeping instance id, EIP and volumes. |
| +resource "aws_cloudwatch_metric_alarm" "host_auto_recover" { |
| + count = var.enable_auto_recovery ? 1 : 0 |
| + alarm_name = "${var.name}-auto-recover" |
| + namespace = "AWS/EC2" |
| + metric_name = "StatusCheckFailed_System" |
| + statistic = "Maximum" |
| + period = 60 |
| + evaluation_periods = 2 |
| + threshold = 1 |
| + comparison_operator = "GreaterThanOrEqualToThreshold" |
| + alarm_actions = ["arn:aws:automate:${data.aws_region.current.name}:ec2:recover"] |
| + dimensions = { |
| + InstanceId = aws_instance.host.id |
| + } |
| +} |
| +``` |
| + |
| +If the module has no `data "aws_region" "current"`, add it: |
| +```hcl |
| +data "aws_region" "current" {} |
| +``` |
| + |
| +- [ ] **Step 3: Format check** |
| + |
| +Run: `tofu fmt infra/modules/region-host/` (or `terraform fmt` if tofu is absent; if neither binary exists, hand-verify two-space HCL alignment). |
| +Expected: no diff output beyond the files just written. |
| + |
| +- [ ] **Step 4: Commit** |
| + |
| +```bash |
| +git add infra/modules/region-host/ |
| +git commit -m "infra: EC2 auto-recovery alarm on region-host boxes (HA rung 0)" |
| +``` |
| + |
| +Note for the PR body: pinning callers reference this module by `?ref=main`, so the alarm arms on Paul's next apply per deployment — record that as a queued operator action, not something this PR performs. |
| + |
| +--- |
| + |
| +### Task 5: platform PR |
| + |
| +- [ ] **Step 1: Re-run the full Task 3 Step 3 gate** (all packages, darwin cross-compile, gofmt). Expected: clean. |
| + |
| +- [ ] **Step 2: Push and open the PR** |
| + |
| +```bash |
| +git push -u origin ha-snapshot |
| +gh pr create --title "host: registry snapshot to bucket + auto-recovery alarm (HA rung 0/1)" --body "Implements the platform half of Part I of the HA posture spec (PR #154, #153). |
| + |
| +- \`registry.SnapshotTo\` — consistent VACUUM INTO copy on a fresh connection |
| +- 5-min snapshot pass: \`control/registry-snapshot.db\` + sidecar + 8 pruned generations, CAS-safe, platform boxes only |
| +- EC2 auto-recovery alarm added to the region-host module (was absent fleet-wide) |
| + |
| +Companion PR in platform-infrastructure adds the count=0 standby box + failover runbook. Queued operator action: applies per deployment arm the alarm. |
| + |
| +🤖 Generated with [Claude Code](https://claude.com/claude-code)" |
| +``` |
| + |
| +--- |
| + |
| +### Task 6: standby box in platform-infrastructure |
| + |
| +**Files (repo `/home/paulca/github.com/carlosframework/platform-infrastructure`, branch `edge-standby` off its default branch):** |
| +- Create: `tito/standby-ie.tf` |
| + |
| +**Interfaces:** |
| +- Consumes: the `region-host` module exactly as `tito/region-ie.tf:12-28` uses it (source `git::https://github.com/carlosframework/platform.git//infra/modules/region-host?ref=main`). |
| +- Produces: `module.host_ie_standby` with `count = 0` — applying with count=1 is the runbook's step 1. |
| + |
| +- [ ] **Step 1: Read `tito/region-ie.tf` in full**, then write `tito/standby-ie.tf` copying its module block exactly, with these deltas: |
| + |
| +```hcl |
| +# Cold-standby edge box (HA ladder rung 1, platform spec 2026-08-12). |
| +# count stays 0 in normal operation; the failover runbook (RUNBOOK-edge- |
| +# failover.md) flips it to 1. It reuses host-init.sh so a fresh standby |
| +# converges from the bucket like any new box; promotion is the EIP remap. |
| +module "host_ie_standby" { |
| + count = 0 |
| + source = "git::https://github.com/carlosframework/platform.git//infra/modules/region-host?ref=main" |
| + name = "carlos-tito-ie-standby" |
| + sg_description = "carlos tito IE standby host" |
| + iam_name = "carlos-tito-host-ie-standby" |
| + instance_type = "t4g.medium" |
| + root_volume_size = 40 |
| + user_data = file("${path.module}/host-init.sh") |
| + deployment_bucket_arn = module.buckets.deployment_bucket_arn |
| + data_bucket_arn = module.buckets.data_bucket_arn |
| + ssm_parameter_arn = "arn:aws:ssm:eu-west-1:${local.account}:parameter/carlos/*" |
| + route53_zone_ids = ["ZEBE064UEO9O8"] |
| +} |
| +``` |
| + |
| +Copy any argument present in the live `module "host_ie"` block that this sketch omits, verbatim — the live file wins over this sketch wherever they differ (it may have gained arguments since this plan was written). The module mints its own EIP (`aws_eip.host`, region-host/main.tf:178); the standby's unused EIP while count=1 is a documented few-dollars cost in the runbook, not something to engineer away tonight. |
| + |
| +- [ ] **Step 2: Format + commit** |
| + |
| +```bash |
| +tofu fmt tito/ |
| +git add tito/standby-ie.tf |
| +git commit -m "tito: count=0 cold-standby edge box (HA rung 1)" |
| +``` |
| + |
| +(Do NOT run `tofu plan`/`apply` — memory records live drift in this repo; targeted-apply discipline is Paul's.) |
| + |
| +--- |
| + |
| +### Task 7: the failover runbook |
| + |
| +**Files:** |
| +- Create: `tito/RUNBOOK-edge-failover.md` (platform-infrastructure repo, same branch) |
| + |
| +**Interfaces:** |
| +- Consumes: key names from Task 2 (`control/registry-snapshot.db`, `control/registry-snapshot.json`), the standby module from Task 6. |
| + |
| +- [ ] **Step 1: Write the runbook** with exactly these sections (prose adjusted to what Task 6 actually shipped; commands verified against the real resource/module names in this repo — read `tito/region-ie.tf`, `tito/outputs.tf`, `tito/dns.tf` first and correct any name below that drifted): |
| + |
| +```markdown |
| +# Edge-box failover — carlos-tito-ie (cold standby, HA rung 1) |
| + |
| +Scope: the edge box died (hardware, AZ, unbootable). EC2 auto-recovery |
| +(same instance, same volumes) is the first responder and needs nobody; |
| +this runbook is for when the box itself is unrecoverable. |
| + |
| +What you lose and don't lose |
| +- Lost: instance state on the dead box's disk that had no Replication |
| + setting; up to 5 minutes of registry changes (snapshot cadence); |
| + in-flight requests. |
| +- Not lost: releases/channels/pointers (bucket), the registry as of the |
| + last snapshot (control/registry-snapshot.db — check |
| + control/registry-snapshot.json for created_at/fingerprint before |
| + trusting it), replicated instance DBs, TLS (re-minted). |
| + |
| +Steps |
| +1. Provision the standby: in tito/standby-ie.tf set count = 1; |
| + tofu apply -target=module.host_ie_standby. Wait for SSM to see the |
| + instance (aws ssm describe-instance-information). |
| +2. Restore the registry BEFORE first adoption: via SSM on the standby — |
| + systemctl stop carlos-edge.service (if running); |
| + aws s3 cp s3://<deployment-bucket>/control/registry-snapshot.db /data/registry.db; |
| + chown carlos:carlos /data/registry.db; then systemctl start |
| + carlos-edge.service. (host-init.sh + update-platform-binary.sh have |
| + already armed the path units; verify carlos-stage.path and |
| + carlos-restart.path are active.) |
| +3. Remap the EIP: aws ec2 disassociate-address --association-id <assoc of |
| + the dead box's EIP>; aws ec2 associate-address --allocation-id |
| + <allocation-id from tofu state: module.host_ie.aws_eip.host> |
| + --instance-id <standby instance id>. DNS does not change — every |
| + record points at the EIP. |
| +4. Let convergence run: the standby's 2s serial poll + full pass set |
| + rebuilds instances from the bucket. Certs re-mint per host on first |
| + traffic (HTTP-01 now lands on the standby); wildcards via the DNS-01/ |
| + DNSimple path (token at SSM /carlos/dnsimple-token). |
| +5. Verify — never against a cached answer: |
| + curl -sI --resolve go.tito.io:443:<EIP> https://go.tito.io | grep -i x-carlos-version |
| + then a console login at console.platform.tito.io, then |
| + carlos instances against the console API. |
| +6. Afterwards: the old box's row cleanup, standby promotion to primary in |
| + tofu (rename/moved), and a fresh count=0 standby are follow-up work in |
| + daylight — record what the rehearsal actually required here. |
| + |
| +Rehearsal |
| +Run this end-to-end on the dev deployment (Kass account 640012953448) |
| +before trusting it; record the measured RTO here. Target from the spec: |
| +under 60 minutes. |
| +- Rehearsed: (date, RTO, notes) — NOT YET DONE |
| +``` |
| + |
| +The tito wrinkle from memory that MUST be reflected when writing step 2: the box carries a second registry at `/data/titogo/registry.db` (titogo's own app DB, holds its auth) — the platform snapshot does NOT cover it; note that titogo's DB restores via its own litestream/Replication path (`tito/litestream-read.tf` is the read side) and the runbook must say so explicitly in "What you lose and don't lose". |
| + |
| +- [ ] **Step 2: Commit + PR** |
| + |
| +```bash |
| +git add tito/RUNBOOK-edge-failover.md |
| +git commit -m "tito: edge-box cold-standby failover runbook (HA rung 1)" |
| +git push -u origin edge-standby |
| +gh pr create --title "tito: cold-standby edge box + failover runbook (HA rung 1)" --body "Companion to platform PR (registry snapshot pass) — Part I of the HA posture spec (platform PR #154, platform #153). |
| + |
| +- count=0 standby region-host module instance |
| +- failover runbook: provision -> restore snapshot -> EIP remap -> converge -> verify |
| +- rehearsal on the dev deployment is the closing condition for rung 1 (queued; needs applies) |
| + |
| +NOT applied — no tofu run against live state (recorded drift; targeted applies are Paul's). |
| + |
| +🤖 Generated with [Claude Code](https://claude.com/claude-code)" |
| +``` |
| + |
| +--- |
| + |
| +## Self-review notes (already applied) |
| + |
| +- Spec coverage: snapshot pass (Tasks 1-3), auto-recovery action item (Task 4), standby tofu + runbook incl. tenant-data honesty and titogo second-registry wrinkle (Tasks 6-7). The rehearsal itself needs live applies and is recorded as queued for Paul, not a task here. |
| +- Flagship deployment: its box also uses the region-host module (alarm arms on apply); a flagship standby `.tf` is deliberately not written tonight — tito is the deployment #153 names, and the runbook pattern transfers. Recorded in the pinfra PR body as follow-up. |
| +- Type consistency: `registrySnapshotPass(ctx, s, reg, box, now, &lastFP)` matches between Tasks 2 and 3; key constants named identically in Tasks 2 and 7. |
diff --git a/docs/superpowers/specs/2026-08-12-ha-posture-and-multi-upstream-routes-design.md b/docs/superpowers/specs/2026-08-12-ha-posture-and-multi-upstream-routes-design.md| new file mode 100644 |
| index 000000000..1d80df87c |
| --- /dev/null |
| +++ b/docs/superpowers/specs/2026-08-12-ha-posture-and-multi-upstream-routes-design.md |
| @@ -0,0 +1,251 @@ |
| +# HA posture and multi-upstream routes |
| + |
| +**Date:** 2026-08-12 |
| +**Status:** Approved design |
| +**Prompted by:** #152 (multi-upstream, health-checked addr routes) and #153 |
| +(the edge box is a per-deployment SPOF), both filed out of the Rails-behind- |
| +the-edge PoC's §7 HA fallout (`2026-08-11-rails-behind-carlos-edge-poc-design.md`, |
| +PR #150). This doc answers both at once because they are two altitudes of |
| +one problem: #152 raises the availability of the *upstream* layer, #153 is |
| +the availability ceiling above it — the edge box that terminates TLS for |
| +every route on a deployment. |
| + |
| +## Context |
| + |
| +- Every deployment (flagship, tito, dev) routes all traffic through one |
| + edge box. On tito that box (`carlos-tito-ie`) fronts go.tito.io, every |
| + instance, the console alias, and — since the Rails PoC — potentially |
| + Tito Classic/Pro staging. New tenants widen the blast radius of an |
| + existing, deliberately recorded risk; they don't create it. |
| +- An `addr` route today is one SQLite row with one `Addr` |
| + (`internal/registry/registry.go`). The edge's `tcpProxy` |
| + (`internal/edge/edge.go`) sets no dial timeout and no `ErrorHandler`: a |
| + dead upstream surfaces as the stdlib's bare 502, still stamped with the |
| + route-level `X-Carlos-Version`. The only "resolve a live target per |
| + request" precedent is `fleetProxy`'s tunnel-channel resolution; the only |
| + retry machinery is unix-socket-specific (`dialUnixWithRetry`). |
| +- The control bucket already holds releases, channels, pointers, and — |
| + once PR #137 (route records) merges — provisioned-route desired state. |
| + Genuinely box-local state: the registry DB (`/data/registry.db`), minted |
| + TLS certs, instance data dirs, and the running processes. #137's |
| + `shapeMismatch` deliberately refuses record authority over box-scoped |
| + rows (`Backing`, `Tunnel`, `Caps`, `Platform`, `Canary`), so records |
| + alone cannot reconstruct a full registry today. |
| +- Addr routes are forbidden from `Hibernate` |
| + (`provision.ValidateShape`, on the #137 branch), so probing an addr |
| + upstream can never trigger a wake. That invariant is load-bearing for |
| + this design; relaxing it later would reintroduce the wake hazard. |
| + |
| +## Decisions taken during design (Paul, 2026-08-12) |
| + |
| +- **One doc for both issues** — the #153 ceiling decided first, #152 |
| + designed within it; implementation still lands separately. |
| +- **#153 is a ladder, not a point choice**: accepted-risk now, cold |
| + standby documented and built, warm standby sketched behind explicit |
| + triggers. |
| +- **#152 designed implementation-ready**, riding #137's record shape |
| + additively — not roadmap-priced shape-only. |
| +- **#152 mechanism: active probe + in-request dial retry** (option C). |
| + Active probing owns fail-away/fail-back and dashboard truth; a dial |
| + failure additionally ejects the upstream and retries in-request so box |
| + death never surfaces a 502 even between probe ticks. Probe-only and |
| + passive-only were considered and rejected (the first 502s during the |
| + probe gap; the second has dial-level health only and no clean |
| + rolling-deploy story). |
| +- **Cold-standby enabler: registry snapshot to the bucket now**; |
| + records-complete stays the long-term trajectory and retires the |
| + snapshot when it gets there. Records-complete-first was rejected as a |
| + large scope increase on an unmerged PR; runbook-only was rejected as |
| + rot-prone. |
| + |
| +--- |
| + |
| +## Part I — the availability ladder (#153) |
| + |
| +### Rung 0: today, written down as accepted risk |
| + |
| +One edge box per deployment terminates TLS for everything on it. Box |
| +death = every route on that deployment dark until recovery. Mitigations |
| +that are already real: EC2 auto-recovery (action item: verify it is |
| +actually enabled in tofu for both production boxes — it is not believed |
| +to be explicit today), the control bucket, and 2s serial convergence. |
| +This is not an oversight; the single-box posture is the recorded |
| +"multi-node failover is where an orchestrator earns its keep" decision. |
| +This rung exists so the risk is written where the migration evaluation |
| +can cite it. |
| + |
| +### Rung 1: cold standby (built by this design) |
| + |
| +**1. Registry snapshot to the bucket.** The box's converge loop gains a |
| +snapshot pass: every 5 minutes, if `registry.Fingerprint()` changed since |
| +the last snapshot, `VACUUM INTO` a temp copy and PUT it to the control |
| +bucket: current at `control/registry-snapshot.db`, with a sidecar JSON |
| +(`control/registry-snapshot.json`: created-at, fingerprint, box |
| +identity); prior generations at |
| +`control/registry-snapshots/<created-at>.db`, pruned beyond the newest |
| +8. This |
| +covers every row, including the box-scoped ones records cannot rebuild. |
| +Worst-case staleness is one interval; after restore, the reconcile pass |
| +corrects any record-backed drift. The snapshot is additive bucket |
| +content — nothing existing parses it, so no wire-shape risk. |
| + |
| +**2. TLS: re-mint, don't replicate.** Private keys never enter the |
| +bucket. A replacement box re-mints via HTTP-01 the moment it holds the |
| +EIP; wildcards go via the existing DNS-01/DNSimple path (token in SSM). |
| +Cost: a brief per-host TLS brownout on first hits after failover — |
| +accepted at this rung. Let's Encrypt duplicate-certificate limits are |
| +generous enough for disaster recovery. |
| + |
| +**3. Runbook + rehearsal.** A pre-written standby `.tf` (count = 0) lands |
| +in platform-infrastructure per deployment. The runbook, kept next to it: |
| + |
| +1. `tofu apply` the standby box. |
| +2. Remap the EIP to it. |
| +3. Restore `control/registry-snapshot.db` to `/data/registry.db`. |
| +4. Arm the path units (`update-platform-binary.sh` does this). |
| +5. Verify: `X-Carlos-Version` over the remapped EIP via |
| + `curl -sI --resolve`, then a console login. Never a cached answer. |
| + |
| +The runbook explicitly separates platform recovery from tenant data: |
| +instance DBs come back only via their own `Replication` setting; |
| +unreplicated instance state on the dead box is lost, and the runbook says |
| +so plainly. The rung is not "done" until rehearsed once on the dev |
| +deployment (Kass account) with the measured RTO recorded in this doc's |
| +follow-up. Target: under 60 minutes; the rehearsal replaces this guess |
| +with a number. |
| + |
| +### Rung 2: warm standby (trigger-gated, sketched only) |
| + |
| +A second box converging the same channels, promoted by EIP remap for a |
| +~1–2 minute RTO. Known open problems, recorded but deliberately not |
| +solved here: |
| + |
| +- A standby cannot answer HTTP-01 without the EIP — pre-minted certs need |
| + DNS-01-everything or a cert-sync channel. |
| +- The console must stay single-writer; a standby console is read-only or |
| + off. |
| +- Exec/unit-backed tenants cannot run on two boxes at once, so an honest |
| + warm standby is closer to a continuously-rehearsed cold one for |
| + everything except addr/static/tunnel routes. |
| + |
| +**Triggers to design rung 2 for real:** the first paying or SLA-bearing |
| +traffic on a deployment (the Rails production cutover is the named |
| +candidate), or the first time measured rung-1 RTO exceeds what a real |
| +tenant conversation can bear. |
| + |
| +--- |
| + |
| +## Part II — multi-upstream, health-checked addr routes (#152) |
| + |
| +### Record and registry shape (additive, rides #137) |
| + |
| +The route record gains two optional fields: |
| + |
| +```json |
| +{ |
| + "addrs": ["10.0.1.10:3000", "10.0.2.11:3000"], |
| + "health": { |
| + "path": "/up", |
| + "status": 200, |
| + "host_header": "", |
| + "interval_ms": 2000, |
| + "fail_after": 3, |
| + "rise_after": 2 |
| + } |
| +} |
| +``` |
| + |
| +Defaults: path `/`, status `200`, host header = route host, 2000 ms, 3, |
| +2. **Implementation amendment (2026-08-12, overnight build):** the record's |
| +wire key for this object is `probe`, not `health` — the instances list API |
| +already ships a computed `health` status string (dashboard truth+scale), |
| +and embedding both collided. Go names keep the spec's `Health` vocabulary; |
| +only the JSON tag differs. Flagged for Paul's review. The legacy singular `addr` stays; `AddrsOrDefault()` folds it into a |
| +one-element list, so every record written before this change parses and |
| +behaves identically (the wire-shape rule). `health` on a single-addr |
| +route is legal and useful: it buys dashboard state and a 503 with |
| +`Retry-After` instead of today's bare 502. |
| + |
| +Registry side: two new TEXT columns — `addrs` (JSON array) and `health` |
| +(JSON blob) — both folded into `Fingerprint()` so the 2s watch notices |
| +changes, and both compared in `shapeMismatch` with canonicalized |
| +ordering. `routeFor`, the single shared mint/adopt builder on the #137 |
| +branch, is the one place the widening happens. |
| + |
| +### The prober |
| + |
| +For each route with `health` set (or `len(addrs) > 1`, which implies the |
| +default health config), the edge runs one background prober: direct |
| +`GET http://<addr><path>` per upstream with the route host as `Host` |
| +header and a 2 s client timeout. `fail_after` consecutive failures → |
| +upstream leaves the live set; `rise_after` consecutive passes → rejoins. |
| +Transitions are logged with the reason (dial error vs wrong status). |
| +Probes dial upstreams directly, never the edge's public host — combined |
| +with the no-hibernate invariant for addr routes, there is no wake hazard. |
| +Probers start and stop at `Reload()` when the fingerprint moves. |
| + |
| +### The request path |
| + |
| +Live-upstream resolution happens at RoundTrip time (the `fleetProxy` |
| +precedent), round-robin across the live set — these upstreams are |
| +replicas, so the edge does the balancing an NLB would have. On a **dial** |
| +failure (no bytes sent, so safe for any method) the upstream is ejected |
| +immediately — not waiting for the probe to notice — and the request |
| +retries once against another live upstream. Never a retry after bytes |
| +have flowed. Empty live set → 503 + `Retry-After: 2`, mirroring |
| +`errFleetDark`; the bare-502 path disappears for health-configured |
| +routes. `X-Carlos-Version` stays route-level as today: addr upstreams are |
| +processes CARLOS didn't build, and per-upstream version attribution is |
| +out of scope. |
| + |
| +### Surfaces |
| + |
| +Per-upstream state (up/down, last transition, consecutive-failure count) |
| +flows into the existing readings/sweep path, so the dashboard's instance |
| +page lists each upstream with its state. The instances API carries the |
| +same, and `carlos instances` shows `2/3 healthy`-style summaries. Route |
| +mutation goes through the console API / route-records door like |
| +everything else post-#137 — no box-local-only verb (the CLI no-infra |
| +rule). |
| + |
| +### Rolling deploys |
| + |
| +V1 needs no drain verb: the deploy stops the app on box 1 → the probe |
| +fails it away → deploy → probe passes → fail-back → next box. |
| +`fail_after × interval_ms` (~6 s at defaults) is the window in which |
| +requests can still land on a stopping box; the in-request dial retry |
| +covers them. An explicit per-upstream drain/maintenance flag is named as |
| +a future nicety, not built. |
| + |
| +--- |
| + |
| +## Testing |
| + |
| +- Unit tests for prober state transitions (fail-away at `fail_after`, |
| + fail-back at `rise_after`, dial-error vs wrong-status reasons) against |
| + `httptest` upstreams. |
| +- An edge e2e with two backends where killing one mid-request-stream |
| + produces zero 5xx responses. |
| +- Golden compatibility tests proving pre-change records and registry rows |
| + still parse and produce today's behavior (the rule this repo has paid |
| + for three times). |
| +- Snapshot pass: a test that a restored snapshot plus a reconcile pass |
| + reproduces the source registry's routes. |
| + |
| +## Sequencing |
| + |
| +1. PR #137 (route records) merges first — #152's shape rides it. |
| +2. Part I's snapshot + runbook are independent of #137 and #152 and can |
| + proceed in parallel; the rehearsal on the dev deployment closes rung 1. |
| +3. #152 edge work follows #137; the Rails staging boxes are the first |
| + live consumer, replacing their internal-NLB rung when it lands. |
| + |
| +## Out of scope |
| + |
| +- An orchestrator, in any form (#153 says so explicitly). |
| +- Data HA: Postgres/Redis/instance-DB replication have their own stories |
| + (RDS/Elasticache multi-AZ; the route `Replication` column). |
| +- Warm-standby design (rung 2) until a trigger fires. |
| +- Per-upstream version attribution in `X-Carlos-Version`. |
| +- A drain/maintenance verb for upstreams (recorded as future nicety). |
| +- Relaxing the addr-routes-never-hibernate invariant. |