| index 34741bc8a..743c01469 100644 |
| --- a/docs/superpowers/plans/2026-08-22-multi-region-backhaul.md |
| +++ b/docs/superpowers/plans/2026-08-22-multi-region-backhaul.md |
| @@ -29,7 +29,11 @@ as the house idiom for bucket-fed edge state. |
| - Spec: `docs/superpowers/specs/2026-08-20-multi-region-ingress-design.md` |
| Part 3. Key shapes verbatim: `control/edges/<box>.json`, |
| `control/edge-mesh-key`. Hop headers verbatim: `X-Carlos-Hop: 1` and |
| - `X-Carlos-Hop-Auth: <hex hmac>`. |
| + `X-Carlos-Hop-Auth: <unix>:<hex>` (**amendment, 2026-08-22, T7**: this |
| + line originally read `<hex hmac>`, omitting the leading unix timestamp |
| + `signHop` actually prepends — File Structure's own Task 2 Interfaces |
| + section below had the right shape all along; ledgered as a Task 2 minor |
| + finding, fixed here rather than left to drift from the code). |
| - **Nothing writes `control/serial`.** Task 1 amends the spec's Part 3 |
| line that said the edges file is serial-bumped (same correction the |
| certs took in stage 1-2): edges files ride their own 30s poll. |
| @@ -455,15 +459,245 @@ func TestHomeMapFoldsAndExcludesOwnBox(t *testing.T) { |
| - Modify: the spec (Part 3 rows → implemented-dormant with commits); |
| this plan (rollout notes below). |
| |
| -- [ ] Fill the rollout notes: stage-5 arming now needs — per-box edges |
| +- [x] Fill the rollout notes: stage-5 arming now needs — per-box edges |
| files verified present for every armed box BEFORE steering (plan 3) |
| ever points DNS at a second edge; mesh-key presence check; the |
| backhaul path exercised on the dev/Kass deployment before flagship AU |
| (first two-real-boxes deployment); #244's bare-edge note extends to |
| the mesh key (a bare edge with a bucket can read it — same trust |
| envelope as certs, record it). |
| -- [ ] Full gate; commit `docs: backhaul rollout notes`. |
| +- [x] Full gate; commit `docs: backhaul rollout notes`. |
| |
| ## Rollout notes (filled by Task 7 at execution) |
| |
| -- (filled at execution) |
| +Synthesized from the SDD ledger (`progress.md`) across all six |
| +implementation tasks and their reviews. This plan ships stages 1–3 (the |
| +spec's Part 3 rows) dormant on single-box deployments; nothing below is |
| +armed by merging this branch — it is the checklist and the open-questions |
| +list for the rollout that follows, including the stage-5/6 arming this |
| +plan's own spec Part 4 describes. |
| + |
| +### Stage-5 arming checklist (before flagship AU, before any real second box) |
| + |
| +1. **Per-box edges files verified present for every armed box, BEFORE |
| + plan 3 (steering) ever points DNS at a second edge.** The home map is |
| + built by folding `control/edges/<box>.json` for every box the mirror |
| + can list (`internal/edge/homemap.go`, Task 3); a box that never |
| + published one is invisible to every other edge's map, so a route |
| + homed there is neither served locally by strangers' hellos (fine) nor |
| + backhaul-reachable (NOT fine, once steering can legitimately land |
| + traffic on a box that has nothing to route it with). No CLI/console |
| + surface exists for this today — same box-local-only shape as the |
| + registry-snapshot check plan 1's rollout notes carry — so it is a |
| + direct-bucket list, consistent with CLAUDE.md's operate-via-API-and-CLI |
| + carve-out for platform-internal work: |
| + |
| + ```bash |
| + aws --profile <profile> s3 ls s3://<control-bucket>/control/edges/ |
| + ``` |
| + |
| + Expect exactly one `<box>.json` per box that is supposed to be armed. |
| + A missing entry means that box's `edgesFileLoop` has not ticked yet |
| + (5min + jitter cadence) or is not running — resolve before arming |
| + steering, not after. |
| +2. **Mesh-key presence check.** `control/edge-mesh-key` must exist and |
| + every armed box's mirror must have it loaded before any hop can verify |
| + (`internal/edge/mesh.go`, Task 2). Same direct-bucket check: |
| + `aws s3 ls s3://<control-bucket>/control/edge-mesh-key`. Absence means |
| + no box has yet won the housekeeper election with `mintMeshKeyLoop` |
| + armed — check the housekeeper's acquisition log line |
| + (`internal/host/housekeeper.go`) on whichever box should hold it. |
| + **Rotation runbook pointer:** there is no standalone rotation runbook |
| + document yet — the only written procedure is this plan's own Task 2 |
| + Global Constraints section above (`control/edge-mesh-key.next`, |
| + dual-accept read on both `verifyHop` callers, promotion = operator |
| + copies `.next` over the key and deletes `.next`). File a real runbook |
| + doc before the first live rotation rather than pointing operators at a |
| + plan file; until then, this paragraph IS the pointer. **Propagation |
| + window is ≤1 poll** (30s, the home-map/mesh-key mirror's shared |
| + cadence) per Task 2's ledgered honest-header fix (`0c07490`) — during |
| + that window some edges verify hops under the old key, some under |
| + `.next`; dual-accept on both is what makes that safe, not a race to |
| + avoid. |
| +3. **Exercise the backhaul path on dev/Kass before flagship AU.** |
| + Flagship AU is the spec's designated first target because a steering |
| + mistake there "burns nobody's ticket sales" (spec Part 1 decisions) — |
| + but nothing in this branch has run across two REAL boxes; Task 6's |
| + e2e (`67de75f`) is in-process with edges files written directly into a |
| + fake store (see the spec's Part 3 residual note, and Task 6's report |
| + §4.1). The dev/Kass deployment is the first two-real-boxes proving |
| + ground: arm a second box there, verify items 1–2 above hold live (not |
| + just in a fake store), drive one real cross-box hop with `curl |
| + --resolve` against the far box's EIP for a host homed on the near box, |
| + and confirm the marker/XFF/one-hop behaviour Task 6 pinned in-process |
| + also holds over a real network hop — before repeating any of this on |
| + flagship. |
| +4. **#244's bare-edge note extends to the mesh key.** Plan 1's rollout |
| + notes (item 5) ruled that a standalone `carlos edge` run outside the |
| + agent on a bucket-backed box is ungated and publishes to |
| + `control/certs/**` unconditionally — "don't run a bare `carlos edge` |
| + against a deployment bucket on a multi-edge deployment," tracked as |
| + issue #244. The same box, run the same way, also mirrors |
| + `control/edges/` and `control/edge-mesh-key` (Task 3's home-map mirror |
| + has no issuer gate — it is read-only bucket state, not the issuance |
| + path) — so a bare edge can read the mesh key and the whole |
| + deployment's home map exactly as it can read certs. This is not a new |
| + hole: it is the same trust statement Part 2 already makes explicit |
| + ("any box with deployment-bucket read already serves this deployment's |
| + traffic... holding TLS keys does not extend what a compromised box can |
| + do") extended to the one new key type Part 3 adds. Recorded here so |
| + #244's operational rule ("don't run a bare `carlos edge` against a |
| + deployment bucket on a multi-edge deployment") is read as covering the |
| + mesh key too, not just certs. |
| + |
| +### Open ruling requested for Paul |
| + |
| +5. **T7 egress ruling — under-metering of backhauled egress, including |
| + an abuse case.** By Task 3's hard constraint (T3 review) the ONLY |
| + usage meter (`edge.go:840`) sits below a successful local |
| + `lookupUpstream`; a backhauling edge misses it by construction, so a |
| + route's cross-region backhaul traffic is billed (AWS egress) on the |
| + SENDING box's account but never metered (CARLOS usage/billing) on |
| + either box — the bytes leave the sending box's NIC twice as expensive |
| + as a locally-served request and the platform's own metering sees none |
| + of it. Task 4 (`bdcf6f6`..`0193912`) confirmed this is not merely a |
| + missed-revenue gap: it is an **abuse surface**. Backhaul dials any |
| + host the home map names, on an unverified hello, before any hop trust |
| + check applies to the INBOUND request that triggered the dial — so a |
| + scanner sending forged `Host:` headers at a non-home edge for hosts |
| + that happen to be real elsewhere in the deployment generates real |
| + cross-region egress cost on the sending box's AWS bill, with no |
| + member, no account, and no CARLOS usage row to attribute it to. |
| + **Containment option on the table, not yet decided:** a per-home-box |
| + in-flight semaphore bounding concurrent backhaul dials toward any one |
| + home box — cheap (one counter, no new storage), does not fix |
| + attribution, but bounds the blast radius of a scanning flood to a |
| + fixed number of concurrent cross-region dials rather than unbounded. |
| + This needs a ruling before Part 4 (steering) sends real, sustained |
| + traffic through this path — today it is dormant/low-volume by |
| + construction (no steering, no DNS pointing anywhere but the home |
| + region), which is the window in which to decide, not after. Cross-refs |
| + unit-economics (the meter placement is `internal/usage`'s territory, |
| + not `internal/edge`'s to relocate unilaterally) and the object of this |
| + ruling should be recorded as a numbered issue once Paul rules on the |
| + containment approach. |
| + |
| +### Constraint for Part 4 / plan 3 (steering) to honor |
| + |
| +6. **Pool narrowing: steered non-shared-pool routes must not get |
| + out-of-pool latency records.** Task 5 (`15c3673`..`af006be`) confirmed |
| + `inScope` tests `poolServed` FIRST and unchanged — the everywhere-class |
| + widening in this plan only ever widens the REGION axis, never the pool |
| + axis, so a `tito`-pool static record is still refused by a box serving |
| + a different pool. Part 4's Route53 record generation must respect the |
| + same boundary: a steered route in a non-shared pool must only get |
| + latency records for the armed edges that actually serve its pool, not |
| + every armed edge in the deployment — otherwise DNS would advertise a |
| + path to a box that has never adopted the route and cannot serve it |
| + (the everywhere-class widening's pool refusal would make that box 404 |
| + or refuse the hop, not backhaul it, since the route never landed |
| + there). Stated as a plan-3 design constraint, not implemented here. |
| + |
| +### Operator/dashboard notes at second-region arming |
| + |
| +7. **`DNSStatus: "elsewhere"` false-alarm risk at second-region arming.** |
| + `dnsStatus` (`internal/domains/reconcile.go:465`) classifies a route's |
| + resolved IPs against `cfg.EdgeIPs` — the deployment's edge address list, |
| + sourced from `PLATFORM_EDGE_IPS` per box (`internal/host/agent.go:1371`). |
| + Each box runs its OWN `domains.Reconcile` pass and stamps its OWN |
| + registry. Once Task 5's everywhere-class adoption puts the SAME static |
| + routes on every box (not just the home box), every armed box now |
| + independently DNS-checks those hosts — and any box whose own |
| + `PLATFORM_EDGE_IPS` has not been updated to include the new region's |
| + EIP will read a real, correctly-resolving DNS answer (still pointing |
| + only at the original region, since Part 4 steering has not run yet) as |
| + `"elsewhere"`, because the resolved IP is not in that box's edge set. |
| + **Rollout action:** update `PLATFORM_EDGE_IPS` on every box to the |
| + full deployment-wide edge IP list at the same time a new region is |
| + armed, not after — otherwise the fleet dashboard (which aggregates |
| + per-box readings) shows real DNS noise for every everywhere-class host |
| + the instant the second box starts reconciling them, indistinguishable |
| + at a glance from an actual DNS problem. |
| +8. **Legacy public-IP single-`Addr` rows: enumerate before regions |
| + apply.** Task 5's `publiclyDialable` gate reclassifies any addr record |
| + whose upstreams are ALL public IP literals as everywhere-class the |
| + moment a second box exists — no code deploy required, just another |
| + box appearing. Loopback and RFC1918/ULA private addrs are excluded by |
| + design (home-boxed, unchanged), but a legacy single-`Addr` row that |
| + happens to carry a real public IP literal would silently start being |
| + probed and served from every armed box the day regions apply. Expected |
| + empty in both production deployments today, but unverified — enumerate |
| + before the first regions-apply: |
| + ```sql |
| + -- against each deployment's registry (read-only) |
| + SELECT host, addr FROM routes WHERE addr <> '' AND kind = ''; |
| + ``` |
| + then eyeball each `addr` against `publiclyDialable`'s rule |
| + (`net.ParseIP` non-nil, `IsGlobalUnicast()`, not private/loopback/ |
| + link-local) — any hit is a route to look at BEFORE arming, not after. |
| +9. **Edges-file `role: "home"` over-claims for static hosts — wants a |
| + `"replica"` role before Part 4 leans on the map.** Task 5's own |
| + concern (report §4.1): once static routes are everywhere-class, EVERY |
| + box that adopts one publishes it with `role: "home"` in its edges |
| + file (Task 1's schema has only ever written `"home"`), so the home map |
| + folds multiple boxes all claiming to home the same static host. The |
| + fold's duplicate rule (first box in sorted order wins) makes this |
| + harmless TODAY, because static routes never backhaul (Part 3's table: |
| + Everywhere class is "adopt and serve locally... zero backhaul") — the |
| + role is never consulted for a routing decision on this class. It stops |
| + being harmless the moment Part 4's read-local anticipation (spec |
| + "Anticipated, not built" §1) starts reading the `role` field for |
| + real — a reader that cannot distinguish "the box that actually homes |
| + this process" from "a box that adopted a static copy" cannot implement |
| + read-local correctly. The role vocabulary already has the room (spec: |
| + `{"role":"home"}` object form, not a bool) — this is a note for |
| + whoever designs Part 4's writer change, not a code change here. |
| +10. **Domains: a static app's apex row appears on far boxes — operator- |
| + surprise note, not a DNS-writer change.** Task 5 confirmed (report |
| + §4.1, item 3) that `domains.Reconcile` mints the platform apex on a |
| + far box too, since Task 5 makes its source route exist there. DNS |
| + itself still points only at the home region until Part 4 steering — |
| + **this is NOT a new DNS writer and does not change what answers a |
| + real client sees**, disproving any read of it as a DNS-authority |
| + change. It is purely a local-registry/console-visibility surprise: an |
| + operator inspecting a far box's own domain rows will now see an apex |
| + entry for an app they may not expect that box to know about at all. |
| + Worth a line in the fleet dashboard / `carlos apps` docs before |
| + stage 5, not a blocker. |
| + |
| +### Cross-referenced issues |
| + |
| +11. **#250 — everywhere-class rows with operator-touched |
| + Backing/Tunnel/Caps survive record deletion on far boxes ("far-box |
| + survivors").** Filed during Task 5's review round 1 (fix commit |
| + `af006be`). Since `15c3673`, a static or public-addr record is |
| + materialized on every platform box, and the removal phase reads the |
| + class off the ROW (`everywhereRoute`), which deliberately answers |
| + "home-boxed" for any row carrying `Backing`, `Tunnel` or `Caps` — a |
| + conservative choice that also means: mint a static record on a far |
| + box, `carlos route -grant`/`-backing`/`-tunnel` that far copy, delete |
| + the record — the home box retires its route, the far box keeps |
| + serving the retired site forever with its grants intact and no record |
| + anywhere to explain why. The gate stays conservative on purpose (the |
| + alternative hands an unattended bucket-record deletion the |
| + capability-grant purge's human-gated contract), so the fix is |
| + visibility + a supported cleanup verb, not a gate change. Relevant to |
| + this rollout because everywhere-class adoption (this plan) is what |
| + makes far-box copies routine for the first time — #250 was inert |
| + before this branch. |
| +12. **#179 — deployment-wide secrets key, tracked as a mesh blocker.** |
| + Task 5's concern (report §4.1, item 2): `appconfig.Sync` now renders |
| + an everywhere-class app's `<host>.env` on every box that adopts it, |
| + which is more copies of a tenant's rendered config at rest — not a |
| + new READ capability, since every box already holds the |
| + deployment-wide secrets key and could always read the whole store, |
| + the exact coupling #179 tracks. #179 is a pre-existing blocker (also |
| + surfaced on the trust-placement branch's mesh design work), not |
| + something this branch introduces; recorded here as the cross-ref |
| + Task 5's review asked for, not as new scope for this plan. |
| + |
| +**Full gate** (env `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod |
| +GOPROXY=off`): `go build ./...`, `go vet ./...`, `GOOS=darwin go build |
| +./...`, `go test ./... -count=1`, `gofmt -l` on touched files — all |
| +clean; this task's changes are docs-only (spec + this plan file), so |
| +the gate is trivially green (no `.go` file in the diff). |
amadan