amadan
docs: final-review fix wave — honest replay/heal prose, XFF amendment, rollout gates
Five doc/comment-only corrections from the whole-branch final review: - homemap.go replay paragraph: a captured Hop-Auth DOES buy XFF belief for that host within its window; the real defense is that capturing one requires breaking the hop's TLS or holding the mesh key outright. - staleHopRefusal doc: the real heal path is the edges-file cycle (5min+jitter publish + 30s mirror), not a 2s serial converge; Retry-After: 2 is a keep-trying hint, not a heal estimate. - spec Part 3: dated amendment — Task 3 shipped verbatim-restore of the inbound X-Forwarded-For chain, not append. - plan rollout note 8: add the missing pre-regions-apply scan for bucket records whose Region mismatches the deployment's single region. - plan rollout notes: new top item recording the final review's two MUST-FIX-BEFORE-STAGE-5 gates — seed the edges-file *last from the bucket at loop arm, and add visibility for the two silent stuck-state paths before steering sends sustained traffic.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
3 files changed,
+80
−5
docs/superpowers/plans/2026-08-22-multi-region-backhaul.md+58 −0docs/superpowers/specs/2026-08-20-multi-region-ingress-design.md+6 −0internal/edge/homemap.go+16 −5
diff --git a/docs/superpowers/plans/2026-08-22-multi-region-backhaul.md b/docs/superpowers/plans/2026-08-22-multi-region-backhaul.md| index 20c1ef82d..79e338ea9 100644 |
| --- a/docs/superpowers/plans/2026-08-22-multi-region-backhaul.md |
| +++ b/docs/superpowers/plans/2026-08-22-multi-region-backhaul.md |
| @@ -477,6 +477,42 @@ armed by merging this branch — it is the checklist and the open-questions |
| list for the rollout that follows, including the stage-5/6 arming this |
| plan's own spec Part 4 describes. |
| +### Gates from the whole-branch final review (2026-08-22) |
| + |
| +The whole-branch final review (2026-08-22) marked two items |
| +MUST-FIX-BEFORE-STAGE-5. Recorded here as gates on the checklist below, |
| +not left to stand alone in the review record: |
| + |
| +a. **Seed the edges-file change-gate (`*last`) from the bucket at loop |
| + arm.** `edgesFilePass`'s change-gate (`internal/host/edgesfile.go`, |
| + Task 1) compares each pass's body against `*last`, but `*last` starts |
| + nil at process start, in memory only. Two consequences follow from |
| + that until it is seeded: a box that restarts during a transient |
| + `readPublicIP()` miss (an IMDS blip) publishes `"eip":""` over its own |
| + last-good file instead of falling back to it — the `eip == "" && *last |
| + != nil` carry-forward this file already has only helps once `*last` |
| + is populated, and right after a restart it never is — and separately, |
| + every restart forces one gate-crossing write regardless of whether |
| + anything about the box's routes actually changed, churning the |
| + object's ETag on every box bounce. Seeding `*last` from the bucket's |
| + current `control/edges/<box>.json` at loop arm (read-old-then-loop, |
| + the same shape the loop already uses for its steady-state reads) |
| + closes both. Land this as the FIRST commit of the stage-5 arming |
| + branch, before anything else in the checklist below runs against a |
| + real second box. |
| +b. **Visibility for the two stuck-state paths before steering sends |
| + sustained traffic.** Both are silent today. `staleHopRefusal` |
| + (`internal/edge/homemap.go:551`) answers a sender stuck dialing a home |
| + box whose map has moved on with a correct 503, but nothing is LOGGED |
| + on either end, so a sender looping into it during a stuck map |
| + propagation produces zero signal. The backhaul 502 `ErrorHandler` is |
| + unthrottled the other direction — one log line per failed dial, not a |
| + throttled cadence (detail and cross-ref: item 14 below). Neither is a |
| + correctness bug, but both need eyes before Part 4 (steering) can |
| + legitimately point real, sustained traffic at this path — the |
| + dormant/low-volume window this plan ships into is where to add the |
| + logging, not after a stuck-move incident goes unseen in production. |
| + |
| ### Stage-5 arming checklist (before flagship AU, before any real second box) |
| 1. **Per-box edges files verified present for every armed box, BEFORE |
| @@ -642,6 +678,28 @@ plan's own spec Part 4 describes. |
| then eyeball each `addr` against `publiclyDialable`'s rule |
| (`net.ParseIP` non-nil, `IsGlobalUnicast()`, not private/loopback/ |
| link-local) — any hit is a route to look at BEFORE arming, not after. |
| + |
| + The same before-first-regions-apply window wants a second, adjacent |
| + scan that the SQL above cannot reach: bucket **records** |
| + (`apps/<sqid>/<app>/instances/<host>.json`, `Region` field — |
| + `internal/provision/provision.go:40`) whose region does not match the |
| + deployment's own single region. That mismatch is the only shape that |
| + would make Task 5's widened everywhere-class adoption gate fire on the |
| + very first roll, on a record no one has looked at since it was |
| + written, since adoption reads the ROW's region, not the box's |
| + assumption of it. Expect empty, same as the addr scan above, but |
| + unverified — enumerate with a bucket-wide key scan in the spirit of |
| + the bucket-writer audit's own enumeration method |
| + (`docs/superpowers/specs/2026-08-20-bucket-writer-audit.md`): |
| + |
| + ```bash |
| + aws --profile <profile> s3 sync s3://<control-bucket>/apps/ /tmp/apps-scan/ \ |
| + --exclude "*" --include "*/instances/*.json" |
| + grep -rl '"region"' /tmp/apps-scan | xargs -r grep -L '"region":"<deployment-region>"' |
| + ``` |
| + |
| + Any file the second `grep` prints carries a region other than the |
| + deployment's own — a record to look at BEFORE arming, not after. |
| 9. **Edges-file `role: "home"` over-claims for static hosts — wants a |
| `"replica"` role before Part 4 leans on the map.** Task 5's own |
| concern (report §4.1): once static routes are everywhere-class, EVERY |
diff --git a/docs/superpowers/specs/2026-08-20-multi-region-ingress-design.md b/docs/superpowers/specs/2026-08-20-multi-region-ingress-design.md| index fa911ee7c..19e15f475 100644 |
| --- a/docs/superpowers/specs/2026-08-20-multi-region-ingress-design.md |
| +++ b/docs/superpowers/specs/2026-08-20-multi-region-ingress-design.md |
| @@ -253,6 +253,12 @@ forwarded headers stripped, normal serving rules. Timestamp skew bound |
| ±30s (boxes run NTP). Key rotation: dual-accept window, old + new both |
| verify until every edge converges the new key. |
| +**Amendment (2026-08-22):** Task 3 shipped verbatim-restore, not append — |
| +on a verified hop the receiving edge replaces its own computed |
| +`X-Forwarded-For` with the sending edge's inbound chain unchanged (the |
| +last element stays the client); the sending edge's own address is never |
| +appended (`internal/edge/homemap.go`'s `setForwarded`). |
| + |
| **Hibernation** composes unchanged: the home edge owns wake (wake-once |
| singleflight, #199), and a backhaul hop is just another first client. |
diff --git a/internal/edge/homemap.go b/internal/edge/homemap.go| index 14ead04e0..dec82865a 100644 |
| --- a/internal/edge/homemap.go |
| +++ b/internal/edge/homemap.go |
| @@ -33,9 +33,15 @@ package edge |
| // captured X-Carlos-Hop-Auth is good for at most 2×hopSkew against ONE |
| // host, and what it buys a replayer is a request to a route the edge |
| // already serves to the whole internet — no new reachability, no |
| -// privilege. What it must NOT buy is the ability to state who the client |
| -// was, which is why verification gates exactly one thing: whether the |
| -// inbound X-Forwarded-For is believed. |
| +// privilege — PLUS, for that one host and that one window, XFF belief: |
| +// verification gates exactly one thing, whether the inbound |
| +// X-Forwarded-For is believed, and a valid MAC is what grants that |
| +// belief. What keeps the loose envelope affordable is not that XFF |
| +// belief is withheld — it is that capturing a valid Hop-Auth in the |
| +// first place requires breaking the hop's TLS or holding |
| +// control/edge-mesh-key outright, either of which already hands an |
| +// attacker strictly more than "who does this one host think the client |
| +// was for 2×hopSkew seconds." |
| // |
| // - XFF is trusted as an inbound chain to CARRY, never one to extend. On a |
| // verified hop the chain the sending edge built is passed through |
| @@ -538,8 +544,13 @@ func stripForwardedHeaders(r *http.Request) { |
| // staleHopRefusal answers a VERIFIED hop that named a host this box does not |
| // serve: the sending edge's home map is stale, mid-move. 503 + Retry-After: 2 |
| -// is the same "come back shortly" answer a dark fleet gets (fleetProxy), and |
| -// converge heals it within a publish cycle. |
| +// is the same "come back shortly" answer a dark fleet gets (fleetProxy). |
| +// Retry-After: 2 is a keep-trying hint, not a heal estimate: there is no 2s |
| +// serial-converge path here (edges never bump control/serial), so what |
| +// actually heals a stale map is the edges-file cycle — the sending box's own |
| +// 5min+jitter publish loop (internal/host's edgesFileLoop) followed by this |
| +// box's 30s mirror poll — which can leave a sender retrying into this refusal |
| +// for minutes during a home move, not seconds. |
| // |
| // The body says nothing. It is reachable only by a party holding the mesh key |
| // — an unauthenticated stranger asking for the same host gets the ordinary, |