| index c2e6bc1f6..e48fbaaf3 100644 |
| --- a/docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md |
| +++ b/docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md |
| @@ -1,6 +1,8 @@ |
| # Region HA, spec 2: two-box regions |
| |
| -Design, 2026-10-15, **rev 4**. DRAFT. Nothing here is built. |
| +Design, 2026-10-15, **rev 5**. DRAFT. Nothing here is built. |
| + |
| +**Rev 5** answers Astra's round 4 (`CODEX-FINDINGS-REGION-HA-SPEC2-4.md`, cited **A4-n**), two P2s, both accepted: stage 2 is gated on an armed host floor, and the hop credential follows the mesh key's existing current-and-next rotation. §17 maps each. |
| |
| **Rev 4** answers Astra's round 3 (`CODEX-FINDINGS-REGION-HA-SPEC2-3.md`, cited **A3-n**): 1 P1 and 2 P2, all accepted. They cover the forwarding failure contract, a staged rollout of the new forwarding protocol, and the refresh worker's liveness. §16 maps each. |
| |
| @@ -221,11 +223,16 @@ Anything else is joining mode. Joining resumes from whatever it finds: no invita |
| - **Between regions**, a box forwards to the home region's **ingress address**: the first entry of that region's ingress list. That's the region's EIP, which is on the NLB once the region is migrated, or still on the single box before then. |
| - The box that receives a cross-region forward forwards it **once more, within its region, to the private address** of the home box, if it isn't the home box itself. |
| - **Hop limit 2.** A request may be forwarded at most twice, and the second hop only to a private address in the receiving box's own region. Anything else is refused with 503, as one hop is today. |
| - - **The hop credential is versioned and binds the count** (A3-1). Today's credential is a MAC over the host and a timestamp (`internal/edge/mesh.go:241`). Version 2 is a MAC, under the mesh key, over the version, the host, the timestamp, the hop count, and the region the hop was addressed to. The receiver checks it with the current and the previous mesh key, so a rotation in flight still verifies. |
| + - **The hop credential is versioned and binds the count** (A3-1). Today's credential is a MAC over the host and a timestamp (`internal/edge/mesh.go:241`). Version 2 is a MAC, under the mesh key, over the version, the host, the timestamp, the hop count, and the region the hop was addressed to. |
| + - **Rotation follows today's protocol** (A4-2). Each edge loads the **current** key and, when published, the **`.next`** key, and signs with `.next` as soon as it has it (`internal/edge/mesh.go:167`, `homemap.go:812`). The receiver accepts a credential under either. A rotation publishes `.next`, waits until every box has refreshed (one map interval plus margin), promotes it to current, and then removes the old current. Spec 2 doesn't change this protocol, only what's signed. |
| - **A request carrying a hop credential that fails is never forwarded again** (A3-1). That covers an invalid signature, an unknown version, a malformed count or an expired timestamp. Today such a credential is stripped and the request is treated as a fresh client request (`internal/edge/homemap.go:930`), which would reset the budget and let two boxes with disagreeing maps pass a request back and forth. Instead, the receiver serves it only if the host is local to it, and otherwise answers 503. The failure is logged once per peer and key version. |
| - **The protocol is rolled out in three stages** (A3-2). Readers today discard a box whose edges file has no `eip` (`internal/edge/homemap.go:570`), so removing it at once would break forwarding to boxes still running old code. |
| 1. **Publish both.** Edges files add `private_ip`, `region` and `fwd_protocol: 2` while still publishing `eip`. Every edge accepts version 2 hop credentials, and the second private hop, alongside version 1. Forwarders still send version 1 to the published `eip`. This ships in a normal release and must reach every box in the deployment. |
| - 2. **Forward with version 2.** A deployment setting in the control bucket, `forwarding_protocol: 2`, switches forwarders to §6's rules: region ingress between regions, private within a region, version 2 credentials. The console sets it only when every enrolled box's edges file reports `fwd_protocol: 2`, and refuses otherwise. |
| + 2. **Forward with version 2.** A deployment setting in the control bucket, `forwarding_protocol: 2`, switches forwarders to §6's rules: region ingress between regions, private within a region, version 2 credentials. |
| + - **It's gated on an armed host floor, not on what boxes advertise** (A4-1). A box that advertises version 2 could still return to an older binary through rollback or boot recovery, which restores `carlos.prev` (`internal/host/systemapply.go:1604`). |
| + - So, first, the host floor (`internal/hostfloor`) is raised to the release that introduced version 2. |
| + - Every enrolled box must then report its guard armed for that floor (`FloorGuardReady` with the floor's digest in its census report, `internal/hostactors/census.go:68`). From then on its guard refuses to activate any binary below the floor, `carlos.prev` included. |
| + - The console sets `forwarding_protocol: 2` only when every enrolled box reports that, and refuses otherwise. A box that joins later already satisfies it: it installs the channel release, which is at or above the floor (§5). |
| 3. **Stop publishing `eip`.** A later release drops it, once the setting has been at 2 for one full release cycle. Readers by then ignore its absence. |
| - Group boxes join only after stage 2. §3's prerequisites gain stage 2 as a fourth condition. |
| - **What this removes, once stage 3 is done:** boxes no longer publish a public dial address at all, so there's nothing to re-probe, filter or override. The cut-over (§9) needs no second address for the old box, and rollback releases nothing forwarding depends on. |
| @@ -383,7 +390,8 @@ Each is closed by a code change, or by showing it's safe under claim mode, befor |
| - **Forwarding.** |
| - Private-address forwarding within a region, and cross-region forwarding to the ingress address followed by one private hop. |
| - Refusals: a third hop; a second hop to a non-private or out-of-region target; a hop credential with a bad signature, unknown version, malformed count or expired timestamp, which is served only if local and otherwise refused, never forwarded. |
| - - Key rotation: a credential made with the previous key verifies. |
| + - Key rotation, the existing protocol: `.next` published, boxes refreshing at different times, promotion, then removal of the old key. Credentials verify at every point, whichever key the sender signed with. |
| + - The stage 2 gate: refused while any box lacks an armed floor covering version 2, and a recovery to an older `carlos.prev` refused by the guard once the floor is armed, including when the two race. |
| - The three rollout stages: a version 1 box and a version 2 box forward to each other at stage 1, and the console refuses stage 2 while any box reports version 1. |
| - Platform routes forwarded from a group box to the platform box. |
| - **Placement.** Unit tests for each eligibility rule, the spread preference, the headroom ordering, the stale-headroom cut-off, the platform role, and unplaced with its retry triggers. |
| @@ -452,3 +460,10 @@ All three were answered in Astra's round 1 and folded in: |
| | A3-2 P2: removing published addresses breaks old readers | Accepted: three stages (publish both, switch by deployment setting once every box reports version 2, then drop `eip`), with stage 2 a prerequisite (§3, §6) | |
| | A3-3 P2: a stalled refresh worker isn't detected | Accepted: the worker must complete an attempt, successful or not, within 5 minutes (§6) | |
| | Stale summaries | §10's enrolment-failure row states the first-build exception, and §12 says what is actually deferred about zone B | |
| + |
| +## 17. Astra round 4 dispositions (rev 5) |
| + |
| +| Finding | Disposition | |
| +|---|---| |
| +| A4-1 P2: stage 2 needs a release floor, not an advertisement | Accepted: the host floor is raised to the version 2 release, and stage 2 needs every enrolled box's guard armed for it, which also blocks rollback and recovery below it (§6) | |
| +| A4-2 P2: "current and previous" conflicts with mesh-key rotation | Accepted: the existing current-and-`.next` protocol is kept and named, with its full rotation sequence tested (§6) | |
amadan