| index 047277873..c2e6bc1f6 100644 |
| --- a/docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md |
| +++ b/docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md |
| @@ -1,6 +1,8 @@ |
| # Region HA, spec 2: two-box regions |
| |
| -Design, 2026-10-15, **rev 3**. DRAFT. Nothing here is built. |
| +Design, 2026-10-15, **rev 4**. DRAFT. Nothing here is built. |
| + |
| +**Rev 4** answers Astra's round 3 (`CODEX-FINDINGS-REGION-HA-SPEC2-3.md`, cited **A3-n**): 1 P1 and 2 P2, all accepted. They cover the forwarding failure contract, a staged rollout of the new forwarding protocol, and the refresh worker's liveness. §16 maps each. |
| |
| **Rev 3** answers Astra's round 2 (`CODEX-FINDINGS-REGION-HA-SPEC2-2.md`, |
| cited **A2-n**): 3 P1 and 6 P2, all accepted, plus the partial |
| @@ -67,8 +69,9 @@ instance's local database copy only while a region has one box |
| 1. **Claim mode is active in the deployment.** That means `CARLOS_HOST_CLAIM`, `CARLOS_HOST_CLAIM_ORIGINS` and #250's backfill, plus the open claim asks on #236. |
| 2. **Region HA is on in the region** (`CARLOS_REGION_HA`), and relocation has been rehearsed there. |
| 3. **Every single-box assumption in §8 is closed.** |
| +4. **The forwarding protocol is at stage 2** (§6): every box accepts and sends version 2, and forwards by region ingress and private address. |
| |
| -No region gets a second box before all three hold. |
| +No region gets a second box before all four hold. |
| |
| ## 4. Infrastructure: `infra/modules/region-host` |
| |
| @@ -192,6 +195,7 @@ Anything else is joining mode. Joining resumes from whatever it finds: no invita |
| - **Forwarding is armed:** the forwarding map and the mesh key have each loaded successfully **at least once** since the edge started (A1-10). This is latched, not a freshness check. After that, a failed refresh keeps the cached map and key, as today (`internal/edge/homemap.go:235`), and doesn't affect health. |
| - The TLS certificate cache has loaded, with the default certificate available. |
| - The edge's request loop and the activator's hosting loop have each reported a heartbeat within the last 60 seconds. These are local liveness: a hung process stops its own heartbeat. |
| + - The forwarding map and key refresh worker (`internal/edge/homemap.go:838`) has **completed an attempt**, whether it succeeded or failed, within the last 5 minutes (A3-3). Completion counts as progress, so a bucket outage, where every attempt fails quickly, keeps it satisfied. A worker stuck inside one attempt doesn't. The 5 minutes covers the refresh interval plus the existing two-minute request timeout. |
| - **Why freshness was dropped.** Rev 2 required refreshes within 2 minutes. That made a bucket outage over two minutes mark every box unhealthy, and the group would terminate working boxes whose instances spec 2 can't move yet. A shared-dependency outage is an alert, from the existing bucket and housekeeper alerting, never a termination. |
| - Each condition is a timestamp or flag that its own loop updates. The answer only reads them, and does no disk, bucket or network work per request. |
| |
| @@ -216,8 +220,15 @@ Anything else is joining mode. Joining resumes from whatever it finds: no invita |
| - **Within a region**, a box forwards to the home box's **private IPv4**. Each edges file publishes it, from IMDS. |
| - **Between regions**, a box forwards to the home region's **ingress address**: the first entry of that region's ingress list. That's the region's EIP, which is on the NLB once the region is migrated, or still on the single box before then. |
| - The box that receives a cross-region forward forwards it **once more, within its region, to the private address** of the home box, if it isn't the home box itself. |
| - - **Hop limit 2.** The signed forwarding header carries a hop count. A request may be forwarded at most twice, and the second hop only to a private address in the receiving box's own region. Anything else is refused with 503, as one hop is today. |
| -- **What this removes:** boxes no longer publish a public dial address at all, so there's nothing to re-probe, filter or override. The cut-over (§9) needs no second address for the old box, and rollback releases nothing forwarding depends on. |
| + - **Hop limit 2.** A request may be forwarded at most twice, and the second hop only to a private address in the receiving box's own region. Anything else is refused with 503, as one hop is today. |
| + - **The hop credential is versioned and binds the count** (A3-1). Today's credential is a MAC over the host and a timestamp (`internal/edge/mesh.go:241`). Version 2 is a MAC, under the mesh key, over the version, the host, the timestamp, the hop count, and the region the hop was addressed to. The receiver checks it with the current and the previous mesh key, so a rotation in flight still verifies. |
| + - **A request carrying a hop credential that fails is never forwarded again** (A3-1). That covers an invalid signature, an unknown version, a malformed count or an expired timestamp. Today such a credential is stripped and the request is treated as a fresh client request (`internal/edge/homemap.go:930`), which would reset the budget and let two boxes with disagreeing maps pass a request back and forth. Instead, the receiver serves it only if the host is local to it, and otherwise answers 503. The failure is logged once per peer and key version. |
| +- **The protocol is rolled out in three stages** (A3-2). Readers today discard a box whose edges file has no `eip` (`internal/edge/homemap.go:570`), so removing it at once would break forwarding to boxes still running old code. |
| + 1. **Publish both.** Edges files add `private_ip`, `region` and `fwd_protocol: 2` while still publishing `eip`. Every edge accepts version 2 hop credentials, and the second private hop, alongside version 1. Forwarders still send version 1 to the published `eip`. This ships in a normal release and must reach every box in the deployment. |
| + 2. **Forward with version 2.** A deployment setting in the control bucket, `forwarding_protocol: 2`, switches forwarders to §6's rules: region ingress between regions, private within a region, version 2 credentials. The console sets it only when every enrolled box's edges file reports `fwd_protocol: 2`, and refuses otherwise. |
| + 3. **Stop publishing `eip`.** A later release drops it, once the setting has been at 2 for one full release cycle. Readers by then ignore its absence. |
| + - Group boxes join only after stage 2. §3's prerequisites gain stage 2 as a fourth condition. |
| +- **What this removes, once stage 3 is done:** boxes no longer publish a public dial address at all, so there's nothing to re-probe, filter or override. The cut-over (§9) needs no second address for the old box, and rollback releases nothing forwarding depends on. |
| - The security group gains 443 from the region's VPC CIDR, which is already exported as `vpc_cidr`. |
| - Readers that still use the published public address are listed in §8 and moved to these rules. |
| |
| @@ -354,7 +365,7 @@ Each is closed by a code change, or by showing it's safe under claim mode, befor |
| | Both group boxes in a region fail | The region's instances are down. The group replaces both, and the first replacement is confirmed from another region (H17). | |
| | The NLB's availability zone fails | The region's ingress is down until the zone recovers (§6). | |
| | A replacement can't boot | It's never healthy, the group keeps replacing it, and the eligible-capacity alarm fires at 25 minutes. | |
| -| A box boots but can't enrol | It stays in joining mode, its health answer stays 503, and it takes no traffic. The group replaces it at the end of the grace period, and the eligible-capacity alarm fires at 25 minutes. | |
| +| A box boots but can't enrol | It stays in joining mode, its health answer stays 503, and it takes no traffic. The group replaces it at the end of the grace period, except during a region's first build (§5). The eligible-capacity alarm fires at 25 minutes either way. | |
| | One zone is unavailable | The NLB serves through the other zone's node only for clients that use zone B's address, so the region's published address (zone A) is down until the zone recovers. The group doesn't move boxes between zones (`AZRebalance` is suspended). | |
| | A forwarding mesh key or map refresh stalls on a box | Its health answer turns 503, and the NLB stops sending it traffic. If the stall lasts past the health-check threshold, the group replaces it. | |
| | No box has headroom | New instances are unplaced, and an alert fires (§7). | |
| @@ -368,8 +379,13 @@ Each is closed by a code change, or by showing it's safe under claim mode, befor |
| - **Install.** A fresh box installs the channel release into both protected slots, never the bootstrap binary, refuses a release below the host floor, and then passes `guard-arm`. |
| - **Mode selection.** A restart at each joining state (no invitation, invited, guard installed, guard-ready reported) resumes joining. Only an active roster entry plus an armed guard starts hosting. |
| - **Unplaced.** Pending create, retry on enrolment and on headroom, cancellation while pending, the hostname reservation shared with the immediate door, and exactly one claim create per pending row. That includes a crash after the claim write and before the row delete, reconciled by operation ID. |
| -- **Health answer.** Edge tests for `/.carlos/lb-health`: any host, exact path, GET and HEAD only, and 503 for each failing condition (joining mode, forwarding never armed, no certificate cache, a stalled heartbeat). A test that an unreachable bucket after arming leaves the answer at 200. |
| -- **Forwarding.** Private-address forwarding within a region. Cross-region forwarding to the ingress address followed by one private hop. A third hop, or a second hop to a non-private or out-of-region target, is refused. Platform routes forwarded from a group box to the platform box. |
| +- **Health answer.** Edge tests for `/.carlos/lb-health`: any host, exact path, GET and HEAD only, and 503 for each failing condition (joining mode, forwarding never armed, no certificate cache, a stalled heartbeat). A test that an unreachable bucket after arming leaves the answer at 200, because failed refresh attempts still count as progress, and that a refresh worker blocked inside one attempt turns it 503. |
| +- **Forwarding.** |
| + - Private-address forwarding within a region, and cross-region forwarding to the ingress address followed by one private hop. |
| + - Refusals: a third hop; a second hop to a non-private or out-of-region target; a hop credential with a bad signature, unknown version, malformed count or expired timestamp, which is served only if local and otherwise refused, never forwarded. |
| + - Key rotation: a credential made with the previous key verifies. |
| + - The three rollout stages: a version 1 box and a version 2 box forward to each other at stage 1, and the console refuses stage 2 while any box reports version 1. |
| + - Platform routes forwarded from a group box to the platform box. |
| - **Placement.** Unit tests for each eligibility rule, the spread preference, the headroom ordering, the stale-headroom cut-off, the platform role, and unplaced with its retry triggers. |
| - **Rehearsal on dev:** |
| - the full §9 sequence, with the EIP cut-over window measured, the zone-B rehearsal, and the rollback sequence run once; |
| @@ -382,7 +398,7 @@ Each is closed by a code change, or by showing it's safe under claim mode, befor |
| - **Spec 3:** relocating a terminated box's instances (the placer loop), waking always-on and scheduled instances, the failover gap target. |
| - **Spec 4:** |
| - automatic scale-out and scale-in, and rebalancing; |
| - - a second availability zone for ingress (a second EIP); |
| + - offering zone B's ingress address to customers, so the region's published addresses span both zones (§6). The NLB already has a node in each zone; |
| - latency steering; |
| - H5's per-region minimum as anything other than a fixed group size; |
| - the Auto Scaling activity-history evidence source (S1 A2-4). |
| @@ -427,3 +443,12 @@ All three were answered in Astra's round 1 and folded in: |
| | A2-8 P2: no permission for the capacity metric | Accepted: a namespace-scoped grant on the platform box, missing data as breaching (§4) | |
| | A2-9 P2: rollback removes needed addresses | Accepted: forwarding doesn't depend on per-box addresses, and IE's step 0 isn't rolled back (§9) | |
| | Round 1 partials | A1-1: §6 and §12 wording updated. A1-11: the first-build exception stated (§5). A1-12 and A1-13: superseded by the forwarding change | |
| + |
| +## 16. Astra round 3 dispositions (rev 4) |
| + |
| +| Finding | Disposition | |
| +|---|---| |
| +| A3-1 P1: a failed hop credential resets the forwarding budget | Accepted: a version 2 credential binding the count and region, the current and previous key accepted, and a failed credential never forwarded again (§6) | |
| +| A3-2 P2: removing published addresses breaks old readers | Accepted: three stages (publish both, switch by deployment setting once every box reports version 2, then drop `eip`), with stage 2 a prerequisite (§3, §6) | |
| +| A3-3 P2: a stalled refresh worker isn't detected | Accepted: the worker must complete an attempt, successful or not, within 5 minutes (§6) | |
| +| Stale summaries | §10's enrolment-failure row states the first-build exception, and §12 says what is actually deferred about zone B | |
amadan