| index e48fbaaf3..6ebbc9e56 100644 |
| --- a/docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md |
| +++ b/docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md |
| @@ -1,6 +1,8 @@ |
| # Region HA, spec 2: two-box regions |
| |
| -Design, 2026-10-15, **rev 5**. DRAFT. Nothing here is built. |
| +Design, 2026-10-15, **rev 6**. DRAFT. Nothing here is built. |
| + |
| +**Rev 6** simplifies forwarding (§6). Rounds 3 to 5 each found a new edge in rev 3's two-hop protocol: the hop budget, the staged rollout, the host floor, then key rotation. Rev 6 keeps today's protocol instead: one hop, today's credential, today's mesh-key rotation. Only the address the forwarder dials changes, and instances leave the old box before the region's EIP moves. A3-1, A3-2, A4-1, A4-2, A5-1 and A5-2 are moot. §18 maps them. |
| |
| **Rev 5** answers Astra's round 4 (`CODEX-FINDINGS-REGION-HA-SPEC2-4.md`, cited **A4-n**), two P2s, both accepted: stage 2 is gated on an armed host floor, and the hop credential follows the mesh key's existing current-and-next rotation. §17 maps each. |
| |
| @@ -71,9 +73,8 @@ instance's local database copy only while a region has one box |
| 1. **Claim mode is active in the deployment.** That means `CARLOS_HOST_CLAIM`, `CARLOS_HOST_CLAIM_ORIGINS` and #250's backfill, plus the open claim asks on #236. |
| 2. **Region HA is on in the region** (`CARLOS_REGION_HA`), and relocation has been rehearsed there. |
| 3. **Every single-box assumption in §8 is closed.** |
| -4. **The forwarding protocol is at stage 2** (§6): every box accepts and sends version 2, and forwards by region ingress and private address. |
| |
| -No region gets a second box before all four hold. |
| +No region gets a second box before all three hold. |
| |
| ## 4. Infrastructure: `infra/modules/region-host` |
| |
| @@ -88,6 +89,7 @@ that don't want HA. |
| - Same AMI lookup, instance type, root volume (`delete_on_termination = true`), security group and instance profile as the fixed box. |
| - The start-up script is §5's. |
| - IMDSv2 required, with a hop limit of 1. |
| + - An auto-assigned public IPv4 address on the box's own interface, so each group box has a public address of its own for cross-region forwarding (§6). It's never the region's EIP. |
| - **An Auto Scaling group.** |
| - `min = max = desired = var.box_count`, which defaults to 2. |
| - Spread over two availability zones. |
| @@ -216,28 +218,18 @@ Anything else is joining mode. Joining resumes from whatever it finds: no invita |
| - Outbound mail from the platform box leaves from the new address too, so the SPF records and reverse DNS for the platform box's sending address move with it. |
| - Its web traffic arrives through the NLB like every other box's. |
| |
| -**Forwarding between boxes** (A2-7, A2-9: rev 3 replaces rev 2's own-address rule). |
| -- Spec 1's forwarding (the edges file's home map, `internal/edge/homemap.go`) is on main and active today. It dials the home box's published public EIP on 443 (`edge.go:2779`), and a forwarded request is never forwarded again (`edge.go:1505`). |
| -- Group boxes have no fixed public address, and an NLB address reaches an arbitrary box. So forwarding changes: |
| - - **Within a region**, a box forwards to the home box's **private IPv4**. Each edges file publishes it, from IMDS. |
| - - **Between regions**, a box forwards to the home region's **ingress address**: the first entry of that region's ingress list. That's the region's EIP, which is on the NLB once the region is migrated, or still on the single box before then. |
| - - The box that receives a cross-region forward forwards it **once more, within its region, to the private address** of the home box, if it isn't the home box itself. |
| - - **Hop limit 2.** A request may be forwarded at most twice, and the second hop only to a private address in the receiving box's own region. Anything else is refused with 503, as one hop is today. |
| - - **The hop credential is versioned and binds the count** (A3-1). Today's credential is a MAC over the host and a timestamp (`internal/edge/mesh.go:241`). Version 2 is a MAC, under the mesh key, over the version, the host, the timestamp, the hop count, and the region the hop was addressed to. |
| - - **Rotation follows today's protocol** (A4-2). Each edge loads the **current** key and, when published, the **`.next`** key, and signs with `.next` as soon as it has it (`internal/edge/mesh.go:167`, `homemap.go:812`). The receiver accepts a credential under either. A rotation publishes `.next`, waits until every box has refreshed (one map interval plus margin), promotes it to current, and then removes the old current. Spec 2 doesn't change this protocol, only what's signed. |
| - - **A request carrying a hop credential that fails is never forwarded again** (A3-1). That covers an invalid signature, an unknown version, a malformed count or an expired timestamp. Today such a credential is stripped and the request is treated as a fresh client request (`internal/edge/homemap.go:930`), which would reset the budget and let two boxes with disagreeing maps pass a request back and forth. Instead, the receiver serves it only if the host is local to it, and otherwise answers 503. The failure is logged once per peer and key version. |
| -- **The protocol is rolled out in three stages** (A3-2). Readers today discard a box whose edges file has no `eip` (`internal/edge/homemap.go:570`), so removing it at once would break forwarding to boxes still running old code. |
| - 1. **Publish both.** Edges files add `private_ip`, `region` and `fwd_protocol: 2` while still publishing `eip`. Every edge accepts version 2 hop credentials, and the second private hop, alongside version 1. Forwarders still send version 1 to the published `eip`. This ships in a normal release and must reach every box in the deployment. |
| - 2. **Forward with version 2.** A deployment setting in the control bucket, `forwarding_protocol: 2`, switches forwarders to §6's rules: region ingress between regions, private within a region, version 2 credentials. |
| - - **It's gated on an armed host floor, not on what boxes advertise** (A4-1). A box that advertises version 2 could still return to an older binary through rollback or boot recovery, which restores `carlos.prev` (`internal/host/systemapply.go:1604`). |
| - - So, first, the host floor (`internal/hostfloor`) is raised to the release that introduced version 2. |
| - - Every enrolled box must then report its guard armed for that floor (`FloorGuardReady` with the floor's digest in its census report, `internal/hostactors/census.go:68`). From then on its guard refuses to activate any binary below the floor, `carlos.prev` included. |
| - - The console sets `forwarding_protocol: 2` only when every enrolled box reports that, and refuses otherwise. A box that joins later already satisfies it: it installs the channel release, which is at or above the floor (§5). |
| - 3. **Stop publishing `eip`.** A later release drops it, once the setting has been at 2 for one full release cycle. Readers by then ignore its absence. |
| - - Group boxes join only after stage 2. §3's prerequisites gain stage 2 as a fourth condition. |
| -- **What this removes, once stage 3 is done:** boxes no longer publish a public dial address at all, so there's nothing to re-probe, filter or override. The cut-over (§9) needs no second address for the old box, and rollback releases nothing forwarding depends on. |
| +**Forwarding between boxes** (rev 6). |
| +- Spec 1's forwarding is on main and active today. The edges file's home map (`internal/edge/homemap.go`) tells each edge which box runs each host. A request for a host on another box is forwarded **once**, signed with the mesh key, to that box (`edge.go:2779`), and a forwarded request is never forwarded again (`edge.go:1505`). Spec 2 keeps all of that, including the credential and the mesh-key rotation, unchanged. |
| +- **The only change is which address the forwarder dials:** |
| + - **Same region:** the home box's **private IPv4**. Each edges file adds a `private_ip` field, read from IMDS. Readers that don't know the field ignore it, so it ships in an ordinary release with no ordering between boxes. A reader that finds no `private_ip` dials the public address, as today. |
| + - **Another region:** the home box's own **public address**, as today. For a group box, that's its auto-assigned public IP (§4). |
| +- **A box publishes only an address IMDS reports as its own** (A1-12). The publisher re-reads IMDS `public-ipv4` before every publication. A blank read publishes no public address, instead of keeping the previous one (`internal/host/edgesfile.go:589`). The region's EIP is published only by a box that actually holds it, which before the cut-over is the old fixed box. Once the EIP is on the NLB, IMDS no longer reports it to any box. |
| +- **Why this is enough:** every request reaches the home box in one hop: |
| + - from the NLB's box to the home box in the same region, privately; |
| + - from any box in another region to the home box's own public address. |
| + - No box ever needs to forward a forwarded request. |
| +- **The one transition** is the old box's address at the cut-over. §9 moves every instance off the old box first, so when the region's EIP leaves it, nothing is reached through that address. The exception is IE's platform routes, covered in §9 step 3. |
| - The security group gains 443 from the region's VPC CIDR, which is already exported as `vpc_cidr`. |
| -- Readers that still use the published public address are listed in §8 and moved to these rules. |
| |
| **Platform routes.** |
| - The console and other unit- or socket-backed routes run only on the platform box (S1 §12: never relocated). |
| @@ -309,7 +301,7 @@ Each is closed by a code change, or by showing it's safe under claim mode, befor |
| | `internal/activator/activator.go:2264-2272` | The local database copy is a warm cache only while a region has one box | H13: under the claim every admission is cold, and the warm path is never used. Verify that the claim-mode activator never reads this cache, and add a test | |
| | `internal/console/apiinstances.go:523-527` | Hosts are unique to one box | Holds under the claim (one executor). Add a test | |
| | `internal/console/view.go:177-237`, `server.go:1120-1147` | EIP list as A-record targets | §6: the region ingress list | |
| -| `internal/edge/homemap.go:549-613`, `edge.go:2779`, `internal/host/agent.go:2951-2963`, `edgesfile.go:589, 806, 823` | Forwarding dials each box's published public EIP, one hop | §6: private addresses in a region, the region ingress address between regions, hop limit 2 | |
| +| `internal/edge/homemap.go:549-613`, `edge.go:2779`, `internal/host/agent.go:2951-2963`, `edgesfile.go:589, 806, 823` | Forwarding dials each box's published public EIP, one hop | §6: dial the private address within a region and the box's own public address between regions; publish only an address IMDS reports | |
| | `cmd/console/main.go:1390-1400` | `consoleBoxLabel`: exactly one box | Holds: the console runs on the platform box only. Document it | |
| | Flagship tofu `outputs.tf:39-68`, `dns*.tf`, `keymail.tf`, `mail.tf`, `scanner.tf` | One instance and EIP per region | Rewrite for the group, the NLB EIP and the platform box | |
| | `internal/console/apirestartscope.go:45` | Scoped restart refuses hosts the console's box doesn't know (A1-9) | Resolve the host's executor from the claim and act through it | |
| @@ -330,30 +322,28 @@ Each is closed by a code change, or by showing it's safe under claim mode, befor |
| - Set the new EIP's reverse DNS once its forward record exists, and add it to SPF beside the old address. |
| 1. **Build.** |
| - Write the SSM parameters. |
| - - Apply the module in group mode. The NLB is created with **only zone B's mapping**, carrying its new EIP (A1-§13). |
| - - The two group boxes boot, join, enrol and reach hosting mode. The NLB is rehearsed through zone B's address. That checks zone B's targets only: zone A's are first exercised after step 2. |
| - - The old fixed box keeps the region's EIP and all public traffic. It's also registered as an NLB target. |
| -2. **Cut over the EIP.** |
| - 1. **Check forwarding is ready.** Every box in the deployment forwards with §6's rules: cross-region forwards go to the region's ingress address, and the receiving box forwards privately. No box holds a published-public-address dependency on the old box, so moving the EIP doesn't strand cross-region forwarding (A2-7). |
| - 2. **Disassociate the region's EIP** from the old box. In IE, immediately associate the new EIP with it, switch the relay and MX records to the new address, and restart the relay with that address advertised. |
| - 3. **Add zone A's mapping** to the NLB with the region's EIP (`SetSubnets`, which accepts an EIP when adding a mapping). |
| - 4. **The cost:** the region's public address is down from step ii until the zone A node is active. Expect seconds to a few minutes, scheduled at a quiet hour and announced. The window is measured on dev first. |
| - 5. The old box keeps serving its instances throughout, reached through the NLB and through forwarding. |
| - 6. In IE, before the window closes: check that mail arrives on the new address, and that a TURN allocation advertises it and carries media. |
| -3. **Migrate.** Relocate every instance on the old box to §7's choice, one at a time, with spec 1's relocation. |
| + - Apply the module in group mode. The NLB is created with **only zone B's mapping**, carrying its new EIP. |
| + - The two group boxes boot, join, enrol and reach hosting mode. The NLB is rehearsed through zone B's address. That checks zone B's targets only: zone A's are first exercised after step 3. |
| + - The old fixed box keeps the region's EIP and all public traffic. It forwards to the group boxes over their private addresses as instances move to them. |
| +2. **Migrate.** Relocate every instance on the old box to §7's choice, one at a time, with spec 1's relocation. Each relocated instance is reached the usual ways: through the old box, which still holds the region's EIP, privately, and from other regions at the group box's own public address. When this step ends, the old box hosts no customer instance. |
| +3. **Cut over the EIP.** The old box is empty by now, so moving its address strands nothing. |
| + 1. **Disassociate the region's EIP** from the old box. In IE, immediately associate the new EIP with it, switch the relay and MX records to the new address, and restart the relay with that address advertised. |
| + 2. **Add zone A's mapping** to the NLB with the region's EIP (`SetSubnets`, which accepts an EIP when adding a mapping). |
| + 3. **The cost:** the region's public address is down from step i until the zone A node is active. Expect seconds to a few minutes, scheduled at a quiet hour and announced. The window is measured on dev first. |
| + 4. **In IE, platform routes:** the platform box republishes at once with its new address (a forced edges republication). Until other regions' maps have it, which takes up to a minute, a cross-region forward to the console lands on the NLB and is refused. That's inside the window. Requests for platform hostnames from browsers go to the region's EIP and reach the platform box privately through the NLB, so they're unaffected once zone A is active. |
| + 5. In IE, before the window closes: check that mail arrives on the new address, and that a TURN allocation advertises it and carries media. |
| 4. **Accept.** |
| - - The old box stays running, deregistered from the NLB and holding no instances, for **seven days**. That's the rollback window. |
| - - After acceptance, UK, US and AU terminate it. |
| - - IE keeps it as the platform box: it's enrolled with the `platform` role and keeps its new EIP from step 0. |
| + - **UK, US and AU:** the old box stays running, deregistered from the NLB and empty, for **seven days**. That's the rollback window. Then it's terminated. |
| + - **IE:** the old box is the platform box. It's enrolled with the `platform` role, keeps its new EIP from step 0, and stays registered as an NLB target. |
| |
| **Rollback** (A1-13). The region's EIP is a standalone `aws_eip` with `prevent_destroy`, so the allocation is never lost. |
| -- **Before step 2:** destroy the group and NLB. The region is back to today's state. |
| -- **From step 2 until acceptance:** |
| +- **Before step 3:** relocate instances back to the old box, then destroy the group and NLB. The region is back to today's state. |
| +- **From step 3 until acceptance:** |
| 1. Remove zone A's mapping from the NLB, or delete the NLB. An address on a requester-managed interface can't be disassociated directly. |
| 2. Wait until the EIP shows as unassociated. |
| - 3. Associate it with the old box. Forwarding needs no change: cross-region forwards to the region's ingress address now land on the old box, which forwards privately to the group boxes if needed. |
| - - **In IE** (A2-9), associating the region's EIP with the old box takes its new EIP off it, because one private address holds one public address. So the relay and MX records are switched back to the region's address, and the relay is restarted advertising it. The new EIP's allocation is kept for the next attempt. |
| - 4. Relocate any migrated instances back. |
| + 3. Associate it with the old box, which republishes it. |
| + 4. **In IE** (A2-9), associating the region's EIP with the old box takes its new EIP off it, because one private address holds one public address. So the relay and MX records are switched back to the region's address, and the relay is restarted advertising it. The new EIP's allocation is kept for the next attempt. |
| + 5. Relocate instances back to the old box if the group is the problem. Otherwise the group boxes keep them, reached through the old box privately. |
| - **After acceptance:** there's no rollback to the old box, because it's gone. Recovery is forward, within the group. |
| |
| **Cost.** |
| @@ -388,15 +378,14 @@ Each is closed by a code change, or by showing it's safe under claim mode, befor |
| - **Unplaced.** Pending create, retry on enrolment and on headroom, cancellation while pending, the hostname reservation shared with the immediate door, and exactly one claim create per pending row. That includes a crash after the claim write and before the row delete, reconciled by operation ID. |
| - **Health answer.** Edge tests for `/.carlos/lb-health`: any host, exact path, GET and HEAD only, and 503 for each failing condition (joining mode, forwarding never armed, no certificate cache, a stalled heartbeat). A test that an unreachable bucket after arming leaves the answer at 200, because failed refresh attempts still count as progress, and that a refresh worker blocked inside one attempt turns it 503. |
| - **Forwarding.** |
| - - Private-address forwarding within a region, and cross-region forwarding to the ingress address followed by one private hop. |
| - - Refusals: a third hop; a second hop to a non-private or out-of-region target; a hop credential with a bad signature, unknown version, malformed count or expired timestamp, which is served only if local and otherwise refused, never forwarded. |
| - - Key rotation, the existing protocol: `.next` published, boxes refreshing at different times, promotion, then removal of the old key. Credentials verify at every point, whichever key the sender signed with. |
| - - The stage 2 gate: refused while any box lacks an armed floor covering version 2, and a recovery to an older `carlos.prev` refused by the guard once the floor is armed, including when the two race. |
| - - The three rollout stages: a version 1 box and a version 2 box forward to each other at stage 1, and the console refuses stage 2 while any box reports version 1. |
| + - Private-address forwarding within a region, and own-public-address forwarding between regions. |
| + - A reader without `private_ip` dials the public address. |
| + - The publisher publishes only what IMDS reports, and publishes no public address on a blank read. |
| + - A forwarded request is still never forwarded again. |
| - Platform routes forwarded from a group box to the platform box. |
| - **Placement.** Unit tests for each eligibility rule, the spread preference, the headroom ordering, the stale-headroom cut-off, the platform role, and unplaced with its retry triggers. |
| - **Rehearsal on dev:** |
| - - the full §9 sequence, with the EIP cut-over window measured, the zone-B rehearsal, and the rollback sequence run once; |
| + - the full §9 sequence, with migration before the cut-over, the cut-over window measured, the zone-B rehearsal, and the rollback sequence run once; |
| - one group box terminated by hand, with replacement and re-enrolment observed; |
| - a box made unhealthy, observed being terminated. |
| - **Failover drills** belong to spec 3. |
| @@ -467,3 +456,13 @@ All three were answered in Astra's round 1 and folded in: |
| |---|---| |
| | A4-1 P2: stage 2 needs a release floor, not an advertisement | Accepted: the host floor is raised to the version 2 release, and stage 2 needs every enrolled box's guard armed for it, which also blocks rollback and recovery below it (§6) | |
| | A4-2 P2: "current and previous" conflicts with mesh-key rotation | Accepted: the existing current-and-`.next` protocol is kept and named, with its full rotation sequence tested (§6) | |
| + |
| +## 18. Astra round 5 dispositions, and the rev 6 simplification |
| + |
| +| Finding | Disposition | |
| +|---|---| |
| +| A5-1 P2: the host floor can't be raised once published | Moot: rev 6 has no new forwarding protocol, so there's nothing to gate on a floor | |
| +| A5-2 P2: early `.next` signing fails during staggered refresh | Moot: rev 6 needs no second hop. Today's single hop goes straight to the home box, and today's rotation behaviour is unchanged | |
| +| A3-1, A3-2, A4-1, A4-2 | Moot for the same reason. Rev 3's version 2 credential, three-stage rollout and floor gate are removed | |
| +| A2-7, A2-9 | Still met: a box publishes only an address IMDS reports, and instances leave the old box before its address moves (§6, §9) | |
| +| A3-3 refresh-worker liveness | Kept (§6) | |
amadan