| index 6ebbc9e56..831a6df04 100644 |
| --- a/docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md |
| +++ b/docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md |
| @@ -1,6 +1,8 @@ |
| # Region HA, spec 2: two-box regions |
| |
| -Design, 2026-10-15, **rev 6**. DRAFT. Nothing here is built. |
| +Design, 2026-10-15, **rev 7**. DRAFT. Nothing here is built. |
| + |
| +**Rev 7** answers Astra's round 6 (`CODEX-FINDINGS-REGION-HA-SPEC2-6.md`, cited **A6-n**), two P2s in the rollout, both accepted. The old box is set `draining` before migration and checked empty before the cut-over and before termination. IE's old box takes the `platform` role only at acceptance, so rollback can move instances back to it. §19 maps them. |
| |
| **Rev 6** simplifies forwarding (§6). Rounds 3 to 5 each found a new edge in rev 3's two-hop protocol: the hop budget, the staged rollout, the host floor, then key rotation. Rev 6 keeps today's protocol instead: one hop, today's credential, today's mesh-key rotation. Only the address the forwarder dials changes, and instances leave the old box before the region's EIP moves. A3-1, A3-2, A4-1, A4-2, A5-1 and A5-2 are moot. §18 maps them. |
| |
| @@ -187,7 +189,7 @@ Anything else is joining mode. Joining resumes from whatever it finds: no invita |
| - **Any step:** the health answer never reaches 200. The group replaces the box when the grace period ends, **except** while the group's target group has never had a healthy target: AWS doesn't replace ELB-unhealthy instances then (A1-11). That case happens only when a region is first built. |
| - **A loop** (a bad bootstrap object, a missing parameter, enrolment that never completes) shows as eligible capacity below `box_count` for 25 minutes, and the alarm fires (§4). That alarm also covers the first-build case, where the group won't replace on its own. |
| |
| -**The platform box** keeps its hand-managed state (keymail, woodstar, relay masters, the console database) and isn't rebuilt by this script. Its `host.env` moves to the same parameters, so a rebuild needs fewer hand steps, but it stays a fixed box. It is enrolled once by an operator, with the `platform` role (§7). |
| +**The platform box** keeps its hand-managed state (keymail, woodstar, relay masters, the console database) and isn't rebuilt by this script. Its `host.env` moves to the same parameters, so a rebuild needs fewer hand steps, but it stays a fixed box. During the rollout it's an ordinary enrolled box, set `draining` while its instances move (§9). It takes the `platform` role at acceptance. |
| |
| ## 6. Ingress, DNS and forwarding |
| |
| @@ -325,7 +327,11 @@ Each is closed by a code change, or by showing it's safe under claim mode, befor |
| - Apply the module in group mode. The NLB is created with **only zone B's mapping**, carrying its new EIP. |
| - The two group boxes boot, join, enrol and reach hosting mode. The NLB is rehearsed through zone B's address. That checks zone B's targets only: zone A's are first exercised after step 3. |
| - The old fixed box keeps the region's EIP and all public traffic. It forwards to the group boxes over their private addresses as instances move to them. |
| -2. **Migrate.** Relocate every instance on the old box to §7's choice, one at a time, with spec 1's relocation. Each relocated instance is reached the usual ways: through the old box, which still holds the region's EIP, privately, and from other regions at the group box's own public address. When this step ends, the old box hosts no customer instance. |
| +2. **Migrate.** |
| + - **First, set the old box `draining`** (A6-1). This is spec 1's existing placement exclusion (`internal/boxchoice/boxchoice.go:133`), set through the box by the placer (S1 §5). It's persistent, and it makes the box ineligible for every new placement, pending creates included. |
| + - **Settle selections already made.** Pending creates in `starting` that chose the old box (§7) are reconciled to completion first, so their instances join the migration. |
| + - **Then relocate** every instance on the old box to §7's choice, one at a time, with spec 1's relocation. Each relocated instance is reached the usual ways: through the old box, which still holds the region's EIP, privately, and from other regions at the group box's own public address. When this step ends, the old box hosts no customer instance. |
| + - **Check it's empty:** the claim names no host whose executor is the old box, and the console has no `starting` pending create for it. The check is repeated immediately before step 3 and again before the box is terminated. |
| 3. **Cut over the EIP.** The old box is empty by now, so moving its address strands nothing. |
| 1. **Disassociate the region's EIP** from the old box. In IE, immediately associate the new EIP with it, switch the relay and MX records to the new address, and restart the relay with that address advertised. |
| 2. **Add zone A's mapping** to the NLB with the region's EIP (`SetSubnets`, which accepts an EIP when adding a mapping). |
| @@ -334,16 +340,17 @@ Each is closed by a code change, or by showing it's safe under claim mode, befor |
| 5. In IE, before the window closes: check that mail arrives on the new address, and that a TURN allocation advertises it and carries media. |
| 4. **Accept.** |
| - **UK, US and AU:** the old box stays running, deregistered from the NLB and empty, for **seven days**. That's the rollback window. Then it's terminated. |
| - - **IE:** the old box is the platform box. It's enrolled with the `platform` role, keeps its new EIP from step 0, and stays registered as an NLB target. |
| + - **IE:** the old box becomes the platform box. **Only now** does its roster role change from `draining` to `platform` (A6-2), after the seven-day window and the emptiness check. It keeps its new EIP from step 0 and stays registered as an NLB target. |
| |
| **Rollback** (A1-13). The region's EIP is a standalone `aws_eip` with `prevent_destroy`, so the allocation is never lost. |
| -- **Before step 3:** relocate instances back to the old box, then destroy the group and NLB. The region is back to today's state. |
| +- **Before step 3:** clear `draining` on the old box, relocate instances back to it, then destroy the group and NLB. The region is back to today's state. |
| - **From step 3 until acceptance:** |
| 1. Remove zone A's mapping from the NLB, or delete the NLB. An address on a requester-managed interface can't be disassociated directly. |
| 2. Wait until the EIP shows as unassociated. |
| 3. Associate it with the old box, which republishes it. |
| 4. **In IE** (A2-9), associating the region's EIP with the old box takes its new EIP off it, because one private address holds one public address. So the relay and MX records are switched back to the region's address, and the relay is restarted advertising it. The new EIP's allocation is kept for the next attempt. |
| - 5. Relocate instances back to the old box if the group is the problem. Otherwise the group boxes keep them, reached through the old box privately. |
| + 5. **Clear `draining`** on the old box, which makes it eligible again. In IE that works because it doesn't take the `platform` role until acceptance (A6-2). |
| + 6. Relocate instances back to the old box if the group is the problem. Otherwise the group boxes keep them, reached through the old box privately. |
| - **After acceptance:** there's no rollback to the old box, because it's gone. Recovery is forward, within the group. |
| |
| **Cost.** |
| @@ -386,6 +393,8 @@ Each is closed by a code change, or by showing it's safe under claim mode, befor |
| - **Placement.** Unit tests for each eligibility rule, the spread preference, the headroom ordering, the stale-headroom cut-off, the platform role, and unplaced with its retry triggers. |
| - **Rehearsal on dev:** |
| - the full §9 sequence, with migration before the cut-over, the cut-over window measured, the zone-B rehearsal, and the rollback sequence run once; |
| + - a create racing the start of migration: the draining box is never chosen, and a `starting` selection for it is settled before the emptiness check; |
| + - IE's role order: `draining`, rollback clearing it and relocating back, and `platform` only at acceptance; |
| - one group box terminated by hand, with replacement and re-enrolment observed; |
| - a box made unhealthy, observed being terminated. |
| - **Failover drills** belong to spec 3. |
| @@ -466,3 +475,10 @@ All three were answered in Astra's round 1 and folded in: |
| | A3-1, A3-2, A4-1, A4-2 | Moot for the same reason. Rev 3's version 2 credential, three-stage rollout and floor gate are removed | |
| | A2-7, A2-9 | Still met: a box publishes only an address IMDS reports, and instances leave the old box before its address moves (§6, §9) | |
| | A3-3 refresh-worker liveness | Kept (§6) | |
| + |
| +## 19. Astra round 6 dispositions (rev 7) |
| + |
| +| Finding | Disposition | |
| +|---|---| |
| +| A6-1 P2: migration doesn't stop new placements on the old box | Accepted: `draining` is set first, `starting` selections are settled, and emptiness is checked before the cut-over and before termination (§9) | |
| +| A6-2 P2: IE rollback targets a box the policy forbids | Accepted: the IE box is `draining`, not `platform`, until acceptance, and rollback clears `draining` (§5, §9) | |
amadan