| index 18854292f..e844a5a70 100644 |
| --- a/docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md |
| +++ b/docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md |
| @@ -1,6 +1,8 @@ |
| # Region HA, spec 2: two-box regions |
| |
| -Design, 2026-10-15, **rev 9**. DRAFT. Nothing here is built. |
| +Design, 2026-10-15, **rev 10**. DRAFT. Nothing here is built. |
| + |
| +**Rev 10** follows Paul's ruling **H25**: spec 2 no longer empties the old boxes. Rounds 6 to 9 each found another gap in draining a live box, because that was spec 4's drain pulled forward. Instead, each region's old box keeps its existing instances, takes no new ones, and swaps to a new public address of its own at the cut-over. The live drain returns to spec 4, with rounds 6 to 9's findings recorded as its inputs. §22 maps them. |
| |
| **Rev 9** answers Astra's round 8 (`CODEX-FINDINGS-REGION-HA-SPEC2-8.md`, cited **A8-n**). Rounds 6 to 8 each found another way an instance could reach a draining box. So the settlement barrier is now one general rule over the claim, not a list of operation kinds. Never-woken instances get one platform wake before relocation. §21 maps both. |
| |
| @@ -58,7 +60,8 @@ from 2026-10-15: |
| | H21 | **Ingress is one Network Load Balancer per region**, passing TCP 80 and 443 straight through. TLS stays on the boxes. | |
| | H22 | **Every carloku.com region runs two boxes**: dev first, then UK, US, AU and IE. (H5's per-region minimum is set to 2 everywhere.) | |
| | H23 | **The specs are recut.** Spec 2 covers two-box regions: this document. Spec 3 covers failover. Spec 4 shrinks to automatic scale-out and scale-in, rebalancing and latency steering. S1 §12's split is superseded where they differ. | |
| -| H24 | **IE's current box becomes the platform box.** It's fixed, outside the group, and keeps the console, keymail, woodstar, the relay and the scanner grants. It hosts no customer instances once migration ends. | |
| +| H24 | **IE's current box becomes the platform box.** It's fixed, outside the group, and keeps the console, keymail, woodstar, the relay and the scanner grants. It takes no new customer instances. Its existing ones leave with spec 4's drain (H25). | |
| +| H25 | **(2026-10-16) Spec 2 doesn't empty the old boxes.** Each region's old fixed box keeps the instances it has and takes no new ones. Those instances keep today's single-box behaviour until spec 4's drain moves them, or until spec 3's failover moves them if AWS terminates the box. Each region runs three boxes until then. | |
| |
| Design choices Paul approved in section review on the same day, recorded |
| here so they aren't reopened: |
| @@ -193,7 +196,7 @@ Anything else is joining mode. Joining resumes from whatever it finds: no invita |
| - **Any step:** the health answer never reaches 200. The group replaces the box when the grace period ends, **except** while the group's target group has never had a healthy target: AWS doesn't replace ELB-unhealthy instances then (A1-11). That case happens only when a region is first built. |
| - **A loop** (a bad bootstrap object, a missing parameter, enrolment that never completes) shows as eligible capacity below `box_count` for 25 minutes, and the alarm fires (§4). That alarm also covers the first-build case, where the group won't replace on its own. |
| |
| -**The platform box** keeps its hand-managed state (keymail, woodstar, relay masters, the console database) and isn't rebuilt by this script. Its `host.env` moves to the same parameters, so a rebuild needs fewer hand steps, but it stays a fixed box. During the rollout it's an ordinary enrolled box, set `draining` while its instances move (§9). It takes the `platform` role at acceptance. |
| +**The platform box** keeps its hand-managed state (keymail, woodstar, relay masters, the console database) and isn't rebuilt by this script. Its `host.env` moves to the same parameters, so a rebuild needs fewer hand steps, but it stays a fixed box. It takes the `platform` role at the rollout's step 2 (§9). It keeps the customer instances it already has until spec 4's drain (H25). |
| |
| ## 6. Ingress, DNS and forwarding |
| |
| @@ -291,27 +294,11 @@ The choice is made once and recorded in the create intent, as now (H12). Recover |
| - **The hostname answers with the edge's unknown-host response.** The host isn't in any edges file, so it isn't routed. |
| - **Copy.** The new console text and the alert go through Paul's copy review. |
| |
| -**Draining** (A7-1). S1 assigned the writer of a box's `draining` state to spec 4. `boxchoice.Eligible` already honours it (`internal/boxchoice/boxchoice.go:133`), but nothing sets it: the edges publisher always writes it false (`internal/host/edgesfile.go:778`). Spec 2 builds the control, because the rollout (§9) needs it: |
| -- **Where it lives:** a field on the box's roster entry, `draining: true`, written by the console. The roster is signed and persistent, so the state survives restarts of the box and the console. It is never derived from the edges file. |
| -- **How it's set and cleared:** `carlos system box drain <ref>` and `carlos system box undrain <ref>`, through a console API that only an operator may call. Both are idempotent. |
| -- **Who reads it:** placement reads the roster when choosing (§7's eligibility), so a drained box is excluded from the next choice the console makes. Placement and the drain control live in the same console process, so there's no propagation delay. |
| -- **What it shows:** the box mirrors the roster value into its edges file's `draining` field for display only. Readers that decide anything read the roster. |
| -- **The acknowledgement** that drain has taken effect is the settlement barrier below. Nothing needs to wait for a box to republish. |
| - |
| -**The settlement barrier** (A7-2, A8-1) runs after drain is set, before relocation begins, and again before the cut-over and before termination. Rather than listing the ways an instance can come to land on a box, it uses one rule. **A box is settled when nothing anywhere still names it as a destination:** |
| -1. **No claim record names it, in any role.** Every claim record, of every operation kind and in every phase, is scanned. None may name the box as: |
| - - its executor (`op.exec`); |
| - - its selected executor (`intent.selected_executor`, `internal/hostclaim/create.go:115`); |
| - - a relocation target (`intent.target_box`, `internal/hostrelocate/worker.go:577`); |
| - - a move target (`intent.target`, `internal/hostclaim/move.go:58`); |
| - - any other box field that a later claim revision adds. The scan is written against the claim's box-reference fields, not a list of operation kinds, and a test fails if a new box field is added without the scan reading it. |
| -2. **No request chose it before the drain and is still in flight.** That means: |
| - - the console's create and move doors, and its pending creates in `starting` (§7). The console tracks every request between choosing a box and writing the claim, and the barrier waits until every request that began before the drain has finished; |
| - - box-side starters, such as box-started relocations (S1 H18). They choose with `boxchoice`, which reads the roster's `draining` (above). The barrier waits one roster refresh interval after the drain, plus margin, so every box has seen it. |
| +**Old boxes** (H25). Each region's old fixed box takes the roster role **`legacy`**; IE's takes **`platform`**. Both roles are never eligible for a new placement, a relocation target or a move target. That's a static exclusion, read from the roster by every chooser (`boxchoice.Eligible`), and it needs no settlement: |
| +- the box's existing instances stay where they are; |
| +- operations already heading to it before the role was set simply finish there, and the instance joins the box's existing ones. Nothing needs to be empty, so nothing needs to be waited for. |
| |
| -An operation found in step 1 isn't cancelled. It's allowed to finish, and whatever it lands on the box then joins the migration. |
| - |
| -**Never-woken instances** (A8-2). Relocation refuses a host whose holder has never had a session (`Holder.Session == 0`, `internal/hostrelocate/worker.go:329`, kept by `worker_test.go:482`). A create that completed but never received traffic is such a host. So migration first **wakes each never-woken instance once on the old box**: a platform-initiated wake through the activator's normal admission, with no customer request. It then has a session, and relocation proceeds as for any other instance. The wake runs the customer's app once, as a scheduled wake would. If a wake fails, the instance stays and the emptiness check doesn't pass, and the operator sees which one. |
| +The roles are set by an operator, through `carlos system box role <ref> legacy|platform|default`, a console API only an operator may call, and they persist in the roster. Spec 4's drain later moves a `legacy` box's instances to the group, and then the box is retired. |
| |
| **Rebalancing:** none automatic. An operator moves an instance between boxes in a region with spec 1's relocation. Automatic rebalancing is spec 4's. |
| |
| @@ -344,46 +331,35 @@ Each is closed by a code change, or by showing it's safe under claim mode, befor |
| **Order:** dev, then UK, US, AU and IE, one region at a time. Each region needs §3 to hold first. |
| |
| **Per region:** |
| -0. **IE only: prepare the relay and MX** (A1-4, A2-6). |
| - - Allocate the platform box's new EIP, unattached. |
| - - Lower the TTLs of `relay.eu-west-1.oncarlos.com` and the MX target to 60 seconds. Wait at least twice their old TTL before the cut-over. |
| - - Set the new EIP's reverse DNS once its forward record exists, and add it to SPF beside the old address. |
| +0. **Prepare addresses.** |
| + - Allocate the old box's **new EIP**, unattached. It's the old box's own address after the cut-over. |
| + - **IE only:** lower the TTLs of `relay.eu-west-1.oncarlos.com` and the MX target to 60 seconds, and wait at least twice their old TTL before the cut-over. Set the new EIP's reverse DNS once its forward record exists, and add it to SPF beside the old address (A1-4, A2-6). |
| 1. **Build.** |
| - Write the SSM parameters. |
| - Apply the module in group mode. The NLB is created with **only zone B's mapping**, carrying its new EIP. |
| - The two group boxes boot, join, enrol and reach hosting mode. The NLB is rehearsed through zone B's address. That checks zone B's targets only: zone A's are first exercised after step 3. |
| - - The old fixed box keeps the region's EIP and all public traffic. It forwards to the group boxes over their private addresses as instances move to them. |
| -2. **Migrate.** |
| - - **First, drain the old box** (A6-1, A7-1): `carlos system box drain <ref>`. The drain is persistent, and the box is ineligible for every new placement, pending creates included (§7). |
| - - **Run the settlement barrier** (§7): no claim record names the box in any role, and no request that chose it before the drain is in flight. Operations already heading to it finish, and whatever they land joins the migration. |
| - - **Wake never-woken instances once** (§7, A8-2), so each has a session to relocate. |
| - - **Then relocate** every instance on the old box to §7's choice, one at a time, with spec 1's relocation. Each relocated instance is reached the usual ways: through the old box, which still holds the region's EIP, privately, and from other regions at the group box's own public address. When this step ends, the old box hosts no customer instance. |
| - - **Check it's empty:** the full settlement barrier (§7) is clear. It's repeated immediately before step 3 and again before the box is terminated. |
| -3. **Cut over the EIP.** The old box is empty by now, so moving its address strands nothing. |
| - 1. **Disassociate the region's EIP** from the old box. In IE, immediately associate the new EIP with it, switch the relay and MX records to the new address, and restart the relay with that address advertised. |
| - 2. **Add zone A's mapping** to the NLB with the region's EIP (`SetSubnets`, which accepts an EIP when adding a mapping). |
| - 3. **The cost:** the region's public address is down from step i until the zone A node is active. Expect seconds to a few minutes, scheduled at a quiet hour and announced. The window is measured on dev first. |
| - 4. **In IE, platform routes:** the platform box republishes at once with its new address (a forced edges republication). Until other regions' maps have it, which takes up to a minute, a cross-region forward to the console lands on the NLB and is refused. That's inside the window. Requests for platform hostnames from browsers go to the region's EIP and reach the platform box privately through the NLB, so they're unaffected once zone A is active. |
| + - The old box keeps the region's EIP and all public traffic. It's also registered as an NLB target, and stays one: it keeps serving its own instances and forwarding. |
| +2. **Exclude the old box from placement** (H25): `carlos system box role <ref> legacy`, or `platform` in IE. New instances now land on the group boxes, reached through the old box privately until the cut-over. The old box's own instances are untouched. |
| +3. **Cut over the EIP.** |
| + 1. **Disassociate the region's EIP** from the old box, and immediately **associate the old box's new EIP** with it. In IE, also switch the relay and MX records to the new address and restart the relay with that address advertised. |
| + 2. **Republish at once:** the old box publishes its new address with a forced edges republication. Other regions' maps pick it up within a minute. Until then, a cross-region forward to one of the old box's instances dials the region's EIP, lands on the NLB, and is refused, because a forwarded request isn't forwarded again. That's inside the window. |
| + 3. **Add zone A's mapping** to the NLB with the region's EIP (`SetSubnets`, which accepts an EIP when adding a mapping). |
| + 4. **The cost:** the region's public address is down from step i until the zone A node is active, plus up to a minute more for cross-region requests to the old box's instances. Expect a few minutes in all, scheduled at a quiet hour and announced. The window is measured on dev first. |
| 5. In IE, before the window closes: check that mail arrives on the new address, and that a TURN allocation advertises it and carries media. |
| -4. **Accept.** |
| - - **UK, US and AU:** the old box stays running, deregistered from the NLB and empty, for **seven days**. That's the rollback window. Then it's terminated. |
| - - **IE:** the old box becomes the platform box. **Only now** does its roster role change from `draining` to `platform` (A6-2), after the seven-day window and the emptiness check. It keeps its new EIP from step 0 and stays registered as an NLB target. |
| +4. **Accept.** After seven days with no rollback, the region is done. The old box stays, serving its instances, until spec 4's drain. |
| |
| **Rollback** (A1-13). The region's EIP is a standalone `aws_eip` with `prevent_destroy`, so the allocation is never lost. |
| -- **Before step 3:** clear `draining` on the old box, relocate instances back to it, then destroy the group and NLB. The region is back to today's state. |
| +- **Before step 3:** set the old box's role back to `default`, and destroy the group and NLB. Any instance created on the group boxes meanwhile is relocated to the old box first (it's eligible again). The region is back to today's state. |
| - **From step 3 until acceptance:** |
| 1. Remove zone A's mapping from the NLB, or delete the NLB. An address on a requester-managed interface can't be disassociated directly. |
| 2. Wait until the EIP shows as unassociated. |
| - 3. Associate it with the old box, which republishes it. |
| - 4. **In IE** (A2-9), associating the region's EIP with the old box takes its new EIP off it, because one private address holds one public address. So the relay and MX records are switched back to the region's address, and the relay is restarted advertising it. The new EIP's allocation is kept for the next attempt. |
| - 5. **Undrain the old box** (`carlos system box undrain <ref>`), which makes it eligible again. In IE that works because it doesn't take the `platform` role until acceptance (A6-2). |
| - 6. Relocate instances back to the old box if the group is the problem. Otherwise the group boxes keep them, reached through the old box privately. |
| -- **After acceptance:** there's no rollback to the old box, because it's gone. Recovery is forward, within the group. |
| + 3. Swap the old box's addresses back: disassociate its new EIP, associate the region's EIP, and force a republication. In IE, switch the relay and MX records back and restart the relay advertising the region's address (A2-9). |
| + 4. The group boxes keep their instances, reached through the old box privately. To remove the group as well, continue as for "before step 3". |
| +- **After acceptance:** recovery is forward, within the group. |
| |
| **Cost.** |
| -- UK, US and AU each go from one box to two, plus an NLB (about $20 a month before traffic) and zone B's EIP. |
| -- IE goes from one box to three: the platform box plus two group boxes. |
| -- Each old box costs seven more days during acceptance. |
| +- Every region goes from one box to three until spec 4's drain retires the old box: the old box plus two group boxes. It also gains an NLB (about $20 a month before traffic) and two new EIPs (zone B's, and the old box's own). |
| +- After the drain: two group boxes per region, plus IE's platform box. |
| - Dev matches whichever layout it rehearses. |
| |
| ## 10. Failure modes |
| @@ -392,7 +368,7 @@ Each is closed by a code change, or by showing it's safe under claim mode, befor |
| |---|---| |
| | A group box hangs or fails its health check | The NLB stops sending it traffic within about 20s, and the group terminates it once its health check has failed (after the grace period, for a new box). Its instances are down until spec 3 relocates them. A replacement boots, enrols and becomes placeable with no hand steps. | |
| | A group box is terminated by AWS | As above, without the health-check delay. | |
| -| The platform box dies | The console and platform services are down. EC2 auto-recovery covers hardware faults, and otherwise it's an operator job (S1 §12). Customer instances are unaffected once migration is done. | |
| +| An old box (`legacy` or `platform`) dies | Its own instances are down, as today. EC2 auto-recovery covers hardware faults. If AWS terminates it, spec 3's failover relocates them. Otherwise it's an operator job. In IE, the console and platform services are down too (S1 §12). Instances on the group boxes are unaffected. | |
| | Both group boxes in a region fail | The region's instances are down. The group replaces both, and the first replacement is confirmed from another region (H17). | |
| | The NLB's availability zone fails | The region's ingress is down until the zone recovers (§6). | |
| | A replacement can't boot | It's never healthy, the group keeps replacing it, and the eligible-capacity alarm fires at 25 minutes. | |
| @@ -417,16 +393,11 @@ Each is closed by a code change, or by showing it's safe under claim mode, befor |
| - The publisher publishes only what IMDS reports, and publishes no public address on a blank read. |
| - A forwarded request is still never forwarded again. |
| - Platform routes forwarded from a group box to the platform box. |
| -- **Placement.** Unit tests for each eligibility rule, the spread preference, the headroom ordering, the stale-headroom cut-off, the platform role, and unplaced with its retry triggers. |
| +- **Placement.** Unit tests for each eligibility rule, the spread preference, the headroom ordering, the stale-headroom cut-off, the `platform` and `legacy` roles, and unplaced with its retry triggers. |
| - **Rehearsal on dev:** |
| - - the full §9 sequence, with migration before the cut-over, the cut-over window measured, the zone-B rehearsal, and the rollback sequence run once; |
| - - a create racing the start of migration, through each path separately: an immediate create that selected the box before the drain and completes after it; one that stalls and completes through recovery; a console request between choosing and the claim write; and a `starting` pending create. Each is found by the settlement barrier and joins the migration, and the box is never reported empty while one is outstanding; |
| - - drain and undrain surviving a console restart and a box restart; |
| - - a relocation and a move that chose the box before the drain, each stalled and then resumed: both found by the claim scan, and each landed instance migrated; |
| - - a box-started relocation choosing just before the drain, covered by the roster-refresh wait; |
| - - a never-woken instance on the box: woken once, then relocated. A failed wake leaves the box not empty, and says why; |
| - - the claim scan's coverage test: adding a box-reference field to the claim without the scan reading it fails the test; |
| - - IE's role order: `draining`, rollback clearing it and relocating back, and `platform` only at acceptance; |
| + - the full §9 sequence, with the cut-over window measured, the zone-B rehearsal, and both rollback sequences run once; |
| + - the old box's address swap at the cut-over: forced republication, and cross-region forwards to its instances recovering within a minute; |
| + - the `legacy` and `platform` roles excluding a box from creates, relocation targets and move targets, persisting across console and box restarts, and an operation already heading to the box finishing there; |
| - one group box terminated by hand, with replacement and re-enrolment observed; |
| - a box made unhealthy, observed being terminated. |
| - **Failover drills** belong to spec 3. |
| @@ -439,7 +410,8 @@ Each is closed by a code change, or by showing it's safe under claim mode, befor |
| - offering zone B's ingress address to customers, so the region's published addresses span both zones (§6). The NLB already has a node in each zone; |
| - latency steering; |
| - H5's per-region minimum as anything other than a fixed group size; |
| - - the Auto Scaling activity-history evidence source (S1 A2-4). |
| + - the Auto Scaling activity-history evidence source (S1 A2-4); |
| + - **draining a live box** (H25): moving a `legacy` box's instances to the group and retiring it, which also empties IE's platform box of customers. Spec 2's rounds 6 to 9 are its starting inputs (§22): a durable drain control, a settlement protocol that acknowledges outstanding requests rather than waiting out time, first-wake support for never-woken instances, and claim box references sorted into live and historical. |
| - **Not planned:** relocating the console or platform services (S1 §12). The platform box is fixed, and its HA remains the HA ladder's. |
| |
| ## 13. Questions from rev 1 |
| @@ -528,3 +500,13 @@ All three were answered in Astra's round 1 and folded in: |
| |---|---| |
| | A8-1 P2: the barrier misses moves and relocations heading to the box | Accepted, generalised: one rule over every claim box-reference field, enforced by a coverage test, plus in-flight requests from the console doors and box-side starters (§7) | |
| | A8-2 P2: relocation refuses never-woken instances | Accepted: each is woken once on the old box before relocation, and a failure blocks the emptiness check visibly (§7, §9) | |
| + |
| +## 22. Astra round 9 dispositions, and H25 |
| + |
| +| Finding | Disposition | |
| +|---|---| |
| +| A9-1 P2: a timed wait doesn't settle box-side starters | Moot under H25: nothing has to be empty. Recorded for spec 4's drain | |
| +| A9-2 P2: a never-woken claim can't be woken by the normal path | Moot under H25. Recorded for spec 4's drain | |
| +| A9-3 P2: the barrier needs migration to have finished | Moot under H25. Recorded for spec 4's drain | |
| +| A9-4 P2: historical box references never clear | Moot under H25. Recorded for spec 4's drain | |
| +| A6-1, A7-1, A7-2, A8-1, A8-2 | Their machinery (the drain control, the settlement barrier, the one-off wake) is removed from spec 2 and recorded as spec 4's inputs. A6-2, IE rollback, is met differently: the IE box isn't drained, so rollback moves no instances back to it | |
amadan