| index 831a6df04..23977dc9c 100644 |
| --- a/docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md |
| +++ b/docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md |
| @@ -1,6 +1,8 @@ |
| # Region HA, spec 2: two-box regions |
| |
| -Design, 2026-10-15, **rev 7**. DRAFT. Nothing here is built. |
| +Design, 2026-10-15, **rev 8**. DRAFT. Nothing here is built. |
| + |
| +**Rev 8** answers Astra's round 7 (`CODEX-FINDINGS-REGION-HA-SPEC2-7.md`, cited **A7-n**): spec 2 builds the durable `draining` control that S1 left to spec 4, and the emptiness check covers in-flight creates. §20 maps both. |
| |
| **Rev 7** answers Astra's round 6 (`CODEX-FINDINGS-REGION-HA-SPEC2-6.md`, cited **A6-n**), two P2s in the rollout, both accepted. The old box is set `draining` before migration and checked empty before the cut-over and before termination. IE's old box takes the `platform` role only at acceptance, so rollback can move instances back to it. §19 maps them. |
| |
| @@ -287,6 +289,21 @@ The choice is made once and recorded in the create intent, as now (H12). Recover |
| - **The hostname answers with the edge's unknown-host response.** The host isn't in any edges file, so it isn't routed. |
| - **Copy.** The new console text and the alert go through Paul's copy review. |
| |
| +**Draining** (A7-1). S1 assigned the writer of a box's `draining` state to spec 4. `boxchoice.Eligible` already honours it (`internal/boxchoice/boxchoice.go:133`), but nothing sets it: the edges publisher always writes it false (`internal/host/edgesfile.go:778`). Spec 2 builds the control, because the rollout (§9) needs it: |
| +- **Where it lives:** a field on the box's roster entry, `draining: true`, written by the console. The roster is signed and persistent, so the state survives restarts of the box and the console. It is never derived from the edges file. |
| +- **How it's set and cleared:** `carlos system box drain <ref>` and `carlos system box undrain <ref>`, through a console API that only an operator may call. Both are idempotent. |
| +- **Who reads it:** placement reads the roster when choosing (§7's eligibility), so a drained box is excluded from the next choice the console makes. Placement and the drain control live in the same console process, so there's no propagation delay. |
| +- **What it shows:** the box mirrors the roster value into its edges file's `draining` field for display only. Readers that decide anything read the roster. |
| +- **The acknowledgement** that drain has taken effect is the settlement barrier below. Nothing needs to wait for a box to republish. |
| + |
| +**The settlement barrier** (A7-2) runs after drain is set, before relocation begins, and again before the cut-over and before termination. It covers every way an instance can come to land on the box: |
| +1. **Creates in flight in the console.** The console tracks every create request between choosing an executor and writing the claim. The barrier waits until every request that began before the drain was set has finished, one way or the other. |
| +2. **Claim creates not yet complete.** An immediate create records its chosen box in `intent.selected_executor` (`internal/hostclaim/create.go:115`) and only sets `op.exec` when it completes. Recovery keeps that selection without checking `draining` (`internal/hostops/production_create.go:257`). So the barrier scans the claim for incomplete creates whose `selected_executor` is the drained box, and waits for each to complete or be abandoned. A completed one then joins the migration. |
| +3. **Pending creates in `starting`** that chose the box (§7): reconciled to completion first, likewise. |
| +4. **Completed hosts:** the claim names no host whose executor is the box. |
| + |
| +The box is empty only when all four are clear. |
| + |
| **Rebalancing:** none automatic. An operator moves an instance between boxes in a region with spec 1's relocation. Automatic rebalancing is spec 4's. |
| |
| **Failover** (spec 3) uses this same choice for each displaced instance. |
| @@ -328,10 +345,10 @@ Each is closed by a code change, or by showing it's safe under claim mode, befor |
| - The two group boxes boot, join, enrol and reach hosting mode. The NLB is rehearsed through zone B's address. That checks zone B's targets only: zone A's are first exercised after step 3. |
| - The old fixed box keeps the region's EIP and all public traffic. It forwards to the group boxes over their private addresses as instances move to them. |
| 2. **Migrate.** |
| - - **First, set the old box `draining`** (A6-1). This is spec 1's existing placement exclusion (`internal/boxchoice/boxchoice.go:133`), set through the box by the placer (S1 §5). It's persistent, and it makes the box ineligible for every new placement, pending creates included. |
| - - **Settle selections already made.** Pending creates in `starting` that chose the old box (§7) are reconciled to completion first, so their instances join the migration. |
| + - **First, drain the old box** (A6-1, A7-1): `carlos system box drain <ref>`. The drain is persistent, and the box is ineligible for every new placement, pending creates included (§7). |
| + - **Run the settlement barrier** (§7, A7-2): in-flight console creates, incomplete claim creates that selected the box, and `starting` pending creates. Each one that lands on the box joins the migration. |
| - **Then relocate** every instance on the old box to §7's choice, one at a time, with spec 1's relocation. Each relocated instance is reached the usual ways: through the old box, which still holds the region's EIP, privately, and from other regions at the group box's own public address. When this step ends, the old box hosts no customer instance. |
| - - **Check it's empty:** the claim names no host whose executor is the old box, and the console has no `starting` pending create for it. The check is repeated immediately before step 3 and again before the box is terminated. |
| + - **Check it's empty:** the full settlement barrier (§7) is clear. It's repeated immediately before step 3 and again before the box is terminated. |
| 3. **Cut over the EIP.** The old box is empty by now, so moving its address strands nothing. |
| 1. **Disassociate the region's EIP** from the old box. In IE, immediately associate the new EIP with it, switch the relay and MX records to the new address, and restart the relay with that address advertised. |
| 2. **Add zone A's mapping** to the NLB with the region's EIP (`SetSubnets`, which accepts an EIP when adding a mapping). |
| @@ -349,7 +366,7 @@ Each is closed by a code change, or by showing it's safe under claim mode, befor |
| 2. Wait until the EIP shows as unassociated. |
| 3. Associate it with the old box, which republishes it. |
| 4. **In IE** (A2-9), associating the region's EIP with the old box takes its new EIP off it, because one private address holds one public address. So the relay and MX records are switched back to the region's address, and the relay is restarted advertising it. The new EIP's allocation is kept for the next attempt. |
| - 5. **Clear `draining`** on the old box, which makes it eligible again. In IE that works because it doesn't take the `platform` role until acceptance (A6-2). |
| + 5. **Undrain the old box** (`carlos system box undrain <ref>`), which makes it eligible again. In IE that works because it doesn't take the `platform` role until acceptance (A6-2). |
| 6. Relocate instances back to the old box if the group is the problem. Otherwise the group boxes keep them, reached through the old box privately. |
| - **After acceptance:** there's no rollback to the old box, because it's gone. Recovery is forward, within the group. |
| |
| @@ -393,7 +410,8 @@ Each is closed by a code change, or by showing it's safe under claim mode, befor |
| - **Placement.** Unit tests for each eligibility rule, the spread preference, the headroom ordering, the stale-headroom cut-off, the platform role, and unplaced with its retry triggers. |
| - **Rehearsal on dev:** |
| - the full §9 sequence, with migration before the cut-over, the cut-over window measured, the zone-B rehearsal, and the rollback sequence run once; |
| - - a create racing the start of migration: the draining box is never chosen, and a `starting` selection for it is settled before the emptiness check; |
| + - a create racing the start of migration, through each path separately: an immediate create that selected the box before the drain and completes after it; one that stalls and completes through recovery; a console request between choosing and the claim write; and a `starting` pending create. Each is found by the settlement barrier and joins the migration, and the box is never reported empty while one is outstanding; |
| + - drain and undrain surviving a console restart and a box restart; |
| - IE's role order: `draining`, rollback clearing it and relocating back, and `platform` only at acceptance; |
| - one group box terminated by hand, with replacement and re-enrolment observed; |
| - a box made unhealthy, observed being terminated. |
| @@ -482,3 +500,10 @@ All three were answered in Astra's round 1 and folded in: |
| |---|---| |
| | A6-1 P2: migration doesn't stop new placements on the old box | Accepted: `draining` is set first, `starting` selections are settled, and emptiness is checked before the cut-over and before termination (§9) | |
| | A6-2 P2: IE rollback targets a box the policy forbids | Accepted: the IE box is `draining`, not `platform`, until acceptance, and rollback clears `draining` (§5, §9) | |
| + |
| +## 20. Astra round 7 dispositions (rev 8) |
| + |
| +| Finding | Disposition | |
| +|---|---| |
| +| A7-1 P2: the draining control is still spec 4's | Accepted: spec 2 builds it, as a roster field written by an operator-only console API and CLI. It's persistent, and placement reads it directly (§7) | |
| +| A7-2 P2: settlement misses immediate creates | Accepted: a settlement barrier over in-flight console creates, incomplete claim creates by `selected_executor`, `starting` pending creates and completed hosts. It's run before relocation, before the cut-over and before termination (§7, §9) | |
amadan