amadan
region HA spec 2 (two-box regions), rev 1 draft, and its Astra brief
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2 files changed,
+326
−0
docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md+297 −0docs/superpowers/specs/CODEX-REVIEW-BRIEF-REGION-HA-SPEC2.md+29 −0
CI cancelled — superseded by a newer push — run details
CI log
=== gate === cd launchctx && go vet ./... && go test ./... -count=1 ok amadan.net/carlos/platform.git/launchctx 0.011s ? amadan.net/carlos/platform.git/launchctx/controlname [no test files] ? amadan.net/carlos/platform.git/launchctx/wireid [no test files] go vet ./... gofmt -l .: clean go test ./... -count=1 -timeout 30m ok github.com/carlosframework/platform 279.732s ? github.com/carlosframework/platform/cmd/changelog [no test files] ok github.com/carlosframework/platform/cmd/console 2.823s ok github.com/carlosframework/platform/cmd/moveprobe 92.344s ok github.com/carlosframework/platform/cmd/zddprobe 0.700s ok github.com/carlosframework/platform/internal/activator 248.660s ok github.com/carlosframework/platform/internal/agentsock 0.015s ok github.com/carlosframework/platform/internal/alerting 0.003s ok github.com/carlosframework/platform/internal/analytics 0.018s ok github.com/carlosframework/platform/internal/appconfig 2.458s ok github.com/carlosframework/platform/internal/apperr 0.069s ok github.com/carlosframework/platform/internal/applog 0.121s ok github.com/carlosframework/platform/internal/awsinv 0.003s ok github.com/carlosframework/platform/internal/boxchoice 0.006s ok github.com/carlosframework/platform/internal/boxid 0.629s ? github.com/carlosframework/platform/internal/boxid/boxidtest [no test files] ? github.com/carlosframework/platform/internal/boxid/internal/signtest [no test files] ok github.com/carlosframework/platform/internal/boxvolumes 0.008s ok github.com/carlosframework/platform/internal/caps 0.011s ok github.com/carlosframework/platform/internal/certwarm 0.001s ok github.com/carlosframework/platform/internal/cgroup 0.693s ok github.com/carlosframework/platform/internal/changelog 0.301s ok github.com/carlosframework/platform/internal/clamd 0.055s ? github.com/carlosframework/platform/internal/cliflag [no test files] ok github.com/carlosframework/platform/internal/console 387.030s ? github.com/carlosframework/platform/internal/controlname [no test files] ? github.com/carlosframework/platform/internal/controlsock [no test files] ok github.com/carlosframework/platform/internal/dbdoor 6.372s ok github.com/carlosframework/platform/internal/dbdoor/dbdoortest 0.008s ok github.com/carlosframework/platform/internal/dbverify 0.004s ok github.com/carlosframework/platform/internal/dnsrec 0.002s ok github.com/carlosframework/platform/internal/docsite 0.005s ok github.com/carlosframework/platform/internal/domains 2.331s ok github.com/carlosframework/platform/internal/doorkey 0.006s ok github.com/carlosframework/platform/internal/economics 0.004s ok github.com/carlosframework/platform/internal/edge 26.551s ? github.com/carlosframework/platform/internal/edgeproof [no test files] ok github.com/carlosframework/platform/internal/edgesfile 0.007s ok github.com/carlosframework/platform/internal/fleetapi 0.127s ok github.com/carlosframework/platform/internal/gate 1.654s ok github.com/carlosframework/platform/internal/genselect 0.014s ok github.com/carlosframework/platform/internal/geo 0.013s ok github.com/carlosframework/platform/internal/geoip 0.029s ok github.com/carlosframework/platform/internal/host 132.767s ok github.com/carlosframework/platform/internal/hostactors 0.579s ok github.com/carlosframework/platform/internal/hostbox 2.185s ok github.com/carlosframework/platform/internal/hostclaim 0.491s ok github.com/carlosframework/platform/internal/hostevidence 0.005s ok github.com/carlosframework/platform/internal/hostexec 4.908s ok github.com/carlosframework/platform/internal/hostfence 0.030s ok github.com/carlosframework/platform/internal/hostfiles 10.536s ok github.com/carlosframework/platform/internal/hostfloor 0.013s ok github.com/carlosframework/platform/internal/hostguard 1.010s ok github.com/carlosframework/platform/internal/hostops 1.550s ? github.com/carlosframework/platform/internal/hostops/fencerelease [no test files] ok github.com/carlosframework/platform/internal/hostops/internal/fencerelease 0.002s ok github.com/carlosframework/platform/internal/hostquarantine 0.002s ok github.com/carlosframework/platform/internal/hostrelocate 0.438s ok github.com/carlosframework/platform/internal/hostresolve 0.137s ok github.com/carlosframework/platform/internal/hostwrite 0.002s ok github.com/carlosframework/platform/internal/infraview 0.002s ok github.com/carlosframework/platform/internal/instancectl 4.171s ok github.com/carlosframework/platform/internal/instanceexec 2.693s ok github.com/carlosframework/platform/internal/instancefs 0.005s ok github.com/carlosframework/platform/internal/instancekey 0.003s ok github.com/carlosframework/platform/internal/instanceupdates 0.008s ok github.com/carlosframework/platform/internal/layout 1.491s ok github.com/carlosframework/platform/internal/lease 9.795s ok github.com/carlosframework/platform/internal/lease/leasetest 0.006s ok github.com/carlosframework/platform/internal/ledger 0.017s ok github.com/carlosframework/platform/internal/mail 0.172s ok github.com/carlosframework/platform/internal/maintenance 0.086s ok github.com/carlosframework/platform/internal/maintenanceexec 5.498s ok github.com/carlosframework/platform/internal/maintenanceguard 0.072s ok github.com/carlosframework/platform/internal/metrics 0.005s ok github.com/carlosframework/platform/internal/moves 20.958s ok github.com/carlosframework/platform/internal/moves/archive 0.779s ok github.com/carlosframework/platform/internal/objstore 0.179s ok github.com/carlosframework/platform/internal/ops 0.178s ok github.com/carlosframework/platform/internal/park 0.201s ok github.com/carlosframework/platform/internal/privatefile 0.006s ok github.com/carlosframework/platform/internal/provision 16.746s ok github.com/carlosframework/platform/internal/publicrelease 0.010s ? github.com/carlosframework/platform/internal/regionname [no test files] ok github.com/carlosframework/platform/internal/registry 4.917s ok github.com/carlosframework/platform/internal/relay 7.243s ok github.com/carlosframework/platform/internal/release 19.745s ok github.com/carlosframework/platform/internal/rollout 3.738s ok github.com/carlosframework/platform/internal/rolloutwords 0.003s ok github.com/carlosframework/platform/internal/rootaws 0.009s ok github.com/carlosframework/platform/internal/rootparticipation 0.006s ok github.com/carlosframework/platform/internal/rootstate 0.025s ok github.com/carlosframework/platform/internal/runs 0.592s ok github.com/carlosframework/platform/internal/s3compat 0.005s ok github.com/carlosframework/platform/internal/safefile 0.004s ok github.com/carlosframework/platform/internal/schedule 2.508s ok github.com/carlosframework/platform/internal/seeds 0.011s ok github.com/carlosframework/platform/internal/sim/hostlife 9.397s ok github.com/carlosframework/platform/internal/sim/litestreamfacts 4.296s ok github.com/carlosframework/platform/internal/status 0.041s ok github.com/carlosframework/platform/internal/steering 0.008s ok github.com/carlosframework/platform/internal/stopevidence 0.168s ok github.com/carlosframework/platform/internal/svcrole 0.002s ok github.com/carlosframework/platform/internal/telemetry 0.004s ok github.com/carlosframework/platform/internal/termfence 0.004s ok github.com/carlosframework/platform/internal/tunnelclient 0.004s ok github.com/carlosframework/platform/internal/unixdial 0.002s ok github.com/carlosframework/platform/internal/updatewords 0.004s ok github.com/carlosframework/platform/internal/usage 0.004s ok github.com/carlosframework/platform/internal/version 0.002s ok github.com/carlosframework/platform/internal/vet 1.548s === cross === --- shard 0 of 5 --- cross-compiling darwin/arm64 go: downloading github.com/ncruces/go-strftime v1.0.0 amadan: job cancelled — a newer push superseded it --- shard 1 of 5 --- cross-compiling darwin/amd64 go: downloading github.com/ncruces/go-strftime v1.0.0 amadan: job cancelled — a newer push superseded it --- shard 2 of 5 --- cross-compiling linux/arm64 amadan: job cancelled — a newer push superseded it --- shard 3 of 5 --- cross-compiling windows/amd64 go: downloading github.com/ncruces/go-strftime v1.0.0 amadan: job cancelled — a newer push superseded it --- shard 4 of 5 --- cross-compiling windows/arm64 go: downloading github.com/ncruces/go-strftime v1.0.0 amadan: job cancelled — a newer push superseded it
diff --git a/docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md b/docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md| new file mode 100644 |
| index 000000000..9fa8512f2 |
| --- /dev/null |
| +++ b/docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md |
| @@ -0,0 +1,297 @@ |
| +# Region HA, spec 2: two-box regions |
| + |
| +Design, 2026-10-15, **rev 1**. DRAFT. Nothing here is built. |
| + |
| +This spec builds on **spec 1** (`2026-10-04-region-ha-box-placement-design.md`, |
| +cited **S1 §n**), which puts several boxes in a region under the host |
| +lifecycle claim. Spec 1's code, and relocation (plan 1b-next), are on |
| +main and dormant. This spec turns every carloku.com region into a group |
| +of two interchangeable boxes behind a load balancer, with a placement |
| +policy for which box takes each instance. |
| + |
| +## 1. Purpose |
| + |
| +carloku.com instances should keep serving when one box in a region |
| +fails, with no one paged, in the region that sold them. Today each |
| +region runs one fixed EC2 box. If it dies, its instances are down until |
| +it's recovered by hand. EC2 auto-recovery restarts it in place, but only |
| +for AWS hardware faults. |
| + |
| +Spec 2 gives each region two boxes. A sick box is terminated and |
| +replaced automatically, and new instances are spread across the two |
| +boxes. **Spec 3** (failover) then moves a terminated box's instances to |
| +the survivor. Spec 2 is useful without spec 3: a replaced box rejoins |
| +with no hand steps, and new instances keep landing on the healthy box. |
| +But a box's instances still go down with it until spec 3 lands. |
| + |
| +## 2. Rulings |
| + |
| +Paul's rulings of 2026-10-04 (S1 §2, H1 to H19) stand. These are new, |
| +from 2026-10-15: |
| + |
| +| # | Ruling | |
| +|---|---| |
| +| H20 | **The first HA release recovers hung boxes as well as lost ones.** H1 makes termination the only proof of death. A hung box is terminated only by something that health-checks it, so each region's boxes run in an **Auto Scaling group** with health checks. Group size is fixed for now. Automatic scale-out and scale-in stay in spec 4. | |
| +| H21 | **Ingress is one Network Load Balancer per region**, passing TCP 80 and 443 straight through. TLS stays on the boxes. | |
| +| H22 | **Every carloku.com region runs two boxes**: dev first, then UK, US, AU and IE. (H5's per-region minimum is set to 2 everywhere.) | |
| +| H23 | **The specs are recut.** Spec 2 covers two-box regions: this document. Spec 3 covers failover. Spec 4 shrinks to automatic scale-out and scale-in, rebalancing and latency steering. S1 §12's split is superseded where they differ. | |
| +| H24 | **IE's current box becomes the platform box.** It's fixed, outside the group, and keeps the console, keymail, woodstar, the relay and the scanner grants. It hosts no customer instances once migration ends. | |
| + |
| +Design choices Paul approved in section review on the same day, recorded |
| +here so they aren't reopened: |
| + |
| +- **Boot (§5).** New boxes boot with no hand steps. Configuration comes from SSM parameters, and a box takes no instance until it's enrolled. |
| +- **Ingress (§6).** The NLB uses the region's existing EIP. The platform box gets its own EIP for the relay and MX. The console tells customers to point at configured ingress IPs. |
| +- **Placement (§7).** The rules are eligibility thresholds, soft spread, then the most free memory. When nothing qualifies, the instance is unplaced and an alert fires. No automatic rebalancing. |
| +- **Rollout (§9).** Each region needs claim mode active first. The EIP cut-over has a few minutes of announced downtime per region. |
| + |
| +## 3. Prerequisites |
| + |
| +Two-box regions depend on the claim. With claim mode off, every box in a |
| +region competes for each instance through the per-host lease |
| +(`internal/activator/activator.go:1080`). The activator trusts an |
| +instance's local database copy only while a region has one box |
| +(`activator.go:2264-2272`). So: |
| + |
| +1. **Claim mode is active in the deployment.** That means `CARLOS_HOST_CLAIM`, `CARLOS_HOST_CLAIM_ORIGINS` and #250's backfill, plus the open claim asks on #236. |
| +2. **Region HA is on in the region** (`CARLOS_REGION_HA`), and relocation has been rehearsed there. |
| +3. **Every single-box assumption in §8 is closed.** |
| + |
| +No region gets a second box before all three hold. |
| + |
| +## 4. Infrastructure: `infra/modules/region-host` |
| + |
| +The module currently makes one `aws_instance` and one `aws_eip` |
| +(`main.tf:482-517`). It gains a **group mode**, chosen by a variable. |
| +The fixed-instance mode stays for the platform box and for deployments |
| +that don't want HA. |
| + |
| +**Group mode adds:** |
| + |
| +- **A launch template.** |
| + - Same AMI lookup, instance type, root volume (`delete_on_termination = true`), security group and instance profile as the fixed box. |
| + - The start-up script is §5's. |
| + - IMDSv2 required, with a hop limit of 1. |
| +- **An Auto Scaling group.** |
| + - `min = max = desired = var.box_count`, which defaults to 2. |
| + - Spread over two availability zones. |
| + - Health check type `ELB`, using the NLB target group, plus EC2 status. |
| + - A grace period long enough for §5's boot, about 10 minutes, to be set from dev measurements. |
| + - Instance refresh is not enabled. Releases reach boxes through `carlos system update`, never by replacing them. |
| + - Scale-in protection is not used, because there is no scale-in yet. |
| +- **A Network Load Balancer.** |
| + - **One subnet mapping, carrying the region's existing EIP allocation** (§9 step 2). |
| + - Cross-zone load balancing on. |
| + - Listeners: TCP 80 and TCP 443. |
| + - One target group per port, with instance targets (the client IP is preserved) and deregistration delay at 30s. |
| + - Health check: HTTP on port 80, path `/.carlos/lb-health` (§6), every 10s, 2 healthy and 2 unhealthy. |
| + - The group registers its instances itself. A fixed box, such as IE's platform box, is registered by a separate attachment. |
| +- **Alarms.** |
| + - `GroupInServiceInstances < desired` for 15 minutes: this catches a launch loop. |
| + - `UnHealthyHostCount > 0` for 10 minutes. |
| + - Both go to the module's existing SNS topic. |
| +- **Termination evidence.** The EventBridge rule and SQS queue already on main (`main.tf:738-804`) cover group instances unchanged, because they match every EC2 `terminated` event in the account and region. |
| +- **IAM.** |
| + - The box role gains `autoscaling:DescribeAutoScalingInstances` and `autoscaling:DescribeAutoScalingGroups` (read-only), for H16's membership confirmation. |
| + - The box role gains `ssm:GetParameter` and `GetParametersByPath` on exactly `/carlos/<deployment>/<region>/*` and `/carlos/<deployment>/shared/*`, with `kms:Decrypt` for that path's key only. |
| + - The existing SSM-parameter Deny (`main.tf:456-475`) is kept and narrowed so it doesn't cover these paths. |
| + |
| +**Group mode removes** the per-box `aws_eip`. The per-instance alarms |
| +(`host_auto_recover`, `host_memory_low`, `main.tf:521-581`) are bound to |
| +one InstanceId. In group mode they are replaced by the two group alarms |
| +above, plus the memory alarm reworked to use the group dimension. |
| +Auto-recovery has nothing to do once the group replaces sick boxes. |
| + |
| +**The platform box** stays in fixed mode (`aws_instance` plus its own new `aws_eip`). It is also attached to the region's NLB target groups, so that console and platform routes are reachable through the NLB (§6). |
| + |
| +## 5. Boot and enrolment |
| + |
| +The launch template's start-up script is the only way a group box is |
| +configured. It replaces `scripts/bootstrap-box.sh`, |
| +`set-alert-sns-topic.sh` and the hardening scripts, which are run by hand |
| +over SSM today. Steps run in this order. A failed step stops the script, |
| +and nothing later runs. |
| + |
| +1. **Harden.** Apply what `harden-box.sh` and `harden-host.sh` do today, including the nftables gate that keeps tenant processes away from IMDS credentials. Nothing tenant-facing starts before this. |
| +2. **Install.** |
| + - Download `s3://<deployment-bucket>/bootstrap/carlos`. |
| + - Check its sha256 against the SSM parameter `/carlos/<deployment>/shared/bootstrap-sha256`, and refuse on a mismatch. |
| + - Run `carlos bootstrap`, which installs the release that the signed channel pointer names, verified with the release key. So a new box joins on its neighbours' version, not on whatever the bootstrap object holds. |
| +3. **Configure.** Assemble `/etc/carlos/host.env` from SecureString parameters. Every value a hosting box needs today (`bootstrap-box.sh:49-57`, `README.md:339-342`) has one: |
| + - the buckets and their regions; |
| + - `CARLOS_RELEASE_PUBKEY` and `CARLOS_SECRETS_KEY`; |
| + - `CARLOS_REGION` and `CARLOS_SYSTEM_CHANNEL`; |
| + - the alert topic; |
| + - `CARLOS_HOST_CLAIM`, `CARLOS_HOST_CLAIM_ORIGINS` and `CARLOS_REGION_HA`. |
| + |
| + The file is written mode 0600, owned by root. A missing required parameter stops the boot. |
| +4. **Start the edge.** `carlos-edge` starts and serves at once. It forwards (§6) without being enrolled, so the NLB health check can pass, and the box can carry traffic for its neighbour while it waits to enrol. |
| +5. **Enrol.** |
| + - The agent publishes its signed identity record, `control/boxes/<ref>.json` (`internal/host/boxrecord.go`, already built). |
| + - Confirmation (H16): a regional worker, meaning any enrolled box in the same region, reads the record. Using the new read-only Auto Scaling describe, it confirms that the instance is a current member of the region's group, then writes the create-only confirmation `control/box-enrollment/<ref>/<attempt>/<confirmation>.json`. |
| + - In a region with no enrolled box, a box in another region confirms it (H17, S1 §5.1). |
| + - The console calls `hostactors.EnrollAutomatic` (built, with no caller yet), and the box runs `carlos system guard-arm`. |
| + - Only then is the box `active` in the roster and eligible for placement (§7). |
| + |
| +**What spec 2 builds here:** the start-up script, the SSM parameter layout and a `carlos` command that writes it from today's `host.env`, the membership-confirmation worker (S1 called this spec 4's producer), and the console's enrolment pass. |
| + |
| +**When boot fails:** |
| +- **Before step 4:** the health check never passes, and the group replaces the box after the grace period. |
| +- **A loop** (a bad bootstrap object, a missing parameter) shows as `GroupInServiceInstances < desired` for 15 minutes, and the alarm fires. |
| +- **Enrolment fails:** the box serves and forwards but is never placeable. The console lists it as pending enrolment, and the same alarm path alerts if it stays that way for 15 minutes. |
| + |
| +**The platform box** keeps its hand-managed state (keymail, woodstar, relay masters, the console database) and isn't rebuilt by this script. Its `host.env` moves to the same parameters, so a rebuild needs fewer hand steps, but it stays a fixed box. |
| + |
| +## 6. Ingress, DNS and forwarding |
| + |
| +**The load balancer health answer.** |
| +- NLB HTTP health checks send the target's address as the Host header, so the steering probe (`internal/edge/steeringprobe.go`, which answers only for its own hostname) can't serve them. |
| +- The edge gains one reserved answer: `GET /.carlos/lb-health` on port 80, from any host. It replies 200 when the edge is serving and its forwarding map has loaded at least once since start, and 503 otherwise. |
| +- It reads nothing from disk or the bucket per request, and uses the same narrow matching as the steering probe: exact path, GET or HEAD only. |
| + |
| +**The region's IP.** |
| +- The NLB takes the region's existing EIP (§9 step 2). So the carloku.com latency records (`dns-carloku.tf:44-106`), `oncarlos.com` and its wildcard, `<region>.oncarlos.com`, `platform.carlosframework.com`, and customers' A records all keep resolving to the same address. |
| +- The NLB uses one subnet, so this single address lives in one availability zone. **Losing that whole zone still takes the region's ingress down.** |
| +- A second zone means a second EIP and customers changing their DNS. That's deferred to spec 4. |
| + |
| +**The platform box's own EIP.** |
| +- The relay (3478 and 49152-65535) and keymail's MX (port 25) aren't web traffic, and the NLB carries only 80 and 443. So the platform box gets a new EIP, and `relay.eu-west-1.oncarlos.com` and the MX target move to it in flagship tofu. |
| +- Its web traffic arrives through the NLB like every other box's. |
| + |
| +**Forwarding between boxes.** |
| +- Spec 1's forwarding (the edges file's home map, `internal/edge/homemap.go`) is on main and active today. It dials the home box's published public EIP on 443 (`edge.go:2779`). |
| +- Group boxes have temporary public IPs, and traffic within a region shouldn't leave the VPC. So each edges file also publishes the box's **private IPv4** from IMDS. Forwarding to a box in the same region dials the private address, and forwarding to another region dials the public one. |
| +- The security group gains 443 from the region's VPC CIDR, which is already exported as `vpc_cidr`. |
| +- Forwarding stays one hop, as now. |
| + |
| +**Platform routes.** |
| +- The console and other unit- or socket-backed routes run only on the platform box (S1 §12: never relocated). |
| +- With the NLB, their requests reach any box, so a group box must forward them to the platform box. Today those routes have only ever been served on the box that received them. |
| +- Spec 2 makes this a requirement with its own tests: a request for `console.<domain>`, or any platform hostname, arriving at a group box is forwarded to the box that publishes the route. |
| + |
| +**What the console tells customers.** |
| +- The console lists every EIP folded from the edges files as A-record targets, plus `PLATFORM_EDGE_IPS` (`internal/console/view.go:177-237`, `server.go:1120-1147`). With group boxes, that list would include temporary addresses. |
| +- It changes to a configured **region ingress list**: each region's NLB EIP, from a deployment setting written by tofu. |
| +- The domain check's "pointing" verdict, from the `domains reconcile` duty, uses the same list. |
| +- The new console text goes through Paul's copy review. |
| + |
| +**Unchanged:** |
| +- **Certificates.** A challenge landing on any box is answered by reading through to `control/certs/` (`internal/edge/certsync.go:1594-1622`). |
| +- **The housekeeper election** (`internal/host/housekeeper.go`). If the housekeeper's box dies, issuance pauses for up to the lease TTL, 15 minutes, while serving continues. |
| +- **Per-box latency steering** stays off. It's spec 4's. |
| + |
| +## 7. Placement policy (H4) |
| + |
| +Spec 1's `boxchoice.Choose` (`internal/boxchoice/boxchoice.go:146-182`) |
| +prefers a box running none of the app, then the fewest hosts, then the |
| +P5-D2 hash. Its candidates already carry `MemFreeMiB`, `DiskFreeMiB` and |
| +`Awake`, unused (S1 §5, "until spec 2"). Spec 2 sets the policy. |
| + |
| +**Eligibility** is spec 1's `Eligible` (enrolled, `active`, in the instance's region, not fenced), and in addition: |
| + |
| +- **Not the platform box.** The roster gains a box role, `platform`, which is never eligible. The IE platform box is enrolled with it. |
| +- **Memory.** `mem_free_mib` is at least 15% of the box's total memory, the same line as the memory alarm. Boxes publish total memory alongside headroom. |
| +- **Disk.** `disk_free_mib` is at least 10% of the data volume's size. |
| +- **Fresh headroom.** A box whose edges file is older than 5 minutes is not eligible, so stale headroom is never trusted. |
| + |
| +**Choice:** |
| +1. Prefer a box running no instance of the same app. This is never a reason to refuse. |
| +2. Then the most `mem_free_mib`. |
| +3. Then the P5-D2 hash. |
| + |
| +The choice is made once and recorded in the create intent, as now (H12). Recovery never re-chooses. |
| + |
| +**Accepted limit.** `mem_free_mib` is memory free now (MemAvailable), so a box full of sleeping instances looks roomy until they wake. Counting each instance's memory limit would be more accurate, but limits aren't set everywhere. Spec 2 uses live memory, and the memory alarm and spec 4's scale-out cover overcommitment. |
| + |
| +**Unplaced.** |
| +- When no box is eligible, the create intent is recorded as `unplaced`. No executor is named. |
| +- The console shows the instance waiting for capacity, and an operator alert fires once per instance. |
| +- The placement pass retries every minute, and immediately when a box enrols or publishes new headroom. |
| +- An unplaced instance has no box, so it serves 503 with Retry-After, as an instance whose executor is unreachable does today. |
| +- When placed, it proceeds exactly as a fresh create. |
| + |
| +**Rebalancing:** none automatic. An operator moves an instance between boxes in a region with spec 1's relocation. Automatic rebalancing is spec 4's. |
| + |
| +**Failover** (spec 3) uses this same choice for each displaced instance. |
| + |
| +## 8. Single-box assumptions to close |
| + |
| +Each is closed by a code change, or by showing it's safe under claim mode, before §3's third prerequisite holds. |
| + |
| +| Where | Assumption | Plan | |
| +|---|---|---| |
| +| `internal/console/apiinstancemove.go:614-618`, `boxselect.go:114-118`, `internal/host/movesphase1.go:517` | The move door refuses a region with more than one box | Moves' code: ask on #236 for multi-box support, choosing the destination box with §7's choice | |
| +| `internal/console/seedfloor.go:25-29, 88` | Seed import refuses a region with more than one box | Ask the seeded-instances owner: import onto §7's choice | |
| +| `internal/hostfence/onebox.go:9-30` | `CheckOneBoxRegion` | Read its callers. Each caller either requires one box, and is refused in two-box regions by design, or gets the region's box set | |
| +| `internal/activator/activator.go:2264-2272` | The local database copy is a warm cache only while a region has one box | Show that claim mode makes the executor the only writer (claim §7.1), so the cache is safe, or disable the cache in multi-box regions | |
| +| `internal/console/apiinstances.go:523-527` | Hosts are unique to one box | Holds under the claim (one executor). Add a test | |
| +| `internal/console/view.go:177-237`, `server.go:1120-1147` | EIP list as A-record targets | §6: the region ingress list | |
| +| `cmd/console/main.go:1390-1400` | `consoleBoxLabel`: exactly one box | Holds: the console runs on the platform box only. Document it | |
| +| Flagship tofu `outputs.tf:39-68`, `dns*.tf`, `keymail.tf`, `mail.tf`, `scanner.tf` | One instance and EIP per region | Rewrite for the group, the NLB EIP and the platform box | |
| +| Flagship scripts (`bootstrap-box.sh`, `update-platform-binary.sh`, `harden-*.sh`, `roll-console.sh`, `verify-console.sh`, `rollout-status-flagship.sh`, the woodstar and relay scripts) | One instance ID, or a hard-coded one | Group boxes: replaced by §5 and `carlos system update`. Platform box: keep the scripts, pointed at the platform box | |
| + |
| +## 9. Rollout |
| + |
| +**Order:** dev, then UK, US, AU and IE, one region at a time. Each region needs §3 to hold first. |
| + |
| +**Per region:** |
| +1. **Build.** Write the SSM parameters, then apply the module in group mode with the NLB created **without** the EIP, on a temporary address. The two group boxes boot, enrol and serve through forwarding. The old fixed box keeps the EIP and all traffic. |
| +2. **Cut over the EIP.** |
| + - An NLB takes an EIP only when it's created. So the temporary NLB is destroyed, the EIP is disassociated from the old box, and the final NLB is created with the EIP. Its target groups hold the two group boxes and the old box. |
| + - **The region's ingress is down for a few minutes.** That's scheduled at a quiet hour and announced. The window is measured on dev first. |
| + - The old box keeps serving its instances throughout, reached through the NLB like any other target. |
| +3. **Migrate.** Relocate every instance on the old box to §7's choice, one at a time, with spec 1's relocation. |
| +4. **Retire.** |
| + - **UK, US and AU:** deregister the old box and terminate it. |
| + - **IE:** the old box stays as the platform box. It's enrolled with the `platform` role, gets its new EIP, and the relay and MX records move to it. |
| + |
| +**Rollback.** |
| +- Before step 2, destroying the group and NLB returns the region to today's state. |
| +- After step 2, moving the EIP back to the old box reverses ingress. Instances already relocated in step 3 can be relocated back. |
| + |
| +**Cost.** Each of UK, US and AU goes from one box to two, plus an NLB (about $20 a month before traffic). IE goes from one box to three: the platform box plus two group boxes. Dev matches whichever layout it rehearses. |
| + |
| +## 10. Failure modes |
| + |
| +| What happens | Result | |
| +|---|---| |
| +| A group box hangs or fails its health check | The NLB stops sending it traffic within about 20s, and the group terminates it after its health check fails. Its instances are down until spec 3 relocates them. A replacement boots, enrols and becomes placeable with no hand steps. | |
| +| A group box is terminated by AWS | As above, without the health-check delay. | |
| +| The platform box dies | The console and platform services are down. EC2 auto-recovery covers hardware faults, and otherwise it's an operator job (S1 §12). Customer instances are unaffected once migration is done. | |
| +| Both group boxes in a region fail | The region's instances are down. The group replaces both, and the first replacement is confirmed from another region (H17). | |
| +| The NLB's availability zone fails | The region's ingress is down until the zone recovers (§6). | |
| +| A replacement can't boot | It's never healthy, the group keeps replacing it, and the in-service alarm fires at 15 minutes. | |
| +| A box boots but can't enrol | It serves and forwards, takes no new instances, and is listed as pending. The alarm fires at 15 minutes. | |
| +| No box has headroom | New instances are unplaced, and an alert fires (§7). | |
| +| The housekeeper's box dies | Certificate issuance pauses for up to 15 minutes, and serving continues. | |
| + |
| +## 11. Testing |
| + |
| +- **Tofu.** The module's tests cover group mode: the launch template, group, NLB, target groups, the health check path, IAM scoped to the parameter path, the alarms, and fixed-mode attachment to the target groups. Fixed mode stays identical when group mode is off. |
| +- **Boot.** The start-up script is tested on dev from a clean launch, and every failure path is checked: bad sha256, missing parameter, failed hardening. Each must leave the box unhealthy. |
| +- **Enrolment.** Unit tests for the membership-confirmation worker against a fake Auto Scaling describe, covering a member, a non-member, a terminated instance, and confirmation from another region (H17). On dev, a real launch enrols with no operator. |
| +- **Health answer.** Edge tests for `/.carlos/lb-health`: any host, exact path, GET and HEAD only, 503 before the forwarding map loads. |
| +- **Forwarding.** Private-address forwarding within a region and public-address forwarding across regions. Platform routes forwarded from a group box to the platform box. |
| +- **Placement.** Unit tests for each eligibility rule, the spread preference, the headroom ordering, the stale-headroom cut-off, the platform role, and unplaced with its retry triggers. |
| +- **Rehearsal on dev:** |
| + - the full §9 sequence, with the EIP cut-over window measured; |
| + - one group box terminated by hand, with replacement and re-enrolment observed; |
| + - a box made unhealthy, observed being terminated. |
| +- **Failover drills** belong to spec 3. |
| + |
| +## 12. Out of scope |
| + |
| +- **Spec 3:** relocating a terminated box's instances (the placer loop), waking always-on and scheduled instances, the failover gap target. |
| +- **Spec 4:** |
| + - automatic scale-out and scale-in, and rebalancing; |
| + - a second availability zone for ingress (a second EIP); |
| + - latency steering; |
| + - H5's per-region minimum as anything other than a fixed group size; |
| + - the Auto Scaling activity-history evidence source (S1 A2-4). |
| +- **Not planned:** relocating the console or platform services (S1 §12). The platform box is fixed, and its HA remains the HA ladder's. |
| + |
| +## 13. Open questions for review |
| + |
| +1. **Fresh headroom.** Is 5 minutes the right cut-off, given edges files republish every 30s? |
| +2. **Temporary NLB address (§9 step 1).** Is there a cheaper rehearsal of the NLB before the cut-over than a throwaway NLB? |
| +3. **SSM parameter layout and key.** Should `CARLOS_SECRETS_KEY` share the region path's KMS key, or have its own key with a narrower policy? |
diff --git a/docs/superpowers/specs/CODEX-REVIEW-BRIEF-REGION-HA-SPEC2.md b/docs/superpowers/specs/CODEX-REVIEW-BRIEF-REGION-HA-SPEC2.md| new file mode 100644 |
| index 000000000..891df5788 |
| --- /dev/null |
| +++ b/docs/superpowers/specs/CODEX-REVIEW-BRIEF-REGION-HA-SPEC2.md |
| @@ -0,0 +1,29 @@ |
| +# Astra: review region HA spec 2 (two-box regions), rev 1 |
| + |
| +Spec: `docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md` on branch `design-region-ha-two-box`. It builds on spec 1 (`docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md`), whose code is on main and dormant. |
| + |
| +Paul's rulings H1 to H24, and the section choices in §2, are settled. Don't reopen them. Judge whether the design achieves them safely, against the code on main. |
| + |
| +**Focus areas:** |
| +1. **Safety under the claim.** |
| + - Can two boxes in a region ever run the same instance, or let a stale local copy win? |
| + - Is §8's list of single-box assumptions complete? Grep main for others. |
| +2. **Boot and enrolment (§5).** |
| + - Can a box take instances before it's hardened, configured or enrolled? |
| + - Is H16's confirmation sound with the new read-only Auto Scaling describe? |
| + - Is the SSM and KMS scope tight enough, given the `host_buckets` statements and the existing SSM Deny in `infra/modules/region-host/main.tf`? |
| +3. **Ingress (§6).** |
| + - Is the NLB health answer correct and narrow? |
| + - Do private-address forwarding and platform-route forwarding work with `internal/edge/homemap.go` and `edge.go`'s single-hop backhaul? |
| + - Does anything break when the region's EIP moves to an NLB: GeoIP, rate limits, the IMDS `public-ipv4` that edges files publish, certificate issuance, or the domain check? |
| +4. **Placement (§7).** |
| + - Are the eligibility rules and the unplaced state consistent with `internal/boxchoice` and `internal/hostops/executor.go`? |
| + - Is `unplaced` representable in the claim's create intent without breaking recovery? |
| +5. **Rollout (§9).** |
| + - Is the EIP cut-over sequence correct for AWS: an NLB subnet-mapping EIP can be set only at creation? |
| + - Is the rollback real? |
| + - Does the old box keep serving correctly as an NLB target? |
| +6. **Failure modes (§10).** Is anything missing or wrong? |
| +7. **The open questions in §13.** |
| + |
| +Give findings P1/P2/P3 with file:line, each with a concrete failure scenario. End with a verdict: ready for a plan, or not. |