amadan
region HA spec 2, rev 15: H27 - no pending creates; a refused create writes nothing and alerts; certificate readiness is the mirror's first load
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+16
−19
docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md+16 −19
diff --git a/docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md b/docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md| index 36b5601be..e694e0514 100644 |
| --- a/docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md |
| +++ b/docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md |
| @@ -1,6 +1,8 @@ |
| # Region HA, spec 2: two-box regions |
| -Design, 2026-10-16, **rev 13**. Approved by Paul on 2026-10-16, subject to this revision. Nothing here is built. |
| +Design, 2026-10-16, **rev 15**. Approved by Paul on 2026-10-16. Nothing here is built. |
| + |
| +**Rev 15** records Paul's ruling **H27**: no pending creates. When no box qualifies, the create door refuses with a clear message and alerts the operator. It also settles the certificate-readiness wording in §6. Plans: 2a (core), 2b (joining and enrolment), 2c (infrastructure and boot), 2d (single-box fixes). |
| **Rev 13** adds §25, the provider boundary, at Paul's request. Spec 2 must not make the platform depend on AWS: a deployment stays "one object-store bucket + N hosts running the `carlos` binary + DNS" (README), and its AWS-specific parts are adapters. |
| @@ -71,13 +73,14 @@ from 2026-10-15: |
| | H24 | **IE's current box becomes the platform box.** It's fixed, outside the group, and keeps the console, keymail, woodstar, the relay and the scanner grants. It takes no new customer instances. Its existing ones leave with spec 4's drain (H25). | |
| | H25 | **(2026-10-16) Spec 2 doesn't empty the old boxes.** Each region's old fixed box keeps the instances it has and takes no new ones. Those instances keep today's single-box behaviour until spec 4's drain moves them, or until spec 3's failover moves them if AWS terminates the box. Each region runs three boxes until then. | |
| | H26 | **(2026-10-16) Multi-box regions stay AWS-only until a provider-neutral box identity exists** (#299). Box identity, enrolment, roster validation and placement eligibility rest on AWS's signed instance document on main today. Spec 2 doesn't make that worse: its own new AWS parts are adapters (§25). | |
| +| H27 | **(2026-10-16) No pending creates; this supersedes H4's "waits as unplaced".** When no box qualifies, every door that chooses a box refuses with HTTP 503 and a clear message: no capacity in that region right now, try again shortly. The operator is alerted, at most once per region per hour. No create without an executor is ever written to the claim. Sidecars run on their instance's box and choose nothing, so they're unaffected. | |
| Design choices Paul approved in section review on the same day, recorded |
| here so they aren't reopened: |
| - **Boot (§5).** New boxes boot with no hand steps. Configuration comes from SSM parameters, and a box takes no instance until it's enrolled. |
| - **Ingress (§6).** The NLB uses the region's existing EIP. The platform box gets its own EIP for the relay and MX. The console tells customers to point at configured ingress IPs. |
| -- **Placement (§7).** The rules are eligibility thresholds, soft spread, then the most free memory. When nothing qualifies, the instance is unplaced and an alert fires. No automatic rebalancing. |
| +- **Placement (§7).** The rules are eligibility thresholds, soft spread, then the most free memory. When nothing qualifies, the request is refused and an alert fires (H27, which replaced the original "unplaced and alert"). No automatic rebalancing. |
| - **Rollout (§9).** Each region needs claim mode active first. The EIP cut-over has a few minutes of announced downtime per region. |
| ## 3. Prerequisites |
| @@ -215,7 +218,7 @@ Anything else is joining mode. Joining resumes from whatever it finds: no invita |
| - It replies **200 only when every condition holds**, and 503 otherwise. Every condition is this box's own, so an outage of a shared dependency like the deployment bucket never turns a working box unhealthy (A2-3). |
| - The agent is in **hosting mode** (§5): enrolled, guard armed. |
| - **Forwarding is armed:** the forwarding map and the mesh key have each loaded successfully **at least once** since the edge started (A1-10). This is latched, not a freshness check. After that, a failed refresh keeps the cached map and key, as today (`internal/edge/homemap.go:235`), and doesn't affect health. |
| - - The TLS certificate cache has loaded, with the default certificate available. |
| + - The TLS certificate cache has loaded: the certificate mirror has completed its first load. Certificates are per host, so there is no single default certificate to check. |
| - The edge's request loop and the activator's hosting loop have each reported a heartbeat within the last 60 seconds. These are local liveness: a hung process stops its own heartbeat. |
| - The forwarding map and key refresh worker (`internal/edge/homemap.go:838`) has **completed an attempt**, whether it succeeded or failed, within the last 5 minutes (A3-3). Completion counts as progress, so a bucket outage, where every attempt fails quickly, keeps it satisfied. A worker stuck inside one attempt doesn't. The 5 minutes covers the refresh interval plus the existing two-minute request timeout. |
| - **Why freshness was dropped.** Rev 2 required refreshes within 2 minutes. That made a bucket outage over two minutes mark every box unhealthy, and the group would terminate working boxes whose instances spec 2 can't move yet. A shared-dependency outage is an alert, from the existing bucket and housekeeper alerting, never a termination. |
| @@ -288,20 +291,7 @@ The choice is made once and recorded in the create intent, as now (H12). Recover |
| **Accepted limit.** `mem_free_mib` is memory free now (MemAvailable), so a box full of sleeping instances looks roomy until they wake. Counting each instance's memory limit would be more accurate, but limits aren't set everywhere. Spec 2 uses live memory, and the memory alarm and spec 4's scale-out cover overcommitment. |
| -**Unplaced** (H4). The claim is never asked to hold a create with no executor (A1-7). `BeginCreateOn` refuses an empty executor (`internal/hostclaim/create.go:109`), and recovery needs `selected_executor` and `executor_enrollment` (`internal/hostops/production_create.go:257`). |
| -- **The request waits in the console.** When no box is eligible, the console records a **pending create** in its own database: the requested host, app, region, plan and requesting member. It starts no claim operation. |
| -- **The customer and operator see it.** The console shows the instance waiting for capacity, and an operator alert fires once per pending create. |
| -- **It is retried.** The placement pass runs every minute, and immediately when a box enrols or publishes new headroom. |
| -- **The handoff is recoverable, not atomic** (A2-5). The claim write is in the bucket and the pending row is in the console database, so no single transaction covers both. Instead: |
| - 1. When a pending create finds an eligible box, the console first commits the row as **`starting`**, with the chosen executor and a **stable operation identity** generated then. |
| - 2. Then it runs today's create path with that identity. The create path gains an option to take a caller-supplied operation ID instead of generating one (`internal/console/claim_create.go:50`). |
| - 3. On success, the row is deleted. |
| - 4. After a restart, or on any error, a `starting` row is **reconciled** before anything else touches it: the console reads the claim for that operation ID. If the create exists, the row is deleted. If it provably doesn't, the start is retried with the same identity, and the create is idempotent on it. |
| -- **Only one placer.** The console is the only placer. The database row lock lets one pass at a time move a row to `starting`. |
| -- **The hostname is reserved** from the moment the pending row exists. The immediate create door and the pending path check and take the same reservation, so a hostname can't be pending and created twice. |
| -- **It can be cancelled while it's still `pending`.** That deletes the row and its reservation. A `starting` row can't be cancelled until it's reconciled. If its create exists, the member cancels the instance through the ordinary delete. |
| -- **The hostname answers with the edge's unknown-host response.** The host isn't in any edges file, so it isn't routed. |
| -- **Copy.** The new console text and the alert go through Paul's copy review. |
| +**No capacity** (H27). When no box is eligible, the door refuses the request with HTTP 503 and a clear message that there's no capacity in that region right now, so the caller should try again shortly. The message goes through Paul's copy review. An operator alert fires, at most once per region per hour, so a retry loop doesn't flood it. Nothing is written to the claim, so there is nothing to recover or cancel. The eligible-capacity metric (§4) is published by a small console pass every minute. |
| **Old boxes** (H25). Each region's old fixed box takes the roster role **`legacy`**; IE's takes **`platform`**. Both roles are never eligible for a new placement, a relocation target or a move target. That's a static exclusion, read from the roster by every chooser (`boxchoice.Eligible`), and it needs no settlement: |
| - the box's existing instances stay where they are; |
| @@ -403,7 +393,7 @@ Each is closed by a code change, or by showing it's safe under claim mode, befor |
| - **Enrolment.** Unit tests for the membership-confirmation worker against a fake Auto Scaling describe, covering a member, a non-member, a terminated instance, and confirmation from another region (H17). Tests that joining mode never publishes an edges file or admits an instance. A test that the full sequence (confirm, invite, guard-ready, enrol, hosting) is satisfied against `membership.go`'s rule when a floor is published. On dev, a real launch enrols with no operator. |
| - **Install.** A fresh box installs the channel release into both protected slots, never the bootstrap binary, refuses a release below the host floor, and then passes `guard-arm`. |
| - **Mode selection.** A restart at each joining state (no invitation, invited, guard installed, guard-ready reported) resumes joining. Only an active roster entry plus an armed guard starts hosting. |
| -- **Unplaced.** Pending create, retry on enrolment and on headroom, cancellation while pending, the hostname reservation shared with the immediate door, and exactly one claim create per pending row. That includes a crash after the claim write and before the row delete, reconciled by operation ID. |
| +- **No capacity.** A create with every box ineligible is refused with 503, writes nothing to the claim, and alerts at most once per region per hour. |
| - **Health answer.** Edge tests for `/.carlos/lb-health`: any host, exact path, GET and HEAD only, and 503 for each failing condition (joining mode, forwarding never armed, no certificate cache, a stalled heartbeat). A test that an unreachable bucket after arming leaves the answer at 200, because failed refresh attempts still count as progress, and that a refresh worker blocked inside one attempt turns it 503. |
| - **Forwarding.** |
| - Private-address forwarding within a region, and own-public-address forwarding between regions. |
| @@ -411,7 +401,7 @@ Each is closed by a code change, or by showing it's safe under claim mode, befor |
| - The publisher publishes only what IMDS reports, and publishes no public address on a blank read. |
| - A forwarded request is still never forwarded again. |
| - Platform routes forwarded from a group box to the platform box. |
| -- **Placement.** Unit tests for each eligibility rule, the spread preference, the headroom ordering, the stale-headroom cut-off, the `platform` and `legacy` roles, and unplaced with its retry triggers. |
| +- **Placement.** Unit tests for each eligibility rule, the spread preference, the headroom ordering, the stale-headroom cut-off, and the `platform` and `legacy` roles. |
| - **Rehearsal on dev:** |
| - the full §9 sequence, with the cut-over window measured, the zone-B rehearsal, and both rollback sequences run once. Rollback leaves the group in service with its instances, including one that was never woken; |
| - outbound moves from a `legacy` box and from the `platform` box; |
| @@ -586,3 +576,10 @@ A deployment without AWS runs one box per region with claim mode off, unaffected |
| | A13-1 P1: operator approval doesn't remove the AWS identity requirement | Accepted as a fact about main. Under H26, multi-box stays AWS-only until #299, and §25 says so | |
| | A13-2 P2: placement eligibility excludes non-AWS hosts | Accepted: spec 2's policy runs over identity-filtered candidates, so #299 changes only the filter. The import test doesn't claim to cover this (§25) | |
| | A13-3 P2: the marker client ignores `CARLOS_S3_ENDPOINT` | Accepted: a bug on main, filed as #298 for the claim's code. Spec 2's own clients use `s3compat.Options()`, with a call-site test (§25) | |
| + |
| +## 27. H27 (rev 15) |
| + |
| +| Item | Disposition | |
| +|---|---| |
| +| A1-7, A2-5 (the unplaced claim contract, the pending-create handoff) | Superseded by H27: no pending state exists, and a refused create writes nothing | |
| +| §7's pending creates and the placement retry pass | Removed. The eligible-capacity metric is published by its own console pass | |