| index ffd35ae81..047277873 100644 |
| --- a/docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md |
| +++ b/docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md |
| @@ -1,6 +1,14 @@ |
| # Region HA, spec 2: two-box regions |
| |
| -Design, 2026-10-15, **rev 2**. DRAFT. Nothing here is built. |
| +Design, 2026-10-15, **rev 3**. DRAFT. Nothing here is built. |
| + |
| +**Rev 3** answers Astra's round 2 (`CODEX-FINDINGS-REGION-HA-SPEC2-2.md`, |
| +cited **A2-n**): 3 P1 and 6 P2, all accepted, plus the partial |
| +dispositions of round 1. The largest change is to forwarding (§6, A2-7, |
| +A2-9). Forwarding between regions now targets the destination region's |
| +ingress address, and the box there forwards once more over the private |
| +network. No box ever needs a public address of its own. §15 maps each |
| +finding. |
| |
| **Rev 2** answers Astra's round 1 (`CODEX-FINDINGS-REGION-HA-SPEC2-1.md`, |
| cited **A1-n**): 4 P1 and 9 P2, all accepted. §14 maps each. |
| @@ -78,8 +86,8 @@ that don't want HA. |
| - **An Auto Scaling group.** |
| - `min = max = desired = var.box_count`, which defaults to 2. |
| - Spread over two availability zones. |
| - - Health check type `ELB`, using the NLB target group, plus EC2 status. |
| - - A grace period long enough for §5's whole boot, enrolment included: about 20 minutes, to be set from dev measurements. The health answer (§6) stays 503 until the box is fully ready, so the grace period must cover enrolment, not just start-up. |
| + - Health check type `ELB`, using the NLB target group, plus EC2 status. The health answer (§6) measures only this box's own liveness, never a shared dependency, so a deployment-bucket outage can't mark healthy boxes for termination (A2-3). |
| + - A grace period long enough for §5's whole boot, enrolment included: about 20 minutes, to be set from dev measurements. The health answer (§6) stays 503 until the box reaches hosting mode, so the grace period must cover enrolment, not just start-up. |
| - Instance refresh is not enabled. Releases reach boxes through `carlos system update`, never by replacing them. |
| - **The `AZRebalance` process is suspended** (A1-3). Rebalancing between zones launches and terminates boxes without changing desired capacity, and could terminate a healthy box that's hosting instances. With fixed size and no failover until spec 3, the only terminations are of unhealthy boxes. The `Launch`, `Terminate`, `HealthCheck` and `ReplaceUnhealthy` processes stay on. |
| - **A Network Load Balancer.** |
| @@ -94,6 +102,8 @@ that don't want HA. |
| - **Alarms.** All go to the module's existing SNS topic. |
| - `UnHealthyHostCount > 0` for 10 minutes. |
| - **Eligible capacity** (A1-11): the console's placement pass publishes `Carlos/Region EligibleBoxes` per region (enrolled, active, placeable boxes, §7). An alarm fires when it is below `box_count` for 25 minutes, longer than the grace period. |
| + - The platform box's role gains `cloudwatch:PutMetricData` conditioned on `cloudwatch:namespace = Carlos/Region`. The current grant covers only `Carlos/Host` (`main.tf:418`, A2-8). |
| + - Missing data is treated as **breaching**: if the console stops publishing, the alarm fires, which is also worth an alert. |
| - This catches what the group's own metrics can't: a box that's in service and passing health checks but never enrolled, and a launch loop on a group whose target group has never had a healthy target (in that state, AWS doesn't replace ELB-unhealthy instances). |
| - The group's `GroupInServiceInstances < desired` is kept as a second signal. |
| - **Termination evidence.** The EventBridge rule and SQS queue already on main (`main.tf:738-804`) cover group instances unchanged, because they match every EC2 `terminated` event in the account and region. |
| @@ -118,7 +128,14 @@ configured. It replaces `scripts/bootstrap-box.sh`, |
| `set-alert-sns-topic.sh` and the hardening scripts, which are run by hand |
| over SSM today. A failed step stops the script, and nothing later runs. |
| |
| -A group box boots in two modes. **Joining** publishes evidence and installs the guard, but hosts nothing and publishes no edges file. **Hosting** is today's agent under the claim, entered only once the box is enrolled. Today the claim-mode agent refuses to start without `/etc/carlos/host-actor.json` and a matching roster entry (`internal/host/claimproduction.go:87`). And a box added after the floor is published needs its guard-ready report first (`internal/hostactors/membership.go:40`). So a joining mode is required (A1-2). |
| +A group box boots in two modes. **Joining** publishes evidence and installs the guard, but hosts nothing and publishes no edges file. **Hosting** is today's agent under the claim, entered only once the box is enrolled. |
| + |
| +**Mode selection** happens at every agent start, never from one file's presence (A2-4). The agent is in hosting mode only when all three hold: |
| +- `host-actor.json` exists; |
| +- the roster has a matching, active entry for this box; |
| +- the guard is armed for that enrolment. |
| + |
| +Anything else is joining mode. Joining resumes from whatever it finds: no invitation yet, invited, guard installed, or guard-ready reported. A restart at any point repeats the next step. Today the claim-mode agent refuses to start without `/etc/carlos/host-actor.json` and a matching roster entry (`internal/host/claimproduction.go:87`). And a box added after the floor is published needs its guard-ready report first (`internal/hostactors/membership.go:40`). So a joining mode is required (A1-2). |
| |
| 1. **Harden.** Apply what `harden-box.sh` and `harden-host.sh` do today, including the nftables gate that keeps tenant processes away from IMDS credentials. Nothing tenant-facing starts before this. |
| 2. **Configure.** Assemble `/etc/carlos/host.env` from SecureString parameters. Every value a hosting box needs today (`bootstrap-box.sh:49-57`, `README.md:339-342`) has one: |
| @@ -132,19 +149,24 @@ A group box boots in two modes. **Joining** publishes evidence and installs the |
| 3. **Install the signed channel release** (A1-6). |
| - Download `s3://<deployment-bucket>/bootstrap/carlos` and check its sha256 against the SSM parameter `/carlos/<deployment>/shared/bootstrap-sha256`, refusing on a mismatch. |
| - Run `carlos bootstrap` for users, directories and units (`internal/host/bootstrap.go`). It doesn't install a release. |
| - - Then install the release named by the signed channel pointer, verified with `CARLOS_RELEASE_PUBKEY`, through the same staging and verification path `carlos system update` uses (`internal/host/systemleg.go`, `internal/release/systempointer.go`). The bootstrap binary is kept as the previous binary. |
| + - Then install the release named by the signed channel pointer, verified with `CARLOS_RELEASE_PUBKEY`, through the same staging and verification path `carlos system update` uses (`internal/host/systemleg.go`, `internal/release/systempointer.go`). |
| + - **Both protected slots, the current binary and `carlos.prev`, are the verified channel artifact** (A2-2). `guard-arm` checks both installed binaries against the supplied identity and the exact artifact hash (`internal/host/guardarm.go:75`, `internal/hostfloor/floor.go:138`). A bootstrap binary of a different build in either slot would make the guard refuse. The bootstrap binary is deleted once the install verifies. |
| - The result: the box runs its neighbours' release, never the bootstrap object's. The box refuses to continue if the release is below the published host floor. |
| - The plan names the exact command. If no existing command does exactly this, spec 2 adds `carlos system install-channel`. |
| 4. **Join.** Start the agent in joining mode, a new mode, selected because no `host-actor.json` exists yet. It: |
| - publishes the signed identity record, `control/boxes/<ref>.json` (`internal/host/boxrecord.go`, already built); |
| - does **not** publish an edges file (A1-5). An edges file naming the box would make it referenced (`internal/hostactors/references.go:63`), and automatic enrolment refuses a referenced box (`enroll.go:37`); |
| - does not admit or serve any instance. The edge runs, but its health answer stays 503 (§6), so the NLB sends it nothing. |
| -5. **Enrol**, in this order: |
| - 1. **Confirm membership (H16).** A regional worker, meaning any hosting box in the same region, reads the identity record. With the new read-only Auto Scaling describe, it confirms the instance is a current member of the region's group, then writes the create-only confirmation `control/box-enrollment/<ref>/<attempt>/<confirmation>.json`. In a region with no hosting box, a box in another region confirms it (H17, S1 §5.1). |
| - 2. **Invite.** The console's enrolment pass sees the confirmation and issues a signed invitation (`hostactors.Invite`, built, with no caller yet). |
| - 3. **Install the guard.** The joining agent fetches the invitation, writes `host-actor.json`, runs `carlos system guard-arm` with the identity and previous-binary material from step 3, and publishes its guard-ready report. |
| - 4. **Enrol.** The console sees the guard-ready report and calls `hostactors.EnrollAutomatic` (built, with no caller yet). The membership rule at `membership.go:40` is then satisfied. |
| - 5. **Restart in hosting mode.** The agent finds itself enrolled in the roster and restarts as the claim-mode agent. Only now does it publish edges files and answer health 200 (§6), and only then is it eligible for placement (§7). |
| +5. **Enrol** (A2-1). `EnrollAutomatic` needs a matching `Roster.EnrollmentAttempts` entry, which only `BeginEnrollmentAttempt` creates. It also needs confirmation evidence **no older than one minute** (`internal/hostactors/enrollment_confirmation.go:66, 111`). So membership is confirmed twice, once to decide whether to invite and once, fresh, to enrol: |
| + 1. **First attempt.** The console's enrolment pass sees the identity record and calls `BeginEnrollmentAttempt`. A regional worker confirms membership for that attempt (H16): |
| + - any hosting box in the same region reads the identity record; |
| + - using the new read-only Auto Scaling describe, it confirms the instance is a current member of the region's group; |
| + - it writes the create-only confirmation `control/box-enrollment/<ref>/<attempt>/<confirmation>.json`; |
| + - in a region with no hosting box, a box in another region confirms it (H17, S1 §5.1). |
| + 2. **Invite.** On that confirmation, the console issues a signed invitation (`hostactors.Invite`, built, with no caller yet). The invitation is stable: later attempts reuse it. |
| + 3. **Install the guard.** The joining agent fetches the invitation, writes `host-actor.json`, runs `carlos system guard-arm` with the identity and binaries from step 3, and publishes its guard-ready report. |
| + 4. **Fresh attempt, then enrol.** When the guard-ready report appears, the console begins a fresh attempt, the worker confirms it, and the console calls `hostactors.EnrollAutomatic` (built, with no caller yet) within the one-minute window. If the window lapses (a slow worker, a console restart), the pass simply begins another attempt. Attempts are cheap and the invitation is unchanged. With the guard-ready report present, `membership.go:40`'s rule is satisfied. |
| + 5. **Hosting mode.** The agent's next start check finds the roster entry and the armed guard, and it restarts as the claim-mode agent. Only now does it publish edges files and answer health 200 (§6), and only then is it eligible for placement (§7). |
| |
| **What spec 2 builds here:** |
| - the start-up script, and the SSM parameter layout with a `carlos` command that writes it from today's `host.env`; |
| @@ -155,8 +177,8 @@ A group box boots in two modes. **Joining** publishes evidence and installs the |
| - the eligible-capacity metric. |
| |
| **When boot fails:** |
| -- **Any step:** the health answer never reaches 200, and the group replaces the box when the grace period ends. |
| -- **A loop** (a bad bootstrap object, a missing parameter, enrolment that never completes) shows as eligible capacity below `box_count` for 25 minutes, and the alarm fires (§4). The first boot of a new group is covered too, before its target group has ever had a healthy target. |
| +- **Any step:** the health answer never reaches 200. The group replaces the box when the grace period ends, **except** while the group's target group has never had a healthy target: AWS doesn't replace ELB-unhealthy instances then (A1-11). That case happens only when a region is first built. |
| +- **A loop** (a bad bootstrap object, a missing parameter, enrolment that never completes) shows as eligible capacity below `box_count` for 25 minutes, and the alarm fires (§4). That alarm also covers the first-build case, where the group won't replace on its own. |
| |
| **The platform box** keeps its hand-managed state (keymail, woodstar, relay masters, the console database) and isn't rebuilt by this script. Its `host.env` moves to the same parameters, so a rebuild needs fewer hand steps, but it stays a fixed box. It is enrolled once by an operator, with the `platform` role (§7). |
| |
| @@ -165,34 +187,39 @@ A group box boots in two modes. **Joining** publishes evidence and installs the |
| **The load balancer health answer.** |
| - NLB HTTP health checks send the target's address as the Host header, so the steering probe (`internal/edge/steeringprobe.go`, which answers only for its own hostname) can't serve them. |
| - The edge gains one reserved answer: `GET /.carlos/lb-health` on port 80, from any host. It uses the same narrow matching as the steering probe: exact path, GET or HEAD only. |
| -- It replies **200 only when every condition holds**, and 503 otherwise (A1-10): |
| - - the agent is in **hosting mode** (§5): enrolled, guard armed; |
| - - the forwarding map has refreshed successfully within the last 2 minutes; |
| - - the forwarding mesh key is present and was refreshed within its own refresh window (`internal/edge/homemap.go:259`). Forwarding refuses without it; |
| - - the TLS certificate cache has loaded, with the default certificate available; |
| - - the activator's hosting loop has reported a heartbeat within the last 60 seconds. |
| +- It replies **200 only when every condition holds**, and 503 otherwise. Every condition is this box's own, so an outage of a shared dependency like the deployment bucket never turns a working box unhealthy (A2-3). |
| + - The agent is in **hosting mode** (§5): enrolled, guard armed. |
| + - **Forwarding is armed:** the forwarding map and the mesh key have each loaded successfully **at least once** since the edge started (A1-10). This is latched, not a freshness check. After that, a failed refresh keeps the cached map and key, as today (`internal/edge/homemap.go:235`), and doesn't affect health. |
| + - The TLS certificate cache has loaded, with the default certificate available. |
| + - The edge's request loop and the activator's hosting loop have each reported a heartbeat within the last 60 seconds. These are local liveness: a hung process stops its own heartbeat. |
| +- **Why freshness was dropped.** Rev 2 required refreshes within 2 minutes. That made a bucket outage over two minutes mark every box unhealthy, and the group would terminate working boxes whose instances spec 2 can't move yet. A shared-dependency outage is an alert, from the existing bucket and housekeeper alerting, never a termination. |
| - Each condition is a timestamp or flag that its own loop updates. The answer only reads them, and does no disk, bucket or network work per request. |
| |
| **The region's IP.** |
| - The NLB takes the region's existing EIP (§9 step 2). So the carloku.com latency records (`dns-carloku.tf:44-106`), `oncarlos.com` and its wildcard, `<region>.oncarlos.com`, `platform.carlosframework.com`, and customers' A records all keep resolving to the same address. |
| -- The NLB uses one subnet, so this single address lives in one availability zone. **Losing that whole zone still takes the region's ingress down.** |
| -- A second zone means a second EIP and customers changing their DNS. That's deferred to spec 4. |
| +- Customers and our DNS use only zone A's address. The NLB has a node in each zone (§4), but **losing zone A still takes the region's published address down**. Clients using zone B's address keep working. |
| +- Publishing zone B's address too means customers changing their DNS. That's deferred to spec 4. |
| |
| **The platform box's own EIP.** |
| - The relay (3478 and 49152-65535) and keymail's MX (port 25) aren't web traffic, and the NLB carries only 80 and 443. So the platform box gets a new EIP, and `relay.eu-west-1.oncarlos.com` and the MX target move to it in flagship tofu. |
| -- **This happens before the region's EIP moves** (§9 step 0, A1-4). Otherwise the relay and inbound mail would fail from the cut-over until DNS caches expire. |
| +- **It moves in the same window as the region's EIP, prepared in advance** (§9 steps 0 and 2, A1-4). One instance can't simply hold both addresses: a second EIP needs a secondary private address and OS routing, the same difficulty as A2-7. So: |
| + - the relay and MX records' TTLs are lowered to 60 seconds well before the cut-over; |
| + - during the cut-over window, the platform box's public address is swapped from the region's EIP to its new one, and the records are switched with it. |
| + - The relay and inbound mail are down for the same few minutes as web traffic. Mail senders retry for days, so nothing is lost. |
| +- **The relay advertises its address in every allocation**, read once at start-up (`internal/relay/serve.go:175`, `server.go:645`). So the swap configures the relay's public address explicitly to the new EIP and restarts it, not just DNS (A2-6). Allocations made on the old address end with the window. A TURN allocation and a media exchange through the new address are checked before the window closes. |
| - Outbound mail from the platform box leaves from the new address too, so the SPF records and reverse DNS for the platform box's sending address move with it. |
| - Its web traffic arrives through the NLB like every other box's. |
| |
| -**Forwarding between boxes.** |
| -- Spec 1's forwarding (the edges file's home map, `internal/edge/homemap.go`) is on main and active today. It dials the home box's published public EIP on 443 (`edge.go:2779`). |
| -- Group boxes have temporary public IPs, and traffic within a region shouldn't leave the VPC. So each edges file also publishes the box's **private IPv4** from IMDS. Forwarding to a box in the same region dials the private address, and forwarding to another region dials the public one. |
| -- **A box's published public address must be its own** (A1-12). An address in the region ingress list belongs to an NLB, and a forwarded request landing on an NLB reaches an arbitrary box that can't forward again (one hop, `edge.go:1505`). So: |
| - - the publisher never publishes an address that appears in the region ingress list; |
| - - it re-reads IMDS `public-ipv4` before every publication instead of keeping the previous address across a blank read (`internal/host/edgesfile.go:589`); |
| - - a forced publication can be triggered (`carlos system republish-edges`, or the existing serial bump) for §9's cut-over. |
| +**Forwarding between boxes** (A2-7, A2-9: rev 3 replaces rev 2's own-address rule). |
| +- Spec 1's forwarding (the edges file's home map, `internal/edge/homemap.go`) is on main and active today. It dials the home box's published public EIP on 443 (`edge.go:2779`), and a forwarded request is never forwarded again (`edge.go:1505`). |
| +- Group boxes have no fixed public address, and an NLB address reaches an arbitrary box. So forwarding changes: |
| + - **Within a region**, a box forwards to the home box's **private IPv4**. Each edges file publishes it, from IMDS. |
| + - **Between regions**, a box forwards to the home region's **ingress address**: the first entry of that region's ingress list. That's the region's EIP, which is on the NLB once the region is migrated, or still on the single box before then. |
| + - The box that receives a cross-region forward forwards it **once more, within its region, to the private address** of the home box, if it isn't the home box itself. |
| + - **Hop limit 2.** The signed forwarding header carries a hop count. A request may be forwarded at most twice, and the second hop only to a private address in the receiving box's own region. Anything else is refused with 503, as one hop is today. |
| +- **What this removes:** boxes no longer publish a public dial address at all, so there's nothing to re-probe, filter or override. The cut-over (§9) needs no second address for the old box, and rollback releases nothing forwarding depends on. |
| - The security group gains 443 from the region's VPC CIDR, which is already exported as `vpc_cidr`. |
| -- Forwarding stays one hop, as now. |
| +- Readers that still use the published public address are listed in §8 and moved to these rules. |
| |
| **Platform routes.** |
| - The console and other unit- or socket-backed routes run only on the platform box (S1 §12: never relocated). |
| @@ -236,9 +263,15 @@ The choice is made once and recorded in the create intent, as now (H12). Recover |
| **Unplaced** (H4). The claim is never asked to hold a create with no executor (A1-7). `BeginCreateOn` refuses an empty executor (`internal/hostclaim/create.go:109`), and recovery needs `selected_executor` and `executor_enrollment` (`internal/hostops/production_create.go:257`). |
| - **The request waits in the console.** When no box is eligible, the console records a **pending create** in its own database: the requested host, app, region, plan and requesting member. It starts no claim operation. |
| - **The customer and operator see it.** The console shows the instance waiting for capacity, and an operator alert fires once per pending create. |
| -- **It is retried.** The placement pass runs every minute, and immediately when a box enrols or publishes new headroom. A pending create that finds an eligible box begins the claim create with that executor, through today's create path unchanged, and leaves the pending state in the same database transaction that records the started create. |
| -- **Only one placer.** The console is the only placer, and one pending create starts at most one claim create. The console database's row lock makes that atomic. |
| -- **It can be cancelled.** A member can cancel a pending create, which deletes it. Nothing exists outside the console to clean up. |
| +- **It is retried.** The placement pass runs every minute, and immediately when a box enrols or publishes new headroom. |
| +- **The handoff is recoverable, not atomic** (A2-5). The claim write is in the bucket and the pending row is in the console database, so no single transaction covers both. Instead: |
| + 1. When a pending create finds an eligible box, the console first commits the row as **`starting`**, with the chosen executor and a **stable operation identity** generated then. |
| + 2. Then it runs today's create path with that identity. The create path gains an option to take a caller-supplied operation ID instead of generating one (`internal/console/claim_create.go:50`). |
| + 3. On success, the row is deleted. |
| + 4. After a restart, or on any error, a `starting` row is **reconciled** before anything else touches it: the console reads the claim for that operation ID. If the create exists, the row is deleted. If it provably doesn't, the start is retried with the same identity, and the create is idempotent on it. |
| +- **Only one placer.** The console is the only placer. The database row lock lets one pass at a time move a row to `starting`. |
| +- **The hostname is reserved** from the moment the pending row exists. The immediate create door and the pending path check and take the same reservation, so a hostname can't be pending and created twice. |
| +- **It can be cancelled while it's still `pending`.** That deletes the row and its reservation. A `starting` row can't be cancelled until it's reconciled. If its create exists, the member cancels the instance through the ordinary delete. |
| - **The hostname answers with the edge's unknown-host response.** The host isn't in any edges file, so it isn't routed. |
| - **Copy.** The new console text and the alert go through Paul's copy review. |
| |
| @@ -258,6 +291,7 @@ Each is closed by a code change, or by showing it's safe under claim mode, befor |
| | `internal/activator/activator.go:2264-2272` | The local database copy is a warm cache only while a region has one box | H13: under the claim every admission is cold, and the warm path is never used. Verify that the claim-mode activator never reads this cache, and add a test | |
| | `internal/console/apiinstances.go:523-527` | Hosts are unique to one box | Holds under the claim (one executor). Add a test | |
| | `internal/console/view.go:177-237`, `server.go:1120-1147` | EIP list as A-record targets | §6: the region ingress list | |
| +| `internal/edge/homemap.go:549-613`, `edge.go:2779`, `internal/host/agent.go:2951-2963`, `edgesfile.go:589, 806, 823` | Forwarding dials each box's published public EIP, one hop | §6: private addresses in a region, the region ingress address between regions, hop limit 2 | |
| | `cmd/console/main.go:1390-1400` | `consoleBoxLabel`: exactly one box | Holds: the console runs on the platform box only. Document it | |
| | Flagship tofu `outputs.tf:39-68`, `dns*.tf`, `keymail.tf`, `mail.tf`, `scanner.tf` | One instance and EIP per region | Rewrite for the group, the NLB EIP and the platform box | |
| | `internal/console/apirestartscope.go:45` | Scoped restart refuses hosts the console's box doesn't know (A1-9) | Resolve the host's executor from the claim and act through it | |
| @@ -272,25 +306,26 @@ Each is closed by a code change, or by showing it's safe under claim mode, befor |
| **Order:** dev, then UK, US, AU and IE, one region at a time. Each region needs §3 to hold first. |
| |
| **Per region:** |
| -0. **IE only: move the relay and MX first** (A1-4). |
| - - Give the platform box its new EIP, as a second address alongside the region's. |
| - - Move `relay.eu-west-1.oncarlos.com`, the MX target, SPF and reverse DNS to it. |
| - - Wait at least twice the old records' TTL, and check that relay and mail traffic arrive on the new address. |
| +0. **IE only: prepare the relay and MX** (A1-4, A2-6). |
| + - Allocate the platform box's new EIP, unattached. |
| + - Lower the TTLs of `relay.eu-west-1.oncarlos.com` and the MX target to 60 seconds. Wait at least twice their old TTL before the cut-over. |
| + - Set the new EIP's reverse DNS once its forward record exists, and add it to SPF beside the old address. |
| 1. **Build.** |
| - Write the SSM parameters. |
| - Apply the module in group mode. The NLB is created with **only zone B's mapping**, carrying its new EIP (A1-§13). |
| - - The two group boxes boot, join, enrol and reach hosting mode. The NLB is rehearsed through zone B's address. |
| + - The two group boxes boot, join, enrol and reach hosting mode. The NLB is rehearsed through zone B's address. That checks zone B's targets only: zone A's are first exercised after step 2. |
| - The old fixed box keeps the region's EIP and all public traffic. It's also registered as an NLB target. |
| 2. **Cut over the EIP.** |
| - 1. **Give the old box a direct address.** It gets a temporary EIP of its own (in IE, step 0's new EIP is that address) and publishes it with a forced edges republication (A1-12). Wait until every box's home map shows it, which is checked through each box's published map digest. Cross-region forwarding to the old box then never dials the region's EIP. |
| - 2. **Disassociate the region's EIP** from the old box. |
| + 1. **Check forwarding is ready.** Every box in the deployment forwards with §6's rules: cross-region forwards go to the region's ingress address, and the receiving box forwards privately. No box holds a published-public-address dependency on the old box, so moving the EIP doesn't strand cross-region forwarding (A2-7). |
| + 2. **Disassociate the region's EIP** from the old box. In IE, immediately associate the new EIP with it, switch the relay and MX records to the new address, and restart the relay with that address advertised. |
| 3. **Add zone A's mapping** to the NLB with the region's EIP (`SetSubnets`, which accepts an EIP when adding a mapping). |
| 4. **The cost:** the region's public address is down from step ii until the zone A node is active. Expect seconds to a few minutes, scheduled at a quiet hour and announced. The window is measured on dev first. |
| 5. The old box keeps serving its instances throughout, reached through the NLB and through forwarding. |
| + 6. In IE, before the window closes: check that mail arrives on the new address, and that a TURN allocation advertises it and carries media. |
| 3. **Migrate.** Relocate every instance on the old box to §7's choice, one at a time, with spec 1's relocation. |
| 4. **Accept.** |
| - The old box stays running, deregistered from the NLB and holding no instances, for **seven days**. That's the rollback window. |
| - - After acceptance, UK, US and AU terminate it and release its temporary EIP. |
| + - After acceptance, UK, US and AU terminate it. |
| - IE keeps it as the platform box: it's enrolled with the `platform` role and keeps its new EIP from step 0. |
| |
| **Rollback** (A1-13). The region's EIP is a standalone `aws_eip` with `prevent_destroy`, so the allocation is never lost. |
| @@ -298,7 +333,8 @@ Each is closed by a code change, or by showing it's safe under claim mode, befor |
| - **From step 2 until acceptance:** |
| 1. Remove zone A's mapping from the NLB, or delete the NLB. An address on a requester-managed interface can't be disassociated directly. |
| 2. Wait until the EIP shows as unassociated. |
| - 3. Associate it with the old box, and remove the old box's temporary EIP after a forced republication. |
| + 3. Associate it with the old box. Forwarding needs no change: cross-region forwards to the region's ingress address now land on the old box, which forwards privately to the group boxes if needed. |
| + - **In IE** (A2-9), associating the region's EIP with the old box takes its new EIP off it, because one private address holds one public address. So the relay and MX records are switched back to the region's address, and the relay is restarted advertising it. The new EIP's allocation is kept for the next attempt. |
| 4. Relocate any migrated instances back. |
| - **After acceptance:** there's no rollback to the old box, because it's gone. Recovery is forward, within the group. |
| |
| @@ -329,10 +365,11 @@ Each is closed by a code change, or by showing it's safe under claim mode, befor |
| - **Tofu.** The module's tests cover group mode: the launch template, group, NLB, target groups, the health check path, IAM scoped to the parameter path, the alarms, and fixed-mode attachment to the target groups. Fixed mode stays identical when group mode is off. |
| - **Boot.** The start-up script is tested on dev from a clean launch, and every failure path is checked: bad sha256, missing parameter, failed hardening. Each must leave the box unhealthy. |
| - **Enrolment.** Unit tests for the membership-confirmation worker against a fake Auto Scaling describe, covering a member, a non-member, a terminated instance, and confirmation from another region (H17). Tests that joining mode never publishes an edges file or admits an instance. A test that the full sequence (confirm, invite, guard-ready, enrol, hosting) is satisfied against `membership.go`'s rule when a floor is published. On dev, a real launch enrols with no operator. |
| -- **Install.** A fresh box installs the channel release, not the bootstrap binary, and refuses a release below the host floor. |
| -- **Unplaced.** Pending create, retry on enrolment and on headroom, cancellation, and exactly one claim create per pending row under concurrent passes. |
| -- **Health answer.** Edge tests for `/.carlos/lb-health`: any host, exact path, GET and HEAD only, and 503 for each failing condition (joining mode, stale map, stale or missing mesh key, no certificate cache, no hosting heartbeat). |
| -- **Forwarding.** Private-address forwarding within a region and public-address forwarding across regions. Platform routes forwarded from a group box to the platform box. The publisher never publishes an address from the region ingress list, and re-reads IMDS on every publication. |
| +- **Install.** A fresh box installs the channel release into both protected slots, never the bootstrap binary, refuses a release below the host floor, and then passes `guard-arm`. |
| +- **Mode selection.** A restart at each joining state (no invitation, invited, guard installed, guard-ready reported) resumes joining. Only an active roster entry plus an armed guard starts hosting. |
| +- **Unplaced.** Pending create, retry on enrolment and on headroom, cancellation while pending, the hostname reservation shared with the immediate door, and exactly one claim create per pending row. That includes a crash after the claim write and before the row delete, reconciled by operation ID. |
| +- **Health answer.** Edge tests for `/.carlos/lb-health`: any host, exact path, GET and HEAD only, and 503 for each failing condition (joining mode, forwarding never armed, no certificate cache, a stalled heartbeat). A test that an unreachable bucket after arming leaves the answer at 200. |
| +- **Forwarding.** Private-address forwarding within a region. Cross-region forwarding to the ingress address followed by one private hop. A third hop, or a second hop to a non-private or out-of-region target, is refused. Platform routes forwarded from a group box to the platform box. |
| - **Placement.** Unit tests for each eligibility rule, the spread preference, the headroom ordering, the stale-headroom cut-off, the platform role, and unplaced with its retry triggers. |
| - **Rehearsal on dev:** |
| - the full §9 sequence, with the EIP cut-over window measured, the zone-B rehearsal, and the rollback sequence run once; |
| @@ -373,5 +410,20 @@ All three were answered in Astra's round 1 and folded in: |
| | A1-9 P2: local-registry consumers | Accepted: five rows added to §8, plus a full audit as the plan's first task | |
| | A1-10 P2: one map load doesn't prove readiness | Accepted: five live conditions (§6) | |
| | A1-11 P2: pending enrolment is invisible to alarms | Accepted: the eligible-capacity metric and alarm, a 503 health answer until hosting, and a grace period that covers enrolment (§4, §5) | |
| -| A1-12 P2: a published address can be the NLB's | Accepted: a box publishes only its own address, the old box gets a direct address before cut-over, forced republication (§6, §9) | |
| +| A1-12 P2: a published address can be the NLB's | Accepted in rev 2; superseded in rev 3: no box publishes a public dial address (§6) | |
| | A1-13 P2: rollback can't reassociate an NLB-owned EIP | Accepted: a standalone EIP with `prevent_destroy`, a seven-day acceptance window, and an explicit release-then-associate sequence (§9) | |
| + |
| +## 15. Astra round 2 dispositions (rev 3) |
| + |
| +| Finding | Disposition | |
| +|---|---| |
| +| A2-1 P1: enrolment omits the attempt and confirmation expiry | Accepted: a first attempt to decide on the invitation, then a fresh attempt, confirmation and `EnrollAutomatic` within the minute, repeated on expiry (§5) | |
| +| A2-2 P1: the bootstrap binary as `.prev` blocks guard-arm | Accepted: both slots hold the verified channel artifact (§5) | |
| +| A2-3 P1: a bucket outage would terminate working boxes | Accepted: the health answer is local liveness only, with forwarding latched once armed (§4, §6) | |
| +| A2-4 P2: a restart after writing `host-actor.json` | Accepted: mode selected from roster and guard, with resumable joining states (§5) | |
| +| A2-5 P2: the pending-create handoff isn't atomic | Accepted: a `starting` state with a stable operation identity, reconciliation by operation ID, and a shared hostname reservation (§7) | |
| +| A2-6 P2: the relay advertises its start-up address | Accepted: the relay is configured and restarted on the new address inside the cut-over window, and a TURN allocation and media exchange are checked before it closes (§6, §9). Rev 3 also moves IE's relay and MX swap into that window instead of a second address beforehand, which would need a secondary private IP | |
| +| A2-7 P2: no way to publish a second address | Superseded by the forwarding change: no box publishes a public dial address (§6) | |
| +| A2-8 P2: no permission for the capacity metric | Accepted: a namespace-scoped grant on the platform box, missing data as breaching (§4) | |
| +| A2-9 P2: rollback removes needed addresses | Accepted: forwarding doesn't depend on per-box addresses, and IE's step 0 isn't rolled back (§9) | |
| +| Round 1 partials | A1-1: §6 and §12 wording updated. A1-11: the first-build exception stated (§5). A1-12 and A1-13: superseded by the forwarding change | |
amadan