amadan
region HA spec 2, rev 2: answers Astra round 1 (4 P1, 9 P2, all accepted); round 1 findings committed unedited
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2 files changed,
+204
−67
docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md+147 −67docs/superpowers/specs/CODEX-FINDINGS-REGION-HA-SPEC2-1.md+57 −0
CI cancelled — superseded by a newer push — run details
diff --git a/docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md b/docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md| index 9fa8512f2..ffd35ae81 100644 |
| --- a/docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md |
| +++ b/docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md |
| @@ -1,6 +1,9 @@ |
| # Region HA, spec 2: two-box regions |
| -Design, 2026-10-15, **rev 1**. DRAFT. Nothing here is built. |
| +Design, 2026-10-15, **rev 2**. DRAFT. Nothing here is built. |
| + |
| +**Rev 2** answers Astra's round 1 (`CODEX-FINDINGS-REGION-HA-SPEC2-1.md`, |
| +cited **A1-n**): 4 P1 and 9 P2, all accepted. §14 maps each. |
| This spec builds on **spec 1** (`2026-10-04-region-ha-box-placement-design.md`, |
| cited **S1 §n**), which puts several boxes in a region under the host |
| @@ -76,27 +79,31 @@ that don't want HA. |
| - `min = max = desired = var.box_count`, which defaults to 2. |
| - Spread over two availability zones. |
| - Health check type `ELB`, using the NLB target group, plus EC2 status. |
| - - A grace period long enough for §5's boot, about 10 minutes, to be set from dev measurements. |
| + - A grace period long enough for §5's whole boot, enrolment included: about 20 minutes, to be set from dev measurements. The health answer (§6) stays 503 until the box is fully ready, so the grace period must cover enrolment, not just start-up. |
| - Instance refresh is not enabled. Releases reach boxes through `carlos system update`, never by replacing them. |
| - - Scale-in protection is not used, because there is no scale-in yet. |
| + - **The `AZRebalance` process is suspended** (A1-3). Rebalancing between zones launches and terminates boxes without changing desired capacity, and could terminate a healthy box that's hosting instances. With fixed size and no failover until spec 3, the only terminations are of unhealthy boxes. The `Launch`, `Terminate`, `HealthCheck` and `ReplaceUnhealthy` processes stay on. |
| - **A Network Load Balancer.** |
| - - **One subnet mapping, carrying the region's existing EIP allocation** (§9 step 2). |
| - - Cross-zone load balancing on. |
| + - **Two subnet mappings, one in each of the group's two zones** (A1-1). An NLB sends traffic only to targets in its enabled zones, even with cross-zone balancing on, so a single mapping would leave the second zone's box unused. |
| + - Zone A's mapping carries the region's **existing EIP** (§9 step 2). This is the address customers and our DNS use. |
| + - Zone B's mapping carries a **new EIP** of its own. It's in the region ingress list (§6), so it can be offered later, but nothing points at it yet. |
| + - Cross-zone load balancing on, so the node behind the existing EIP reaches both boxes. |
| - Listeners: TCP 80 and TCP 443. |
| - One target group per port, with instance targets (the client IP is preserved) and deregistration delay at 30s. |
| - Health check: HTTP on port 80, path `/.carlos/lb-health` (§6), every 10s, 2 healthy and 2 unhealthy. |
| - The group registers its instances itself. A fixed box, such as IE's platform box, is registered by a separate attachment. |
| -- **Alarms.** |
| - - `GroupInServiceInstances < desired` for 15 minutes: this catches a launch loop. |
| +- **Alarms.** All go to the module's existing SNS topic. |
| - `UnHealthyHostCount > 0` for 10 minutes. |
| - - Both go to the module's existing SNS topic. |
| + - **Eligible capacity** (A1-11): the console's placement pass publishes `Carlos/Region EligibleBoxes` per region (enrolled, active, placeable boxes, §7). An alarm fires when it is below `box_count` for 25 minutes, longer than the grace period. |
| + - This catches what the group's own metrics can't: a box that's in service and passing health checks but never enrolled, and a launch loop on a group whose target group has never had a healthy target (in that state, AWS doesn't replace ELB-unhealthy instances). |
| + - The group's `GroupInServiceInstances < desired` is kept as a second signal. |
| - **Termination evidence.** The EventBridge rule and SQS queue already on main (`main.tf:738-804`) cover group instances unchanged, because they match every EC2 `terminated` event in the account and region. |
| - **IAM.** |
| - The box role gains `autoscaling:DescribeAutoScalingInstances` and `autoscaling:DescribeAutoScalingGroups` (read-only), for H16's membership confirmation. |
| - - The box role gains `ssm:GetParameter` and `GetParametersByPath` on exactly `/carlos/<deployment>/<region>/*` and `/carlos/<deployment>/shared/*`, with `kms:Decrypt` for that path's key only. |
| - - The existing SSM-parameter Deny (`main.tf:456-475`) is kept and narrowed so it doesn't cover these paths. |
| + - The box role gains `ssm:GetParameter` and `GetParametersByPath` on exactly `/carlos/<deployment>/<region>/*` and `/carlos/<deployment>/shared/*`. |
| + - **Encryption** (A1-§13): one dedicated KMS key per deployment for its configuration parameters. The box role's `kms:Decrypt` on it is conditioned on `kms:ViaService = ssm.<region>.amazonaws.com` and on the encryption context `PARAMETER_ARN` matching the two paths. A separate key for the secrets key adds nothing, because the same boxes read both. |
| + - **The existing explicit SSM Deny** (`main.tf:456-475`) is kept, covering `GetParameter`, `GetParameters`, `GetParametersByPath` and `GetParameterHistory` on every parameter **outside** the two paths. A narrow Allow alone wouldn't constrain the managed-policy grants (`AmazonSSMManagedInstanceCore`). |
| -**Group mode removes** the per-box `aws_eip`. The per-instance alarms |
| +**Group mode removes** the per-box `aws_eip`. The region's existing EIP becomes a standalone `aws_eip` resource with `prevent_destroy`, so the allocation survives NLB changes and rollback (A1-13). The per-instance alarms |
| (`host_auto_recover`, `host_memory_low`, `main.tf:521-581`) are bound to |
| one InstanceId. In group mode they are replaced by the two group alarms |
| above, plus the memory alarm reworked to use the group dimension. |
| @@ -109,45 +116,62 @@ Auto-recovery has nothing to do once the group replaces sick boxes. |
| The launch template's start-up script is the only way a group box is |
| configured. It replaces `scripts/bootstrap-box.sh`, |
| `set-alert-sns-topic.sh` and the hardening scripts, which are run by hand |
| -over SSM today. Steps run in this order. A failed step stops the script, |
| -and nothing later runs. |
| +over SSM today. A failed step stops the script, and nothing later runs. |
| + |
| +A group box boots in two modes. **Joining** publishes evidence and installs the guard, but hosts nothing and publishes no edges file. **Hosting** is today's agent under the claim, entered only once the box is enrolled. Today the claim-mode agent refuses to start without `/etc/carlos/host-actor.json` and a matching roster entry (`internal/host/claimproduction.go:87`). And a box added after the floor is published needs its guard-ready report first (`internal/hostactors/membership.go:40`). So a joining mode is required (A1-2). |
| 1. **Harden.** Apply what `harden-box.sh` and `harden-host.sh` do today, including the nftables gate that keeps tenant processes away from IMDS credentials. Nothing tenant-facing starts before this. |
| -2. **Install.** |
| - - Download `s3://<deployment-bucket>/bootstrap/carlos`. |
| - - Check its sha256 against the SSM parameter `/carlos/<deployment>/shared/bootstrap-sha256`, and refuse on a mismatch. |
| - - Run `carlos bootstrap`, which installs the release that the signed channel pointer names, verified with the release key. So a new box joins on its neighbours' version, not on whatever the bootstrap object holds. |
| -3. **Configure.** Assemble `/etc/carlos/host.env` from SecureString parameters. Every value a hosting box needs today (`bootstrap-box.sh:49-57`, `README.md:339-342`) has one: |
| +2. **Configure.** Assemble `/etc/carlos/host.env` from SecureString parameters. Every value a hosting box needs today (`bootstrap-box.sh:49-57`, `README.md:339-342`) has one: |
| - the buckets and their regions; |
| - `CARLOS_RELEASE_PUBKEY` and `CARLOS_SECRETS_KEY`; |
| - `CARLOS_REGION` and `CARLOS_SYSTEM_CHANNEL`; |
| - the alert topic; |
| - `CARLOS_HOST_CLAIM`, `CARLOS_HOST_CLAIM_ORIGINS` and `CARLOS_REGION_HA`. |
| - The file is written mode 0600, owned by root. A missing required parameter stops the boot. |
| -4. **Start the edge.** `carlos-edge` starts and serves at once. It forwards (§6) without being enrolled, so the NLB health check can pass, and the box can carry traffic for its neighbour while it waits to enrol. |
| -5. **Enrol.** |
| - - The agent publishes its signed identity record, `control/boxes/<ref>.json` (`internal/host/boxrecord.go`, already built). |
| - - Confirmation (H16): a regional worker, meaning any enrolled box in the same region, reads the record. Using the new read-only Auto Scaling describe, it confirms that the instance is a current member of the region's group, then writes the create-only confirmation `control/box-enrollment/<ref>/<attempt>/<confirmation>.json`. |
| - - In a region with no enrolled box, a box in another region confirms it (H17, S1 §5.1). |
| - - The console calls `hostactors.EnrollAutomatic` (built, with no caller yet), and the box runs `carlos system guard-arm`. |
| - - Only then is the box `active` in the roster and eligible for placement (§7). |
| - |
| -**What spec 2 builds here:** the start-up script, the SSM parameter layout and a `carlos` command that writes it from today's `host.env`, the membership-confirmation worker (S1 called this spec 4's producer), and the console's enrolment pass. |
| + The file is written mode 0600, owned by root. A missing required parameter stops the boot. Configuration comes before the install, because the install verifies the release with the key it holds. |
| +3. **Install the signed channel release** (A1-6). |
| + - Download `s3://<deployment-bucket>/bootstrap/carlos` and check its sha256 against the SSM parameter `/carlos/<deployment>/shared/bootstrap-sha256`, refusing on a mismatch. |
| + - Run `carlos bootstrap` for users, directories and units (`internal/host/bootstrap.go`). It doesn't install a release. |
| + - Then install the release named by the signed channel pointer, verified with `CARLOS_RELEASE_PUBKEY`, through the same staging and verification path `carlos system update` uses (`internal/host/systemleg.go`, `internal/release/systempointer.go`). The bootstrap binary is kept as the previous binary. |
| + - The result: the box runs its neighbours' release, never the bootstrap object's. The box refuses to continue if the release is below the published host floor. |
| + - The plan names the exact command. If no existing command does exactly this, spec 2 adds `carlos system install-channel`. |
| +4. **Join.** Start the agent in joining mode, a new mode, selected because no `host-actor.json` exists yet. It: |
| + - publishes the signed identity record, `control/boxes/<ref>.json` (`internal/host/boxrecord.go`, already built); |
| + - does **not** publish an edges file (A1-5). An edges file naming the box would make it referenced (`internal/hostactors/references.go:63`), and automatic enrolment refuses a referenced box (`enroll.go:37`); |
| + - does not admit or serve any instance. The edge runs, but its health answer stays 503 (§6), so the NLB sends it nothing. |
| +5. **Enrol**, in this order: |
| + 1. **Confirm membership (H16).** A regional worker, meaning any hosting box in the same region, reads the identity record. With the new read-only Auto Scaling describe, it confirms the instance is a current member of the region's group, then writes the create-only confirmation `control/box-enrollment/<ref>/<attempt>/<confirmation>.json`. In a region with no hosting box, a box in another region confirms it (H17, S1 §5.1). |
| + 2. **Invite.** The console's enrolment pass sees the confirmation and issues a signed invitation (`hostactors.Invite`, built, with no caller yet). |
| + 3. **Install the guard.** The joining agent fetches the invitation, writes `host-actor.json`, runs `carlos system guard-arm` with the identity and previous-binary material from step 3, and publishes its guard-ready report. |
| + 4. **Enrol.** The console sees the guard-ready report and calls `hostactors.EnrollAutomatic` (built, with no caller yet). The membership rule at `membership.go:40` is then satisfied. |
| + 5. **Restart in hosting mode.** The agent finds itself enrolled in the roster and restarts as the claim-mode agent. Only now does it publish edges files and answer health 200 (§6), and only then is it eligible for placement (§7). |
| + |
| +**What spec 2 builds here:** |
| +- the start-up script, and the SSM parameter layout with a `carlos` command that writes it from today's `host.env`; |
| +- the channel-release install step; |
| +- joining mode; |
| +- the membership-confirmation worker (S1 called it spec 4's producer); |
| +- the console's enrolment pass (invitation, then enrolment); |
| +- the eligible-capacity metric. |
| **When boot fails:** |
| -- **Before step 4:** the health check never passes, and the group replaces the box after the grace period. |
| -- **A loop** (a bad bootstrap object, a missing parameter) shows as `GroupInServiceInstances < desired` for 15 minutes, and the alarm fires. |
| -- **Enrolment fails:** the box serves and forwards but is never placeable. The console lists it as pending enrolment, and the same alarm path alerts if it stays that way for 15 minutes. |
| +- **Any step:** the health answer never reaches 200, and the group replaces the box when the grace period ends. |
| +- **A loop** (a bad bootstrap object, a missing parameter, enrolment that never completes) shows as eligible capacity below `box_count` for 25 minutes, and the alarm fires (§4). The first boot of a new group is covered too, before its target group has ever had a healthy target. |
| -**The platform box** keeps its hand-managed state (keymail, woodstar, relay masters, the console database) and isn't rebuilt by this script. Its `host.env` moves to the same parameters, so a rebuild needs fewer hand steps, but it stays a fixed box. |
| +**The platform box** keeps its hand-managed state (keymail, woodstar, relay masters, the console database) and isn't rebuilt by this script. Its `host.env` moves to the same parameters, so a rebuild needs fewer hand steps, but it stays a fixed box. It is enrolled once by an operator, with the `platform` role (§7). |
| ## 6. Ingress, DNS and forwarding |
| **The load balancer health answer.** |
| - NLB HTTP health checks send the target's address as the Host header, so the steering probe (`internal/edge/steeringprobe.go`, which answers only for its own hostname) can't serve them. |
| -- The edge gains one reserved answer: `GET /.carlos/lb-health` on port 80, from any host. It replies 200 when the edge is serving and its forwarding map has loaded at least once since start, and 503 otherwise. |
| -- It reads nothing from disk or the bucket per request, and uses the same narrow matching as the steering probe: exact path, GET or HEAD only. |
| +- The edge gains one reserved answer: `GET /.carlos/lb-health` on port 80, from any host. It uses the same narrow matching as the steering probe: exact path, GET or HEAD only. |
| +- It replies **200 only when every condition holds**, and 503 otherwise (A1-10): |
| + - the agent is in **hosting mode** (§5): enrolled, guard armed; |
| + - the forwarding map has refreshed successfully within the last 2 minutes; |
| + - the forwarding mesh key is present and was refreshed within its own refresh window (`internal/edge/homemap.go:259`). Forwarding refuses without it; |
| + - the TLS certificate cache has loaded, with the default certificate available; |
| + - the activator's hosting loop has reported a heartbeat within the last 60 seconds. |
| +- Each condition is a timestamp or flag that its own loop updates. The answer only reads them, and does no disk, bucket or network work per request. |
| **The region's IP.** |
| - The NLB takes the region's existing EIP (§9 step 2). So the carloku.com latency records (`dns-carloku.tf:44-106`), `oncarlos.com` and its wildcard, `<region>.oncarlos.com`, `platform.carlosframework.com`, and customers' A records all keep resolving to the same address. |
| @@ -156,11 +180,17 @@ and nothing later runs. |
| **The platform box's own EIP.** |
| - The relay (3478 and 49152-65535) and keymail's MX (port 25) aren't web traffic, and the NLB carries only 80 and 443. So the platform box gets a new EIP, and `relay.eu-west-1.oncarlos.com` and the MX target move to it in flagship tofu. |
| +- **This happens before the region's EIP moves** (§9 step 0, A1-4). Otherwise the relay and inbound mail would fail from the cut-over until DNS caches expire. |
| +- Outbound mail from the platform box leaves from the new address too, so the SPF records and reverse DNS for the platform box's sending address move with it. |
| - Its web traffic arrives through the NLB like every other box's. |
| **Forwarding between boxes.** |
| - Spec 1's forwarding (the edges file's home map, `internal/edge/homemap.go`) is on main and active today. It dials the home box's published public EIP on 443 (`edge.go:2779`). |
| - Group boxes have temporary public IPs, and traffic within a region shouldn't leave the VPC. So each edges file also publishes the box's **private IPv4** from IMDS. Forwarding to a box in the same region dials the private address, and forwarding to another region dials the public one. |
| +- **A box's published public address must be its own** (A1-12). An address in the region ingress list belongs to an NLB, and a forwarded request landing on an NLB reaches an arbitrary box that can't forward again (one hop, `edge.go:1505`). So: |
| + - the publisher never publishes an address that appears in the region ingress list; |
| + - it re-reads IMDS `public-ipv4` before every publication instead of keeping the previous address across a blank read (`internal/host/edgesfile.go:589`); |
| + - a forced publication can be triggered (`carlos system republish-edges`, or the existing serial bump) for §9's cut-over. |
| - The security group gains 443 from the region's VPC CIDR, which is already exported as `vpc_cidr`. |
| - Forwarding stays one hop, as now. |
| @@ -192,7 +222,7 @@ P5-D2 hash. Its candidates already carry `MemFreeMiB`, `DiskFreeMiB` and |
| - **Not the platform box.** The roster gains a box role, `platform`, which is never eligible. The IE platform box is enrolled with it. |
| - **Memory.** `mem_free_mib` is at least 15% of the box's total memory, the same line as the memory alarm. Boxes publish total memory alongside headroom. |
| - **Disk.** `disk_free_mib` is at least 10% of the data volume's size. |
| -- **Fresh headroom.** A box whose edges file is older than 5 minutes is not eligible, so stale headroom is never trusted. |
| +- **Fresh headroom.** A box whose edges file is older than 15 minutes is not eligible, so stale headroom is never trusted. Edges files republish every 5 minutes plus up to a minute of jitter (`internal/host/edgesfile.go:794-798`), so a healthy box always has one within two intervals plus margin (A1-8). |
| **Choice:** |
| 1. Prefer a box running no instance of the same app. This is never a reason to refuse. |
| @@ -203,12 +233,14 @@ The choice is made once and recorded in the create intent, as now (H12). Recover |
| **Accepted limit.** `mem_free_mib` is memory free now (MemAvailable), so a box full of sleeping instances looks roomy until they wake. Counting each instance's memory limit would be more accurate, but limits aren't set everywhere. Spec 2 uses live memory, and the memory alarm and spec 4's scale-out cover overcommitment. |
| -**Unplaced.** |
| -- When no box is eligible, the create intent is recorded as `unplaced`. No executor is named. |
| -- The console shows the instance waiting for capacity, and an operator alert fires once per instance. |
| -- The placement pass retries every minute, and immediately when a box enrols or publishes new headroom. |
| -- An unplaced instance has no box, so it serves 503 with Retry-After, as an instance whose executor is unreachable does today. |
| -- When placed, it proceeds exactly as a fresh create. |
| +**Unplaced** (H4). The claim is never asked to hold a create with no executor (A1-7). `BeginCreateOn` refuses an empty executor (`internal/hostclaim/create.go:109`), and recovery needs `selected_executor` and `executor_enrollment` (`internal/hostops/production_create.go:257`). |
| +- **The request waits in the console.** When no box is eligible, the console records a **pending create** in its own database: the requested host, app, region, plan and requesting member. It starts no claim operation. |
| +- **The customer and operator see it.** The console shows the instance waiting for capacity, and an operator alert fires once per pending create. |
| +- **It is retried.** The placement pass runs every minute, and immediately when a box enrols or publishes new headroom. A pending create that finds an eligible box begins the claim create with that executor, through today's create path unchanged, and leaves the pending state in the same database transaction that records the started create. |
| +- **Only one placer.** The console is the only placer, and one pending create starts at most one claim create. The console database's row lock makes that atomic. |
| +- **It can be cancelled.** A member can cancel a pending create, which deletes it. Nothing exists outside the console to clean up. |
| +- **The hostname answers with the edge's unknown-host response.** The host isn't in any edges file, so it isn't routed. |
| +- **Copy.** The new console text and the alert go through Paul's copy review. |
| **Rebalancing:** none automatic. An operator moves an instance between boxes in a region with spec 1's relocation. Automatic rebalancing is spec 4's. |
| @@ -216,18 +248,23 @@ The choice is made once and recorded in the create intent, as now (H12). Recover |
| ## 8. Single-box assumptions to close |
| -Each is closed by a code change, or by showing it's safe under claim mode, before §3's third prerequisite holds. |
| +Each is closed by a code change, or by showing it's safe under claim mode, before §3's third prerequisite holds. The list comes from Astra's round 1 and a grep of main. The plan's first task repeats that audit for every reader of the local registry (`registry.Open` in console and housekeeper code), because those are the places that see only one box. |
| | Where | Assumption | Plan | |
| |---|---|---| |
| | `internal/console/apiinstancemove.go:614-618`, `boxselect.go:114-118`, `internal/host/movesphase1.go:517` | The move door refuses a region with more than one box | Moves' code: ask on #236 for multi-box support, choosing the destination box with §7's choice | |
| | `internal/console/seedfloor.go:25-29, 88` | Seed import refuses a region with more than one box | Ask the seeded-instances owner: import onto §7's choice | |
| | `internal/hostfence/onebox.go:9-30` | `CheckOneBoxRegion` | Read its callers. Each caller either requires one box, and is refused in two-box regions by design, or gets the region's box set | |
| -| `internal/activator/activator.go:2264-2272` | The local database copy is a warm cache only while a region has one box | Show that claim mode makes the executor the only writer (claim §7.1), so the cache is safe, or disable the cache in multi-box regions | |
| +| `internal/activator/activator.go:2264-2272` | The local database copy is a warm cache only while a region has one box | H13: under the claim every admission is cold, and the warm path is never used. Verify that the claim-mode activator never reads this cache, and add a test | |
| | `internal/console/apiinstances.go:523-527` | Hosts are unique to one box | Holds under the claim (one executor). Add a test | |
| | `internal/console/view.go:177-237`, `server.go:1120-1147` | EIP list as A-record targets | §6: the region ingress list | |
| | `cmd/console/main.go:1390-1400` | `consoleBoxLabel`: exactly one box | Holds: the console runs on the platform box only. Document it | |
| | Flagship tofu `outputs.tf:39-68`, `dns*.tf`, `keymail.tf`, `mail.tf`, `scanner.tf` | One instance and EIP per region | Rewrite for the group, the NLB EIP and the platform box | |
| +| `internal/console/apirestartscope.go:45` | Scoped restart refuses hosts the console's box doesn't know (A1-9) | Resolve the host's executor from the claim and act through it | |
| +| `internal/console/domainattach.go:150` | Domain attach looks for the primary route in the console's local registry | Read routes from the deployment-wide host records, checking ownership | |
| +| `internal/console/economicspass.go:308` | Processes on other boxes are attributed to platform overhead | Attribute by each box's published usage and the claim's executor | |
| +| `internal/status/fleet.go:154` | Fleet app and service listings use the elected publisher's local routes | Fold every box's edges file | |
| +| `internal/host/retention.go:49` | Retention lists apps from the housekeeper's local registry | List from the deployment-wide host records | |
| | Flagship scripts (`bootstrap-box.sh`, `update-platform-binary.sh`, `harden-*.sh`, `roll-console.sh`, `verify-console.sh`, `rollout-status-flagship.sh`, the woodstar and relay scripts) | One instance ID, or a hard-coded one | Group boxes: replaced by §5 and `carlos system update`. Platform box: keep the scripts, pointed at the platform box | |
| ## 9. Rollout |
| @@ -235,33 +272,55 @@ Each is closed by a code change, or by showing it's safe under claim mode, befor |
| **Order:** dev, then UK, US, AU and IE, one region at a time. Each region needs §3 to hold first. |
| **Per region:** |
| -1. **Build.** Write the SSM parameters, then apply the module in group mode with the NLB created **without** the EIP, on a temporary address. The two group boxes boot, enrol and serve through forwarding. The old fixed box keeps the EIP and all traffic. |
| +0. **IE only: move the relay and MX first** (A1-4). |
| + - Give the platform box its new EIP, as a second address alongside the region's. |
| + - Move `relay.eu-west-1.oncarlos.com`, the MX target, SPF and reverse DNS to it. |
| + - Wait at least twice the old records' TTL, and check that relay and mail traffic arrive on the new address. |
| +1. **Build.** |
| + - Write the SSM parameters. |
| + - Apply the module in group mode. The NLB is created with **only zone B's mapping**, carrying its new EIP (A1-§13). |
| + - The two group boxes boot, join, enrol and reach hosting mode. The NLB is rehearsed through zone B's address. |
| + - The old fixed box keeps the region's EIP and all public traffic. It's also registered as an NLB target. |
| 2. **Cut over the EIP.** |
| - - An NLB takes an EIP only when it's created. So the temporary NLB is destroyed, the EIP is disassociated from the old box, and the final NLB is created with the EIP. Its target groups hold the two group boxes and the old box. |
| - - **The region's ingress is down for a few minutes.** That's scheduled at a quiet hour and announced. The window is measured on dev first. |
| - - The old box keeps serving its instances throughout, reached through the NLB like any other target. |
| + 1. **Give the old box a direct address.** It gets a temporary EIP of its own (in IE, step 0's new EIP is that address) and publishes it with a forced edges republication (A1-12). Wait until every box's home map shows it, which is checked through each box's published map digest. Cross-region forwarding to the old box then never dials the region's EIP. |
| + 2. **Disassociate the region's EIP** from the old box. |
| + 3. **Add zone A's mapping** to the NLB with the region's EIP (`SetSubnets`, which accepts an EIP when adding a mapping). |
| + 4. **The cost:** the region's public address is down from step ii until the zone A node is active. Expect seconds to a few minutes, scheduled at a quiet hour and announced. The window is measured on dev first. |
| + 5. The old box keeps serving its instances throughout, reached through the NLB and through forwarding. |
| 3. **Migrate.** Relocate every instance on the old box to §7's choice, one at a time, with spec 1's relocation. |
| -4. **Retire.** |
| - - **UK, US and AU:** deregister the old box and terminate it. |
| - - **IE:** the old box stays as the platform box. It's enrolled with the `platform` role, gets its new EIP, and the relay and MX records move to it. |
| - |
| -**Rollback.** |
| -- Before step 2, destroying the group and NLB returns the region to today's state. |
| -- After step 2, moving the EIP back to the old box reverses ingress. Instances already relocated in step 3 can be relocated back. |
| - |
| -**Cost.** Each of UK, US and AU goes from one box to two, plus an NLB (about $20 a month before traffic). IE goes from one box to three: the platform box plus two group boxes. Dev matches whichever layout it rehearses. |
| +4. **Accept.** |
| + - The old box stays running, deregistered from the NLB and holding no instances, for **seven days**. That's the rollback window. |
| + - After acceptance, UK, US and AU terminate it and release its temporary EIP. |
| + - IE keeps it as the platform box: it's enrolled with the `platform` role and keeps its new EIP from step 0. |
| + |
| +**Rollback** (A1-13). The region's EIP is a standalone `aws_eip` with `prevent_destroy`, so the allocation is never lost. |
| +- **Before step 2:** destroy the group and NLB. The region is back to today's state. |
| +- **From step 2 until acceptance:** |
| + 1. Remove zone A's mapping from the NLB, or delete the NLB. An address on a requester-managed interface can't be disassociated directly. |
| + 2. Wait until the EIP shows as unassociated. |
| + 3. Associate it with the old box, and remove the old box's temporary EIP after a forced republication. |
| + 4. Relocate any migrated instances back. |
| +- **After acceptance:** there's no rollback to the old box, because it's gone. Recovery is forward, within the group. |
| + |
| +**Cost.** |
| +- UK, US and AU each go from one box to two, plus an NLB (about $20 a month before traffic) and zone B's EIP. |
| +- IE goes from one box to three: the platform box plus two group boxes. |
| +- Each old box costs seven more days during acceptance. |
| +- Dev matches whichever layout it rehearses. |
| ## 10. Failure modes |
| | What happens | Result | |
| |---|---| |
| -| A group box hangs or fails its health check | The NLB stops sending it traffic within about 20s, and the group terminates it after its health check fails. Its instances are down until spec 3 relocates them. A replacement boots, enrols and becomes placeable with no hand steps. | |
| +| A group box hangs or fails its health check | The NLB stops sending it traffic within about 20s, and the group terminates it once its health check has failed (after the grace period, for a new box). Its instances are down until spec 3 relocates them. A replacement boots, enrols and becomes placeable with no hand steps. | |
| | A group box is terminated by AWS | As above, without the health-check delay. | |
| | The platform box dies | The console and platform services are down. EC2 auto-recovery covers hardware faults, and otherwise it's an operator job (S1 §12). Customer instances are unaffected once migration is done. | |
| | Both group boxes in a region fail | The region's instances are down. The group replaces both, and the first replacement is confirmed from another region (H17). | |
| | The NLB's availability zone fails | The region's ingress is down until the zone recovers (§6). | |
| -| A replacement can't boot | It's never healthy, the group keeps replacing it, and the in-service alarm fires at 15 minutes. | |
| -| A box boots but can't enrol | It serves and forwards, takes no new instances, and is listed as pending. The alarm fires at 15 minutes. | |
| +| A replacement can't boot | It's never healthy, the group keeps replacing it, and the eligible-capacity alarm fires at 25 minutes. | |
| +| A box boots but can't enrol | It stays in joining mode, its health answer stays 503, and it takes no traffic. The group replaces it at the end of the grace period, and the eligible-capacity alarm fires at 25 minutes. | |
| +| One zone is unavailable | The NLB serves through the other zone's node only for clients that use zone B's address, so the region's published address (zone A) is down until the zone recovers. The group doesn't move boxes between zones (`AZRebalance` is suspended). | |
| +| A forwarding mesh key or map refresh stalls on a box | Its health answer turns 503, and the NLB stops sending it traffic. If the stall lasts past the health-check threshold, the group replaces it. | |
| | No box has headroom | New instances are unplaced, and an alert fires (§7). | |
| | The housekeeper's box dies | Certificate issuance pauses for up to 15 minutes, and serving continues. | |
| @@ -269,12 +328,14 @@ Each is closed by a code change, or by showing it's safe under claim mode, befor |
| - **Tofu.** The module's tests cover group mode: the launch template, group, NLB, target groups, the health check path, IAM scoped to the parameter path, the alarms, and fixed-mode attachment to the target groups. Fixed mode stays identical when group mode is off. |
| - **Boot.** The start-up script is tested on dev from a clean launch, and every failure path is checked: bad sha256, missing parameter, failed hardening. Each must leave the box unhealthy. |
| -- **Enrolment.** Unit tests for the membership-confirmation worker against a fake Auto Scaling describe, covering a member, a non-member, a terminated instance, and confirmation from another region (H17). On dev, a real launch enrols with no operator. |
| -- **Health answer.** Edge tests for `/.carlos/lb-health`: any host, exact path, GET and HEAD only, 503 before the forwarding map loads. |
| -- **Forwarding.** Private-address forwarding within a region and public-address forwarding across regions. Platform routes forwarded from a group box to the platform box. |
| +- **Enrolment.** Unit tests for the membership-confirmation worker against a fake Auto Scaling describe, covering a member, a non-member, a terminated instance, and confirmation from another region (H17). Tests that joining mode never publishes an edges file or admits an instance. A test that the full sequence (confirm, invite, guard-ready, enrol, hosting) is satisfied against `membership.go`'s rule when a floor is published. On dev, a real launch enrols with no operator. |
| +- **Install.** A fresh box installs the channel release, not the bootstrap binary, and refuses a release below the host floor. |
| +- **Unplaced.** Pending create, retry on enrolment and on headroom, cancellation, and exactly one claim create per pending row under concurrent passes. |
| +- **Health answer.** Edge tests for `/.carlos/lb-health`: any host, exact path, GET and HEAD only, and 503 for each failing condition (joining mode, stale map, stale or missing mesh key, no certificate cache, no hosting heartbeat). |
| +- **Forwarding.** Private-address forwarding within a region and public-address forwarding across regions. Platform routes forwarded from a group box to the platform box. The publisher never publishes an address from the region ingress list, and re-reads IMDS on every publication. |
| - **Placement.** Unit tests for each eligibility rule, the spread preference, the headroom ordering, the stale-headroom cut-off, the platform role, and unplaced with its retry triggers. |
| - **Rehearsal on dev:** |
| - - the full §9 sequence, with the EIP cut-over window measured; |
| + - the full §9 sequence, with the EIP cut-over window measured, the zone-B rehearsal, and the rollback sequence run once; |
| - one group box terminated by hand, with replacement and re-enrolment observed; |
| - a box made unhealthy, observed being terminated. |
| - **Failover drills** belong to spec 3. |
| @@ -290,8 +351,27 @@ Each is closed by a code change, or by showing it's safe under claim mode, befor |
| - the Auto Scaling activity-history evidence source (S1 A2-4). |
| - **Not planned:** relocating the console or platform services (S1 §12). The platform box is fixed, and its HA remains the HA ladder's. |
| -## 13. Open questions for review |
| +## 13. Questions from rev 1 |
| + |
| +All three were answered in Astra's round 1 and folded in: |
| +1. **Fresh headroom:** 15 minutes, because of the 5-minute publication interval (§7, A1-8). |
| +2. **Rehearsing the NLB:** create it with zone B's mapping only, then add zone A's with `SetSubnets` (§9, A1-§13). |
| +3. **SSM and KMS:** one configuration key per deployment, with `ViaService` and encryption-context conditions, and the explicit Deny kept (§4). |
| -1. **Fresh headroom.** Is 5 minutes the right cut-off, given edges files republish every 30s? |
| -2. **Temporary NLB address (§9 step 1).** Is there a cheaper rehearsal of the NLB before the cut-over than a throwaway NLB? |
| -3. **SSM parameter layout and key.** Should `CARLOS_SECRETS_KEY` share the region path's KMS key, or have its own key with a narrower policy? |
| +## 14. Astra round 1 dispositions (rev 2) |
| + |
| +| Finding | Disposition | |
| +|---|---| |
| +| A1-1 P1: one enabled zone leaves the second box unused | Accepted: two subnet mappings, the existing EIP in zone A and a new EIP in zone B (§4) | |
| +| A1-2 P1: boot and enrolment are circular | Accepted: joining mode, then confirm, invite, guard-ready, enrol, hosting (§5) | |
| +| A1-3 P1: AZRebalance can terminate a healthy box | Accepted: AZRebalance suspended (§4) | |
| +| A1-4 P1: relay and MX break at the cut-over | Accepted: IE step 0 moves them first and waits for TTLs (§6, §9) | |
| +| A1-5 P2: an edges file makes a joining box referenced | Accepted: joining mode publishes no edges file (§5) | |
| +| A1-6 P2: `carlos bootstrap` doesn't install the release | Accepted: an explicit signed channel-release install, below-floor refusal (§5) | |
| +| A1-7 P2: `unplaced` has no claim contract | Accepted, by keeping it out of the claim: a pending create in the console, started once an executor is chosen (§7) | |
| +| A1-8 P2: the freshness cut-off is below the publication interval | Accepted: 15 minutes (§7) | |
| +| A1-9 P2: local-registry consumers | Accepted: five rows added to §8, plus a full audit as the plan's first task | |
| +| A1-10 P2: one map load doesn't prove readiness | Accepted: five live conditions (§6) | |
| +| A1-11 P2: pending enrolment is invisible to alarms | Accepted: the eligible-capacity metric and alarm, a 503 health answer until hosting, and a grace period that covers enrolment (§4, §5) | |
| +| A1-12 P2: a published address can be the NLB's | Accepted: a box publishes only its own address, the old box gets a direct address before cut-over, forced republication (§6, §9) | |
| +| A1-13 P2: rollback can't reassociate an NLB-owned EIP | Accepted: a standalone EIP with `prevent_destroy`, a seven-day acceptance window, and an explicit release-then-associate sequence (§9) | |
diff --git a/docs/superpowers/specs/CODEX-FINDINGS-REGION-HA-SPEC2-1.md b/docs/superpowers/specs/CODEX-FINDINGS-REGION-HA-SPEC2-1.md| new file mode 100644 |
| index 000000000..64145fbfa |
| --- /dev/null |
| +++ b/docs/superpowers/specs/CODEX-FINDINGS-REGION-HA-SPEC2-1.md |
| @@ -0,0 +1,57 @@ |
| +Reviewed `ddd0617f` against its code base, `origin/main` at `70e57449`, and the supplied flagship checkout. No files changed; no builds, tests or infrastructure commands run. |
| + |
| +1. **P1 — The NLB cannot use both boxes with only one enabled zone.** |
| + [Spec:83](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/design-region-ha-two-box/docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md:83) specifies one subnet mapping but instance targets spread across two zones. Cross-zone balancing reaches targets in **enabled** zones; the second zone’s target remains `unused`. If the first box hangs while its zone remains available, the healthy second box cannot receive ingress. Enable both target zones; customers can retain their existing A record. [AWS target-health documentation](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/target-group-health-checks.html). |
| + |
| +2. **P1 — Boot and enrolment have a circular dependency.** |
| + [Spec:128–134](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/design-region-ha-two-box/docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md:128) starts the claim-enabled agent before enrolment, then enrols before arming the guard. Today, agent construction requires `/etc/carlos/host-actor.json` and a matching roster entry ([claimproduction.go:87](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/design-region-ha-two-box/internal/host/claimproduction.go:87)); failure exits before the identity-publication loop starts. Separately, adding a box after floor publication requires its guard-ready report ([membership.go:40](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/design-region-ha-two-box/internal/hostactors/membership.go:40)). A clean replacement therefore cannot complete either ordering. Specify a non-hosting enrolment mode, signed invitation, guard installation and readiness report **before** final enrolment. |
| + |
| +3. **P1 — Fixed group size does not prevent healthy-box termination.** |
| + [Spec:81](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/design-region-ha-two-box/docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md:81) omits scale-in protection because scaling is deferred. However, `AZRebalance` can launch and terminate instances without changing desired capacity. If an unavailable zone forces both boxes into the surviving zone, recovery can trigger termination of a healthy box hosting customer instances. Spec 2 then leaves those instances down until spec 3. Explicitly suspend zone rebalancing or provide a relocation-aware protection mechanism. [AWS process behaviour](https://docs.aws.amazon.com/autoscaling/ec2/userguide/understand-how-suspending-processes-affects-other-processes.html). |
| + |
| +4. **P1 — IE’s relay and MX move too late.** |
| + [Spec:246](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/design-region-ha-two-box/docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md:246) assigns the platform EIP and changes relay/MX DNS only after all customer instances migrate. From step 2 onward, their old address belongs to an NLB accepting only 80/443. Relay traffic and inbound mail therefore fail for the entire migration, plus DNS cache expiry—not merely the announced web cut-over. The current relay record directly uses the IE EIP ([dns53.tf:103](/home/paulca/github.com/carlosframework/carloku-infrastructure/dns53.tf:103)). Move these services and account for DNS TTLs before transferring that address. |
| + |
| +5. **P2 — Pre-enrolment publication can permanently disqualify automatic enrolment.** |
| + [Spec:130](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/design-region-ha-two-box/docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md:130) needs an explicit publication boundary. Once pre-enrolment startup is enabled, the normal agent also starts `edgesFileLoop`. An edges file naming the new box makes `Directory.Referenced` true, and automatic enrolment explicitly rejects referenced boxes ([enroll.go:37](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/design-region-ha-two-box/internal/hostactors/enroll.go:37), [references.go:63](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/design-region-ha-two-box/internal/hostactors/references.go:63)). A delayed confirmation can consequently turn an otherwise valid replacement into an operator-only case. Suppress ordinary edges publication until enrolment, or define a safely distinguishable provisional record. |
| + |
| +6. **P2 — `carlos bootstrap` does not install the signed channel release.** |
| + [Spec:119](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/design-region-ha-two-box/docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md:119) attributes release installation to a command that installs users, directories and units. It finishes by instructing the operator to copy the binary and configure the host ([bootstrap.go:619](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/design-region-ha-two-box/internal/host/bootstrap.go:619)). A replacement would run the bootstrap object’s version, potentially below the published floor. Specify the signed-release installation step, including identity and previous-binary material required by guard-arm; obtain its verification configuration beforehand. |
| + |
| +7. **P2 — `unplaced` needs a claim transition and recovery contract.** |
| + [Spec:207](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/design-region-ha-two-box/docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md:207) cannot use today’s create path unchanged. `BeginCreateOn` rejects an empty executor, and production recovery requires both `selected_executor` and `executor_enrollment` ([create.go:109](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/design-region-ha-two-box/internal/hostclaim/create.go:109), [production_create.go:257](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/design-region-ha-two-box/internal/hostops/production_create.go:257)). A persisted executor-less create would remain unrecoverable after capacity returns. Define reservation, cancellation, concurrent placer CAS, one-time executor selection, and old-reader behaviour. Also define how edges recognise an unplaced hostname and return the promised 503 without a published home. |
| + |
| +8. **P2 — Healthy boxes will repeatedly fail the freshness threshold.** |
| + [Spec:195](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/design-region-ha-two-box/docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md:195) rejects edges files older than five minutes, but publication is **five minutes plus up to one minute of jitter**, not every 30 seconds ([edgesfile.go:794–798](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/design-region-ha-two-box/internal/host/edgesfile.go:794)). Quiet, healthy boxes periodically become ineligible; overlapping windows produce false capacity alerts and unplaced instances. Change the publication cadence explicitly or set the cutoff above its maximum interval plus delivery margin. |
| + |
| +9. **P2 — §8 omits customer operations and accounting that depend on the console’s local registry.** |
| + [Spec:217](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/design-region-ha-two-box/docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md:217) does not close these concrete regressions when IE customers leave the console box: |
| + |
| + - Scoped restart refuses hosts known only to another box ([apirestartscope.go:45](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/design-region-ha-two-box/internal/console/apirestartscope.go:45)). |
| + - Domain attachment finds no existing primary route and takes the first-route path ([domainattach.go:150](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/design-region-ha-two-box/internal/console/domainattach.go:150)). |
| + - Remote customer processes fall into platform-overhead attribution ([economicspass.go:308](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/design-region-ha-two-box/internal/console/economicspass.go:308)). |
| + - Fleet app/service listings use the elected publisher’s local routes ([fleet.go:154](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/design-region-ha-two-box/internal/status/fleet.go:154)); retention enumerates apps from the housekeeper’s local registry ([retention.go:49](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/design-region-ha-two-box/internal/host/retention.go:49)). |
| + |
| + Add these consumers to the prerequisite audit and give them complete, ownership-checked inputs. |
| + |
| +10. **P2 — Loading the forwarding map once does not prove forwarding readiness.** |
| + [Spec:149](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/design-region-ha-two-box/docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md:149) permits a permanent 200 after one map load. Map and mesh-key refreshes fail independently ([homemap.go:259](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/design-region-ha-two-box/internal/edge/homemap.go:259)); forwarding refuses without a key. A new forwarding-only box can therefore pass health checks while every forwarded request fails. The same answer does not establish TLS readiness or detect a stalled hosting worker. Define cached readiness and local liveness conditions for the services this health check is meant to protect. |
| + |
| +11. **P2 — Pending enrolment is invisible to the proposed alarms.** |
| + [Spec:141](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/design-region-ha-two-box/docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md:141) promises an alert after 15 minutes, but an unenrolled forwarding box can be both ASG `InService` and NLB healthy. Both proposed alarm predicates remain false indefinitely. Add an enrolment-age or eligible-capacity signal. Initial boot failure also needs qualification: AWS does not replace ELB-unhealthy instances while an attached target group has never reached its first healthy target. [ASG metrics](https://docs.aws.amazon.com/autoscaling/ec2/userguide/ec2-auto-scaling-metrics.html), [attachment-state behaviour](https://docs.aws.amazon.com/autoscaling/ec2/userguide/load-balancer-status.html). |
| + |
| +12. **P2 — EIP cut-over needs a direct-address publication barrier.** |
| + [Spec:240–242](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/design-region-ha-two-box/docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md:240) moves the old box’s published dial address onto the NLB. The publisher re-probes it only on timed passes and retains the previous address across blank IMDS reads ([edgesfile.go:589](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/design-region-ha-two-box/internal/host/edgesfile.go:589)). During that interval, another region forwards a signed hop to the NLB; a group box receiving it cannot forward again and returns 503 ([edge.go:1505](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/design-region-ha-two-box/internal/edge/edge.go:1505)). Require a direct public address, forced publication and reader convergence for the old box during cut-over, as well as private forwarding within its region. |
| + |
| +13. **P2 — Rollback omits releasing the NLB-owned address and retaining the old executor.** |
| + [Spec:250](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/design-region-ha-two-box/docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md:250) cannot be executed as a direct EIP reassociation: AWS forbids disassociating addresses from requester-managed interfaces. The owning NLB/subnet mapping must release it first. After step 4 terminates the old box, the stated rollback is no longer available at all. Define the rollback boundary, preserve the allocation, retain the old box until acceptance, and specify the release/wait/reassociate sequence. [AWS interface restrictions](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/requester-managed-eni.html). |
| + |
| +For §13: |
| + |
| +- **Fresh headroom:** resolve finding 8 first. Five minutes is workable only with an explicitly faster publisher. |
| +- **Temporary NLB:** the creation-only assertion is too broad. `SetSubnets` accepts an EIP when adding a mapping. A temporary mapping in the other zone, followed by adding the existing-EIP mapping, could retain the rehearsal NLB and address finding 1. Rehearse that exact sequence on dev. [AWS API](https://docs.aws.amazon.com/elasticloadbalancing/latest/APIReference/API_SetSubnets.html). |
| +- **SSM/KMS:** a separate secrets key only adds isolation if its authorised readers differ. A dedicated deployment configuration key can suffice, with exact parameter-path restrictions and decrypt conditions binding SSM and `PARAMETER_ARN`. Keep the existing explicit Deny covering all four read operations outside the permitted paths; adding a narrow Allow alone does not constrain the managed-policy grants. [AWS encryption-context documentation](https://docs.aws.amazon.com/systems-manager/latest/userguide/secure-string-parameter-kms-encryption.html). |
| + |
| +I found no new two-writer path in the existing cold-admission mechanism. Preserve H13: exclusive executorship alone does **not** make an old local database safe after relocation. Certificate challenge read-through supports the proposed ingress, and preserving client IPs avoids an inherent NLB-induced GeoIP change. |
| + |
| +**Verdict: not ready for a plan.** |
| \ No newline at end of file |