Region HA, spec 2: two-box regions

Design, 2026-10-17, rev 21. Rev 15 was approved by Paul on 2026-10-16; rev 16's ruling H28 was approved in direction the same day, and platform-1e's census ruling region-ha-2b was RULED by the Plan 14 controller the same evening, with conditions. Nothing here is built.

Rev 18 answers Astra's third review of plan 2b (docs/superpowers/plans/CODEX-REVIEW-REGION-HA-2B-3.md, cited B3-n) and the census ruling's conditions where the spec changes: the write deadline's anchor, edge and single enforcement point, and the read-back of an unknown enrolment (§5 step 5); a group box publishes no status until its reference resolves, and the status reader's listing (§5 step 5); host.env's allowlist and cross-check (§5 step 2). §30 maps them.

Rev 19 answers Astra's fourth review of plan 2b (docs/superpowers/plans/CODEX-REVIEW-REGION-HA-2B-4.md, cited B4-n) with two simplifications the claim owner and platform-1e agreed: an unknown enrolment is settled by the next pass's roster read, with no read-back and no page, and a detached-but-running instance is spec 3's (§5 step 5, §12); every store call of a bound write has a hard 15-second timeout, and a success after the deadline is unknown (§5 step 5; rev 20 corrects what that bounds); host.env's path is fixed and written only through directories root alone can write, held by descriptor (§5 step 2). §31 maps them.

Rev 20 answers Astra's fifth review of plan 2b (docs/superpowers/plans/CODEX-REVIEW-REGION-HA-2B-5.md, cited B5-n) and platform-1e's acceptance of revision 4: the 15-second timeout bounds how long the console waits, not when a write commits, and rev 19's 75-second commit ceiling is withdrawn; every write in the invitation and enrolment paths is conditional, so a late commit lands only where it was aimed (§5 step 5). §32 maps them.

Rev 21 answers Astra's final review of the plan 2b build (docs/superpowers/plans/CODEX-FINAL-REVIEW-REGION-HA-2B.md, cited BF-n): each membership describe is cut off at 15 seconds as unknown (§5 step 5); host.env's allowlist leaves out the alert webhook and the status destination overrides (§5 step 2); and automatic enrolment gains two activation gates (§5). §33 maps them.

Rev 17 answers Astra's second review of plan 2b (docs/superpowers/plans/CODEX-REVIEW-REGION-HA-2B-2.md, cited B2-n) where the spec changes: the membership read's 60 seconds are a deadline on the write itself, not a check before it (§5 step 5); a box its region's group launched is referenced by a group reference, not a status label (§5); operator seeding needs kms:Decrypt (§5 step 2); and automatic enrolment has two activation gates (§5). §29 maps them.

Rev 16 records Paul's ruling H28: the console is the only membership confirmer. It confirms a group box's membership itself, with its own read-only Auto Scaling and EC2 describes, immediately before it invites and before it enrolls; no box writes or contributes membership evidence, and H17's cross-region box confirmer is withdrawn. It answers Astra's review of plan 2b (docs/superpowers/plans/CODEX-REVIEW-REGION-HA-2B-1.md, cited B1-n) where the spec changes: §4's IAM, §5's boot order and enrolment, §10, §11 and §25. §28 maps them.

Rev 15 records Paul's ruling H27: no pending creates. When no box qualifies, the create door refuses with a clear message and alerts the operator. It also settles the certificate-readiness wording in §6. Plans: 2a (core), 2b (joining and enrolment), 2c (infrastructure and boot), 2d (single-box fixes).

Rev 13 adds §25, the provider boundary, at Paul's request. Spec 2 must not make the platform depend on AWS: a deployment stays "one object-store bucket + N hosts running the carlos binary + DNS" (README), and its AWS-specific parts are adapters.

Rev 14 corrects §25 after Astra's round 13 (CODEX-FINDINGS-REGION-HA-SPEC2-13.md, A13-n). Multi-box regions are already AWS-only on main, because a box's identity is AWS's signed instance document. Paul ruled H26 (2026-10-16): they stay AWS-only until a provider-neutral identity spec lands (#299). Spec 2 keeps its own new AWS parts as adapters, so that spec only has to replace identity.

Rev 12 answers Astra's round 11 (CODEX-FINDINGS-REGION-HA-SPEC2-11.md, A11-1, P2, accepted). Rollback's exclusion is set on the region's group, and every box the group launches inherits it at enrolment, so a replacement can't reopen placement. §24.

Rev 11 answers Astra's round 10, a full pass (CODEX-FINDINGS-REGION-HA-SPEC2-10.md, cited A10-n): three P2s and one P3, all accepted. Rollback now restores only the ingress and never destroys a group that has hosted an instance, so retiring a group is spec 4's drain too. Moves take their source from the claim, not from placement eligibility. §23 maps them.

Rev 10 follows Paul's ruling H25: spec 2 no longer empties the old boxes. Rounds 6 to 9 each found another gap in draining a live box, because that was spec 4's drain pulled forward. Instead, each region's old box keeps its existing instances, takes no new ones, and swaps to a new public address of its own at the cut-over. The live drain returns to spec 4, with rounds 6 to 9's findings recorded as its inputs. §22 maps them.

Rev 9 answers Astra's round 8 (CODEX-FINDINGS-REGION-HA-SPEC2-8.md, cited A8-n). Rounds 6 to 8 each found another way an instance could reach a draining box. So the settlement barrier is now one general rule over the claim, not a list of operation kinds. Never-woken instances get one platform wake before relocation. §21 maps both.

Rev 8 answers Astra's round 7 (CODEX-FINDINGS-REGION-HA-SPEC2-7.md, cited A7-n): spec 2 builds the durable draining control that S1 left to spec 4, and the emptiness check covers in-flight creates. §20 maps both.

Rev 7 answers Astra's round 6 (CODEX-FINDINGS-REGION-HA-SPEC2-6.md, cited A6-n), two P2s in the rollout, both accepted. The old box is set draining before migration and checked empty before the cut-over and before termination. IE's old box takes the platform role only at acceptance, so rollback can move instances back to it. §19 maps them.

Rev 6 simplifies forwarding (§6). Rounds 3 to 5 each found a new edge in rev 3's two-hop protocol: the hop budget, the staged rollout, the host floor, then key rotation. Rev 6 keeps today's protocol instead: one hop, today's credential, today's mesh-key rotation. Only the address the forwarder dials changes, and instances leave the old box before the region's EIP moves. A3-1, A3-2, A4-1, A4-2, A5-1 and A5-2 are moot. §18 maps them.

Rev 5 answers Astra's round 4 (CODEX-FINDINGS-REGION-HA-SPEC2-4.md, cited A4-n), two P2s, both accepted: stage 2 is gated on an armed host floor, and the hop credential follows the mesh key's existing current-and-next rotation. §17 maps each.

Rev 4 answers Astra's round 3 (CODEX-FINDINGS-REGION-HA-SPEC2-3.md, cited A3-n): 1 P1 and 2 P2, all accepted. They cover the forwarding failure contract, a staged rollout of the new forwarding protocol, and the refresh worker's liveness. §16 maps each.

Rev 3 answers Astra's round 2 (CODEX-FINDINGS-REGION-HA-SPEC2-2.md, cited A2-n): 3 P1 and 6 P2, all accepted, plus the partial dispositions of round 1. The largest change is to forwarding (§6, A2-7, A2-9). Forwarding between regions now targets the destination region's ingress address, and the box there forwards once more over the private network. No box ever needs a public address of its own. §15 maps each finding.

Rev 2 answers Astra's round 1 (CODEX-FINDINGS-REGION-HA-SPEC2-1.md, cited A1-n): 4 P1 and 9 P2, all accepted. §14 maps each.

This spec builds on spec 1 (2026-10-04-region-ha-box-placement-design.md, cited S1 §n), which puts several boxes in a region under the host lifecycle claim. Spec 1's code, and relocation (plan 1b-next), are on main and dormant. This spec turns every carloku.com region into a group of two interchangeable boxes behind a load balancer, with a placement policy for which box takes each instance.

1. Purpose

carloku.com instances should keep serving when one box in a region fails, with no one paged, in the region that sold them. Today each region runs one fixed EC2 box. If it dies, its instances are down until it's recovered by hand. EC2 auto-recovery restarts it in place, but only for AWS hardware faults.

Spec 2 gives each region two boxes. A sick box is terminated and replaced automatically, and new instances are spread across the two boxes. Spec 3 (failover) then moves a terminated box's instances to the survivor. Spec 2 is useful without spec 3: a replaced box rejoins with no hand steps, and new instances keep landing on the healthy box. But a box's instances still go down with it until spec 3 lands.

2. Rulings

Paul's rulings of 2026-10-04 (S1 §2, H1 to H19) stand, except H17, which H28 withdraws. These are new, from 2026-10-15 and 2026-10-16:

| # | Ruling | |---|---| | H20 | The first HA release recovers hung boxes as well as lost ones. H1 makes termination the only proof of death. A hung box is terminated only by something that health-checks it, so each region's boxes run in an Auto Scaling group with health checks. Group size is fixed for now. Automatic scale-out and scale-in stay in spec 4. | | H21 | Ingress is one Network Load Balancer per region, passing TCP 80 and 443 straight through. TLS stays on the boxes. | | H22 | Every carloku.com region runs two boxes: dev first, then UK, US, AU and IE. (H5's per-region minimum is set to 2 everywhere.) | | H23 | The specs are recut. Spec 2 covers two-box regions: this document. Spec 3 covers failover. Spec 4 shrinks to automatic scale-out and scale-in, rebalancing and latency steering. S1 §12's split is superseded where they differ. | | H24 | IE's current box becomes the platform box. It's fixed, outside the group, and keeps the console, keymail, woodstar, the relay and the scanner grants. It takes no new customer instances. Its existing ones leave with spec 4's drain (H25). | | H25 | (2026-10-16) Spec 2 doesn't empty the old boxes. Each region's old fixed box keeps the instances it has and takes no new ones. Those instances keep today's single-box behaviour until spec 4's drain moves them, or until spec 3's failover moves them if AWS terminates the box. Each region runs three boxes until then. | | H26 | (2026-10-16) Multi-box regions stay AWS-only until a provider-neutral box identity exists (#299). Box identity, enrolment, roster validation and placement eligibility rest on AWS's signed instance document on main today. Spec 2 doesn't make that worse: its own new AWS parts are adapters (§25). | | H27 | (2026-10-16) No pending creates; this supersedes H4's "waits as unplaced". When no box qualifies, every door that chooses a box refuses with HTTP 503 and a clear message: no capacity in that region right now, try again shortly. The operator is alerted, at most once per region per hour per console process. A console restart may send one extra alert, which is accepted rather than persisting the limit. The door chooses before it registers anything, so a refusal leaves no claim write and no recovery work. The doors that choose a box are: instance create, domain attach (first route), standalone sidecar create, and the move door's destination. A sidecar attached to an instance runs on that instance's box and chooses nothing. | | H28 | (2026-10-16) The console is the only membership confirmer. "H16 stands (automatic enrolment by the console). The console confirms group membership itself, with its own credentials, by read-only Auto Scaling and EC2 describes in the region's AWS region, immediately before it invites and before it enrols. No box writes or contributes membership evidence. H17's cross-region box confirmer is withdrawn as unnecessary: the console can read any region, so an emptied region recovers the same way." With platform-1e's authority line: a box never writes, approves or mints roster membership; the console verifies and signs. |

Design choices Paul approved in section review on the same day, recorded here so they aren't reopened:

• Boot (§5). New boxes boot with no hand steps. Configuration comes from SSM parameters, and a box takes no instance until it's enrolled.
• Ingress (§6). The NLB uses the region's existing EIP. The platform box gets its own EIP for the relay and MX. The console tells customers to point at configured ingress IPs.
• Placement (§7). The rules are eligibility thresholds, soft spread, then the most free memory. When nothing qualifies, the request is refused and an alert fires (H27, which replaced the original "unplaced and alert"). No automatic rebalancing.
• Rollout (§9). Each region needs claim mode active first. The EIP cut-over has a few minutes of announced downtime per region.

3. Prerequisites

Two-box regions depend on the claim. With claim mode off, every box in a region competes for each instance through the per-host lease (internal/activator/activator.go:1080). The activator trusts an instance's local database copy only while a region has one box (activator.go:2264-2272). So:

1. Claim mode is active in the deployment. That means CARLOS_HOST_CLAIM, CARLOS_HOST_CLAIM_ORIGINS and #250's backfill, plus the open claim asks on #236.
2. Region HA is on in the region (CARLOS_REGION_HA), and relocation has been rehearsed there.
3. Every single-box assumption in §8 is closed.

No region gets a second box before all three hold.

4. Infrastructure: infra/modules/region-host

The module currently makes one aws_instance and one aws_eip (main.tf:482-517). It gains a group mode, chosen by a variable. The fixed-instance mode stays for the platform box and for deployments that don't want HA.

Group mode adds:

• A launch template. - Same AMI lookup, instance type, root volume (delete_on_termination = true), security group and instance profile as the fixed box. - The start-up script is §5's. - IMDSv2 required, with a hop limit of 1. - An auto-assigned public IPv4 address on the box's own interface, so each group box has a public address of its own for cross-region forwarding (§6). It's never the region's EIP.
• An Auto Scaling group. - min = max = desired = var.box_count, which defaults to 2. - Spread over two availability zones. - Health check type ELB, using the NLB target group, plus EC2 status. The health answer (§6) measures only this box's own liveness, never a shared dependency, so a deployment-bucket outage can't mark healthy boxes for termination (A2-3). - A grace period long enough for §5's whole boot, enrolment included: about 20 minutes, to be set from dev measurements. The health answer (§6) stays 503 until the box reaches hosting mode, so the grace period must cover enrolment, not just start-up. - Instance refresh is not enabled. Releases reach boxes through carlos system update, never by replacing them. - The AZRebalance process is suspended (A1-3). Rebalancing between zones launches and terminates boxes without changing desired capacity, and could terminate a healthy box that's hosting instances. With fixed size and no failover until spec 3, the only terminations are of unhealthy boxes. The Launch, Terminate, HealthCheck and ReplaceUnhealthy processes stay on.
• A Network Load Balancer. - Two subnet mappings, one in each of the group's two zones (A1-1). An NLB sends traffic only to targets in its enabled zones, even with cross-zone balancing on, so a single mapping would leave the second zone's box unused. - Zone A's mapping carries the region's existing EIP (§9 step 2). This is the address customers and our DNS use. - Zone B's mapping carries a new EIP of its own. It's in the region ingress list (§6), so it can be offered later, but nothing points at it yet. - Cross-zone load balancing on, so the node behind the existing EIP reaches both boxes. - Listeners: TCP 80 and TCP 443. - One target group per port, with instance targets (the client IP is preserved) and deregistration delay at 30s. - Health check: HTTP on port 80, path /.carlos/lb-health (§6), every 10s, 2 healthy and 2 unhealthy. - The group registers its instances itself. A fixed box, such as IE's platform box, is registered by a separate attachment.
• Alarms. All go to the module's existing SNS topic. - UnHealthyHostCount > 0 for 10 minutes. - Eligible capacity (A1-11): the console's placement pass publishes Carlos/Region EligibleBoxes per region (enrolled, active, placeable boxes, §7). An alarm fires when it is below box_count for 25 minutes, longer than the grace period. - The platform box's role gains cloudwatch:PutMetricData conditioned on cloudwatch:namespace = Carlos/Region. The current grant covers only Carlos/Host (main.tf:418, A2-8). - Missing data is treated as breaching: if the console stops publishing, the alarm fires, which is also worth an alert. - This catches what the group's own metrics can't: a box that's in service and passing health checks but never enrolled, and a launch loop on a group whose target group has never had a healthy target (in that state, AWS doesn't replace ELB-unhealthy instances). - The group's GroupInServiceInstances < desired is kept as a second signal.
• Termination evidence. The EventBridge rule and SQS queue already on main (main.tf:738-804) cover group instances unchanged, because they match every EC2 terminated event in the account and region.
• IAM. - Membership is the console's to read (H28). The console's role gains autoscaling:DescribeAutoScalingInstances and ec2:DescribeInstances, read-only, conditioned on aws:RequestedRegion being one of the deployment's AWS regions. Neither action takes a resource ARN, so the region condition is the limit. DescribeAutoScalingGroups isn't needed: the read binds to the group name and ARN the signed roster names. - The box role gains no Auto Scaling or EC2 describe, and no write under control/box-enrollment/. 2c's infrastructure test checks both denials with a box's credentials. - The box role gains ssm:GetParameter and GetParametersByPath on exactly /carlos/<deployment>/<region>/* and /carlos/<deployment>/shared/*. The start-up script reads /carlos/<deployment>/shared/bootstrap-sha256 with the AWS CLI before any carlos binary is on the box. - Encryption (A1-§13): one dedicated KMS key per deployment for its configuration parameters. The box role's kms:Decrypt on it is conditioned on kms:ViaService = ssm.<region>.amazonaws.com and on the encryption context PARAMETER_ARN matching the two paths. A separate key for the secrets key adds nothing, because the same boxes read both. - The existing explicit SSM Deny (main.tf:456-475) is kept, covering GetParameter, GetParameters, GetParametersByPath and GetParameterHistory on every parameter outside the two paths. A narrow Allow alone wouldn't constrain the managed-policy grants (AmazonSSMManagedInstanceCore).

Group mode removes the per-box aws_eip. The region's existing EIP becomes a standalone aws_eip resource with prevent_destroy, so the allocation survives NLB changes and rollback (A1-13). The per-instance alarms (host_auto_recover, host_memory_low, main.tf:521-581) are bound to one InstanceId. In group mode they are replaced by the two group alarms above, plus the memory alarm reworked to use the group dimension. Auto-recovery has nothing to do once the group replaces sick boxes.

The platform box stays in fixed mode (aws_instance plus its own new aws_eip). It is also attached to the region's NLB target groups, so that console and platform routes are reachable through the NLB (§6).

5. Boot and enrolment

The launch template's start-up script is the only way a group box is configured. It replaces scripts/bootstrap-box.sh, set-alert-sns-topic.sh and the hardening scripts, which are run by hand over SSM today. A failed step stops the script, and nothing later runs.

A group box boots in two modes. Joining publishes evidence and installs the guard, but hosts nothing and publishes no edges file. Hosting is today's agent under the claim, entered only once the box is enrolled.

Mode selection happens at every agent start, never from one file's presence (A2-4). The agent is in hosting mode only when all three hold:

• host-actor.json exists;
• the roster has a matching, active entry for this box;
• the guard is armed for that enrolment.

For a box its region's group launched, a fourth: the instance is still the launch it was enrolled on. A stopped group box that is started again finds all three intact, so it would otherwise resume hosting on a stale enrolment. EC2 sets the identity document's pendingTime at every start, never at a reboot. When the current document's pendingTime differs from the one in the identity record the roster entry binds, the box is held in joining mode: it answers 503, and the group replaces it. It can't prove membership again under the same reference while its roster entry stands. A box an operator enrolled isn't asked, because fixed boxes are stopped and started for maintenance.

Anything else is joining mode. Joining resumes from whatever it finds: no invitation yet, invited, guard installed, or guard-ready reported. A restart at any point repeats the next step. Today the claim-mode agent refuses to start without /etc/carlos/host-actor.json and a matching roster entry (internal/host/claimproduction.go:87). And a box added after the floor is published needs its guard-ready report first (internal/hostactors/membership.go:40). So a joining mode is required (A1-2).

1. Harden. Apply what harden-box.sh and harden-host.sh do today, including the nftables gate that keeps tenant processes away from IMDS credentials. Nothing tenant-facing starts before this.
2. Configure. - First, the bootstrap binary (B1-5). The launch template's user data names the deployment, its bucket, the region and its AWS region, so this needs no carlos binary. With the AWS CLI, download s3://<deployment-bucket>/bootstrap/carlos, read /carlos/<deployment>/shared/bootstrap-sha256 (aws ssm get-parameter --with-decryption), and check the download's sha256 against it. A mismatch, or a read that fails, stops the boot. - Then run the bootstrap binary's carlos system host-env write as root, which assembles /etc/carlos/host.env from SecureString parameters. Every value a hosting box needs today (bootstrap-box.sh:49-57, README.md:339-342) has one: - the buckets and their regions; - CARLOS_RELEASE_PUBKEY and CARLOS_SECRETS_KEY; - CARLOS_REGION and CARLOS_SYSTEM_CHANNEL; - the alert topic; - CARLOS_HOST_CLAIM, CARLOS_HOST_CLAIM_ORIGINS and CARLOS_REGION_HA.

The file is written mode 0600, owned by root. A missing required parameter stops the boot. Configuration comes before the install, because the install verifies the release with the key it holds. Shared values are seeded create-only, so two seeds can't leave different shared values behind. Only an allowlist of variable names is taken from the parameters (any other is refused by name), values are limited to a safe set, and the file is replaced whole. The allowlist leaves out credentials, CARLOS_ALERT_WEBHOOK among them, since a webhook URL can carry one (BF-3), and the status destination overrides, CARLOS_STATUS_BUCKET, CARLOS_STATUS_PREFIX and CARLOS_STATUS_REGION: a group box reports to the deployment bucket, unprefixed, where the console reads its guard-ready report, while joining and once hosting (BF-2). Its path is fixed, /etc/carlos/host.env, with no flag to move it (B4-4). /etc and /etc/carlos are each opened by descriptor, without following a symlink, and refused unless they are directories root owns that no group or other can write (a missing /etc/carlos is created 0755, then opened and checked the same way); the new file is created 0600 inside the held directory, fsynced, renamed over the old one within it, and the directory fsynced. Off Linux the write refuses. The two values that select a box's authority, the group marker below and CARLOS_REGION, are checked against the signed roster before a group box hosts, not at write, since at boot the box has no roster entry yet. Box roles never write the parameters. Seeding reads the shared values back decrypted, so the operator needs ssm:GetParametersByPath and ssm:PutParameter on the two paths, kms:Encrypt on the configuration key, and kms:Decrypt on it conditioned on kms:ViaService = ssm.<region>.amazonaws.com and the PARAMETER_ARN encryption context of the two paths (B2-new-2). host-env write also adds CARLOS_BOX_REFERENCE=group, which marks the box as one its group launched; it is never a parameter.

1. Install the signed channel release (A1-6). - Run the bootstrap binary's carlos bootstrap for users, directories and units (internal/host/bootstrap.go). It doesn't install a release. - Then install the release named by the signed channel pointer, verified with CARLOS_RELEASE_PUBKEY, through the same staging and verification path carlos system update uses (internal/host/systemleg.go, internal/release/systempointer.go): carlos system install-channel. The directories it creates under /data are handed to the service user, as bootstrap's own are, so the agent's later updates can download there. - Both protected slots, the current binary and carlos.prev, are the verified channel artifact (A2-2). guard-arm checks both installed binaries against the supplied identity and the exact artifact hash (internal/host/guardarm.go:75, internal/hostfloor/floor.go:138). A bootstrap binary of a different build in either slot would make the guard refuse. The bootstrap binary is deleted once the install verifies. - The result: the box runs its neighbours' release, never the bootstrap object's. The box refuses to continue if the release is below the published host floor.
2. Join. Start the agent in joining mode, a new mode, selected because no host-actor.json exists yet. It: - publishes the signed identity record, control/boxes/<ref>.json (internal/host/boxrecord.go, already built); - does not publish an edges file (A1-5). An edges file naming the box would make it referenced (internal/hostactors/references.go:63), and automatic enrolment refuses a referenced box (enroll.go:37); - does not admit or serve any instance. The edge runs, but its health answer stays 503 (§6), so the NLB sends it nothing.
3. Enrol (A2-1, H28). EnrollAutomatic needs a matching Roster.EnrollmentAttempts entry, which only BeginEnrollmentAttempt creates, and the attempt is single-use (Consumed). The console is the only membership confirmer: it reads membership itself, through its own read-only client, immediately before each write that membership allows. 1. Attempt. The console's enrolment pass sees the identity record and calls BeginEnrollmentAttempt. 2. Confirm, then invite. The console describes the attempt's instance, with DescribeAutoScalingInstances and DescribeInstances in the region's AWS region. The instance is a member only when it is in the region's group, Pending or InService, pending or running, owned by the region's account, in its AWS region, with a request id from each call. On that read the console issues a signed invitation (hostactors.Invite). The invitation is stable: later attempts reuse it. 3. Install the guard. The joining agent fetches the invitation, writes host-actor.json, runs carlos system guard-arm with the identity and binaries from step 3, and publishes its guard-ready report. 4. Confirm again, then enrol. When the guard-ready report appears, the console calls hostactors.EnrollAutomatic, which reads membership again itself and makes the enrolling write. With the guard-ready report present, membership.go:40's rule is satisfied. 5. Hosting mode. The agent's next start check finds the roster entry and the armed guard, and it restarts as the claim-mode agent. Only now does it publish edges files and answer health 200 (§6), and only then is it eligible for placement (§7).

The rules of the read. - Freshness: 60 seconds, enforced at the write, and a 15-second hard timeout per call (B2-new-1, B3, B4-1). A fresh describe runs immediately before each write, the invitation and the enrolment. The write deadline is anchored at the completion of the membership describe or earlier, never later (2b anchors it at the describe's start), and a write 60 s or more after the anchor is refused: the write is issued no later than 60 s after the membership read. Each store call issued before the deadline has a hard timeout of 15 s. What that guarantees (B5-1): every store call of the invite and the enrolling write is issued before the deadline (60 s from the start of the membership describe); the console waits at most 15 s for any call; a write whose result the console did not see is unknown and is settled on a later pass by reading the roster. A cancelled write can still commit later; such an enrolment rests on a membership read that was true when the write was issued, and is the same exposure as an instance detached after enrolment, which spec 3's worker reconciles. The invitation and the enrolling write, with every bucket read and signature they make on the way (the sealed Invite, checkMembership and Replace unchanged), run on a store bound to the deadline, the single enforcement point. It guarantees that no store call, the roster Put included, is issued at or after the deadline; its context cancels a call in flight at the deadline; a call not answered 15 s after it was issued is cut off by the wrapper itself, whether or not the store honours its context; and a call that answers only after the deadline, even with success, is unknown. So a write whose commit can't be confirmed before the deadline is unknown. The hard timeout is the call's own context, which the S3 client enforces at the HTTP layer; the console's shared S3 client is not changed and no second client is built. Every write on these paths is conditional: the roster write is a compare-and-swap on the ETag the pass read, and an invitation is created only if absent, through every wrapper of the console's store. So a write that commits late lands only on the exact roster revision it was built against, and never overwrites a newer revision or another enrolment. A missed deadline keeps the attempt and its backoff; a describe that itself took 60 seconds or more counts against it. The describe has the same 15-second cut-off (BF-1): neither AWS call sets a deadline of its own, and a stalled connection would otherwise hold the serial pass and every box behind it. A describe cut off is unknown, its attempt kept and its backoff unchanged, and the pass goes on to the next box. - Unknown outcomes are settled by the roster (B4-2, B4-3). An outcome the bound can't confirm is resolved the ordinary way, by the next pass's read of the signed roster: a box the roster lists is enrolled and done; one it doesn't list goes on with the same attempt while that attempt is still valid, under the same rules, and an unknown doesn't grow its backoff. Nothing is read back, remembered or paged. A terminated instance is kept out of hosting and capacity by its termination fence, a stopped one by the stale-edges rule and the restart hold. One that left its group but still runs keeps its enrolment until an operator removes it; reconciling that is spec 3's (§12). - Unforgeable. EnrollAutomatic and its check take the console's reader and make the describe themselves; the confirmation is built inside hostactors from that answer and never leaves it. No exported path takes a confirmation, so nothing a box writes can stand in for one. - Unknown is not invalid (B1-7). A read that fails, or isn't an answer, is unknown: the pass asks again with the same attempt, and its backoff doesn't grow. Only the cloud's positive answer that the instance isn't a current member, or a read that went stale before its write, counts against the attempt. The pass then keeps the attempt, without describing again, for a wait that doubles with each refusal up to 32 times its base, before it begins another. - Audit. The roster has no field for the describes' request ids, and none is added: the console logs every membership read with the box's reference, its attempt id, its instance id, both request ids, the read's time and what came of it.

Group references (B2-3, census ruling region-ha-2b). A box its region's group launched is referenced by g- and the first 20 hex digits of sha256("carlos-box-ref-v1|" + account + "|" + instance), never a status label: labels come from 1,600 values, identity records are never deleted, and a box whose label another record holds could never publish its own. The box publishes its identity record, its status reports and its edges files under it; statusLoop's tick publishes under it, the one change to that sealed unit, under platform-1e's ruling. Until the reference resolves, a group box publishes no status report at all, and never under a status label, which may be another box's key (B3-1); it asks again every tick, whatever the claim and HA switches say. The console never trusts the key a box chose: it derives the reference from the instance the identity record proves (the attempt's, then the roster's pinned one), refuses a record or request under any other key, and reads the box's reports under the derived key, so a report another box writes there fails on its instance and enrollment. The public status/fleet.json shows only the pseudonym. Every box an operator enrolled, fleet boxes included, keeps its label and status key unchanged.

The status listing (B3-2). Each replaced group box leaves a report under its own reference. The status reader drops reports no box has rewritten in a day, from the listing's write times, before any read and before its key cap, so the cap bounds current reports only. It's an optimisation, not authority: every reader still judges a report's own age as before. The deployment bucket also expires status/boxes/ objects after 30 days, as hygiene; nothing reads old reports for history, and nothing depends on the rule, so a store without lifecycle rules works the same.

Activation gates. Automatic enrolment isn't switched on for any region until (1) a release with group references runs on every box of that region and on the console: without them, a region in its first build, whose group never replaces an unhealthy box, could stall on a label collision that an alarm reports but nothing repairs (B2-3); (2) the pendingTime behaviour behind the stopped-box rule has been seen on dev, before the first region: a group box stopped and started is held and replaced, one rebooted hosts again; (3) on dev, a fresh group box goes guard-arm, then carlos-system-recover, then guard-exec across the joining-to-hosting restart, and comes up hosting, since static inspection can't establish that systemd transition (BF); and (4) carloku.com's host.env files have been checked against the allowlist, which now excludes the status destination overrides and the alert webhook, before the first seed (BF-2, BF-3).

What spec 2 builds here:

• the start-up script, and the SSM parameter layout with a carlos command that writes it from today's host.env;
• the channel-release install step;
• joining mode;
• the console's membership read (S1 called the producer spec 4's; H28 makes it the console's);
• the console's enrolment pass (invitation, then enrolment);
• the eligible-capacity metric.

When boot fails:

• Any step: the health answer never reaches 200. The group replaces the box when the grace period ends, except while the group's target group has never had a healthy target: AWS doesn't replace ELB-unhealthy instances then (A1-11). That case happens only when a region is first built.
• A loop (a bad bootstrap object, a missing parameter, enrolment that never completes) shows as eligible capacity below box_count for 25 minutes, and the alarm fires (§4). That alarm also covers the first-build case, where the group won't replace on its own.

The platform box keeps its hand-managed state (keymail, woodstar, relay masters, the console database) and isn't rebuilt by this script. Its host.env moves to the same parameters, so a rebuild needs fewer hand steps, but it stays a fixed box. It takes the platform role at the rollout's step 2 (§9). It keeps the customer instances it already has until spec 4's drain (H25).

6. Ingress, DNS and forwarding

The load balancer health answer.

• NLB HTTP health checks send the target's address as the Host header, so the steering probe (internal/edge/steeringprobe.go, which answers only for its own hostname) can't serve them.
• The edge gains one reserved answer: GET /.carlos/lb-health on port 80, from any host. It uses the same narrow matching as the steering probe: exact path, GET or HEAD only.
• It replies 200 only when every condition holds, and 503 otherwise. Every condition is this box's own, so an outage of a shared dependency like the deployment bucket never turns a working box unhealthy (A2-3). - The agent is in hosting mode (§5): enrolled, guard armed. - Forwarding is armed: the forwarding map and the mesh key have each loaded successfully at least once since the edge started (A1-10). This is latched, not a freshness check. After that, a failed refresh keeps the cached map and key, as today (internal/edge/homemap.go:235), and doesn't affect health. - The TLS certificate cache has loaded: the certificate mirror has completed its first load. Certificates are per host, so there is no single default certificate to check. - The edge's request loop and the activator's hosting loop have each reported a heartbeat within the last 60 seconds. These are local liveness: a hung process stops its own heartbeat. - The forwarding map and key refresh worker (internal/edge/homemap.go:838) has completed an attempt, whether it succeeded or failed, within the last 5 minutes (A3-3). Completion counts as progress, so a bucket outage, where every attempt fails quickly, keeps it satisfied. A worker stuck inside one attempt doesn't. The 5 minutes covers the refresh interval plus the existing two-minute request timeout.
• Why freshness was dropped. Rev 2 required refreshes within 2 minutes. That made a bucket outage over two minutes mark every box unhealthy, and the group would terminate working boxes whose instances spec 2 can't move yet. A shared-dependency outage is an alert, from the existing bucket and housekeeper alerting, never a termination.
• Each condition is a timestamp or flag that its own loop updates. The answer only reads them, and does no disk, bucket or network work per request.

The region's IP.

• The NLB takes the region's existing EIP (§9 step 2). So the carloku.com latency records (dns-carloku.tf:44-106), oncarlos.com and its wildcard, <region>.oncarlos.com, platform.carlosframework.com, and customers' A records all keep resolving to the same address.
• Customers and our DNS use only zone A's address. The NLB has a node in each zone (§4), but losing zone A still takes the region's published address down. Clients using zone B's address keep working.
• Publishing zone B's address too means customers changing their DNS. That's deferred to spec 4.

The platform box's own EIP.

• The relay (3478 and 49152-65535) and keymail's MX (port 25) aren't web traffic, and the NLB carries only 80 and 443. So the platform box gets a new EIP, and relay.eu-west-1.oncarlos.com and the MX target move to it in flagship tofu.
• It moves in the same window as the region's EIP, prepared in advance (§9 steps 0 and 2, A1-4). One instance can't simply hold both addresses: a second EIP needs a secondary private address and OS routing, the same difficulty as A2-7. So: - the relay and MX records' TTLs are lowered to 60 seconds well before the cut-over; - during the cut-over window, the platform box's public address is swapped from the region's EIP to its new one, and the records are switched with it. - The relay and inbound mail are down for the same few minutes as web traffic. Mail senders retry for days, so nothing is lost.
• The relay advertises its address in every allocation, read once at start-up (internal/relay/serve.go:175, server.go:645). So the swap configures the relay's public address explicitly to the new EIP and restarts it, not just DNS (A2-6). Allocations made on the old address end with the window. A TURN allocation and a media exchange through the new address are checked before the window closes.
• Outbound mail from the platform box leaves from the new address too, so the SPF records and reverse DNS for the platform box's sending address move with it.
• Its web traffic arrives through the NLB like every other box's.

Forwarding between boxes (rev 6).

• Spec 1's forwarding is on main and active today. The edges file's home map (internal/edge/homemap.go) tells each edge which box runs each host. A request for a host on another box is forwarded once, signed with the mesh key, to that box (edge.go:2779), and a forwarded request is never forwarded again (edge.go:1505). Spec 2 keeps all of that, including the credential and the mesh-key rotation, unchanged.
• The only change is which address the forwarder dials: - Same region: the home box's private IPv4. Each edges file adds a private_ip field, read from IMDS. Readers that don't know the field ignore it, so it ships in an ordinary release with no ordering between boxes. A reader that finds no private_ip dials the public address, as today. - Another region: the home box's own public address, as today. For a group box, that's its auto-assigned public IP (§4).
• A box publishes only an address IMDS reports as its own (A1-12). The publisher re-reads IMDS public-ipv4 before every publication. A blank read publishes no public address, instead of keeping the previous one (internal/host/edgesfile.go:589). The region's EIP is published only by a box that actually holds it, which before the cut-over is the old fixed box. Once the EIP is on the NLB, IMDS no longer reports it to any box.
• Why this is enough: every request reaches the home box in one hop: - from the NLB's box to the home box in the same region, privately; - from any box in another region to the home box's own public address. - No box ever needs to forward a forwarded request.
• The one transition is the old box's address at the cut-over (A10-4). Its instances stay (H25), so §9 step 3 swaps it to a new EIP of its own and republishes at once. Cross-region readers pick up the new address within a minute, inside the cut-over window. Until then, a cross-region forward to one of its instances lands on the NLB and is refused, because a forwarded request isn't forwarded again.
• The security group gains 443 from the region's VPC CIDR, which is already exported as vpc_cidr.

Platform routes.

• The console and other unit- or socket-backed routes run only on the platform box (S1 §12: never relocated).
• With the NLB, their requests reach any box, so a group box must forward them to the platform box. Today those routes have only ever been served on the box that received them.
• Spec 2 makes this a requirement with its own tests: a request for console.<domain>, or any platform hostname, arriving at a group box is forwarded to the box that publishes the route.

What the console tells customers.

• The console lists every EIP folded from the edges files as A-record targets, plus PLATFORM_EDGE_IPS (internal/console/view.go:177-237, server.go:1120-1147). With group boxes, that list would include temporary addresses.
• It changes to a configured region ingress list: each region's NLB EIP, from a deployment setting written by tofu.
• The domain check's "pointing" verdict, from the domains reconcile duty, uses the same list.
• The new console text goes through Paul's copy review.

Unchanged:

• Certificates. A challenge landing on any box is answered by reading through to control/certs/ (internal/edge/certsync.go:1594-1622).
• The housekeeper election (internal/host/housekeeper.go). If the housekeeper's box dies, issuance pauses for up to the lease TTL, 15 minutes, while serving continues.
• Per-box latency steering stays off. It's spec 4's.

7. Placement policy (H4)

Spec 1's boxchoice.Choose (internal/boxchoice/boxchoice.go:146-182) prefers a box running none of the app, then the fewest hosts, then the P5-D2 hash. Its candidates already carry MemFreeMiB, DiskFreeMiB and Awake, unused (S1 §5, "until spec 2"). Spec 2 sets the policy.

Eligibility is spec 1's Eligible (enrolled, active, in the instance's region, not fenced), and in addition:

• Not the platform box. The roster gains a box role, platform, which is never eligible. The IE platform box is enrolled with it.
• Memory. mem_free_mib is at least 15% of the box's total memory, the same line as the memory alarm. Boxes publish total memory alongside headroom.
• Disk. disk_free_mib is at least 10% of the data volume's size.
• Fresh headroom. A box whose edges file is older than 15 minutes is not eligible, so stale headroom is never trusted. Edges files republish every 5 minutes plus up to a minute of jitter (internal/host/edgesfile.go:794-798), so a healthy box always has one within two intervals plus margin (A1-8).

Choice:

1. Prefer a box running no instance of the same app. This is never a reason to refuse.
2. Then the most mem_free_mib.
3. Then the P5-D2 hash.

The choice is made once and recorded in the create intent, as now (H12). Recovery never re-chooses.

Accepted limit. mem_free_mib is memory free now (MemAvailable), so a box full of sleeping instances looks roomy until they wake. Counting each instance's memory limit would be more accurate, but limits aren't set everywhere. Spec 2 uses live memory, and the memory alarm and spec 4's scale-out cover overcommitment.

No capacity (H27). When no box is eligible, the door refuses the request with HTTP 503 and a clear message that there's no capacity in that region right now, so the caller should try again shortly. The message goes through Paul's copy review. An operator alert fires, at most once per region per hour per console process, so a retry loop doesn't flood it. The door chooses the box before it registers the operation, so a refusal leaves nothing registered, nothing written to the claim, and nothing for recovery to settle. This applies to every door that chooses a box (H27). The eligible-capacity metric (§4) is published by a small console pass every minute.

Old boxes (H25). Each region's old fixed box takes the roster role legacy; IE's takes platform. Both roles are never eligible for a new placement, a relocation target or a move target. That's a static exclusion, read from the roster by every chooser (boxchoice.Eligible), and it needs no settlement:

• the box's existing instances stay where they are;
• operations already heading to it before the role was set simply finish there, and the instance joins the box's existing ones. Nothing needs to be empty, so nothing needs to be waited for.

The roles are set by an operator, through carlos system box role <ref> legacy|platform|default, a console API only an operator may call, and they persist in the roster.

A group-level role (A11-1). Each region's group has its own entry in the roster: the Region struct already carries GroupName and GroupARN. That entry gains a default role for its boxes, normally default, set with carlos system group role <region> legacy|default.

• The console's enrolment pass (§5) gives every box it enrols from that group the group's role, in the same roster write that enrols it.
• So a replacement launched after the group was excluded enrols already excluded, and is never placeable even for a moment.
• Changing the group's role also changes the role of every box currently enrolled from it, in one roster write. Spec 4's drain later moves a legacy box's instances to the group, and then the box is retired.

Rebalancing: none automatic. An operator moves an instance between boxes in a region with spec 1's relocation. Automatic rebalancing is spec 4's.

Failover (spec 3) uses this same choice for each displaced instance.

8. Single-box assumptions to close

Each is closed by a code change, or by showing it's safe under claim mode, before §3's third prerequisite holds. The list comes from Astra's round 1 and a grep of main. The plan's first task repeats that audit for every reader of the local registry (registry.Open in console and housekeeper code), because those are the places that see only one box.

| Where | Assumption | Plan | |---|---|---| | internal/console/apiinstancemove.go:614-618, 671, boxselect.go:114-118, internal/host/movesphase1.go:517 | The move door refuses a region with more than one box, and selectMoveBoxes applies the placement predicate to both regions | Moves' code; ask on #236. The source comes from the claim's executor, validated only as enrolled, active and unfenced, never against placement eligibility, so instances on legacy and platform boxes can still move out (A10-3). Only the destination gets §7's choice. Tests cover outbound moves from both excluded roles | | internal/console/seedfloor.go:25-29, 88 | Seed import refuses a region with more than one box | Ask the seeded-instances owner: import onto §7's choice | | internal/hostfence/onebox.go:9-30 | CheckOneBoxRegion | Read its callers. Each caller either requires one box, and is refused in two-box regions by design, or gets the region's box set | | internal/activator/activator.go:2264-2272 | The local database copy is a warm cache only while a region has one box | H13: under the claim every admission is cold, and the warm path is never used. Verify that the claim-mode activator never reads this cache, and add a test | | internal/console/apiinstances.go:523-527 | Hosts are unique to one box | Holds under the claim (one executor). Add a test | | internal/console/view.go:177-237, server.go:1120-1147 | EIP list as A-record targets | §6: the region ingress list | | internal/edge/homemap.go:549-613, edge.go:2779, internal/host/agent.go:2951-2963, edgesfile.go:589, 806, 823 | Forwarding dials each box's published public EIP, one hop | §6: dial the private address within a region and the box's own public address between regions; publish only an address IMDS reports | | cmd/console/main.go:1390-1400 | consoleBoxLabel: exactly one box | Holds: the console runs on the platform box only. Document it | | Flagship tofu outputs.tf:39-68, dns*.tf, keymail.tf, mail.tf, scanner.tf | One instance and EIP per region | Rewrite for the group, the NLB EIP and the platform box | | internal/console/apirestartscope.go:45 | Scoped restart refuses hosts the console's box doesn't know (A1-9) | Resolve the host's executor from the claim and act through it | | internal/console/domainattach.go:150 | Domain attach looks for the primary route in the console's local registry | Read routes from the deployment-wide host records, checking ownership | | internal/console/economicspass.go:308 | Processes on other boxes are attributed to platform overhead | Attribute by each box's published usage and the claim's executor | | internal/status/fleet.go:154 | Fleet app and service listings use the elected publisher's local routes | Fold every box's edges file | | internal/host/retention.go:49 | Retention lists apps from the housekeeper's local registry | List from the deployment-wide host records | | Flagship scripts (bootstrap-box.sh, update-platform-binary.sh, harden-*.sh, roll-console.sh, verify-console.sh, rollout-status-flagship.sh, the woodstar and relay scripts) | One instance ID, or a hard-coded one | Group boxes: replaced by §5 and carlos system update. Platform box: keep the scripts, pointed at the platform box |

9. Rollout

Order: dev, then UK, US, AU and IE, one region at a time. Each region needs §3 to hold first.

Per region:

1. Prepare addresses. - Allocate the old box's new EIP, unattached. It's the old box's own address after the cut-over. - IE only: lower the TTLs of relay.eu-west-1.oncarlos.com and the MX target to 60 seconds, and wait at least twice their old TTL before the cut-over. Set the new EIP's reverse DNS once its forward record exists, and add it to SPF beside the old address (A1-4, A2-6).
2. Build. - Write the SSM parameters. - Apply the module in group mode. The NLB is created with only zone B's mapping, carrying its new EIP. - The two group boxes boot, join, enrol and reach hosting mode. The NLB is rehearsed through zone B's address. That checks zone B's targets only: zone A's are first exercised after step 3. - The old box keeps the region's EIP and all public traffic. It's also registered as an NLB target, and stays one: it keeps serving its own instances and forwarding.
3. Exclude the old box from placement (H25): carlos system box role <ref> legacy, or platform in IE. New instances now land on the group boxes, reached through the old box privately until the cut-over. The old box's own instances are untouched.
4. Cut over the EIP. 1. Disassociate the region's EIP from the old box, and immediately associate the old box's new EIP with it. In IE, also switch the relay and MX records to the new address and restart the relay with that address advertised. 2. Republish at once: the old box publishes its new address with a forced edges republication. Other regions' maps pick it up within a minute. Until then, a cross-region forward to one of the old box's instances dials the region's EIP, lands on the NLB, and is refused, because a forwarded request isn't forwarded again. That's inside the window. 3. Add zone A's mapping to the NLB with the region's EIP (SetSubnets, which accepts an EIP when adding a mapping). 4. The cost: the region's public address is down from step i until the zone A node is active, plus up to a minute more for cross-region requests to the old box's instances. Expect a few minutes in all, scheduled at a quiet hour and announced. The window is measured on dev first. 5. In IE, before the window closes: check that mail arrives on the new address, and that a TURN allocation advertises it and carries media.
5. Accept. After seven days with no rollback, the region is done. The old box stays, serving its instances, until spec 4's drain.

Rollback (A1-13, A10-1, A10-2). Rollback restores the ingress. It never destroys a group that has hosted an instance. Retiring a group means moving its instances, which is a drain, and that's spec 4's. The region's EIP is a standalone aws_eip with prevent_destroy, so the allocation is never lost.

• Before any group box reaches hosting mode (step 1 failing): destroy the group and NLB freely. Nothing has been placed on them.
• After that, before step 3: - set the old box's role back to default, so new instances can land on it again; - set the group's role to legacy (carlos system group role <region> legacy). That excludes both current group boxes and every replacement the group launches later (A11-1). - The group boxes keep whatever they host. They stay in service, behind the old box's forwarding, until spec 4's drain retires them.
• From step 3 until acceptance: 1. Remove zone A's mapping from the NLB. An address on a requester-managed interface can't be disassociated directly. The NLB itself is kept, because the group boxes are its targets. 2. Wait until the EIP shows as unassociated. 3. Swap the old box's addresses back: disassociate its new EIP, associate the region's EIP, and force a republication. In IE, switch the relay and MX records back and restart the relay advertising the region's address (A2-9). 4. Set the roles as for "before step 3", the group's role included. Group instances stay where they are, reached through the old box privately.
• After acceptance: recovery is forward, within the group.

Cost.

• Every region goes from one box to three until spec 4's drain retires the old box: the old box plus two group boxes. It also gains an NLB (about $20 a month before traffic) and two new EIPs (zone B's, and the old box's own).
• After the drain: two group boxes per region, plus IE's platform box.
• Dev matches whichever layout it rehearses.

10. Failure modes

| What happens | Result | |---|---| | A group box hangs or fails its health check | The NLB stops sending it traffic within about 20s, and the group terminates it once its health check has failed (after the grace period, for a new box). Its instances are down until spec 3 relocates them. A replacement boots, enrols and becomes placeable with no hand steps. | | A group box is terminated by AWS | As above, without the health-check delay. | | An old box (legacy or platform) dies | Its own instances are down, as today. EC2 auto-recovery covers hardware faults. If AWS terminates it, spec 3's failover relocates them. Otherwise it's an operator job. In IE, the console and platform services are down too (S1 §12). Instances on the group boxes are unaffected. | | Both group boxes in a region fail | The region's instances are down. The group replaces both. The console confirms each replacement's membership itself, so an emptied region recovers like any other (H28). | | The NLB's availability zone fails | The region's ingress is down until the zone recovers (§6). | | A replacement can't boot | It's never healthy, the group keeps replacing it, and the eligible-capacity alarm fires at 25 minutes. | | A box boots but can't enrol | It stays in joining mode, its health answer stays 503, and it takes no traffic. The group replaces it at the end of the grace period, except during a region's first build (§5). The eligible-capacity alarm fires at 25 minutes either way. | | One zone is unavailable | The NLB serves through the other zone's node only for clients that use zone B's address, so the region's published address (zone A) is down until the zone recovers. The group doesn't move boxes between zones (AZRebalance is suspended). | | A forwarding mesh key or map refresh stalls on a box | Its health answer turns 503, and the NLB stops sending it traffic. If the stall lasts past the health-check threshold, the group replaces it. | | No box has headroom | New instances are unplaced, and an alert fires (§7). | | The housekeeper's box dies | Certificate issuance pauses for up to 15 minutes, and serving continues. |

11. Testing

• Tofu. The module's tests cover group mode: the launch template, group, NLB, target groups, the health check path, IAM scoped to the parameter path, the alarms, and fixed-mode attachment to the target groups. Fixed mode stays identical when group mode is off.
• Boot. The start-up script is tested on dev from a clean launch, and every failure path is checked: bad sha256, missing parameter, failed hardening. Each must leave the box unhealthy. With a box's credentials, a write under control/box-enrollment/ and an Auto Scaling describe are both denied.
• Enrolment. Unit tests for the console's membership read against a fake reader: a member, a non-member, a terminated instance, a read that fails (unknown, never a refusal, and the attempt kept), a read over a minute old at its write, a store call that ignores its context (cut off at the hard timeout, unknown), a write that answers success after the deadline (unknown, settled by the next pass's roster read), a replayed attempt, and no exported path that takes a confirmation. Tests that joining mode never publishes an edges file or admits an instance. A test that the full sequence (confirm, invite, guard-ready, enrol, hosting) is satisfied against membership.go's rule when a floor is published. On dev, a real launch enrols with no operator.
• Install. A fresh box installs the channel release into both protected slots, never the bootstrap binary, refuses a release below the host floor, and then passes guard-arm.
• Mode selection. A restart at each joining state (no invitation, invited, guard installed, guard-ready reported) resumes joining. Only an active roster entry plus an armed guard starts hosting. A group box stopped and started since its enrolment is held; one rebooted hosts. On dev, both are tried once.
• No capacity. A create with every box ineligible is refused with 503, writes nothing to the claim, and alerts at most once per region per hour.
• Health answer. Edge tests for /.carlos/lb-health: any host, exact path, GET and HEAD only, and 503 for each failing condition (joining mode, forwarding never armed, no certificate cache, a stalled heartbeat). A test that an unreachable bucket after arming leaves the answer at 200, because failed refresh attempts still count as progress, and that a refresh worker blocked inside one attempt turns it 503.
• Forwarding. - Private-address forwarding within a region, and own-public-address forwarding between regions. - A reader without private_ip dials the public address. - The publisher publishes only what IMDS reports, and publishes no public address on a blank read. - A forwarded request is still never forwarded again. - Platform routes forwarded from a group box to the platform box.
• Placement. Unit tests for each eligibility rule, the spread preference, the headroom ordering, the stale-headroom cut-off, and the platform and legacy roles.
• Rehearsal on dev: - the full §9 sequence, with the cut-over window measured, the zone-B rehearsal, and both rollback sequences run once. Rollback leaves the group in service with its instances, including one that was never woken; - outbound moves from a legacy box and from the platform box; - a group box replaced after rollback, and one already joining during rollback: each enrols with the group's legacy role and is never placeable; - the old box's address swap at the cut-over: forced republication, and cross-region forwards to its instances recovering within a minute; - the legacy and platform roles excluding a box from creates, relocation targets and move targets, persisting across console and box restarts, and an operation already heading to the box finishing there; - one group box terminated by hand, with replacement and re-enrolment observed; - a box made unhealthy, observed being terminated.
• Failover drills belong to spec 3.

12. Out of scope

• Spec 3: relocating a terminated box's instances (the placer loop), waking always-on and scheduled instances, the failover gap target. Spec 2 hands it one requirement (B4-2, platform-1e): the regional worker reconciles roster membership against Auto Scaling group membership, including detached-but-running instances. Spec 2 enrols a box only on the console's own membership read, but removes nothing from the roster: an instance detached from its group that keeps running stays enrolled until an operator removes it or spec 3 reconciles it.
• Spec 4: - automatic scale-out and scale-in, and rebalancing; - offering zone B's ingress address to customers, so the region's published addresses span both zones (§6). The NLB already has a node in each zone; - latency steering; - H5's per-region minimum as anything other than a fixed group size; - the Auto Scaling activity-history evidence source (S1 A2-4); - draining a live box (H25): moving a legacy box's instances to the group and retiring it, which also empties IE's platform box of customers. Spec 2's rounds 6 to 9 are its starting inputs (§22): a durable drain control, a settlement protocol that acknowledges outstanding requests rather than waiting out time, first-wake support for never-woken instances, and claim box references sorted into live and historical.
• Not planned: relocating the console or platform services (S1 §12). The platform box is fixed, and its HA remains the HA ladder's.

13. Questions from rev 1

All three were answered in Astra's round 1 and folded in:

1. Fresh headroom: 15 minutes, because of the 5-minute publication interval (§7, A1-8).
2. Rehearsing the NLB: create it with zone B's mapping only, then add zone A's with SetSubnets (§9, A1-§13).
3. SSM and KMS: one configuration key per deployment, with ViaService and encryption-context conditions, and the explicit Deny kept (§4).

14. Astra round 1 dispositions (rev 2)

| Finding | Disposition | |---|---| | A1-1 P1: one enabled zone leaves the second box unused | Accepted: two subnet mappings, the existing EIP in zone A and a new EIP in zone B (§4) | | A1-2 P1: boot and enrolment are circular | Accepted: joining mode, then confirm, invite, guard-ready, enrol, hosting (§5) | | A1-3 P1: AZRebalance can terminate a healthy box | Accepted: AZRebalance suspended (§4) | | A1-4 P1: relay and MX break at the cut-over | Accepted: IE step 0 moves them first and waits for TTLs (§6, §9) | | A1-5 P2: an edges file makes a joining box referenced | Accepted: joining mode publishes no edges file (§5) | | A1-6 P2: carlos bootstrap doesn't install the release | Accepted: an explicit signed channel-release install, below-floor refusal (§5) | | A1-7 P2: unplaced has no claim contract | Accepted, by keeping it out of the claim: a pending create in the console, started once an executor is chosen (§7) | | A1-8 P2: the freshness cut-off is below the publication interval | Accepted: 15 minutes (§7) | | A1-9 P2: local-registry consumers | Accepted: five rows added to §8, plus a full audit as the plan's first task | | A1-10 P2: one map load doesn't prove readiness | Accepted: five live conditions (§6) | | A1-11 P2: pending enrolment is invisible to alarms | Accepted: the eligible-capacity metric and alarm, a 503 health answer until hosting, and a grace period that covers enrolment (§4, §5) | | A1-12 P2: a published address can be the NLB's | Accepted in rev 2; superseded in rev 3: no box publishes a public dial address (§6) | | A1-13 P2: rollback can't reassociate an NLB-owned EIP | Accepted: a standalone EIP with prevent_destroy, a seven-day acceptance window, and an explicit release-then-associate sequence (§9) |

15. Astra round 2 dispositions (rev 3)

| Finding | Disposition | |---|---| | A2-1 P1: enrolment omits the attempt and confirmation expiry | Accepted: a first attempt to decide on the invitation, then a fresh attempt, confirmation and EnrollAutomatic within the minute, repeated on expiry (§5) | | A2-2 P1: the bootstrap binary as .prev blocks guard-arm | Accepted: both slots hold the verified channel artifact (§5) | | A2-3 P1: a bucket outage would terminate working boxes | Accepted: the health answer is local liveness only, with forwarding latched once armed (§4, §6) | | A2-4 P2: a restart after writing host-actor.json | Accepted: mode selected from roster and guard, with resumable joining states (§5) | | A2-5 P2: the pending-create handoff isn't atomic | Accepted: a starting state with a stable operation identity, reconciliation by operation ID, and a shared hostname reservation (§7) | | A2-6 P2: the relay advertises its start-up address | Accepted: the relay is configured and restarted on the new address inside the cut-over window, and a TURN allocation and media exchange are checked before it closes (§6, §9). Rev 3 also moves IE's relay and MX swap into that window instead of a second address beforehand, which would need a secondary private IP | | A2-7 P2: no way to publish a second address | Superseded by the forwarding change: no box publishes a public dial address (§6) | | A2-8 P2: no permission for the capacity metric | Accepted: a namespace-scoped grant on the platform box, missing data as breaching (§4) | | A2-9 P2: rollback removes needed addresses | Accepted: forwarding doesn't depend on per-box addresses, and IE's step 0 isn't rolled back (§9) | | Round 1 partials | A1-1: §6 and §12 wording updated. A1-11: the first-build exception stated (§5). A1-12 and A1-13: superseded by the forwarding change |

16. Astra round 3 dispositions (rev 4)

| Finding | Disposition | |---|---| | A3-1 P1: a failed hop credential resets the forwarding budget | Accepted: a version 2 credential binding the count and region, the current and previous key accepted, and a failed credential never forwarded again (§6) | | A3-2 P2: removing published addresses breaks old readers | Accepted: three stages (publish both, switch by deployment setting once every box reports version 2, then drop eip), with stage 2 a prerequisite (§3, §6) | | A3-3 P2: a stalled refresh worker isn't detected | Accepted: the worker must complete an attempt, successful or not, within 5 minutes (§6) | | Stale summaries | §10's enrolment-failure row states the first-build exception, and §12 says what is actually deferred about zone B |

17. Astra round 4 dispositions (rev 5)

| Finding | Disposition | |---|---| | A4-1 P2: stage 2 needs a release floor, not an advertisement | Accepted: the host floor is raised to the version 2 release, and stage 2 needs every enrolled box's guard armed for it, which also blocks rollback and recovery below it (§6) | | A4-2 P2: "current and previous" conflicts with mesh-key rotation | Accepted: the existing current-and-.next protocol is kept and named, with its full rotation sequence tested (§6) |

18. Astra round 5 dispositions, and the rev 6 simplification

| Finding | Disposition | |---|---| | A5-1 P2: the host floor can't be raised once published | Moot: rev 6 has no new forwarding protocol, so there's nothing to gate on a floor | | A5-2 P2: early .next signing fails during staggered refresh | Moot: rev 6 needs no second hop. Today's single hop goes straight to the home box, and today's rotation behaviour is unchanged | | A3-1, A3-2, A4-1, A4-2 | Moot for the same reason. Rev 3's version 2 credential, three-stage rollout and floor gate are removed | | A2-7, A2-9 | Still met: a box publishes only an address IMDS reports, and instances leave the old box before its address moves (§6, §9) | | A3-3 refresh-worker liveness | Kept (§6) |

19. Astra round 6 dispositions (rev 7)

| Finding | Disposition | |---|---| | A6-1 P2: migration doesn't stop new placements on the old box | Accepted: draining is set first, starting selections are settled, and emptiness is checked before the cut-over and before termination (§9) | | A6-2 P2: IE rollback targets a box the policy forbids | Accepted: the IE box is draining, not platform, until acceptance, and rollback clears draining (§5, §9) |

20. Astra round 7 dispositions (rev 8)

| Finding | Disposition | |---|---| | A7-1 P2: the draining control is still spec 4's | Accepted: spec 2 builds it, as a roster field written by an operator-only console API and CLI. It's persistent, and placement reads it directly (§7) | | A7-2 P2: settlement misses immediate creates | Accepted: a settlement barrier over in-flight console creates, incomplete claim creates by selected_executor, starting pending creates and completed hosts. It's run before relocation, before the cut-over and before termination (§7, §9) |

21. Astra round 8 dispositions (rev 9)

| Finding | Disposition | |---|---| | A8-1 P2: the barrier misses moves and relocations heading to the box | Accepted, generalised: one rule over every claim box-reference field, enforced by a coverage test, plus in-flight requests from the console doors and box-side starters (§7) | | A8-2 P2: relocation refuses never-woken instances | Accepted: each is woken once on the old box before relocation, and a failure blocks the emptiness check visibly (§7, §9) |

22. Astra round 9 dispositions, and H25

| Finding | Disposition | |---|---| | A9-1 P2: a timed wait doesn't settle box-side starters | Moot under H25: nothing has to be empty. Recorded for spec 4's drain | | A9-2 P2: a never-woken claim can't be woken by the normal path | Moot under H25. Recorded for spec 4's drain | | A9-3 P2: the barrier needs migration to have finished | Moot under H25. Recorded for spec 4's drain | | A9-4 P2: historical box references never clear | Moot under H25. Recorded for spec 4's drain | | A6-1, A7-1, A7-2, A8-1, A8-2 | Their machinery (the drain control, the settlement barrier, the one-off wake) is removed from spec 2 and recorded as spec 4's inputs. A6-2, IE rollback, is met differently: the IE box isn't drained, so rollback moves no instances back to it |

23. Astra round 10 dispositions (rev 11)

| Finding | Disposition | |---|---| | A10-1 P2: destroying the group during rollback needs settlement | Accepted: rollback never destroys a group that has hosted an instance. It sets roles and reverts ingress, and retiring the group is spec 4's drain (§9) | | A10-2 P2: rollback can't relocate a never-woken group instance | Accepted by the same change: rollback moves nothing (§9) | | A10-3 P2: moves need source validation separate from destination eligibility | Accepted: the source comes from the claim and is checked only as enrolled, active and unfenced. Only the destination uses §7 (§8) | | A10-4 P3: §6 still requires emptying the old box | Accepted: §6 describes the address swap (§6) |

24. Astra round 11 dispositions (rev 12)

| Finding | Disposition | |---|---| | A11-1 P2: rollback's exclusion doesn't survive box replacement | Accepted: a role on the region's group entry in the roster, inherited at enrolment in the same write, and applied to current members when changed (§7, §9) |

25. The provider boundary (rev 14)

Spec 2 is AWS-first because carloku.com runs on AWS. The platform isn't: a self-hosted deployment is one object-store bucket (S3-compatible, MinIO included), N hosts running carlos, and DNS.

What's true on main before spec 2 (A13-1, A13-2). Multi-box regions already need AWS. Under the claim, a box's identity is AWS's signed instance identity document:

• enrolment requires an AWS region and verifies the AWS-signed evidence (internal/hostactors/enroll.go:29);
• roster validation requires Cloud.Provider == "aws" (internal/hostactors/roster.go:111);
• hosting re-verifies it (internal/hostbox/identity.go:45);
• placement eligibility requires an AWS region and Cloud.Complete() (internal/boxchoice/boxchoice.go:110, internal/boxid/verify.go:30).

A deployment without AWS runs one box per region with claim mode off, unaffected by any of this. H26: multi-box stays AWS-only until a provider-neutral identity spec (#299) replaces that identity. Operator approval alone can't stand in for it, because the claim re-verifies identity at every admission.

What spec 2 adds, and on which side it sits. Spec 2 writes everything it adds so that #299 only has to replace identity:

| Spec 2 adds | Side | Notes | |---|---|---| | Placement policy, box and group roles, pending creates (§7) | Core | Spec 2's new rules live in a function that takes candidates already filtered for identity. #299 then changes only the identity filter (boxchoice.Eligible's AWS checks), not the policy | | The health answer /.carlos/lb-health (§6) | Core | Plain HTTP | | Forwarding by private address in a region, own public address between regions (§6) | Core, with an address source | The AWS adapter reads IMDS. The core fallback is CARLOS_PRIVATE_IP and CARLOS_PUBLIC_IP in host.env, or the primary interface's address | | Joining and hosting modes, and the console's enrolment (§5) | Core, with a membership reader | The console confirms membership through a reader interface; its AWS adapter is the Auto Scaling and EC2 describe (H16, H28). Joining itself names no provider | | host.env assembly from SSM and KMS (§5) | AWS adapter | host.env stays a file. SSM is one way of writing it | | The channel-release install and guard (§5) | Core | They read the bucket and the release key | | The region-host module's group mode (§4) | AWS adapter | Tofu, AWS by nature | | Termination evidence (S1, spec 3) | AWS adapter | |

Bucket access (A13-3). Every bucket client spec 2 adds is built with s3compat.Options(), so CARLOS_S3_ENDPOINT applies, exactly as the existing clients do. One existing claim client doesn't: the generation-marker client (internal/host/claimproduction_marker.go:21). That's filed as #298 and fixed in claim code, separately from spec 2.

Enforcement.

• An import test: spec 2's new core files import no github.com/aws/... package. Adapters live in clearly named files or packages, for example internal/host/awsaddr.go and internal/hostactors/awsmembership.
• A test that every s3.NewFromConfig call site in the packages spec 2 touches passes s3compat.Options().
• Unit tests for each core piece with the AWS adapters absent: the address source from host.env, the health answer, the placement policy over identity-filtered candidates, and the console's enrolment with a fake membership reader.
• A full multi-box lifecycle test without AWS waits for #299, because it needs a non-AWS identity.

26. Astra round 13 dispositions (rev 14)

| Finding | Disposition | |---|---| | A13-1 P1: operator approval doesn't remove the AWS identity requirement | Accepted as a fact about main. Under H26, multi-box stays AWS-only until #299, and §25 says so | | A13-2 P2: placement eligibility excludes non-AWS hosts | Accepted: spec 2's policy runs over identity-filtered candidates, so #299 changes only the filter. The import test doesn't claim to cover this (§25) | | A13-3 P2: the marker client ignores CARLOS_S3_ENDPOINT | Accepted: a bug on main, filed as #298 for the claim's code. Spec 2's own clients use s3compat.Options(), with a call-site test (§25) |

27. H27 (rev 15)

| Item | Disposition | |---|---| | A1-7, A2-5 (the unplaced claim contract, the pending-create handoff) | Superseded by H27: no pending state exists, and a refused create writes nothing | | §7's pending creates and the placement retry pass | Removed. The eligible-capacity metric is published by its own console pass |

28. Astra plan 2b round 1 dispositions (rev 16)

Findings from Astra's review of plan 2b's first draft that change this spec. The rest (B1-2, B1-4, B1-6, B1-8) are the plan's alone. B1-3 (box references outside the label space) waited on a census ruling in rev 16; rev 17 builds it (§29).

| Finding | Disposition | |---|---| | B1-1 P1: box-written confirmations don't authenticate their writer | Accepted, by H28: the console is the only confirmer, reading membership itself immediately before it invites and before it enrols; no box writes evidence or holds a describe or control/box-enrollment/ grant (§2, §4, §5 step 5, §10, §11, §25) | | B1-5 P2: the boot order runs a binary before fetching it | Accepted: the bootstrap binary is fetched and checked with the AWS CLI first, then assembles host.env, then bootstraps and installs (§5 steps 2 and 3) | | B1-7 P3: an unreadable confirmation supersedes a valid attempt | Accepted: a read that fails is unknown, keeps the attempt and doesn't grow the backoff; only a positive non-member answer or a stale read counts against it (§5 step 5) | | The claim owner's conditions on H28 (C1 to C4) | Accepted: the confirmation can't be supplied by a caller; a 60-second bound at the moment of each write, with request ids logged; a group box started again is held and replaced (§5); the console's describes are region-limited and a box has none, checked by 2c's infrastructure test (§4, §11) |

29. Astra plan 2b round 2 dispositions (rev 17)

Findings from Astra's second review of plan 2b that change this spec. B2's partly open B1-2 (ownership on a retry) is the plan's alone.

| Finding | Disposition | |---|---| | B2-3 (B1-3 open): label collisions; activation must wait for the fix | Accepted, with platform-1e's ruling: group references, derived by the console from the proven instance, and statusLoop re-sealed to publish under them; automatic enrolment gated on their release (§5 step 5) | | B2-new-1 P2: freshness checked before intervening I/O, not at the write | Accepted: the read sets a 60-second deadline on the write itself, enforced on every store call of the sealed writers; a missed deadline is unknown (§5 step 5) | | B2-new-2 P2: the seeding contract omits decryption | Accepted: seeding's permissions name kms:Decrypt, conditioned as the box's is, and the SSM read (§5 step 2) | | B2's note on C3: pendingTime is documented as launch time, not the exact stop/start invariant | Accepted as an activation gate: verified on dev before the first region (§5 step 5) |

30. Astra plan 2b round 3 dispositions, and the census ruling's conditions (rev 18)

| Finding or condition | Disposition | |---|---| | B3-1 P2: an unresolved group reference overwrites another box's status | Accepted: a group box publishes no status until its reference resolves, never under a status label, asked again every tick (§5 step 5) | | B3-2 P2: replaced boxes' reports exceed the status reader's key cap | Accepted: the reader drops reports no box has rewritten in a day before its cap; a 30-day expiry as hygiene (§5 step 5) | | B3: the two redundant freshness checks | Accepted: removed; the deadline-bound store is the single enforcement point (§5 step 5) | | The claim owner: the deadline's anchor and edge, and the read-back | Accepted: anchored no later than the describe's completion, refused at 60 seconds or more, in-flight calls cancelled and unknown; a landed unknown enrolment accepted only on a fresh describe, paged once if no longer a member (§5 step 5) | | platform-1e: host.env's conditions | Accepted: box roles never write the parameters; an allowlist of names, safe values, a whole write through a checked directory; the group marker and CARLOS_REGION checked against the roster before a group box hosts (§5 step 2) | | platform-1e: the join trigger | Accepted: the path unit's file is only a trigger; root reads nothing from it |

31. Astra plan 2b round 4 dispositions, and two simplifications (rev 19)

| Finding or ruling | Disposition | |---|---| | B4-1 P2: a success that returns after the deadline escapes the bound | Accepted (U2): every store call of a bound write has a hard 15-second timeout; the wrapper returns on its own when the store doesn't answer, and a call that answers after the deadline, even with success, is unknown. Rev 20 corrects the bound this row claimed: the timeout bounds the console's waiting, not the commit (§5 step 5, §32) | | B4-2 P2: the read-back accepts a stale describe | Superseded (U1): the read-back is removed. An unknown outcome is settled by the next pass's roster read, under the same rules as any other; spec 3 reconciles detached-but-running instances (§5 step 5, §12) | | B4-3 P2: a failed page discards the only escalation | Superseded (U1): the page is removed with the read-back; no page is sent, so none is lost. The requirement moves to spec 3 (§12) | | B4-4 P2: host.env's directory check doesn't protect the write path | Accepted (U3): the path is fixed; /etc and /etc/carlos are held by descriptor, without following symlinks, and refused unless only root can write them; the file is written and renamed inside the held directory, and the directory synced. Off Linux the write refuses (§5 step 2) | | platform-1e: spec 3's requirement | Accepted: "the regional worker reconciles roster membership against Auto Scaling group membership, including detached-but-running instances" (§12) |

32. Astra plan 2b round 5 dispositions, and platform-1e's acceptance of revision 4 (rev 20)

| Finding or ruling | Disposition | |---|---| | B5-1 P2: the timeout does not establish a 75-second commit ceiling | Accepted: the claim is withdrawn and no mechanism added. The guarantee: every store call of the invite and the enrolling write is issued before the deadline (60 s from the start of the membership describe); the console waits at most 15 s for any call; a write whose result the console did not see is unknown and is settled on a later pass by reading the roster. A cancelled write can still commit later; such an enrolment rests on a membership read that was true when the write was issued, and is the same exposure as an instance detached after enrolment, which spec 3's worker reconciles. (§5 step 5) | | platform-1e: the per-call context reaches the SDK | Accepted: plan 2b tests it through the console's production stores, down to the HTTP request | | platform-1e: late commits land only where they were aimed | Confirmed from main's code: the roster write is a compare-and-swap and an invitation is create-only, through every wrapper of the console's store (§5 step 5) | | platform-1e: the two host.env units | Ruled A,J4 under census ruling region-ha-2b |

33. Astra's final review of the plan 2b build (rev 21)

| Finding | Disposition | |---|---| | BF-1 P2: a stalled membership read stops enrolment across all regions | Accepted: each describe gets the 15-second cut-off each bound store call has; a describe cut off is unknown, the attempt kept, its backoff unchanged, and the pass goes on to the next box (§5 step 5) | | BF-2 P2: accepted status overrides break the group-report contract | Accepted, refused up front: CARLOS_STATUS_BUCKET, CARLOS_STATUS_PREFIX and CARLOS_STATUS_REGION leave host.env's allowlist. A group box reports to the deployment bucket, unprefixed, where the console reads. A deployment that needs a separate status bucket can't use group boxes until joining, hosting and the console share that destination (§5 step 2) | | BF-3 P3: the allowlist admits credentials embedded in webhook URLs | Accepted: CARLOS_ALERT_WEBHOOK leaves the allowlist, on the rule that keeps the DNSimple token off it. Group boxes alert through SNS until a reviewed exception or a credential store exists (§5 step 2) | | BF, before activation: the guard's restart transition | Accepted as activation gate 3: on dev, a fresh group box goes guard-arm, then carlos-system-recover, then guard-exec across the joining-to-hosting restart, and comes up hosting. The stop/start and reboot check stays gate 2 (§5) | | BF, before activation: the allowlist against real boxes | Activation gate 4: carloku.com's host.env files checked against the allowlist before the first seed (§5) |