Region HA 2a: The Provider-Neutral Core Implementation Plan (rev 4)

> For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Build region HA spec 2's provider-neutral core, dormant behind CARLOS_REGION_HA: box and group roles, the placement policy, H27's no-room refusal on every door that chooses a box, the load balancer health answer, private-address forwarding with a validated address source, and the eligible-capacity metric.

Architecture: Every new rule sits on spec 1's code without changing what it proves. The placement policy runs over boxchoice.Eligible's identity-filtered candidates inside hostops.ExecutorChooser, which creates and relocation targets share, and which the create and move doors ask whether a region has room; roles live in the signed roster. A create door chooses its box before its first write, so a refusal leaves nothing behind. The health answer is an in-memory state (internal/lbhealth) that the agent's own loops feed and the edge reads on :80. Forwarding changes only the address an edge dials. Each AWS part is an adapter in a file named for AWS (internal/host/awsaddr.go, internal/console/awsregionmetric.go), and a test holds the core to that.

Tech Stack: Go 1.26, module github.com/carlosframework/platform; internal/hostactors, internal/boxchoice, internal/edgesfile, internal/hostops, internal/host, internal/edge, internal/activator, internal/console, the carlos CLI (repo root); AWS SDK v2 CloudWatch in one adapter.

Spec: docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md, rev 15 (approved), which records Paul's ruling H27 (no pending creates; a refused request writes nothing and alerts; choose before registering; the four choosing doors; the alert limit is per console process) and settles §6's certificate readiness as the mirror's first load. Spec 1, for context: docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md.

Rev 2: Astra round 1 and census ruling region-ha-2a

Rev 2 answers Astra's review of rev 1 (docs/superpowers/plans/CODEX-REVIEW-REGION-HA-2A-1.md, seven P2s, all accepted, with the coordinator's rulings now in spec rev 15's H27) and platform-1e's census ruling region-ha-2a (approved in principle with conditions, amadan #236).

| # | Finding or condition | Rev 2 | Where | |---|---|---|---| | A1 | The no-capacity path left a registration behind; the test called StartCreate directly | Every create door chooses before RegisterStart: beginClaimCreate calls chooseExecutorFirst, which refuses before anything is registered and otherwise carries the choice to the create (hostops.WithChosenExecutor, honoured by SelectExecutor). Tested through the HTTP door and a recovery pass: no registration, no claim, nothing written by recovery | Task 11 | | A2 | H27 was missing from the other choosing doors | One refusal (noCapacityError, answered by claimResponseRefusal) on instance create, domain attach's first route (API and browser, through attachClaimFirstRoute, unchanged), standalone sidecar create, and the move door's destination. Each has an HTTP or door-level test. Sidecars attached to an instance choose nothing and are unchanged | Tasks 11, 12 | | A3 | Instance create's earlier region check refused before the chooser | With region HA on, an app-derived region is judged by the chooser's eligible set (PlaceableBoxes); an empty set is the no-room refusal. HTTP tests: every box stale (503 and an alert, not 400), and a misconfigured sibling beside an eligible box (the create goes ahead; 400 with region HA off, as on main) | Task 11 | | A4 | Move role filtering ran with region HA off | The move destination is filtered only through placementChooser, which exists only with region HA on; no roster is read otherwise. Test: host claims on, region HA off, a legacy destination and an unreadable roster, both main's answer | Task 12 | | A5 | dialAddress used private addresses with the reader's region HA off | The home map dials private only with the reader's own switch on (homeMap.privateDial). Tested both ways | Task 8 | | A6 | The alert limit did not survive a restart | H27 (rev 15) makes the limit per console process and accepts one extra alert after a restart. The limiter stays in memory; the wording says so | Task 11 | | A7 | Pending copy reached CLI help and docs | The copy gate is real: an id in regionHACopyBuilt must be an approved row. No pending row is ever marked approved for checking. Tasks 10 and 11 wait for their rows' approval | Tasks 1, 10, 11 | | B1 | private_ip: root-configured source; validated (IPv4, private, inside the region's VPC CIDR, not loopback or link-local); omitted and refused otherwise; routing data only; tests | The source is IMDS on AWS (the VPC's CIDR blocks from network/interfaces/macs/<mac>/vpc-ipv4-cidr-blocks) or host.env (CARLOS_PRIVATE_IP with a required CARLOS_PRIVATE_CIDR). edgesfile.PrivateAddress validates on both sides: the writer omits what fails, and a reader dials a private address only when it passes against the reader's own published network. Tests: a malformed, a metadata-range and an out-of-CIDR address, each omitted and refused | Tasks 7, 8 | | B2 | Server.boxRoster: operator-only, read-only, the same roster source, no role feeds root's claim authority | boxRoster is called only by the two role doors, past apiSystemRoller; the move door no longer reads roles through it (it uses the chooser) | Task 10 | | B3 | The overlay loads last; values recomputed on the actual build base, after Plan 14 Parts 2 and 3 | Stated in the census section below; the recompute tooling is kept | Census |

Task numbers moved: the move door is its own task (12), and the metric is now Task 13.

Rev 3: Astra round 2 and census ruling region-ha-2a rev 2

Rev 3 answers Astra's round 2 (docs/superpowers/plans/CODEX-REVIEW-REGION-HA-2A-2.md: rev 2's findings closed but one, and two new P2s, all accepted) and platform-1e's ruling region-ha-2a rev 2, which approved items 3 and 4 below with conditions.

| # | Finding or condition | Rev 3 | Where | |---|---|---|---| | R1 | The move door's earlier placement checks (a destination with no edges file, or only blank public addresses) still answered 400 with no alert | With region HA on, roomIn asks the chooser before those checks: a destination the roster places boxes in, with no box that has room, is H27's refusal (503, RHA-C17, the alert). The instance create door's armed-placement gate gets the same treatment, the same defect. HTTP tests through the whole move door (both shapes, region HA on and off, and with room) and the create door | Tasks 11, 12 | | R2 | handleAPIInstanceCreate bound the environment (bindEnvironmentCreate, which can clear a seed's mark on a store) before beginClaimCreate's early choice | The create door chooses before its first write (chooseCreateFirst, before the binding) and passes the choice through; beginClaimCreate keeps a choice the context carries and does not choose again. Tests: a refused create leaves the store's bytes unchanged and writes nothing, both when every box is stale and when the last box goes between the region check and the choice; with room, the same create takes the store | Task 11 | | R3 | The capacity metric took each region's AWS region from edges files | ExecutorChooser.Capacity also answers the AWS region the signed roster names for the placement, the mapping the chooser reads, and the metric goes there. Tests: a conflicting file (another AWS region, sorting last) does not move the count, unit and end to end | Task 13 | | C3 | Ruling item 3: chooseExecutorFirst takes no lease and makes no write; its executor is passed through and never chosen again; a refusal leaves nothing behind | TestChooseFirstWritesNothingAndChoosesOnce: no bucket or backend event while choosing, the choice is the authority's own choice for the same record, a carried choice is kept without calling the build, and SelectExecutor returns the carried choice. Every refusal test also checks that nothing was written (wroteNothing) | Task 11 | | C4 | Ruling item 4: the move room check only refuses, never picks or changes the destination | moveRoomCheck replaces rev 2's placeableMoveTargets: it refuses when no destination candidate is placeable and otherwise leaves the candidates as main's. TestMoveRoomCheckOnlyRefuses: with room the selection equals region HA off's; with none, the refusal; a legacy box beside a default one gives main's several-boxes refusal, not a pick | Task 12 | | C5 | Record-write re-digests stand only if no write site changes; a new write site needs its own line | No write site changes and none is added: the record-write census reports digest changes only (handleAPIInstanceCreate, handleAPIInstanceMoveRequest, registerCLIRoutes, Directory.Replace) | Census |

One new census item: Server.chooseCreateFirst (Task 11), a new accepted-legacy unit whose one root read is the claim read beginClaimCreate makes, made before the binding. The census tooling's unitdigest build line is corrected (rev 2 built it over its own source directory).

Rev 4: census ruling region-ha-2a item 5

Astra's round 3 (docs/superpowers/plans/CODEX-REVIEW-REGION-HA-2A-3.md) found nothing and calls the plan ready to build. platform-1e approved item 5, Server.chooseCreateFirst, in the current structure (not the hook alternative), with one condition (amadan #236), which rev 4 meets:

| # | Condition | Rev 4 | Where | |---|---|---|---| | C6 | chooseCreateFirst returns its choice with the claim observation it decided on; beginClaimCreate requires its own claim read to be that observation, and refuses or re-chooses otherwise, never proceeding on the old choice | chooseCreateFirst carries the choice and the observation (the claim's ETag, or its absence) in the context. beginClaimCreate passes its own read to chooseExecutorFirst, which keeps a carried choice only when that read is the same observation. Otherwise it refuses: errChoiceStale, which is hostclaim.ErrConflict, so the door answers main's retryable busy refusal (409, Retry-After: 2) before anything is registered or written. Refusing is the smaller change: no new copy, and no second choice to prove. TestAStaleEarlyChoiceIsNeverUsed: a claim write landing between the choice and begin is refused, with nothing chosen again, registered or written; without the check the test fails (begin goes on past RegisterStart with the old choice) | Task 11 |

Census values move with it: new digests for Server.chooseCreateFirst and Server.beginClaimCreate (kinds and read sites unchanged), their @legacy fingerprints, internal/console's @coverage row in Tasks 11 to 13, and the seal in Tasks 11 and 12. Open question 2 is closed as decided.

Scope

Spec 2 is four plans. This is 2a. The others, and the seams 2a leaves them:

| Plan | Builds | 2a's seam | |---|---|---| | 2b | Joining mode, the membership-confirmation worker, the console's enrolment pass | lbhealth.Default.SetHosting (Task 9): 2b's mode selection owns it, and joining mode leaves it false. hostactors.Directory.EnrollAutomatic already gives an admitted box its group's role in the enrolling write (Task 3) | | 2c | The region-host module's group mode, the start-up script, SSM | The health path /.carlos/lb-health on port 80 (Task 9). The metric Carlos/Region EligibleBoxes, dimension Region = the placement code, in each region's own CloudWatch, the AWS region the signed roster names for it (Task 13): 2c grants the platform box's role cloudwatch:PutMetricData conditioned on cloudwatch:namespace = Carlos/Region, and sets CARLOS_REGION_METRICS=cloudwatch on the console. The box role's IMDS access must cover network/interfaces/macs/* (Task 7), and the security group opens 443 from the region's VPC CIDR | | 2d | The §8 single-box fixes | With region HA on the move door already refuses a destination with no placeable box (Task 12), without narrowing its candidates; 2d's rework of the door (its multi-box regions, the source from the claim) builds on it |

Not in 2a: forwarding of platform routes from a group box to the platform box (§6 "Platform routes"), the region ingress list in the console (§6), and every row of §8; those are 2d's or 2c's.

Global Constraints

• Everything new is dormant with CARLOS_REGION_HA off: with it off, every door, file, edge and box behaves exactly as on main. The health answer is not served, the role doors answer 404, the edges file publishes no private_ip and keeps its EIP hold, the home map dials public addresses only, the create and move doors choose as on main (no early choice, no room check, no roster read), and the capacity pass does not start.
• Multi-box regions stay AWS-only until #299 (H26). Spec 2's new core files import no github.com/aws/... package; AWS parts are adapters in files named for AWS (§25). region_ha_boundary_test.go holds both, plus s3compat.Options() on every S3 client in the packages 2a touches.
• Wire shapes are additive: private_ip, private_cidr, mem_total_mib, disk_total_mib (edges file), group and role (roster box), role (roster region) are all omitempty, and a reader that predates them is unaffected.
• private_ip is routing data only: an edge dials it, and nothing grants anything on it. It comes only from IMDS or root-owned host.env, never from a file the agent can write, and it is published and dialled only when edgesfile.PrivateAddress passes it.
• The roster's default role is stored as "". Role values: box "", legacy, platform; group "", legacy. Roles are read by the placement chooser alone and never feed root's claim authority.
• §7's thresholds: free memory at least 15% of total (boxchoice.MinMemPercent), free disk at least 10% of the volume (MinDiskPercent), an edges file no older than 15 minutes (MaxEdgesAge). Choice: spread, then most mem_free_mib, then the P5-D2 hash.
• H27: when no box qualifies, every door that chooses a box (instance create, domain attach's first route, standalone sidecar create, the move door's destination) answers 503 with RHA-C17, and alerts at most once per region an hour per console process. A door's own earlier placement checks answer a region with no room as H27 too. A create door chooses before its first write (registration, environment binding or any other) and passes that choice through; nothing chooses again.
• §6's health answer: GET or HEAD /.carlos/lb-health, any host, port 80. 200 only when hosting, forwarding armed (map and key each loaded once, latched), the certificate mirror loaded once (latched), the edge's and the activator's beats within 60 seconds, and a home map refresh attempt completed within 5 minutes.
• User-facing copy is in docs/superpowers/plans/2026-10-16-region-ha-2a-copy.md, gated by approved_region_ha_copy_test.go. No pending row reaches code, CLI help or docs. No em dashes.
• Go commands: export GOFLAGS=-buildvcs=false GOCACHE=$TMPDIR/gocache. Run tests with nice -n 19 and a -run filter; never the whole internal/host suite (CI's make test is the full run). Tests that bind unix sockets fail inside the sandbox ("socket: operation not permitted"); that is the sandbox, not the change.
• Census: every task that changes a non-test Go file under a package directory re-pins that directory's @coverage row in internal/provision/testdata/plan14-root-reads-region-ha-2a.tsv.

Census items for platform-1e (census ruling region-ha-2a)

Approved in principle with conditions (amadan #236). The overlay asks for exactly these:

| Task | Unit | Class | Change | Seal | |---|---|---|---|---| | 7 | internal/host/edgesfile.go:edgesFileWriteClaims | authority, F,J1 (kept) | Re-digest, class, task, basis and sites unchanged: with region HA on it reads the address source before every publication | stands | | 10 | internal/console/apiboxroles.go:Server.boxRoster | accepted-legacy, J4 root-input | New: the role doors' read of the signed roster (Directory.Read), operator-only, read-only | moves | | 11 | internal/console/claim_create.go:Server.beginClaimCreate | accepted-legacy, J4 root-input | Re-freeze: one call (chooseExecutorFirst) before RegisterStart, given this unit's own claim read, which keeps a carried choice only when that read is the observation the choice was decided on; its one read site unchanged (approved, ruling rev 2 item 3) | moves | | 11 | internal/console/nocapacity.go:Server.chooseCreateFirst | accepted-legacy, J4 root-input | New in rev 3, approved (ruling item 5) with its condition met in rev 4: the instance create door's choice before its first write. Its one root read is the host's claim (hostclaim.Read), the read beginClaimCreate makes, feeding only the placement choice; it takes no lease and writes nothing, and returns its choice with the claim observation it decided on | moves | | 12 | internal/console/apiinstancemove.go:Server.handleAPIInstanceMoveRequest | accepted-legacy | Re-freeze: two refusals added before any write, both no room in the destination (roomIn before the placement checks, moveRoomCheck through selectMoveBoxes); both only refuse; read sites unchanged, offsets moved (approved, ruling rev 2 item 4; the basis text names both refusals) | moves |

Server.attachClaimFirstRoute is not changed: the first route's refusal comes through beginClaimCreate and claimResponseRefusal, which is not a root-read unit. Record-write census (B8) digests change for Directory.Replace (Task 3), registerCLIRoutes (Task 10), handleAPIInstanceCreate (Task 11) and handleAPIInstanceMoveRequest (Task 12). No write site changes and none is added (ruling rev 2): each re-digest is a change of the unit's other lines; the census reports no new or changed site.

Build base (condition 3). plan14-root-reads-region-ha-2a.tsv loads last at build time. The values in this plan were computed at this branch's base, main 70e57449; the build base will be later (after Plan 14 Parts 2 and 3 land), so recompute every value there with the tooling below: each @coverage row, each B8 digest, each unit's kinds and digest, each @legacy fingerprint, and each seal. The rows to write stay the same.

Census tooling

Create these in $TMPDIR/census (never in the repo), and delete the directory when the plan is done.

coverage.sh prints a directory's @coverage row exactly as rootReadProduction computes it (run from the repo root):

````bash #!/bin/bash

usage: coverage.sh <dir> ... ; run from the repo root. Prints the Plan 14

@coverage line of each directory, as rootReadProduction computes it.

for dir in "$@"; do hash=$(for f in "$dir"/; do [ -f "$f" ] || continue case "$f" in test.go) continue;; .go|.s|.sh|.service|.path|.socket|.timer|.conf|.tf|.tftpl|.tmpl|/go.mod|/go.sum) ;; ) continue;; esac printf '%s\0%s\n' "${f#./}" "$(sha256sum "$f" | cut -d' ' -f1)" done | LCALL=C sort | head -c -1 | sha256sum | cut -d' ' -f1) printf '@coverage\t%s\t%s\n' "$dir" "$hash" done ````

unitdigest/main.go prints a top-level unit's digest (the sha256 of its declaration's source), the value both censuses pin. Build it with cd $TMPDIR/census/unitdigest && GOFLAGS= go build -o ../unitdigest.bin main.go (not -o ../unitdigest, which is the source directory: Go would write the binary inside it), then run $TMPDIR/census/unitdigest.bin <file> <unit> from the repo root:

````go // unitdigest prints the census digest (sha256 of the declaration's source) // of each named top-level unit: unitdigest <file> <unit> [<unit> ...]. // Unit names are the censuses': Func, Recv.Method, "type T", "var v". package main

import ( "crypto/sha256" "encoding/hex" "fmt" "go/ast" "go/parser" "go/token" "os" )

func recv(e ast.Expr) string { for { switch x := e.(type) { case ast.StarExpr: e = x.X case ast.ParenExpr: e = x.X case ast.IndexExpr: e = x.X case ast.IndexListExpr: e = x.X case *ast.Ident: return x.Name default: return "?" } } }

func main() { src, err := os.ReadFile(os.Args[1]) if err != nil { panic(err) } fset := token.NewFileSet() f, err := parser.ParseFile(fset, os.Args[1], src, 0) if err != nil { panic(err) } want := map[string]bool{} for , u := range os.Args[2:] { want[u] = true } for , d := range f.Decls { name := "" switch d := d.(type) { case ast.FuncDecl: name = d.Name.Name if d.Recv != nil { name = recv(d.Recv.List[0].Type) + "." + name } case ast.GenDecl: if d.Tok == token.IMPORT || d.Lparen.IsValid() { continue } switch s := d.Specs[0].(type) { case ast.ValueSpec: name = d.Tok.String() + " " + s.Names[0].Name case ast.TypeSpec: name = "type " + s.Name.Name } } if want[name] { sum := sha256.Sum256(src[fset.Position(d.Pos()).Offset:fset.Position(d.End()).Offset]) fmt.Printf("%s\t%s\t%s\n", os.Args[1], name, hex.EncodeToString(sum[:])) } } } ````

legacyrow.py prints the @legacy row and the entry row for a unit (python3 legacyrow.py <file> <unit> <kinds> <digest> <basis>):

````python import hashlib, sys

legacyrow.py <file> <unit> <kinds> <digest> <basis>: prints the @legacy row and the entry row.

f,u,k,d,b = sys.argv[1:6] fields=[f,u,"legacy","accepted-legacy",b,k,d] fp=hashlib.sha256("\t".join(fields).encode()).hexdigest() print("\t".join(["@legacy",fp]+fields)) print("\t".join(fields)) ````

A unit's kinds string (for an unclassified or changed root read) is printed by a scratch test. Create internal/provision/zz_census_scratch_test.go with this content, run it, and delete it by exact path before committing:

````go package provision

import ( "os" "strings" "testing" )

// Scratch: prints the census scan of the named units (RHA2AUNITS, comma // separated "file:unit"). Never committed. func TestZZCensusScratchUnits(t *testing.T) { want := os.Getenv("RHA2AUNITS") if want == "" { t.Skip() } got, err := rootReadProduction(censusRoot(t)) if err != nil { t.Fatal(err) } for , e := range got.Entries { for , w := range strings.Split(want, ",") { if e.File+":"+e.Unit == w { t.Logf("ENTRY\t%s\t%s\t%s\t%s", e.File, e.Unit, e.Kinds, e.Digest) } } } } ````

```bash RHA2A_UNITS="internal/console/apiboxroles.go:Server.boxRoster" nice -n 19 go test ./internal/provision/ -run TestZZCensusScratchUnits -count=1 -v | grep ENTRY ```

The seal after a task's legacy rows is rootReadLegacySeal over every overlay in the test's load order. Add this to the scratch file to print it (the list is the load order at 70e57449; on a newer base, copy the order from TestPlan14RootReadCensus):

```go func TestZZCensusScratchSeal(t *testing.T) { if os.Getenv("RHA2A_SEAL") == "" { t.Skip() } root := censusRoot(t) inv := rootReadInventory{Entries: map[string]classifiedRootRead{}, Coverage: map[string]string{}} files := []string{"plan14-root-reads.tsv"} if _, err := os.Stat(root + "/internal/host/movesheldrelease_production.go"); err == nil { files = append(files, "plan14-root-reads-moves.tsv") } files = append(files, "plan14-root-reads-merge.tsv", "plan14-root-reads-fix1.tsv", "plan14-root-reads-fix2.tsv", "plan14-root-reads-merge3.tsv", "plan14-root-reads-fix4.tsv", "plan14-root-reads-fix5.tsv", "plan14-root-reads-fix6.tsv", "plan14-root-reads-zddr3.tsv", "plan14-root-reads-hotfix281.tsv", "plan14-root-reads-zddr4.tsv", "plan14-root-reads-reserved285.tsv", "plan14-root-reads-292.tsv", "plan14-root-reads-census-ruling-275.tsv", "plan14-root-reads-census-ruling-owed.tsv", "plan14-root-reads-carlos-run.tsv", "plan14-root-reads-seeded-instances.tsv", "plan14-root-reads-hostfiles-cap-236.tsv", "plan14-root-reads-moves-overcap-cut-236.tsv", "plan14-root-reads-region-ha-2a.tsv") for i, f := range files { if err := loadRootReadInventory(root+"/internal/provision/testdata/"+f, &inv, i > 0); err != nil { t.Fatal(f, err) } } t.Logf("SEAL\t%s", rootReadLegacySeal(inv)) } ```

```bash RHA2A_SEAL=1 nice -n 19 go test ./internal/provision/ -run TestZZCensusScratchSeal -count=1 -v | grep SEAL ```

The overlay is tab-separated: keep the tabs in every row this plan gives, or the loader refuses the row.

The census checks every task runs:

```bash nice -n 19 go test ./internal/provision/ -run 'TestPlan9B8RecordWriteCensus|TestPlan14RootReadCensus$' -count=1 ```

Review Focus

1. A roster write by another path must keep a box's role (a quarantine, an activation, an enrolment): a box that silently drops back to default takes new instances the operator excluded. Task 3's TestOtherRosterWritesKeepTheRole pins the quarantine path; Replace compares every other field, so no write may change Group.
2. IMDS answering blank on one pass must publish no public address rather than an old one, and the box then drops out of Eligible until the next pass, never forwarding to an address it no longer holds. Task 7's TestEdgesFilePublishesOnlyWhatTheSourceReports.
3. An edges file stamped by a box whose clock runs ahead is believed for a minute of skew and no more; a stale or unreadable generation is never trusted. Task 4's TestPlaceableAppliesEachRule (the future and unreadable cases).
4. A bucket outage after a box has armed must leave its health answer at 200 (no termination for a shared outage), while a refresh stuck inside one attempt turns it 503. Task 2's TestBucketOutageAfterArmingStaysHealthy and Task 9's TestHomeMapFeedsTheHealthState.
5. A client retrying a refused create in a loop must not page the operator on every retry, and must leave nothing for recovery each time. Task 11's TestCreateDoorsRefuseWithNoRoomBeforeRegistering (three refusals, one page, no registration, no claim after a recovery pass).
6. A create refused for want of room must change nothing, even when the fleet changes under it. The choice comes before the environment binding, and the create names the box that was chosen. Task 11's TestRefusedCreateLeavesTheEnvironmentsStoreAlone (the last box goes between the region check and the choice) and TestChooseFirstWritesNothingAndChoosesOnce. Nor may a create go on with a choice whose claim changed: TestAStaleEarlyChoiceIsNeverUsed.
7. The move door's room check only refuses. A regression that narrows the candidates would quietly pick a destination main's selection refuses (a region with several boxes). Task 12's TestMoveRoomCheckOnlyRefuses.

Dry run

Every code block below was written into this branch on main 70e57449 (Tasks 1 to 10 at aa5b6b07 for rev 2, replayed unchanged for rev 3; Tasks 11 to 13 rewritten at 0a31a455 for rev 3, and again at e5a3a3de for rev 4), task by task in order, built, vetted with GOOS=linux and GOOS=darwin (and built with GOOS=windows), and its tests run as each task's steps say; the census values are those runs'. The copy gate was checked without approving anything: with every row pending, TestRegionHABuiltCopyIsApplied, TestRegionHACopyRecordIsComplete and TestRegionHACopyHasNoEmDash pass, and TestRegionHABuiltCopyIsApproved fails naming exactly the built ids (RHA-C1 to RHA-C16 after Task 10, RHA-C1 to RHA-C19 after Task 11). That failure is what an implementer sees until Paul approves the rows. Each rev 3 regression test was first run against rev 2's code and failed (the store test's refused create wrote the store; the move door answered 400; a legacy box beside a default one was picked; the metric went to the edges file's region). Rev 4's TestAStaleEarlyChoiceIsNeverUsed was run with the observation check removed and failed. The whole TestClaimConsoleMutationInvariant ran once on the final tree. The plan text was then replayed onto a clean export, task by task, and every intermediate state built and passed both censuses. The code was then removed.

---

Task 1: The copy record and its gate

Files:

• Create: docs/superpowers/plans/2026-10-16-region-ha-2a-copy.md
• Create: approved_region_ha_copy_test.go

Interfaces:

• Consumes: the run and moves gates' helpers at the repo root (parseCopyTable, checkMovesInventory, checkMovesNoEmDash, checkRunBuilt, checkRunApplied, builtRows, reportAll, copyRow).
• Produces: regionHACopyBuilt []string, which Tasks 10 and 11 extend with the ids whose text they write; the record's ids RHA-C1 to RHA-C19.

Every user-facing string 2a writes is drafted here first, all pending. The gate is real: an id in regionHACopyBuilt must be an approved row, and its text must be in the file the row names. Nothing marks a pending row approved for checking.

• [ ] Step 1: Write the record

docs/superpowers/plans/2026-10-16-region-ha-2a-copy.md:

````markdown

Region HA plan 2a: copy record (pending Paul's review)

Every user-facing string plan 2a (2026-10-16-region-ha-2a-core.md) writes: console refusals, CLI output and usage lines, the operator alert, and the docs prose. The gate is approved_region_ha_copy_test.go at the repo root, which reuses the moves gate's parser, so the format is the moves record's: one table per heading, columns id, file, text, status.

The rules:

• A row is pending until Paul approves it in the copy review (the copy-review skill). His edits are applied to the text here verbatim and the row becomes approved.
• No pending string reaches code, CLI help or docs. The task that writes a row's text adds its id to regionHACopyBuilt in the gate in the same commit, and from then on the gate fails unless the row is approved and its text is in the file it names. Tasks 10 and 11 therefore wait for their rows' approval.
• In a Go file the text is one string literal, or one fmt.Sprintf format, equal to the row with %s where the row has a <placeholder>. A row whose file cell says "a fragment" is a piece of a larger literal (the carlos system usage text).
• No em dashes. A full stop and a new sentence instead. The role doors' refusals start lower case, as the console's other CLI refusals do; the no-capacity refusal is shared with the browser's domain door, so it is a capitalised sentence, as claimResponseUnavailable is.

Placeholders: <box> a box's roster reference or label; <role> a role name (default, legacy or platform); <region> a placement code or the home region's name.

Box and group roles (Task 10)

| id | file | text | status | |---|---|---|---| | RHA-C1 | internal/console/apiboxroles.go | role must be default, legacy or platform | pending | | RHA-C2 | internal/console/apiboxroles.go | a group's role must be default or legacy | pending | | RHA-C3 | internal/console/apiboxroles.go | could not read the box roster. Try again. | pending | | RHA-C4 | internal/console/apiboxroles.go | the box roster changed while the role was being saved, and nothing was changed. Try again. | pending | | RHA-C5 | internal/console/apiboxroles.go | could not save the role. Try again. | pending | | RHA-C6 | internal/console/apiboxroles.go | no enrolled box is named <box> | pending | | RHA-C7 | internal/console/apiboxroles.go | <region> has no group | pending | | RHA-C8 | internal/console/apiboxroles.go | <box> is retired, so it takes no role | pending | | RHA-C9 | systemroles.go | <box> now has the <role> role | pending | | RHA-C10 | systemroles.go | the group in <region> now has the <role> role | pending | | RHA-C11 | system.go (a fragment of the usage) | whether new instances may be placed on a box. A legacy or | pending | | RHA-C12 | system.go (a fragment of the usage) | platform box keeps the instances it has and takes no new ones | pending | | RHA-C13 | system.go (a fragment of the usage) | the same for every box a region's group has enrolled, and for | pending | | RHA-C14 | system.go (a fragment of the usage) | every box it enrolls later | pending | | RHA-C15 | docs/site/cli.md | Box and group roles | pending | | RHA-C16 | docs/site/cli.md | carlos system box role says whether new instances may be placed on a box. A box with the default role can take them. A legacy or platform box keeps the instances it has and takes no new ones: no new instance, relocation or move lands on it. carlos system group role sets the role of every box a region's group has enrolled, and of every box it enrolls later. Both work only on a console with region HA on. | pending |

No box has room (Task 11)

One refusal for every door that chooses a box (H27): instance create, domain attach, standalone sidecar create and the move door's destination.

| id | file | text | status | |---|---|---|---| | RHA-C17 | internal/console/nocapacity.go | No box in <region> has room right now. Try again in a few minutes. | pending | | RHA-C18 | internal/console/nocapacity.go (the operator alert's subject) | No box in <region> has room | pending | | RHA-C19 | internal/console/nocapacity.go (the operator alert's body) | A request in <region> was refused because no box there can take it. Every box is full, stale or excluded by its role. Add a box, or free memory or disk on one, and check the eligible-capacity alarm for <region>. | pending | ````

• [ ] Step 2: Write the gate

approved_region_ha_copy_test.go:

````go // approvedregionhacopytest.go package main

import ( "os" "regexp" "strconv" "testing" )

// The copy gate of region HA plan 2a (Task 1). The record is a markdown // table in the moves record's format, read with the moves gate's parser // and checked with the run gate's helpers, so "approved" and "applied" // mean what they mean there: // // - TestRegionHACopyRecordIsComplete: the record names exactly RHA-C1 to // RHA-C19; // - TestRegionHACopyHasNoEmDash; // - TestRegionHABuiltCopyIsApproved: an id in regionHACopyBuilt is an // approved row. No pending string reaches code, CLI help or docs; // - TestRegionHABuiltCopyIsApplied: a built row's text is in the file it // names. const regionHACopyRecordPath = "docs/superpowers/plans/2026-10-16-region-ha-2a-copy.md"

var regionHACopyIDPattern = regexp.MustCompile(^RHA-C[0-9]+$)

// regionHACopyIDs is the record's inventory: RHA-C1 to RHA-C19, pinned. var regionHACopyIDs = func() []string { ids := make([]string, 0, 19) for i := 1; i <= 19; i++ { ids = append(ids, "RHA-C"+strconv.Itoa(i)) } return ids }()

// regionHACopyBuilt is every id whose text the code or docs carry. The // task that writes a row's text adds its id here in the same commit, once // Paul has approved the row. var regionHACopyBuilt = []string{}

func mustParseRegionHARecord(t *testing.T) []copyRow { t.Helper() raw, err := os.ReadFile(regionHACopyRecordPath) if err != nil { t.Fatal(err) } rows, err := parseCopyTable(string(raw), regionHACopyRecordPath, regionHACopyIDPattern, "an RHA-C<n>") if err != nil { t.Fatal(err) } return rows }

func TestRegionHACopyRecordIsComplete(t *testing.T) { reportAll(t, checkMovesInventory(mustParseRegionHARecord(t), regionHACopyIDs)) }

func TestRegionHACopyHasNoEmDash(t *testing.T) { reportAll(t, checkMovesNoEmDash(mustParseRegionHARecord(t))) }

func TestRegionHABuiltCopyIsApproved(t *testing.T) { reportAll(t, checkRunBuilt(mustParseRegionHARecord(t), regionHACopyBuilt)) }

func TestRegionHABuiltCopyIsApplied(t *testing.T) { reportAll(t, checkRunApplied(builtRows(mustParseRegionHARecord(t), regionHACopyBuilt), ".")) } ````

• [ ] Step 3: Run it

Run: nice -n 19 go test . -run 'RegionHACopy|RegionHABuilt' -count=1 Expected: PASS (nothing is built yet).

• [ ] Step 4: Put the record in front of Paul

Run the copy-review skill over the record's rows. Apply his edits verbatim to the record, mark each row he approves approved, and commit that on its own. Tasks 10 and 11 wire their rows only once they are approved, with Paul's text: where his text differs from a draft below, use his in the code too. Tasks 2 to 9 do not wait.

• [ ] Step 5: Commit

git add docs/superpowers/plans/2026-10-16-region-ha-2a-copy.md approved_region_ha_copy_test.go
git commit -m "region HA 2a: the copy record and its gate"

No package envelope changes: neither file is a non-test Go file.

---

Task 2: The health state (internal/lbhealth), the provider-boundary tests and the census overlay

Files:

• Create: internal/lbhealth/lbhealth.go, internal/lbhealth/lbhealth_test.go
• Create: region_ha_boundary_test.go
• Create: internal/provision/testdata/plan14-root-reads-region-ha-2a.tsv
• Modify: internal/provision/plan14_root_reads_test.go (load the new overlay last)

Interfaces:

• Produces (internal/lbhealth): const Path = "/.carlos/lb-health"; HeartbeatWindow = 60 * time.Second; RefreshWindow = 5 * time.Minute; the condition names CondHosting, CondForwarding, CondCerts, CondEdge, CondActivator, CondRefresh; type State; var Default = &State{}; methods (all safe on a nil *State) SetHosting(bool), Hosting() bool, MapLoaded(), KeyLoaded(), CertsLoaded(), EdgeBeat(time.Time), RefreshAttempted(time.Time), SetActivator(func() time.Time), Failing(now time.Time) []string; func Serve(w http.ResponseWriter, r *http.Request, s *State, now time.Time) bool.
• Produces (repo root): regionHACoreFiles, regionHAAdapterFiles, regionHAPackages, regionHAS3Exceptions in region_ha_boundary_test.go, which later tasks extend.
• Produces: the overlay file every later task's census step edits, and the seal = ... line after it, which Tasks 10, 11 and 12 change.

• [ ] Step 1: Write the failing tests

internal/lbhealth/lbhealth_test.go:

````go package lbhealth

import ( "net/http" "net/http/httptest" "reflect" "strings" "testing" "time" )

var t0 = time.Date(2026, 10, 16, 12, 0, 0, 0, time.UTC)

// healthy is a state with every condition met at t0. func healthy() *State { s := &State{} s.SetHosting(true) s.MapLoaded() s.KeyLoaded() s.CertsLoaded() s.EdgeBeat(t0) s.RefreshAttempted(t0) s.SetActivator(func() time.Time { return t0 }) return s }

func TestFailingNamesEachCondition(t testing.T) { if got := healthy().Failing(t0); len(got) != 0 { t.Fatalf("healthy box fails %v", got) } if got := (&State{}).Failing(t0); !reflect.DeepEqual(got, []string{CondHosting, CondForwarding, CondCerts, CondEdge, CondActivator, CondRefresh}) { t.Fatalf("zero state fails %v", got) } var nilState State if got := nilState.Failing(t0); len(got) != 6 { t.Fatalf("nil state fails %v", got) } cases := []struct { name string spoil func(State) now time.Time want string }{ {"joining", func(s State) { s.SetHosting(false) }, t0, CondHosting}, {"map never loaded", func(s State) { s.mapLoaded.Store(false) }, t0, CondForwarding}, {"key never loaded", func(s State) { s.keyLoaded.Store(false) }, t0, CondForwarding}, {"no certificate cache", func(s State) { s.certs.Store(false) }, t0, CondCerts}, {"edge beat stale", func(s State) { s.EdgeBeat(t0.Add(-61 time.Second)) }, t0, CondEdge}, {"activator beat stale", func(s State) { s.SetActivator(func() time.Time { return t0.Add(-61 time.Second) }) }, t0, CondActivator}, {"no activator", func(s State) { s.SetActivator(nil) }, t0, CondActivator}, {"refresh stuck", func(s State) { s.RefreshAttempted(t0.Add(-5time.Minute - time.Second)) }, t0, CondRefresh}, } for _, c := range cases { t.Run(c.name, func(t *testing.T) { s := healthy() c.spoil(s) if got := s.Failing(c.now); !reflect.DeepEqual(got, []string{c.want}) { t.Fatalf("Failing = %v, want [%s]", got, c.want) } }) } // The windows are inclusive: a beat exactly 60s old and a refresh // exactly 5 minutes old still count. s := healthy() s.EdgeBeat(t0.Add(-HeartbeatWindow)) s.RefreshAttempted(t0.Add(-RefreshWindow)) if got := s.Failing(t0); len(got) != 0 { t.Fatalf("boundary fails %v", got) } }

// A bucket outage after arming: refreshes keep failing, but each failed // attempt is still a completed attempt, and the latches hold. The box // stays healthy (A2-3, A3-3). func TestBucketOutageAfterArmingStaysHealthy(t testing.T) { s := healthy() for i := 1; i <= 20; i++ { now := t0.Add(time.Duration(i) 30 * time.Second) s.RefreshAttempted(now) // a failed attempt: no latch is re-set s.EdgeBeat(now) s.SetActivator(func() time.Time { return now }) if got := s.Failing(now); len(got) != 0 { t.Fatalf("after %d failed refreshes: %v", i, got) } } }

func TestServe(t testing.T) { get := func(s State, method, target, host string) (*httptest.ResponseRecorder, bool) { w := httptest.NewRecorder() r := httptest.NewRequest(method, target, nil) r.Host = host return w, Serve(w, r, s, t0) } w, ok := get(healthy(), http.MethodGet, Path, "10.0.1.7") if !ok || w.Code != http.StatusOK || w.Body.String() != "ok\n" || w.Header().Get("Cache-Control") != "no-store" { t.Fatalf("healthy GET: ok=%v %d %q %v", ok, w.Code, w.Body.String(), w.Header()) } w, ok = get(healthy(), http.MethodHead, Path, "any.example.com") if !ok || w.Code != http.StatusOK || w.Body.Len() != 0 { t.Fatalf("healthy HEAD: ok=%v %d %q", ok, w.Code, w.Body.String()) } joining := healthy() joining.SetHosting(false) w, ok = get(joining, http.MethodGet, Path, "10.0.1.7") if !ok || w.Code != http.StatusServiceUnavailable || !strings.Contains(w.Body.String(), CondHosting) { t.Fatalf("joining GET: ok=%v %d %q", ok, w.Code, w.Body.String()) } for , c := range []struct{ method, target string }{ {http.MethodPost, Path}, {http.MethodGet, Path + "/"}, {http.MethodGet, "/.carlos/lb-healthz"}, {http.MethodGet, "/"}, } { if , ok := get(healthy(), c.method, c.target, "10.0.1.7"); ok { t.Fatalf("%s %s answered", c.method, c.target) } } }

func TestNilStateIgnoresUpdates(t testing.T) { var s State s.SetHosting(true) s.MapLoaded() s.KeyLoaded() s.CertsLoaded() s.EdgeBeat(t0) s.RefreshAttempted(t0) s.SetActivator(func() time.Time { return t0 }) if s.Hosting() { t.Fatal("nil state hosting") } } ````

region_ha_boundary_test.go:

````go package main

import ( "go/ast" "go/parser" "go/token" "os" "path/filepath" "sort" "strconv" "strings" "testing" )

// Region HA spec 2 §25: the provider boundary. Spec 2's core names no // cloud provider; its AWS parts are adapters in files named for AWS. Each // plan task that adds a core file lists it here in the same commit.

// regionHACoreFiles are spec 2's provider-neutral files. None may import // an AWS package. var regionHACoreFiles = []string{ "internal/lbhealth/lbhealth.go", }

// regionHAAdapterFiles are spec 2's AWS adapters. Each is named for AWS so // a reader can see where the provider lives. var regionHAAdapterFiles = []string{}

// regionHAPackages are the packages spec 2 touches. Every S3 client built // in them goes through s3compat.Options(), so CARLOSS3ENDPOINT applies. var regionHAPackages = []string{ "internal/activator", "internal/boxchoice", "internal/console", "internal/edge", "internal/edgesfile", "internal/host", "internal/hostactors", "internal/hostops", "internal/lbhealth", }

// regionHAS3Exceptions are S3 clients on main that skip s3compat, each // with the discussion that fixes it outside spec 2. A listed file that has // since been fixed passes. var regionHAS3Exceptions = map[string]string{ "internal/host/claimproduction_marker.go": "#298", }

func TestRegionHACoreImportsNoAWS(t *testing.T) { for , f := range regionHACoreFiles { file, err := parser.ParseFile(token.NewFileSet(), f, nil, parser.ImportsOnly) if err != nil { t.Fatalf("%s: %v", f, err) } for , imp := range file.Imports { path, _ := strconv.Unquote(imp.Path.Value) if strings.HasPrefix(path, "github.com/aws/") { t.Errorf("%s imports %s: spec 2's core names no provider (§25); move it to an AWS adapter", f, path) } } } }

func TestRegionHAAdaptersAreNamedForAWS(t *testing.T) { for , f := range regionHAAdapterFiles { if , err := os.Stat(f); err != nil { t.Errorf("%s: %v", f, err) } if !strings.Contains(filepath.Base(f), "aws") && !strings.Contains(filepath.Dir(f), "aws") { t.Errorf("%s is an AWS adapter but its name does not say so", f) } } }

// s3NewFromConfigCalls returns the line of each s3.NewFromConfig call in // src that does not pass s3compat.Options() as an argument. func s3NewFromConfigCalls(t testing.T, path string) []int { t.Helper() fset := token.NewFileSet() f, err := parser.ParseFile(fset, path, nil, 0) if err != nil { t.Fatalf("%s: %v", path, err) } s3Names, compatNames := map[string]bool{}, map[string]bool{} for , imp := range f.Imports { p, := strconv.Unquote(imp.Path.Value) name := filepath.Base(p) if imp.Name != nil { name = imp.Name.Name } switch p { case "github.com/aws/aws-sdk-go-v2/service/s3": s3Names[name] = true case "github.com/carlosframework/platform/internal/s3compat": compatNames[name] = true } } var bad []int ast.Inspect(f, func(n ast.Node) bool { call, ok := n.(ast.CallExpr) if !ok { return true } sel, ok := call.Fun.(ast.SelectorExpr) if !ok || sel.Sel.Name != "NewFromConfig" { return true } pkg, ok := sel.X.(ast.Ident) if !ok || !s3Names[pkg.Name] { return true } for _, arg := range call.Args { inner, ok := arg.(ast.CallExpr) if !ok { continue } isel, ok := inner.Fun.(ast.SelectorExpr) if !ok || isel.Sel.Name != "Options" { continue } if id, ok := isel.X.(*ast.Ident); ok && compatNames[id.Name] { return true } } bad = append(bad, fset.Position(call.Pos()).Line) return true }) return bad }

func TestRegionHAS3ClientsUseS3Compat(t testing.T) { for _, dir := range regionHAPackages { paths, err := filepath.Glob(filepath.Join(dir, ".go")) if err != nil { t.Fatal(err) } sort.Strings(paths) for , p := range paths { if strings.HasSuffix(p, "test.go") { continue } if , ok := regionHAS3Exceptions[filepath.ToSlash(p)]; ok { continue } for , line := range s3NewFromConfigCalls(t, p) { t.Errorf("%s:%d builds an S3 client without s3compat.Options() (§25): CARLOSS3ENDPOINT would not apply", p, line) } } } }

// The checker itself: a bare client is caught, one with Options() is not, // whatever the import names. func TestRegionHAS3CheckerCatchesABareClient(t *testing.T) { dir := t.TempDir() write := func(name, body string) string { p := filepath.Join(dir, name) if err := os.WriteFile(p, []byte(body), 0o644); err != nil { t.Fatal(err) } return p } bare := write("bare.go", package x import awss3 "github.com/aws/aws-sdk-go-v2/service/s3" func f(cfg any) { _ = awss3.NewFromConfig(cfg) } ) good := write("good.go", package x import ( "github.com/aws/aws-sdk-go-v2/service/s3" "github.com/carlosframework/platform/internal/s3compat" ) func f(cfg any) { _ = s3.NewFromConfig(cfg, s3compat.Options()) } ) if got := s3NewFromConfigCalls(t, bare); len(got) != 1 { t.Fatalf("bare client: %v", got) } if got := s3NewFromConfigCalls(t, good); len(got) != 0 { t.Fatalf("compat client flagged: %v", got) } } ````

• [ ] Step 2: Run them to see them fail

Run: nice -n 19 go test ./internal/lbhealth/ -count=1 Expected: FAIL, the package does not build (undefined: State). go test . -run 'TestRegionHA(Core|Adapters|S3)' fails the same way on internal/lbhealth/lbhealth.go not existing.

• [ ] Step 3: Write the package

internal/lbhealth/lbhealth.go:

````go // Package lbhealth is a box's own liveness, as the load balancer's health // check reads it (region HA spec 2 §6). Every fact here is this box's own: // a flag or a timestamp that one of its own loops sets. Nothing here reads // a disk, a bucket or the network, so an outage of something the boxes // share (the deployment bucket) never turns a working box unhealthy and // never gets it terminated. // // The edge answers GET or HEAD /.carlos/lb-health on port 80 from State. // The loops that feed it are the agent's: the claim agent sets hosting, // the home map latches forwarding and stamps its refresh attempts, the // certificate mirror latches its first load, and the edge's registry watch // and the activator's sweeper each beat. package lbhealth

import ( "io" "net/http" "strings" "sync" "sync/atomic" "time" )

// Path is the health check's path, matched exactly. const Path = "/.carlos/lb-health"

// HeartbeatWindow is how recent the edge's and the activator's beats must // be. Both loops beat every 30 seconds or faster. const HeartbeatWindow = 60 * time.Second

// RefreshWindow is how recent the home map's last completed refresh // attempt must be: its 30-second interval plus its two-minute request // timeout, with room to spare. A failed attempt counts, so a bucket outage // where every attempt fails fast keeps this satisfied; a worker stuck // inside one attempt does not. const RefreshWindow = 5 * time.Minute

// The condition names a 503 lists, in this order. const ( CondHosting = "hosting" CondForwarding = "forwarding" CondCerts = "certificates" CondEdge = "edge" CondActivator = "activator" CondRefresh = "refresh" )

// State is one box's liveness facts. The zero value is a box that has // proven nothing, and answers 503. A nil *State ignores every update. type State struct { hosting atomic.Bool mapLoaded atomic.Bool keyLoaded atomic.Bool certs atomic.Bool edgeBeat atomic.Int64 // unix nanoseconds; 0 is never refreshed atomic.Int64 // unix nanoseconds; 0 is never

mu sync.RWMutex activator func() time.Time }

// Default is the process's state. The agent runs the edge and the // activator in one process, so the claim wiring, the edge and the // activator share it. var Default = &State{}

// SetHosting records whether the agent is in hosting mode: enrolled, with // its guard armed (§5). Region HA spec 2's plan 2b adds joining mode, whose // mode selection is what sets this false; until then the claim agent sets // it true once its claim wiring is running (ClaimWiring.startRegional). func (s *State) SetHosting(on bool) { if s != nil { s.hosting.Store(on) } }

// Hosting reports what SetHosting last recorded. func (s *State) Hosting() bool { return s != nil && s.hosting.Load() }

// MapLoaded latches the first successful load of the forwarding map. func (s *State) MapLoaded() { if s != nil { s.mapLoaded.Store(true) } }

// KeyLoaded latches the first successful load of a mesh key. func (s *State) KeyLoaded() { if s != nil { s.keyLoaded.Store(true) } }

// CertsLoaded latches the first successful load of the certificate mirror // into the local cache the TLS handshake reads. func (s *State) CertsLoaded() { if s != nil { s.certs.Store(true) } }

// EdgeBeat records that the edge's registry watch completed a poll at t. func (s *State) EdgeBeat(t time.Time) { if s != nil { s.edgeBeat.Store(t.UnixNano()) } }

// RefreshAttempted records that the home map's refresh finished an // attempt at t, whether it succeeded or failed. func (s *State) RefreshAttempted(t time.Time) { if s != nil { s.refreshed.Store(t.UnixNano()) } }

// SetActivator gives the state the activator's own beat: the time its // sweeper last finished a sweep. nil means there is no activator, which // fails the activator condition. func (s *State) SetActivator(last func() time.Time) { if s == nil { return } s.mu.Lock() s.activator = last s.mu.Unlock() }

func (s *State) activatorBeat() time.Time { s.mu.RLock() last := s.activator s.mu.RUnlock() if last == nil { return time.Time{} } return last() }

// within reports whether at is set and no older than window at now. func within(at, now time.Time, window time.Duration) bool { return !at.IsZero() && now.Sub(at) <= window }

func stamp(n int64) time.Time { if n == 0 { return time.Time{} } return time.Unix(0, n) }

// Failing lists the conditions that do not hold at now, in the order of // the Cond names. An empty list is a healthy box. A nil State fails // every condition. func (s State) Failing(now time.Time) []string { if s == nil { return []string{CondHosting, CondForwarding, CondCerts, CondEdge, CondActivator, CondRefresh} } var out []string if !s.hosting.Load() { out = append(out, CondHosting) } if !s.mapLoaded.Load() || !s.keyLoaded.Load() { out = append(out, CondForwarding) } if !s.certs.Load() { out = append(out, CondCerts) } if !within(stamp(s.edgeBeat.Load()), now, HeartbeatWindow) { out = append(out, CondEdge) } if !within(s.activatorBeat(), now, HeartbeatWindow) { out = append(out, CondActivator) } if !within(stamp(s.refreshed.Load()), now, RefreshWindow) { out = append(out, CondRefresh) } return out }

// Serve answers the health check when r is one, and reports whether it // did. It matches as narrowly as the steering probe: the exact path, GET // or HEAD, any host. It answers 200 "ok" when every condition holds and // 503 naming the failing conditions otherwise, never cached. It only reads // s. func Serve(w http.ResponseWriter, r http.Request, s State, now time.Time) bool { if r.URL.Path != Path || (r.Method != http.MethodGet && r.Method != http.MethodHead) { return false } body, code := "ok\n", http.StatusOK if failing := s.Failing(now); len(failing) > 0 { body, code = "unavailable: "+strings.Join(failing, ",")+"\n", http.StatusServiceUnavailable } w.Header().Set("Content-Type", "text/plain; charset=utf-8") w.Header().Set("Cache-Control", "no-store") w.WriteHeader(code) if r.Method == http.MethodGet { , = io.WriteString(w, body) } return true } ````

• [ ] Step 4: Run the tests

Run: nice -n 19 go test ./internal/lbhealth/ -count=1 && nice -n 19 go test . -run 'TestRegionHA(Core|Adapters|S3)' -count=1 Expected: PASS. The S3 call-site test passes with internal/host/claimproduction_marker.go excepted for #298.

• [ ] Step 5: Census

Create the overlay:

internal/provision/testdata/plan14-root-reads-region-ha-2a.tsv:

````text

Plan 14 census overlay region-ha-2a (region HA spec 2, plan 2a: docs/superpowers/plans/2026-10-16-region-ha-2a-core.md), loaded last.

Task 2: the new internal/lbhealth package (the load balancer health state). Envelope-only: no root-read unit changed or was added, nothing runs as root that did not before, and no registry, /data, release or claim read was added, so the legacy seal stands.

@coverage internal/lbhealth ad1211a01a4eb3d02cd8ed95cbff0f8b17b3976ce9c0279ebd2cae2f20c07124 ````

Load it last in TestPlan14RootReadCensus:

```diff --- a/internal/provision/plan14_root_reads_test.go +++ b/internal/provision/plan14_root_reads_test.go @@ -759,6 +759,17 @@ t.Fatal(err) } seal = "fc38db001ebd8825709a8ca9c41cc0e8f165c1b1967e5967347dc9bd3556afec" + if errs := checkRootReadLegacy(inv, seal); len(errs) != 0 { + t.Fatal(errs) + } + // Census overlay region-ha-2a (region HA spec 2, plan 2a), loaded last. + // Its header says what each plan task added. The accepted-legacy rows + // it adds are census ruling region-ha-2a's (platform-1e), and move the + // seal. + if err = loadRootReadInventory(filepath.Join(root, "internal/provision/testdata/plan14-root-reads-region-ha-2a.tsv"), &inv, true); err != nil { + t.Fatal(err) + } + seal = "fc38db001ebd8825709a8ca9c41cc0e8f165c1b1967e5967347dc9bd3556afec" errs := append(compareRootReads(got, inv), checkRootReadLegacy(inv, seal)...) errs = append(errs, checkRootReadDeclarations(got, inv)...) if len(errs) > 0 { ```

Run the census checks. Expected: PASS.

• [ ] Step 6: Commit

git add internal/lbhealth region_ha_boundary_test.go internal/provision/testdata/plan14-root-reads-region-ha-2a.tsv internal/provision/plan14_root_reads_test.go
git commit -m "region HA 2a: the load balancer health state and the provider boundary tests"

---

Task 3: Box and group roles in the roster

Files:

• Create: internal/hostactors/roles.go, internal/hostactors/roles_test.go
• Modify: internal/hostactors/roster.go (Box.Group, Box.Role; Validate; Replace lets the role change)
• Modify: internal/hostactors/enrollment_confirmation.go (Region.Role)
• Modify: internal/hostactors/enroll.go (automatic enrollment inherits the group's role)
• Modify: region_ha_boundary_test.go, internal/provision/plan9_b8_census_test.go, the overlay

Interfaces:

• Consumes: Directory.Replace, Directory.Read, Clone, ErrRoster.
• Produces: RoleDefault = "", RoleLegacy = "legacy", RolePlatform = "platform"; var ErrRole; ParseBoxRole(string) (string, error) (default, legacy, platform); ParseGroupRole(string) (string, error) (default, legacy); RoleName(string) string; func (d Directory) SetBoxRole(ctx context.Context, sg release.Signer, o Observed, ref, role string) (Observed, error); func GroupMembers(r Roster, region string) []string; func (d Directory) SetGroupRole(ctx context.Context, sg release.Signer, o Observed, region, role string) (Observed, []string, error); fields Box.Group, Box.Role (json group, role, omitempty), Region.Role (json role, omitempty).

A role is written only by the console (the only root signer). Replace keeps comparing every other field of an enrolled box, so Group never changes once set. Automatic enrollment (2b's caller) sets Group and Role from the region's group in the same write that admits the box (A11-1).

• [ ] Step 1: Write the failing tests

internal/hostactors/roles_test.go:

````go package hostactors

import ( "context" "crypto" "crypto/rand" "crypto/rsa" "crypto/sha256" "crypto/x509" "encoding/base64" "encoding/json" "errors" "reflect" "strings" "testing" "time"

"github.com/carlosframework/platform/internal/boxid" "github.com/carlosframework/platform/internal/release" )

func TestParseRoles(t *testing.T) { for in, want := range map[string]string{"default": RoleDefault, "legacy": RoleLegacy, "platform": RolePlatform, " legacy ": RoleLegacy} { if got, err := ParseBoxRole(in); err != nil || got != want { t.Fatalf("ParseBoxRole(%q) = %q, %v", in, got, err) } } for , in := range []string{"", "Legacy", "draining", "none"} { if , err := ParseBoxRole(in); !errors.Is(err, ErrRole) { t.Fatalf("ParseBoxRole(%q) accepted", in) } } if _, err := ParseGroupRole("platform"); !errors.Is(err, ErrRole) { t.Fatal("a group took the platform role") } if got, err := ParseGroupRole("legacy"); err != nil || got != RoleLegacy { t.Fatalf("ParseGroupRole(legacy) = %q, %v", got, err) } if RoleName(RoleDefault) != "default" || RoleName(RolePlatform) != "platform" { t.Fatal("RoleName") } }

func TestBoxRoleIsSignedAndPersists(t *testing.T) { ctx := context.Background() d, sg, o := fixture(t) next, err := d.SetBoxRole(ctx, sg, o, "box", RoleLegacy) if err != nil { t.Fatal(err) } read, err := d.Read(ctx) if err != nil || read.Roster.Boxes["box"].Role != RoleLegacy || read.ETag != next.ETag { t.Fatalf("read back: %+v %v", read.Roster.Boxes["box"], err) } // The same role again writes nothing. same, err := d.SetBoxRole(ctx, sg, next, "box", RoleLegacy) if err != nil || same.ETag != next.ETag { t.Fatalf("idempotent set wrote: %v", err) } // A stale observation loses the CAS. if , err = d.SetBoxRole(ctx, sg, o, "box", RolePlatform); err == nil { t.Fatal("stale roster accepted") } back, err := d.SetBoxRole(ctx, sg, next, "box", RoleDefault) if err != nil || back.Roster.Boxes["box"].Role != RoleDefault { t.Fatalf("back to default: %+v %v", back.Roster.Boxes["box"], err) } if , err = d.SetBoxRole(ctx, sg, back, "nobody", RoleLegacy); !errors.Is(err, ErrRoster) { t.Fatalf("unknown box: %v", err) } if , err = d.SetBoxRole(ctx, sg, back, "box", "boss"); !errors.Is(err, ErrRole) { t.Fatalf("unknown role: %v", err) } // A retired box keeps no role. retired := Clone(back.Roster) b := retired.Boxes["box"] b.State = "retired" retired.Boxes["box"] = b gone, err := d.Replace(ctx, sg, back, retired) if err != nil { t.Fatal(err) } if , err = d.SetBoxRole(ctx, sg, gone, "box", RoleLegacy); !errors.Is(err, ErrRoster) { t.Fatalf("retired box: %v", err) } }

func TestRosterRefusesUnknownRolesAndAGroupChange(t *testing.T) { ctx := context.Background() d, sg, o := fixture(t) bad := Clone(o.Roster) b := bad.Boxes["box"] b.Role = "boss" bad.Boxes["box"] = b if , err := d.Replace(ctx, sg, o, bad); !errors.Is(err, ErrRoster) { t.Fatalf("unknown box role written: %v", err) } moved := Clone(o.Roster) b = moved.Boxes["box"] b.Group = "routers" moved.Boxes["box"] = b if , err := d.Replace(ctx, sg, o, moved); !errors.Is(err, ErrRoster) { t.Fatalf("group rewritten: %v", err) } noGroup := Clone(o.Roster) noGroup.Regions = map[string]Region{"ie": {Account: "123", AWSRegion: "eu-west-1", Role: RoleLegacy}} if , err := d.Replace(ctx, sg, o, noGroup); !errors.Is(err, ErrRoster) { t.Fatalf("a role on a region with no group: %v", err) } platformGroup := Clone(o.Roster) platformGroup.Regions = map[string]Region{"ie": {Account: "123", AWSRegion: "eu-west-1", GroupName: "routers", Role: RolePlatform}} if , err := d.Replace(ctx, sg, o, platformGroup); !errors.Is(err, ErrRoster) { t.Fatalf("a platform group: %v", err) } }

// groupFixture is fixture plus a group in ie and two boxes it enrolled. func groupFixture(t *testing.T) (Directory, release.Signer, Observed) { t.Helper() ctx := context.Background() d, sg, o := fixture(t) next := Clone(o.Roster) next.Regions = map[string]Region{ "ie": {Account: "123", AWSRegion: "eu-west-1", GroupName: "routers", GroupARN: "group-arn", Namespaces: []string{"platform"}, Pools: []string{"default"}}, "uk": {Account: "123", AWSRegion: "eu-west-2"}, } for i, ref := range []string{"g1", "g2"} { next.Boxes[ref] = Box{Enrollment: strings.Repeat(string(rune('c'+i)), 32), Namespace: "platform", BoxID: ref, Label: ref, Region: "ie", Pool: "default", Cloud: Cloud{Provider: "aws", Account: "123", Region: "eu-west-1", InstanceID: "i-" + ref}, EvidenceSHA256: strings.Repeat("b", 64), State: "active", LossIDs: []string{}, Group: "routers"} } o, err := d.Replace(ctx, sg, o, next) if err != nil { t.Fatal(err) } return d, sg, o }

func TestGroupRoleSetsEveryMemberInOneWrite(t *testing.T) { ctx := context.Background() d, sg, o := groupFixture(t) next, set, err := d.SetGroupRole(ctx, sg, o, "ie", RoleLegacy) if err != nil { t.Fatal(err) } if !reflect.DeepEqual(set, []string{"g1", "g2"}) { t.Fatalf("set %v", set) } if next.Roster.Revision != o.Roster.Revision+1 { t.Fatalf("revision %d, want one write", next.Roster.Revision) } r := next.Roster if r.Regions["ie"].Role != RoleLegacy || r.Boxes["g1"].Role != RoleLegacy || r.Boxes["g2"].Role != RoleLegacy { t.Fatalf("roles: %+v %+v %+v", r.Regions["ie"], r.Boxes["g1"], r.Boxes["g2"]) } // The operator-enrolled box in the same region is not the group's. if r.Boxes["box"].Role != RoleDefault { t.Fatalf("operator box took the group's role: %+v", r.Boxes["box"]) } again, , err := d.SetGroupRole(ctx, sg, next, "ie", RoleLegacy) if err != nil || again.ETag != next.ETag { t.Fatalf("idempotent group set wrote: %v", err) } back, , err := d.SetGroupRole(ctx, sg, next, "ie", RoleDefault) if err != nil || back.Roster.Boxes["g1"].Role != RoleDefault || back.Roster.Regions["ie"].Role != RoleDefault { t.Fatalf("back to default: %v", err) } if , , err = d.SetGroupRole(ctx, sg, back, "uk", RoleLegacy); !errors.Is(err, ErrRoster) { t.Fatalf("a region with no group: %v", err) } if , , err = d.SetGroupRole(ctx, sg, back, "ie", RolePlatform); !errors.Is(err, ErrRole) { t.Fatalf("a platform group: %v", err) } }

// Automatic enrollment gives the new box the group's role in the same // roster write (A11-1): a replacement for an excluded group is never // placeable, even for a moment. func TestAutomaticEnrollmentInheritsTheGroupRole(t testing.T) { for _, role := range []string{RoleDefault, RoleLegacy} { t.Run(RoleName(role), func(t testing.T) { ctx := context.Background() d, sg, o := fixture(t) key, err := rsa.GenerateKey(rand.Reader, 2048) if err != nil { t.Fatal(err) } doc := []byte({"accountId":"123","region":"eu-west-1","instanceId":"i-new"}) sum := sha256.Sum256(doc) sig, err := rsa.SignPKCS1v15(rand.Reader, key, crypto.SHA256, sum[:]) if err != nil { t.Fatal(err) } raw, := json.Marshal(boxid.Record{V: 1, Box: "new", IdentityDocument: base64.StdEncoding.EncodeToString(doc), IdentitySignature: base64.StdEncoding.EncodeToString(sig)}) v := boxid.Verifier{Account: "123", Certs: map[string]*x509.Certificate{"eu-west-1": {PublicKey: &key.PublicKey, PublicKeyAlgorithm: x509.RSA}}} in := EnrollmentRequest{Reference: "new", Namespace: "platform", Label: "new", Region: "ie", Pool: "default", AWSRegion: "eu-west-1"} next := Clone(o.Roster) next.Regions = map[string]Region{"ie": {Account: "123", AWSRegion: "eu-west-1", GroupName: "routers", GroupARN: "group-arn", Namespaces: []string{"platform"}, Pools: []string{"default"}, Role: role}} if o, err = d.Replace(ctx, sg, o, next); err != nil { t.Fatal(err) } if , err = d.Invite(ctx, sg, in.Reference, raw); err != nil { t.Fatal(err) } o, a, err := d.BeginEnrollmentAttempt(ctx, sg, o, in, raw, "i-new") if err != nil { t.Fatal(err) } now := time.Now().UTC() c := MembershipConfirmation{PassID: "pass-1", AttemptID: a.AttemptID, ConfirmationID: a.ConfirmationID, EvidenceSHA256: a.EvidenceSHA256, TargetPlacement: "ie", AWSRegion: "eu-west-1", GroupName: "routers", GroupARN: "group-arn", InstanceID: "i-new", Account: "123", AvailabilityZoneRegion: "eu-west-1", LifecycleState: "InService", InstanceState: "running", ObservedAt: now, RequestIDs: []string{"asg", "ec2"}} c.Confirmer.Box, c.Confirmer.Enrollment, c.Confirmer.Region = "box", o.Roster.Boxes["box"].Enrollment, "ie" wire, _ := json.Marshal(c) if err := release.CreateExact(ctx, d.Store, ConfirmationKey(in.Reference, a.AttemptID, a.ConfirmationID), wire); err != nil { t.Fatal(err) } in.AttemptID, in.ConfirmationID = a.AttemptID, a.ConfirmationID admitted, err := d.EnrollAutomatic(ctx, sg, o, v, in, raw, "i-new", now) if err != nil { t.Fatal(err) } b := admitted.Roster.Boxes["new"] if b.Group != "routers" || b.Role != role { t.Fatalf("admitted box group %q role %q, want routers %q", b.Group, b.Role, role) } if admitted.Roster.Revision != o.Roster.Revision+1 { t.Fatal("the role was not set in the enrolling write") } }) } }

// Every other roster write keeps the role: a quarantine, for one, rewrites // the box's entry and must not reopen placement on it. func TestOtherRosterWritesKeepTheRole(t *testing.T) { ctx := context.Background() d, sg, o := fixture(t) o, err := d.SetBoxRole(ctx, sg, o, "box", RoleLegacy) if err != nil { t.Fatal(err) } q, err := d.Quarantine(ctx, sg, o, "box", o.Roster.Boxes["box"].Enrollment, "loss-1") if err != nil { t.Fatal(err) } if b := q.Roster.Boxes["box"]; b.Role != RoleLegacy || b.State != "quarantined" { t.Fatalf("after a quarantine: %+v", b) } read, err := d.Read(ctx) if err != nil || read.Roster.Boxes["box"].Role != RoleLegacy { t.Fatalf("read back: %+v %v", read.Roster.Boxes["box"], err) } } ````

• [ ] Step 2: Run them to see them fail

Run: nice -n 19 go test ./internal/hostactors/ -run 'Role|Roster|Group|Inherits' -count=1 Expected: FAIL to build (undefined: ParseBoxRole, b.Role undefined).

• [ ] Step 3: Write the roles

internal/hostactors/roles.go:

````go package hostactors

import ( "context" "errors" "sort" "strings"

"github.com/carlosframework/platform/internal/release" )

// Box and group roles (region HA spec 2 §7, A11-1). A box's role says // whether placement may choose it: the default role may be chosen, and // legacy and platform never are, for a create, a relocation target or a // move target. The box keeps the instances it has (H25). // // The region's group entry carries a role too. Every box automatic // enrollment admits from the group takes the group's role in the same // roster write, and changing the group's role changes the role of every // box enrolled from it, so a replacement launched after the group was // excluded is never placeable, even for a moment. // // The roster stores the default role as "" so a roster written before // roles existed reads as all-default. const ( RoleDefault = "" RoleLegacy = "legacy" RolePlatform = "platform" )

// ErrRole is a role name a box or group cannot take. var ErrRole = errors.New("hostactors: not a role this box or group can take")

// ParseBoxRole reads an operator's box role: default, legacy or platform. func ParseBoxRole(s string) (string, error) { switch strings.TrimSpace(s) { case "default": return RoleDefault, nil case RoleLegacy: return RoleLegacy, nil case RolePlatform: return RolePlatform, nil } return "", ErrRole }

// ParseGroupRole reads an operator's group role: default or legacy. A // group is never the platform box. func ParseGroupRole(s string) (string, error) { switch strings.TrimSpace(s) { case "default": return RoleDefault, nil case RoleLegacy: return RoleLegacy, nil } return "", ErrRole }

// RoleName is the operator's word for a stored role. func RoleName(role string) string { if role == RoleDefault { return "default" } return role }

func validBoxRole(role string) bool { return role == RoleDefault || role == RoleLegacy || role == RolePlatform }

func validGroupRole(role string) bool { return role == RoleDefault || role == RoleLegacy }

// SetBoxRole sets one box's role in a fresh roster write over o. The box // must be enrolled and not retired. Setting the role it already has // changes nothing and returns o. func (d Directory) SetBoxRole(ctx context.Context, sg release.Signer, o Observed, ref, role string) (Observed, error) { if !validBoxRole(role) { return Observed{}, ErrRole } b, ok := o.Roster.Boxes[ref] if !ok || b.State == "retired" { return Observed{}, ErrRoster } if b.Role == role { return o, nil } r := Clone(o.Roster) b = r.Boxes[ref] b.Role = role r.Boxes[ref] = b return d.Replace(ctx, sg, o, r) }

// GroupMembers returns, sorted, the boxes enrolled from region's group // that are not retired: the boxes a group role change applies to. func GroupMembers(r Roster, region string) []string { g, ok := r.Regions[region] if !ok || g.GroupName == "" { return nil } var out []string for ref, b := range r.Boxes { if b.Region == region && b.Group == g.GroupName && b.State != "retired" { out = append(out, ref) } } sort.Strings(out) return out }

// SetGroupRole sets region's group role and, in the same roster write, the // role of every box enrolled from that group (GroupMembers). The region // must have a group. It returns the roster and the boxes it set. func (d Directory) SetGroupRole(ctx context.Context, sg release.Signer, o Observed, region, role string) (Observed, []string, error) { if !validGroupRole(role) { return Observed{}, nil, ErrRole } g, ok := o.Roster.Regions[region] if !ok || g.GroupName == "" { return Observed{}, nil, ErrRoster } members := GroupMembers(o.Roster, region) changed := g.Role != role for , ref := range members { changed = changed || o.Roster.Boxes[ref].Role != role } if !changed { return o, members, nil } r := Clone(o.Roster) g.Role = role r.Regions[region] = g for , ref := range members { b := r.Boxes[ref] b.Role = role r.Boxes[ref] = b } next, err := d.Replace(ctx, sg, o, r) if err != nil { return Observed{}, nil, err } return next, members, nil } ````

```diff --- a/internal/hostactors/roster.go +++ b/internal/hostactors/roster.go @@ -39,6 +39,14 @@ State string json:"state" QuarantineRevision uint64 json:"quarantinerevision" LossIDs []string json:"lossids" + // Group is the Auto Scaling group automatic enrollment admitted the box + // from (the region's GroupName then); "" for a box an operator + // enrolled. It never changes. + Group string json:"group,omitempty" + // Role is the box's placement role (roles.go): "" (default), legacy or + // platform. It is the one field besides the state fields a roster + // write may change on an enrolled box. + Role string json:"role,omitempty" } type Console struct { EnrolledAt time.Time json:"enrolledat,omitzero"` @@ -114,11 +122,19 @@ if b.State != "active" && b.State != "quarantined" && b.State != "retired" { return ErrRoster } + if !validBoxRole(b.Role) { + return ErrRoster + } if b.State == "quarantined" && (b.QuarantineRevision == 0 || len(b.LossIDs) == 0) { return ErrRoster } ids[b.Enrollment] = true clouds[b.Cloud] = true + } + for , g := range r.Regions { + if !validGroupRole(g.Role) || (g.Role != RoleDefault && g.GroupName == "") { + return ErrRoster + } } for id, c := range r.Consoles { if !segment(id) || !token.MatchString(c.Enrollment) || ids[c.Enrollment] || c.InstanceID != id || c.DeploymentBucket != bucket || len(c.Namespaces) == 0 { @@ -246,6 +262,8 @@ z.QuarantineRevision = 0 a.LossIDs = nil z.LossIDs = nil + a.Role = "" + z.Role = "" if !reflect.DeepEqual(a, z) || b.QuarantineRevision < old.QuarantineRevision { return Observed{}, ErrRoster } ````

```diff --- a/internal/hostactors/enrollment_confirmation.go +++ b/internal/hostactors/enrollment_confirmation.go @@ -18,6 +18,9 @@ AWSRegion string json:"awsregion" GroupName string json:"groupname" GroupARN string json:"grouparn" + // Role is the role every box automatic enrollment admits from the + // group takes (roles.go): "" (default) or legacy. + Role string json:"role,omitempty" } type TerminatedEnrollment struct { PassID string json:"passid" ```

````diff --- a/internal/hostactors/enroll.go +++ b/internal/hostactors/enroll.go @@ -68,7 +68,15 @@ a.Consumed = true r.EnrollmentAttempts[in.Reference] = a }

• r.Boxes[in.Reference] = Box{Enrollment: enrollment, Namespace: in.Namespace, BoxID: in.Reference, Label: in.Label, Region: in.Region, Pool: in.Pool, Cloud: cloud, EvidenceSHA256: evidence.RecordSHA256, State: "active", LossIDs: []string{}} + box := Box{Enrollment: enrollment, Namespace: in.Namespace, BoxID: in.Reference, Label: in.Label, Region: in.Region, Pool: in.Pool, Cloud: cloud, EvidenceSHA256: evidence.RecordSHA256, State: "active", LossIDs: []string{}} + if automatic { + // A box the group launched takes the group's role in this same + // write (region HA spec 2 §7, A11-1): a replacement for an + // excluded group is never placeable, even for a moment. + g := r.Regions[in.Region] + box.Group, box.Role = g.GroupName, g.Role + } + r.Boxes[in.Reference] = box return d.Replace(ctx, sg, o, r) }

````

• [ ] Step 4: Run the tests

Run: nice -n 19 go test ./internal/hostactors/ -count=1 Expected: PASS (the package's whole suite, about 15 seconds).

• [ ] Step 5: Boundary list and census

````diff --- a/regionhaboundarytest.go +++ b/regionhaboundarytest.go @@ -20,6 +20,7 @@ // an AWS package. var regionHACoreFiles = []string{ "internal/lbhealth/lbhealth.go", + "internal/hostactors/roles.go", }

// regionHAAdapterFiles are spec 2's AWS adapters. Each is named for AWS so ````

Directory.Replace is in the record-write census (it names Put). Its digest changes:

````diff --- a/internal/provision/plan9b8censustest.go +++ b/internal/provision/plan9b8censustest.go @@ -791,7 +791,7 @@ {"internal/host/snapshot_loop.go", "putReplacing", "nonrecord", "sel:Put", "11e8639effce142eab707b67cbdd425d0d7cea510ddf95d292bdba8f79c9b510"}, {"internal/hostactors/censuskey.go", "Directory.BindCensusKey", "nonrecord", "sel:Put", "7d9d673435e8c8a6aac3105e3a754bde2adcbc5d603945dd599cef703f334a5b"}, {"internal/hostactors/invitation.go", "Directory.Invite", "nonrecord", "sel:Put", "356618a75a23d0157d8e0402e20187a42956f67a4332d698bb22bf05133ac489"},

• {"internal/hostactors/roster.go", "Directory.Replace", "nonrecord", "sel:Put", "677bd94746cc5d3a5cd966d221f80b509ae096083a8ff4a0ee817d4c12c4a75f"}, + {"internal/hostactors/roster.go", "Directory.Replace", "nonrecord", "sel:Put", "e6e8ac0070650dfacc4a438b080cb69738530b3d9013de55c1bcf06431b0a847"}, {"internal/hostbox/regionalworker.go", "RegionalWorker.Census", "key-site", "lit:instances", "477bcb896b5ae8ec80af592d68eceb7992ad8a23cd6cff6043ad03b3ef2a8085"}, {"internal/hostbox/requeststore.go", "RequestStore.CopyVersion", "versioned-transport", "def:CopyVersion,sel:CopyVersion", "3a50049ce7ca7f2cbc770bef46218944cdc7000205a86c20e4932f0b510da28f"}, {"internal/hostbox/requeststore.go", "RequestStore.PutIfAbsent", "versioned-transport", "def:PutIfAbsent,sel:PutIfAbsent", "3666cbe5d2b91c1c652ffb4459c10a374fbd92e325e4e5c574fa3525d0ce2ffe"}, ````

Add this comment line after the overlay's last comment line:

````text

Task 3: box and group roles in the signed roster (internal/hostactors roles.go; Box.Role and Box.Group, Region.Role, Replace lets the role change, automatic enrollment inherits the group's role). Envelope-only: Directory.Replace, Roster.Validate and Directory.enroll are not root-read units, no root-read unit changed or was added, and the legacy seal stands.

````

Then set these rows. A @coverage row replaces the overlay's row for the same directory, or is appended when there is none; every other row is appended:

```text @coverage internal/hostactors 6736698380a247764ada687f2476f89382fc32a1183a50c06afe75967d716eab ```

Run the census checks and go test . -run 'TestRegionHA(Core|Adapters|S3)' -count=1. Expected: PASS.

• [ ] Step 6: Commit

git add internal/hostactors region_ha_boundary_test.go internal/provision
git commit -m "region HA 2a: box and group roles in the signed roster"

---

Task 4: The placement policy

Files:

• Create: internal/boxchoice/policy.go, internal/boxchoice/policy_test.go
• Modify: internal/boxchoice/boxchoice.go (headroom totals and the generation carried through Eligible)
• Modify: internal/edgesfile/snapshot.go (the reader's half: totals, generation, BoxChoice)
• Create: internal/edgesfile/snapshot_headroom_test.go
• Modify: region_ha_boundary_test.go, the overlay

Interfaces:

• Consumes: boxchoice.Candidate, Load, Eligible, Choose (its hash rule).
• Produces: Headroom.MemTotalMiB, Headroom.DiskTotalMiB (json mem_total_mib, disk_total_mib); Box.Generation string (json generation); Candidate.MemTotalMiB, Candidate.DiskTotalMiB, Candidate.Generation; MinMemPercent = 15, MinDiskPercent = 10, MaxEdgesAge = 15 * time.Minute; func ExcludeRoles(cands []Candidate, roles map[string]string) []Candidate; func Placeable(cands []Candidate, roles map[string]string, now time.Time) []Candidate; func ChooseRoomiest(host string, cands []Candidate, running map[string]Load, app string) (Candidate, bool); edgesfile.PlacementView.Generation and its headroom totals.

Eligible only carries the new fields through; the new rules are Placeable and ChooseRoomiest, over candidates Eligible has already filtered for identity (§25). Any role other than the default excludes a box, so a role this build does not know fails closed.

• [ ] Step 1: Write the failing tests

internal/boxchoice/policy_test.go:

````go package boxchoice

import ( "reflect" "testing" "time" )

var policyNow = time.Date(2026, 10, 16, 12, 0, 0, 0, time.UTC)

// roomy is a candidate every §7 rule passes at policyNow. func roomy(id string, memFree int64) Candidate { return Candidate{BoxID: id, Region: "eu-west-2", MemFreeMiB: memFree, MemTotalMiB: 8000, DiskFreeMiB: 50000, DiskTotalMiB: 100000, Generation: policyNow.Add(-5 * time.Minute).Format(time.RFC3339)} }

func TestPlaceableAppliesEachRule(t testing.T) { gen := func(d time.Duration) string { return policyNow.Add(d).Format(time.RFC3339) } cases := []struct { name string mut func(Candidate) role string want bool }{ {"every rule holds", nil, "", true}, {"memory exactly 15%", func(c Candidate) { c.MemFreeMiB = 1200 }, "", true}, {"memory under 15%", func(c Candidate) { c.MemFreeMiB = 1199 }, "", false}, {"no memory total", func(c Candidate) { c.MemTotalMiB = 0 }, "", false}, {"disk exactly 10%", func(c Candidate) { c.DiskFreeMiB = 10000 }, "", true}, {"disk under 10%", func(c Candidate) { c.DiskFreeMiB = 9999 }, "", false}, {"no disk total", func(c Candidate) { c.DiskTotalMiB = 0 }, "", false}, {"edges file exactly 15 minutes old", func(c Candidate) { c.Generation = gen(-15 time.Minute) }, "", true}, {"edges file older than 15 minutes", func(c Candidate) { c.Generation = gen(-15time.Minute - time.Second) }, "", false}, {"edges file a minute in the future", func(c Candidate) { c.Generation = gen(time.Minute) }, "", true}, {"edges file further in the future", func(c Candidate) { c.Generation = gen(time.Minute + time.Second) }, "", false}, {"no generation", func(c Candidate) { c.Generation = "" }, "", false}, {"unreadable generation", func(c Candidate) { c.Generation = "yesterday" }, "", false}, {"legacy role", nil, "legacy", false}, {"platform role", nil, "platform", false}, {"a role this build does not know", nil, "draining", false}, } for _, tc := range cases { t.Run(tc.name, func(t *testing.T) { c := roomy("i-0aaa", 4000) if tc.mut != nil { tc.mut(&c) } got := Placeable([]Candidate{c}, map[string]string{"i-0aaa": tc.role}, policyNow) if (len(got) == 1) != tc.want { t.Fatalf("placeable = %v, want %v", ids(got), tc.want) } }) } }

func TestExcludeRolesKeepsOrderAndDefaults(t *testing.T) { cands := []Candidate{roomy("c", 1), roomy("a", 1), roomy("b", 1)} got := ExcludeRoles(cands, map[string]string{"a": "legacy", "b": ""}) if !reflect.DeepEqual(ids(got), []string{"c", "b"}) { t.Fatalf("got %v", ids(got)) } if got := ExcludeRoles(cands, nil); len(got) != 3 { t.Fatalf("no roles excluded %v", ids(got)) } }

func TestChooseRoomiestSpreadsThenMostMemoryThenHash(t *testing.T) { running := map[string]Load{"a": {Hosts: 1, Apps: map[string]int{"bab/jam": 1}}} // a has the most memory but already runs the app: spread wins. got, ok := ChooseRoomiest("h.example", []Candidate{roomy("a", 7000), roomy("b", 2000), roomy("c", 3000)}, running, "bab/jam") if !ok || got.BoxID != "c" { t.Fatalf("spread then memory: got %q", got.BoxID) } // Spread never refuses: every box runs the app, so memory decides. all := map[string]Load{"a": running["a"], "b": running["a"]} got, = ChooseRoomiest("h.example", []Candidate{roomy("a", 1000), roomy("b", 2000)}, all, "bab/jam") if got.BoxID != "b" { t.Fatalf("all run the app: got %q", got.BoxID) } // Fewer hosts is no longer a rule: memory beats it. busy := map[string]Load{"a": {Hosts: 50}, "b": {Hosts: 1}} got, = ChooseRoomiest("h.example", []Candidate{roomy("a", 5000), roomy("b", 4000)}, busy, "bab/jam") if got.BoxID != "a" { t.Fatalf("memory over host count: got %q", got.BoxID) } // Equal memory: the P5-D2 hash, whatever the order. one, := ChooseRoomiest("h.example", []Candidate{roomy("a", 1000), roomy("b", 1000)}, nil, "bab/jam") two, := ChooseRoomiest("h.example", []Candidate{roomy("b", 1000), roomy("a", 1000)}, nil, "bab/jam") if one.BoxID != two.BoxID { t.Fatalf("order changed the answer: %q %q", one.BoxID, two.BoxID) } want, := Choose("h.example", []Candidate{roomy("a", 1000), roomy("b", 1000)}, nil, "bab/jam") if one.BoxID != want.BoxID { t.Fatalf("hash tie-break differs from spec 1's: %q, want %q", one.BoxID, want.BoxID) } if , ok := ChooseRoomiest("h.example", nil, nil, "bab/jam"); ok { t.Fatal("chose from nothing") } }

func TestEligibleCarriesTotalsAndGeneration(t *testing.T) { body := haBox("i-0aaa", func(m map[string]any) { m["headroom"] = map[string]any{"memfreemib": 1000, "diskfreemib": 2000, "memtotalmib": 8000, "disktotalmib": 40000, "awake": 1} m["generation"] = "2026-10-16T11:55:00Z" }) got := Eligible(views(t, body), rosterFor("i-0aaa"), "uk", "eu-west-2", "ie", testKeyFP, nil) want := Candidate{BoxID: "i-0aaa", Region: "eu-west-2", MemFreeMiB: 1000, DiskFreeMiB: 2000, MemTotalMiB: 8000, DiskTotalMiB: 40000, Awake: 1, Generation: "2026-10-16T11:55:00Z"} if len(got) != 1 || got[0] != want { t.Fatalf("got %+v, want %+v", got, want) } } ````

internal/edgesfile/snapshot_headroom_test.go:

````go package edgesfile

import ( "encoding/json" "testing" )

// Region HA spec 2 §7: the placement policy reads the headroom totals and // the file's generation, so BoxChoice carries both. func TestBoxChoiceCarriesHeadroomTotalsAndGeneration(t *testing.T) { var f PlacementView body := {"v":1,"box":"box-a","box_id":"i-a","generation":"2026-10-16T11:55:00Z","headroom":{"mem_free_mib":512,"disk_free_mib":2048,"mem_total_mib":4096,"disk_total_mib":20480,"awake":3},"hosts":{}} if err := json.Unmarshal([]byte(body), &f); err != nil { t.Fatal(err) } f.Fresh = true got := Snapshot{Files: []PlacementView{f}, Complete: true, BoxesComplete: true}.BoxChoice() if len(got.Boxes) != 1 { t.Fatalf("boxes %+v", got.Boxes) } b := got.Boxes[0] if b.Generation != "2026-10-16T11:55:00Z" || b.Headroom == nil || b.Headroom.MemTotalMiB != 4096 || b.Headroom.DiskTotalMiB != 20480 || b.Headroom.MemFreeMiB != 512 || b.Headroom.DiskFreeMiB != 2048 || b.Headroom.Awake != 3 { t.Fatalf("box lost a field: %+v %+v", b, b.Headroom) } } ````

• [ ] Step 2: Run them to see them fail

Run: nice -n 19 go test ./internal/boxchoice/ ./internal/edgesfile/ -count=1 Expected: FAIL to build (undefined: Placeable, unknown field MemTotalMiB).

• [ ] Step 3: Write the policy

internal/boxchoice/policy.go:

````go package boxchoice

import ( "crypto/sha256" "encoding/hex" "sort" "time" )

// Region HA spec 2's placement policy (§7). Its rules run over candidates // Eligible has already filtered for identity, placement and configuration // (§25): a provider-neutral box identity (#299) changes only Eligible, // never these.

// The thresholds of §7. const ( // MinMemPercent: a box is placeable only while its free memory is at // least this share of its total, the memory alarm's own line. MinMemPercent = 15 // MinDiskPercent: and its data volume's free space at least this // share of the volume. MinDiskPercent = 10 // MaxEdgesAge: and its edges file, which carries the headroom, is no // older than this. Files republish every 5 minutes plus up to a // minute of jitter, so a healthy box always has one within two // intervals and a margin (A1-8). MaxEdgesAge = 15 * time.Minute // maxEdgesSkew is how far in the future a file's generation may be // before it is not believed: box clocks are synchronised, not equal. maxEdgesSkew = time.Minute )

// ExcludeRoles drops every candidate whose roster role is not the default // one (roles is box id -> role, "" for the default). Legacy and platform // boxes, and any role this build does not know, take no new instances. // The order of cands is kept. func ExcludeRoles(cands []Candidate, roles map[string]string) []Candidate { var out []Candidate for _, c := range cands { if roles[c.BoxID] == "" { out = append(out, c) } } return out }

// headroomFresh reports whether a candidate's edges file is recent enough // at now for its headroom to be trusted. func headroomFresh(generation string, now time.Time) bool { at, err := time.Parse(time.RFC3339, generation) if err != nil { return false } age := now.Sub(at) return age <= MaxEdgesAge && age >= -maxEdgesSkew }

// Placeable is §7's eligibility on top of Eligible's: the default role, // free memory at least MinMemPercent of the total, free disk at least // MinDiskPercent of the volume, and an edges file no older than // MaxEdgesAge at now. A box that publishes no totals is not placeable: its // headroom cannot be judged. The order of cands is kept. func Placeable(cands []Candidate, roles map[string]string, now time.Time) []Candidate { var out []Candidate for _, c := range ExcludeRoles(cands, roles) { switch { case c.MemTotalMiB <= 0 || c.MemFreeMiB100 < c.MemTotalMiBMinMemPercent: case c.DiskTotalMiB <= 0 || c.DiskFreeMiB100 < c.DiskTotalMiBMinDiskPercent: case !headroomFresh(c.Generation, now): default: out = append(out, c) } } return out }

// ChooseRoomiest is §7's choice among placeable candidates: prefer a box // running no instance of app (soft spread: it never refuses), then the // most free memory, then the greatest hexadecimal SHA-256 of host + "\x00" // + box id (claim P5-D2), then the smallest box id. The answer does not // depend on the order of cands. The bool is false only when cands is // empty. func ChooseRoomiest(host string, cands []Candidate, running map[string]Load, app string) (Candidate, bool) { if len(cands) == 0 { return Candidate{}, false } tie := func(ref string) string { sum := sha256.Sum256([]byte(host + "\x00" + ref)) return hex.EncodeToString(sum[:]) } sorted := append([]Candidate(nil), cands...) sort.SliceStable(sorted, func(i, j int) bool { a, b := sorted[i], sorted[j] aRuns, bRuns := running[a.BoxID].Apps[app] > 0, running[b.BoxID].Apps[app] > 0 switch { case aRuns != bRuns: return !aRuns case a.MemFreeMiB != b.MemFreeMiB: return a.MemFreeMiB > b.MemFreeMiB case tie(a.BoxID) != tie(b.BoxID): return tie(a.BoxID) > tie(b.BoxID) } return a.BoxID < b.BoxID }) return sorted[0], true } ````

````diff --- a/internal/boxchoice/boxchoice.go +++ b/internal/boxchoice/boxchoice.go @@ -21,11 +21,15 @@ // region HA executor. const Capability = "region-ha-1"

-// Headroom is what a box reports it has free. +// Headroom is what a box reports it has free, and (region HA spec 2 §7) +// the totals the free figures are fractions of. A total of 0 is a writer +// that predates it, or a reading that failed. type Headroom struct {

• MemFreeMiB int64 json:"mem_free_mib"
• DiskFreeMiB int64 json:"disk_free_mib"
• Awake int json:"awake" + MemFreeMiB int64 json:"mem_free_mib" + DiskFreeMiB int64 json:"disk_free_mib" + MemTotalMiB int64 json:"mem_total_mib" + DiskTotalMiB int64 json:"disk_total_mib" + Awake int json:"awake" }

// Box is one box's edges file (control/edges/<box>.json, written by @@ -48,6 +52,9 @@ Capabilities []string json:"capabilities" Draining bool json:"draining" Headroom *Headroom json:"headroom" + // Generation is when the box last wrote the file (RFC 3339). Spec 2's + // policy trusts headroom only from a recent file (Placeable). + Generation string json:"generation" // Fresh is the caller's verdict that the box's own status snapshot is // still fresh. An edges file outlives its box, so the file alone is // not liveness. @@ -76,10 +83,13 @@ // Candidate is one box that may be named as a host's executor (region HA // spec 1 §2, "eligible box"). type Candidate struct {

• BoxID string
• Region string
• MemFreeMiB, DiskFreeMiB int64
• Awake int + BoxID string + Region string + MemFreeMiB, DiskFreeMiB int64 + MemTotalMiB, DiskTotalMiB int64 + Awake int + // Generation is the box's edges file's (Box.Generation). + Generation string }

// Load is what claims already place on a box. @@ -132,9 +142,10 @@ case !slices.Contains(f.Capabilities, Capability): case f.Draining, lost[f.BoxID]: default:

• c := Candidate{BoxID: f.BoxID, Region: rb.Cloud.Region} + c := Candidate{BoxID: f.BoxID, Region: rb.Cloud.Region, Generation: f.Generation} if f.Headroom != nil { c.MemFreeMiB, c.DiskFreeMiB, c.Awake = f.Headroom.MemFreeMiB, f.Headroom.DiskFreeMiB, f.Headroom.Awake + c.MemTotalMiB, c.DiskTotalMiB = f.Headroom.MemTotalMiB, f.Headroom.DiskTotalMiB } out = append(out, c) } ````

```diff --- a/internal/edgesfile/snapshot.go +++ b/internal/edgesfile/snapshot.go @@ -58,11 +58,18 @@ Capabilities []string json:"capabilities" Draining bool json:"draining"` Headroom *struct {

• MemFreeMiB int64 json:"mem_free_mib"
• DiskFreeMiB int64 json:"disk_free_mib"
• Awake int json:"awake" + MemFreeMiB int64 json:"mem_free_mib" + DiskFreeMiB int64 json:"disk_free_mib" + MemTotalMiB int64 json:"mem_total_mib" + DiskTotalMiB int64 json:"disk_total_mib" + Awake int json:"awake" } json:"headroom" Hosts map[string]json.RawMessage json:"hosts" + + // Generation is when the box last wrote the file (RFC 3339), which + // region HA spec 2's placement policy reads as headroom's age. + Generation string json:"generation" + // The moves protocol (moves spec §11), read only by the Phase 1 move // door's floor check (selectMoveBoxes); Task 13 adds the writer's side. // MovesProtocol is the protocol the box speaks (absent: 0). @@ -246,10 +253,11 @@ Box: f.Box, BoxID: f.BoxID, EIP: f.EIP, Region: f.Region, Placement: f.Placement, HibernationConfigured: f.HibernationConfigured, ServesSharedPool: f.ServesSharedPool, AcceptsInbound: f.AcceptsInbound, SecretsKeyFP: f.SecretsKeyFP, Capabilities: f.Capabilities,
• Draining: f.Draining, Fresh: f.Fresh, + Draining: f.Draining, Generation: f.Generation, Fresh: f.Fresh, } if f.Headroom != nil {
• b.Headroom = &boxchoice.Headroom{MemFreeMiB: f.Headroom.MemFreeMiB, DiskFreeMiB: f.Headroom.DiskFreeMiB, Awake: f.Headroom.Awake} + b.Headroom = &boxchoice.Headroom{MemFreeMiB: f.Headroom.MemFreeMiB, DiskFreeMiB: f.Headroom.DiskFreeMiB, + MemTotalMiB: f.Headroom.MemTotalMiB, DiskTotalMiB: f.Headroom.DiskTotalMiB, Awake: f.Headroom.Awake} } out.Boxes = append(out.Boxes, b) } ````

• [ ] Step 4: Run the tests

Run: nice -n 19 go test ./internal/boxchoice/ ./internal/edgesfile/ -count=1 && nice -n 19 go test ./internal/console/ -run 'Box|Eligib|Executor|RegionCand' -count=1 Expected: PASS.

• [ ] Step 5: Boundary list and census

````diff --- a/regionhaboundarytest.go +++ b/regionhaboundarytest.go @@ -21,6 +21,7 @@ var regionHACoreFiles = []string{ "internal/lbhealth/lbhealth.go", "internal/hostactors/roles.go", + "internal/boxchoice/policy.go", }

// regionHAAdapterFiles are spec 2's AWS adapters. Each is named for AWS so ````

Add this comment line after the overlay's last comment line:

````text

Task 4: the placement policy (internal/boxchoice policy.go; headroom totals and the edges file's generation carried through Eligible and edgesfile.Snapshot.BoxChoice). Envelope-only: no root-read unit changed or was added, and the legacy seal stands.

````

Then set these rows. A @coverage row replaces the overlay's row for the same directory, or is appended when there is none; every other row is appended:

```text @coverage internal/boxchoice 1bfd561bcb48b51e60e63f99e086fd44d3b1be2160e28b3ed0253a14bbb9ff66 @coverage internal/edgesfile da42c799d62e065817b5cce4f9a3ead1308dbd379615a85a7b14533c16d7e804 ```

Run the census checks and go test . -run 'TestRegionHA(Core|Adapters|S3)' -count=1. Expected: PASS.

• [ ] Step 6: Commit

git add internal/boxchoice internal/edgesfile region_ha_boundary_test.go internal/provision/testdata/plan14-root-reads-region-ha-2a.tsv
git commit -m "region HA 2a: the placement policy over the eligible set"

---

Task 5: Boxes publish their total memory and data volume size

Files:

• Modify: internal/host/headroom.go (headroomFor fills the totals; capacityMiB)
• Modify: internal/host/edgesfile.go (edgeHeadroom's two fields)
• Modify: internal/host/headroom_test.go
• Modify: the overlay

Interfaces:

• Consumes: readHeadroom, readMemBytes, diskBytes (unchanged).
• Produces: edgeHeadroom.MemTotalMiB, edgeHeadroom.DiskTotalMiB (json mem_total_mib, disk_total_mib, omitempty), the fields Task 4's reader takes; func capacityMiB(mem func() (total, avail *int64), disk func() (total, avail uint64, err error)) (memMiB, diskMiB int64).

Headroom is published only with the region HA capability (headroomFor returns nil without it), so with region HA off the edges file is byte-identical to main's. readHeadroom and readMemBytes are accepted-legacy units and stay as they are: headroomFor calls the existing readers.

• [ ] Step 1: Write the failing tests

````diff --- a/internal/host/headroomtest.go +++ b/internal/host/headroomtest.go @@ -1,6 +1,13 @@ package host

-import "testing" +import ( + "context" + "encoding/json" + "errors" + "testing" + + "github.com/carlosframework/platform/internal/release" +)

func TestParseMemAvailable(t testing.T) { const meminfo = "MemTotal: 16000000 kB\nMemFree: 500000 kB\nMemAvailable: 2263040 kB\n" @@ -24,3 +31,47 @@ t.Fatal("flag on: headroom is nil") } } + +// Region HA spec 2 §7: the totals ride beside the free figures, and a +// failed reading is 0, never a guess. +func TestCapacityMiBReadsTotalsAndZeroesFailures(t testing.T) { + gib := int64(8 << 30) + mem := func() (int64, int64) { return &gib, nil } + disk := func() (uint64, uint64, error) { return 100 << 30, 1 << 30, nil } + if m, d := capacityMiB(mem, disk); m != 8192 || d != 102400 { + t.Fatalf("capacity = %d, %d; want 8192, 102400", m, d) + } + noMem := func() (int64, int64) { return nil, nil } + noDisk := func() (uint64, uint64, error) { return 0, 0, errors.New("statfs") } + if m, d := capacityMiB(noMem, noDisk); m != 0 || d != 0 { + t.Fatalf("failed readings = %d, %d; want 0, 0", m, d) + } +} + +func TestHeadroomCarriesTheDataVolumeSize(t testing.T) { + h := headroomFor([]string{capRegionHA1}, t.TempDir(), edgesTestRegistry(t)) + if h == nil || h.DiskTotalMiB <= 0 { + t.Fatalf("headroom %+v: want the data volume's size", h) + } +} + +func TestEdgesFileWireCarriesHeadroomTotals(t testing.T) { + s, reg := release.NewDirStore(t.TempDir()), edgesTestRegistry(t) + var last []byte + ident := edgesFile{Box: "annie", BoxID: "i-0abc", Capabilities: []string{"region-ha-1"}, + Headroom: &edgeHeadroom{MemFreeMiB: 2210, DiskFreeMiB: 31200, Awake: 14, MemTotalMiB: 7900, DiskTotalMiB: 61000}} + if err := edgesFilePass(context.Background(), s, reg, ident, "1.2.3.4", &last); err != nil { + t.Fatal(err) + } + body, , err := s.Get(context.Background(), "control/edges/annie.json") + if err != nil { + t.Fatal(err) + } + var raw map[string]json.RawMessage + if err := json.Unmarshal(body, &raw); err != nil { + t.Fatal(err) + } + if want := `{"memfreemib":2210,"diskfreemib":31200,"awake":14,"memtotalmib":7900,"disktotal_mib":61000}; string(raw["headroom"]) != want { + t.Fatalf("headroom = %s, want %s", raw["headroom"], want) + } +} ```

• [ ] Step 2: Run them to see them fail

Run: nice -n 19 go test ./internal/host/ -run 'Headroom|CapacityMiB' -count=1 Expected: FAIL to build (undefined: capacityMiB, unknown field MemTotalMiB).

• [ ] Step 3: Write the totals

````diff --- a/internal/host/headroom.go +++ b/internal/host/headroom.go @@ -14,7 +14,23 @@ if len(caps) == 0 { return nil }

• return readHeadroom(dataDir, awakeCount(reg)) + h := readHeadroom(dataDir, awakeCount(reg)) + h.MemTotalMiB, h.DiskTotalMiB = capacityMiB(readMemBytes, func() (uint64, uint64, error) { return diskBytes(dataDir) }) + return h +} + +// capacityMiB is the box's total memory and its data volume's size, in +// MiB: the totals region HA spec 2's placement policy judges the free +// figures against (§7). A reading that fails is 0, which the policy reads +// as a box it cannot judge, and so does not place on. +func capacityMiB(mem func() (total, avail int64), disk func() (total, avail uint64, err error)) (memMiB, diskMiB int64) { + if total, _ := mem(); total != nil && total > 0 { + memMiB = *total / (1 << 20) + } + if total, _, err := disk(); err == nil { + diskMiB = int64(total / (1 << 20)) + } + return memMiB, diskMiB }

// parseMemAvailableMiB returns the MemAvailable line in MiB, or 0. ````

```diff --- a/internal/host/edgesfile.go +++ b/internal/host/edgesfile.go @@ -151,6 +151,11 @@ MemFreeMiB int64 json:"memfreemib" DiskFreeMiB int64 json:"diskfreemib" Awake int json:"awake" + // MemTotalMiB and DiskTotalMiB are what the free figures are shares + // of (region HA spec 2 §7). Additive and omitted when 0, a reading + // that failed: a reader takes absent and 0 alike, as unknown. + MemTotalMiB int64 json:"memtotalmib,omitempty" + DiskTotalMiB int64 json:"disktotalmib,omitempty"` }

const capRegionHA1 = "region-ha-1" ````

• [ ] Step 4: Run the tests

Run: nice -n 19 go test ./internal/host/ -run 'Edges|Headroom|CapacityMiB|ParseMemAvailable' -count=1 Expected: PASS, including main's TestEdgesFileRawWireBodyCarriesBoxEvidence (the totals are omitted when zero).

• [ ] Step 5: Census

Add this comment line after the overlay's last comment line:

````text

Task 5: the edges file's headroom carries the box's total memory and data volume size (headroomFor and capacityMiB, internal/host/headroom.go; edgeHeadroom's two additive fields). headroomFor calls the existing readMemBytes and diskBytes and adds no read of its own. Envelope-only: readHeadroom and readMemBytes are unchanged, no root-read unit changed or was added, and the legacy seal stands.

````

Then set these rows. A @coverage row replaces the overlay's row for the same directory, or is appended when there is none; every other row is appended:

```text @coverage internal/host c1cbcb478b2d64b3114a3c33be7d847cf8b266694f42f0fab4e9b28a51bc7f9c ```

Run the census checks. Expected: PASS.

• [ ] Step 6: Commit

git add internal/host/headroom.go internal/host/headroom_test.go internal/host/edgesfile.go internal/provision/testdata/plan14-root-reads-region-ha-2a.tsv
git commit -m "region HA 2a: edges files carry total memory and disk"

---

Task 6: Creates and relocation targets choose by the policy

Files:

• Modify: internal/hostops/executor.go (Choose, Permits, PlaceableBoxes, Capacity, placeable)
• Modify: internal/hostops/executor_test.go, internal/hostrelocate/chooser_test.go (fixtures publish roomy, fresh headroom)
• Modify: the overlay

Interfaces:

• Consumes: boxchoice.Placeable, boxchoice.ChooseRoomiest (Task 4); hostactors.Box.Role (Task 3).
• Produces: func (x ExecutorChooser) PlaceableBoxes(ctx context.Context, placement string) ([]string, error) (Tasks 11 and 12 read it) and func (x ExecutorChooser) Capacity(ctx context.Context, placement string) (int, error) (Task 13 publishes it, and extends it to answer the roster's AWS region); Choose and Permits now apply the roles and thresholds. ExecutorChooser exists only with CARLOS_REGION_HA on (the console binds it only then, and the box only in its region HA relocation runtime), so the policy is dormant with it off.

ExecutorChooser.eligible is an accepted-legacy unit (merge ruling 2) and does not change: the policy runs over its result. Relocation's RegionChooser uses Choose and Permits, so relocation targets follow the same policy.

• [ ] Step 1: Write the failing tests

````diff --- a/internal/hostops/executortest.go +++ b/internal/hostops/executortest.go @@ -43,7 +43,12 @@ roster.Boxes[ref] = hostactors.Box{Enrollment: strings.Repeat(string("abcdef"[i]), 32), Namespace: "platform", BoxID: ref, Label: ref, Region: "ie", Pool: "default", Cloud: hostactors.Cloud{Provider: "aws", Account: "123", Region: "eu-west-1", InstanceID: "i-" + ref}, EvidenceSHA256: strings.Repeat("c", 64), State: "active"} snap.Boxes = append(snap.Boxes, boxchoice.Box{BoxID: ref, EIP: "203.0.113.1", Region: "eu-west-1", Placement: "ie", Fresh: true,

• HibernationConfigured: &yes, ServesSharedPool: &yes, AcceptsInbound: &yes, SecretsKeyFP: "fp", Capabilities: []string{boxchoice.Capability}}) + HibernationConfigured: &yes, ServesSharedPool: &yes, AcceptsInbound: &yes, SecretsKeyFP: "fp", Capabilities: []string{boxchoice.Capability}, + // Equal, roomy headroom from a file a minute old: spec 2's + // policy (§7) admits every box, and the choice falls through + // to the hash unless a test changes it. + Headroom: &boxchoice.Headroom{MemFreeMiB: 4000, MemTotalMiB: 8000, DiskFreeMiB: 50000, DiskTotalMiB: 100000}, + Generation: "2026-10-07T08:59:00Z"}) } if edit != nil { edit(&roster) @@ -68,9 +73,9 @@ Body: []byte({"host":"shop.bab.oncarlos.com","account":"bab","app":"jam","region":"ie"})} }

-func TestExecutorChooserSpreadsThenBalancesThenHashes(t *testing.T) {

• ctx := context.Background()
• x, , , loads := chooserFixture(t, "box-a", "box-b") +func TestExecutorChooserSpreadsThenPrefersMemoryThenHashes(t *testing.T) { + ctx := context.Background() + x, , snap, loads := chooserFixture(t, "box-a", "box-b") absent := hostclaim.Observed{Host: "shop.bab.oncarlos.com", Absent: true} first, err := x.Choose(ctx, absent, chooserRecord()) if err != nil { @@ -85,10 +90,80 @@ if got, := x.Choose(ctx, absent, chooserRecord()); got.Box != other { t.Fatalf("soft spread: got %s, want %s", got.Box, other) }
• // Both run the app: the fewest hosts wins. + // Both run the app: the most free memory wins (spec 2 §7), whatever + // the host counts say. loads[other] = boxchoice.Load{Hosts: 5, Apps: map[string]int{"bab/jam": 1}}
• if got, _ := x.Choose(ctx, absent, chooserRecord()); got.Box != first.Box {
• t.Fatalf("headroom: got %s, want %s", got.Box, first.Box) + for i := range snap.Boxes { + if snap.Boxes[i].BoxID == other { + snap.Boxes[i].Headroom.MemFreeMiB = 6000 + } + } + if got, := x.Choose(ctx, absent, chooserRecord()); got.Box != other { + t.Fatalf("headroom: got %s, want %s", got.Box, other) + } +} + +// withRole sets one box's roster role before the roster is signed. +func withRole(ref, role string) func(hostactors.Roster) { + return func(r hostactors.Roster) { + b := r.Boxes[ref] + b.Role = role + r.Boxes[ref] = b + } +} + +// Region HA spec 2 §7: the chooser, and so creates and relocation +// targets, applies the roster's roles and the policy's thresholds after +// Eligible. Permits answers the same set, and Capacity counts it. +func TestExecutorChooserAppliesRolesAndHeadroom(t testing.T) { + ctx := context.Background() + absent := hostclaim.Observed{Host: "shop.bab.oncarlos.com", Absent: true} + cases := []struct { + name string + roster func(hostactors.Roster) + edges func(boxchoice.Box) + }{ + {"legacy role", withRole("box-a", hostactors.RoleLegacy), nil}, + {"platform role", withRole("box-a", hostactors.RolePlatform), nil}, + {"memory under 15%", nil, func(b boxchoice.Box) { b.Headroom.MemFreeMiB = 1199 }}, + {"disk under 10%", nil, func(b boxchoice.Box) { b.Headroom.DiskFreeMiB = 9999 }}, + {"edges file 16 minutes old", nil, func(b boxchoice.Box) { b.Generation = "2026-10-07T08:44:00Z" }}, + {"no headroom", nil, func(b *boxchoice.Box) { b.Headroom = nil }}, + } + for , tc := range cases { + t.Run(tc.name, func(t *testing.T) { + x, , snap, := chooserFixtureWith(t, tc.roster, "box-a", "box-b") + for i := range snap.Boxes { + if snap.Boxes[i].BoxID == "box-a" && tc.edges != nil { + tc.edges(&snap.Boxes[i]) + } + } + for range 4 { + got, err := x.Choose(ctx, absent, chooserRecord()) + if err != nil || got.Box != "box-b" { + t.Fatalf("chose %+v %v, want box-b", got, err) + } + } + if err := x.Permits(ctx, absent, chooserRecord(), "box-a"); !errors.Is(err, ErrNoEligibleBox) { + t.Fatalf("Permits(box-a) = %v, want ErrNoEligibleBox", err) + } + if err := x.Permits(ctx, absent, chooserRecord(), "box-b"); err != nil { + t.Fatalf("Permits(box-b) = %v", err) + } + if n, err := x.Capacity(ctx, "ie"); err != nil || n != 1 { + t.Fatalf("Capacity = %d, %v; want 1", n, err) + } + if boxes, err := x.PlaceableBoxes(ctx, "ie"); err != nil || len(boxes) != 1 || boxes[0] != "box-b" { + t.Fatalf("PlaceableBoxes = %v, %v; want [box-b]", boxes, err) + } + }) + } + x, , , := chooserFixture(t, "box-a", "box-b") + if n, err := x.Capacity(ctx, "ie"); err != nil || n != 2 { + t.Fatalf("Capacity with both placeable = %d, %v; want 2", n, err) + } + if , err := x.Capacity(ctx, "us"); !errors.Is(err, ErrContract) { + t.Fatalf("Capacity of a region the roster lacks = %v, want ErrContract", err) } }

````

````diff --- a/internal/hostrelocate/choosertest.go +++ b/internal/hostrelocate/choosertest.go @@ -5,6 +5,7 @@ "errors" "slices" "testing" + "time"

"github.com/carlosframework/platform/internal/boxchoice" "github.com/carlosframework/platform/internal/hostclaim" @@ -13,13 +14,16 @@

// scopedChooser plugs create's chooser into the fixture's worker, scoped // to namespace, with every fixture box fresh and eligible on its edges -// file and spa the most loaded. +// file, equally roomy under spec 2's policy (§7), and spa the most loaded. func scopedChooser(f *fixture, namespace string) RegionChooser { yes := true snap := boxchoice.Snapshot{Complete: true} + generation := f.w.Now().Add(-time.Minute).UTC().Format(time.RFC3339) for _, ref := range []string{"src", "dst", "spa", "sp2"} { snap.Boxes = append(snap.Boxes, boxchoice.Box{BoxID: ref, EIP: "203.0.113.1", Region: "eu-west-1", Placement: "ie", Fresh: true,

• HibernationConfigured: &yes, ServesSharedPool: &yes, AcceptsInbound: &yes, SecretsKeyFP: "fp", Capabilities: []string{boxchoice.Capability}}) + HibernationConfigured: &yes, ServesSharedPool: &yes, AcceptsInbound: &yes, SecretsKeyFP: "fp", Capabilities: []string{boxchoice.Capability}, + Headroom: &boxchoice.Headroom{MemFreeMiB: 4000, MemTotalMiB: 8000, DiskFreeMiB: 50000, DiskTotalMiB: 100000}, + Generation: generation}) } loads := map[string]boxchoice.Load{"spa": {Hosts: 9, Apps: map[string]int{"bab/jam": 3}}} return RegionChooser{Records: f.s, Namespace: namespace, Region: "ie", ````

• [ ] Step 2: Run them to see them fail

Run: nice -n 19 go test ./internal/hostops/ -run 'Executor|Chooser' -count=1 Expected: FAIL to build (x.Capacity undefined, x.PlaceableBoxes undefined).

• [ ] Step 3: Apply the policy

````diff --- a/internal/hostops/executor.go +++ b/internal/hostops/executor.go @@ -48,17 +48,19 @@ // r's placement is created on. Identity, placement and AWS region come from // the roster; a box with a valid termination fence, or not active, is never // chosen; a box on the claim's lost list, or owing a park-local chore on -// it, is never chosen. +// it, is never chosen. Among the rest, region HA spec 2's policy (§7) +// decides: placeable boxes only (placeable), then boxchoice.ChooseRoomiest. func (x ExecutorChooser) Choose(ctx context.Context, obs hostclaim.Observed, r Record) (hostclaim.ExecutorSelection, error) { o, cands, err := x.eligible(ctx, obs, r) if err != nil { return hostclaim.ExecutorSelection{}, err } + cands = x.placeable(o, cands) loads, err := x.Loads(ctx) if err != nil { return hostclaim.ExecutorSelection{}, err }

• best, ok := boxchoice.Choose(r.Holder.Host, cands, loads, r.Holder.Account+"/"+r.Holder.App) + best, ok := boxchoice.ChooseRoomiest(r.Holder.Host, cands, loads, r.Holder.Account+"/"+r.Holder.App) if !ok { return hostclaim.ExecutorSelection{}, ErrNoEligibleBox } @@ -69,18 +71,56 @@ // in r's placement now (spec 1 §2, §5.8 "The start CAS ... refused ... // when to is not eligible for the host"): the same read Choose makes, // without choosing. A stale edges file, a missing capability or secrets -// key, a draining box, a fence, or an incomplete snapshot refuses. +// key, a draining box, a fence, or an incomplete snapshot refuses, and so +// does a box spec 2's policy does not place on (placeable): a legacy or +// platform role, too little memory or disk, or stale headroom. func (x ExecutorChooser) Permits(ctx context.Context, obs hostclaim.Observed, r Record, box string) error {
• _, cands, err := x.eligible(ctx, obs, r) + o, cands, err := x.eligible(ctx, obs, r) if err != nil { return err }
• for , c := range cands { + for , c := range x.placeable(o, cands) { if c.BoxID == box { return nil } } return fmt.Errorf("%w: %s", ErrNoEligibleBox, box) +} + +// PlaceableBoxes is every box in placement that could take a new +// instance in the chooser's namespace and pool now, sorted: Choose's set, +// before the choice. The console's move door takes a move's destination +// from it (region HA spec 2 §7, H27). +func (x ExecutorChooser) PlaceableBoxes(ctx context.Context, placement string) ([]string, error) { + r := Record{Holder: hostclaim.NewHolder{Region: placement}} + o, cands, err := x.eligible(ctx, hostclaim.Observed{Absent: true}, r) + if err != nil { + return nil, err + } + boxes := []string{} + for , c := range x.placeable(o, cands) { + boxes = append(boxes, c.BoxID) + } + return boxes, nil +} + +// Capacity is how many boxes PlaceableBoxes names. The console publishes +// it per region as the eligible-capacity metric (region HA spec 2 §4, +// A1-11). +func (x ExecutorChooser) Capacity(ctx context.Context, placement string) (int, error) { + boxes, err := x.PlaceableBoxes(ctx, placement) + return len(boxes), err +} + +// placeable is region HA spec 2's policy over the eligible set (§7): +// each box's roster role, then boxchoice.Placeable's thresholds at +// x.Now(). Identity was Eligible's; nothing here reads a provider. +func (x ExecutorChooser) placeable(o hostactors.Observed, cands []boxchoice.Candidate) []boxchoice.Candidate { + roles := make(map[string]string, len(cands)) + for , c := range cands { + roles[c.BoxID] = o.Roster.Boxes[c.BoxID].Role + } + return boxchoice.Placeable(cands, roles, x.Now()) }

// eligible is one read of the roster, the fences and the edges files, ````

• [ ] Step 4: Run the tests

Run: nice -n 19 go test ./internal/hostops/ -run 'Executor|Chooser|SelectExecutor|ProductionCreate' -count=1 && nice -n 19 go test ./internal/hostrelocate/ -count=1 Expected: PASS.

• [ ] Step 5: Census

Add this comment line after the overlay's last comment line:

````text

Task 6: hostops.ExecutorChooser applies spec 2's policy in Choose and Permits (creates and relocation targets) and names the placeable set in PlaceableBoxes and Capacity. ExecutorChooser.eligible, the accepted-legacy unit (merge ruling 2), is unchanged: the policy runs over its result. Envelope-only, and the legacy seal stands.

````

Then set these rows. A @coverage row replaces the overlay's row for the same directory, or is appended when there is none; every other row is appended:

```text @coverage internal/hostops 3dc23aaff7bdae0341a8c9a6fbac076a2f57ca1e4676d83543433ff9ad7d1c79 ```

Run the census checks. Expected: PASS.

• [ ] Step 6: Commit

git add internal/hostops/executor.go internal/hostops/executor_test.go internal/hostrelocate/chooser_test.go internal/provision/testdata/plan14-root-reads-region-ha-2a.tsv
git commit -m "region HA 2a: creates and relocations choose by the placement policy"

---

Task 7: The box's own addresses, validated, and private_ip in the edges file

Files:

• Create: internal/edgesfile/private.go, internal/edgesfile/private_test.go (PrivateAddress, the one validator the writer and the reader share)
• Create: internal/host/boxaddress.go (core: BoxAddresses, AddressSource, the host.env source)
• Create: internal/host/awsaddr.go (the AWS adapter: IMDSv2 local-ipv4, public-ipv4, mac and the VPC's vpc-ipv4-cidr-blocks)
• Create: internal/host/boxaddress_test.go
• Modify: internal/host/edgesclaims.go (edgesClaims.addrs)
• Modify: internal/host/edgesfile.go (edgesFile.PrivateIP, PrivateCIDR; edgesFileWriteClaims reads the source with region HA on)
• Modify: internal/host/edgesclaims_test.go, region_ha_boundary_test.go, the overlay

Interfaces:

• Consumes: imdsToken, imdsLeaf (agent.go, unchanged).
• Produces: func edgesfile.PrivateAddress(addr string, cidrs []string) (string, string); type BoxAddresses struct{ Private, PrivateCIDR, Public string }; type AddressSource interface{ BoxAddresses() BoxAddresses }; envAddresses{getenv, provider}; imdsAddresses; func boxAddressSource(getenv func(string) string) AddressSource; edgesFile.PrivateIP, edgesFile.PrivateCIDR (json private_ip, private_cidr, omitempty), which Task 8 reads.

Census ruling region-ha-2a's first condition, in full:

• Source. The address comes from the same root-configured source as the rest of the box's entry: IMDS on AWS, or root-owned host.env otherwise (CARLOS_PRIVATE_IP with CARLOS_PRIVATE_CIDR, which is required beside it; CARLOS_PUBLIC_IP). Never from a file the agent can write.
• Validation before writing (PrivateAddress): a well-formed IPv4 address, in a private range, inside the region's VPC CIDR (from IMDS network/interfaces/macs/<mac>/vpc-ipv4-cidr-blocks on AWS, CARLOS_PRIVATE_CIDR otherwise), and not loopback, link-local or 169.254.0.0/16. Otherwise it is omitted, and forwarding to the box stays public.
• Routing data only. It is dialled, and nothing grants anything on it.

With region HA on (the edges claims cache exists exactly then), every publication reads the address source afresh and publishes only what it reports: a blank read publishes no address, instead of holding the last EIP, which may since have moved to the region's load balancer (A1-12). With region HA off the publisher is main's, EIP hold included. A host.env setting, when present, is the answer for its field, and one that does not validate publishes nothing rather than fall back to IMDS.

Census (platform-1e): edgesFileWriteClaims is a kept authority row (F,J1). Its sites do not move; its digest does. The overlay re-digests it with its class, task and basis unchanged. No seal moves.

• [ ] Step 1: Write the failing tests

internal/edgesfile/private_test.go:

````go package edgesfile

import "testing"

// Census ruling region-ha-2a: a private address passes only when it is a // well-formed private IPv4 address inside the region network's CIDR, and // never loopback or link-local (the metadata range included). func TestPrivateAddress(t testing.T) { vpc := []string{"10.0.0.0/16"} cases := []struct { name, addr string cidrs []string want, cidr string }{ {"inside the region network", "10.0.1.5", vpc, "10.0.1.5", "10.0.0.0/16"}, {"inside the second block", "172.31.4.9", []string{"10.0.0.0/16", " 172.31.0.0/16 "}, "172.31.4.9", "172.31.0.0/16"}, {"malformed", "10.0.1", vpc, "", ""}, {"a hostname", "box.internal", vpc, "", ""}, {"the metadata range", "169.254.169.254", []string{"169.254.0.0/16"}, "", ""}, {"loopback", "127.0.0.1", []string{"127.0.0.0/8"}, "", ""}, {"outside the region network", "10.1.1.5", vpc, "", ""}, {"a public address", "203.0.113.4", []string{"203.0.113.0/24"}, "", ""}, {"IPv6", "fd00::5", []string{"fd00::/8"}, "", ""}, {"no network to check it against", "10.0.1.5", nil, "", ""}, {"a malformed network", "10.0.1.5", []string{"10.0.0.0/33"}, "", ""}, } for _, c := range cases { t.Run(c.name, func(t testing.T) { got, cidr := PrivateAddress(c.addr, c.cidrs) if got != c.want || cidr != c.cidr { t.Fatalf("PrivateAddress(%q, %v) = %q, %q; want %q, %q", c.addr, c.cidrs, got, cidr, c.want, c.cidr) } }) } } ````

internal/host/boxaddress_test.go:

````go package host

import ( "context" "encoding/json" "net/http" "testing"

"github.com/carlosframework/platform/internal/release" )

// staticAddresses is a fixed AddressSource. type staticAddresses BoxAddresses

func (a staticAddresses) BoxAddresses() BoxAddresses { return BoxAddresses(a) }

// countingAddresses counts reads, answering each from its list in turn. type countingAddresses struct { answers []BoxAddresses reads int }

func (c *countingAddresses) BoxAddresses() BoxAddresses { a := c.answers[min(c.reads, len(c.answers)-1)] c.reads++ return a }

func envOf(m map[string]string) func(string) string { return func(k string) string { return m[k] } }

// The core source works with no provider adapter at all (§25). func TestEnvAddressesWithoutAProvider(t *testing.T) { got := envAddresses{getenv: envOf(map[string]string{"CARLOSPRIVATEIP": " 10.0.1.5 ", "CARLOSPRIVATECIDR": "10.0.0.0/16", "CARLOSPUBLICIP": "198.51.100.7"})}.BoxAddresses() if got != (BoxAddresses{Private: "10.0.1.5", PrivateCIDR: "10.0.0.0/16", Public: "198.51.100.7"}) { t.Fatalf("got %+v", got) } if got := (envAddresses{getenv: envOf(nil)}).BoxAddresses(); got != (BoxAddresses{}) { t.Fatalf("nothing set: %+v", got) } // CARLOSPRIVATEIP needs CARLOSPRIVATECIDR beside it. if got := (envAddresses{getenv: envOf(map[string]string{"CARLOSPRIVATEIP": "10.0.1.5"})}).BoxAddresses(); got.Private != "" { t.Fatalf("a private address with no network: %+v", got) } got = envAddresses{getenv: envOf(map[string]string{"CARLOSPRIVATEIP": "169.254.169.254", "CARLOSPRIVATECIDR": "169.254.0.0/16", "CARLOSPUBLICIP": "box.example.com"})}.BoxAddresses() if got != (BoxAddresses{}) { t.Fatalf("invalid settings: %+v", got) } }

func TestEnvAddressesOverrideTheProviderFieldByField(t *testing.T) { provider := &countingAddresses{answers: []BoxAddresses{{Private: "10.0.9.9", PrivateCIDR: "10.0.0.0/16", Public: "203.0.113.9"}}} got := envAddresses{getenv: envOf(map[string]string{"CARLOSPUBLICIP": "198.51.100.7"}), provider: provider}.BoxAddresses() if got != (BoxAddresses{Private: "10.0.9.9", PrivateCIDR: "10.0.0.0/16", Public: "198.51.100.7"}) { t.Fatalf("got %+v", got) } // Both set: the provider is not asked. provider.reads = 0 envAddresses{getenv: envOf(map[string]string{"CARLOSPRIVATEIP": "10.0.1.5", "CARLOSPRIVATECIDR": "10.0.0.0/16", "CARLOSPUBLICIP": "198.51.100.7"}), provider: provider}.BoxAddresses() if provider.reads != 0 { t.Fatalf("provider asked %d times with both settings present", provider.reads) } // A bad private setting publishes no private address; the provider's // answer does not replace it. got = envAddresses{getenv: envOf(map[string]string{"CARLOSPRIVATEIP": "not-an-ip", "CARLOSPRIVATECIDR": "10.0.0.0/16"}), provider: provider}.BoxAddresses() if got != (BoxAddresses{Public: "203.0.113.9"}) { t.Fatalf("bad setting: %+v", got) } // The provider's answer is validated too. bad := &countingAddresses{answers: []BoxAddresses{{Private: "10.9.9.9", PrivateCIDR: "10.0.0.0/16"}}} if got := (envAddresses{getenv: envOf(nil), provider: bad}).BoxAddresses(); got.Private != "" { t.Fatalf("an out-of-network provider address: %+v", got) } }

func TestIMDSAddressesReadsTheLeavesAndTheVPCNetwork(t *testing.T) { leaves := map[string]string{ "/latest/meta-data/local-ipv4": "10.0.1.5", "/latest/meta-data/public-ipv4": "203.0.113.4", "/latest/meta-data/mac": "0e:12:34:56:78:9a", "/latest/meta-data/network/interfaces/macs/0e:12:34:56:78:9a/vpc-ipv4-cidr-blocks": "172.31.0.0/16\n10.0.0.0/16", } s := imdsAddresses{token: func() string { return "tok" }, leaf: func(token, path string) (string, int) { if token != "tok" { t.Fatalf("token %q", token) } v, ok := leaves[path] if !ok { return "", http.StatusNotFound } return v, http.StatusOK }} if got := s.BoxAddresses(); got != (BoxAddresses{Private: "10.0.1.5", PrivateCIDR: "10.0.0.0/16", Public: "203.0.113.4"}) { t.Fatalf("got %+v", got) } // An address outside the VPC's blocks is not published. leaves["/latest/meta-data/local-ipv4"] = "10.9.1.5" if got := s.BoxAddresses(); got.Private != "" || got.PrivateCIDR != "" { t.Fatalf("out of network: %+v", got) } // No VPC network answer: no private address. leaves["/latest/meta-data/local-ipv4"] = "10.0.1.5" delete(leaves, "/latest/meta-data/network/interfaces/macs/0e:12:34:56:78:9a/vpc-ipv4-cidr-blocks") if got := s.BoxAddresses(); got.Private != "" { t.Fatalf("no network: %+v", got) } // No public address (a box without one, or the EIP gone): blank. delete(leaves, "/latest/meta-data/public-ipv4") if got := s.BoxAddresses(); got.Public != "" { t.Fatalf("no public leaf: %+v", got) } noToken := imdsAddresses{token: func() string { return "" }, leaf: s.leaf} if got := noToken.BoxAddresses(); got != (BoxAddresses{}) { t.Fatalf("no token: %+v", got) } }

func readEdgesFile(t *testing.T, s release.Store, box string) edgesFile { t.Helper() body, _, err := s.Get(context.Background(), edgesFileKey(box)) if err != nil { t.Fatal(err) } var f edgesFile if err := json.Unmarshal(body, &f); err != nil { t.Fatal(err) } return f }

// Region HA spec 2 §6 (A1-12): with region HA on, every publication reads // the address source afresh, and a blank read publishes no address. The // last published EIP is never held: it may be the region's, which the // load balancer has taken. func TestEdgesFilePublishesOnlyWhatTheSourceReports(t *testing.T) { s, reg := release.NewDirStore(t.TempDir()), edgesTestRegistry(t) seedRoute(t, reg, "app.example.com") claims := newEdgesClaims("annie", (&claimsBook{}).read) src := &countingAddresses{answers: []BoxAddresses{ {Private: "10.0.1.5", PrivateCIDR: "10.0.0.0/16", Public: "203.0.113.9"}, {Private: "10.0.1.5", PrivateCIDR: "10.0.0.0/16"}, }} claims.addrs = src var last []byte ident := edgesFile{Box: "annie", BoxID: "annie"} // The loop's own reading (the eip argument) is not what is published. if err := edgesFileWriteClaims(context.Background(), s, reg, ident, "198.51.100.1", &last, false, claims); err != nil { t.Fatal(err) } if f := readEdgesFile(t, s, "annie"); f.EIP != "203.0.113.9" || f.PrivateIP != "10.0.1.5" || f.PrivateCIDR != "10.0.0.0/16" { t.Fatalf("first publication: %+v", f) } if err := edgesFileWriteClaims(context.Background(), s, reg, ident, "", &last, false, claims); err != nil { t.Fatal(err) } if f := readEdgesFile(t, s, "annie"); f.EIP != "" || f.PrivateIP != "10.0.1.5" { t.Fatalf("after a blank public read: eip %q private %q, want no public address", f.EIP, f.PrivateIP) } if src.reads != 2 { t.Fatalf("source read %d times, want once per publication", src.reads) } }

// With region HA off there is no claims cache, and the publisher is // main's: no private address, and a blank probe holds the last EIP. func TestEdgesFileWithRegionHAOffKeepsTheEIPHold(t *testing.T) { s, reg := release.NewDirStore(t.TempDir()), edgesTestRegistry(t) seedRoute(t, reg, "app.example.com") var last []byte ident := edgesFile{Box: "annie"} if err := edgesFileWriteClaims(context.Background(), s, reg, ident, "203.0.113.9", &last, false, nil); err != nil { t.Fatal(err) } if err := edgesFileWriteClaims(context.Background(), s, reg, ident, "", &last, true, nil); err != nil { t.Fatal(err) } if f := readEdgesFile(t, s, "annie"); f.EIP != "203.0.113.9" || f.PrivateIP != "" || f.PrivateCIDR != "" { t.Fatalf("%+v", f) } } ````

The existing claims tests get a fixed address source, never IMDS:

````diff --- a/internal/host/edgesclaimstest.go +++ b/internal/host/edgesclaimstest.go @@ -50,6 +50,11 @@

func publishWithClaims(t testing.T, reg registry.Registry, claims *edgesClaims) map[string]edgeHost { t.Helper() + // These tests are about the hosts; the box's addresses come from a + // fixed source, never from IMDS. nil claims is region HA off. + if claims != nil { + claims.addrs = staticAddresses{Public: "1.2.3.4"} + } st := release.NewDirStore(t.TempDir()) var last []byte if err := edgesFileWriteClaims(context.Background(), st, reg, edgesFile{Box: "annie", BoxID: "annie"}, "1.2.3.4", &last, false, claims); err != nil { ````

• [ ] Step 2: Run them to see them fail

Run: nice -n 19 go test ./internal/edgesfile/ -run PrivateAddress -count=1; nice -n 19 go test ./internal/host/ -run 'Address|IMDS|EdgesFile' -count=1 Expected: FAIL to build (undefined: PrivateAddress, undefined: BoxAddresses, claims.addrs undefined).

• [ ] Step 3: Write the validator, the source and the publisher's change

internal/edgesfile/private.go:

````go package edgesfile

import ( "net" "strings" )

// PrivateAddress validates a box's private address against its region // network's CIDR blocks (region HA spec 2 §6, census ruling region-ha-2a): // a well-formed IPv4 address, in a private range, neither loopback nor // link-local (169.254.0.0/16, the metadata range, included), and inside // one of cidrs. It returns the address and the block that holds it, both // canonical, or "", "". The writer publishes only what it passes, and a // reader dials only what it passes against its own region's network. func PrivateAddress(addr string, cidrs []string) (string, string) { ip := net.ParseIP(strings.TrimSpace(addr)) if ip == nil || ip.To4() == nil { return "", "" } ip = ip.To4() if !ip.IsPrivate() || ip.IsLoopback() || ip.IsLinkLocalUnicast() || ip.IsUnspecified() { return "", "" } for , c := range cidrs { , block, err := net.ParseCIDR(strings.TrimSpace(c)) if err != nil || block.IP.To4() == nil { continue } if block.Contains(ip) { return ip.String(), block.String() } } return "", "" } ````

internal/host/boxaddress.go:

````go package host

import ( "net" "strings"

"github.com/carlosframework/platform/internal/edgesfile" )

// BoxAddresses are the addresses this box publishes in its edges file for // other edges to dial (region HA spec 2 §6). Public is for an edge in // another region. Private, with PrivateCIDR (the region network it lies // in), is for an edge in the same region. They are routing data only: an // edge dials them, and nothing grants anything on them. "" is an address // the source could not report, or one that failed validation, and nothing // is published for it. type BoxAddresses struct { Private string PrivateCIDR string Public string }

// AddressSource reads this box's addresses afresh on every call. The edges // file publisher asks it before every publication and publishes only what // it reports, so an address the box no longer holds is never republished. // Every source is root-configured: the cloud's metadata service, or // host.env. None reads a file the agent can write. type AddressSource interface { BoxAddresses() BoxAddresses }

// envAddresses is the provider-neutral source (§25), from host.env: // CARLOSPRIVATEIP with CARLOSPRIVATECIDR, and CARLOSPUBLICIP. A field // whose setting is present is the answer for that field, and a value that // does not validate publishes nothing: an operator's setting is never // silently replaced. CARLOSPRIVATEIP without CARLOSPRIVATECIDR is // invalid (census ruling region-ha-2a). A field left unset comes from // provider, the cloud's adapter, when there is one. type envAddresses struct { getenv func(string) string provider AddressSource }

func (s envAddresses) BoxAddresses() BoxAddresses { ip := strings.TrimSpace(s.getenv("CARLOSPRIVATEIP")) cidr := strings.TrimSpace(s.getenv("CARLOSPRIVATECIDR")) public := strings.TrimSpace(s.getenv("CARLOSPUBLICIP")) privateSet, publicSet := ip != "" || cidr != "", public != "" out := BoxAddresses{} if privateSet { out.Private, out.PrivateCIDR = edgesfile.PrivateAddress(ip, []string{cidr}) } if publicSet { out.Public = ipv4Only(public) } if (!privateSet || !publicSet) && s.provider != nil { p := s.provider.BoxAddresses() if !privateSet { out.Private, out.PrivateCIDR = edgesfile.PrivateAddress(p.Private, []string{p.PrivateCIDR}) } if !publicSet { out.Public = ipv4Only(p.Public) } } return out }

// ipv4Only is s when it is an IPv4 address, in its canonical form, and "" // otherwise. func ipv4Only(s string) string { ip := net.ParseIP(strings.TrimSpace(s)) if ip == nil || ip.To4() == nil { return "" } return ip.To4().String() } ````

internal/host/awsaddr.go:

````go package host

import ( "net/http" "strings" "time"

"github.com/carlosframework/platform/internal/edgesfile" )

// imdsAddresses is the AWS adapter of AddressSource (region HA spec 2 // §25): IMDSv2's local-ipv4, public-ipv4, and the VPC's CIDR blocks for // the primary interface (network/interfaces/macs/<mac>/vpc-ipv4-cidr-blocks), // read on every call. IMDS reports only addresses this instance holds, so // once the region's EIP has moved to the load balancer no box reports it // (A1-12). A leaf that does not answer 200 is blank. type imdsAddresses struct { token func() string leaf func(token, path string) (string, int) }

func newIMDSAddresses() imdsAddresses { client := &http.Client{Timeout: 2 * time.Second} return imdsAddresses{ token: func() string { return imdsToken(client) }, leaf: func(token, path string) (string, int) { return imdsLeaf(client, token, path) }, } }

func (s imdsAddresses) BoxAddresses() BoxAddresses { token := s.token() if token == "" { return BoxAddresses{} } read := func(path string) string { v, code := s.leaf(token, path) if code != http.StatusOK { return "" } return v } out := BoxAddresses{Public: read("/latest/meta-data/public-ipv4")} ip := read("/latest/meta-data/local-ipv4") mac := read("/latest/meta-data/mac") if ip == "" || mac == "" || strings.ContainsAny(mac, "/\\ ") { return out } blocks := strings.Fields(read("/latest/meta-data/network/interfaces/macs/" + mac + "/vpc-ipv4-cidr-blocks")) out.Private, out.PrivateCIDR = edgesfile.PrivateAddress(ip, blocks) return out }

// boxAddressSource is what the edges file publisher reads with region HA // on: host.env's settings, then IMDS on EC2. func boxAddressSource(getenv func(string) string) AddressSource { return envAddresses{getenv: getenv, provider: newIMDSAddresses()} } ````

````diff --- a/internal/host/edgesclaims.go +++ b/internal/host/edgesclaims.go @@ -4,6 +4,7 @@ "context" "log" "math" + "os" "sort" "time"

@@ -71,11 +72,15 @@ passDeadline time.Duration logf func(format string, a ...any) failingRun int // consecutive passes with a failed claim read + // addrs is the box's own addresses, read before every publication + // (region HA spec 2 §6). It rides here because this cache exists + // exactly when region HA is on. + addrs AddressSource }

func newEdgesClaims(self string, read func(context.Context, string) (hostclaim.Observed, error)) *edgesClaims { return &edgesClaims{self: self, read: read, now: time.Now, entries: map[string]edgesClaimEntry{},

• passDeadline: edgesClaimPassDeadline, logf: log.Printf} + passDeadline: edgesClaimPassDeadline, logf: log.Printf, addrs: boxAddressSource(os.Getenv)} }

// relocatableRow is spec 1 §2's relocatable host as a registry row shows ````

```diff --- a/internal/host/edgesfile.go +++ b/internal/host/edgesfile.go @@ -205,6 +205,13 @@ Box string json:"box" EIP string json:"eip" Region string json:"region,omitempty" + // PrivateIP is the address an edge in this box's region dials (region + // HA spec 2 §6), and PrivateCIDR the region network that holds it, + // both validated by privateAddress before they are written. Routing + // data only. Additive and omitempty: a reader that does not know them, + // or finds them absent, dials EIP, as before. + PrivateIP string json:"privateip,omitempty" + PrivateCIDR string json:"privatecidr,omitempty"` // Placement is this box's CARLOS placement code — CARLOS_REGION verbatim // ("ie", "uk"), the OTHER region vocabulary (see Region above, issue // #329). Additive and omitempty like Region: a pre-placement body parses @@ -591,7 +598,15 @@ hosts[key] = edgeHost{Role: role, Wildcard: true, Alias: alias, Account: acct, App: app, Stamp: stamp} }

• if eip == "" && last != nil { + var private BoxAddresses + if claims != nil && claims.addrs != nil { + // Region HA spec 2 §6 (A1-12): publish only what the box's address + // source reports now. A blank read publishes no address, never the + // last one: that may be the region's EIP, which the load balancer + // has since taken. + private = claims.addrs.BoxAddresses() + eip = private.Public + } else if eip == "" && last != nil { var prior edgesFile if err := json.Unmarshal(*last, &prior); err == nil { eip = prior.EIP @@ -600,6 +615,7 @@

f := ident f.V, f.EIP, f.Hosts, f.Generation = 1, eip, hosts, "" + f.PrivateIP, f.PrivateCIDR = private.Private, private.PrivateCIDR gateCopy := f gateCopy.Headroom = nil // a changing reading is not, by itself, a reason to write gate, err := json.Marshal(gateCopy) ````

• [ ] Step 4: Run the tests

Run: nice -n 19 go test ./internal/edgesfile/ -count=1 && nice -n 19 go test ./internal/host/ -run 'Address|IMDS|EdgesFile|Edges|Headroom' -count=1 Expected: PASS: a malformed, a metadata-range, a loopback, a public, an IPv6 and an out-of-network address are each omitted.

• [ ] Step 5: Boundary list and census

````diff --- a/regionhaboundarytest.go +++ b/regionhaboundarytest.go @@ -22,11 +22,15 @@ "internal/lbhealth/lbhealth.go", "internal/hostactors/roles.go", "internal/boxchoice/policy.go", + "internal/host/boxaddress.go", + "internal/edgesfile/private.go", }

// regionHAAdapterFiles are spec 2's AWS adapters. Each is named for AWS so // a reader can see where the provider lives. -var regionHAAdapterFiles = []string{} +var regionHAAdapterFiles = []string{ + "internal/host/awsaddr.go", +}

// regionHAPackages are the packages spec 2 touches. Every S3 client built // in them goes through s3compat.Options(), so CARLOSS3ENDPOINT applies. ````

Add this comment line after the overlay's last comment line:

````text

Task 7: the box's own addresses (internal/host boxaddress.go, the provider-neutral source over CARLOSPRIVATEIP with CARLOSPRIVATECIDR, and CARLOSPUBLICIP, all from root-owned host.env; awsaddr.go, the IMDS adapter, which reads the VPC's CIDR blocks too) and the edges file's privateip and privatecidr, validated before they are written by edgesfile.PrivateAddress (census ruling region-ha-2a, condition 1). edgesFileWriteClaims (authority, F,J1) is re-digested with its class, task, basis and sites unchanged: with region HA on it reads the address source before every publication and no longer holds a blank EIP. It adds no registry, /data, release or claim read, and nothing runs as root that did not before; the legacy seal stands.

````

Then set these rows. A @coverage row replaces the overlay's row for the same directory, or is appended when there is none; every other row is appended:

```text @coverage internal/edgesfile f62c3973bb48fa36f561d26f496c0ba9a14fe882fe5843867e9f7a9eaf0abbdf @coverage internal/host 02e98726846daefb57c11e2731344f4f7ae7eb4d4a6ec32388290bf1369091a0 internal/host/edgesfile.go edgesFileWriteClaims authority F,J1 root-route;kept=F;scope=protected-decision-only registry:Account@10835,registry:Alias@5297,registry:Alias@5332,registry:Alias@5754,registry:Alias@5911,registry:Alias@5981,registry:Alias@6013,registry:Alias@6482,registry:Alias@8772,registry:App@10847,registry:DNSStatus@5429,registry:DNSStatus@5920,registry:Host@10480,registry:Host@2878,registry:Host@5389,registry:Host@6234,registry:Host@6567,registry:Host@8819,registry:Host@8866,registry:Host@8916,registry:Host@8951,registry:Host@9000,registry:Platform@2921,registry:Provisioned@2935,registry:Replica@2959,registry:Status@877,whole:Route@3212,whole:Route@323,whole:Route@3505,whole:Route@351,whole:RouteAlias@5282,whole:RouteAlias@8725,whole:RoutePath@3520,whole:WildcardDomain@8905 48bad9a5f49c5fca53fcf7bd6a241576fd2cd332974114e463b2cb3832e3598d ```

The appended row is edgesFileWriteClaims's, re-digested: its kinds are main's row's, unchanged (check with the scratch test). Run the census checks and go test . -run 'TestRegionHA(Core|Adapters|S3)' -count=1. Expected: PASS.

• [ ] Step 6: Commit

git add internal/edgesfile/private.go internal/edgesfile/private_test.go internal/host/boxaddress.go internal/host/awsaddr.go internal/host/boxaddress_test.go internal/host/edgesclaims.go internal/host/edgesclaims_test.go internal/host/edgesfile.go region_ha_boundary_test.go internal/provision/testdata/plan14-root-reads-region-ha-2a.tsv
git commit -m "region HA 2a: publish the box's validated private address and only what its source reports"

---

Task 8: Forwarding dials the private address within a region, with the reader's region HA on

Files:

• Modify: internal/edge/homemap.go (edgesFileView reads private_ip, private_cidr, region, placement; dialAddress; regionHAFromEnv; homeMap.privateDial; foldHomes)
• Modify: internal/edge/homemap_test.go (foldHomes's new argument)
• Create: internal/edge/homemap_private_test.go
• Modify: the overlay

Interfaces:

• Consumes: edgesfile.PrivateAddress and private_ip/private_cidr from Task 7.
• Produces: func dialAddress(f, self edgesFileView, selfKnown, privateDial bool) string; func regionHAFromEnv(getenv func(string) string) bool (Task 9 reuses it); foldHomes(files, self, privateDial). homeEntry.eip is now the address to dial, so Home, ExactHome and Owner answer it, and the backhaul transport, keyed by that address, dials it.

The hop is unchanged: one hop, today's credential and mesh key, the TLS name the host's. Only the address dialled changes, and only with this edge's own CARLOS_REGION_HA on (Astra finding 5): a box in this box's own AWS region and placement, publishing a private_ip that passes PrivateAddress against the network this box's own file publishes, is dialled there. Every other box, and every box with this edge's switch off, is dialled at its EIP, as before; so is every box when this edge cannot find its own file.

• [ ] Step 1: Write the failing tests

internal/edge/homemap_private_test.go:

````go package edge

import "testing"

// privateFleet publishes this edge's own file (annie, in ie, network // 10.0.0.0/16) and one file per case. func privateFleet(t testing.T, m homeMap) { t.Helper() putJSON(t, m.s, "control/edges/annie.json", {"v":1,"box":"annie","eip":"1.1.1.1","private_ip":"10.0.1.1","private_cidr":"10.0.0.0/16","region":"eu-west-1","placement":"ie","hosts":{}}) putJSON(t, m.s, "control/edges/betty.json", {"v":1,"box":"betty","eip":"2.2.2.2","private_ip":"10.0.1.2","private_cidr":"10.0.0.0/16","region":"eu-west-1","placement":"ie","hosts":{"b.test":{"role":"home"}}}) putJSON(t, m.s, "control/edges/carla.json", {"v":1,"box":"carla","eip":"3.3.3.3","private_ip":"10.1.1.3","private_cidr":"10.1.0.0/16","region":"eu-west-2","placement":"uk","hosts":{"c.test":{"role":"home"}}}) putJSON(t, m.s, "control/edges/dora.json", {"v":1,"box":"dora","eip":"4.4.4.4","region":"eu-west-1","placement":"ie","hosts":{"d.test":{"role":"home"}}}) putJSON(t, m.s, "control/edges/edna.json", {"v":1,"box":"edna","eip":"5.5.5.5","private_ip":"10.0.1.5","region":"eu-west-1","placement":"fleet/shop","hosts":{"e.test":{"role":"home"}}}) putJSON(t, m.s, "control/edges/fern.json", {"v":1,"box":"fern","eip":"","private_ip":"10.0.1.6","region":"eu-west-1","placement":"ie","hosts":{"f.test":{"role":"home"}}}) // Addresses a reader refuses: each falls back to the public address. putJSON(t, m.s, "control/edges/gail.json", {"v":1,"box":"gail","eip":"7.7.7.7","private_ip":"10.0.1","region":"eu-west-1","placement":"ie","hosts":{"g.test":{"role":"home"}}}) putJSON(t, m.s, "control/edges/hana.json", {"v":1,"box":"hana","eip":"8.8.8.8","private_ip":"169.254.169.254","private_cidr":"169.254.0.0/16","region":"eu-west-1","placement":"ie","hosts":{"h.test":{"role":"home"}}}) putJSON(t, m.s, "control/edges/iris.json", {"v":1,"box":"iris","eip":"9.9.9.9","private_ip":"10.9.1.9","private_cidr":"10.9.0.0/16","region":"eu-west-1","placement":"ie","hosts":{"i.test":{"role":"home"}}}) refreshHomeMap(t, m) }

// Region HA spec 2 §6: with this edge's region HA on, a box in the same // region is dialled at its private address and a box in another region at // its own public address. An address that fails validation against this // edge's own network is refused, and the dial is public. func TestHomeMapDialsPrivateWithinTheRegionAndPublicBetween(t *testing.T) { m := newHomeMap(meshTestStore(t), "annie") m.privateDial = true privateFleet(t, m) for host, want := range map[string]string{ "b.test": "10.0.1.2", // same region and placement: private "c.test": "3.3.3.3", // another region: its own public address "d.test": "4.4.4.4", // a writer with no privateip: public, as before "e.test": "5.5.5.5", // same AWS region, another placement: public "f.test": "10.0.1.6", // no public address, but private within the region "g.test": "7.7.7.7", // a malformed private address: refused "h.test": "8.8.8.8", // the metadata range: refused "i.test": "9.9.9.9", // outside this region's network: refused } { if addr, , ok := m.Home(host); !ok || addr != want { t.Errorf("%s dials %q (ok=%v), want %q", host, addr, ok, want) } } // An edge whose own file is not in the map knows no region or network // of its own: every dial is public, and a box with no public address // is no target. ghost := newHomeMap(m.s, "ghost") ghost.privateDial = true refreshHomeMap(t, ghost) if addr, , := ghost.Home("b.test"); addr != "2.2.2.2" { t.Errorf("without its own file: b.test dials %q, want 2.2.2.2", addr) } if , , ok := ghost.Home("f.test"); ok { t.Error("without its own file, a box with no public address became a target") } }

// With this edge's region HA off, a peer publishing privateip changes // nothing: every dial is public, as on main, during a mixed roll or a // rollback. func TestHomeMapWithRegionHAOffDialsPublicOnly(t *testing.T) { m := newHomeMap(meshTestStore(t), "annie") privateFleet(t, m) for host, want := range map[string]string{"b.test": "2.2.2.2", "c.test": "3.3.3.3", "d.test": "4.4.4.4"} { if addr, , ok := m.Home(host); !ok || addr != want { t.Errorf("%s dials %q (ok=%v), want %q", host, addr, ok, want) } } if , , ok := m.Home("f.test"); ok { t.Error("a box with only a private address became a target with region HA off") } }

func TestEdgeHomeMapFollowsRegionHA(t *testing.T) { e := &Edge{staticStore: meshTestStore(t), box: "annie"} t.Setenv("CARLOSREGIONHA", "") if e.newHomeMap().privateDial { t.Fatal("region HA unset dials private") } t.Setenv("CARLOSREGIONHA", "1") if !e.newHomeMap().privateDial { t.Fatal("region HA on does not dial private") } } ````

• [ ] Step 2: Run them to see them fail

Run: nice -n 19 go test ./internal/edge/ -run 'DialsPrivate|PublicOnly|FollowsRegionHA' -count=1 Expected: FAIL to build (m.privateDial undefined).

• [ ] Step 3: Choose the dial address

````diff --- a/internal/edge/homemap.go +++ b/internal/edge/homemap.go @@ -80,10 +80,12 @@ "os" "slices" "sort" + "strconv" "strings" "sync" "time"

+ "github.com/carlosframework/platform/internal/edgesfile" "github.com/carlosframework/platform/internal/release" )

@@ -132,6 +134,39 @@ Box string json:"box" EIP string json:"eip" Hosts map[string]edgesHostView json:"hosts" + // PrivateIP, PrivateCIDR, Region and Placement choose the dial address + // (region HA spec 2 §6, dialAddress). All are absent from a writer that + // predates them, and the dial is then EIP, as before. + PrivateIP string json:"private_ip" + PrivateCIDR string json:"private_cidr" + Region string json:"region" + Placement string json:"placement" +} + +// dialAddress is where this edge dials f's box (region HA spec 2 §6). With +// privateDial (this edge's own region HA switch) on, a box that publishes +// the same AWS region and placement as this one is dialled at its private +// address, so the dial stays inside the region's network, but only when +// that address passes edgesfile.PrivateAddress against the network this +// box's own file publishes (census ruling region-ha-2a). Every other box, +// and every box with privateDial off, is dialled at its public address, +// as before. self is this box's own file, which selfKnown says was found. +// "" is nothing to dial. +func dialAddress(f, self edgesFileView, selfKnown, privateDial bool) string { + if privateDial && selfKnown && f.Region != "" && f.Placement != "" && + f.Region == self.Region && f.Placement == self.Placement { + if addr, := edgesfile.PrivateAddress(f.PrivateIP, []string{self.PrivateCIDR}); addr != "" { + return addr + } + } + return f.EIP +} + +// regionHAFromEnv is CARLOSREGIONHA as strconv.ParseBool reads it, the +// reading every region HA switch makes. +func regionHAFromEnv(getenv func(string) string) bool { + on, err := strconv.ParseBool(strings.TrimSpace(getenv("CARLOSREGION_HA"))) + return err == nil && on }

// edgesHostView is the per-host object. An object rather than a bare string is @@ -161,6 +196,8 @@

// homeEntry is one folded answer: where to dial, and which box that is. type homeEntry struct { + // eip is the address to dial: dialAddress's answer, the home box's + // private address within its region and its EIP from anywhere else. eip string box string // platform carries the publisher's class mark through the fold. @@ -238,6 +275,10 @@ keyMu sync.RWMutex keyCurrent []byte keyNext []byte + + // privateDial is this edge's region HA switch: only with it on does + // the fold dial a same-region box's private address (dialAddress). + privateDial bool }

func newHomeMap(s release.Store, self string) *homeMap { @@ -253,7 +294,9 @@ if e.staticStore == nil { return nil }

• return newHomeMap(e.staticStore, e.box) + m := newHomeMap(e.staticStore, e.box) + m.privateDial = regionHAFromEnv(os.Getenv) + return m }

// Refresh rebuilds the host fold and re-reads the mesh keys. The two halves @@ -330,7 +373,7 @@ if failed > 0 { m.logPartialPoll(failed, tried, lastErr) }

• hosts, tied := foldHomes(next, m.self) + hosts, tied := foldHomes(next, m.self, m.privateDial) if len(tied) > 0 { // One claim revision names one executor, so two boxes publishing // one host at the same stamp is a fault upstream. The fold stays @@ -532,9 +575,11 @@ // target would have the edge dial itself. Excluded by the file's own box // field rather than by its object name, because the field is what the // publisher stamps and what self is derived to match. -// - NO EIP. Task 1 publishes a box with no discoverable public IP as -// eip:"" so the DR runbook still gets its hosts list. There is nothing to -// dial, so it is not a target. +// - NO DIAL ADDRESS. Task 1 publishes a box with no discoverable public IP +// as eip:"" so the DR runbook still gets its hosts list. There is +// nothing to dial, so it is not a target. The address is dialAddress's: +// a box in this box's own region is dialled at its private address when +// it publishes one (region HA spec 2 §6), every other at its EIP. // - A ROLE THIS BUILD DOES NOT KNOW. Only "home" routes. The role field // exists so read-local roles can be added later; a reader that treated // every role as a home would send traffic to a box that had declared it @@ -546,28 +591,36 @@ // split-and-wrong: the whole fleet aiming at one box is a condition that heals // on the next publish, where half the fleet aiming at each is a condition that // looks fine from any single box. -func foldHomes(files map[string]edgesFileView, self string) (map[string]homeEntry, []string) { +func foldHomes(files map[string]edgesFileView, self string, privateDial bool) (map[string]homeEntry, []string) { keys := make([]string, 0, len(files)) for k := range files { keys = append(keys, k) } slices.Sort(keys) + var own edgesFileView + ownKnown := false + for , k := range keys { + if f := files[k]; self != "" && f.Box == self { + own, ownKnown = f, true + } + } hosts := make(map[string]homeEntry) for , k := range keys { f := files[k] if f.Box == "" || f.Box == self { continue } + addr := dialAddress(f, own, ownKnown, privateDial) for host, h := range f.Hosts { if host == "" { continue } switch h.Role { case edgeRoleHome:
• // A dial target needs an address. A file with no EIP (a box + // A dial target needs an address. A file with none (a box // with no discoverable public IP) still carries its cert // halves below; its dial halves are simply not dialable.
• if f.EIP == "" { + if addr == "" { if !h.Wildcard || !strings.HasPrefix(host, "*.") { continue } @@ -583,7 +636,7 @@ default: continue }
• in := homeEntry{eip: f.EIP, box: f.Box, platform: h.Platform && !h.Alias && !h.Wildcard, alias: h.Alias, wildcard: h.Wildcard, dial: h.Role == edgeRoleHome, + in := homeEntry{eip: addr, box: f.Box, platform: h.Platform && !h.Alias && !h.Wildcard, alias: h.Alias, wildcard: h.Wildcard, dial: h.Role == edgeRoleHome, account: h.Account, app: h.App, stamp: h.Stamp} prev, dup := hosts[host] if !dup { ````

````diff --- a/internal/edge/homemaptest.go +++ b/internal/edge/homemaptest.go @@ -1715,7 +1715,7 @@ "control/edges/betty.json": file("betty", "2.2.2.2", map[string]edgesHostView{ "tied.test": {Role: "home", Stamp: 7}, "fine.test": {Role: "home", Stamp: 9}, "plain.test": {Role: "home"}}), }

• hosts, tied := foldHomes(files, "carol") + hosts, tied := foldHomes(files, "carol", false) if len(tied) != 1 || tied[0] != "tied.test" { t.Fatalf("tied = %v, want [tied.test]: unstamped duplicates are normal (static sites run on every box) and are not reported", tied) } @@ -1735,7 +1735,7 @@ "control/edges/betty.json": file("betty", "2.2.2.2", map[string]edgesHostView{"h.test": {Role: "home", Stamp: 7}}), "control/edges/carol.json": file("carol", "3.3.3.3", map[string]edgesHostView{"h.test": {Role: "home", Stamp: 7}}), }
• hosts, tied := foldHomes(files, "dave") + hosts, tied := foldHomes(files, "dave", false) if len(tied) != 1 || tied[0] != "h.test" { t.Fatalf("tied = %v, want [h.test]", tied) } @@ -1751,7 +1751,7 @@ "control/edges/annie.json": file("annie", "1.1.1.1", map[string]edgesHostView{"h.test": {Role: "home", Stamp: 7}}), "control/edges/betty.json": file("betty", "2.2.2.2", map[string]edgesHostView{"h.test": {Role: "home", Stamp: 7}}), }
• hosts, tied := foldHomes(files, "annie") + hosts, tied := foldHomes(files, "annie", false) if len(tied) != 1 || tied[0] != "h.test" { t.Fatalf("tied = %v, want [h.test]", tied) } ````

• [ ] Step 4: Run the tests

Run: nice -n 19 go test ./internal/edge/ -run 'HomeMap|Backhaul|Hop|KnownHost|DialsPrivate|PublicOnly|FollowsRegionHA' -count=1 Expected: PASS.

• [ ] Step 5: Census

Add this comment line after the overlay's last comment line:

````text

Task 8: with the reader's own region HA on, forwarding dials a same-region box's private address when it passes edgesfile.PrivateAddress against the reader's own published network, and every other box at its public address (internal/edge homemap.go: dialAddress, regionHAFromEnv, edgesFileView's privateip, privatecidr, region and placement). The home map is not a root-read unit. Envelope-only, and the legacy seal stands.

````

Then set these rows. A @coverage row replaces the overlay's row for the same directory, or is appended when there is none; every other row is appended:

```text @coverage internal/edge 2c474d685471295daf20f061e595becab78175c5754dbef0d2187f0f8519fbe3 ```

Run the census checks. Expected: PASS.

• [ ] Step 6: Commit

git add internal/edge/homemap.go internal/edge/homemap_test.go internal/edge/homemap_private_test.go internal/provision/testdata/plan14-root-reads-region-ha-2a.tsv
git commit -m "region HA 2a: dial a box's private address within its region"

---

Task 9: The load balancer health answer

Files:

• Create: internal/edge/lbhealth.go, internal/edge/lbhealth_test.go
• Modify: internal/edge/edge.go (Edge.lbHealth, set in New; Watch beats; httpFallbackHandler answers first)
• Modify: internal/edge/homemap.go (homeMap.health; Refresh latches and stamps)
• Modify: internal/edge/certsync.go (certMirror.health; a landed Refresh latches)
• Modify: internal/activator/activator.go (lastSweep, LastSweep)
• Create: internal/activator/lastsweep_test.go
• Modify: internal/host/claimwiring.go (startRegional marks the box hosting)
• Create: internal/host/claimwiring_hosting_test.go
• Modify: region_ha_boundary_test.go, the overlay

Interfaces:

• Consumes: Task 2's lbhealth.State and lbhealth.Serve; Task 8's regionHAFromEnv.
• Produces: func lbHealthFromEnv(getenv func(string) string, act Activator) *lbhealth.State (nil with region HA off); func (e *Edge) lbHealthAnswer(w http.ResponseWriter, r *http.Request) bool; func (a *activator.Service) LastSweep() time.Time; the hosting hook lbhealth.Default.SetHosting(true) in ClaimWiring.startRegional. Seam for 2b: 2b's mode selection takes SetHosting over; joining mode never sets it.

The answer is served on port 80 only (the :80 handler, httpFallbackHandler), ahead of every host-based branch, and reads only in-memory state. Each condition is fed by the loop it describes:

| Condition | Fed by | |---|---| | hosting | ClaimWiring.startRegional: the claim agent only starts with its host-actor enrollment and a matching roster row | | forwarding | homeMap.Refresh: the first successful map load and the first successful mesh key load, latched | | certificates | certMirror.Refresh: the first refresh that lands control/certs/ in the local cache the handshake reads, latched (spec rev 15: certificates are per host, so there is no single default certificate) | | edge | Edge.Watch: every completed registry poll (every 2 seconds) | | activator | activator.Service.LastSweep: every finished sweep (at most 30 seconds apart) | | refresh | homeMap.Refresh: every completed attempt, failed or not |

RunWithActivatorReady, Edge.ServeHTTP and runAgent are accepted-legacy or kept authority units; none changes. The edge's state is lbhealth.Default (process-wide) because the agent runs the edge and the activator in one process; tests give each edge its own State.

• [ ] Step 1: Write the failing tests

internal/edge/lbhealth_test.go:

````go package edge

import ( "context" "errors" "net/http" "net/http/httptest" "strings" "testing" "time"

"golang.org/x/crypto/acme/autocert"

"github.com/carlosframework/platform/internal/lbhealth" "github.com/carlosframework/platform/internal/release" )

// sweeper is an Activator whose only working method is the sweep beat. type sweeper struct { Activator at time.Time }

func (s sweeper) LastSweep() time.Time { return s.at }

func lbEnv(v string) func(string) string { return func(k string) string { if k == "CARLOSREGIONHA" { return v } return "" } }

func TestLBHealthStateFollowsRegionHA(t *testing.T) { if s := lbHealthFromEnv(lbEnv(""), nil); s != nil { t.Fatal("region HA unset armed the health answer") } if s := lbHealthFromEnv(lbEnv("0"), nil); s != nil { t.Fatal("region HA off armed the health answer") } at := time.Now() s := lbHealthFromEnv(lbEnv("1"), nil) if s != lbhealth.Default { t.Fatal("region HA on: not the process's state") } t.Cleanup(func() { lbhealth.Default.SetActivator(nil) }) lbHealthFromEnv(lbEnv("true"), sweeper{at: at}) if got := lbhealth.Default.Failing(at); strings.Contains(strings.Join(got, ","), lbhealth.CondActivator) { t.Fatalf("the activator's sweep was not wired: %v", got) } }

// fallbackGet drives the :80 handler with one request. func fallbackGet(e Edge, method, path, host string) httptest.ResponseRecorder { w := httptest.NewRecorder() r := httptest.NewRequest(method, path, nil) r.Host = host e.httpFallbackHandler().ServeHTTP(w, r) return w }

// healthyState meets every condition now. func healthyState() *lbhealth.State { s := &lbhealth.State{} now := time.Now() s.SetHosting(true) s.MapLoaded() s.KeyLoaded() s.CertsLoaded() s.EdgeBeat(now) s.RefreshAttempted(now) s.SetActivator(func() time.Time { return now }) return s }

// The answer is on :80, for any host (the load balancer sends the // target's address), GET or HEAD, ahead of the redirect and unknown-host // branches. With region HA off the path is ordinary :80 traffic. func TestLBHealthAnswersOnPort80ForAnyHost(t *testing.T) { e, err := New(openReg(t), nil) if err != nil { t.Fatal(err) } t.Cleanup(e.probers.Stop) e.lbHealth = healthyState() for _, host := range []string{"10.0.1.7", "10.0.1.7:80", "unknown.example.com"} { w := fallbackGet(e, http.MethodGet, lbhealth.Path, host) if w.Code != http.StatusOK || w.Body.String() != "ok\n" { t.Fatalf("host %s: %d %q", host, w.Code, w.Body.String()) } } if w := fallbackGet(e, http.MethodHead, lbhealth.Path, "10.0.1.7"); w.Code != http.StatusOK { t.Fatalf("HEAD: %d", w.Code) } joining := healthyState() joining.SetHosting(false) e.lbHealth = joining if w := fallbackGet(e, http.MethodGet, lbhealth.Path, "10.0.1.7"); w.Code != http.StatusServiceUnavailable || !strings.Contains(w.Body.String(), lbhealth.CondHosting) { t.Fatalf("joining: %d %q", w.Code, w.Body.String()) } // A POST, or another path, is ordinary :80 traffic. if w := fallbackGet(e, http.MethodPost, lbhealth.Path, "10.0.1.7"); w.Code == http.StatusOK || w.Code == http.StatusServiceUnavailable { t.Fatalf("POST answered by the health check: %d", w.Code) } e.lbHealth = nil if w := fallbackGet(e, http.MethodGet, lbhealth.Path, "10.0.1.7"); w.Code == http.StatusOK || strings.Contains(w.Body.String(), "unavailable:") { t.Fatalf("region HA off answered the health check: %d %q", w.Code, w.Body.String()) } }

// blockingStore makes every List wait until release is closed, and fail // while down is set: a refresh stuck inside one attempt, and a bucket // outage. type blockingStore struct { release.Store block chan struct{} down bool entered chan struct{} }

func (s *blockingStore) List(ctx context.Context, prefix string) ([]string, error) { if s.block != nil { s.entered <- struct{}{} <-s.block } if s.down { return nil, errors.New("bucket unreachable") } return s.Store.List(ctx, prefix) }

func (s *blockingStore) Get(ctx context.Context, key string) ([]byte, string, error) { if s.down { return nil, "", errors.New("bucket unreachable") } return s.Store.Get(ctx, key) }

// The home map latches its first loaded map and key, stamps every attempt // it finishes, failed or not, and stamps nothing while one is stuck. func TestHomeMapFeedsTheHealthState(t *testing.T) { inner := meshTestStore(t) if err := mintMeshKey(context.Background(), inner); err != nil { t.Fatal(err) } putJSON(t, inner, "control/edges/betty.json", {"v":1,"box":"betty","eip":"2.2.2.2","hosts":{"b.test":{"role":"home"}}}) s := &blockingStore{Store: inner} h := &lbhealth.State{} m := newHomeMap(s, "annie") m.health = h refreshHomeMap(t, m) now := time.Now() failing := strings.Join(h.Failing(now), ",") if strings.Contains(failing, lbhealth.CondForwarding) || strings.Contains(failing, lbhealth.CondRefresh) { t.Fatalf("after a good refresh: %s", failing) }

// A bucket outage: the attempt fails but finishes, and the latches hold. s.down = true before := time.Now() if err := m.Refresh(context.Background()); err == nil { t.Fatal("refresh against a down bucket succeeded") } if got := strings.Join(h.Failing(before.Add(lbhealth.RefreshWindow)), ","); strings.Contains(got, lbhealth.CondForwarding) || strings.Contains(got, lbhealth.CondRefresh) { t.Fatalf("a failed attempt dropped forwarding or went unstamped: %s", got) }

// A stuck attempt stamps nothing. s.down = false s.block, s.entered = make(chan struct{}), make(chan struct{}, 1) done := make(chan struct{}) go func() { _ = m.Refresh(context.Background()); close(done) }() <-s.entered stuckAt := time.Now() if got := strings.Join(h.Failing(stuckAt.Add(lbhealth.RefreshWindow+time.Second)), ","); !strings.Contains(got, lbhealth.CondRefresh) { t.Fatalf("a refresh stuck for longer than the window still counts: %s", got) } close(s.block) <-done }

// A key that never loads leaves forwarding unarmed, whatever the map did. func TestHomeMapWithNoMeshKeyIsNotArmed(t *testing.T) { s := meshTestStore(t) putJSON(t, s, "control/edges/betty.json", {"v":1,"box":"betty","eip":"2.2.2.2","hosts":{}}) h := &lbhealth.State{} m := newHomeMap(s, "annie") m.health = h refreshHomeMap(t, m) if got := strings.Join(h.Failing(time.Now()), ","); !strings.Contains(got, lbhealth.CondForwarding) { t.Fatalf("no mesh key, yet forwarding armed: %s", got) } }

func TestCertMirrorLatchesItsFirstLoad(t *testing.T) { s := meshTestStore(t) h := &lbhealth.State{} m := newCertMirror(s, autocert.DirCache(t.TempDir())) m.health = h if !strings.Contains(strings.Join(h.Failing(time.Now()), ","), lbhealth.CondCerts) { t.Fatal("certificates counted before any load") } if err := m.Refresh(context.Background()); err != nil { t.Fatal(err) } if strings.Contains(strings.Join(h.Failing(time.Now()), ","), lbhealth.CondCerts) { t.Fatal("a loaded mirror did not latch the certificate condition") } }

// The edge's registry watch beats on every poll it completes. func TestRegistryWatchBeats(t testing.T) { e, err := New(openReg(t), nil) if err != nil { t.Fatal(err) } t.Cleanup(e.probers.Stop) h := &lbhealth.State{} e.lbHealth = h stop := make(chan struct{}) t.Cleanup(func() { close(stop) }) go e.Watch(stop) deadline := time.Now().Add(5 time.Second) for strings.Contains(strings.Join(h.Failing(time.Now()), ","), lbhealth.CondEdge) { if time.Now().After(deadline) { t.Fatal("the registry watch never beat") } time.Sleep(50 * time.Millisecond) } } ````

internal/activator/lastsweep_test.go:

````go package activator

import ( "context" "testing" "time" )

// Region HA spec 2 §6: the sweeper's finished sweeps are the hosting // loop's beat. Nothing has swept before Start. func TestLastSweepBeatsAfterEachSweep(t testing.T) { a, , , _ := testActivator(t, 400time.Millisecond) // sweeps every 100ms if !a.LastSweep().IsZero() { t.Fatal("swept before Start") } var none Service if !none.LastSweep().IsZero() { t.Fatal("a nil service swept") } a.Start() t.Cleanup(func() { a.Shutdown(context.Background()) }) waitFor(t, "a sweep", func() bool { return !a.LastSweep().IsZero() }) first := a.LastSweep() if time.Since(first) > 5time.Second { t.Fatalf("beat %v is not wall-clock time", first) } waitFor(t, "a later sweep", func() bool { return a.LastSweep().After(first) }) } ````

internal/host/claimwiring_hosting_test.go:

````go package host

import ( "testing"

"github.com/carlosframework/platform/internal/hostwrite" "github.com/carlosframework/platform/internal/lbhealth" )

// Region HA spec 2 §5, §6: a claim agent that is running is enrolled, so // it is hosting; an agent with claim mode off never says so. Plan 2b's // joining mode is what leaves this false. func TestClaimWiringStartMarksTheBoxHosting(t *testing.T) { lbhealth.Default.SetHosting(false) t.Cleanup(func() { lbhealth.Default.SetHosting(false) }) off := &ClaimWiring{} if err := off.startRegional(t.Context(), nil, nil); err != nil { t.Fatal(err) } if lbhealth.Default.Hosting() { t.Fatal("claim mode off marked the box hosting") } loop := testRegionalLoop{started: make(chan struct{})} w := &ClaimWiring{Writes: hostwrite.Config{Enabled: true}, Providers: ClaimProviders{Regional: loop}} if err := w.startRegional(t.Context(), nil, nil); err != nil { t.Fatal(err) } <-loop.started if !lbhealth.Default.Hosting() { t.Fatal("a running claim agent is not hosting") } if err := w.close(t.Context()); err != nil { t.Fatal(err) } } ````

• [ ] Step 2: Run them to see them fail

Run: nice -n 19 go test ./internal/edge/ -run 'LBHealth|FeedsTheHealth|NoMeshKeyIsNotArmed|CertMirrorLatches|RegistryWatchBeats' -count=1 Expected: FAIL to build (undefined: lbHealthFromEnv, e.lbHealth undefined). The activator test fails to build on a.LastSweep undefined; the host test builds and fails (a running claim agent is not hosting).

• [ ] Step 3: Write the answer and its feeds

internal/edge/lbhealth.go:

````go package edge

import ( "net/http" "time"

"github.com/carlosframework/platform/internal/lbhealth" )

// The load balancer health answer (region HA spec 2 §6): GET or HEAD // /.carlos/lb-health on port 80, from any host. The network load // balancer's HTTP check sends the target's address as the Host header, so // the steering probe, which answers only for its own hostname, cannot // serve it. Every condition is this box's own; lbhealth has the list.

// lbHealthFromEnv is the state the edge answers from: the process's // (lbhealth.Default) with CARLOSREGIONHA on, and nil with it off, which // leaves the path to the ordinary :80 handling exactly as before. The // activator's sweep is the hosting loop's beat; an edge with no activator // (carlos edge) has none, and its answer is never 200. func lbHealthFromEnv(getenv func(string) string, act Activator) *lbhealth.State { if !regionHAFromEnv(getenv) { return nil } s := lbhealth.Default if a, ok := act.(interface{ LastSweep() time.Time }); ok { s.SetActivator(a.LastSweep) } return s }

// lbHealthAnswer answers the health check when r is one and region HA is // on, reporting whether it did. func (e Edge) lbHealthAnswer(w http.ResponseWriter, r http.Request) bool { if e.lbHealth == nil { return false } return lbhealth.Serve(w, r, e.lbHealth, time.Now()) } ````

````diff --- a/internal/edge/edge.go +++ b/internal/edge/edge.go @@ -56,6 +56,7 @@ "github.com/carlosframework/platform/internal/gate" "github.com/carlosframework/platform/internal/geo" "github.com/carlosframework/platform/internal/geoip" + "github.com/carlosframework/platform/internal/lbhealth" "github.com/carlosframework/platform/internal/provision" "github.com/carlosframework/platform/internal/registry" "github.com/carlosframework/platform/internal/release" @@ -131,6 +132,10 @@ // environment; see steeringprobe.go for why it is a serving concern at // all. steeringHost string + // lbHealth is the load balancer health state (region HA spec 2 §6): + // lbhealth.Default with CARLOSREGIONHA on, nil and inert with it + // off. Set once in New. + lbHealth lbhealth.State // appsDomain is the platform apps domain (PLATFORMAPPSDOMAIN, // normalized), read once in New; cookieLogged is the set of route hosts // whose cookie rewrite has been logged. Both serve cookiescope.go. "" @@ -368,7 +373,8 @@ // New builds an Edge over reg and performs the startup reload. act may be nil // (no hibernation). func New(reg registry.Registry, act Activator) (*Edge, error) {

• e := &Edge{reg: reg, act: act, gate: gate.New(), probers: newProberSet(), gzipOn: gzipEnabled(), steeringHost: steeringProbeHostFromEnv(), appsDomain: appsDomainFromEnv()} + e := &Edge{reg: reg, act: act, gate: gate.New(), probers: newProberSet(), gzipOn: gzipEnabled(), steeringHost: steeringProbeHostFromEnv(), appsDomain: appsDomainFromEnv(), + lbHealth: lbHealthFromEnv(os.Getenv, act)} e.geo = geo.NewSwappable() e.geoLookup = e.geo.Lookup // Generations (zero-downtime spec §7.2), before the startup Reload so @@ -2226,6 +2232,12 @@ if h, _, err := net.SplitHostPort(host); err == nil { host = h } + // The load balancer's health check (region HA spec 2 §6, lbhealth.go): + // any host, so it comes before every host-based branch. Inert with + // region HA off. + if e.lbHealthAnswer(w, r) { + return + } // The steering hostname answers here too (ServeHTTP's own first // branch, and the same reasoning): the health check is HTTPS today, // but a :80 probe — or an operator's curl — must not be handed this @@ -2305,6 +2317,9 @@ continue } errs = 0 + // The edge's own loop is alive and its routing table readable + // (region HA spec 2 §6). A wedged poll stops beating. + e.lbHealth.EdgeBeat(time.Now()) if fp == last { continue } ````

````diff --- a/internal/edge/homemap.go +++ b/internal/edge/homemap.go @@ -86,6 +86,7 @@ "time"

"github.com/carlosframework/platform/internal/edgesfile" + "github.com/carlosframework/platform/internal/lbhealth" "github.com/carlosframework/platform/internal/release" )

@@ -279,6 +280,11 @@ // privateDial is this edge's region HA switch: only with it on does // the fold dial a same-region box's private address (dialAddress). privateDial bool + + // health is the load balancer health state (region HA spec 2 §6), + // nil without region HA: Refresh latches the first loaded map and key + // and stamps every attempt it completes. + health *lbhealth.State }

func newHomeMap(s release.Store, self string) *homeMap { @@ -296,6 +302,7 @@ } m := newHomeMap(e.staticStore, e.box) m.privateDial = regionHAFromEnv(os.Getenv) + m.health = e.lbHealth return m }

@@ -303,10 +310,23 @@ // fail INDEPENDENTLY — a bucket that answered the edges list but not the key // object should still land the fold — so both run and their errors join. // Whatever fails leaves its own last-good in place; the caller logs. +// +// For the load balancer's health answer (region HA spec 2 §6) each half +// latches its first success, and every call that returns stamps an +// attempt, failed or not: a bucket outage after arming leaves the box +// healthy, and only a refresh stuck inside one attempt goes stale. func (m *homeMap) Refresh(ctx context.Context) error { m.refreshMu.Lock() defer m.refreshMu.Unlock()

• return errors.Join(m.refreshHosts(ctx), m.refreshKeys(ctx)) + hostsErr, keysErr := m.refreshHosts(ctx), m.refreshKeys(ctx) + if hostsErr == nil { + m.health.MapLoaded() + } + if current, _ := m.meshKeys(); keysErr == nil && len(current) > 0 { + m.health.KeyLoaded() + } + m.health.RefreshAttempted(time.Now()) + return errors.Join(hostsErr, keysErr) }

// refreshHosts rebuilds the whole map from one list plus a get per key — no ````

````diff --- a/internal/edge/certsync.go +++ b/internal/edge/certsync.go @@ -50,6 +50,7 @@ "golang.org/x/crypto/acme/autocert" "golang.org/x/net/idna"

+ "github.com/carlosframework/platform/internal/lbhealth" "github.com/carlosframework/platform/internal/release" )

@@ -516,6 +517,10 @@ // first-and-every-tenth log cadence. partialPolls int

+ // health is the load balancer health state the first landed refresh + // latches (region HA spec 2 §6); nil without region HA. + health *lbhealth.State + // synced remembers, per name, the mirrored bytes last reconciled into the // local cache, so a steady state costs no local I/O at all. syncMu sync.Mutex @@ -546,6 +551,7 @@ return nil } m := newCertMirror(e.staticStore, local) + m.health = e.lbHealth m.gated = func() bool { _, held := e.issuerGate() return !held @@ -752,6 +758,10 @@ m.snap = snap m.mu.Unlock() m.syncLocal(ctx, snap) + // The local cache the handshake reads now holds what the bucket does: + // the certificate condition is met for good (§6). A later failed + // refresh keeps serving this copy and leaves the latch set. + m.health.CertsLoaded() if m.wake != nil && wildcardMaterialChanged(prev, snap) { // A takeover (or a first wildcard for a new account) reaches the // serving path in one poll instead of up to twelve hours. ````

````diff --- a/internal/activator/activator.go +++ b/internal/activator/activator.go @@ -1237,6 +1237,10 @@

sweepStop chan struct{} sweepDone chan struct{} + // lastSweep is when the sweeper last finished a sweep, in unix + // nanoseconds (0: never): the hosting loop's own beat, which the load + // balancer's health answer reads (region HA spec 2 §6, LastSweep). + lastSweep atomic.Int64

// gens is the generations switch (SetGenerations) as read at New. Off: // every spawn binds the canonical socket and no retire hook fires. @@ -4439,9 +4443,25 @@ return case <-t.C: a.sweepOnce() + a.lastSweep.Store(time.Now().UnixNano()) } } }() +} + +// LastSweep is when the sweeper last finished a sweep, or the zero time +// before its first. The sweep runs at least every 30 seconds, so a box +// whose last sweep is older than a minute has a hosting loop that is stuck +// (region HA spec 2 §6). A nil service has never swept. +func (a *Service) LastSweep() time.Time { + if a == nil { + return time.Time{} + } + n := a.lastSweep.Load() + if n == 0 { + return time.Time{} + } + return time.Unix(0, n) }

// markCleanParkLocked records that a clean-exit park is stopping e's current ````

```diff --- a/internal/host/claimwiring.go +++ b/internal/host/claimwiring.go @@ -10,6 +10,7 @@ "github.com/carlosframework/platform/internal/hostclaim" "github.com/carlosframework/platform/internal/hostguard" "github.com/carlosframework/platform/internal/hostwrite" + "github.com/carlosframework/platform/internal/lbhealth" "github.com/carlosframework/platform/internal/provision" "github.com/carlosframework/platform/internal/schedule" ) @@ -94,6 +95,12 @@ w.cancel = cancel w.joined = make(chan struct{}) go func() { defer close(w.joined); w.Providers.Regional.Loop(workerCtx, serial, report) }() + // The claim agent is running: it refused to start without its + // host-actor enrollment and a matching roster row, so this box is + // hosting (region HA spec 2 §5, §6). Plan 2b's mode selection owns + // this flag from then on: joining mode leaves it false, and only an + // active roster entry with an armed guard sets it. + lbhealth.Default.SetHosting(true) return nil } func (w *ClaimWiring) close(ctx context.Context) error { ```

• [ ] Step 4: Run the tests

nice -n 19 go test ./internal/edge/ -run 'LBHealth|FeedsTheHealth|NoMeshKeyIsNotArmed|CertMirrorLatches|RegistryWatchBeats|HomeMap|Backhaul|Hop|HTTPFallback|FollowsRegionHA|PublicOnly' -count=1
nice -n 19 go test ./internal/activator/ -run LastSweep -count=1
nice -n 19 go test ./internal/host/ -run ClaimWiring -count=1

Expected: PASS.

• [ ] Step 5: Boundary list and census

````diff --- a/regionhaboundarytest.go +++ b/regionhaboundarytest.go @@ -24,6 +24,7 @@ "internal/boxchoice/policy.go", "internal/host/boxaddress.go", "internal/edgesfile/private.go", + "internal/edge/lbhealth.go", }

// regionHAAdapterFiles are spec 2's AWS adapters. Each is named for AWS so ````

Add this comment line after the overlay's last comment line:

````text

Task 9: the load balancer health answer (internal/edge lbhealth.go, answered in httpFallbackHandler; the home map and the certificate mirror latch their first loads, Edge.Watch beats, the activator's sweeper beats through Service.LastSweep, and ClaimWiring.startRegional marks the box hosting). None of the changed units is a root-read unit, and the answer reads only in-memory state. Envelope-only, and the legacy seal stands.

````

Then set these rows. A @coverage row replaces the overlay's row for the same directory, or is appended when there is none; every other row is appended:

```text @coverage internal/host 87d13f40c52d39585cd9579eafea61b772577c6d3bf23e9e3951af778711e19a @coverage internal/edge 8e47251c54e3c5ee9cb6e8b9b1ad3ae87da528436c6a870cadb07ca3a8ac3146 @coverage internal/activator c2eb11024375a706d70652dee68d446202a264927f819deab6321b652e82c2bb ```

Run the census checks and go test . -run 'TestRegionHA(Core|Adapters|S3)' -count=1. Expected: PASS.

• [ ] Step 6: Commit

git add internal/edge internal/activator internal/host/claimwiring.go internal/host/claimwiring_hosting_test.go region_ha_boundary_test.go internal/provision/testdata/plan14-root-reads-region-ha-2a.tsv
git commit -m "region HA 2a: the load balancer health answer"

---

Task 10: The role doors and carlos system box role and group role

Files:

• Create: internal/console/apiboxroles.go, internal/console/apiboxroles_test.go
• Modify: internal/console/authcli.go (two routes, with host claims on)
• Modify: internal/console/svcroutes_test.go, internal/console/claim_invariant_fixtures_test.go, internal/console/claim_invariant_probes_nonapp_test.go, internal/console/testdata/claim-routes.txt
• Create: system_roles.go, system_roles_test.go
• Modify: system.go (the box and group verbs and their usage)
• Modify: docs/site/cli.md (a "Box and group roles" section)
• Modify: approved_region_ha_copy_test.go (wire RHA-C1 to RHA-C16), region_ha_boundary_test.go
• Modify: internal/provision/plan9_b8_census_test.go, internal/provision/plan14_root_reads_test.go (the seal), the overlay

Interfaces:

• Consumes: Task 3's ParseBoxRole, ParseGroupRole, RoleName, SetBoxRole, SetGroupRole; apiSystemRoller (ops account owners and CARLOS_SYSTEM_OPERATORS).
• Produces: POST /api/cli/system/boxes/{box}/role and POST /api/cli/system/groups/{region}/role, body {"role": "<name>"}, answering {"box", "role"} and {"region", "group", "role", "boxes"}; func (s *Server) boxRoster(ctx) (hostactors.Directory, release.Signer, hostactors.Observed, error); CLI setBoxRole, setGroupRole, runSystemBoxVerb, runSystemGroupVerb.

Wait for the copy. This task writes RHA-C1 to RHA-C16 into console replies, CLI help and docs/site/cli.md. Start it only once Paul has approved those rows (Task 1, Step 4), and use his text where it differs from the drafts below. Until then TestRegionHABuiltCopyIsApproved fails, by design; do not mark a row approved to get past it.

The doors are registered only with host claims on (the roster exists only then), closed to every service credential, and answer 404 unless CARLOS_REGION_HA is on too. A box is named by its roster reference or its label.

Census ruling region-ha-2a, condition 2. Server.boxRoster reads the signed roster (Directory.Read, kind file:Read) through the same source and verification as every other roster read. It is operator-only (it is reached only past apiSystemRoller, from the two role doors), it is read-only (the doors write through Task 3's SetBoxRole and SetGroupRole, which verify and sign as Directory.Replace always has), and the roles it carries are read by the placement chooser alone and never feed root's claim authority. The move door does not call it (Task 12 reads roles through the chooser). It is a new accepted-legacy unit (J4 root input), and the seal moves.

• [ ] Step 1: Write the failing tests

internal/console/apiboxroles_test.go:

````go package console

import ( "bytes" "context" "encoding/json" "errors" "net/http" "net/http/httptest" "reflect" "testing"

"github.com/carlosframework/platform/internal/hostactors" "github.com/carlosframework/platform/internal/release" )

// roleFixture is a console with region HA on and a signed roster: in ie, // an operator-enrolled box ie-old and two boxes the group "routers" // enrolled; uk has no group. func roleFixture(t testing.T) consoleFixture { t.Helper() fx := newConsoleFixture(t) fx.srv.cfg.RegionHA = true old := moveRosterBox(ieBox("ie-old")) g1, g2 := moveRosterBox(ieBox("ie-g1")), moveRosterBox(ieBox("ie-g2")) g1.Group, g2.Group = "routers", "routers" putMoveRoster(t, fx, hostactors.Roster{ Boxes: map[string]hostactors.Box{"ie-old": old, "ie-g1": g1, "ie-g2": g2}, Regions: map[string]hostactors.Region{ "ie": {Account: moveTestAccount, AWSRegion: "eu-west-1", GroupName: "routers", GroupARN: "arn:group"}, "uk": {Account: moveTestAccount, AWSRegion: "eu-west-2"}, }, }) return fx }

// roleCall drives one role door as addr, past cliRoute. func roleCall(t *testing.T, h http.HandlerFunc, addr, pathKey, pathValue, body string) (int, map[string]any) { t.Helper() r := httptest.NewRequest(http.MethodPost, "/api/cli/system/x/y/role", bytes.NewBufferString(body)) r.Header.Set("Content-Type", "application/json") r.SetPathValue(pathKey, pathValue) r = r.WithContext(context.WithValue(r.Context(), cliRouteKey{}, Principal{Kind: PrincipalUser, Address: addr})) w := httptest.NewRecorder() h(w, r) var out map[string]any _ = json.Unmarshal(w.Body.Bytes(), &out) return w.Code, out }

func rosterNow(t testing.T, fx consoleFixture) hostactors.Roster { t.Helper() , , o, err := fx.srv.boxRoster(context.Background()) if err != nil { t.Fatal(err) } return o.Roster }

func TestBoxRoleDoor(t *testing.T) { fx := roleFixture(t) h := fx.srv.handleAPISystemBoxRole if st, := roleCall(t, h, "someone@keymail.dev", "box", "ie-old", {"role":"legacy"}); st != http.StatusNotFound { t.Fatalf("a non-operator = %d, want 404", st) } fx.srv.cfg.RegionHA = false if st, := roleCall(t, h, "paul@keymail.dev", "box", "ie-old", {"role":"legacy"}); st != http.StatusNotFound { t.Fatalf("region HA off = %d, want 404", st) } fx.srv.cfg.RegionHA = true if st, out := roleCall(t, h, "paul@keymail.dev", "box", "ie-old", {"role":"draining"}); st != http.StatusBadRequest || out["error"] != boxRoleRefusal { t.Fatalf("unknown role = %d %v", st, out) } if st, out := roleCall(t, h, "paul@keymail.dev", "box", "nobody", {"role":"legacy"}); st != http.StatusNotFound || out["error"] != noEnrolledBox("nobody") { t.Fatalf("unknown box = %d %v", st, out) } // By label, as an operator names it. st, out := roleCall(t, h, "paul@keymail.dev", "box", "ie-old-label", {"role":"legacy"}) if st != http.StatusOK || out["box"] != "ie-old" || out["role"] != "legacy" { t.Fatalf("set = %d %v", st, out) } if got := rosterNow(t, fx).Boxes["ie-old"].Role; got != hostactors.RoleLegacy { t.Fatalf("the roster says %q", got) } st, out = roleCall(t, h, "paul@keymail.dev", "box", "ie-old", {"role":"default"}) if st != http.StatusOK || out["role"] != "default" || rosterNow(t, fx).Boxes["ie-old"].Role != hostactors.RoleDefault { t.Fatalf("back to default = %d %v", st, out) } }

func TestGroupRoleDoor(t *testing.T) { fx := roleFixture(t) h := fx.srv.handleAPISystemGroupRole if st, out := roleCall(t, h, "paul@keymail.dev", "region", "ie", {"role":"platform"}); st != http.StatusBadRequest || out["error"] != groupRoleRefusal { t.Fatalf("platform group = %d %v", st, out) } if st, out := roleCall(t, h, "paul@keymail.dev", "region", "uk", {"role":"legacy"}); st != http.StatusNotFound || out["error"] != noGroupIn("uk") { t.Fatalf("no group = %d %v", st, out) } st, out := roleCall(t, h, "paul@keymail.dev", "region", "ie", {"role":"legacy"}) if st != http.StatusOK || out["group"] != "routers" || out["role"] != "legacy" || !reflect.DeepEqual(out["boxes"], []any{"ie-g1", "ie-g2"}) { t.Fatalf("set = %d %v", st, out) } r := rosterNow(t, fx) if r.Regions["ie"].Role != hostactors.RoleLegacy || r.Boxes["ie-g1"].Role != hostactors.RoleLegacy || r.Boxes["ie-g2"].Role != hostactors.RoleLegacy || r.Boxes["ie-old"].Role != hostactors.RoleDefault { t.Fatalf("roster after the group role: %+v", r.Boxes) } }

func TestRoleWriteErrors(t *testing.T) { for , c := range []struct { err error code int msg string }{ {release.ErrCASConflict, http.StatusConflict, boxRosterChanged}, {hostactors.ErrRoster, http.StatusConflict, boxRosterChanged}, {errors.New("SlowDown"), http.StatusServiceUnavailable, boxRoleWriteFailure}, } { w := httptest.NewRecorder() roleWriteError(w, "test", c.err) var out map[string]string = json.Unmarshal(w.Body.Bytes(), &out) if w.Code != c.code || out["error"] != c.msg { t.Fatalf("%v = %d %v", c.err, w.Code, out) } } }

// The doors exist only on a host-claim console, operator-only and 404 // with region HA off. func TestRoleDoorsAreRegisteredWithHostClaims(t *testing.T) { st := openGatedStore(t) rel := release.NewDirStore(t.TempDir()) path := "/api/cli/system/boxes/ie-old/role" off, err := NewServer(Config{Origin: "https://console.example", Keymail: "https://keymail.example", Releases: rel}, st) if err != nil { t.Fatal(err) } ghostStatus, ghostBody := serveHostClaim(t, off, "/unknown-role-route", "") if status, body := serveHostClaim(t, off, path, ""); status != ghostStatus || !bytes.Equal(body, ghostBody) { t.Fatalf("claims off = %d %q, want the unknown route's %d %q", status, body, ghostStatus, ghostBody) } token := mintHostClaimToken(t, st, "paul@keymail.dev") on, err := NewServer(Config{Origin: "https://console.example", Keymail: "https://keymail.example", Releases: rel, HostClaim: true, SystemOperators: []string{"paul@keymail.dev"}}, st) if err != nil { t.Fatal(err) } if status, := serveHostClaim(t, on, path, ""); status != http.StatusUnauthorized { t.Fatalf("unsigned = %d", status) } if status, := serveHostClaim(t, on, path, token); status != http.StatusNotFound { t.Fatalf("region HA off = %d, want 404", status) } } ````

system_roles_test.go:

````go package main

import ( "context" "errors" "net/http" "reflect" "testing" )

func TestSystemVerbKnowsBoxAndGroup(t *testing.T) { for , verb := range []string{"box", "group"} { got, rest, := systemVerb([]string{verb, "role", "x", "legacy"}) if got != verb || len(rest) != 3 || rest[0] != "role" { t.Fatalf("systemVerb(%s) = %q %q", verb, got, rest) } } }

// fakeRolePost answers one role call with status and body, recording what // was asked. type fakeRolePost struct { status int body string path string in any }

func (f *fakeRolePost) post(_ context.Context, path string, in any) (int, []byte, error) { f.path, f.in = path, in return f.status, []byte(f.body), nil }

func TestSetBoxRole(t *testing.T) { f := &fakeRolePost{status: http.StatusOK, body: {"box":"i-0abc","role":"legacy"}} line, err := setBoxRole(context.Background(), f.post, "old box", "legacy") if err != nil || line != "i-0abc now has the legacy role" { t.Fatalf("line %q, %v", line, err) } if f.path != "/api/cli/system/boxes/old%20box/role" || !reflect.DeepEqual(f.in, map[string]string{"role": "legacy"}) { t.Fatalf("asked %s %v", f.path, f.in) } f = &fakeRolePost{status: http.StatusBadRequest, body: {"error":"role must be default, legacy or platform"}} if , err := setBoxRole(context.Background(), f.post, "x", "boss"); err == nil || err.Error() != "role must be default, legacy or platform" { t.Fatalf("refusal: %v", err) } failing := func(context.Context, string, any) (int, []byte, error) { return 0, nil, errors.New("offline") } if , err := setBoxRole(context.Background(), failing, "x", "legacy"); err == nil { t.Fatal("a failed call printed a line") } }

func TestSetGroupRole(t *testing.T) { f := &fakeRolePost{status: http.StatusOK, body: {"region":"ie","group":"routers","role":"legacy","boxes":["i-1","i-2"]}} lines, err := setGroupRole(context.Background(), f.post, "ie", "legacy") if err != nil { t.Fatal(err) } want := []string{"the group in ie now has the legacy role", "i-1 now has the legacy role", "i-2 now has the legacy role"} if !reflect.DeepEqual(lines, want) || f.path != "/api/cli/system/groups/ie/role" { t.Fatalf("lines %q path %s", lines, f.path) } f = &fakeRolePost{status: http.StatusOK, body: {"region":"ie","group":"routers","role":"default","boxes":[]}} if lines, _ := setGroupRole(context.Background(), f.post, "ie", "default"); len(lines) != 1 { t.Fatalf("an empty group: %q", lines) } } ````

The route censuses learn the two doors:

```diff --- a/internal/console/svcroutes_test.go +++ b/internal/console/svcroutes_test.go @@ -286,6 +286,10 @@ hostClaimDoors := []string{ "POST /api/cli/ops/hosts/{host}/resolve/inspect", "POST /api/cli/ops/hosts/{host}/resolve", + // Box and group roles (region HA spec 2 §7): operator-only, and 404 + // unless CARLOS_REGION_HA is on as well. + "POST /api/cli/system/boxes/{box}/role", + "POST /api/cli/system/groups/{region}/role", } movesDoors := []string{ "POST /api/cli/apps/{account}/{app}/instances/{host}/move/resolve", ```

```diff --- a/internal/console/claim_invariant_fixtures_test.go +++ b/internal/console/claim_invariant_fixtures_test.go @@ -1129,6 +1129,8 @@ }}, "route POST /api/cli/ops/hosts/{host}/resolve/inspect": claimRead("operator host-loss inspection: reads the claim, the record and the roster and signs a challenge; writes nothing"), "route POST /api/cli/system/antivirus": claimNonApp("the deployment scanning switch (system/antivirus.json)"), + "route POST /api/cli/system/boxes/{box}/role": claimNonApp("operator box role (region HA spec 2 §7): one signed roster write to control/host-actors.json; names no app"), + "route POST /api/cli/system/groups/{region}/role": claimNonApp("operator group role (region HA spec 2 §7): one signed roster write to control/host-actors.json, the group and the boxes it enrolled; names no app"), "route POST /api/cli/system/layout": claimNonApp("the deployment layout intent (control/layout.json)"), "route POST /api/cli/system/activation/cancel": claimNonApp("cancels an unpublished system activation"), "route POST /api/cli/system/pointer-digests": claimNonApp("operator switch: writes or removes each account's root-signed pointer digest marker (releases/<acct>/pointer-digests), classified exactly by the mutation boundary"), ```

```diff --- a/internal/console/claim_invariant_probes_nonapp_test.go +++ b/internal/console/claim_invariant_probes_nonapp_test.go @@ -674,6 +674,54 @@ return "" }, }, + // Box and group roles (region HA spec 2 §7): one signed roster write, + // only with region HA on. + "route POST /api/cli/system/boxes/{box}/role": { + setup: func(t *testing.T, cx *claimCase) { cx.srv.cfg.RegionHA = true }, + do: claimAPIReply(http.MethodPost, "/api/cli/system/boxes/box-ie-default/role", {"role":"legacy"}), + body: []string{"role":"legacy"}, + effect: func(t *testing.T, cx *claimCase, r claimReply) string { + o, err := cx.claimFixtureDirectory().Read(context.Background()) + if err != nil || o.Roster.Boxes["box-ie-default"].Role != hostactors.RoleLegacy { + return "box-ie-default does not have the legacy role in the signed roster" + } + return "" + }, + }, + "route POST /api/cli/system/groups/{region}/role": { + setup: func(t *testing.T, cx *claimCase) { + cx.srv.cfg.RegionHA = true + ctx := context.Background() + o, err := cx.claimFixtureDirectory().Read(ctx) + if err != nil { + t.Fatal(err) + } + next := hostactors.Clone(o.Roster) + g := next.Regions["ie"] + g.GroupName, g.GroupARN = "routers", "arn:aws:autoscaling:eu-west-1:123456789012:autoScalingGroup:routers" + next.Regions["ie"] = g + b := next.Boxes["box-ie-default"] + b.Group = "routers" + next.Boxes["box-ie-default"] = b + next.Revision++ + raw, err := release.SignRootEnvelope(release.RootSigner(cx.priv), next) + if err != nil { + t.Fatal(err) + } + if _, err = cx.rel.Put(ctx, hostactors.Key, raw, o.ETag); err != nil { + t.Fatal(err) + } + }, + do: claimAPIReply(http.MethodPost, "/api/cli/system/groups/ie/role", {"role":"legacy"}), + body: []string{"boxes":["box-ie-default"]}, + effect: func(t *testing.T, cx *claimCase, r claimReply) string { + o, err := cx.claimFixtureDirectory().Read(context.Background()) + if err != nil || o.Roster.Regions["ie"].Role != hostactors.RoleLegacy || o.Roster.Boxes["box-ie-default"].Role != hostactors.RoleLegacy { + return "the group in ie and its box do not have the legacy role in the signed roster" + } + return "" + }, + }, "route POST /api/cli/system/seeds-floor": { do: claimAPIReply(http.MethodPost, "/api/cli/system/seeds-floor", {"minrelease":"v0.27.0"}), body: []string{"minrelease":"v0.27.0"}, ```

```diff --- a/internal/console/testdata/claim-routes.txt +++ b/internal/console/testdata/claim-routes.txt @@ -237,6 +237,8 @@ route POST /api/cli/services/{account}/{name}/rotate route POST /api/cli/system/activation/cancel route POST /api/cli/system/antivirus +route POST /api/cli/system/boxes/{box}/role +route POST /api/cli/system/groups/{region}/role route POST /api/cli/system/layout route POST /api/cli/system/legacy-routes route POST /api/cli/system/pointer-digests ```

• [ ] Step 2: Run them to see them fail

Run: nice -n 19 go test ./internal/console/ -run 'BoxRoleDoor|GroupRoleDoor|RoleWriteErrors|RoleDoorsAreRegistered' -count=1 Expected: FAIL to build (undefined: handleAPISystemBoxRole, undefined: boxRoster). go test . -run 'SystemVerbKnowsBoxAndGroup|SetBoxRole|SetGroupRole' fails to build on undefined: setBoxRole.

• [ ] Step 3: Write the doors and the CLI

internal/console/apiboxroles.go:

````go package console

import ( "context" "errors" "fmt" "log" "net/http"

"github.com/carlosframework/platform/internal/hostactors" "github.com/carlosframework/platform/internal/release" )

// Box and group roles (region HA spec 2 §7, A11-1): carlos system box role // <box> <role> and carlos system group role <region> <role>. A legacy or // platform box takes no new instance, relocation or move; it keeps the // ones it has (H25). A group's role is every box it enrolled, now and // later. The roles live in the signed roster, so the console, the only // holder of the root key, is the only writer. // // Operator-only (apiSystemRoller, the update doors' authority), registered // only with host claims on, and answering 404 unless CARLOSREGIONHA is on // too: with region HA off nothing reads a role.

const ( boxRoleRefusal = "role must be default, legacy or platform" groupRoleRefusal = "a group's role must be default or legacy" boxRosterUnread = "could not read the box roster. Try again." boxRosterChanged = "the box roster changed while the role was being saved, and nothing was changed. Try again." boxRoleWriteFailure = "could not save the role. Try again." )

func noEnrolledBox(name string) string { return fmt.Sprintf("no enrolled box is named %s", name) }

func noGroupIn(region string) string { return fmt.Sprintf("%s has no group", region) }

func retiredBoxRole(box string) string { return fmt.Sprintf("%s is retired, so it takes no role", box) }

// boxRoster reads the signed roster with the root signer that writes it, // for the role doors alone, which have already let only an operator in // (roleOperator). It is a read, of the one roster every reader reads // (control/host-actors.json through hostactors.Directory). Census ruling // region-ha-2a's conditions hold it to that: operator-only, read-only, the // same roster, and no role it reads feeds root's claim authority (roles // are read by the placement chooser alone). func (s *Server) boxRoster(ctx context.Context) (hostactors.Directory, release.Signer, hostactors.Observed, error) { if s.cfg.Releases == nil || s.cfg.Bucket == "" || len(s.cfg.Key) == 0 { return hostactors.Directory{}, release.Signer{}, hostactors.Observed{}, errors.New("console: no deployment bucket or root key") } signer := release.RootSigner(s.cfg.Key) dir := hostactors.Directory{Store: s.cfg.Releases, PublicKey: signer.RootPublicKey(), Bucket: s.cfg.Bucket} o, err := dir.Read(ctx) return dir, signer, o, err }

// rosterRef finds a box by reference, or by the one label no retired box // shares (hostactors.Directory.Resolve's rule, over a roster in hand). func rosterRef(r hostactors.Roster, name string) (string, bool) { if _, ok := r.Boxes[name]; ok { return name, true } found := "" for ref, b := range r.Boxes { if b.Label == name && b.State != "retired" { if found != "" { return "", false } found = ref } } return found, found != "" }

// roleOperator is the role doors' gate: an operator, with region HA on. func (s Server) roleOperator(w http.ResponseWriter, r http.Request) (string, bool) { addr, _, ok := s.apiSystemRoller(w, r) if !ok { return "", false } if !s.cfg.RegionHA { http.NotFound(w, r) return "", false } return addr, true }

// roleWriteError answers a roster write that did not land. func roleWriteError(w http.ResponseWriter, what string, err error) { if errors.Is(err, release.ErrCASConflict) || errors.Is(err, hostactors.ErrRoster) { jsonError(w, http.StatusConflict, boxRosterChanged) return } log.Printf("console: %s: %v", what, err) jsonError(w, http.StatusServiceUnavailable, boxRoleWriteFailure) }

func (s Server) handleAPISystemBoxRole(w http.ResponseWriter, r http.Request) { by, ok := s.roleOperator(w, r) if !ok { return } var req struct { Role string json:"role" } if !readJSON(w, r, &req) { return } role, err := hostactors.ParseBoxRole(req.Role) if err != nil { jsonError(w, http.StatusBadRequest, boxRoleRefusal) return } dir, signer, o, err := s.boxRoster(r.Context()) if err != nil { log.Printf("console: box role: roster: %v", err) jsonError(w, http.StatusServiceUnavailable, boxRosterUnread) return } name := r.PathValue("box") ref, ok := rosterRef(o.Roster, name) if !ok { jsonError(w, http.StatusNotFound, noEnrolledBox(name)) return } if o.Roster.Boxes[ref].State == "retired" { jsonError(w, http.StatusConflict, retiredBoxRole(ref)) return } next, err := dir.SetBoxRole(r.Context(), signer, o, ref, role) if err != nil { roleWriteError(w, "box role "+ref, err) return } log.Printf("console: box %s has the %s role (by %s, roster revision %d)", ref, hostactors.RoleName(role), by, next.Roster.Revision) writeJSON(w, http.StatusOK, map[string]string{"box": ref, "role": hostactors.RoleName(role)}) }

func (s Server) handleAPISystemGroupRole(w http.ResponseWriter, r http.Request) { by, ok := s.roleOperator(w, r) if !ok { return } var req struct { Role string json:"role" } if !readJSON(w, r, &req) { return } role, err := hostactors.ParseGroupRole(req.Role) if err != nil { jsonError(w, http.StatusBadRequest, groupRoleRefusal) return } dir, signer, o, err := s.boxRoster(r.Context()) if err != nil { log.Printf("console: group role: roster: %v", err) jsonError(w, http.StatusServiceUnavailable, boxRosterUnread) return } region := r.PathValue("region") g, ok := o.Roster.Regions[region] if !ok || g.GroupName == "" { jsonError(w, http.StatusNotFound, noGroupIn(region)) return } next, boxes, err := dir.SetGroupRole(r.Context(), signer, o, region, role) if err != nil { roleWriteError(w, "group role "+region, err) return } log.Printf("console: the group in %s (%s) and its boxes %v have the %s role (by %s, roster revision %d)", region, g.GroupName, boxes, hostactors.RoleName(role), by, next.Roster.Revision) if boxes == nil { boxes = []string{} } writeJSON(w, http.StatusOK, map[string]any{"region": region, "group": g.GroupName, "role": hostactors.RoleName(role), "boxes": boxes}) } ````

```diff --- a/internal/console/authcli.go +++ b/internal/console/authcli.go @@ -512,6 +512,10 @@ // ActorDirectory both routes stop at the same inert refusal. s.cliRoute("POST /api/cli/ops/hosts/{host}/resolve/inspect", svcrole.Set{}, s.handleAPIHostResolveInspect) s.cliRoute("POST /api/cli/ops/hosts/{host}/resolve", svcrole.Set{}, s.handleAPIHostResolve) + // Box and group roles (region HA spec 2 §7, apiboxroles.go): + // operator-only, and 404 unless CARLOS_REGION_HA is on too. + s.cliRoute("POST /api/cli/system/boxes/{box}/role", svcrole.Set{}, s.handleAPISystemBoxRole) + s.cliRoute("POST /api/cli/system/groups/{region}/role", svcrole.Set{}, s.handleAPISystemGroupRole) } // carlos status --layout (plan 4, task 5): which layout each instance is // on, box by box. Same posture as the door above — an empty role set, the ```

system_roles.go:

````go package main

import ( "context" "encoding/json" "fmt" "log" "net/http" "net/url" "os" )

// carlos system box role <box> <role> and carlos system group role // <region> <role> (region HA spec 2 §7, A11-1). A box's role says whether // placement may choose it: legacy and platform boxes take no new // instances and keep the ones they have. A group's role is every box it // has enrolled and every box it enrolls later. Operators only, and only on // a console with region HA on.

// rolePost is the one console call a role change makes. type rolePost func(ctx context.Context, path string, in any) (int, []byte, error)

// boxRoleLine is what the CLI says once a box has its role. func boxRoleLine(box, role string) string { return fmt.Sprintf("%s now has the %s role", box, role) }

// groupRoleLine is what it says once a group has its role. func groupRoleLine(region, role string) string { return fmt.Sprintf("the group in %s now has the %s role", region, role) }

// setBoxRole asks the console to give box the role, and returns the line // to print. func setBoxRole(ctx context.Context, post rolePost, box, role string) (string, error) { status, body, err := post(ctx, "/api/cli/system/boxes/"+url.PathEscape(box)+"/role", map[string]string{"role": role}) if err != nil { return "", err } if status != http.StatusOK { return "", apiErrorFromBody(status, body) } var out struct { Box string json:"box" Role string json:"role" } if err := json.Unmarshal(body, &out); err != nil { return "", fmt.Errorf("unreadable response: %w", err) } return boxRoleLine(out.Box, out.Role), nil }

// setGroupRole asks the console to give region's group the role, and // returns the lines to print: the group's, then one per box it set. func setGroupRole(ctx context.Context, post rolePost, region, role string) ([]string, error) { status, body, err := post(ctx, "/api/cli/system/groups/"+url.PathEscape(region)+"/role", map[string]string{"role": role}) if err != nil { return nil, err } if status != http.StatusOK { return nil, apiErrorFromBody(status, body) } var out struct { Region string json:"region" Role string json:"role" Boxes []string json:"boxes" } if err := json.Unmarshal(body, &out); err != nil { return nil, fmt.Errorf("unreadable response: %w", err) } lines := []string{groupRoleLine(out.Region, out.Role)} for _, b := range out.Boxes { lines = append(lines, boxRoleLine(b, out.Role)) } return lines, nil }

// roleArgs reads role <name> <role> with an optional --console flag // after them, and exits with the usage on anything else. func roleArgs(rest []string) (name, role, console string) { if len(rest) < 3 || rest[0] != "role" { systemUsage() os.Exit(2) } name, role = rest[1], rest[2] tail := rest[3:] switch { case len(tail) == 0: case len(tail) == 2 && (tail[0] == "--console" || tail[0] == "-console"): console = tail[1] default: systemUsage() os.Exit(2) } return name, role, console }

func runSystemBoxVerb(rest []string) { box, role, consoleFlag := roleArgs(rest) sess := requireDeployAPISession(consoleFlag) printUsing(stdout, sess) line, err := setBoxRole(context.Background(), sess.postJSON, box, role) if err != nil { log.Fatalf("carlos system box role: %v", err) } fmt.Fprintln(stdout, line) }

func runSystemGroupVerb(rest []string) { region, role, consoleFlag := roleArgs(rest) sess := requireDeployAPISession(consoleFlag) printUsing(stdout, sess) lines, err := setGroupRole(context.Background(), sess.postJSON, region, role) if err != nil { log.Fatalf("carlos system group role: %v", err) } for _, l := range lines { fmt.Fprintln(stdout, l) } } ````

````diff --- a/system.go +++ b/system.go @@ -49,7 +49,7 @@ // untouched in rest for runSystemAntivirusVerb to read. // layout is a family too: migrate|revert|status stay in rest for // runSystemLayout.

• case "guard-arm", "guard-exec", "journal-migrate", "floor-allows-prev", "update", "status", "rollback", "cancel-activation", "apply", "antivirus", "layout", "geoip", "seeds-floor", "legacy-routes", "help", "-h", "--help": + case "guard-arm", "guard-exec", "journal-migrate", "floor-allows-prev", "update", "status", "rollback", "cancel-activation", "apply", "antivirus", "layout", "geoip", "seeds-floor", "legacy-routes", "box", "group", "help", "-h", "--help": return args[0], args[1:], component } return "", args, component @@ -88,6 +88,10 @@ runSystemSeedsFloorVerb(rest) case "legacy-routes": runSystemLegacyRoutesVerb(rest) + case "box": + runSystemBoxVerb(rest) + case "group": + runSystemGroupVerb(rest) case "help", "-h", "--help": systemUsage() default: @@ -122,6 +126,8 @@ carlos system geoip <publish|status|off> [flags] carlos system seeds-floor publish --min-release <release> carlos system legacy-routes assign --name <name> --to <account>/<app> + carlos system box role <box> <default|legacy|platform> + carlos system group role <region> <default|legacy>

The platform's own binary (ops account owners and CARLOSSYSTEMOPERATORS):

@@ -138,6 +144,12 @@ the first release whose boxes can place a seed, such as v0.27.0 legacy-routes assign --name <name> --to <account>/<app> +systemLegacyRoutesVerb+ + box role <box> <default|legacy|platform> + whether new instances may be placed on a box. A legacy or + platform box keeps the instances it has and takes no new ones + group role <region> <default|legacy> + the same for every box a region's group has enrolled, and for + every box it enrolls later

Deployment-wide settings (CARLOSSTOREOPERATORS):

````

```diff --- a/docs/site/cli.md +++ b/docs/site/cli.md @@ -2200,6 +2200,17 @@ carlos system legacy-routes assign --name shop --to bac/shop ``

+#### Box and group roles + +carlos system box role says whether new instances may be placed on a box. A box with the default role can take them. A legacy or platform box keeps the instances it has and takes no new ones: no new instance, relocation or move lands on it. carlos system group role sets the role of every box a region's group has enrolled, and of every box it enrolls later. Both work only on a console with region HA on. + +``sh +carlos system box role i-0123456789abcdef0 legacy +carlos system group role ie default +` + +Ops account owners and CARLOSSYSTEMOPERATORS`. + ### carlos move

Moves one instance to another region. The instance keeps serving while its ````

• [ ] Step 4: Run the tests

nice -n 19 go test ./internal/console/ -run 'BoxRoleDoor|GroupRoleDoor|RoleWriteErrors|RoleDoorsAreRegistered|EveryCLIDoor|ClaimConsoleRouteCensus|NoRoleReaches' -count=1
nice -n 19 go test ./internal/console/ -run 'TestClaimConsoleMutationInvariant$/probes/route_POST_/api/cli/system/' -count=1 -v | grep -E -- '--- (PASS|FAIL).*role'
nice -n 19 go test . -run 'SystemVerb|SetBoxRole|SetGroupRole' -count=1
nice -n 19 go test ./internal/docsite/ -count=1

Expected: PASS, and both role probes PASS. The second command runs only the probes, so its parent test reports app-scoped writers "reached by no door's open run": that is the filter, not the change. The whole TestClaimConsoleMutationInvariant takes longer than the default ten-minute test timeout; run it once with -timeout 90m before handing off (it is part of CI's make test).

• [ ] Step 5: Copy gate, boundary list and census

````diff --- a/approvedregionhacopytest.go +++ b/approvedregionhacopytest.go @@ -36,7 +36,7 @@ // regionHACopyBuilt is every id whose text the code or docs carry. The // task that writes a row's text adds its id here in the same commit, once // Paul has approved the row. -var regionHACopyBuilt = []string{} +var regionHACopyBuilt = []string{"RHA-C1", "RHA-C2", "RHA-C3", "RHA-C4", "RHA-C5", "RHA-C6", "RHA-C7", "RHA-C8", "RHA-C9", "RHA-C10", "RHA-C11", "RHA-C12", "RHA-C13", "RHA-C14", "RHA-C15", "RHA-C16"}

func mustParseRegionHARecord(t *testing.T) []copyRow { t.Helper() ````

Run: nice -n 19 go test . -run 'RegionHACopy|RegionHABuilt' -count=1 Expected: PASS once RHA-C1 to RHA-C16 are approved. While any is pending, TestRegionHABuiltCopyIsApproved fails naming it, and the task is not done.

````diff --- a/regionhaboundarytest.go +++ b/regionhaboundarytest.go @@ -25,6 +25,8 @@ "internal/host/boxaddress.go", "internal/edgesfile/private.go", "internal/edge/lbhealth.go", + "internal/console/apiboxroles.go", + "system_roles.go", }

// regionHAAdapterFiles are spec 2's AWS adapters. Each is named for AWS so ````

registerCLIRoutes is in the record-write census (a key site). Its digest changes:

````diff --- a/internal/provision/plan9b8censustest.go +++ b/internal/provision/plan9b8censustest.go @@ -680,7 +680,7 @@ // Registers instance and sidecar HTTP paths; record handlers use the reviewed provision writers. // Re-reviewed at the merge of main into carlos-run (2026-10-13): main's routes (the retry door among them) with the five run doors registered beside them; record writes stay in the reviewed handlers. // Re-reviewed at the merge of carlos-run into seeded-instances (2026-10-13): main's routes and the run doors with the four import doors, the seeds-floor door and the legacy-routes door registered beside them; record writes stay in the reviewed handlers.

• {"internal/console/authcli.go", "Server.registerCLIRoutes", "key-site", "lit:instances,lit:sidecars", "bb32a85461c3253b2b20f4191ae5fb1f2aa26e5e6c038bc338b6d16a14e7301a"}, + {"internal/console/authcli.go", "Server.registerCLIRoutes", "key-site", "lit:instances,lit:sidecars", "b55987735c833b952a7ef835302dba728c803379ecb474c3d465bb268cf4c781"}, {"internal/console/browserrestart.go", "Server.handleBrowserRestart", "key-site", "lit:instances", "955d71813504f2cfec1e1f64817d5aa5a921aa38e1044c0e903d2cfbe4d073f4"}, // Builds the instance page path and picks the confirmation line (a declined box's own line); the only write is the reviewed retry door (requestHostRetry), no record write here. {"internal/console/browserretry.go", "Server.handleInstanceRetry", "key-site", "lit:instances", "4d85d392f3660373f3c282c6391f58d8e25b6df738257ccc067b816e24dfe7ff"}, ````

The Server.boxRoster rows are census ruling region-ha-2a's (platform-1e). If the ruling's own basis text differs from the row's, use it, then recompute the fingerprint with legacyrow.py and the seal.

Add this comment line after the overlay's last comment line:

````text

Task 10: the operator's box and group role doors (internal/console apiboxroles.go, registered with host claims on in registerCLIRoutes) and the CLI (system_roles.go). Census ruling region-ha-2a (platform-1e, approved with conditions) covers one accepted-legacy unit, Server.boxRoster: the role doors' read of the signed roster (Directory.Read), operator-only (it is reached only past apiSystemRoller), read-only, the same roster every reader reads, and no role it reads feeds root's claim authority. Its basis is main's for a console root input (J4). The seal moves.

````

Then set these rows. A @coverage row replaces the overlay's row for the same directory, or is appended when there is none; every other row is appended:

```text @coverage internal/console bffe4b4c505d98b077d2e5e7906c32615ed62efb369da70eb18b4cc7084a1882 @coverage . 011b3548e5615fa1bb046e5a6d0b0e4345206ac7774c571a8320cfd124ba9af3 @legacy b02b534b6617f57381ba12e9d9a2bf1bd63f737dc68005de926ff18b7db93dd3 internal/console/apiboxroles.go Server.boxRoster legacy accepted-legacy accepted-legacy;rev3=J4;basis=root-input;census-ruling-region-ha-2a file:Read@484 9be9925de14dbbd7d86123bf9faf1535c7b116575cadcb8ba983a665af0513f4 internal/console/apiboxroles.go Server.boxRoster legacy accepted-legacy accepted-legacy;rev3=J4;basis=root-input;census-ruling-region-ha-2a file:Read@484 9be9925de14dbbd7d86123bf9faf1535c7b116575cadcb8ba983a665af0513f4 ```

````diff --- a/internal/provision/plan14rootreadstest.go +++ b/internal/provision/plan14rootreadstest.go @@ -769,7 +769,7 @@ if err = loadRootReadInventory(filepath.Join(root, "internal/provision/testdata/plan14-root-reads-region-ha-2a.tsv"), &inv, true); err != nil { t.Fatal(err) }

• seal = "fc38db001ebd8825709a8ca9c41cc0e8f165c1b1967e5967347dc9bd3556afec" + seal = "a57250c350c20e4116e5c87b559cfda71f3e0c65d59a2858d83c5f78bea4e9b5" errs := append(compareRootReads(got, inv), checkRootReadLegacy(inv, seal)...) errs = append(errs, checkRootReadDeclarations(got, inv)...) if len(errs) > 0 { ````

Run the census checks and go test . -run 'TestRegionHA(Core|Adapters|S3)' -count=1. Expected: PASS.

• [ ] Step 6: Commit

git add internal/console/apiboxroles.go internal/console/apiboxroles_test.go internal/console/authcli.go internal/console/svcroutes_test.go internal/console/claim_invariant_fixtures_test.go internal/console/claim_invariant_probes_nonapp_test.go internal/console/testdata/claim-routes.txt system.go system_roles.go system_roles_test.go docs/site/cli.md approved_region_ha_copy_test.go region_ha_boundary_test.go internal/provision
git commit -m "region HA 2a: box and group role doors and the carlos system verbs"

---

Task 11: The create doors choose first, and refuse when no box has room (H27)

Files:

• Create: internal/hostops/chosen.go (WithChosenExecutor, ChosenExecutor)
• Modify: internal/hostops/backfill.go (ProductionAuthority.SelectExecutor honours the door's choice)
• Modify: internal/hostops/executor_test.go
• Create: internal/console/nocapacity.go, internal/console/nocapacity_test.go
• Modify: internal/console/claim_create.go (beginClaimCreate calls chooseExecutorFirst before RegisterStart)
• Modify: internal/console/claim_response.go (claimResponseRefusal answers noCapacityError with a 503)
• Modify: internal/console/apiinstances.go (handleAPIInstanceCreate: the app-derived region check uses the chooser, the armed-placement gate asks roomIn first, and chooseCreateFirst runs before the environment binding)
• Modify: internal/console/server.go (Server.noCapacity)
• Modify: approved_region_ha_copy_test.go (wire RHA-C17 to RHA-C19), region_ha_boundary_test.go, internal/provision/plan9_b8_census_test.go, internal/provision/plan14_root_reads_test.go (the seal), the overlay

Interfaces:

• Consumes: Task 6's ExecutorChooser.PlaceableBoxes and Choose (through ProductionAuthority.SelectExecutor); hostops.ErrNoEligibleBox; alerting.Page; Server.homeRegionOut.
• Produces: func hostops.WithChosenExecutor(ctx, host string, sel hostclaim.ExecutorSelection) context.Context; func hostops.ChosenExecutor(ctx, host string) (hostclaim.ExecutorSelection, bool); type noCapacityError struct{ Region string } (errors.Is(err, hostops.ErrNoEligibleBox)); type noCapacityAlerts (due(region string, now time.Time) bool); func (s *Server) noRoom(ctx, region string) error; func (s *Server) placementChooser(pool string) *hostops.ExecutorChooser (Tasks 12 and 13 use it); func (s *Server) roomIn(ctx, region, pool string) error (Task 12 uses it); func (s *Server) chooseExecutorFirst(ctx, host string, before hostclaim.Observed, build func(time.Time, string) (hostops.Record, error)) (context.Context, error); func (s *Server) chooseCreateFirst(ctx, host string, build ...) (context.Context, error) (the returned context carries the choice and the claim observation it was decided on); var errChoiceStale (errors.Is(err, hostclaim.ErrConflict)); var pageNoCapacity = alerting.Page (the test seam).

Wait for the copy. RHA-C17 to RHA-C19 are the refusal and the alert. Start once Paul has approved them, and use his text.

H27 (spec rev 15), as Astra's findings ask (round 1: 1, 2, 3 and 6; round 2: 1 and 2):

• Choose before the first write. Every create through the claim starts in beginClaimCreate. With region HA on, chooseExecutorFirst builds the record the create would write, asks SelectExecutor for its executor, and returns before RegisterStart when there is none. Otherwise it puts the choice in the context, and SelectExecutor returns that choice to StartCreate (claim Plan 5's checks still run on it), so the create names exactly the box the door checked. The instance create door has a write before beginClaimCreate: bindEnvironmentCreate can clear a seed's mark on the environment's store (takeSeedStore). So that door calls chooseCreateFirst before the binding; it reads the host's claim and chooses the same way, and beginClaimCreate finds the choice in the context and keeps it. The domain attach and standalone sidecar doors write nothing before beginClaimCreate, and the seed doors refuse on a claim console (seedRegionReady).
• The observation travels with the choice (ruling item 5). chooseCreateFirst returns the choice together with the claim observation it decided on (the claim's ETag, or its absence). beginClaimCreate hands its own claim read to chooseExecutorFirst, which keeps the carried choice only when that read is the same observation. If the claim changed in between, the create is refused with errChoiceStale (hostclaim.ErrConflict: main's retryable busy answer) before anything is registered or written; the old choice is never used, and nothing is chosen again. Refusing is the smaller of the two options the ruling allows: it adds no copy and no second choice.
• No lease, no write, chosen once (ruling rev 2, item 3). The choice is SelectExecutor's read of the roster, the fences and the edges files. It takes no lease and writes nothing; a choice the context carries for the host is never chosen again; a refusal leaves no registration, no claim, no store change and nothing for a recovery pass.
• Every choosing door. One error, noCapacityError, answered in one place, claimResponseRefusal: instance create, domain attach's first route (the API's handleAPIDomainAttach and the browser's handleAppDomainAttach, both through attachClaimFirstRoute, which is unchanged), and standalone sidecar create. The move door's destination is Task 12. A sidecar attached to an instance chooses nothing (it lives on its instance's box) and is unchanged.
• The earlier region checks. handleAPIInstanceCreate judges an app-derived region (regionCandidates) by the chooser's eligible set instead, and asks roomIn before the armed-placement gate for an explicit region. roomIn only refuses: a region the roster places boxes in, with no placeable box (no edges file, no public address, stale, full or excluded), is the no-room refusal rather than a 400; a region the roster does not know is left to the gate.
• The alert limit. At most one alert per region an hour, remembered in the console process; a restart may send one more, which H27 accepts.

With region HA off nothing here runs: roomIn, chooseCreateFirst and chooseExecutorFirst return at once, and every create is main's.

Census (platform-1e): Server.beginClaimCreate is an accepted-legacy unit (J4 root input). It gains one call before RegisterStart; its one read site is unchanged. The overlay re-freezes it with the ruling's tag. Server.chooseCreateFirst is new: its one root read is the host's claim, the read beginClaimCreate makes, made before the binding and feeding only the placement choice. It returns its choice with that observation, and beginClaimCreate refuses when its own read differs (ruling item 5). The overlay asks for it as accepted-legacy (J4 root input), as approved. The seal moves. Server.attachClaimFirstRoute does not change. The record-write census digest of handleAPIInstanceCreate changes; no write site is added or changed.

• [ ] Step 1: Write the failing tests

internal/console/nocapacity_test.go:

````go package console

import ( "context" "encoding/json" "errors" "fmt" "net/http" "strings" "testing" "time"

"github.com/carlosframework/platform/internal/boxchoice" "github.com/carlosframework/platform/internal/hostclaim" "github.com/carlosframework/platform/internal/hostops" "github.com/carlosframework/platform/internal/objstore" "github.com/carlosframework/platform/internal/provision" "github.com/carlosframework/platform/internal/status" )

func TestNoCapacityAlertsRepeatAfterAnHour(t testing.T) { var a noCapacityAlerts t0 := time.Date(2026, 10, 16, 12, 0, 0, 0, time.UTC) if !a.due("uk", t0) || a.due("uk", t0.Add(59time.Minute)) || !a.due("ie", t0) || !a.due("uk", t0.Add(time.Hour)) { t.Fatal("once per region an hour") } }

func TestClaimRefusalAnswersNoRoom(t *testing.T) { ref := claimResponseRefusal(fmt.Errorf("begin: %w", &noCapacityError{Region: "uk"})) if ref.Status != http.StatusServiceUnavailable || ref.Msg != noCapacityRefusal("uk") { t.Fatalf("refusal %+v", ref) } var err error = &noCapacityError{Region: "uk"} if !errors.Is(err, hostops.ErrNoEligibleBox) { t.Fatal("a no-room refusal is not ErrNoEligibleBox") } }

// pagesTo replaces the alert sender for the test and returns what it sent. func pagesTo(t *testing.T) chan string { t.Helper() pages := make(chan string, 16) old := pageNoCapacity pageNoCapacity = func( context.Context, level, subject, string) error { pages <- level + " " + subject return nil } t.Cleanup(func() { pageNoCapacity = old }) return pages }

func expectPages(t testing.T, pages chan string, want ...string) { t.Helper() for _, w := range want { select { case got := <-pages: if got != w { t.Fatalf("page %q, want %q", got, w) } case <-time.After(5 time.Second): t.Fatalf("no page %q", w) } } select { case got := <-pages: t.Fatalf("an extra page: %q", got) case <-time.After(100 * time.Millisecond): } }

func noRoomSubject(region string) string { subject, _ := noCapacityAlert(region) return "warning " + subject }

// haClaimCase is a host-claim console with region HA on: its create doors // choose through the region HA chooser over the edges files the test // publishes (publishHABox). home names the console's home region ("" for // none): a first route is placed in the home. func haClaimCase(t testing.T, home string) claimCase { t.Helper() cx := newClaimCaseWith(t, func(c *Config) { c.HomeRegion = home }) cx.srv.cfg.RegionHA = true a := cx.srv.cfg.ClaimAuthority a.Executors = cx.srv.executorChooser(a.Directory) return cx }

// publishHABox publishes (or republishes) a roster box's edges file and // status snapshot: placeable in ie unless mut changes it, and stale when // stale is set. func publishHABox(t testing.T, cx claimCase, box string, stale bool, mut func(map[string]any)) { t.Helper() ctx := context.Background() m := map[string]any{ "v": 1, "box": box, "boxid": box, "eip": "203.0.113.7", "region": "eu-west-1", "placement": "ie", "hibernationconfigured": true, "servessharedpool": true, "acceptsinbound": true, "secretskeyfp": cx.srv.expectedSecretsFP(), "capabilities": []string{"region-ha-1"}, "headroom": map[string]any{"memfreemib": 4000, "memtotalmib": 8000, "diskfreemib": 50000, "disktotalmib": 100000, "awake": 0}, "generation": time.Now().UTC().Format(time.RFC3339), "hosts": map[string]any{}, } if mut != nil { mut(m) } body, err := json.Marshal(m) if err != nil { t.Fatal(err) } key := "control/edges/" + box + ".json" , tag, err := cx.rel.Get(ctx, key) if err != nil { tag = "" } if _, err := cx.rel.Put(ctx, key, body, tag); err != nil { t.Fatal(err) } at := time.Now() if stale { at = at.Add(-time.Hour) } if err := status.WriteBox(ctx, cx.rel, status.Box{Label: box, GeneratedAt: at}); err != nil { t.Fatal(err) } }

// registrations counts account/app's registered operations. func registrations(t testing.T, cx claimCase, account, app string) int { t.Helper() a, err := hostops.ReadCanonicalAdmission(context.Background(), cx.rel, account, app) if err != nil { t.Fatal(err) } return len(a.Value.Registrations) }

// refusedBeforeRegistering checks what H27 promises of a refused create: // nothing registered, no claim, and nothing recovery writes afterwards. func refusedBeforeRegistering(t testing.T, cx claimCase, host string, registered int) { t.Helper() ctx := context.Background() if n := registrations(t, cx, "bab", "titogo"); n != registered { t.Fatalf("registrations %d after the refusal, want %d: the door registered before choosing", n, registered) } if obs, err := hostclaim.Read(ctx, cx.rel, host); err != nil || !obs.Absent { t.Fatalf("claim after the refusal: %+v %v", obs, err) } if err := cx.srv.claimRecoveryPass(ctx); err != nil { t.Fatalf("recovery: %v", err) } if obs, err := hostclaim.Read(ctx, cx.rel, host); err != nil || !obs.Absent { t.Fatalf("claim after recovery: %+v %v", obs, err) } if n := registrations(t, cx, "bab", "titogo"); n != registered { t.Fatalf("registrations %d after recovery, want %d", n, registered) } }

// wroteNothing checks that nothing reached the bucket or a backend since // mark: no lease, no registration, no record and no store write. func wroteNothing(t testing.T, cx claimCase, mark int) { t.Helper() if events := cx.rec.since(mark); len(events) != 0 { t.Fatalf("a refused request wrote %v", events) } }

// Region HA spec 2 §7 (H27): every create door that chooses a box refuses // with a 503 when none is eligible, before it registers anything, and the // operator hears of it once per region an hour. Every box here is stale. func TestCreateDoorsRefuseWithNoRoomBeforeRegistering(t testing.T) { t.Run("instance create", func(t testing.T) { pages := pagesTo(t) cx := haClaimCase(t, "") cx.instanceWorld(t) publishHABox(t, cx, "box-ie-tito", true, nil) before := registrations(t, cx, "bab", "titogo") mark := cx.rec.mark() for range 3 { // a retry loop: refused every time, paged once r := cx.apiReply(t, http.MethodPost, "/api/cli/apps/bab/titogo/instances", {"host":"acme.go.tito.io","region":"ie"}) if r.Status != http.StatusServiceUnavailable || !strings.Contains(r.Body, noCapacityRefusal("ie")) { t.Fatalf("create = %d %s", r.Status, r.Body) } } wroteNothing(t, cx, mark) refusedBeforeRegistering(t, cx, "acme.go.tito.io", before) expectPages(t, pages, noRoomSubject("ie")) }) t.Run("standalone sidecar create", func(t testing.T) { pages := pagesTo(t) cx := haClaimCase(t, "") cx.sidecarWorld(t) before := registrations(t, cx, "bab", "titogo") mark := cx.rec.mark() r := cx.apiReply(t, http.MethodPost, "/api/cli/apps/bab/titogo/sidecars", {"name":"bot","region":"ie"}) if r.Status != http.StatusServiceUnavailable || !strings.Contains(r.Body, noCapacityRefusal("ie")) { t.Fatalf("sidecar create = %d %s", r.Status, r.Body) } wroteNothing(t, cx, mark) refusedBeforeRegistering(t, cx, provision.SidecarHost("bot", "titogo", "bab"), before) expectPages(t, pages, noRoomSubject("ie")) }) t.Run("domain attach, a first route", func(t testing.T) { pages := pagesTo(t) cx := haClaimCase(t, "ie") shipVersion(t, cx.consoleFixture, "titogo", "v1") seedChannel(t, cx.consoleFixture, "titogo", "edge", "v1") before := registrations(t, cx, "bab", "titogo") mark := cx.rec.mark() r := cx.apiReply(t, http.MethodPost, "/api/cli/apps/bab/titogo/domains", {"host":"first.titogo-claim.example","kind":"static"}) if r.Status != http.StatusServiceUnavailable || !strings.Contains(r.Body, noCapacityRefusal("ie")) { t.Fatalf("attach = %d %s", r.Status, r.Body) } wroteNothing(t, cx, mark) refusedBeforeRegistering(t, cx, "first.titogo-claim.example", before) expectPages(t, pages, noRoomSubject("ie")) }) t.Run("browser domain attach, a first route", func(t *testing.T) { pages := pagesTo(t) cx := haClaimCase(t, "ie") shipVersion(t, cx.consoleFixture, "titogo", "v1") seedChannel(t, cx.consoleFixture, "titogo", "edge", "v1") before := registrations(t, cx, "bab", "titogo") mark := cx.rec.mark() r := cx.formReply(t, "/a/bab/apps/titogo/domains/attach", "host", "first.titogo-claim.example", "kind", "static") if r.Status != http.StatusServiceUnavailable || !strings.Contains(r.Body, noCapacityRefusal("ie")) { t.Fatalf("browser attach = %d %s", r.Status, r.Body) } wroteNothing(t, cx, mark) refusedBeforeRegistering(t, cx, "first.titogo-claim.example", before) expectPages(t, pages, noRoomSubject("ie")) }) }

// appDerivedWorld is titogo with no pool and no region allow-list, whose // app region is ie: a create that names no region takes the app's. func appDerivedWorld(t testing.T, cx claimCase) { t.Helper() m := titoManifest() m.Regions = nil m.Pool = "" enableInstances(t, cx.consoleFixture, "titogo", m) cx.srv.appRegionOf = func(int64, string) (string, string, error) { return "ie", "app", nil } }

// Region HA spec 2 §7 (H27, Astra 2a round 1 finding 3): with region HA // on, an app-derived region is judged by the chooser's eligible set. Every // box stale is the no-room refusal, alert included, never the region // refusal's 400. func TestAppRegionWithEveryBoxStaleIsNoRoom(t *testing.T) { pages := pagesTo(t) cx := haClaimCase(t, "") appDerivedWorld(t, cx) publishHABox(t, cx, "box-ie-default", true, nil) before := registrations(t, cx, "bab", "titogo") r := cx.apiReply(t, http.MethodPost, "/api/cli/apps/bab/titogo/instances", {"host":"acme.go.tito.io"}) if r.Status != http.StatusServiceUnavailable || !strings.Contains(r.Body, noCapacityRefusal("ie")) { t.Fatalf("create = %d %s", r.Status, r.Body) } refusedBeforeRegistering(t, cx, "acme.go.tito.io", before) expectPages(t, pages, noRoomSubject("ie")) }

// ... and a misconfigured sibling (a box in ie holding another secrets // key, outside the roster) no longer refuses a region whose eligible set // holds another box: the create goes ahead on that box. With region HA off // the sibling still refuses, as on main. func TestAppRegionWithAMisconfiguredSiblingStillCreates(t *testing.T) { cx := haClaimCase(t, "") appDerivedWorld(t, cx) publishHABox(t, cx, "box-ie-default", false, nil) publishHABox(t, cx, "box-ie-stray", false, func(m map[string]any) { m["secretskeyfp"] = "sha256:another-key" }) off := cx.srv.placementChooser("") cx.srv.cfg.RegionHA = false r := cx.apiReply(t, http.MethodPost, "/api/cli/apps/bab/titogo/instances", {"host":"other.go.tito.io"}) if r.Status != http.StatusBadRequest { t.Fatalf("region HA off: create = %d %s, want main's region refusal", r.Status, r.Body) } cx.srv.cfg.RegionHA = true if off == nil { t.Fatal("no chooser with region HA on") } r = cx.apiReply(t, http.MethodPost, "/api/cli/apps/bab/titogo/instances", {"host":"acme.go.tito.io"}) if r.Status != http.StatusCreated && r.Status != http.StatusAccepted { t.Fatalf("create = %d %s", r.Status, r.Body) } obs, err := hostclaim.Read(context.Background(), cx.rel, "acme.go.tito.io") if err != nil || obs.Absent { t.Fatalf("claim: %+v %v", obs, err) } if exec, chosen := obs.Claim.Op.Exec, obs.Claim.Op.Intent["selected_executor"]; exec != "box-ie-default" && chosen != "box-ie-default" { t.Fatalf("executor %q / %q, want box-ie-default", exec, chosen) } }

// putSeedStore gives titogo's canary environment a store a seed made, and // returns the store record's bytes. func putSeedStore(t testing.T, cx claimCase) (key string, body []byte) { t.Helper() ctx := context.Background() at := time.Now().UTC() if , err := objstore.PutIn(ctx, cx.rel, "bab", "titogo", "canary", &objstore.Record{ Declared: &objstore.Declared{EnvPrefix: "S3", At: at, By: "bob@keymail.dev", Seed: newSeedID(t)}, Grant: &objstore.GrantBlock{Backend: "mem", Bucket: "carlos-bab-titogo-canary", Source: objstore.GrantSourceAccountApproval, At: at, By: "operator@keymail.dev"}, }, ""); err != nil { t.Fatal(err) } key = objstore.KeyIn("bab", "titogo", "canary") body, , err := cx.rel.Get(ctx, key) if err != nil { t.Fatal(err) } return key, body }

// Astra 2a round 2 finding 2: the instance create door chooses before its // first write. A create naming an environment whose store a seed made would // take that store (bindEnvironmentCreate). Refused for want of room, it // leaves the store's bytes as they were and writes nothing at all: when // every box is stale (the region check refuses), and when the region's last // placeable box goes between the region check and the choice (the early // choice refuses, before the binding). With room, the same create takes the // store, so the binding is on the path. func TestRefusedCreateLeavesTheEnvironmentsStoreAlone(t testing.T) { const body = {"host":"acme.go.tito.io","region":"ie","environment":"canary"} refused := func(t testing.T, cx claimCase, pages chan string) { t.Helper() key, stored := putSeedStore(t, cx) before := registrations(t, cx, "bab", "titogo") mark := cx.rec.mark() r := cx.apiReply(t, http.MethodPost, "/api/cli/apps/bab/titogo/instances", body) if r.Status != http.StatusServiceUnavailable || !strings.Contains(r.Body, noCapacityRefusal("ie")) { t.Fatalf("create = %d %s", r.Status, r.Body) } wroteNothing(t, cx, mark) if now, _, err := cx.rel.Get(context.Background(), key); err != nil || string(now) != string(stored) { t.Fatalf("the store after the refusal:\n%s\nwant\n%s (%v)", now, stored, err) } refusedBeforeRegistering(t, cx, "acme.go.tito.io", before) expectPages(t, pages, noRoomSubject("ie")) } t.Run("every box stale", func(t testing.T) { pages := pagesTo(t) cx := haClaimCase(t, "") cx.instanceWorld(t) publishHABox(t, cx, "box-ie-tito", true, nil) refused(t, cx, pages)

publishHABox(t, cx, "box-ie-tito", false, nil) r := cx.apiReply(t, http.MethodPost, "/api/cli/apps/bab/titogo/instances", body) if r.Status != http.StatusCreated && r.Status != http.StatusAccepted { t.Fatalf("create with room = %d %s", r.Status, r.Body) } store, _, err := objstore.GetIn(context.Background(), cx.rel, "bab", "titogo", "canary") if err != nil || store == nil || store.Declared == nil || store.Declared.Seed != "" { t.Fatalf("the store after a create with room: %+v %v, want the seed's mark cleared", store, err) } }) t.Run("the last box goes before the choice", func(t *testing.T) { pages := pagesTo(t) cx := haClaimCase(t, "") cx.instanceWorld(t) publishHABox(t, cx, "box-ie-tito", false, nil) x := cx.srv.cfg.ClaimAuthority.Executors edges, reads := x.Edges, 0 x.Edges = func(ctx context.Context) (boxchoice.Snapshot, error) { if reads++; reads > 1 { return boxchoice.Snapshot{Complete: true}, nil } return edges(ctx) } refused(t, cx, pages) }) }

// Astra 2a round 2 finding 1, at the create door: with region HA on, a // region whose boxes publish no public address (or no edges file) is a // region with no room, not the armed-placement gate's 400. With region HA // off the gate answers, as on main. func TestCreateRegionWithNoReachableBoxIsNoRoom(t *testing.T) { const body = {"host":"acme.go.tito.io","region":"ie"} pages := pagesTo(t) cx := haClaimCase(t, "") cx.instanceWorld(t) publishHABox(t, cx, "box-ie-tito", false, func(m map[string]any) { m["eip"] = "" }) cx.srv.cfg.RegionHA = false if r := cx.apiReply(t, http.MethodPost, "/api/cli/apps/bab/titogo/instances", body); r.Status != http.StatusBadRequest { t.Fatalf("region HA off: create = %d %s, want main's 400", r.Status, r.Body) } cx.srv.cfg.RegionHA = true before := registrations(t, cx, "bab", "titogo") mark := cx.rec.mark() r := cx.apiReply(t, http.MethodPost, "/api/cli/apps/bab/titogo/instances", body) if r.Status != http.StatusServiceUnavailable || !strings.Contains(r.Body, noCapacityRefusal("ie")) { t.Fatalf("create = %d %s", r.Status, r.Body) } wroteNothing(t, cx, mark) refusedBeforeRegistering(t, cx, "acme.go.tito.io", before) expectPages(t, pages, noRoomSubject("ie")) }

// Census ruling region-ha-2a rev 2, item 3: the early choice takes no lease // and writes nothing, and its executor is the one the authority chooses on // its own (the create's choice before the early choice existed). A choice // the context already carries is kept: nothing is chosen again. func TestChooseFirstWritesNothingAndChoosesOnce(t *testing.T) { cx := haClaimCase(t, "") appDerivedWorld(t, cx) publishHABox(t, cx, "box-ie-default", false, nil) ctx := context.Background() host := "acme.go.tito.io" build := func(now time.Time, acquisition string) (hostops.Record, error) { rec := provision.Record{Schema: 1, Host: host, App: "titogo", Account: "bab", Channel: "stable", Kind: "static", CreatedBy: "bob@keymail.dev", Acquisition: acquisition} return hostops.PrepareInstance(rec, now, cx.srv.cfg.claimPlacementRegion("ie")) } probe, err := build(time.Now().UTC(), "") if err != nil { t.Fatal(err) } before, err := hostclaim.Read(ctx, cx.rel, host) if err != nil { t.Fatal(err) } want, chosen, err := cx.srv.cfg.ClaimAuthority.SelectExecutor(ctx, before, probe) if err != nil || !chosen || want.Box != "box-ie-default" { t.Fatalf("the authority's own choice: %+v %v %v", want, chosen, err) }

mark := cx.rec.mark() got, err := cx.srv.chooseCreateFirst(ctx, host, build) if err != nil { t.Fatal(err) } wroteNothing(t, cx, mark) if sel, ok := hostops.ChosenExecutor(got, host); !ok || sel != want { t.Fatalf("early choice %+v (%v), want the authority's %+v", sel, ok, want) }

again, err := cx.srv.chooseExecutorFirst(got, host, before, func(time.Time, string) (hostops.Record, error) { t.Fatal("chose again") return hostops.Record{}, nil }) if err != nil || again != got { t.Fatalf("a carried choice was not kept: %v", err) } if sel, chosen, err := cx.srv.cfg.ClaimAuthority.SelectExecutor(got, before, probe); err != nil || !chosen || sel != want { t.Fatalf("the create's selection %+v %v %v, want the door's %+v", sel, chosen, err, want) } }

// Census ruling region-ha-2a item 5 (amadan #236): the early choice // carries the claim observation it was decided on, and beginClaimCreate // keeps it only if its own read of the claim is that observation. A claim // write landing between the choice and begin refuses the create with // main's retryable busy answer: the old choice is never used, nothing is // chosen again, and nothing is registered or written. func TestAStaleEarlyChoiceIsNeverUsed(t *testing.T) { cx := haClaimCase(t, "") cx.instanceWorld(t) publishHABox(t, cx, "box-ie-tito", false, nil) ctx := context.Background() const host = "acme.go.tito.io" acct, err := cx.st.AccountBySqid("bab") if err != nil || acct == nil { t.Fatalf("account: %+v %v", acct, err) } build := func(now time.Time, acquisition string) (hostops.Record, error) { rec := provision.Record{Schema: 1, Host: host, App: "titogo", Account: "bab", Channel: "stable", Kind: "static", Pool: "tito", Region: "ie", CreatedBy: "bob@keymail.dev", Acquisition: acquisition} return hostops.PrepareInstance(rec, now, cx.srv.cfg.claimPlacementRegion("ie")) } chosen, err := cx.srv.chooseCreateFirst(ctx, host, build) if _, ok := hostops.ChosenExecutor(chosen, host); err != nil || !ok { t.Fatalf("early choice: %v (chosen %v)", err, ok) }

// A claim write lands between the choice and begin. created := time.Now().UTC().Truncate(time.Second) inc, err := hostclaim.Incarnation("bab", "titogo", host, created) if err != nil { t.Fatal(err) } writeServingClaim(t, cx.rel, host, inc, created, "box-ie-tito", "ie") written, , err := cx.rel.Get(ctx, "control/hosts/"+host+".json") if err != nil { t.Fatal(err) } before := registrations(t, cx, "bab", "titogo") mark := cx.rec.mark() // The build runs only to choose again or, past RegisterStart, to write // the record: either is the old choice's create going on. req, err := cx.srv.beginClaimCreate(chosen, acct, "titogo", host, "bob@keymail.dev", func(time.Time, string) (hostops.Record, error) { t.Fatal("begin went on after the claim changed") return hostops.Record{}, nil }) if req != nil { req.Close() } if !errors.Is(err, errChoiceStale) || !errors.Is(err, hostclaim.ErrConflict) { t.Fatalf("begin on a changed claim = %v, want the stale-choice refusal", err) } if ref := claimResponseRefusal(err); ref.Status != http.StatusConflict || ref.Msg != claimRetireBusyMsg { t.Fatalf("refusal %+v, want main's retryable busy answer", ref) } wroteNothing(t, cx, mark) if n := registrations(t, cx, "bab", "titogo"); n != before { t.Fatalf("registrations %d, want %d", n, before) } if now, , err := cx.rel.Get(ctx, "control/hosts/"+host+".json"); err != nil || string(now) != string(written) { t.Fatalf("the claim changed under the refusal: %v", err) } } ````

```diff --- a/internal/hostops/executor_test.go +++ b/internal/hostops/executor_test.go @@ -491,3 +491,31 @@ t.Fatalf("unbound: %v", err) } } + +// Region HA spec 2 §7 (H27): a door that chose before registering holds +// the create to its choice, for that host only; Plan 5's scope still +// binds it (TestProductionCreateSelectsThroughTheChooser). +func TestSelectExecutorHonoursTheDoorsChoice(t *testing.T) { + ctx := context.Background() + x, _, snap, _ := chooserFixture(t, "box-a", "box-b") + a := ProductionAuthority{Namespace: "platform", Executors: &x} + r := chooserRecord() + absent := hostclaim.Observed{Host: r.Holder.Host, Absent: true} + chosen := hostclaim.ExecutorSelection{Box: "box-b", Enrollment: strings.Repeat("b", 32)} + // Every box is now ineligible, yet the door's earlier choice stands. + for i := range snap.Boxes { + snap.Boxes[i].Draining = true + } + sel, ok, err := a.SelectExecutor(WithChosenExecutor(ctx, r.Holder.Host, chosen), absent, r) + if err != nil || !ok || sel != chosen { + t.Fatalf("with the door's choice: %+v %v %v", sel, ok, err) + } + // Another host's choice is not this create's. + if _, _, err := a.SelectExecutor(WithChosenExecutor(ctx, "other.bab.oncarlos.com", chosen), absent, r); !errors.Is(err, ErrNoEligibleBox) { + t.Fatalf("another host's choice was used: %v", err) + } + // No chooser (region HA off): no selection, whatever the context says. + if _, ok, err := (ProductionAuthority{}).SelectExecutor(WithChosenExecutor(ctx, r.Holder.Host, chosen), absent, r); ok || err != nil { + t.Fatalf("region HA off selected: %v %v", ok, err) + } +} ```

• [ ] Step 2: Run them to see them fail

Run: nice -n 19 go test ./internal/console/ -run 'NoCapacity|ClaimRefusalAnswersNoRoom|CreateDoorsRefuse|AppRegionWith|RefusedCreateLeaves|ChooseFirstWrites|CreateRegionWithNoReachable|AStaleEarlyChoice' -count=1 Expected: FAIL to build (undefined: pageNoCapacity, undefined: noCapacityError). The hostops test fails to build on undefined: WithChosenExecutor.

• [ ] Step 3: Write the early choice and the refusal

internal/hostops/chosen.go:

````go package hostops

import ( "context"

"github.com/carlosframework/platform/internal/hostclaim" )

// A door that chooses its executor before it registers anything (region HA // spec 2 §7, H27) carries the choice to the create through the context, so // the create names exactly the box the door checked, and a refusal leaves // nothing registered, nothing written to the claim and nothing for // recovery to settle. selectExecutor still checks the choice against // claim Plan 5's scope (namespace, region, pool, active, unfenced).

type chosenExecutorKey struct{}

type chosenExecutor struct { host string sel hostclaim.ExecutorSelection }

// WithChosenExecutor records sel as host's executor for the create that // ctx carries. func WithChosenExecutor(ctx context.Context, host string, sel hostclaim.ExecutorSelection) context.Context { return context.WithValue(ctx, chosenExecutorKey{}, chosenExecutor{host: host, sel: sel}) }

// ChosenExecutor is the executor ctx carries for host, if any: a door that // already chose finds its choice here and does not choose again. func ChosenExecutor(ctx context.Context, host string) (hostclaim.ExecutorSelection, bool) { c, ok := ctx.Value(chosenExecutorKey{}).(chosenExecutor) if !ok || c.host == "" || c.host != host || c.sel.Box == "" || c.sel.Enrollment == "" { return hostclaim.ExecutorSelection{}, false } return c.sel, true } ````

```diff --- a/internal/hostops/backfill.go +++ b/internal/hostops/backfill.go @@ -107,9 +107,15 @@ // before the create CAS, when a chooser is configured; otherwise no // selection, and the create is today's. The chooser chooses within claim // Plan 5's scope: this authority's namespace and the record's pool. +// +// A door that already chose for this host (WithChosenExecutor) is held to +// that choice, so the create names the box the door checked (H27). func (a ProductionAuthority) SelectExecutor(ctx context.Context, obs hostclaim.Observed, r Record) (hostclaim.ExecutorSelection, bool, error) { if a.Executors == nil { return hostclaim.ExecutorSelection{}, false, nil + } + if sel, ok := ChosenExecutor(ctx, r.Holder.Host); ok { + return sel, true, nil } var shape struct { Pool string json:"pool" ```

internal/console/nocapacity.go:

````go package console

import ( "context" "errors" "fmt" "log" "sync" "time"

"github.com/carlosframework/platform/internal/alerting" "github.com/carlosframework/platform/internal/hostclaim" "github.com/carlosframework/platform/internal/hostops" )

// No box has room (region HA spec 2 §7, ruling H27). Every door that // chooses a box (instance create, domain attach's first route, standalone // sidecar create, and the move door's destination) refuses with a 503 when // the placement policy finds no eligible box, and the operator hears of it // at most once per region an hour per console process: a restart may send // one extra alert, which H27 accepts rather than persist the limit. A // create door chooses before it registers anything (chooseExecutorFirst), // so a refusal leaves nothing registered, nothing written to the claim and // nothing for recovery to settle.

// noCapacityAlertEvery is how often one region's alert may repeat. const noCapacityAlertEvery = time.Hour

func noCapacityRefusal(region string) string { return fmt.Sprintf("No box in %s has room right now. Try again in a few minutes.", region) }

func noCapacityAlert(region string) (subject, body string) { return fmt.Sprintf("No box in %s has room", region), fmt.Sprintf("A request in %s was refused because no box there can take it. Every box is full, stale or excluded by its role. Add a box, or free memory or disk on one, and check the eligible-capacity alarm for %s.", region, region) }

// noCapacityError is a request refused because no box in Region (its // display name) can take it. It is hostops.ErrNoEligibleBox, and // claimResponseRefusal answers it with noCapacityRefusal. type noCapacityError struct{ Region string }

func (e *noCapacityError) Error() string { return "no eligible box in " + e.Region + ": " + hostops.ErrNoEligibleBox.Error() }

func (e *noCapacityError) Is(target error) bool { return target == hostops.ErrNoEligibleBox }

// pageNoCapacity sends the alert; tests replace it. var pageNoCapacity = alerting.Page

// noCapacityAlerts remembers when each region's alert was last sent, in // this console process. The zero value is ready. type noCapacityAlerts struct { mu sync.Mutex last map[string]time.Time }

// due reports whether region's alert should be sent at now, and if so // records it as sent. func (a *noCapacityAlerts) due(region string, now time.Time) bool { a.mu.Lock() defer a.mu.Unlock() if at, ok := a.last[region]; ok && now.Sub(at) < noCapacityAlertEvery { return false } if a.last == nil { a.last = map[string]time.Time{} } a.last[region] = now return true }

// noRoom is the refusal for region (record vocabulary, "" for the home): // it logs, sends the operator alert when one is due, and returns the // error the door answers with. func (s Server) noRoom(ctx context.Context, region string) error { name := s.homeRegionOut(region) log.Printf("console: refused: no eligible box in %s", name) if s.noCapacity.due(name, time.Now()) { subject, body := noCapacityAlert(name) go func() { ctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 30time.Second) defer cancel() if err := pageNoCapacity(ctx, alerting.LevelWarning, subject, body); err != nil { log.Printf("console: no-capacity alert for %s: %v", name, err) } }() } return &noCapacityError{Region: name} }

// placementChooser is the create door's chooser scoped as // ProductionAuthority.SelectExecutor scopes it, for pool ("" the shared // pool), or nil with region HA off or no claim runtime. func (s Server) placementChooser(pool string) hostops.ExecutorChooser { a := s.cfg.ClaimAuthority if !s.cfg.RegionHA || a == nil || a.Executors == nil { return nil } x := *a.Executors x.Namespace, x.Pool = a.Namespace, pool return &x }

// roomIn is H27's question about region (record vocabulary) for pool, asked // with region HA on before a door's own placement checks (Astra 2a round 2 // finding 1): a region the roster places boxes in, whose placeable set is // empty (no edges file, no public address, stale, full or excluded by // role), is the no-room refusal, never a 400. It only refuses: it chooses // nothing and changes nothing the door goes on to do. It answers nil with // region HA off, for a region the roster does not place (the door's own // checks answer that), and when a box has room. func (s *Server) roomIn(ctx context.Context, region, pool string) error { x := s.placementChooser(pool) if x == nil { return nil } boxes, err := x.PlaceableBoxes(ctx, s.cfg.claimPlacementRegion(region)) switch { case errors.Is(err, hostops.ErrContract): return nil case err != nil: return err case len(boxes) == 0: return s.noRoom(ctx, region) } return nil }

// errChoiceStale is an early choice whose claim observation is no longer // the claim: a write landed between the choice and beginClaimCreate's own // read. It is hostclaim.ErrConflict, so the door answers main's retryable // busy refusal, and nothing was registered or written. var errChoiceStale = errors.New("console: the host's claim changed after its executor was chosen")

// chosenOnKey carries the claim observation an early choice was decided // on, beside the choice itself (hostops.WithChosenExecutor). type chosenOnKey struct{}

type chosenOn struct { host, etag string absent bool }

// chooseExecutorFirst is beginClaimCreate's first step with region HA on, // given beginClaimCreate's own read of the claim (before). It chooses the // executor for the record build would make, before anything is // registered, and carries the choice to the create // (hostops.WithChosenExecutor). It takes no lease and writes nothing: the // choice is SelectExecutor's read of the roster and the edges files. No // eligible box is noRoom's refusal. A choice ctx already carries for host // (chooseCreateFirst's) is kept only when before is the claim observation // it was decided on (census ruling region-ha-2a, item 5): otherwise the // create is refused with errChoiceStale, and the old choice is never used // and nothing is chosen again. With region HA off it returns ctx // unchanged, and the create is main's. func (s *Server) chooseExecutorFirst(ctx context.Context, host string, before hostclaim.Observed, build func(time.Time, string) (hostops.Record, error)) (context.Context, error) { a := s.cfg.ClaimAuthority if !s.cfg.RegionHA || a == nil || a.Executors == nil { return ctx, nil } if _, ok := hostops.ChosenExecutor(ctx, host); ok { on, ok := ctx.Value(chosenOnKey{}).(chosenOn) if !ok || on.host != host || on.etag != before.ETag || on.absent != before.Absent { return ctx, fmt.Errorf("%w: %w: %s", hostclaim.ErrConflict, errChoiceStale, host) } return ctx, nil } probe, err := build(time.Now().UTC(), "") if err != nil { return ctx, err } sel, chosen, err := a.SelectExecutor(ctx, before, probe) if errors.Is(err, hostops.ErrNoEligibleBox) { return ctx, s.noRoom(ctx, probe.Holder.RecordRegion) } if err != nil || !chosen { return ctx, err } return hostops.WithChosenExecutor(ctx, host, sel), nil }

// chooseCreateFirst is the instance create door's choice, made before the // door's first write (Astra 2a round 2 finding 2): the environment binding // (bindEnvironmentCreate) can move a seed's store to the new instance, so // a create refused for want of room must be refused before it. It reads // host's claim and chooses as chooseExecutorFirst does, and returns the // choice together with the claim observation it decided on (its ETag, or // its absence). beginClaimCreate keeps the choice only if its own read is // that same observation. With region HA off it returns ctx unchanged and // reads nothing. func (s *Server) chooseCreateFirst(ctx context.Context, host string, build func(time.Time, string) (hostops.Record, error)) (context.Context, error) { a := s.cfg.ClaimAuthority if !s.cfg.RegionHA || a == nil || a.Executors == nil { return ctx, nil } before, err := hostclaim.Read(ctx, s.cfg.Releases, host) if err != nil { return ctx, err } chosen, err := s.chooseExecutorFirst(ctx, host, before, build) if err != nil { return ctx, err } if _, ok := hostops.ChosenExecutor(chosen, host); !ok { return chosen, nil } return context.WithValue(chosen, chosenOnKey{}, chosenOn{host: host, etag: before.ETag, absent: before.Absent}), nil } ````

```diff --- a/internal/console/server.go +++ b/internal/console/server.go @@ -837,6 +837,10 @@ type Server struct { claimLossPageMu sync.Mutex claimLossPageDone chan struct{} + + // noCapacity limits the no-capacity alert to once per region an hour + // in this process (region HA spec 2 §7, H27, nocapacity.go). + noCapacity noCapacityAlerts // Enabled claim runtime (ConfigureClaimRuntime). Unset flag-off. claimRuntimeConfigured bool claimOwner hostops.RegistrationOwner ```

```diff --- a/internal/console/claim_create.go +++ b/internal/console/claim_create.go @@ -37,6 +37,12 @@ } before, err := hostclaim.Read(ctx, s.cfg.Releases, host) if err != nil { + return nil, err + } + // Region HA (spec 2 §7, H27): choose before registering anything. A + // door's earlier choice is kept only if this read is the claim it was + // decided on (census ruling region-ha-2a, item 5). + if ctx, err = s.chooseExecutorFirst(ctx, host, before, build); err != nil { return nil, err } appID, err := s.store.ClaimAppID(acct.ID, app) ```

````diff --- a/internal/console/claimresponse.go +++ b/internal/console/claimresponse.go @@ -15,6 +15,11 @@

func claimResponseRefusal(err error) doorRefusal { log.Printf("console: claim request refused: %v", err) + // No box has room (region HA spec 2 §7, H27): every create door's 503. + var full noCapacityError + if errors.As(err, &full) { + return &doorRefusal{http.StatusServiceUnavailable, noCapacityRefusal(full.Region)} + } if errors.Is(err, hostops.ErrPending) || errors.Is(err, hostclaim.ErrConflict) || errors.Is(err, hostclaim.ErrWrongState) || errors.Is(err, release.ErrCASConflict) { return &doorRefusal{http.StatusConflict, claimRetireBusyMsg} } ````

````diff --- a/internal/console/apiinstances.go +++ b/internal/console/apiinstances.go @@ -1259,6 +1259,25 @@ if c, ok := findCandidate(cands, region); ok { reason = c.unavailableReason() } + // Region HA (spec 2 §7, H27): the region is judged by the eligible + // set the create door's chooser will choose from, never by every + // sibling's own answer, and an empty set is the door's no-room + // refusal, alert included. + if x := s.placementChooser(""); x != nil { + switch boxes, err := x.PlaceableBoxes(ctx, s.cfg.claimPlacementRegion(region)); { + case errors.Is(err, hostops.ErrContract): + reason = "absent" + case err != nil: + log.Printf("console: api instances create %s/%s: region %s placeable boxes: %v", acct.Sqid, app, region, err) + jsonError(w, http.StatusServiceUnavailable, msgAppRegionUnconfirmed(s.homeRegionOut(region))) + return + case len(boxes) == 0: + writeClaimResponseError(w, s.noRoom(ctx, region)) + return + default: + reason = "" + } + } // Each list names only what an explicit request for this same // instance would pass (codex plan-R4 2): instanceChoices. if reason != "" { @@ -1331,6 +1350,12 @@ // provision.Record's own (DerivesPaths), asked of a probe record rather // than restated here, so it cannot drift from what reconcile will mint. if placed == "" && region != "" { + // Region HA (spec 2 §7, H27): a region with no box that has room is + // the no-room refusal, before this gate answers it as a 400. + if err := s.roomIn(ctx, region, m.Pool); err != nil { + writeClaimResponseError(w, err) + return + } probe := provision.Record{Kind: strings.TrimSpace(req.Kind), Addrs: req.Addrs, Role: probeRec.Role} caps, published := placements(ctx, s.cfg.Releases) // The replica placement rides the same gate as the record's own @@ -1568,25 +1593,36 @@ cancel() ctx, cancel = context.WithTimeout(r.Context(), instanceStoreTimeout) defer cancel() + buildClaim := func(now time.Time, acquisition string) (hostops.Record, error) { + rec.CreatedAt = time.Time{} + rec.Acquisition = acquisition + region := s.cfg.claimPlacementRegion(r