amadan

carlos / platform Public

Plan 2a for region HA spec 2: the provider-neutral core

Twelve tasks, dormant behind CARLOS_REGION_HA: the copy record and gate,
the load balancer health state and the provider-boundary tests, box and
group roles in the signed roster, the placement policy over the eligible
set, headroom totals, the chooser applying the policy to creates and
relocation targets, the box's own addresses and private_ip, private
forwarding within a region, the health answer on :80, the role doors and
carlos system verbs, the no-capacity refusal (ruling H27, replacing
pending creates), and the eligible-capacity metric.

Every task's code was written, built, vetted (linux, darwin; windows
build), tested and census-checked at 47b40803, task by task, then
removed. Tasks 7 and 10 ask platform-1e for census ruling region-ha-2a.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Paul Campbell pushed by claude f8bac60712d7c08aae16f3dab5ec86044d7155e6 parent 47b4080
1 file changed, +5533 −0
  • docs/superpowers/plans/2026-10-16-region-ha-2a-core.md +5533 −0
diff --git a/docs/superpowers/plans/2026-10-16-region-ha-2a-core.md b/docs/superpowers/plans/2026-10-16-region-ha-2a-core.md
new file mode 100644
index 000000000..fb9b0c93c
--- /dev/null
+++ b/docs/superpowers/plans/2026-10-16-region-ha-2a-core.md
@@ -0,0 +1,5533 @@
+# Region HA 2a: The Provider-Neutral Core Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Build region HA spec 2's provider-neutral core, dormant behind `CARLOS_REGION_HA`: box and group roles, the placement policy, the create door's refusal when no box has room, the load balancer health answer, private-address forwarding with an address source, and the eligible-capacity metric.
+
+**Architecture:** Every new rule sits on spec 1's code without changing what it proves. The placement policy runs over `boxchoice.Eligible`'s identity-filtered candidates inside `hostops.ExecutorChooser`, which creates and relocation targets already share; roles live in the signed roster. The health answer is an in-memory state (`internal/lbhealth`) that the agent's own loops feed and the edge reads on `:80`. Forwarding changes only the address an edge dials. Each AWS part is an adapter in a file named for AWS (`internal/host/awsaddr.go`, `internal/console/awsregionmetric.go`), and a test holds the core to that.
+
+**Tech Stack:** Go 1.26, module `github.com/carlosframework/platform`; `internal/hostactors`, `internal/boxchoice`, `internal/edgesfile`, `internal/hostops`, `internal/host`, `internal/edge`, `internal/activator`, `internal/console`, the `carlos` CLI (repo root); AWS SDK v2 CloudWatch in one adapter.
+
+**Spec:** `docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md` (rev 14, approved), with Paul's ruling **H27** (2026-10-16), which the spec's rev 15 will record: pending creates are dropped. Spec 1, for context: `docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md`.
+
+## Scope
+
+Spec 2 is four plans. This is **2a**. The others, and the seams 2a leaves them:
+
+| Plan | Builds | 2a's seam |
+|---|---|---|
+| 2b | Joining mode, the membership-confirmation worker, the console's enrolment pass | `lbhealth.Default.SetHosting` (Task 9): 2b's mode selection owns it, and joining mode leaves it false. `hostactors.Directory.EnrollAutomatic` already gives an admitted box its group's role in the enrolling write (Task 3) |
+| 2c | The `region-host` module's group mode, the start-up script, SSM | The health path `/.carlos/lb-health` on port 80 (Task 9). The metric `Carlos/Region EligibleBoxes`, dimension `Region` = the placement code, in each region's own CloudWatch (Task 12). 2c grants the platform box's role `cloudwatch:PutMetricData` conditioned on `cloudwatch:namespace = Carlos/Region`, sets `CARLOS_REGION_METRICS=cloudwatch` on the console, and opens 443 from the region's VPC CIDR for private forwarding |
+| 2d | The §8 single-box fixes | The move door's destination already skips legacy and platform boxes (Task 10); 2d's rework of the door (source from the claim, §7's full choice for the destination, its own sentence for "every box here is excluded") replaces Task 10's interim filter |
+
+**Ruling H27 (Paul, 2026-10-16).** Spec 2's §7 "Unplaced" section (pending creates, the `starting` state, a caller-supplied operation id, reconciliation, a shared reservation, a placement retry pass) is dropped. In its place: when the placement policy finds no eligible box, the create door refuses the create with a 503 and a clear message, and fires the operator alert at most once per region an hour. No executor-less create is written, so the claim is not touched. The eligible-capacity metric stays, published by a small periodic console pass (Task 12).
+
+**Not in 2a:** forwarding of platform routes from a group box to the platform box (§6 "Platform routes"), the region ingress list in the console (§6 "What the console tells customers"), and every row of §8; those are 2d's or 2c's. The sidecar and first-route create doors keep today's refusal when no box is eligible: only the instance create door gets H27's 503 (open question 2).
+
+## Global Constraints
+
+- Everything new is dormant with `CARLOS_REGION_HA` off: with it off, every door, file, edge and box behaves exactly as on main. The health answer is not served, the role doors answer 404, the edges file publishes no `private_ip` and keeps its EIP hold, the no-capacity refusal is inert, and the capacity pass does not start.
+- Multi-box regions stay AWS-only until #299 (H26). Spec 2's new core files import no `github.com/aws/...` package; AWS parts are adapters in files named for AWS (§25). `region_ha_boundary_test.go` holds both, plus `s3compat.Options()` on every S3 client in the packages 2a touches.
+- Wire shapes are additive: `private_ip`, `mem_total_mib`, `disk_total_mib` (edges file), `group` and `role` (roster box), `role` (roster region) are all `omitempty`, and a reader that predates them is unaffected.
+- The roster's default role is stored as `""`. Role values: box `""`, `legacy`, `platform`; group `""`, `legacy`. A role is never `platform` for a group.
+- §7's thresholds: free memory at least 15% of total (`boxchoice.MinMemPercent`), free disk at least 10% of the volume (`MinDiskPercent`), an edges file no older than 15 minutes (`MaxEdgesAge`). Choice: spread, then most `mem_free_mib`, then the P5-D2 hash.
+- §6's health answer: `GET` or `HEAD` `/.carlos/lb-health`, any host, port 80. 200 only when hosting, forwarding armed (map and key each loaded once, latched), the certificate mirror loaded once (latched), the edge's and the activator's beats within 60 seconds, and a home map refresh attempt completed within 5 minutes.
+- User-facing copy is in `docs/superpowers/plans/2026-10-16-region-ha-2a-copy.md`, gated by `approved_region_ha_copy_test.go`. No em dashes. No production console turns `CARLOS_REGION_HA` on before Paul approves every row.
+- Go commands: `export GOFLAGS=-buildvcs=false GOCACHE=$TMPDIR/gocache`. Run tests with `nice -n 19` and a `-run` filter; never the whole `internal/host` suite (CI's `make test` is the full run). Tests that bind unix sockets fail inside the sandbox ("socket: operation not permitted"); that is the sandbox, not the change.
+- Census: every task that changes a non-test Go file under a package directory re-pins that directory's `@coverage` row in `internal/provision/testdata/plan14-root-reads-region-ha-2a.tsv`. Tasks 7 and 10 need platform-1e (below).
+
+## Census rulings this plan asks platform-1e for
+
+- **Task 7** re-digests one authority row, `internal/host/edgesfile.go:edgesFileWriteClaims` (class authority, task F,J1, basis and sites unchanged). No seal moves.
+- **Task 10** adds one accepted-legacy unit, `internal/console/apiboxroles.go:Server.boxRoster` (the console's `Directory.Read` of the signed roster), with basis `accepted-legacy;rev3=J4;basis=root-input;census-ruling-region-ha-2a`. The seal moves from `fc38db00…` to `a57250c3…`.
+- Every other task is envelope-only. No legacy row changes.
+
+The census values below were computed at this branch's base, main `70e57449`. Main has moved since (`a89e35ca` adds overlays `plan14-root-reads-seed.tsv` and `plan14-root-reads-home-part2.tsv`, a new seal, and changes in `internal/host`, `internal/activator` and `internal/console`). On a newer base, keep the overlay last in the load order and recompute every value with the tooling below; the rows to write stay the same.
+
+## Census tooling
+
+Create these in `$TMPDIR/census` (never in the repo), and delete the directory when the plan is done.
+
+`coverage.sh` prints a directory's `@coverage` row exactly as `rootReadProduction` computes it (run from the repo root):
+
+````bash
+#!/bin/bash
+# usage: coverage.sh <dir> ... ; run from the repo root. Prints the Plan 14
+# @coverage line of each directory, as rootReadProduction computes it.
+for dir in "$@"; do
+ hash=$(for f in "$dir"/*; do
+ [ -f "$f" ] || continue
+ case "$f" in
+ *_test.go) continue;;
+ *.go|*.s|*.sh|*.service|*.path|*.socket|*.timer|*.conf|*.tf|*.tftpl|*.tmpl|*/go.mod|*/go.sum) ;;
+ *) continue;;
+ esac
+ printf '%s\0%s\n' "${f#./}" "$(sha256sum "$f" | cut -d' ' -f1)"
+ done | LC_ALL=C sort | head -c -1 | sha256sum | cut -d' ' -f1)
+ printf '@coverage\t%s\t%s\n' "$dir" "$hash"
+done
+````
+
+`unitdigest/main.go` prints a top-level unit's digest (the sha256 of its declaration's source), the value both censuses pin. Build it with `cd $TMPDIR/census/unitdigest && GOFLAGS= go build -o ../unitdigest main.go`, then run `$TMPDIR/census/unitdigest <file> <unit>` from the repo root:
+
+````go
+// unitdigest prints the census digest (sha256 of the declaration's source)
+// of each named top-level unit: unitdigest <file> <unit> [<unit> ...].
+// Unit names are the censuses': Func, Recv.Method, "type T", "var v".
+package main
+
+import (
+ "crypto/sha256"
+ "encoding/hex"
+ "fmt"
+ "go/ast"
+ "go/parser"
+ "go/token"
+ "os"
+)
+
+func recv(e ast.Expr) string {
+ for {
+ switch x := e.(type) {
+ case *ast.StarExpr:
+ e = x.X
+ case *ast.ParenExpr:
+ e = x.X
+ case *ast.IndexExpr:
+ e = x.X
+ case *ast.IndexListExpr:
+ e = x.X
+ case *ast.Ident:
+ return x.Name
+ default:
+ return "?"
+ }
+ }
+}
+
+func main() {
+ src, err := os.ReadFile(os.Args[1])
+ if err != nil {
+ panic(err)
+ }
+ fset := token.NewFileSet()
+ f, err := parser.ParseFile(fset, os.Args[1], src, 0)
+ if err != nil {
+ panic(err)
+ }
+ want := map[string]bool{}
+ for _, u := range os.Args[2:] {
+ want[u] = true
+ }
+ for _, d := range f.Decls {
+ name := ""
+ switch d := d.(type) {
+ case *ast.FuncDecl:
+ name = d.Name.Name
+ if d.Recv != nil {
+ name = recv(d.Recv.List[0].Type) + "." + name
+ }
+ case *ast.GenDecl:
+ if d.Tok == token.IMPORT || d.Lparen.IsValid() {
+ continue
+ }
+ switch s := d.Specs[0].(type) {
+ case *ast.ValueSpec:
+ name = d.Tok.String() + " " + s.Names[0].Name
+ case *ast.TypeSpec:
+ name = "type " + s.Name.Name
+ }
+ }
+ if want[name] {
+ sum := sha256.Sum256(src[fset.Position(d.Pos()).Offset:fset.Position(d.End()).Offset])
+ fmt.Printf("%s\t%s\t%s\n", os.Args[1], name, hex.EncodeToString(sum[:]))
+ }
+ }
+}
+````
+
+`legacyrow.py` prints the `@legacy` row and the entry row for a unit (`python3 legacyrow.py <file> <unit> <kinds> <digest> <basis>`):
+
+````python
+import hashlib, sys
+# legacyrow.py <file> <unit> <kinds> <digest> <basis>: prints the @legacy row and the entry row.
+f,u,k,d,b = sys.argv[1:6]
+fields=[f,u,"legacy","accepted-legacy",b,k,d]
+fp=hashlib.sha256("\t".join(fields).encode()).hexdigest()
+print("\t".join(["@legacy",fp]+fields))
+print("\t".join(fields))
+````
+
+A unit's kinds string (for an unclassified or changed root read) is printed by a scratch test. Create `internal/provision/zz_census_scratch_test.go` with this content, run it, and delete it by exact path before committing:
+
+````go
+package provision
+
+import (
+ "os"
+ "strings"
+ "testing"
+)
+
+// Scratch: prints the census scan of the named units (RHA2A_UNITS, comma
+// separated "file:unit"). Never committed.
+func TestZZCensusScratchUnits(t *testing.T) {
+ want := os.Getenv("RHA2A_UNITS")
+ if want == "" {
+ t.Skip()
+ }
+ got, err := rootReadProduction(censusRoot(t))
+ if err != nil {
+ t.Fatal(err)
+ }
+ for _, e := range got.Entries {
+ for _, w := range strings.Split(want, ",") {
+ if e.File+":"+e.Unit == w {
+ t.Logf("ENTRY\t%s\t%s\t%s\t%s", e.File, e.Unit, e.Kinds, e.Digest)
+ }
+ }
+ }
+}
+````
+
+````bash
+RHA2A_UNITS="internal/console/apiboxroles.go:Server.boxRoster" nice -n 19 go test ./internal/provision/ -run TestZZCensusScratchUnits -count=1 -v | grep ENTRY
+````
+
+The seal after a task's legacy rows is `rootReadLegacySeal` over every overlay in the test's load order. Add this to the scratch file to print it (the list is the load order at `70e57449`; on a newer base, copy the order from `TestPlan14RootReadCensus`):
+
+````go
+func TestZZCensusScratchSeal(t *testing.T) {
+ if os.Getenv("RHA2A_SEAL") == "" {
+ t.Skip()
+ }
+ root := censusRoot(t)
+ inv := rootReadInventory{Entries: map[string]classifiedRootRead{}, Coverage: map[string]string{}}
+ files := []string{"plan14-root-reads.tsv"}
+ if _, err := os.Stat(root + "/internal/host/movesheldrelease_production.go"); err == nil {
+ files = append(files, "plan14-root-reads-moves.tsv")
+ }
+ files = append(files, "plan14-root-reads-merge.tsv", "plan14-root-reads-fix1.tsv", "plan14-root-reads-fix2.tsv", "plan14-root-reads-merge3.tsv",
+ "plan14-root-reads-fix4.tsv", "plan14-root-reads-fix5.tsv", "plan14-root-reads-fix6.tsv", "plan14-root-reads-zddr3.tsv",
+ "plan14-root-reads-hotfix281.tsv", "plan14-root-reads-zddr4.tsv", "plan14-root-reads-reserved285.tsv", "plan14-root-reads-292.tsv",
+ "plan14-root-reads-census-ruling-275.tsv", "plan14-root-reads-census-ruling-owed.tsv", "plan14-root-reads-carlos-run.tsv",
+ "plan14-root-reads-seeded-instances.tsv", "plan14-root-reads-hostfiles-cap-236.tsv", "plan14-root-reads-moves-overcap-cut-236.tsv",
+ "plan14-root-reads-region-ha-2a.tsv")
+ for i, f := range files {
+ if err := loadRootReadInventory(root+"/internal/provision/testdata/"+f, &inv, i > 0); err != nil {
+ t.Fatal(f, err)
+ }
+ }
+ t.Logf("SEAL\t%s", rootReadLegacySeal(inv))
+}
+````
+
+````bash
+RHA2A_SEAL=1 nice -n 19 go test ./internal/provision/ -run TestZZCensusScratchSeal -count=1 -v | grep SEAL
+````
+
+The overlay is tab-separated: keep the tabs in every row this plan gives, or the loader refuses the row.
+
+The census checks every task runs:
+
+````bash
+nice -n 19 go test ./internal/provision/ -run 'TestPlan9B8RecordWriteCensus|TestPlan14RootReadCensus$' -count=1
+````
+
+## Review Focus
+
+1. **A roster write by another path must keep a box's role** (a quarantine, an activation, an enrolment): a box that silently drops back to `default` takes new instances the operator excluded. Task 3's `TestOtherRosterWritesKeepTheRole` pins the quarantine path; `Replace` compares every other field, so no write may change `Group`.
+2. **IMDS answering blank on one pass** must publish no public address rather than an old one, and the box then drops out of `Eligible` until the next pass, never forwarding to an address it no longer holds. Task 7's `TestEdgesFilePublishesOnlyWhatTheSourceReports`.
+3. **An edges file stamped by a box whose clock runs ahead** is believed for a minute of skew and no more; a stale or unreadable generation is never trusted. Task 4's `TestPlaceableAppliesEachRule` (the future and unreadable cases).
+4. **A bucket outage after a box has armed** must leave its health answer at 200 (no termination for a shared outage), while a refresh stuck inside one attempt turns it 503. Task 2's `TestBucketOutageAfterArmingStaysHealthy` and Task 9's `TestHomeMapFeedsTheHealthState`.
+5. **A client retrying a refused create in a loop** must not page the operator on every retry. Task 11's `TestCreateWithNoEligibleBoxIsRefusedAndAlertedOncePerHour`.
+
+## Dry run
+
+Every code block below was written into this branch at `47b40803` (main `70e57449`), built, vetted with `GOOS=linux` and `GOOS=darwin` (and built with `GOOS=windows`), and its tests run as each task's steps say, task by task in order. The census values are those runs'. The code was then removed.
+
+---
+
+### Task 1: The copy record and its gate
+
+**Files:**
+- Create: `docs/superpowers/plans/2026-10-16-region-ha-2a-copy.md`
+- Create: `approved_region_ha_copy_test.go`
+
+**Interfaces:**
+- Consumes: the moves gate's helpers at the repo root (`parseCopyTable`, `checkMovesInventory`, `checkMovesNoEmDash`, `checkRunApplied`, `reportAll`, `copyRow`).
+- Produces: `regionHACopyWired []string`, which Tasks 10 and 11 extend with the ids whose text they write; the record's ids `RHA-C1` to `RHA-C19`.
+
+Every user-facing string 2a writes is drafted here first, all `pending`. The gate requires a wired row's text to be in its file, whatever its status: a pending string may be wired because each is reached only with `CARLOS_REGION_HA` on.
+
+- [ ] **Step 1: Write the record**
+
+`docs/superpowers/plans/2026-10-16-region-ha-2a-copy.md`:
+
+````markdown
+# Region HA plan 2a: copy record (pending Paul's review)
+
+Every user-facing string plan 2a (`2026-10-16-region-ha-2a-core.md`)
+writes: console refusals, CLI output and usage lines, the operator alert,
+and the docs prose. The gate is `approved_region_ha_copy_test.go` at the
+repo root, which reuses the moves gate's parser, so the format is the
+moves record's: one table per heading, columns id, file, text, status.
+
+The rules:
+
+- A row is `pending` until Paul approves it in the copy review (the
+ `copy-review` skill). His edits are applied verbatim, here and in the
+ file the row names, in one commit, and the row becomes `approved`.
+- A `pending` string may be wired while region HA is dormant. Every
+ string here is reached only with `CARLOS_REGION_HA` on (the role doors
+ answer 404 without it, and the no-capacity refusal is inert), so no
+ production console turns region HA on before every row is `approved`.
+- A task that writes a row's text adds its id to `regionHACopyWired` in
+ the gate in the same commit; from then on the gate fails unless the
+ text is in the file the row names, whatever its status.
+- In a Go file the text is one string literal, or one `fmt.Sprintf`
+ format, equal to the row with `%s` or `%d` where the row has a
+ `<placeholder>`. A row whose file cell says "a fragment" is a piece of
+ a larger literal (the `carlos system` usage text).
+- No em dashes. A full stop and a new sentence instead. Console refusals
+ start lower case, as the console's other CLI refusals do.
+
+Placeholders: `<box>` a box's roster reference or label; `<role>` a role
+name (default, legacy or platform); `<region>` a placement code or the
+home region's name.
+
+## Box and group roles (Task 10)
+
+| id | file | text | status |
+|---|---|---|---|
+| RHA-C1 | internal/console/apiboxroles.go | role must be default, legacy or platform | pending |
+| RHA-C2 | internal/console/apiboxroles.go | a group's role must be default or legacy | pending |
+| RHA-C3 | internal/console/apiboxroles.go | could not read the box roster. Try again. | pending |
+| RHA-C4 | internal/console/apiboxroles.go | the box roster changed while the role was being saved, and nothing was changed. Try again. | pending |
+| RHA-C5 | internal/console/apiboxroles.go | could not save the role. Try again. | pending |
+| RHA-C6 | internal/console/apiboxroles.go | no enrolled box is named <box> | pending |
+| RHA-C7 | internal/console/apiboxroles.go | <region> has no group | pending |
+| RHA-C8 | internal/console/apiboxroles.go | <box> is retired, so it takes no role | pending |
+| RHA-C9 | system_roles.go | <box> now has the <role> role | pending |
+| RHA-C10 | system_roles.go | the group in <region> now has the <role> role | pending |
+| RHA-C11 | system.go (a fragment of the usage) | whether new instances may be placed on a box. A legacy or | pending |
+| RHA-C12 | system.go (a fragment of the usage) | platform box keeps the instances it has and takes no new ones | pending |
+| RHA-C13 | system.go (a fragment of the usage) | the same for every box a region's group has enrolled, and for | pending |
+| RHA-C14 | system.go (a fragment of the usage) | every box it enrolls later | pending |
+| RHA-C15 | docs/site/cli.md | Box and group roles | pending |
+| RHA-C16 | docs/site/cli.md | carlos system box role says whether new instances may be placed on a box. A box with the default role can take them. A legacy or platform box keeps the instances it has and takes no new ones: no new instance, relocation or move lands on it. carlos system group role sets the role of every box a region's group has enrolled, and of every box it enrolls later. Both work only on a console with region HA on. | pending |
+
+## A create no box can take (Task 11)
+
+| id | file | text | status |
+|---|---|---|---|
+| RHA-C17 | internal/console/nocapacity.go | no box in <region> can take a new instance right now. Try again in a few minutes. | pending |
+| RHA-C18 | internal/console/nocapacity.go (the operator alert's subject) | No room for new instances in <region> | pending |
+| RHA-C19 | internal/console/nocapacity.go (the operator alert's body) | A create in <region> was refused because no box there can take a new instance. Every box is full, stale or excluded by its role. Add a box, or free memory or disk on one, and check the eligible-capacity alarm for <region>. | pending |
+````
+
+- [ ] **Step 2: Write the gate**
+
+`approved_region_ha_copy_test.go`:
+
+````go
+// approved_region_ha_copy_test.go
+package main
+
+import (
+ "fmt"
+ "os"
+ "regexp"
+ "strconv"
+ "testing"
+)
+
+// The copy gate of region HA plan 2a (Task 1). The record is a markdown
+// table in the moves record's format, read with the moves gate's parser
+// and checked with its helpers:
+//
+// - TestRegionHACopyRecordIsComplete: the record names exactly RHA-C1 to
+// RHA-C19;
+// - TestRegionHACopyHasNoEmDash;
+// - TestRegionHAWiredCopyIsApplied: a wired row's text is in the file it
+// names, whatever its status. A pending row may be wired because every
+// one of these strings is reached only with CARLOS_REGION_HA on, and no
+// production console turns it on before Paul approves the record.
+const regionHACopyRecordPath = "docs/superpowers/plans/2026-10-16-region-ha-2a-copy.md"
+
+var regionHACopyIDPattern = regexp.MustCompile(`^RHA-C[0-9]+$`)
+
+// regionHACopyIDs is the record's inventory: RHA-C1 to RHA-C19, pinned.
+var regionHACopyIDs = func() []string {
+ ids := make([]string, 0, 19)
+ for i := 1; i <= 19; i++ {
+ ids = append(ids, "RHA-C"+strconv.Itoa(i))
+ }
+ return ids
+}()
+
+// regionHACopyWired is every id whose text the code or docs carry. The
+// task that writes a row's text adds its id here in the same commit.
+var regionHACopyWired = []string{}
+
+func mustParseRegionHARecord(t *testing.T) []copyRow {
+ t.Helper()
+ raw, err := os.ReadFile(regionHACopyRecordPath)
+ if err != nil {
+ t.Fatal(err)
+ }
+ rows, err := parseCopyTable(string(raw), regionHACopyRecordPath, regionHACopyIDPattern, "an RHA-C<n>")
+ if err != nil {
+ t.Fatal(err)
+ }
+ return rows
+}
+
+// regionHAWiredRows is the wired rows, each marked approved so the moves
+// gate's application check reads them: a wired pending row must be in its
+// file exactly as an approved one would be.
+func regionHAWiredRows(rows []copyRow, wired []string) ([]copyRow, []string) {
+ byID := map[string]copyRow{}
+ for _, r := range rows {
+ byID[r.ID] = r
+ }
+ var out []copyRow
+ var problems []string
+ seen := map[string]bool{}
+ for _, id := range wired {
+ if seen[id] {
+ problems = append(problems, fmt.Sprintf("regionHACopyWired names %s twice", id))
+ }
+ seen[id] = true
+ r, ok := byID[id]
+ if !ok {
+ problems = append(problems, fmt.Sprintf("regionHACopyWired names %s, which the record has no row for", id))
+ continue
+ }
+ r.Status = "approved"
+ out = append(out, r)
+ }
+ return out, problems
+}
+
+func TestRegionHACopyRecordIsComplete(t *testing.T) {
+ reportAll(t, checkMovesInventory(mustParseRegionHARecord(t), regionHACopyIDs))
+}
+
+func TestRegionHACopyHasNoEmDash(t *testing.T) {
+ reportAll(t, checkMovesNoEmDash(mustParseRegionHARecord(t)))
+}
+
+func TestRegionHAWiredCopyIsApplied(t *testing.T) {
+ rows, problems := regionHAWiredRows(mustParseRegionHARecord(t), regionHACopyWired)
+ reportAll(t, append(problems, checkRunApplied(rows, ".")...))
+}
+````
+
+- [ ] **Step 3: Run it**
+
+Run: `nice -n 19 go test . -run 'RegionHACopy|RegionHAWired' -count=1`
+Expected: PASS (no row is wired yet).
+
+- [ ] **Step 4: Put the record in front of Paul**
+
+Run the `copy-review` skill over the record's rows. Apply his edits verbatim to the record in its own commit, and carry them into the code as Tasks 10 and 11 wire each row. Implementation does not wait for the review: the rows are dormant until region HA is turned on.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add docs/superpowers/plans/2026-10-16-region-ha-2a-copy.md approved_region_ha_copy_test.go
+git commit -m "region HA 2a: the copy record and its gate"
+```
+
+No package envelope changes: neither file is a non-test Go file.
+
+---
+
+### Task 2: The health state (`internal/lbhealth`), the provider-boundary tests and the census overlay
+
+**Files:**
+- Create: `internal/lbhealth/lbhealth.go`, `internal/lbhealth/lbhealth_test.go`
+- Create: `region_ha_boundary_test.go`
+- Create: `internal/provision/testdata/plan14-root-reads-region-ha-2a.tsv`
+- Modify: `internal/provision/plan14_root_reads_test.go` (load the new overlay last)
+
+**Interfaces:**
+- Produces (`internal/lbhealth`): `const Path = "/.carlos/lb-health"`; `HeartbeatWindow = 60 * time.Second`; `RefreshWindow = 5 * time.Minute`; the condition names `CondHosting`, `CondForwarding`, `CondCerts`, `CondEdge`, `CondActivator`, `CondRefresh`; `type State`; `var Default = &State{}`; methods (all safe on a nil `*State`) `SetHosting(bool)`, `Hosting() bool`, `MapLoaded()`, `KeyLoaded()`, `CertsLoaded()`, `EdgeBeat(time.Time)`, `RefreshAttempted(time.Time)`, `SetActivator(func() time.Time)`, `Failing(now time.Time) []string`; `func Serve(w http.ResponseWriter, r *http.Request, s *State, now time.Time) bool`.
+- Produces (repo root): `regionHACoreFiles`, `regionHAAdapterFiles`, `regionHAPackages`, `regionHAS3Exceptions` in `region_ha_boundary_test.go`, which later tasks extend.
+- Produces: the overlay file every later task's census step edits, and the `seal = ...` line after it, which Task 10 changes.
+
+- [ ] **Step 1: Write the failing tests**
+
+`internal/lbhealth/lbhealth_test.go`:
+
+````go
+package lbhealth
+
+import (
+ "net/http"
+ "net/http/httptest"
+ "reflect"
+ "strings"
+ "testing"
+ "time"
+)
+
+var t0 = time.Date(2026, 10, 16, 12, 0, 0, 0, time.UTC)
+
+// healthy is a state with every condition met at t0.
+func healthy() *State {
+ s := &State{}
+ s.SetHosting(true)
+ s.MapLoaded()
+ s.KeyLoaded()
+ s.CertsLoaded()
+ s.EdgeBeat(t0)
+ s.RefreshAttempted(t0)
+ s.SetActivator(func() time.Time { return t0 })
+ return s
+}
+
+func TestFailingNamesEachCondition(t *testing.T) {
+ if got := healthy().Failing(t0); len(got) != 0 {
+ t.Fatalf("healthy box fails %v", got)
+ }
+ if got := (&State{}).Failing(t0); !reflect.DeepEqual(got, []string{CondHosting, CondForwarding, CondCerts, CondEdge, CondActivator, CondRefresh}) {
+ t.Fatalf("zero state fails %v", got)
+ }
+ var nilState *State
+ if got := nilState.Failing(t0); len(got) != 6 {
+ t.Fatalf("nil state fails %v", got)
+ }
+ cases := []struct {
+ name string
+ spoil func(*State)
+ now time.Time
+ want string
+ }{
+ {"joining", func(s *State) { s.SetHosting(false) }, t0, CondHosting},
+ {"map never loaded", func(s *State) { s.mapLoaded.Store(false) }, t0, CondForwarding},
+ {"key never loaded", func(s *State) { s.keyLoaded.Store(false) }, t0, CondForwarding},
+ {"no certificate cache", func(s *State) { s.certs.Store(false) }, t0, CondCerts},
+ {"edge beat stale", func(s *State) { s.EdgeBeat(t0.Add(-61 * time.Second)) }, t0, CondEdge},
+ {"activator beat stale", func(s *State) { s.SetActivator(func() time.Time { return t0.Add(-61 * time.Second) }) }, t0, CondActivator},
+ {"no activator", func(s *State) { s.SetActivator(nil) }, t0, CondActivator},
+ {"refresh stuck", func(s *State) { s.RefreshAttempted(t0.Add(-5*time.Minute - time.Second)) }, t0, CondRefresh},
+ }
+ for _, c := range cases {
+ t.Run(c.name, func(t *testing.T) {
+ s := healthy()
+ c.spoil(s)
+ if got := s.Failing(c.now); !reflect.DeepEqual(got, []string{c.want}) {
+ t.Fatalf("Failing = %v, want [%s]", got, c.want)
+ }
+ })
+ }
+ // The windows are inclusive: a beat exactly 60s old and a refresh
+ // exactly 5 minutes old still count.
+ s := healthy()
+ s.EdgeBeat(t0.Add(-HeartbeatWindow))
+ s.RefreshAttempted(t0.Add(-RefreshWindow))
+ if got := s.Failing(t0); len(got) != 0 {
+ t.Fatalf("boundary fails %v", got)
+ }
+}
+
+// A bucket outage after arming: refreshes keep failing, but each failed
+// attempt is still a completed attempt, and the latches hold. The box
+// stays healthy (A2-3, A3-3).
+func TestBucketOutageAfterArmingStaysHealthy(t *testing.T) {
+ s := healthy()
+ for i := 1; i <= 20; i++ {
+ now := t0.Add(time.Duration(i) * 30 * time.Second)
+ s.RefreshAttempted(now) // a failed attempt: no latch is re-set
+ s.EdgeBeat(now)
+ s.SetActivator(func() time.Time { return now })
+ if got := s.Failing(now); len(got) != 0 {
+ t.Fatalf("after %d failed refreshes: %v", i, got)
+ }
+ }
+}
+
+func TestServe(t *testing.T) {
+ get := func(s *State, method, target, host string) (*httptest.ResponseRecorder, bool) {
+ w := httptest.NewRecorder()
+ r := httptest.NewRequest(method, target, nil)
+ r.Host = host
+ return w, Serve(w, r, s, t0)
+ }
+ w, ok := get(healthy(), http.MethodGet, Path, "10.0.1.7")
+ if !ok || w.Code != http.StatusOK || w.Body.String() != "ok\n" || w.Header().Get("Cache-Control") != "no-store" {
+ t.Fatalf("healthy GET: ok=%v %d %q %v", ok, w.Code, w.Body.String(), w.Header())
+ }
+ w, ok = get(healthy(), http.MethodHead, Path, "any.example.com")
+ if !ok || w.Code != http.StatusOK || w.Body.Len() != 0 {
+ t.Fatalf("healthy HEAD: ok=%v %d %q", ok, w.Code, w.Body.String())
+ }
+ joining := healthy()
+ joining.SetHosting(false)
+ w, ok = get(joining, http.MethodGet, Path, "10.0.1.7")
+ if !ok || w.Code != http.StatusServiceUnavailable || !strings.Contains(w.Body.String(), CondHosting) {
+ t.Fatalf("joining GET: ok=%v %d %q", ok, w.Code, w.Body.String())
+ }
+ for _, c := range []struct{ method, target string }{
+ {http.MethodPost, Path},
+ {http.MethodGet, Path + "/"},
+ {http.MethodGet, "/.carlos/lb-healthz"},
+ {http.MethodGet, "/"},
+ } {
+ if _, ok := get(healthy(), c.method, c.target, "10.0.1.7"); ok {
+ t.Fatalf("%s %s answered", c.method, c.target)
+ }
+ }
+}
+
+func TestNilStateIgnoresUpdates(t *testing.T) {
+ var s *State
+ s.SetHosting(true)
+ s.MapLoaded()
+ s.KeyLoaded()
+ s.CertsLoaded()
+ s.EdgeBeat(t0)
+ s.RefreshAttempted(t0)
+ s.SetActivator(func() time.Time { return t0 })
+ if s.Hosting() {
+ t.Fatal("nil state hosting")
+ }
+}
+````
+
+`region_ha_boundary_test.go`:
+
+````go
+package main
+
+import (
+ "go/ast"
+ "go/parser"
+ "go/token"
+ "os"
+ "path/filepath"
+ "sort"
+ "strconv"
+ "strings"
+ "testing"
+)
+
+// Region HA spec 2 §25: the provider boundary. Spec 2's core names no
+// cloud provider; its AWS parts are adapters in files named for AWS. Each
+// plan task that adds a core file lists it here in the same commit.
+
+// regionHACoreFiles are spec 2's provider-neutral files. None may import
+// an AWS package.
+var regionHACoreFiles = []string{
+ "internal/lbhealth/lbhealth.go",
+}
+
+// regionHAAdapterFiles are spec 2's AWS adapters. Each is named for AWS so
+// a reader can see where the provider lives.
+var regionHAAdapterFiles = []string{}
+
+// regionHAPackages are the packages spec 2 touches. Every S3 client built
+// in them goes through s3compat.Options(), so CARLOS_S3_ENDPOINT applies.
+var regionHAPackages = []string{
+ "internal/activator",
+ "internal/boxchoice",
+ "internal/console",
+ "internal/edge",
+ "internal/edgesfile",
+ "internal/host",
+ "internal/hostactors",
+ "internal/hostops",
+ "internal/lbhealth",
+}
+
+// regionHAS3Exceptions are S3 clients on main that skip s3compat, each
+// with the discussion that fixes it outside spec 2. A listed file that has
+// since been fixed passes.
+var regionHAS3Exceptions = map[string]string{
+ "internal/host/claimproduction_marker.go": "#298",
+}
+
+func TestRegionHACoreImportsNoAWS(t *testing.T) {
+ for _, f := range regionHACoreFiles {
+ file, err := parser.ParseFile(token.NewFileSet(), f, nil, parser.ImportsOnly)
+ if err != nil {
+ t.Fatalf("%s: %v", f, err)
+ }
+ for _, imp := range file.Imports {
+ path, _ := strconv.Unquote(imp.Path.Value)
+ if strings.HasPrefix(path, "github.com/aws/") {
+ t.Errorf("%s imports %s: spec 2's core names no provider (§25); move it to an AWS adapter", f, path)
+ }
+ }
+ }
+}
+
+func TestRegionHAAdaptersAreNamedForAWS(t *testing.T) {
+ for _, f := range regionHAAdapterFiles {
+ if _, err := os.Stat(f); err != nil {
+ t.Errorf("%s: %v", f, err)
+ }
+ if !strings.Contains(filepath.Base(f), "aws") && !strings.Contains(filepath.Dir(f), "aws") {
+ t.Errorf("%s is an AWS adapter but its name does not say so", f)
+ }
+ }
+}
+
+// s3NewFromConfigCalls returns the line of each s3.NewFromConfig call in
+// src that does not pass s3compat.Options() as an argument.
+func s3NewFromConfigCalls(t *testing.T, path string) []int {
+ t.Helper()
+ fset := token.NewFileSet()
+ f, err := parser.ParseFile(fset, path, nil, 0)
+ if err != nil {
+ t.Fatalf("%s: %v", path, err)
+ }
+ s3Names, compatNames := map[string]bool{}, map[string]bool{}
+ for _, imp := range f.Imports {
+ p, _ := strconv.Unquote(imp.Path.Value)
+ name := filepath.Base(p)
+ if imp.Name != nil {
+ name = imp.Name.Name
+ }
+ switch p {
+ case "github.com/aws/aws-sdk-go-v2/service/s3":
+ s3Names[name] = true
+ case "github.com/carlosframework/platform/internal/s3compat":
+ compatNames[name] = true
+ }
+ }
+ var bad []int
+ ast.Inspect(f, func(n ast.Node) bool {
+ call, ok := n.(*ast.CallExpr)
+ if !ok {
+ return true
+ }
+ sel, ok := call.Fun.(*ast.SelectorExpr)
+ if !ok || sel.Sel.Name != "NewFromConfig" {
+ return true
+ }
+ pkg, ok := sel.X.(*ast.Ident)
+ if !ok || !s3Names[pkg.Name] {
+ return true
+ }
+ for _, arg := range call.Args {
+ inner, ok := arg.(*ast.CallExpr)
+ if !ok {
+ continue
+ }
+ isel, ok := inner.Fun.(*ast.SelectorExpr)
+ if !ok || isel.Sel.Name != "Options" {
+ continue
+ }
+ if id, ok := isel.X.(*ast.Ident); ok && compatNames[id.Name] {
+ return true
+ }
+ }
+ bad = append(bad, fset.Position(call.Pos()).Line)
+ return true
+ })
+ return bad
+}
+
+func TestRegionHAS3ClientsUseS3Compat(t *testing.T) {
+ for _, dir := range regionHAPackages {
+ paths, err := filepath.Glob(filepath.Join(dir, "*.go"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ sort.Strings(paths)
+ for _, p := range paths {
+ if strings.HasSuffix(p, "_test.go") {
+ continue
+ }
+ if _, ok := regionHAS3Exceptions[filepath.ToSlash(p)]; ok {
+ continue
+ }
+ for _, line := range s3NewFromConfigCalls(t, p) {
+ t.Errorf("%s:%d builds an S3 client without s3compat.Options() (§25): CARLOS_S3_ENDPOINT would not apply", p, line)
+ }
+ }
+ }
+}
+
+// The checker itself: a bare client is caught, one with Options() is not,
+// whatever the import names.
+func TestRegionHAS3CheckerCatchesABareClient(t *testing.T) {
+ dir := t.TempDir()
+ write := func(name, body string) string {
+ p := filepath.Join(dir, name)
+ if err := os.WriteFile(p, []byte(body), 0o644); err != nil {
+ t.Fatal(err)
+ }
+ return p
+ }
+ bare := write("bare.go", `package x
+import awss3 "github.com/aws/aws-sdk-go-v2/service/s3"
+func f(cfg any) { _ = awss3.NewFromConfig(cfg) }
+`)
+ good := write("good.go", `package x
+import (
+ "github.com/aws/aws-sdk-go-v2/service/s3"
+ "github.com/carlosframework/platform/internal/s3compat"
+)
+func f(cfg any) { _ = s3.NewFromConfig(cfg, s3compat.Options()) }
+`)
+ if got := s3NewFromConfigCalls(t, bare); len(got) != 1 {
+ t.Fatalf("bare client: %v", got)
+ }
+ if got := s3NewFromConfigCalls(t, good); len(got) != 0 {
+ t.Fatalf("compat client flagged: %v", got)
+ }
+}
+````
+
+- [ ] **Step 2: Run them to see them fail**
+
+Run: `nice -n 19 go test ./internal/lbhealth/ -count=1`
+Expected: FAIL, the package does not build (`undefined: State`). `go test . -run TestRegionHA` fails the same way on `internal/lbhealth/lbhealth.go` not existing.
+
+- [ ] **Step 3: Write the package**
+
+`internal/lbhealth/lbhealth.go`:
+
+````go
+// Package lbhealth is a box's own liveness, as the load balancer's health
+// check reads it (region HA spec 2 §6). Every fact here is this box's own:
+// a flag or a timestamp that one of its own loops sets. Nothing here reads
+// a disk, a bucket or the network, so an outage of something the boxes
+// share (the deployment bucket) never turns a working box unhealthy and
+// never gets it terminated.
+//
+// The edge answers GET or HEAD /.carlos/lb-health on port 80 from State.
+// The loops that feed it are the agent's: the claim agent sets hosting,
+// the home map latches forwarding and stamps its refresh attempts, the
+// certificate mirror latches its first load, and the edge's registry watch
+// and the activator's sweeper each beat.
+package lbhealth
+
+import (
+ "io"
+ "net/http"
+ "strings"
+ "sync"
+ "sync/atomic"
+ "time"
+)
+
+// Path is the health check's path, matched exactly.
+const Path = "/.carlos/lb-health"
+
+// HeartbeatWindow is how recent the edge's and the activator's beats must
+// be. Both loops beat every 30 seconds or faster.
+const HeartbeatWindow = 60 * time.Second
+
+// RefreshWindow is how recent the home map's last completed refresh
+// attempt must be: its 30-second interval plus its two-minute request
+// timeout, with room to spare. A failed attempt counts, so a bucket outage
+// where every attempt fails fast keeps this satisfied; a worker stuck
+// inside one attempt does not.
+const RefreshWindow = 5 * time.Minute
+
+// The condition names a 503 lists, in this order.
+const (
+ CondHosting = "hosting"
+ CondForwarding = "forwarding"
+ CondCerts = "certificates"
+ CondEdge = "edge"
+ CondActivator = "activator"
+ CondRefresh = "refresh"
+)
+
+// State is one box's liveness facts. The zero value is a box that has
+// proven nothing, and answers 503. A nil *State ignores every update.
+type State struct {
+ hosting atomic.Bool
+ mapLoaded atomic.Bool
+ keyLoaded atomic.Bool
+ certs atomic.Bool
+ edgeBeat atomic.Int64 // unix nanoseconds; 0 is never
+ refreshed atomic.Int64 // unix nanoseconds; 0 is never
+
+ mu sync.RWMutex
+ activator func() time.Time
+}
+
+// Default is the process's state. The agent runs the edge and the
+// activator in one process, so the claim wiring, the edge and the
+// activator share it.
+var Default = &State{}
+
+// SetHosting records whether the agent is in hosting mode: enrolled, with
+// its guard armed (§5). Region HA spec 2's plan 2b adds joining mode, whose
+// mode selection is what sets this false; until then the claim agent sets
+// it true once its claim wiring is running (ClaimWiring.startRegional).
+func (s *State) SetHosting(on bool) {
+ if s != nil {
+ s.hosting.Store(on)
+ }
+}
+
+// Hosting reports what SetHosting last recorded.
+func (s *State) Hosting() bool { return s != nil && s.hosting.Load() }
+
+// MapLoaded latches the first successful load of the forwarding map.
+func (s *State) MapLoaded() {
+ if s != nil {
+ s.mapLoaded.Store(true)
+ }
+}
+
+// KeyLoaded latches the first successful load of a mesh key.
+func (s *State) KeyLoaded() {
+ if s != nil {
+ s.keyLoaded.Store(true)
+ }
+}
+
+// CertsLoaded latches the first successful load of the certificate mirror
+// into the local cache the TLS handshake reads.
+func (s *State) CertsLoaded() {
+ if s != nil {
+ s.certs.Store(true)
+ }
+}
+
+// EdgeBeat records that the edge's registry watch completed a poll at t.
+func (s *State) EdgeBeat(t time.Time) {
+ if s != nil {
+ s.edgeBeat.Store(t.UnixNano())
+ }
+}
+
+// RefreshAttempted records that the home map's refresh finished an
+// attempt at t, whether it succeeded or failed.
+func (s *State) RefreshAttempted(t time.Time) {
+ if s != nil {
+ s.refreshed.Store(t.UnixNano())
+ }
+}
+
+// SetActivator gives the state the activator's own beat: the time its
+// sweeper last finished a sweep. nil means there is no activator, which
+// fails the activator condition.
+func (s *State) SetActivator(last func() time.Time) {
+ if s == nil {
+ return
+ }
+ s.mu.Lock()
+ s.activator = last
+ s.mu.Unlock()
+}
+
+func (s *State) activatorBeat() time.Time {
+ s.mu.RLock()
+ last := s.activator
+ s.mu.RUnlock()
+ if last == nil {
+ return time.Time{}
+ }
+ return last()
+}
+
+// within reports whether at is set and no older than window at now.
+func within(at, now time.Time, window time.Duration) bool {
+ return !at.IsZero() && now.Sub(at) <= window
+}
+
+func stamp(n int64) time.Time {
+ if n == 0 {
+ return time.Time{}
+ }
+ return time.Unix(0, n)
+}
+
+// Failing lists the conditions that do not hold at now, in the order of
+// the Cond names. An empty list is a healthy box. A nil *State fails
+// every condition.
+func (s *State) Failing(now time.Time) []string {
+ if s == nil {
+ return []string{CondHosting, CondForwarding, CondCerts, CondEdge, CondActivator, CondRefresh}
+ }
+ var out []string
+ if !s.hosting.Load() {
+ out = append(out, CondHosting)
+ }
+ if !s.mapLoaded.Load() || !s.keyLoaded.Load() {
+ out = append(out, CondForwarding)
+ }
+ if !s.certs.Load() {
+ out = append(out, CondCerts)
+ }
+ if !within(stamp(s.edgeBeat.Load()), now, HeartbeatWindow) {
+ out = append(out, CondEdge)
+ }
+ if !within(s.activatorBeat(), now, HeartbeatWindow) {
+ out = append(out, CondActivator)
+ }
+ if !within(stamp(s.refreshed.Load()), now, RefreshWindow) {
+ out = append(out, CondRefresh)
+ }
+ return out
+}
+
+// Serve answers the health check when r is one, and reports whether it
+// did. It matches as narrowly as the steering probe: the exact path, GET
+// or HEAD, any host. It answers 200 "ok" when every condition holds and
+// 503 naming the failing conditions otherwise, never cached. It only reads
+// s.
+func Serve(w http.ResponseWriter, r *http.Request, s *State, now time.Time) bool {
+ if r.URL.Path != Path || (r.Method != http.MethodGet && r.Method != http.MethodHead) {
+ return false
+ }
+ body, code := "ok\n", http.StatusOK
+ if failing := s.Failing(now); len(failing) > 0 {
+ body, code = "unavailable: "+strings.Join(failing, ",")+"\n", http.StatusServiceUnavailable
+ }
+ w.Header().Set("Content-Type", "text/plain; charset=utf-8")
+ w.Header().Set("Cache-Control", "no-store")
+ w.WriteHeader(code)
+ if r.Method == http.MethodGet {
+ _, _ = io.WriteString(w, body)
+ }
+ return true
+}
+````
+
+- [ ] **Step 4: Run the tests**
+
+Run: `nice -n 19 go test ./internal/lbhealth/ -count=1 && nice -n 19 go test . -run TestRegionHA -count=1`
+Expected: PASS. The S3 call-site test passes with `internal/host/claimproduction_marker.go` excepted for #298.
+
+- [ ] **Step 5: Census**
+
+Create the overlay:
+
+`internal/provision/testdata/plan14-root-reads-region-ha-2a.tsv`:
+
+````text
+# Plan 14 census overlay region-ha-2a (region HA spec 2, plan 2a: docs/superpowers/plans/2026-10-16-region-ha-2a-core.md), loaded last.
+# Task 2: the new internal/lbhealth package (the load balancer health state). Envelope-only: no root-read unit changed or was added, nothing runs as root that did not before, and no registry, /data, release or claim read was added, so the legacy seal stands.
+@coverage internal/lbhealth ad1211a01a4eb3d02cd8ed95cbff0f8b17b3976ce9c0279ebd2cae2f20c07124
+````
+
+Load it last in `TestPlan14RootReadCensus`:
+
+````diff
+--- a/internal/provision/plan14_root_reads_test.go
++++ b/internal/provision/plan14_root_reads_test.go
+@@ -759,6 +759,17 @@
+ t.Fatal(err)
+ }
+ seal = "fc38db001ebd8825709a8ca9c41cc0e8f165c1b1967e5967347dc9bd3556afec"
++ if errs := checkRootReadLegacy(inv, seal); len(errs) != 0 {
++ t.Fatal(errs)
++ }
++ // Census overlay region-ha-2a (region HA spec 2, plan 2a), loaded last.
++ // Its header says what each plan task added. The accepted-legacy rows
++ // it adds are census ruling region-ha-2a's (platform-1e), and move the
++ // seal.
++ if err = loadRootReadInventory(filepath.Join(root, "internal/provision/testdata/plan14-root-reads-region-ha-2a.tsv"), &inv, true); err != nil {
++ t.Fatal(err)
++ }
++ seal = "fc38db001ebd8825709a8ca9c41cc0e8f165c1b1967e5967347dc9bd3556afec"
+ errs := append(compareRootReads(got, inv), checkRootReadLegacy(inv, seal)...)
+ errs = append(errs, checkRootReadDeclarations(got, inv)...)
+ if len(errs) > 0 {
+````
+
+Run the census checks. Expected: PASS.
+
+- [ ] **Step 6: Commit**
+
+```bash
+git add internal/lbhealth region_ha_boundary_test.go internal/provision/testdata/plan14-root-reads-region-ha-2a.tsv internal/provision/plan14_root_reads_test.go
+git commit -m "region HA 2a: the load balancer health state and the provider boundary tests"
+```
+
+---
+
+### Task 3: Box and group roles in the roster
+
+**Files:**
+- Create: `internal/hostactors/roles.go`, `internal/hostactors/roles_test.go`
+- Modify: `internal/hostactors/roster.go` (`Box.Group`, `Box.Role`; `Validate`; `Replace` lets the role change)
+- Modify: `internal/hostactors/enrollment_confirmation.go` (`Region.Role`)
+- Modify: `internal/hostactors/enroll.go` (automatic enrollment inherits the group's role)
+- Modify: `region_ha_boundary_test.go`, `internal/provision/plan9_b8_census_test.go`, the overlay
+
+**Interfaces:**
+- Consumes: `Directory.Replace`, `Directory.Read`, `Clone`, `ErrRoster`.
+- Produces: `RoleDefault = ""`, `RoleLegacy = "legacy"`, `RolePlatform = "platform"`; `var ErrRole`; `ParseBoxRole(string) (string, error)` (default, legacy, platform); `ParseGroupRole(string) (string, error)` (default, legacy); `RoleName(string) string`; `func (d Directory) SetBoxRole(ctx context.Context, sg release.Signer, o Observed, ref, role string) (Observed, error)`; `func GroupMembers(r Roster, region string) []string`; `func (d Directory) SetGroupRole(ctx context.Context, sg release.Signer, o Observed, region, role string) (Observed, []string, error)`; fields `Box.Group`, `Box.Role` (json `group`, `role`, omitempty), `Region.Role` (json `role`, omitempty).
+
+A role is written only by the console (the only root signer). `Replace` keeps comparing every other field of an enrolled box, so `Group` never changes once set. Automatic enrollment (2b's caller) sets `Group` and `Role` from the region's group in the same write that admits the box (A11-1).
+
+- [ ] **Step 1: Write the failing tests**
+
+`internal/hostactors/roles_test.go`:
+
+````go
+package hostactors
+
+import (
+ "context"
+ "crypto"
+ "crypto/rand"
+ "crypto/rsa"
+ "crypto/sha256"
+ "crypto/x509"
+ "encoding/base64"
+ "encoding/json"
+ "errors"
+ "reflect"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/carlosframework/platform/internal/boxid"
+ "github.com/carlosframework/platform/internal/release"
+)
+
+func TestParseRoles(t *testing.T) {
+ for in, want := range map[string]string{"default": RoleDefault, "legacy": RoleLegacy, "platform": RolePlatform, " legacy ": RoleLegacy} {
+ if got, err := ParseBoxRole(in); err != nil || got != want {
+ t.Fatalf("ParseBoxRole(%q) = %q, %v", in, got, err)
+ }
+ }
+ for _, in := range []string{"", "Legacy", "draining", "none"} {
+ if _, err := ParseBoxRole(in); !errors.Is(err, ErrRole) {
+ t.Fatalf("ParseBoxRole(%q) accepted", in)
+ }
+ }
+ if _, err := ParseGroupRole("platform"); !errors.Is(err, ErrRole) {
+ t.Fatal("a group took the platform role")
+ }
+ if got, err := ParseGroupRole("legacy"); err != nil || got != RoleLegacy {
+ t.Fatalf("ParseGroupRole(legacy) = %q, %v", got, err)
+ }
+ if RoleName(RoleDefault) != "default" || RoleName(RolePlatform) != "platform" {
+ t.Fatal("RoleName")
+ }
+}
+
+func TestBoxRoleIsSignedAndPersists(t *testing.T) {
+ ctx := context.Background()
+ d, sg, o := fixture(t)
+ next, err := d.SetBoxRole(ctx, sg, o, "box", RoleLegacy)
+ if err != nil {
+ t.Fatal(err)
+ }
+ read, err := d.Read(ctx)
+ if err != nil || read.Roster.Boxes["box"].Role != RoleLegacy || read.ETag != next.ETag {
+ t.Fatalf("read back: %+v %v", read.Roster.Boxes["box"], err)
+ }
+ // The same role again writes nothing.
+ same, err := d.SetBoxRole(ctx, sg, next, "box", RoleLegacy)
+ if err != nil || same.ETag != next.ETag {
+ t.Fatalf("idempotent set wrote: %v", err)
+ }
+ // A stale observation loses the CAS.
+ if _, err = d.SetBoxRole(ctx, sg, o, "box", RolePlatform); err == nil {
+ t.Fatal("stale roster accepted")
+ }
+ back, err := d.SetBoxRole(ctx, sg, next, "box", RoleDefault)
+ if err != nil || back.Roster.Boxes["box"].Role != RoleDefault {
+ t.Fatalf("back to default: %+v %v", back.Roster.Boxes["box"], err)
+ }
+ if _, err = d.SetBoxRole(ctx, sg, back, "nobody", RoleLegacy); !errors.Is(err, ErrRoster) {
+ t.Fatalf("unknown box: %v", err)
+ }
+ if _, err = d.SetBoxRole(ctx, sg, back, "box", "boss"); !errors.Is(err, ErrRole) {
+ t.Fatalf("unknown role: %v", err)
+ }
+ // A retired box keeps no role.
+ retired := Clone(back.Roster)
+ b := retired.Boxes["box"]
+ b.State = "retired"
+ retired.Boxes["box"] = b
+ gone, err := d.Replace(ctx, sg, back, retired)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if _, err = d.SetBoxRole(ctx, sg, gone, "box", RoleLegacy); !errors.Is(err, ErrRoster) {
+ t.Fatalf("retired box: %v", err)
+ }
+}
+
+func TestRosterRefusesUnknownRolesAndAGroupChange(t *testing.T) {
+ ctx := context.Background()
+ d, sg, o := fixture(t)
+ bad := Clone(o.Roster)
+ b := bad.Boxes["box"]
+ b.Role = "boss"
+ bad.Boxes["box"] = b
+ if _, err := d.Replace(ctx, sg, o, bad); !errors.Is(err, ErrRoster) {
+ t.Fatalf("unknown box role written: %v", err)
+ }
+ moved := Clone(o.Roster)
+ b = moved.Boxes["box"]
+ b.Group = "routers"
+ moved.Boxes["box"] = b
+ if _, err := d.Replace(ctx, sg, o, moved); !errors.Is(err, ErrRoster) {
+ t.Fatalf("group rewritten: %v", err)
+ }
+ noGroup := Clone(o.Roster)
+ noGroup.Regions = map[string]Region{"ie": {Account: "123", AWSRegion: "eu-west-1", Role: RoleLegacy}}
+ if _, err := d.Replace(ctx, sg, o, noGroup); !errors.Is(err, ErrRoster) {
+ t.Fatalf("a role on a region with no group: %v", err)
+ }
+ platformGroup := Clone(o.Roster)
+ platformGroup.Regions = map[string]Region{"ie": {Account: "123", AWSRegion: "eu-west-1", GroupName: "routers", Role: RolePlatform}}
+ if _, err := d.Replace(ctx, sg, o, platformGroup); !errors.Is(err, ErrRoster) {
+ t.Fatalf("a platform group: %v", err)
+ }
+}
+
+// groupFixture is fixture plus a group in ie and two boxes it enrolled.
+func groupFixture(t *testing.T) (Directory, release.Signer, Observed) {
+ t.Helper()
+ ctx := context.Background()
+ d, sg, o := fixture(t)
+ next := Clone(o.Roster)
+ next.Regions = map[string]Region{
+ "ie": {Account: "123", AWSRegion: "eu-west-1", GroupName: "routers", GroupARN: "group-arn", Namespaces: []string{"platform"}, Pools: []string{"default"}},
+ "uk": {Account: "123", AWSRegion: "eu-west-2"},
+ }
+ for i, ref := range []string{"g1", "g2"} {
+ next.Boxes[ref] = Box{Enrollment: strings.Repeat(string(rune('c'+i)), 32), Namespace: "platform", BoxID: ref, Label: ref, Region: "ie", Pool: "default", Cloud: Cloud{Provider: "aws", Account: "123", Region: "eu-west-1", InstanceID: "i-" + ref}, EvidenceSHA256: strings.Repeat("b", 64), State: "active", LossIDs: []string{}, Group: "routers"}
+ }
+ o, err := d.Replace(ctx, sg, o, next)
+ if err != nil {
+ t.Fatal(err)
+ }
+ return d, sg, o
+}
+
+func TestGroupRoleSetsEveryMemberInOneWrite(t *testing.T) {
+ ctx := context.Background()
+ d, sg, o := groupFixture(t)
+ next, set, err := d.SetGroupRole(ctx, sg, o, "ie", RoleLegacy)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if !reflect.DeepEqual(set, []string{"g1", "g2"}) {
+ t.Fatalf("set %v", set)
+ }
+ if next.Roster.Revision != o.Roster.Revision+1 {
+ t.Fatalf("revision %d, want one write", next.Roster.Revision)
+ }
+ r := next.Roster
+ if r.Regions["ie"].Role != RoleLegacy || r.Boxes["g1"].Role != RoleLegacy || r.Boxes["g2"].Role != RoleLegacy {
+ t.Fatalf("roles: %+v %+v %+v", r.Regions["ie"], r.Boxes["g1"], r.Boxes["g2"])
+ }
+ // The operator-enrolled box in the same region is not the group's.
+ if r.Boxes["box"].Role != RoleDefault {
+ t.Fatalf("operator box took the group's role: %+v", r.Boxes["box"])
+ }
+ again, _, err := d.SetGroupRole(ctx, sg, next, "ie", RoleLegacy)
+ if err != nil || again.ETag != next.ETag {
+ t.Fatalf("idempotent group set wrote: %v", err)
+ }
+ back, _, err := d.SetGroupRole(ctx, sg, next, "ie", RoleDefault)
+ if err != nil || back.Roster.Boxes["g1"].Role != RoleDefault || back.Roster.Regions["ie"].Role != RoleDefault {
+ t.Fatalf("back to default: %v", err)
+ }
+ if _, _, err = d.SetGroupRole(ctx, sg, back, "uk", RoleLegacy); !errors.Is(err, ErrRoster) {
+ t.Fatalf("a region with no group: %v", err)
+ }
+ if _, _, err = d.SetGroupRole(ctx, sg, back, "ie", RolePlatform); !errors.Is(err, ErrRole) {
+ t.Fatalf("a platform group: %v", err)
+ }
+}
+
+// Automatic enrollment gives the new box the group's role in the same
+// roster write (A11-1): a replacement for an excluded group is never
+// placeable, even for a moment.
+func TestAutomaticEnrollmentInheritsTheGroupRole(t *testing.T) {
+ for _, role := range []string{RoleDefault, RoleLegacy} {
+ t.Run(RoleName(role), func(t *testing.T) {
+ ctx := context.Background()
+ d, sg, o := fixture(t)
+ key, err := rsa.GenerateKey(rand.Reader, 2048)
+ if err != nil {
+ t.Fatal(err)
+ }
+ doc := []byte(`{"accountId":"123","region":"eu-west-1","instanceId":"i-new"}`)
+ sum := sha256.Sum256(doc)
+ sig, err := rsa.SignPKCS1v15(rand.Reader, key, crypto.SHA256, sum[:])
+ if err != nil {
+ t.Fatal(err)
+ }
+ raw, _ := json.Marshal(boxid.Record{V: 1, Box: "new", IdentityDocument: base64.StdEncoding.EncodeToString(doc), IdentitySignature: base64.StdEncoding.EncodeToString(sig)})
+ v := boxid.Verifier{Account: "123", Certs: map[string]*x509.Certificate{"eu-west-1": {PublicKey: &key.PublicKey, PublicKeyAlgorithm: x509.RSA}}}
+ in := EnrollmentRequest{Reference: "new", Namespace: "platform", Label: "new", Region: "ie", Pool: "default", AWSRegion: "eu-west-1"}
+ next := Clone(o.Roster)
+ next.Regions = map[string]Region{"ie": {Account: "123", AWSRegion: "eu-west-1", GroupName: "routers", GroupARN: "group-arn", Namespaces: []string{"platform"}, Pools: []string{"default"}, Role: role}}
+ if o, err = d.Replace(ctx, sg, o, next); err != nil {
+ t.Fatal(err)
+ }
+ if _, err = d.Invite(ctx, sg, in.Reference, raw); err != nil {
+ t.Fatal(err)
+ }
+ o, a, err := d.BeginEnrollmentAttempt(ctx, sg, o, in, raw, "i-new")
+ if err != nil {
+ t.Fatal(err)
+ }
+ now := time.Now().UTC()
+ c := MembershipConfirmation{PassID: "pass-1", AttemptID: a.AttemptID, ConfirmationID: a.ConfirmationID, EvidenceSHA256: a.EvidenceSHA256, TargetPlacement: "ie", AWSRegion: "eu-west-1", GroupName: "routers", GroupARN: "group-arn", InstanceID: "i-new", Account: "123", AvailabilityZoneRegion: "eu-west-1", LifecycleState: "InService", InstanceState: "running", ObservedAt: now, RequestIDs: []string{"asg", "ec2"}}
+ c.Confirmer.Box, c.Confirmer.Enrollment, c.Confirmer.Region = "box", o.Roster.Boxes["box"].Enrollment, "ie"
+ wire, _ := json.Marshal(c)
+ if err := release.CreateExact(ctx, d.Store, ConfirmationKey(in.Reference, a.AttemptID, a.ConfirmationID), wire); err != nil {
+ t.Fatal(err)
+ }
+ in.AttemptID, in.ConfirmationID = a.AttemptID, a.ConfirmationID
+ admitted, err := d.EnrollAutomatic(ctx, sg, o, v, in, raw, "i-new", now)
+ if err != nil {
+ t.Fatal(err)
+ }
+ b := admitted.Roster.Boxes["new"]
+ if b.Group != "routers" || b.Role != role {
+ t.Fatalf("admitted box group %q role %q, want routers %q", b.Group, b.Role, role)
+ }
+ if admitted.Roster.Revision != o.Roster.Revision+1 {
+ t.Fatal("the role was not set in the enrolling write")
+ }
+ })
+ }
+}
+
+// Every other roster write keeps the role: a quarantine, for one, rewrites
+// the box's entry and must not reopen placement on it.
+func TestOtherRosterWritesKeepTheRole(t *testing.T) {
+ ctx := context.Background()
+ d, sg, o := fixture(t)
+ o, err := d.SetBoxRole(ctx, sg, o, "box", RoleLegacy)
+ if err != nil {
+ t.Fatal(err)
+ }
+ q, err := d.Quarantine(ctx, sg, o, "box", o.Roster.Boxes["box"].Enrollment, "loss-1")
+ if err != nil {
+ t.Fatal(err)
+ }
+ if b := q.Roster.Boxes["box"]; b.Role != RoleLegacy || b.State != "quarantined" {
+ t.Fatalf("after a quarantine: %+v", b)
+ }
+ read, err := d.Read(ctx)
+ if err != nil || read.Roster.Boxes["box"].Role != RoleLegacy {
+ t.Fatalf("read back: %+v %v", read.Roster.Boxes["box"], err)
+ }
+}
+````
+
+- [ ] **Step 2: Run them to see them fail**
+
+Run: `nice -n 19 go test ./internal/hostactors/ -run 'Role|Roster|Group|Inherits' -count=1`
+Expected: FAIL to build (`undefined: ParseBoxRole`, `b.Role undefined`).
+
+- [ ] **Step 3: Write the roles**
+
+`internal/hostactors/roles.go`:
+
+````go
+package hostactors
+
+import (
+ "context"
+ "errors"
+ "sort"
+ "strings"
+
+ "github.com/carlosframework/platform/internal/release"
+)
+
+// Box and group roles (region HA spec 2 §7, A11-1). A box's role says
+// whether placement may choose it: the default role may be chosen, and
+// legacy and platform never are, for a create, a relocation target or a
+// move target. The box keeps the instances it has (H25).
+//
+// The region's group entry carries a role too. Every box automatic
+// enrollment admits from the group takes the group's role in the same
+// roster write, and changing the group's role changes the role of every
+// box enrolled from it, so a replacement launched after the group was
+// excluded is never placeable, even for a moment.
+//
+// The roster stores the default role as "" so a roster written before
+// roles existed reads as all-default.
+const (
+ RoleDefault = ""
+ RoleLegacy = "legacy"
+ RolePlatform = "platform"
+)
+
+// ErrRole is a role name a box or group cannot take.
+var ErrRole = errors.New("hostactors: not a role this box or group can take")
+
+// ParseBoxRole reads an operator's box role: default, legacy or platform.
+func ParseBoxRole(s string) (string, error) {
+ switch strings.TrimSpace(s) {
+ case "default":
+ return RoleDefault, nil
+ case RoleLegacy:
+ return RoleLegacy, nil
+ case RolePlatform:
+ return RolePlatform, nil
+ }
+ return "", ErrRole
+}
+
+// ParseGroupRole reads an operator's group role: default or legacy. A
+// group is never the platform box.
+func ParseGroupRole(s string) (string, error) {
+ switch strings.TrimSpace(s) {
+ case "default":
+ return RoleDefault, nil
+ case RoleLegacy:
+ return RoleLegacy, nil
+ }
+ return "", ErrRole
+}
+
+// RoleName is the operator's word for a stored role.
+func RoleName(role string) string {
+ if role == RoleDefault {
+ return "default"
+ }
+ return role
+}
+
+func validBoxRole(role string) bool {
+ return role == RoleDefault || role == RoleLegacy || role == RolePlatform
+}
+
+func validGroupRole(role string) bool { return role == RoleDefault || role == RoleLegacy }
+
+// SetBoxRole sets one box's role in a fresh roster write over o. The box
+// must be enrolled and not retired. Setting the role it already has
+// changes nothing and returns o.
+func (d Directory) SetBoxRole(ctx context.Context, sg release.Signer, o Observed, ref, role string) (Observed, error) {
+ if !validBoxRole(role) {
+ return Observed{}, ErrRole
+ }
+ b, ok := o.Roster.Boxes[ref]
+ if !ok || b.State == "retired" {
+ return Observed{}, ErrRoster
+ }
+ if b.Role == role {
+ return o, nil
+ }
+ r := Clone(o.Roster)
+ b = r.Boxes[ref]
+ b.Role = role
+ r.Boxes[ref] = b
+ return d.Replace(ctx, sg, o, r)
+}
+
+// GroupMembers returns, sorted, the boxes enrolled from region's group
+// that are not retired: the boxes a group role change applies to.
+func GroupMembers(r Roster, region string) []string {
+ g, ok := r.Regions[region]
+ if !ok || g.GroupName == "" {
+ return nil
+ }
+ var out []string
+ for ref, b := range r.Boxes {
+ if b.Region == region && b.Group == g.GroupName && b.State != "retired" {
+ out = append(out, ref)
+ }
+ }
+ sort.Strings(out)
+ return out
+}
+
+// SetGroupRole sets region's group role and, in the same roster write, the
+// role of every box enrolled from that group (GroupMembers). The region
+// must have a group. It returns the roster and the boxes it set.
+func (d Directory) SetGroupRole(ctx context.Context, sg release.Signer, o Observed, region, role string) (Observed, []string, error) {
+ if !validGroupRole(role) {
+ return Observed{}, nil, ErrRole
+ }
+ g, ok := o.Roster.Regions[region]
+ if !ok || g.GroupName == "" {
+ return Observed{}, nil, ErrRoster
+ }
+ members := GroupMembers(o.Roster, region)
+ changed := g.Role != role
+ for _, ref := range members {
+ changed = changed || o.Roster.Boxes[ref].Role != role
+ }
+ if !changed {
+ return o, members, nil
+ }
+ r := Clone(o.Roster)
+ g.Role = role
+ r.Regions[region] = g
+ for _, ref := range members {
+ b := r.Boxes[ref]
+ b.Role = role
+ r.Boxes[ref] = b
+ }
+ next, err := d.Replace(ctx, sg, o, r)
+ if err != nil {
+ return Observed{}, nil, err
+ }
+ return next, members, nil
+}
+````
+
+````diff
+--- a/internal/hostactors/roster.go
++++ b/internal/hostactors/roster.go
+@@ -39,6 +39,14 @@
+ State string `json:"state"`
+ QuarantineRevision uint64 `json:"quarantine_revision"`
+ LossIDs []string `json:"loss_ids"`
++ // Group is the Auto Scaling group automatic enrollment admitted the box
++ // from (the region's GroupName then); "" for a box an operator
++ // enrolled. It never changes.
++ Group string `json:"group,omitempty"`
++ // Role is the box's placement role (roles.go): "" (default), legacy or
++ // platform. It is the one field besides the state fields a roster
++ // write may change on an enrolled box.
++ Role string `json:"role,omitempty"`
+ }
+ type Console struct {
+ EnrolledAt time.Time `json:"enrolled_at,omitzero"`
+@@ -114,11 +122,19 @@
+ if b.State != "active" && b.State != "quarantined" && b.State != "retired" {
+ return ErrRoster
+ }
++ if !validBoxRole(b.Role) {
++ return ErrRoster
++ }
+ if b.State == "quarantined" && (b.QuarantineRevision == 0 || len(b.LossIDs) == 0) {
+ return ErrRoster
+ }
+ ids[b.Enrollment] = true
+ clouds[b.Cloud] = true
++ }
++ for _, g := range r.Regions {
++ if !validGroupRole(g.Role) || (g.Role != RoleDefault && g.GroupName == "") {
++ return ErrRoster
++ }
+ }
+ for id, c := range r.Consoles {
+ if !segment(id) || !token.MatchString(c.Enrollment) || ids[c.Enrollment] || c.InstanceID != id || c.DeploymentBucket != bucket || len(c.Namespaces) == 0 {
+@@ -246,6 +262,8 @@
+ z.QuarantineRevision = 0
+ a.LossIDs = nil
+ z.LossIDs = nil
++ a.Role = ""
++ z.Role = ""
+ if !reflect.DeepEqual(a, z) || b.QuarantineRevision < old.QuarantineRevision {
+ return Observed{}, ErrRoster
+ }
+````
+
+````diff
+--- a/internal/hostactors/enrollment_confirmation.go
++++ b/internal/hostactors/enrollment_confirmation.go
+@@ -18,6 +18,9 @@
+ AWSRegion string `json:"aws_region"`
+ GroupName string `json:"group_name"`
+ GroupARN string `json:"group_arn"`
++ // Role is the role every box automatic enrollment admits from the
++ // group takes (roles.go): "" (default) or legacy.
++ Role string `json:"role,omitempty"`
+ }
+ type TerminatedEnrollment struct {
+ PassID string `json:"pass_id"`
+````
+
+````diff
+--- a/internal/hostactors/enroll.go
++++ b/internal/hostactors/enroll.go
+@@ -68,7 +68,15 @@
+ a.Consumed = true
+ r.EnrollmentAttempts[in.Reference] = a
+ }
+- r.Boxes[in.Reference] = Box{Enrollment: enrollment, Namespace: in.Namespace, BoxID: in.Reference, Label: in.Label, Region: in.Region, Pool: in.Pool, Cloud: cloud, EvidenceSHA256: evidence.RecordSHA256, State: "active", LossIDs: []string{}}
++ box := Box{Enrollment: enrollment, Namespace: in.Namespace, BoxID: in.Reference, Label: in.Label, Region: in.Region, Pool: in.Pool, Cloud: cloud, EvidenceSHA256: evidence.RecordSHA256, State: "active", LossIDs: []string{}}
++ if automatic {
++ // A box the group launched takes the group's role in this same
++ // write (region HA spec 2 §7, A11-1): a replacement for an
++ // excluded group is never placeable, even for a moment.
++ g := r.Regions[in.Region]
++ box.Group, box.Role = g.GroupName, g.Role
++ }
++ r.Boxes[in.Reference] = box
+ return d.Replace(ctx, sg, o, r)
+ }
+
+````
+
+- [ ] **Step 4: Run the tests**
+
+Run: `nice -n 19 go test ./internal/hostactors/ -count=1`
+Expected: PASS (the package's whole suite, about 15 seconds).
+
+- [ ] **Step 5: Boundary list and census**
+
+````diff
+--- a/region_ha_boundary_test.go
++++ b/region_ha_boundary_test.go
+@@ -20,6 +20,7 @@
+ // an AWS package.
+ var regionHACoreFiles = []string{
+ "internal/lbhealth/lbhealth.go",
++ "internal/hostactors/roles.go",
+ }
+
+ // regionHAAdapterFiles are spec 2's AWS adapters. Each is named for AWS so
+````
+
+`Directory.Replace` is in the record-write census (it names `Put`). Its digest changes:
+
+````diff
+--- a/internal/provision/plan9_b8_census_test.go
++++ b/internal/provision/plan9_b8_census_test.go
+@@ -791,7 +791,7 @@
+ {"internal/host/snapshot_loop.go", "putReplacing", "nonrecord", "sel:Put", "11e8639effce142eab707b67cbdd425d0d7cea510ddf95d292bdba8f79c9b510"},
+ {"internal/hostactors/censuskey.go", "Directory.BindCensusKey", "nonrecord", "sel:Put", "7d9d673435e8c8a6aac3105e3a754bde2adcbc5d603945dd599cef703f334a5b"},
+ {"internal/hostactors/invitation.go", "Directory.Invite", "nonrecord", "sel:Put", "356618a75a23d0157d8e0402e20187a42956f67a4332d698bb22bf05133ac489"},
+- {"internal/hostactors/roster.go", "Directory.Replace", "nonrecord", "sel:Put", "677bd94746cc5d3a5cd966d221f80b509ae096083a8ff4a0ee817d4c12c4a75f"},
++ {"internal/hostactors/roster.go", "Directory.Replace", "nonrecord", "sel:Put", "e6e8ac0070650dfacc4a438b080cb69738530b3d9013de55c1bcf06431b0a847"},
+ {"internal/hostbox/regionalworker.go", "RegionalWorker.Census", "key-site", "lit:instances", "477bcb896b5ae8ec80af592d68eceb7992ad8a23cd6cff6043ad03b3ef2a8085"},
+ {"internal/hostbox/requeststore.go", "RequestStore.CopyVersion", "versioned-transport", "def:CopyVersion,sel:CopyVersion", "3a50049ce7ca7f2cbc770bef46218944cdc7000205a86c20e4932f0b510da28f"},
+ {"internal/hostbox/requeststore.go", "RequestStore.PutIfAbsent", "versioned-transport", "def:PutIfAbsent,sel:PutIfAbsent", "3666cbe5d2b91c1c652ffb4459c10a374fbd92e325e4e5c574fa3525d0ce2ffe"},
+````
+
+Add this comment line after the overlay's last comment line:
+
+````text
+# Task 3: box and group roles in the signed roster (internal/hostactors roles.go; Box.Role and Box.Group, Region.Role, Replace lets the role change, automatic enrollment inherits the group's role). Envelope-only: Directory.Replace, Roster.Validate and Directory.enroll are not root-read units, no root-read unit changed or was added, and the legacy seal stands.
+````
+
+Then set these rows. A `@coverage` row replaces the overlay's row for the same directory, or is appended when there is none; every other row is appended:
+
+````text
+@coverage internal/hostactors 6736698380a247764ada687f2476f89382fc32a1183a50c06afe75967d716eab
+````
+
+Run the census checks and `go test . -run TestRegionHA -count=1`. Expected: PASS.
+
+- [ ] **Step 6: Commit**
+
+```bash
+git add internal/hostactors region_ha_boundary_test.go internal/provision
+git commit -m "region HA 2a: box and group roles in the signed roster"
+```
+
+---
+
+### Task 4: The placement policy
+
+**Files:**
+- Create: `internal/boxchoice/policy.go`, `internal/boxchoice/policy_test.go`
+- Modify: `internal/boxchoice/boxchoice.go` (headroom totals and the generation carried through `Eligible`)
+- Modify: `internal/edgesfile/snapshot.go` (the reader's half: totals, generation, `BoxChoice`)
+- Create: `internal/edgesfile/snapshot_headroom_test.go`
+- Modify: `region_ha_boundary_test.go`, the overlay
+
+**Interfaces:**
+- Consumes: `boxchoice.Candidate`, `Load`, `Eligible`, `Choose` (its hash rule).
+- Produces: `Headroom.MemTotalMiB`, `Headroom.DiskTotalMiB` (json `mem_total_mib`, `disk_total_mib`); `Box.Generation string` (json `generation`); `Candidate.MemTotalMiB`, `Candidate.DiskTotalMiB`, `Candidate.Generation`; `MinMemPercent = 15`, `MinDiskPercent = 10`, `MaxEdgesAge = 15 * time.Minute`; `func ExcludeRoles(cands []Candidate, roles map[string]string) []Candidate`; `func Placeable(cands []Candidate, roles map[string]string, now time.Time) []Candidate`; `func ChooseRoomiest(host string, cands []Candidate, running map[string]Load, app string) (Candidate, bool)`; `edgesfile.PlacementView.Generation` and its headroom totals.
+
+`Eligible` only carries the new fields through; the new rules are `Placeable` and `ChooseRoomiest`, over candidates `Eligible` has already filtered for identity (§25). Any role other than the default excludes a box, so a role this build does not know fails closed.
+
+- [ ] **Step 1: Write the failing tests**
+
+`internal/boxchoice/policy_test.go`:
+
+````go
+package boxchoice
+
+import (
+ "reflect"
+ "testing"
+ "time"
+)
+
+var policyNow = time.Date(2026, 10, 16, 12, 0, 0, 0, time.UTC)
+
+// roomy is a candidate every §7 rule passes at policyNow.
+func roomy(id string, memFree int64) Candidate {
+ return Candidate{BoxID: id, Region: "eu-west-2", MemFreeMiB: memFree, MemTotalMiB: 8000, DiskFreeMiB: 50000, DiskTotalMiB: 100000,
+ Generation: policyNow.Add(-5 * time.Minute).Format(time.RFC3339)}
+}
+
+func TestPlaceableAppliesEachRule(t *testing.T) {
+ gen := func(d time.Duration) string { return policyNow.Add(d).Format(time.RFC3339) }
+ cases := []struct {
+ name string
+ mut func(*Candidate)
+ role string
+ want bool
+ }{
+ {"every rule holds", nil, "", true},
+ {"memory exactly 15%", func(c *Candidate) { c.MemFreeMiB = 1200 }, "", true},
+ {"memory under 15%", func(c *Candidate) { c.MemFreeMiB = 1199 }, "", false},
+ {"no memory total", func(c *Candidate) { c.MemTotalMiB = 0 }, "", false},
+ {"disk exactly 10%", func(c *Candidate) { c.DiskFreeMiB = 10000 }, "", true},
+ {"disk under 10%", func(c *Candidate) { c.DiskFreeMiB = 9999 }, "", false},
+ {"no disk total", func(c *Candidate) { c.DiskTotalMiB = 0 }, "", false},
+ {"edges file exactly 15 minutes old", func(c *Candidate) { c.Generation = gen(-15 * time.Minute) }, "", true},
+ {"edges file older than 15 minutes", func(c *Candidate) { c.Generation = gen(-15*time.Minute - time.Second) }, "", false},
+ {"edges file a minute in the future", func(c *Candidate) { c.Generation = gen(time.Minute) }, "", true},
+ {"edges file further in the future", func(c *Candidate) { c.Generation = gen(time.Minute + time.Second) }, "", false},
+ {"no generation", func(c *Candidate) { c.Generation = "" }, "", false},
+ {"unreadable generation", func(c *Candidate) { c.Generation = "yesterday" }, "", false},
+ {"legacy role", nil, "legacy", false},
+ {"platform role", nil, "platform", false},
+ {"a role this build does not know", nil, "draining", false},
+ }
+ for _, tc := range cases {
+ t.Run(tc.name, func(t *testing.T) {
+ c := roomy("i-0aaa", 4000)
+ if tc.mut != nil {
+ tc.mut(&c)
+ }
+ got := Placeable([]Candidate{c}, map[string]string{"i-0aaa": tc.role}, policyNow)
+ if (len(got) == 1) != tc.want {
+ t.Fatalf("placeable = %v, want %v", ids(got), tc.want)
+ }
+ })
+ }
+}
+
+func TestExcludeRolesKeepsOrderAndDefaults(t *testing.T) {
+ cands := []Candidate{roomy("c", 1), roomy("a", 1), roomy("b", 1)}
+ got := ExcludeRoles(cands, map[string]string{"a": "legacy", "b": ""})
+ if !reflect.DeepEqual(ids(got), []string{"c", "b"}) {
+ t.Fatalf("got %v", ids(got))
+ }
+ if got := ExcludeRoles(cands, nil); len(got) != 3 {
+ t.Fatalf("no roles excluded %v", ids(got))
+ }
+}
+
+func TestChooseRoomiestSpreadsThenMostMemoryThenHash(t *testing.T) {
+ running := map[string]Load{"a": {Hosts: 1, Apps: map[string]int{"bab/jam": 1}}}
+ // a has the most memory but already runs the app: spread wins.
+ got, ok := ChooseRoomiest("h.example", []Candidate{roomy("a", 7000), roomy("b", 2000), roomy("c", 3000)}, running, "bab/jam")
+ if !ok || got.BoxID != "c" {
+ t.Fatalf("spread then memory: got %q", got.BoxID)
+ }
+ // Spread never refuses: every box runs the app, so memory decides.
+ all := map[string]Load{"a": running["a"], "b": running["a"]}
+ got, _ = ChooseRoomiest("h.example", []Candidate{roomy("a", 1000), roomy("b", 2000)}, all, "bab/jam")
+ if got.BoxID != "b" {
+ t.Fatalf("all run the app: got %q", got.BoxID)
+ }
+ // Fewer hosts is no longer a rule: memory beats it.
+ busy := map[string]Load{"a": {Hosts: 50}, "b": {Hosts: 1}}
+ got, _ = ChooseRoomiest("h.example", []Candidate{roomy("a", 5000), roomy("b", 4000)}, busy, "bab/jam")
+ if got.BoxID != "a" {
+ t.Fatalf("memory over host count: got %q", got.BoxID)
+ }
+ // Equal memory: the P5-D2 hash, whatever the order.
+ one, _ := ChooseRoomiest("h.example", []Candidate{roomy("a", 1000), roomy("b", 1000)}, nil, "bab/jam")
+ two, _ := ChooseRoomiest("h.example", []Candidate{roomy("b", 1000), roomy("a", 1000)}, nil, "bab/jam")
+ if one.BoxID != two.BoxID {
+ t.Fatalf("order changed the answer: %q %q", one.BoxID, two.BoxID)
+ }
+ want, _ := Choose("h.example", []Candidate{roomy("a", 1000), roomy("b", 1000)}, nil, "bab/jam")
+ if one.BoxID != want.BoxID {
+ t.Fatalf("hash tie-break differs from spec 1's: %q, want %q", one.BoxID, want.BoxID)
+ }
+ if _, ok := ChooseRoomiest("h.example", nil, nil, "bab/jam"); ok {
+ t.Fatal("chose from nothing")
+ }
+}
+
+func TestEligibleCarriesTotalsAndGeneration(t *testing.T) {
+ body := haBox("i-0aaa", func(m map[string]any) {
+ m["headroom"] = map[string]any{"mem_free_mib": 1000, "disk_free_mib": 2000, "mem_total_mib": 8000, "disk_total_mib": 40000, "awake": 1}
+ m["generation"] = "2026-10-16T11:55:00Z"
+ })
+ got := Eligible(views(t, body), rosterFor("i-0aaa"), "uk", "eu-west-2", "ie", testKeyFP, nil)
+ want := Candidate{BoxID: "i-0aaa", Region: "eu-west-2", MemFreeMiB: 1000, DiskFreeMiB: 2000, MemTotalMiB: 8000, DiskTotalMiB: 40000, Awake: 1, Generation: "2026-10-16T11:55:00Z"}
+ if len(got) != 1 || got[0] != want {
+ t.Fatalf("got %+v, want %+v", got, want)
+ }
+}
+````
+
+`internal/edgesfile/snapshot_headroom_test.go`:
+
+````go
+package edgesfile
+
+import (
+ "encoding/json"
+ "testing"
+)
+
+// Region HA spec 2 §7: the placement policy reads the headroom totals and
+// the file's generation, so BoxChoice carries both.
+func TestBoxChoiceCarriesHeadroomTotalsAndGeneration(t *testing.T) {
+ var f PlacementView
+ body := `{"v":1,"box":"box-a","box_id":"i-a","generation":"2026-10-16T11:55:00Z","headroom":{"mem_free_mib":512,"disk_free_mib":2048,"mem_total_mib":4096,"disk_total_mib":20480,"awake":3},"hosts":{}}`
+ if err := json.Unmarshal([]byte(body), &f); err != nil {
+ t.Fatal(err)
+ }
+ f.Fresh = true
+ got := Snapshot{Files: []PlacementView{f}, Complete: true, BoxesComplete: true}.BoxChoice()
+ if len(got.Boxes) != 1 {
+ t.Fatalf("boxes %+v", got.Boxes)
+ }
+ b := got.Boxes[0]
+ if b.Generation != "2026-10-16T11:55:00Z" || b.Headroom == nil || b.Headroom.MemTotalMiB != 4096 || b.Headroom.DiskTotalMiB != 20480 || b.Headroom.MemFreeMiB != 512 || b.Headroom.DiskFreeMiB != 2048 || b.Headroom.Awake != 3 {
+ t.Fatalf("box lost a field: %+v %+v", b, b.Headroom)
+ }
+}
+````
+
+- [ ] **Step 2: Run them to see them fail**
+
+Run: `nice -n 19 go test ./internal/boxchoice/ ./internal/edgesfile/ -count=1`
+Expected: FAIL to build (`undefined: Placeable`, `unknown field MemTotalMiB`).
+
+- [ ] **Step 3: Write the policy**
+
+`internal/boxchoice/policy.go`:
+
+````go
+package boxchoice
+
+import (
+ "crypto/sha256"
+ "encoding/hex"
+ "sort"
+ "time"
+)
+
+// Region HA spec 2's placement policy (§7). Its rules run over candidates
+// Eligible has already filtered for identity, placement and configuration
+// (§25): a provider-neutral box identity (#299) changes only Eligible,
+// never these.
+
+// The thresholds of §7.
+const (
+ // MinMemPercent: a box is placeable only while its free memory is at
+ // least this share of its total, the memory alarm's own line.
+ MinMemPercent = 15
+ // MinDiskPercent: and its data volume's free space at least this
+ // share of the volume.
+ MinDiskPercent = 10
+ // MaxEdgesAge: and its edges file, which carries the headroom, is no
+ // older than this. Files republish every 5 minutes plus up to a
+ // minute of jitter, so a healthy box always has one within two
+ // intervals and a margin (A1-8).
+ MaxEdgesAge = 15 * time.Minute
+ // maxEdgesSkew is how far in the future a file's generation may be
+ // before it is not believed: box clocks are synchronised, not equal.
+ maxEdgesSkew = time.Minute
+)
+
+// ExcludeRoles drops every candidate whose roster role is not the default
+// one (roles is box id -> role, "" for the default). Legacy and platform
+// boxes, and any role this build does not know, take no new instances.
+// The order of cands is kept.
+func ExcludeRoles(cands []Candidate, roles map[string]string) []Candidate {
+ var out []Candidate
+ for _, c := range cands {
+ if roles[c.BoxID] == "" {
+ out = append(out, c)
+ }
+ }
+ return out
+}
+
+// headroomFresh reports whether a candidate's edges file is recent enough
+// at now for its headroom to be trusted.
+func headroomFresh(generation string, now time.Time) bool {
+ at, err := time.Parse(time.RFC3339, generation)
+ if err != nil {
+ return false
+ }
+ age := now.Sub(at)
+ return age <= MaxEdgesAge && age >= -maxEdgesSkew
+}
+
+// Placeable is §7's eligibility on top of Eligible's: the default role,
+// free memory at least MinMemPercent of the total, free disk at least
+// MinDiskPercent of the volume, and an edges file no older than
+// MaxEdgesAge at now. A box that publishes no totals is not placeable: its
+// headroom cannot be judged. The order of cands is kept.
+func Placeable(cands []Candidate, roles map[string]string, now time.Time) []Candidate {
+ var out []Candidate
+ for _, c := range ExcludeRoles(cands, roles) {
+ switch {
+ case c.MemTotalMiB <= 0 || c.MemFreeMiB*100 < c.MemTotalMiB*MinMemPercent:
+ case c.DiskTotalMiB <= 0 || c.DiskFreeMiB*100 < c.DiskTotalMiB*MinDiskPercent:
+ case !headroomFresh(c.Generation, now):
+ default:
+ out = append(out, c)
+ }
+ }
+ return out
+}
+
+// ChooseRoomiest is §7's choice among placeable candidates: prefer a box
+// running no instance of app (soft spread: it never refuses), then the
+// most free memory, then the greatest hexadecimal SHA-256 of host + "\x00"
+// + box id (claim P5-D2), then the smallest box id. The answer does not
+// depend on the order of cands. The bool is false only when cands is
+// empty.
+func ChooseRoomiest(host string, cands []Candidate, running map[string]Load, app string) (Candidate, bool) {
+ if len(cands) == 0 {
+ return Candidate{}, false
+ }
+ tie := func(ref string) string {
+ sum := sha256.Sum256([]byte(host + "\x00" + ref))
+ return hex.EncodeToString(sum[:])
+ }
+ sorted := append([]Candidate(nil), cands...)
+ sort.SliceStable(sorted, func(i, j int) bool {
+ a, b := sorted[i], sorted[j]
+ aRuns, bRuns := running[a.BoxID].Apps[app] > 0, running[b.BoxID].Apps[app] > 0
+ switch {
+ case aRuns != bRuns:
+ return !aRuns
+ case a.MemFreeMiB != b.MemFreeMiB:
+ return a.MemFreeMiB > b.MemFreeMiB
+ case tie(a.BoxID) != tie(b.BoxID):
+ return tie(a.BoxID) > tie(b.BoxID)
+ }
+ return a.BoxID < b.BoxID
+ })
+ return sorted[0], true
+}
+````
+
+````diff
+--- a/internal/boxchoice/boxchoice.go
++++ b/internal/boxchoice/boxchoice.go
+@@ -21,11 +21,15 @@
+ // region HA executor.
+ const Capability = "region-ha-1"
+
+-// Headroom is what a box reports it has free.
++// Headroom is what a box reports it has free, and (region HA spec 2 §7)
++// the totals the free figures are fractions of. A total of 0 is a writer
++// that predates it, or a reading that failed.
+ type Headroom struct {
+- MemFreeMiB int64 `json:"mem_free_mib"`
+- DiskFreeMiB int64 `json:"disk_free_mib"`
+- Awake int `json:"awake"`
++ MemFreeMiB int64 `json:"mem_free_mib"`
++ DiskFreeMiB int64 `json:"disk_free_mib"`
++ MemTotalMiB int64 `json:"mem_total_mib"`
++ DiskTotalMiB int64 `json:"disk_total_mib"`
++ Awake int `json:"awake"`
+ }
+
+ // Box is one box's edges file (control/edges/<box>.json, written by
+@@ -48,6 +52,9 @@
+ Capabilities []string `json:"capabilities"`
+ Draining bool `json:"draining"`
+ Headroom *Headroom `json:"headroom"`
++ // Generation is when the box last wrote the file (RFC 3339). Spec 2's
++ // policy trusts headroom only from a recent file (Placeable).
++ Generation string `json:"generation"`
+ // Fresh is the caller's verdict that the box's own status snapshot is
+ // still fresh. An edges file outlives its box, so the file alone is
+ // not liveness.
+@@ -76,10 +83,13 @@
+ // Candidate is one box that may be named as a host's executor (region HA
+ // spec 1 §2, "eligible box").
+ type Candidate struct {
+- BoxID string
+- Region string
+- MemFreeMiB, DiskFreeMiB int64
+- Awake int
++ BoxID string
++ Region string
++ MemFreeMiB, DiskFreeMiB int64
++ MemTotalMiB, DiskTotalMiB int64
++ Awake int
++ // Generation is the box's edges file's (Box.Generation).
++ Generation string
+ }
+
+ // Load is what claims already place on a box.
+@@ -132,9 +142,10 @@
+ case !slices.Contains(f.Capabilities, Capability):
+ case f.Draining, lost[f.BoxID]:
+ default:
+- c := Candidate{BoxID: f.BoxID, Region: rb.Cloud.Region}
++ c := Candidate{BoxID: f.BoxID, Region: rb.Cloud.Region, Generation: f.Generation}
+ if f.Headroom != nil {
+ c.MemFreeMiB, c.DiskFreeMiB, c.Awake = f.Headroom.MemFreeMiB, f.Headroom.DiskFreeMiB, f.Headroom.Awake
++ c.MemTotalMiB, c.DiskTotalMiB = f.Headroom.MemTotalMiB, f.Headroom.DiskTotalMiB
+ }
+ out = append(out, c)
+ }
+````
+
+````diff
+--- a/internal/edgesfile/snapshot.go
++++ b/internal/edgesfile/snapshot.go
+@@ -58,11 +58,18 @@
+ Capabilities []string `json:"capabilities"`
+ Draining bool `json:"draining"`
+ Headroom *struct {
+- MemFreeMiB int64 `json:"mem_free_mib"`
+- DiskFreeMiB int64 `json:"disk_free_mib"`
+- Awake int `json:"awake"`
++ MemFreeMiB int64 `json:"mem_free_mib"`
++ DiskFreeMiB int64 `json:"disk_free_mib"`
++ MemTotalMiB int64 `json:"mem_total_mib"`
++ DiskTotalMiB int64 `json:"disk_total_mib"`
++ Awake int `json:"awake"`
+ } `json:"headroom"`
+ Hosts map[string]json.RawMessage `json:"hosts"`
++
++ // Generation is when the box last wrote the file (RFC 3339), which
++ // region HA spec 2's placement policy reads as headroom's age.
++ Generation string `json:"generation"`
++
+ // The moves protocol (moves spec §11), read only by the Phase 1 move
+ // door's floor check (selectMoveBoxes); Task 13 adds the writer's side.
+ // MovesProtocol is the protocol the box speaks (absent: 0).
+@@ -246,10 +253,11 @@
+ Box: f.Box, BoxID: f.BoxID, EIP: f.EIP, Region: f.Region, Placement: f.Placement,
+ HibernationConfigured: f.HibernationConfigured, ServesSharedPool: f.ServesSharedPool,
+ AcceptsInbound: f.AcceptsInbound, SecretsKeyFP: f.SecretsKeyFP, Capabilities: f.Capabilities,
+- Draining: f.Draining, Fresh: f.Fresh,
++ Draining: f.Draining, Generation: f.Generation, Fresh: f.Fresh,
+ }
+ if f.Headroom != nil {
+- b.Headroom = &boxchoice.Headroom{MemFreeMiB: f.Headroom.MemFreeMiB, DiskFreeMiB: f.Headroom.DiskFreeMiB, Awake: f.Headroom.Awake}
++ b.Headroom = &boxchoice.Headroom{MemFreeMiB: f.Headroom.MemFreeMiB, DiskFreeMiB: f.Headroom.DiskFreeMiB,
++ MemTotalMiB: f.Headroom.MemTotalMiB, DiskTotalMiB: f.Headroom.DiskTotalMiB, Awake: f.Headroom.Awake}
+ }
+ out.Boxes = append(out.Boxes, b)
+ }
+````
+
+- [ ] **Step 4: Run the tests**
+
+Run: `nice -n 19 go test ./internal/boxchoice/ ./internal/edgesfile/ -count=1 && nice -n 19 go test ./internal/console/ -run 'Box|Eligib|Executor|RegionCand' -count=1`
+Expected: PASS.
+
+- [ ] **Step 5: Boundary list and census**
+
+````diff
+--- a/region_ha_boundary_test.go
++++ b/region_ha_boundary_test.go
+@@ -21,6 +21,7 @@
+ var regionHACoreFiles = []string{
+ "internal/lbhealth/lbhealth.go",
+ "internal/hostactors/roles.go",
++ "internal/boxchoice/policy.go",
+ }
+
+ // regionHAAdapterFiles are spec 2's AWS adapters. Each is named for AWS so
+````
+
+Add this comment line after the overlay's last comment line:
+
+````text
+# Task 4: the placement policy (internal/boxchoice policy.go; headroom totals and the edges file's generation carried through Eligible and edgesfile.Snapshot.BoxChoice). Envelope-only: no root-read unit changed or was added, and the legacy seal stands.
+````
+
+Then set these rows. A `@coverage` row replaces the overlay's row for the same directory, or is appended when there is none; every other row is appended:
+
+````text
+@coverage internal/boxchoice 1bfd561bcb48b51e60e63f99e086fd44d3b1be2160e28b3ed0253a14bbb9ff66
+@coverage internal/edgesfile da42c799d62e065817b5cce4f9a3ead1308dbd379615a85a7b14533c16d7e804
+````
+
+Run the census checks and `go test . -run TestRegionHA -count=1`. Expected: PASS.
+
+- [ ] **Step 6: Commit**
+
+```bash
+git add internal/boxchoice internal/edgesfile region_ha_boundary_test.go internal/provision/testdata/plan14-root-reads-region-ha-2a.tsv
+git commit -m "region HA 2a: the placement policy over the eligible set"
+```
+
+---
+
+### Task 5: Boxes publish their total memory and data volume size
+
+**Files:**
+- Modify: `internal/host/headroom.go` (`headroomFor` fills the totals; `capacityMiB`)
+- Modify: `internal/host/edgesfile.go` (`edgeHeadroom`'s two fields)
+- Modify: `internal/host/headroom_test.go`
+- Modify: the overlay
+
+**Interfaces:**
+- Consumes: `readHeadroom`, `readMemBytes`, `diskBytes` (unchanged).
+- Produces: `edgeHeadroom.MemTotalMiB`, `edgeHeadroom.DiskTotalMiB` (json `mem_total_mib`, `disk_total_mib`, omitempty), the fields Task 4's reader takes; `func capacityMiB(mem func() (total, avail *int64), disk func() (total, avail uint64, err error)) (memMiB, diskMiB int64)`.
+
+Headroom is published only with the region HA capability (`headroomFor` returns nil without it), so with region HA off the edges file is byte-identical to main's. `readHeadroom` and `readMemBytes` are accepted-legacy units and stay as they are: `headroomFor` calls the existing readers.
+
+- [ ] **Step 1: Write the failing tests**
+
+````diff
+--- a/internal/host/headroom_test.go
++++ b/internal/host/headroom_test.go
+@@ -1,6 +1,13 @@
+ package host
+
+-import "testing"
++import (
++ "context"
++ "encoding/json"
++ "errors"
++ "testing"
++
++ "github.com/carlosframework/platform/internal/release"
++)
+
+ func TestParseMemAvailable(t *testing.T) {
+ const meminfo = "MemTotal: 16000000 kB\nMemFree: 500000 kB\nMemAvailable: 2263040 kB\n"
+@@ -24,3 +31,47 @@
+ t.Fatal("flag on: headroom is nil")
+ }
+ }
++
++// Region HA spec 2 §7: the totals ride beside the free figures, and a
++// failed reading is 0, never a guess.
++func TestCapacityMiBReadsTotalsAndZeroesFailures(t *testing.T) {
++ gib := int64(8 << 30)
++ mem := func() (*int64, *int64) { return &gib, nil }
++ disk := func() (uint64, uint64, error) { return 100 << 30, 1 << 30, nil }
++ if m, d := capacityMiB(mem, disk); m != 8192 || d != 102400 {
++ t.Fatalf("capacity = %d, %d; want 8192, 102400", m, d)
++ }
++ noMem := func() (*int64, *int64) { return nil, nil }
++ noDisk := func() (uint64, uint64, error) { return 0, 0, errors.New("statfs") }
++ if m, d := capacityMiB(noMem, noDisk); m != 0 || d != 0 {
++ t.Fatalf("failed readings = %d, %d; want 0, 0", m, d)
++ }
++}
++
++func TestHeadroomCarriesTheDataVolumeSize(t *testing.T) {
++ h := headroomFor([]string{capRegionHA1}, t.TempDir(), edgesTestRegistry(t))
++ if h == nil || h.DiskTotalMiB <= 0 {
++ t.Fatalf("headroom %+v: want the data volume's size", h)
++ }
++}
++
++func TestEdgesFileWireCarriesHeadroomTotals(t *testing.T) {
++ s, reg := release.NewDirStore(t.TempDir()), edgesTestRegistry(t)
++ var last []byte
++ ident := edgesFile{Box: "annie", BoxID: "i-0abc", Capabilities: []string{"region-ha-1"},
++ Headroom: &edgeHeadroom{MemFreeMiB: 2210, DiskFreeMiB: 31200, Awake: 14, MemTotalMiB: 7900, DiskTotalMiB: 61000}}
++ if err := edgesFilePass(context.Background(), s, reg, ident, "1.2.3.4", &last); err != nil {
++ t.Fatal(err)
++ }
++ body, _, err := s.Get(context.Background(), "control/edges/annie.json")
++ if err != nil {
++ t.Fatal(err)
++ }
++ var raw map[string]json.RawMessage
++ if err := json.Unmarshal(body, &raw); err != nil {
++ t.Fatal(err)
++ }
++ if want := `{"mem_free_mib":2210,"disk_free_mib":31200,"awake":14,"mem_total_mib":7900,"disk_total_mib":61000}`; string(raw["headroom"]) != want {
++ t.Fatalf("headroom = %s, want %s", raw["headroom"], want)
++ }
++}
+````
+
+- [ ] **Step 2: Run them to see them fail**
+
+Run: `nice -n 19 go test ./internal/host/ -run 'Headroom|CapacityMiB' -count=1`
+Expected: FAIL to build (`undefined: capacityMiB`, `unknown field MemTotalMiB`).
+
+- [ ] **Step 3: Write the totals**
+
+````diff
+--- a/internal/host/headroom.go
++++ b/internal/host/headroom.go
+@@ -14,7 +14,23 @@
+ if len(caps) == 0 {
+ return nil
+ }
+- return readHeadroom(dataDir, awakeCount(reg))
++ h := readHeadroom(dataDir, awakeCount(reg))
++ h.MemTotalMiB, h.DiskTotalMiB = capacityMiB(readMemBytes, func() (uint64, uint64, error) { return diskBytes(dataDir) })
++ return h
++}
++
++// capacityMiB is the box's total memory and its data volume's size, in
++// MiB: the totals region HA spec 2's placement policy judges the free
++// figures against (§7). A reading that fails is 0, which the policy reads
++// as a box it cannot judge, and so does not place on.
++func capacityMiB(mem func() (total, avail *int64), disk func() (total, avail uint64, err error)) (memMiB, diskMiB int64) {
++ if total, _ := mem(); total != nil && *total > 0 {
++ memMiB = *total / (1 << 20)
++ }
++ if total, _, err := disk(); err == nil {
++ diskMiB = int64(total / (1 << 20))
++ }
++ return memMiB, diskMiB
+ }
+
+ // parseMemAvailableMiB returns the MemAvailable line in MiB, or 0.
+````
+
+````diff
+--- a/internal/host/edgesfile.go
++++ b/internal/host/edgesfile.go
+@@ -151,6 +151,11 @@
+ MemFreeMiB int64 `json:"mem_free_mib"`
+ DiskFreeMiB int64 `json:"disk_free_mib"`
+ Awake int `json:"awake"`
++ // MemTotalMiB and DiskTotalMiB are what the free figures are shares
++ // of (region HA spec 2 §7). Additive and omitted when 0, a reading
++ // that failed: a reader takes absent and 0 alike, as unknown.
++ MemTotalMiB int64 `json:"mem_total_mib,omitempty"`
++ DiskTotalMiB int64 `json:"disk_total_mib,omitempty"`
+ }
+
+ const capRegionHA1 = "region-ha-1"
+````
+
+- [ ] **Step 4: Run the tests**
+
+Run: `nice -n 19 go test ./internal/host/ -run 'Edges|Headroom|CapacityMiB|ParseMemAvailable' -count=1`
+Expected: PASS, including main's `TestEdgesFileRawWireBodyCarriesBoxEvidence` (the totals are omitted when zero).
+
+- [ ] **Step 5: Census**
+
+Add this comment line after the overlay's last comment line:
+
+````text
+# Task 5: the edges file's headroom carries the box's total memory and data volume size (headroomFor and capacityMiB, internal/host/headroom.go; edgeHeadroom's two additive fields). headroomFor calls the existing readMemBytes and diskBytes and adds no read of its own. Envelope-only: readHeadroom and readMemBytes are unchanged, no root-read unit changed or was added, and the legacy seal stands.
+````
+
+Then set these rows. A `@coverage` row replaces the overlay's row for the same directory, or is appended when there is none; every other row is appended:
+
+````text
+@coverage internal/host c1cbcb478b2d64b3114a3c33be7d847cf8b266694f42f0fab4e9b28a51bc7f9c
+````
+
+Run the census checks. Expected: PASS.
+
+- [ ] **Step 6: Commit**
+
+```bash
+git add internal/host/headroom.go internal/host/headroom_test.go internal/host/edgesfile.go internal/provision/testdata/plan14-root-reads-region-ha-2a.tsv
+git commit -m "region HA 2a: edges files carry total memory and disk"
+```
+
+---
+
+### Task 6: Creates and relocation targets choose by the policy
+
+**Files:**
+- Modify: `internal/hostops/executor.go` (`Choose`, `Permits`, `Capacity`, `placeable`)
+- Modify: `internal/hostops/executor_test.go`, `internal/hostrelocate/chooser_test.go` (fixtures publish roomy, fresh headroom)
+- Modify: the overlay
+
+**Interfaces:**
+- Consumes: `boxchoice.Placeable`, `boxchoice.ChooseRoomiest` (Task 4); `hostactors.Box.Role` (Task 3).
+- Produces: `func (x ExecutorChooser) Capacity(ctx context.Context, placement string) (int, error)`, which Task 12 publishes; `Choose` and `Permits` now apply the roles and thresholds. `ExecutorChooser` exists only with `CARLOS_REGION_HA` on (the console binds it only then, and the box only in its region HA relocation runtime), so the policy is dormant with it off.
+
+`ExecutorChooser.eligible` is an accepted-legacy unit (merge ruling 2) and does not change: the policy runs over its result. Relocation's `RegionChooser` uses `Choose` and `Permits`, so relocation targets follow the same policy.
+
+- [ ] **Step 1: Write the failing tests**
+
+````diff
+--- a/internal/hostops/executor_test.go
++++ b/internal/hostops/executor_test.go
+@@ -43,7 +43,12 @@
+ roster.Boxes[ref] = hostactors.Box{Enrollment: strings.Repeat(string("abcdef"[i]), 32), Namespace: "platform", BoxID: ref, Label: ref, Region: "ie", Pool: "default",
+ Cloud: hostactors.Cloud{Provider: "aws", Account: "123", Region: "eu-west-1", InstanceID: "i-" + ref}, EvidenceSHA256: strings.Repeat("c", 64), State: "active"}
+ snap.Boxes = append(snap.Boxes, boxchoice.Box{BoxID: ref, EIP: "203.0.113.1", Region: "eu-west-1", Placement: "ie", Fresh: true,
+- HibernationConfigured: &yes, ServesSharedPool: &yes, AcceptsInbound: &yes, SecretsKeyFP: "fp", Capabilities: []string{boxchoice.Capability}})
++ HibernationConfigured: &yes, ServesSharedPool: &yes, AcceptsInbound: &yes, SecretsKeyFP: "fp", Capabilities: []string{boxchoice.Capability},
++ // Equal, roomy headroom from a file a minute old: spec 2's
++ // policy (§7) admits every box, and the choice falls through
++ // to the hash unless a test changes it.
++ Headroom: &boxchoice.Headroom{MemFreeMiB: 4000, MemTotalMiB: 8000, DiskFreeMiB: 50000, DiskTotalMiB: 100000},
++ Generation: "2026-10-07T08:59:00Z"})
+ }
+ if edit != nil {
+ edit(&roster)
+@@ -68,9 +73,9 @@
+ Body: []byte(`{"host":"shop.bab.oncarlos.com","account":"bab","app":"jam","region":"ie"}`)}
+ }
+
+-func TestExecutorChooserSpreadsThenBalancesThenHashes(t *testing.T) {
+- ctx := context.Background()
+- x, _, _, loads := chooserFixture(t, "box-a", "box-b")
++func TestExecutorChooserSpreadsThenPrefersMemoryThenHashes(t *testing.T) {
++ ctx := context.Background()
++ x, _, snap, loads := chooserFixture(t, "box-a", "box-b")
+ absent := hostclaim.Observed{Host: "shop.bab.oncarlos.com", Absent: true}
+ first, err := x.Choose(ctx, absent, chooserRecord())
+ if err != nil {
+@@ -85,10 +90,77 @@
+ if got, _ := x.Choose(ctx, absent, chooserRecord()); got.Box != other {
+ t.Fatalf("soft spread: got %s, want %s", got.Box, other)
+ }
+- // Both run the app: the fewest hosts wins.
++ // Both run the app: the most free memory wins (spec 2 §7), whatever
++ // the host counts say.
+ loads[other] = boxchoice.Load{Hosts: 5, Apps: map[string]int{"bab/jam": 1}}
+- if got, _ := x.Choose(ctx, absent, chooserRecord()); got.Box != first.Box {
+- t.Fatalf("headroom: got %s, want %s", got.Box, first.Box)
++ for i := range snap.Boxes {
++ if snap.Boxes[i].BoxID == other {
++ snap.Boxes[i].Headroom.MemFreeMiB = 6000
++ }
++ }
++ if got, _ := x.Choose(ctx, absent, chooserRecord()); got.Box != other {
++ t.Fatalf("headroom: got %s, want %s", got.Box, other)
++ }
++}
++
++// withRole sets one box's roster role before the roster is signed.
++func withRole(ref, role string) func(*hostactors.Roster) {
++ return func(r *hostactors.Roster) {
++ b := r.Boxes[ref]
++ b.Role = role
++ r.Boxes[ref] = b
++ }
++}
++
++// Region HA spec 2 §7: the chooser, and so creates and relocation
++// targets, applies the roster's roles and the policy's thresholds after
++// Eligible. Permits answers the same set, and Capacity counts it.
++func TestExecutorChooserAppliesRolesAndHeadroom(t *testing.T) {
++ ctx := context.Background()
++ absent := hostclaim.Observed{Host: "shop.bab.oncarlos.com", Absent: true}
++ cases := []struct {
++ name string
++ roster func(*hostactors.Roster)
++ edges func(*boxchoice.Box)
++ }{
++ {"legacy role", withRole("box-a", hostactors.RoleLegacy), nil},
++ {"platform role", withRole("box-a", hostactors.RolePlatform), nil},
++ {"memory under 15%", nil, func(b *boxchoice.Box) { b.Headroom.MemFreeMiB = 1199 }},
++ {"disk under 10%", nil, func(b *boxchoice.Box) { b.Headroom.DiskFreeMiB = 9999 }},
++ {"edges file 16 minutes old", nil, func(b *boxchoice.Box) { b.Generation = "2026-10-07T08:44:00Z" }},
++ {"no headroom", nil, func(b *boxchoice.Box) { b.Headroom = nil }},
++ }
++ for _, tc := range cases {
++ t.Run(tc.name, func(t *testing.T) {
++ x, _, snap, _ := chooserFixtureWith(t, tc.roster, "box-a", "box-b")
++ for i := range snap.Boxes {
++ if snap.Boxes[i].BoxID == "box-a" && tc.edges != nil {
++ tc.edges(&snap.Boxes[i])
++ }
++ }
++ for range 4 {
++ got, err := x.Choose(ctx, absent, chooserRecord())
++ if err != nil || got.Box != "box-b" {
++ t.Fatalf("chose %+v %v, want box-b", got, err)
++ }
++ }
++ if err := x.Permits(ctx, absent, chooserRecord(), "box-a"); !errors.Is(err, ErrNoEligibleBox) {
++ t.Fatalf("Permits(box-a) = %v, want ErrNoEligibleBox", err)
++ }
++ if err := x.Permits(ctx, absent, chooserRecord(), "box-b"); err != nil {
++ t.Fatalf("Permits(box-b) = %v", err)
++ }
++ if n, err := x.Capacity(ctx, "ie"); err != nil || n != 1 {
++ t.Fatalf("Capacity = %d, %v; want 1", n, err)
++ }
++ })
++ }
++ x, _, _, _ := chooserFixture(t, "box-a", "box-b")
++ if n, err := x.Capacity(ctx, "ie"); err != nil || n != 2 {
++ t.Fatalf("Capacity with both placeable = %d, %v; want 2", n, err)
++ }
++ if _, err := x.Capacity(ctx, "us"); !errors.Is(err, ErrContract) {
++ t.Fatalf("Capacity of a region the roster lacks = %v, want ErrContract", err)
+ }
+ }
+
+````
+
+````diff
+--- a/internal/hostrelocate/chooser_test.go
++++ b/internal/hostrelocate/chooser_test.go
+@@ -5,6 +5,7 @@
+ "errors"
+ "slices"
+ "testing"
++ "time"
+
+ "github.com/carlosframework/platform/internal/boxchoice"
+ "github.com/carlosframework/platform/internal/hostclaim"
+@@ -13,13 +14,16 @@
+
+ // scopedChooser plugs create's chooser into the fixture's worker, scoped
+ // to namespace, with every fixture box fresh and eligible on its edges
+-// file and spa the most loaded.
++// file, equally roomy under spec 2's policy (§7), and spa the most loaded.
+ func scopedChooser(f *fixture, namespace string) RegionChooser {
+ yes := true
+ snap := boxchoice.Snapshot{Complete: true}
++ generation := f.w.Now().Add(-time.Minute).UTC().Format(time.RFC3339)
+ for _, ref := range []string{"src", "dst", "spa", "sp2"} {
+ snap.Boxes = append(snap.Boxes, boxchoice.Box{BoxID: ref, EIP: "203.0.113.1", Region: "eu-west-1", Placement: "ie", Fresh: true,
+- HibernationConfigured: &yes, ServesSharedPool: &yes, AcceptsInbound: &yes, SecretsKeyFP: "fp", Capabilities: []string{boxchoice.Capability}})
++ HibernationConfigured: &yes, ServesSharedPool: &yes, AcceptsInbound: &yes, SecretsKeyFP: "fp", Capabilities: []string{boxchoice.Capability},
++ Headroom: &boxchoice.Headroom{MemFreeMiB: 4000, MemTotalMiB: 8000, DiskFreeMiB: 50000, DiskTotalMiB: 100000},
++ Generation: generation})
+ }
+ loads := map[string]boxchoice.Load{"spa": {Hosts: 9, Apps: map[string]int{"bab/jam": 3}}}
+ return RegionChooser{Records: f.s, Namespace: namespace, Region: "ie",
+````
+
+- [ ] **Step 2: Run them to see them fail**
+
+Run: `nice -n 19 go test ./internal/hostops/ -run 'Executor|Chooser' -count=1`
+Expected: FAIL to build (`x.Capacity undefined`).
+
+- [ ] **Step 3: Apply the policy**
+
+````diff
+--- a/internal/hostops/executor.go
++++ b/internal/hostops/executor.go
+@@ -48,17 +48,19 @@
+ // r's placement is created on. Identity, placement and AWS region come from
+ // the roster; a box with a valid termination fence, or not active, is never
+ // chosen; a box on the claim's lost list, or owing a park-local chore on
+-// it, is never chosen.
++// it, is never chosen. Among the rest, region HA spec 2's policy (§7)
++// decides: placeable boxes only (placeable), then boxchoice.ChooseRoomiest.
+ func (x ExecutorChooser) Choose(ctx context.Context, obs hostclaim.Observed, r Record) (hostclaim.ExecutorSelection, error) {
+ o, cands, err := x.eligible(ctx, obs, r)
+ if err != nil {
+ return hostclaim.ExecutorSelection{}, err
+ }
++ cands = x.placeable(o, cands)
+ loads, err := x.Loads(ctx)
+ if err != nil {
+ return hostclaim.ExecutorSelection{}, err
+ }
+- best, ok := boxchoice.Choose(r.Holder.Host, cands, loads, r.Holder.Account+"/"+r.Holder.App)
++ best, ok := boxchoice.ChooseRoomiest(r.Holder.Host, cands, loads, r.Holder.Account+"/"+r.Holder.App)
+ if !ok {
+ return hostclaim.ExecutorSelection{}, ErrNoEligibleBox
+ }
+@@ -69,18 +71,44 @@
+ // in r's placement now (spec 1 §2, §5.8 "The start CAS ... refused ...
+ // when `to` is not eligible for the host"): the same read Choose makes,
+ // without choosing. A stale edges file, a missing capability or secrets
+-// key, a draining box, a fence, or an incomplete snapshot refuses.
++// key, a draining box, a fence, or an incomplete snapshot refuses, and so
++// does a box spec 2's policy does not place on (placeable): a legacy or
++// platform role, too little memory or disk, or stale headroom.
+ func (x ExecutorChooser) Permits(ctx context.Context, obs hostclaim.Observed, r Record, box string) error {
+- _, cands, err := x.eligible(ctx, obs, r)
++ o, cands, err := x.eligible(ctx, obs, r)
+ if err != nil {
+ return err
+ }
+- for _, c := range cands {
++ for _, c := range x.placeable(o, cands) {
+ if c.BoxID == box {
+ return nil
+ }
+ }
+ return fmt.Errorf("%w: %s", ErrNoEligibleBox, box)
++}
++
++// Capacity is how many boxes in placement could take a new instance in
++// the chooser's namespace and pool now: Choose's set, before the choice.
++// The console publishes it per region as the eligible-capacity metric
++// (region HA spec 2 §4, A1-11).
++func (x ExecutorChooser) Capacity(ctx context.Context, placement string) (int, error) {
++ r := Record{Holder: hostclaim.NewHolder{Region: placement}}
++ o, cands, err := x.eligible(ctx, hostclaim.Observed{Absent: true}, r)
++ if err != nil {
++ return 0, err
++ }
++ return len(x.placeable(o, cands)), nil
++}
++
++// placeable is region HA spec 2's policy over the eligible set (§7):
++// each box's roster role, then boxchoice.Placeable's thresholds at
++// x.Now(). Identity was Eligible's; nothing here reads a provider.
++func (x ExecutorChooser) placeable(o hostactors.Observed, cands []boxchoice.Candidate) []boxchoice.Candidate {
++ roles := make(map[string]string, len(cands))
++ for _, c := range cands {
++ roles[c.BoxID] = o.Roster.Boxes[c.BoxID].Role
++ }
++ return boxchoice.Placeable(cands, roles, x.Now())
+ }
+
+ // eligible is one read of the roster, the fences and the edges files,
+````
+
+- [ ] **Step 4: Run the tests**
+
+Run: `nice -n 19 go test ./internal/hostops/ -run 'Executor|Chooser|SelectExecutor|ProductionCreate' -count=1 && nice -n 19 go test ./internal/hostrelocate/ -count=1`
+Expected: PASS.
+
+- [ ] **Step 5: Census**
+
+Add this comment line after the overlay's last comment line:
+
+````text
+# Task 6: hostops.ExecutorChooser applies spec 2's policy in Choose and Permits (creates and relocation targets) and counts it in Capacity. ExecutorChooser.eligible, the accepted-legacy unit (merge ruling 2), is unchanged: the policy runs over its result. Envelope-only, and the legacy seal stands.
+````
+
+Then set these rows. A `@coverage` row replaces the overlay's row for the same directory, or is appended when there is none; every other row is appended:
+
+````text
+@coverage internal/hostops cede4be93f73d3e14547b19a811dd7fa5d8978d08c7fcf5a91bad80dd5176118
+````
+
+Run the census checks. Expected: PASS.
+
+- [ ] **Step 6: Commit**
+
+```bash
+git add internal/hostops/executor.go internal/hostops/executor_test.go internal/hostrelocate/chooser_test.go internal/provision/testdata/plan14-root-reads-region-ha-2a.tsv
+git commit -m "region HA 2a: creates and relocations choose by the placement policy"
+```
+
+---
+
+### Task 7: The box's own addresses, and `private_ip` in the edges file
+
+**Files:**
+- Create: `internal/host/boxaddress.go` (core: `BoxAddresses`, `AddressSource`, the `CARLOS_PRIVATE_IP` / `CARLOS_PUBLIC_IP` source)
+- Create: `internal/host/awsaddr.go` (the AWS adapter: IMDSv2 `local-ipv4` and `public-ipv4`)
+- Create: `internal/host/boxaddress_test.go`
+- Modify: `internal/host/edgesclaims.go` (`edgesClaims.addrs`)
+- Modify: `internal/host/edgesfile.go` (`edgesFile.PrivateIP`; `edgesFileWriteClaims` reads the source with region HA on)
+- Modify: `internal/host/edgesclaims_test.go`, `region_ha_boundary_test.go`, the overlay
+
+**Interfaces:**
+- Consumes: `imdsToken`, `imdsLeaf` (agent.go, unchanged).
+- Produces: `type BoxAddresses struct{ Private, Public string }`; `type AddressSource interface{ BoxAddresses() BoxAddresses }`; `envAddresses{getenv, provider}`; `imdsAddresses`; `func boxAddressSource(getenv func(string) string) AddressSource`; `edgesFile.PrivateIP` (json `private_ip`, omitempty), which Task 8 reads.
+
+With region HA on (the edges claims cache exists exactly then), every publication reads the address source afresh and publishes only what it reports: a blank read publishes no address, instead of holding the last EIP, which may since have moved to the region's load balancer (A1-12). With region HA off the publisher is main's, EIP hold included. `CARLOS_PRIVATE_IP` and `CARLOS_PUBLIC_IP`, when set, win field by field over IMDS; a set value that is not an IPv4 address publishes nothing for its field.
+
+**Census (flagged):** `edgesFileWriteClaims` is a kept authority row (F,J1). Its sites do not move; its digest does. The overlay re-digests it with its class, task and basis unchanged, and asks platform-1e to confirm. No seal moves.
+
+- [ ] **Step 1: Write the failing tests**
+
+`internal/host/boxaddress_test.go`:
+
+````go
+package host
+
+import (
+ "context"
+ "encoding/json"
+ "net/http"
+ "testing"
+
+ "github.com/carlosframework/platform/internal/release"
+)
+
+// staticAddresses is a fixed AddressSource.
+type staticAddresses BoxAddresses
+
+func (a staticAddresses) BoxAddresses() BoxAddresses { return BoxAddresses(a) }
+
+// countingAddresses counts reads, answering each from its list in turn.
+type countingAddresses struct {
+ answers []BoxAddresses
+ reads int
+}
+
+func (c *countingAddresses) BoxAddresses() BoxAddresses {
+ a := c.answers[min(c.reads, len(c.answers)-1)]
+ c.reads++
+ return a
+}
+
+func envOf(m map[string]string) func(string) string { return func(k string) string { return m[k] } }
+
+// The core source works with no provider adapter at all (§25).
+func TestEnvAddressesWithoutAProvider(t *testing.T) {
+ got := envAddresses{getenv: envOf(map[string]string{"CARLOS_PRIVATE_IP": " 10.0.1.5 ", "CARLOS_PUBLIC_IP": "198.51.100.7"})}.BoxAddresses()
+ if got != (BoxAddresses{Private: "10.0.1.5", Public: "198.51.100.7"}) {
+ t.Fatalf("got %+v", got)
+ }
+ if got := (envAddresses{getenv: envOf(nil)}).BoxAddresses(); got != (BoxAddresses{}) {
+ t.Fatalf("nothing set: %+v", got)
+ }
+ // Not IPv4: published as nothing.
+ got = envAddresses{getenv: envOf(map[string]string{"CARLOS_PRIVATE_IP": "fd00::5", "CARLOS_PUBLIC_IP": "box.example.com"})}.BoxAddresses()
+ if got != (BoxAddresses{}) {
+ t.Fatalf("not IPv4: %+v", got)
+ }
+}
+
+func TestEnvAddressesOverrideTheProviderFieldByField(t *testing.T) {
+ provider := &countingAddresses{answers: []BoxAddresses{{Private: "10.0.9.9", Public: "203.0.113.9"}}}
+ got := envAddresses{getenv: envOf(map[string]string{"CARLOS_PUBLIC_IP": "198.51.100.7"}), provider: provider}.BoxAddresses()
+ if got != (BoxAddresses{Private: "10.0.9.9", Public: "198.51.100.7"}) {
+ t.Fatalf("got %+v", got)
+ }
+ // Both set: the provider is not asked.
+ provider.reads = 0
+ envAddresses{getenv: envOf(map[string]string{"CARLOS_PRIVATE_IP": "10.0.1.5", "CARLOS_PUBLIC_IP": "198.51.100.7"}), provider: provider}.BoxAddresses()
+ if provider.reads != 0 {
+ t.Fatalf("provider asked %d times with both overrides set", provider.reads)
+ }
+ // A bad override publishes nothing for its field; the provider's
+ // answer does not replace it.
+ got = envAddresses{getenv: envOf(map[string]string{"CARLOS_PRIVATE_IP": "not-an-ip"}), provider: provider}.BoxAddresses()
+ if got != (BoxAddresses{Private: "", Public: "203.0.113.9"}) {
+ t.Fatalf("bad override: %+v", got)
+ }
+}
+
+func TestIMDSAddressesReadsBothLeaves(t *testing.T) {
+ leaves := map[string]string{"/latest/meta-data/local-ipv4": "10.0.1.5", "/latest/meta-data/public-ipv4": "203.0.113.4"}
+ s := imdsAddresses{token: func() string { return "tok" }, leaf: func(token, path string) (string, int) {
+ if token != "tok" {
+ t.Fatalf("token %q", token)
+ }
+ v, ok := leaves[path]
+ if !ok {
+ return "", http.StatusNotFound
+ }
+ return v, http.StatusOK
+ }}
+ if got := s.BoxAddresses(); got != (BoxAddresses{Private: "10.0.1.5", Public: "203.0.113.4"}) {
+ t.Fatalf("got %+v", got)
+ }
+ // No public address (a box without one, or the EIP gone): blank.
+ delete(leaves, "/latest/meta-data/public-ipv4")
+ if got := s.BoxAddresses(); got != (BoxAddresses{Private: "10.0.1.5"}) {
+ t.Fatalf("no public leaf: %+v", got)
+ }
+ noToken := imdsAddresses{token: func() string { return "" }, leaf: s.leaf}
+ if got := noToken.BoxAddresses(); got != (BoxAddresses{}) {
+ t.Fatalf("no token: %+v", got)
+ }
+}
+
+func readEdgesFile(t *testing.T, s release.Store, box string) edgesFile {
+ t.Helper()
+ body, _, err := s.Get(context.Background(), edgesFileKey(box))
+ if err != nil {
+ t.Fatal(err)
+ }
+ var f edgesFile
+ if err := json.Unmarshal(body, &f); err != nil {
+ t.Fatal(err)
+ }
+ return f
+}
+
+// Region HA spec 2 §6 (A1-12): with region HA on, every publication reads
+// the address source afresh, and a blank read publishes no address. The
+// last published EIP is never held: it may be the region's, which the
+// load balancer has taken.
+func TestEdgesFilePublishesOnlyWhatTheSourceReports(t *testing.T) {
+ s, reg := release.NewDirStore(t.TempDir()), edgesTestRegistry(t)
+ seedRoute(t, reg, "app.example.com")
+ claims := newEdgesClaims("annie", (&claimsBook{}).read)
+ src := &countingAddresses{answers: []BoxAddresses{
+ {Private: "10.0.1.5", Public: "203.0.113.9"},
+ {Private: "10.0.1.5"},
+ }}
+ claims.addrs = src
+ var last []byte
+ ident := edgesFile{Box: "annie", BoxID: "annie"}
+ // The loop's own reading (the eip argument) is not what is published.
+ if err := edgesFileWriteClaims(context.Background(), s, reg, ident, "198.51.100.1", &last, false, claims); err != nil {
+ t.Fatal(err)
+ }
+ if f := readEdgesFile(t, s, "annie"); f.EIP != "203.0.113.9" || f.PrivateIP != "10.0.1.5" {
+ t.Fatalf("first publication: eip %q private %q", f.EIP, f.PrivateIP)
+ }
+ if err := edgesFileWriteClaims(context.Background(), s, reg, ident, "", &last, false, claims); err != nil {
+ t.Fatal(err)
+ }
+ if f := readEdgesFile(t, s, "annie"); f.EIP != "" || f.PrivateIP != "10.0.1.5" {
+ t.Fatalf("after a blank public read: eip %q private %q, want no public address", f.EIP, f.PrivateIP)
+ }
+ if src.reads != 2 {
+ t.Fatalf("source read %d times, want once per publication", src.reads)
+ }
+}
+
+// With region HA off there is no claims cache, and the publisher is
+// main's: no private address, and a blank probe holds the last EIP.
+func TestEdgesFileWithRegionHAOffKeepsTheEIPHold(t *testing.T) {
+ s, reg := release.NewDirStore(t.TempDir()), edgesTestRegistry(t)
+ seedRoute(t, reg, "app.example.com")
+ var last []byte
+ ident := edgesFile{Box: "annie"}
+ if err := edgesFileWriteClaims(context.Background(), s, reg, ident, "203.0.113.9", &last, false, nil); err != nil {
+ t.Fatal(err)
+ }
+ if err := edgesFileWriteClaims(context.Background(), s, reg, ident, "", &last, true, nil); err != nil {
+ t.Fatal(err)
+ }
+ if f := readEdgesFile(t, s, "annie"); f.EIP != "203.0.113.9" || f.PrivateIP != "" {
+ t.Fatalf("eip %q private %q", f.EIP, f.PrivateIP)
+ }
+}
+````
+
+The existing claims tests get a fixed address source, never IMDS:
+
+````diff
+--- a/internal/host/edgesclaims_test.go
++++ b/internal/host/edgesclaims_test.go
+@@ -50,6 +50,11 @@
+
+ func publishWithClaims(t *testing.T, reg *registry.Registry, claims *edgesClaims) map[string]edgeHost {
+ t.Helper()
++ // These tests are about the hosts; the box's addresses come from a
++ // fixed source, never from IMDS. nil claims is region HA off.
++ if claims != nil {
++ claims.addrs = staticAddresses{Public: "1.2.3.4"}
++ }
+ st := release.NewDirStore(t.TempDir())
+ var last []byte
+ if err := edgesFileWriteClaims(context.Background(), st, reg, edgesFile{Box: "annie", BoxID: "annie"}, "1.2.3.4", &last, false, claims); err != nil {
+````
+
+- [ ] **Step 2: Run them to see them fail**
+
+Run: `nice -n 19 go test ./internal/host/ -run 'Address|IMDS|EdgesFile' -count=1`
+Expected: FAIL to build (`undefined: BoxAddresses`, `claims.addrs undefined`).
+
+- [ ] **Step 3: Write the source and the publisher's change**
+
+`internal/host/boxaddress.go`:
+
+````go
+package host
+
+import (
+ "net"
+ "strings"
+)
+
+// BoxAddresses are the addresses this box publishes in its edges file for
+// other edges to dial (region HA spec 2 §6): Private for an edge in the
+// same region, Public for one in another. "" is an address the source
+// could not report, and nothing is published for it.
+type BoxAddresses struct {
+ Private string
+ Public string
+}
+
+// AddressSource reads this box's addresses afresh on every call. The edges
+// file publisher asks it before every publication and publishes only what
+// it reports, so an address the box no longer holds is never republished.
+type AddressSource interface {
+ BoxAddresses() BoxAddresses
+}
+
+// envAddresses is the provider-neutral source (§25): CARLOS_PRIVATE_IP and
+// CARLOS_PUBLIC_IP from host.env. Each, when set, is the answer for its
+// own field, and a value that is not an IPv4 address publishes nothing for
+// it: an operator's setting is never silently replaced. A field left unset
+// comes from provider, the cloud's adapter, when there is one.
+type envAddresses struct {
+ getenv func(string) string
+ provider AddressSource
+}
+
+func (s envAddresses) BoxAddresses() BoxAddresses {
+ private, privateSet := s.field("CARLOS_PRIVATE_IP")
+ public, publicSet := s.field("CARLOS_PUBLIC_IP")
+ if (!privateSet || !publicSet) && s.provider != nil {
+ p := s.provider.BoxAddresses()
+ if !privateSet {
+ private = ipv4Only(p.Private)
+ }
+ if !publicSet {
+ public = ipv4Only(p.Public)
+ }
+ }
+ return BoxAddresses{Private: private, Public: public}
+}
+
+// field reads one override: its IPv4 address, and whether it was set.
+func (s envAddresses) field(key string) (string, bool) {
+ v := strings.TrimSpace(s.getenv(key))
+ if v == "" {
+ return "", false
+ }
+ return ipv4Only(v), true
+}
+
+// ipv4Only is s when it is an IPv4 address, in its canonical form, and ""
+// otherwise.
+func ipv4Only(s string) string {
+ ip := net.ParseIP(strings.TrimSpace(s))
+ if ip == nil || ip.To4() == nil {
+ return ""
+ }
+ return ip.To4().String()
+}
+````
+
+`internal/host/awsaddr.go`:
+
+````go
+package host
+
+import (
+ "net/http"
+ "time"
+)
+
+// imdsAddresses is the AWS adapter of AddressSource (region HA spec 2
+// §25): IMDSv2's local-ipv4 and public-ipv4, read on every call. IMDS
+// reports only addresses this instance holds, so once the region's EIP has
+// moved to the load balancer no box reports it (A1-12). A leaf that does
+// not answer 200 is blank.
+type imdsAddresses struct {
+ token func() string
+ leaf func(token, path string) (string, int)
+}
+
+func newIMDSAddresses() imdsAddresses {
+ client := &http.Client{Timeout: 2 * time.Second}
+ return imdsAddresses{
+ token: func() string { return imdsToken(client) },
+ leaf: func(token, path string) (string, int) { return imdsLeaf(client, token, path) },
+ }
+}
+
+func (s imdsAddresses) BoxAddresses() BoxAddresses {
+ token := s.token()
+ if token == "" {
+ return BoxAddresses{}
+ }
+ read := func(path string) string {
+ v, code := s.leaf(token, path)
+ if code != http.StatusOK {
+ return ""
+ }
+ return v
+ }
+ return BoxAddresses{Private: read("/latest/meta-data/local-ipv4"), Public: read("/latest/meta-data/public-ipv4")}
+}
+
+// boxAddressSource is what the edges file publisher reads with region HA
+// on: host.env's overrides, then IMDS on EC2.
+func boxAddressSource(getenv func(string) string) AddressSource {
+ return envAddresses{getenv: getenv, provider: newIMDSAddresses()}
+}
+````
+
+````diff
+--- a/internal/host/edgesclaims.go
++++ b/internal/host/edgesclaims.go
+@@ -4,6 +4,7 @@
+ "context"
+ "log"
+ "math"
++ "os"
+ "sort"
+ "time"
+
+@@ -71,11 +72,15 @@
+ passDeadline time.Duration
+ logf func(format string, a ...any)
+ failingRun int // consecutive passes with a failed claim read
++ // addrs is the box's own addresses, read before every publication
++ // (region HA spec 2 §6). It rides here because this cache exists
++ // exactly when region HA is on.
++ addrs AddressSource
+ }
+
+ func newEdgesClaims(self string, read func(context.Context, string) (hostclaim.Observed, error)) *edgesClaims {
+ return &edgesClaims{self: self, read: read, now: time.Now, entries: map[string]edgesClaimEntry{},
+- passDeadline: edgesClaimPassDeadline, logf: log.Printf}
++ passDeadline: edgesClaimPassDeadline, logf: log.Printf, addrs: boxAddressSource(os.Getenv)}
+ }
+
+ // relocatableRow is spec 1 §2's relocatable host as a registry row shows
+````
+
+````diff
+--- a/internal/host/edgesfile.go
++++ b/internal/host/edgesfile.go
+@@ -205,6 +205,10 @@
+ Box string `json:"box"`
+ EIP string `json:"eip"`
+ Region string `json:"region,omitempty"`
++ // PrivateIP is the address an edge in this box's region dials (region
++ // HA spec 2 §6). Additive and omitempty: a reader that does not know
++ // it, or finds it absent, dials EIP, as before.
++ PrivateIP string `json:"private_ip,omitempty"`
+ // Placement is this box's CARLOS placement code — CARLOS_REGION verbatim
+ // ("ie", "uk"), the OTHER region vocabulary (see Region above, issue
+ // #329). Additive and omitempty like Region: a pre-placement body parses
+@@ -591,7 +595,15 @@
+ hosts[key] = edgeHost{Role: role, Wildcard: true, Alias: alias, Account: acct, App: app, Stamp: stamp}
+ }
+
+- if eip == "" && *last != nil {
++ var privateIP string
++ if claims != nil && claims.addrs != nil {
++ // Region HA spec 2 §6 (A1-12): publish only what the box's address
++ // source reports now. A blank read publishes no address, never the
++ // last one: that may be the region's EIP, which the load balancer
++ // has since taken.
++ a := claims.addrs.BoxAddresses()
++ eip, privateIP = a.Public, a.Private
++ } else if eip == "" && *last != nil {
+ var prior edgesFile
+ if err := json.Unmarshal(*last, &prior); err == nil {
+ eip = prior.EIP
+@@ -599,7 +611,7 @@
+ }
+
+ f := ident
+- f.V, f.EIP, f.Hosts, f.Generation = 1, eip, hosts, ""
++ f.V, f.EIP, f.PrivateIP, f.Hosts, f.Generation = 1, eip, privateIP, hosts, ""
+ gateCopy := f
+ gateCopy.Headroom = nil // a changing reading is not, by itself, a reason to write
+ gate, err := json.Marshal(gateCopy)
+````
+
+- [ ] **Step 4: Run the tests**
+
+Run: `nice -n 19 go test ./internal/host/ -run 'Address|IMDS|EdgesFile|Edges|Headroom' -count=1`
+Expected: PASS.
+
+- [ ] **Step 5: Boundary list and census**
+
+````diff
+--- a/region_ha_boundary_test.go
++++ b/region_ha_boundary_test.go
+@@ -22,11 +22,14 @@
+ "internal/lbhealth/lbhealth.go",
+ "internal/hostactors/roles.go",
+ "internal/boxchoice/policy.go",
++ "internal/host/boxaddress.go",
+ }
+
+ // regionHAAdapterFiles are spec 2's AWS adapters. Each is named for AWS so
+ // a reader can see where the provider lives.
+-var regionHAAdapterFiles = []string{}
++var regionHAAdapterFiles = []string{
++ "internal/host/awsaddr.go",
++}
+
+ // regionHAPackages are the packages spec 2 touches. Every S3 client built
+ // in them goes through s3compat.Options(), so CARLOS_S3_ENDPOINT applies.
+````
+
+Add this comment line after the overlay's last comment line:
+
+````text
+# Task 7: the box's own addresses (internal/host boxaddress.go, the provider-neutral source over CARLOS_PRIVATE_IP and CARLOS_PUBLIC_IP; awsaddr.go, the IMDS adapter) and the edges file's private_ip. edgesFileWriteClaims (authority, F,J1) is re-digested with its class, task, basis and sites unchanged: with region HA on it reads the address source before every publication and no longer holds a blank EIP. It adds no registry, /data, release or claim read, and nothing runs as root that did not before; the legacy seal stands. Re-digesting an authority row is flagged for platform-1e's confirmation.
+````
+
+Then set these rows. A `@coverage` row replaces the overlay's row for the same directory, or is appended when there is none; every other row is appended:
+
+````text
+@coverage internal/host 5d9945ec79672e86958261f28dc398f16f1ca51511deb8b48a271a200733ba3a
+internal/host/edgesfile.go edgesFileWriteClaims authority F,J1 root-route;kept=F;scope=protected-decision-only registry:Account@10835,registry:Alias@5297,registry:Alias@5332,registry:Alias@5754,registry:Alias@5911,registry:Alias@5981,registry:Alias@6013,registry:Alias@6482,registry:Alias@8772,registry:App@10847,registry:DNSStatus@5429,registry:DNSStatus@5920,registry:Host@10480,registry:Host@2878,registry:Host@5389,registry:Host@6234,registry:Host@6567,registry:Host@8819,registry:Host@8866,registry:Host@8916,registry:Host@8951,registry:Host@9000,registry:Platform@2921,registry:Provisioned@2935,registry:Replica@2959,registry:Status@877,whole:Route@3212,whole:Route@323,whole:Route@3505,whole:Route@351,whole:RouteAlias@5282,whole:RouteAlias@8725,whole:RoutePath@3520,whole:WildcardDomain@8905 53f9240b5f842da209a8499f821c2c06cabae4c26a1ac5c1a45baa38e1a25d1f
+````
+
+The appended row is `edgesFileWriteClaims`'s, re-digested: its kinds are main's row's, unchanged (check with the scratch test). Run the census checks and `go test . -run TestRegionHA -count=1`. Expected: PASS.
+
+- [ ] **Step 6: Commit**
+
+```bash
+git add internal/host/boxaddress.go internal/host/awsaddr.go internal/host/boxaddress_test.go internal/host/edgesclaims.go internal/host/edgesclaims_test.go internal/host/edgesfile.go region_ha_boundary_test.go internal/provision/testdata/plan14-root-reads-region-ha-2a.tsv
+git commit -m "region HA 2a: publish the box's private address and only what its source reports"
+```
+
+---
+
+### Task 8: Forwarding dials the private address within a region
+
+**Files:**
+- Modify: `internal/edge/homemap.go` (`edgesFileView` reads `private_ip`, `region`, `placement`; `dialAddress`; `foldHomes`)
+- Create: `internal/edge/homemap_private_test.go`
+- Modify: the overlay
+
+**Interfaces:**
+- Consumes: `private_ip` from Task 7's writer.
+- Produces: `func dialAddress(f, self edgesFileView, selfKnown bool) string`. `homeEntry.eip` is now the address to dial, so `Home`, `ExactHome` and `Owner` answer it, and the backhaul transport, keyed by that address, dials it.
+
+The hop is unchanged: one hop, today's credential and mesh key, the TLS name the host's. Only the address dialled changes: a box in this box's own AWS region and placement, publishing `private_ip`, is dialled there; every other box at its EIP, as before. An edge that cannot find its own edges file knows no region of its own and dials public addresses only.
+
+- [ ] **Step 1: Write the failing tests**
+
+`internal/edge/homemap_private_test.go`:
+
+````go
+package edge
+
+import (
+ "net/http"
+ "testing"
+)
+
+// Region HA spec 2 §6: within a region an edge dials the home box at its
+// private address; between regions, at the home box's own public address.
+// A reader of a file with no private_ip dials the public address, as
+// before, and so does an edge that cannot find its own file.
+func TestHomeMapDialsPrivateWithinTheRegionAndPublicBetween(t *testing.T) {
+ s := meshTestStore(t)
+ putJSON(t, s, "control/edges/annie.json", `{"v":1,"box":"annie","eip":"1.1.1.1","private_ip":"10.0.1.1","region":"eu-west-1","placement":"ie","hosts":{}}`)
+ putJSON(t, s, "control/edges/betty.json", `{"v":1,"box":"betty","eip":"2.2.2.2","private_ip":"10.0.1.2","region":"eu-west-1","placement":"ie","hosts":{"b.test":{"role":"home"}}}`)
+ putJSON(t, s, "control/edges/carla.json", `{"v":1,"box":"carla","eip":"3.3.3.3","private_ip":"10.1.1.3","region":"eu-west-2","placement":"uk","hosts":{"c.test":{"role":"home"}}}`)
+ putJSON(t, s, "control/edges/dora.json", `{"v":1,"box":"dora","eip":"4.4.4.4","region":"eu-west-1","placement":"ie","hosts":{"d.test":{"role":"home"}}}`)
+ putJSON(t, s, "control/edges/edna.json", `{"v":1,"box":"edna","eip":"5.5.5.5","private_ip":"10.0.1.5","region":"eu-west-1","placement":"fleet/shop","hosts":{"e.test":{"role":"home"}}}`)
+ putJSON(t, s, "control/edges/fern.json", `{"v":1,"box":"fern","eip":"","private_ip":"10.0.1.6","region":"eu-west-1","placement":"ie","hosts":{"f.test":{"role":"home"}}}`)
+ m := newHomeMap(s, "annie")
+ refreshHomeMap(t, m)
+ for host, want := range map[string]string{
+ "b.test": "10.0.1.2", // same region and placement: private
+ "c.test": "3.3.3.3", // another region: its own public address
+ "d.test": "4.4.4.4", // a writer with no private_ip: public, as before
+ "e.test": "5.5.5.5", // same AWS region, another placement: public
+ "f.test": "10.0.1.6", // no public address, but private within the region
+ } {
+ if addr, _, ok := m.Home(host); !ok || addr != want {
+ t.Errorf("%s dials %q (ok=%v), want %q", host, addr, ok, want)
+ }
+ }
+ // An edge whose own file is not in the map knows no region of its own:
+ // every dial is public, and a box with no public address is no target.
+ m = newHomeMap(s, "ghost")
+ refreshHomeMap(t, m)
+ if addr, _, _ := m.Home("b.test"); addr != "2.2.2.2" {
+ t.Errorf("without its own file: b.test dials %q, want 2.2.2.2", addr)
+ }
+ if _, _, ok := m.Home("f.test"); ok {
+ t.Error("without its own file, a box with no public address became a target")
+ }
+}
+
+// The whole hop, dialled privately: the home box's public address is
+// unroutable (TEST-NET-1), so only a dial to its private address reaches
+// it. It is still one hop.
+func TestBackhaulDialsThePrivateAddressWithinTheRegion(t *testing.T) {
+ e, m, _ := hopEdge(t, "annie", "local.test")
+ rec, private := fakeHomeEdge(t, nil)
+ putJSON(t, m.s, "control/edges/annie.json", `{"v":1,"box":"annie","eip":"192.0.2.10","region":"eu-west-1","placement":"ie","hosts":{"local.test":{"role":"home"}}}`)
+ putJSON(t, m.s, "control/edges/betty.json", `{"v":1,"box":"betty","eip":"192.0.2.11","private_ip":"`+private+`","region":"eu-west-1","placement":"ie","hosts":{"example.com":{"role":"home"}}}`)
+ refreshHomeMap(t, m)
+
+ resp := hopRequest(t, e, "example.com", nil)
+ if resp.StatusCode != http.StatusOK {
+ t.Fatalf("status = %d, want 200 through the private address", resp.StatusCode)
+ }
+ if body := readBody(t, resp); body != homeEdgeBody {
+ t.Fatalf("body = %q", body)
+ }
+ if seen := rec.get(); seen.n != 1 || seen.hop != "1" || seen.sni != "example.com" {
+ t.Fatalf("home edge saw %+v, want one verified hop for example.com", seen)
+ }
+}
+````
+
+- [ ] **Step 2: Run them to see them fail**
+
+Run: `nice -n 19 go test ./internal/edge/ -run 'DialsPrivate|PrivateAddress' -count=1`
+Expected: FAIL: `b.test dials "2.2.2.2" (ok=true), want "10.0.1.2"`, and the private backhaul answers 502 (its dial to the unroutable EIP times out).
+
+- [ ] **Step 3: Choose the dial address**
+
+````diff
+--- a/internal/edge/homemap.go
++++ b/internal/edge/homemap.go
+@@ -132,6 +132,25 @@
+ Box string `json:"box"`
+ EIP string `json:"eip"`
+ Hosts map[string]edgesHostView `json:"hosts"`
++ // PrivateIP, Region and Placement choose the dial address (region HA
++ // spec 2 §6, dialAddress). All three are absent from a writer that
++ // predates them, and the dial is then EIP, as before.
++ PrivateIP string `json:"private_ip"`
++ Region string `json:"region"`
++ Placement string `json:"placement"`
++}
++
++// dialAddress is where this edge dials f's box (region HA spec 2 §6): its
++// private address when it publishes one and both boxes publish the same
++// AWS region and placement, so the dial stays inside the region's
++// network; otherwise its public address, as before. self is this box's
++// own file, which selfKnown says was found. "" is nothing to dial.
++func dialAddress(f, self edgesFileView, selfKnown bool) string {
++ if selfKnown && f.PrivateIP != "" && f.Region != "" && f.Placement != "" &&
++ f.Region == self.Region && f.Placement == self.Placement {
++ return f.PrivateIP
++ }
++ return f.EIP
+ }
+
+ // edgesHostView is the per-host object. An object rather than a bare string is
+@@ -161,6 +180,8 @@
+
+ // homeEntry is one folded answer: where to dial, and which box that is.
+ type homeEntry struct {
++ // eip is the address to dial: dialAddress's answer, the home box's
++ // private address within its region and its EIP from anywhere else.
+ eip string
+ box string
+ // platform carries the publisher's class mark through the fold.
+@@ -532,9 +553,11 @@
+ // target would have the edge dial itself. Excluded by the file's own box
+ // field rather than by its object name, because the field is what the
+ // publisher stamps and what self is derived to match.
+-// - NO EIP. Task 1 publishes a box with no discoverable public IP as
+-// eip:"" so the DR runbook still gets its hosts list. There is nothing to
+-// dial, so it is not a target.
++// - NO DIAL ADDRESS. Task 1 publishes a box with no discoverable public IP
++// as eip:"" so the DR runbook still gets its hosts list. There is
++// nothing to dial, so it is not a target. The address is dialAddress's:
++// a box in this box's own region is dialled at its private address when
++// it publishes one (region HA spec 2 §6), every other at its EIP.
+ // - A ROLE THIS BUILD DOES NOT KNOW. Only "home" routes. The role field
+ // exists so read-local roles can be added later; a reader that treated
+ // every role as a home would send traffic to a box that had declared it
+@@ -552,22 +575,30 @@
+ keys = append(keys, k)
+ }
+ slices.Sort(keys)
++ var own edgesFileView
++ ownKnown := false
++ for _, k := range keys {
++ if f := files[k]; self != "" && f.Box == self {
++ own, ownKnown = f, true
++ }
++ }
+ hosts := make(map[string]homeEntry)
+ for _, k := range keys {
+ f := files[k]
+ if f.Box == "" || f.Box == self {
+ continue
+ }
++ addr := dialAddress(f, own, ownKnown)
+ for host, h := range f.Hosts {
+ if host == "" {
+ continue
+ }
+ switch h.Role {
+ case edgeRoleHome:
+- // A dial target needs an address. A file with no EIP (a box
++ // A dial target needs an address. A file with none (a box
+ // with no discoverable public IP) still carries its cert
+ // halves below; its dial halves are simply not dialable.
+- if f.EIP == "" {
++ if addr == "" {
+ if !h.Wildcard || !strings.HasPrefix(host, "*.") {
+ continue
+ }
+@@ -583,7 +614,7 @@
+ default:
+ continue
+ }
+- in := homeEntry{eip: f.EIP, box: f.Box, platform: h.Platform && !h.Alias && !h.Wildcard, alias: h.Alias, wildcard: h.Wildcard, dial: h.Role == edgeRoleHome,
++ in := homeEntry{eip: addr, box: f.Box, platform: h.Platform && !h.Alias && !h.Wildcard, alias: h.Alias, wildcard: h.Wildcard, dial: h.Role == edgeRoleHome,
+ account: h.Account, app: h.App, stamp: h.Stamp}
+ prev, dup := hosts[host]
+ if !dup {
+````
+
+- [ ] **Step 4: Run the tests**
+
+Run: `nice -n 19 go test ./internal/edge/ -run 'HomeMap|Backhaul|Hop|KnownHost|DialsPrivate|PrivateAddress' -count=1`
+Expected: PASS.
+
+- [ ] **Step 5: Census**
+
+Add this comment line after the overlay's last comment line:
+
+````text
+# Task 8: forwarding dials the home box's private address within its region and its public address between regions (internal/edge homemap.go: dialAddress, edgesFileView's private_ip, region and placement). The home map is not a root-read unit. Envelope-only, and the legacy seal stands.
+````
+
+Then set these rows. A `@coverage` row replaces the overlay's row for the same directory, or is appended when there is none; every other row is appended:
+
+````text
+@coverage internal/edge 11a6e94dbe8025c05d75509e56701e90092a225b191aee0a45dbd7307f660e31
+````
+
+Run the census checks. Expected: PASS.
+
+- [ ] **Step 6: Commit**
+
+```bash
+git add internal/edge/homemap.go internal/edge/homemap_private_test.go internal/provision/testdata/plan14-root-reads-region-ha-2a.tsv
+git commit -m "region HA 2a: dial a box's private address within its region"
+```
+
+---
+
+### Task 9: The load balancer health answer
+
+**Files:**
+- Create: `internal/edge/lbhealth.go`, `internal/edge/lbhealth_test.go`
+- Modify: `internal/edge/edge.go` (`Edge.lbHealth`, set in `New`; `Watch` beats; `httpFallbackHandler` answers first)
+- Modify: `internal/edge/homemap.go` (`homeMap.health`; `Refresh` latches and stamps)
+- Modify: `internal/edge/certsync.go` (`certMirror.health`; a landed `Refresh` latches)
+- Modify: `internal/activator/activator.go` (`lastSweep`, `LastSweep`)
+- Create: `internal/activator/lastsweep_test.go`
+- Modify: `internal/host/claimwiring.go` (`startRegional` marks the box hosting)
+- Create: `internal/host/claimwiring_hosting_test.go`
+- Modify: `region_ha_boundary_test.go`, the overlay
+
+**Interfaces:**
+- Consumes: Task 2's `lbhealth.State` and `lbhealth.Serve`.
+- Produces: `func lbHealthFromEnv(getenv func(string) string, act Activator) *lbhealth.State` (nil with region HA off); `func (e *Edge) lbHealthAnswer(w http.ResponseWriter, r *http.Request) bool`; `func (a *activator.Service) LastSweep() time.Time`; the hosting hook `lbhealth.Default.SetHosting(true)` in `ClaimWiring.startRegional`. **Seam for 2b:** 2b's mode selection takes `SetHosting` over; joining mode never sets it.
+
+The answer is served on port 80 only (the `:80` handler, `httpFallbackHandler`), ahead of every host-based branch, and reads only in-memory state. Each condition is fed by the loop it describes:
+
+| Condition | Fed by |
+|---|---|
+| hosting | `ClaimWiring.startRegional`: the claim agent only starts with its host-actor enrollment and a matching roster row |
+| forwarding | `homeMap.Refresh`: the first successful map load and the first successful mesh key load, latched |
+| certificates | `certMirror.Refresh`: the first refresh that lands `control/certs/` in the local cache the handshake reads, latched |
+| edge | `Edge.Watch`: every completed registry poll (every 2 seconds) |
+| activator | `activator.Service.LastSweep`: every finished sweep (at most 30 seconds apart) |
+| refresh | `homeMap.Refresh`: every completed attempt, failed or not |
+
+`RunWithActivatorReady`, `Edge.ServeHTTP` and `runAgent` are accepted-legacy or kept authority units; none changes. The edge's state is `lbhealth.Default` (process-wide) because the agent runs the edge and the activator in one process; tests give each edge its own `State`.
+
+- [ ] **Step 1: Write the failing tests**
+
+`internal/edge/lbhealth_test.go`:
+
+````go
+package edge
+
+import (
+ "context"
+ "errors"
+ "net/http"
+ "net/http/httptest"
+ "strings"
+ "testing"
+ "time"
+
+ "golang.org/x/crypto/acme/autocert"
+
+ "github.com/carlosframework/platform/internal/lbhealth"
+ "github.com/carlosframework/platform/internal/release"
+)
+
+// sweeper is an Activator whose only working method is the sweep beat.
+type sweeper struct {
+ Activator
+ at time.Time
+}
+
+func (s sweeper) LastSweep() time.Time { return s.at }
+
+func lbEnv(v string) func(string) string {
+ return func(k string) string {
+ if k == "CARLOS_REGION_HA" {
+ return v
+ }
+ return ""
+ }
+}
+
+func TestLBHealthStateFollowsRegionHA(t *testing.T) {
+ if s := lbHealthFromEnv(lbEnv(""), nil); s != nil {
+ t.Fatal("region HA unset armed the health answer")
+ }
+ if s := lbHealthFromEnv(lbEnv("0"), nil); s != nil {
+ t.Fatal("region HA off armed the health answer")
+ }
+ at := time.Now()
+ s := lbHealthFromEnv(lbEnv("1"), nil)
+ if s != lbhealth.Default {
+ t.Fatal("region HA on: not the process's state")
+ }
+ t.Cleanup(func() { lbhealth.Default.SetActivator(nil) })
+ lbHealthFromEnv(lbEnv("true"), sweeper{at: at})
+ if got := lbhealth.Default.Failing(at); strings.Contains(strings.Join(got, ","), lbhealth.CondActivator) {
+ t.Fatalf("the activator's sweep was not wired: %v", got)
+ }
+}
+
+// fallbackGet drives the :80 handler with one request.
+func fallbackGet(e *Edge, method, path, host string) *httptest.ResponseRecorder {
+ w := httptest.NewRecorder()
+ r := httptest.NewRequest(method, path, nil)
+ r.Host = host
+ e.httpFallbackHandler().ServeHTTP(w, r)
+ return w
+}
+
+// healthyState meets every condition now.
+func healthyState() *lbhealth.State {
+ s := &lbhealth.State{}
+ now := time.Now()
+ s.SetHosting(true)
+ s.MapLoaded()
+ s.KeyLoaded()
+ s.CertsLoaded()
+ s.EdgeBeat(now)
+ s.RefreshAttempted(now)
+ s.SetActivator(func() time.Time { return now })
+ return s
+}
+
+// The answer is on :80, for any host (the load balancer sends the
+// target's address), GET or HEAD, ahead of the redirect and unknown-host
+// branches. With region HA off the path is ordinary :80 traffic.
+func TestLBHealthAnswersOnPort80ForAnyHost(t *testing.T) {
+ e, err := New(openReg(t), nil)
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(e.probers.Stop)
+ e.lbHealth = healthyState()
+ for _, host := range []string{"10.0.1.7", "10.0.1.7:80", "unknown.example.com"} {
+ w := fallbackGet(e, http.MethodGet, lbhealth.Path, host)
+ if w.Code != http.StatusOK || w.Body.String() != "ok\n" {
+ t.Fatalf("host %s: %d %q", host, w.Code, w.Body.String())
+ }
+ }
+ if w := fallbackGet(e, http.MethodHead, lbhealth.Path, "10.0.1.7"); w.Code != http.StatusOK {
+ t.Fatalf("HEAD: %d", w.Code)
+ }
+ joining := healthyState()
+ joining.SetHosting(false)
+ e.lbHealth = joining
+ if w := fallbackGet(e, http.MethodGet, lbhealth.Path, "10.0.1.7"); w.Code != http.StatusServiceUnavailable || !strings.Contains(w.Body.String(), lbhealth.CondHosting) {
+ t.Fatalf("joining: %d %q", w.Code, w.Body.String())
+ }
+ // A POST, or another path, is ordinary :80 traffic.
+ if w := fallbackGet(e, http.MethodPost, lbhealth.Path, "10.0.1.7"); w.Code == http.StatusOK || w.Code == http.StatusServiceUnavailable {
+ t.Fatalf("POST answered by the health check: %d", w.Code)
+ }
+ e.lbHealth = nil
+ if w := fallbackGet(e, http.MethodGet, lbhealth.Path, "10.0.1.7"); w.Code == http.StatusOK || strings.Contains(w.Body.String(), "unavailable:") {
+ t.Fatalf("region HA off answered the health check: %d %q", w.Code, w.Body.String())
+ }
+}
+
+// blockingStore makes every List wait until release is closed, and fail
+// while down is set: a refresh stuck inside one attempt, and a bucket
+// outage.
+type blockingStore struct {
+ release.Store
+ block chan struct{}
+ down bool
+ entered chan struct{}
+}
+
+func (s *blockingStore) List(ctx context.Context, prefix string) ([]string, error) {
+ if s.block != nil {
+ s.entered <- struct{}{}
+ <-s.block
+ }
+ if s.down {
+ return nil, errors.New("bucket unreachable")
+ }
+ return s.Store.List(ctx, prefix)
+}
+
+func (s *blockingStore) Get(ctx context.Context, key string) ([]byte, string, error) {
+ if s.down {
+ return nil, "", errors.New("bucket unreachable")
+ }
+ return s.Store.Get(ctx, key)
+}
+
+// The home map latches its first loaded map and key, stamps every attempt
+// it finishes, failed or not, and stamps nothing while one is stuck.
+func TestHomeMapFeedsTheHealthState(t *testing.T) {
+ inner := meshTestStore(t)
+ if err := mintMeshKey(context.Background(), inner); err != nil {
+ t.Fatal(err)
+ }
+ putJSON(t, inner, "control/edges/betty.json", `{"v":1,"box":"betty","eip":"2.2.2.2","hosts":{"b.test":{"role":"home"}}}`)
+ s := &blockingStore{Store: inner}
+ h := &lbhealth.State{}
+ m := newHomeMap(s, "annie")
+ m.health = h
+ refreshHomeMap(t, m)
+ now := time.Now()
+ failing := strings.Join(h.Failing(now), ",")
+ if strings.Contains(failing, lbhealth.CondForwarding) || strings.Contains(failing, lbhealth.CondRefresh) {
+ t.Fatalf("after a good refresh: %s", failing)
+ }
+
+ // A bucket outage: the attempt fails but finishes, and the latches hold.
+ s.down = true
+ before := time.Now()
+ if err := m.Refresh(context.Background()); err == nil {
+ t.Fatal("refresh against a down bucket succeeded")
+ }
+ if got := strings.Join(h.Failing(before.Add(lbhealth.RefreshWindow)), ","); strings.Contains(got, lbhealth.CondForwarding) || strings.Contains(got, lbhealth.CondRefresh) {
+ t.Fatalf("a failed attempt dropped forwarding or went unstamped: %s", got)
+ }
+
+ // A stuck attempt stamps nothing.
+ s.down = false
+ s.block, s.entered = make(chan struct{}), make(chan struct{}, 1)
+ done := make(chan struct{})
+ go func() { _ = m.Refresh(context.Background()); close(done) }()
+ <-s.entered
+ stuckAt := time.Now()
+ if got := strings.Join(h.Failing(stuckAt.Add(lbhealth.RefreshWindow+time.Second)), ","); !strings.Contains(got, lbhealth.CondRefresh) {
+ t.Fatalf("a refresh stuck for longer than the window still counts: %s", got)
+ }
+ close(s.block)
+ <-done
+}
+
+// A key that never loads leaves forwarding unarmed, whatever the map did.
+func TestHomeMapWithNoMeshKeyIsNotArmed(t *testing.T) {
+ s := meshTestStore(t)
+ putJSON(t, s, "control/edges/betty.json", `{"v":1,"box":"betty","eip":"2.2.2.2","hosts":{}}`)
+ h := &lbhealth.State{}
+ m := newHomeMap(s, "annie")
+ m.health = h
+ refreshHomeMap(t, m)
+ if got := strings.Join(h.Failing(time.Now()), ","); !strings.Contains(got, lbhealth.CondForwarding) {
+ t.Fatalf("no mesh key, yet forwarding armed: %s", got)
+ }
+}
+
+func TestCertMirrorLatchesItsFirstLoad(t *testing.T) {
+ s := meshTestStore(t)
+ h := &lbhealth.State{}
+ m := newCertMirror(s, autocert.DirCache(t.TempDir()))
+ m.health = h
+ if !strings.Contains(strings.Join(h.Failing(time.Now()), ","), lbhealth.CondCerts) {
+ t.Fatal("certificates counted before any load")
+ }
+ if err := m.Refresh(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ if strings.Contains(strings.Join(h.Failing(time.Now()), ","), lbhealth.CondCerts) {
+ t.Fatal("a loaded mirror did not latch the certificate condition")
+ }
+}
+
+// The edge's registry watch beats on every poll it completes.
+func TestRegistryWatchBeats(t *testing.T) {
+ e, err := New(openReg(t), nil)
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(e.probers.Stop)
+ h := &lbhealth.State{}
+ e.lbHealth = h
+ stop := make(chan struct{})
+ t.Cleanup(func() { close(stop) })
+ go e.Watch(stop)
+ deadline := time.Now().Add(5 * time.Second)
+ for strings.Contains(strings.Join(h.Failing(time.Now()), ","), lbhealth.CondEdge) {
+ if time.Now().After(deadline) {
+ t.Fatal("the registry watch never beat")
+ }
+ time.Sleep(50 * time.Millisecond)
+ }
+}
+````
+
+`internal/activator/lastsweep_test.go`:
+
+````go
+package activator
+
+import (
+ "context"
+ "testing"
+ "time"
+)
+
+// Region HA spec 2 §6: the sweeper's finished sweeps are the hosting
+// loop's beat. Nothing has swept before Start.
+func TestLastSweepBeatsAfterEachSweep(t *testing.T) {
+ a, _, _, _ := testActivator(t, 400*time.Millisecond) // sweeps every 100ms
+ if !a.LastSweep().IsZero() {
+ t.Fatal("swept before Start")
+ }
+ var none *Service
+ if !none.LastSweep().IsZero() {
+ t.Fatal("a nil service swept")
+ }
+ a.Start()
+ t.Cleanup(func() { a.Shutdown(context.Background()) })
+ waitFor(t, "a sweep", func() bool { return !a.LastSweep().IsZero() })
+ first := a.LastSweep()
+ if time.Since(first) > 5*time.Second {
+ t.Fatalf("beat %v is not wall-clock time", first)
+ }
+ waitFor(t, "a later sweep", func() bool { return a.LastSweep().After(first) })
+}
+````
+
+`internal/host/claimwiring_hosting_test.go`:
+
+````go
+package host
+
+import (
+ "testing"
+
+ "github.com/carlosframework/platform/internal/hostwrite"
+ "github.com/carlosframework/platform/internal/lbhealth"
+)
+
+// Region HA spec 2 §5, §6: a claim agent that is running is enrolled, so
+// it is hosting; an agent with claim mode off never says so. Plan 2b's
+// joining mode is what leaves this false.
+func TestClaimWiringStartMarksTheBoxHosting(t *testing.T) {
+ lbhealth.Default.SetHosting(false)
+ t.Cleanup(func() { lbhealth.Default.SetHosting(false) })
+ off := &ClaimWiring{}
+ if err := off.startRegional(t.Context(), nil, nil); err != nil {
+ t.Fatal(err)
+ }
+ if lbhealth.Default.Hosting() {
+ t.Fatal("claim mode off marked the box hosting")
+ }
+ loop := testRegionalLoop{started: make(chan struct{})}
+ w := &ClaimWiring{Writes: hostwrite.Config{Enabled: true}, Providers: ClaimProviders{Regional: loop}}
+ if err := w.startRegional(t.Context(), nil, nil); err != nil {
+ t.Fatal(err)
+ }
+ <-loop.started
+ if !lbhealth.Default.Hosting() {
+ t.Fatal("a running claim agent is not hosting")
+ }
+ if err := w.close(t.Context()); err != nil {
+ t.Fatal(err)
+ }
+}
+````
+
+- [ ] **Step 2: Run them to see them fail**
+
+Run: `nice -n 19 go test ./internal/edge/ -run 'LBHealth|FeedsTheHealth|NoMeshKeyIsNotArmed|CertMirrorLatches|RegistryWatchBeats' -count=1`
+Expected: FAIL to build (`undefined: lbHealthFromEnv`, `e.lbHealth undefined`). The activator and host tests fail to build the same way (`a.LastSweep undefined`; `lbhealth` not set by `startRegional`, so `TestClaimWiringStartMarksTheBoxHosting` fails).
+
+- [ ] **Step 3: Write the answer and its feeds**
+
+`internal/edge/lbhealth.go`:
+
+````go
+package edge
+
+import (
+ "net/http"
+ "strconv"
+ "strings"
+ "time"
+
+ "github.com/carlosframework/platform/internal/lbhealth"
+)
+
+// The load balancer health answer (region HA spec 2 §6): GET or HEAD
+// /.carlos/lb-health on port 80, from any host. The network load
+// balancer's HTTP check sends the target's address as the Host header, so
+// the steering probe, which answers only for its own hostname, cannot
+// serve it. Every condition is this box's own; lbhealth has the list.
+
+// lbHealthFromEnv is the state the edge answers from: the process's
+// (lbhealth.Default) with CARLOS_REGION_HA on, and nil with it off, which
+// leaves the path to the ordinary :80 handling exactly as before. The
+// activator's sweep is the hosting loop's beat; an edge with no activator
+// (carlos edge) has none, and its answer is never 200.
+func lbHealthFromEnv(getenv func(string) string, act Activator) *lbhealth.State {
+ on, err := strconv.ParseBool(strings.TrimSpace(getenv("CARLOS_REGION_HA")))
+ if err != nil || !on {
+ return nil
+ }
+ s := lbhealth.Default
+ if a, ok := act.(interface{ LastSweep() time.Time }); ok {
+ s.SetActivator(a.LastSweep)
+ }
+ return s
+}
+
+// lbHealthAnswer answers the health check when r is one and region HA is
+// on, reporting whether it did.
+func (e *Edge) lbHealthAnswer(w http.ResponseWriter, r *http.Request) bool {
+ if e.lbHealth == nil {
+ return false
+ }
+ return lbhealth.Serve(w, r, e.lbHealth, time.Now())
+}
+````
+
+````diff
+--- a/internal/edge/edge.go
++++ b/internal/edge/edge.go
+@@ -56,6 +56,7 @@
+ "github.com/carlosframework/platform/internal/gate"
+ "github.com/carlosframework/platform/internal/geo"
+ "github.com/carlosframework/platform/internal/geoip"
++ "github.com/carlosframework/platform/internal/lbhealth"
+ "github.com/carlosframework/platform/internal/provision"
+ "github.com/carlosframework/platform/internal/registry"
+ "github.com/carlosframework/platform/internal/release"
+@@ -131,6 +132,10 @@
+ // environment; see steeringprobe.go for why it is a serving concern at
+ // all.
+ steeringHost string
++ // lbHealth is the load balancer health state (region HA spec 2 §6):
++ // lbhealth.Default with CARLOS_REGION_HA on, nil and inert with it
++ // off. Set once in New.
++ lbHealth *lbhealth.State
+ // appsDomain is the platform apps domain (PLATFORM_APPS_DOMAIN,
+ // normalized), read once in New; cookieLogged is the set of route hosts
+ // whose cookie rewrite has been logged. Both serve cookiescope.go. ""
+@@ -368,7 +373,8 @@
+ // New builds an Edge over reg and performs the startup reload. act may be nil
+ // (no hibernation).
+ func New(reg *registry.Registry, act Activator) (*Edge, error) {
+- e := &Edge{reg: reg, act: act, gate: gate.New(), probers: newProberSet(), gzipOn: gzipEnabled(), steeringHost: steeringProbeHostFromEnv(), appsDomain: appsDomainFromEnv()}
++ e := &Edge{reg: reg, act: act, gate: gate.New(), probers: newProberSet(), gzipOn: gzipEnabled(), steeringHost: steeringProbeHostFromEnv(), appsDomain: appsDomainFromEnv(),
++ lbHealth: lbHealthFromEnv(os.Getenv, act)}
+ e.geo = geo.NewSwappable()
+ e.geoLookup = e.geo.Lookup
+ // Generations (zero-downtime spec §7.2), before the startup Reload so
+@@ -2226,6 +2232,12 @@
+ if h, _, err := net.SplitHostPort(host); err == nil {
+ host = h
+ }
++ // The load balancer's health check (region HA spec 2 §6, lbhealth.go):
++ // any host, so it comes before every host-based branch. Inert with
++ // region HA off.
++ if e.lbHealthAnswer(w, r) {
++ return
++ }
+ // The steering hostname answers here too (ServeHTTP's own first
+ // branch, and the same reasoning): the health check is HTTPS today,
+ // but a :80 probe — or an operator's curl — must not be handed this
+@@ -2305,6 +2317,9 @@
+ continue
+ }
+ errs = 0
++ // The edge's own loop is alive and its routing table readable
++ // (region HA spec 2 §6). A wedged poll stops beating.
++ e.lbHealth.EdgeBeat(time.Now())
+ if fp == last {
+ continue
+ }
+````
+
+````diff
+--- a/internal/edge/homemap.go
++++ b/internal/edge/homemap.go
+@@ -84,6 +84,7 @@
+ "sync"
+ "time"
+
++ "github.com/carlosframework/platform/internal/lbhealth"
+ "github.com/carlosframework/platform/internal/release"
+ )
+
+@@ -259,6 +260,11 @@
+ keyMu sync.RWMutex
+ keyCurrent []byte
+ keyNext []byte
++
++ // health is the load balancer health state (region HA spec 2 §6),
++ // nil without region HA: Refresh latches the first loaded map and key
++ // and stamps every attempt it completes.
++ health *lbhealth.State
+ }
+
+ func newHomeMap(s release.Store, self string) *homeMap {
+@@ -274,17 +280,32 @@
+ if e.staticStore == nil {
+ return nil
+ }
+- return newHomeMap(e.staticStore, e.box)
++ m := newHomeMap(e.staticStore, e.box)
++ m.health = e.lbHealth
++ return m
+ }
+
+ // Refresh rebuilds the host fold and re-reads the mesh keys. The two halves
+ // fail INDEPENDENTLY — a bucket that answered the edges list but not the key
+ // object should still land the fold — so both run and their errors join.
+ // Whatever fails leaves its own last-good in place; the caller logs.
++//
++// For the load balancer's health answer (region HA spec 2 §6) each half
++// latches its first success, and every call that returns stamps an
++// attempt, failed or not: a bucket outage after arming leaves the box
++// healthy, and only a refresh stuck inside one attempt goes stale.
+ func (m *homeMap) Refresh(ctx context.Context) error {
+ m.refreshMu.Lock()
+ defer m.refreshMu.Unlock()
+- return errors.Join(m.refreshHosts(ctx), m.refreshKeys(ctx))
++ hostsErr, keysErr := m.refreshHosts(ctx), m.refreshKeys(ctx)
++ if hostsErr == nil {
++ m.health.MapLoaded()
++ }
++ if current, _ := m.meshKeys(); keysErr == nil && len(current) > 0 {
++ m.health.KeyLoaded()
++ }
++ m.health.RefreshAttempted(time.Now())
++ return errors.Join(hostsErr, keysErr)
+ }
+
+ // refreshHosts rebuilds the whole map from one list plus a get per key — no
+````
+
+````diff
+--- a/internal/edge/certsync.go
++++ b/internal/edge/certsync.go
+@@ -50,6 +50,7 @@
+ "golang.org/x/crypto/acme/autocert"
+ "golang.org/x/net/idna"
+
++ "github.com/carlosframework/platform/internal/lbhealth"
+ "github.com/carlosframework/platform/internal/release"
+ )
+
+@@ -516,6 +517,10 @@
+ // first-and-every-tenth log cadence.
+ partialPolls int
+
++ // health is the load balancer health state the first landed refresh
++ // latches (region HA spec 2 §6); nil without region HA.
++ health *lbhealth.State
++
+ // synced remembers, per name, the mirrored bytes last reconciled into the
+ // local cache, so a steady state costs no local I/O at all.
+ syncMu sync.Mutex
+@@ -546,6 +551,7 @@
+ return nil
+ }
+ m := newCertMirror(e.staticStore, local)
++ m.health = e.lbHealth
+ m.gated = func() bool {
+ _, held := e.issuerGate()
+ return !held
+@@ -752,6 +758,10 @@
+ m.snap = snap
+ m.mu.Unlock()
+ m.syncLocal(ctx, snap)
++ // The local cache the handshake reads now holds what the bucket does:
++ // the certificate condition is met for good (§6). A later failed
++ // refresh keeps serving this copy and leaves the latch set.
++ m.health.CertsLoaded()
+ if m.wake != nil && wildcardMaterialChanged(prev, snap) {
+ // A takeover (or a first wildcard for a new account) reaches the
+ // serving path in one poll instead of up to twelve hours.
+````
+
+````diff
+--- a/internal/activator/activator.go
++++ b/internal/activator/activator.go
+@@ -1237,6 +1237,10 @@
+
+ sweepStop chan struct{}
+ sweepDone chan struct{}
++ // lastSweep is when the sweeper last finished a sweep, in unix
++ // nanoseconds (0: never): the hosting loop's own beat, which the load
++ // balancer's health answer reads (region HA spec 2 §6, LastSweep).
++ lastSweep atomic.Int64
+
+ // gens is the generations switch (SetGenerations) as read at New. Off:
+ // every spawn binds the canonical socket and no retire hook fires.
+@@ -4439,9 +4443,25 @@
+ return
+ case <-t.C:
+ a.sweepOnce()
++ a.lastSweep.Store(time.Now().UnixNano())
+ }
+ }
+ }()
++}
++
++// LastSweep is when the sweeper last finished a sweep, or the zero time
++// before its first. The sweep runs at least every 30 seconds, so a box
++// whose last sweep is older than a minute has a hosting loop that is stuck
++// (region HA spec 2 §6). A nil service has never swept.
++func (a *Service) LastSweep() time.Time {
++ if a == nil {
++ return time.Time{}
++ }
++ n := a.lastSweep.Load()
++ if n == 0 {
++ return time.Time{}
++ }
++ return time.Unix(0, n)
+ }
+
+ // markCleanParkLocked records that a clean-exit park is stopping e's current
+````
+
+````diff
+--- a/internal/host/claimwiring.go
++++ b/internal/host/claimwiring.go
+@@ -10,6 +10,7 @@
+ "github.com/carlosframework/platform/internal/hostclaim"
+ "github.com/carlosframework/platform/internal/hostguard"
+ "github.com/carlosframework/platform/internal/hostwrite"
++ "github.com/carlosframework/platform/internal/lbhealth"
+ "github.com/carlosframework/platform/internal/provision"
+ "github.com/carlosframework/platform/internal/schedule"
+ )
+@@ -94,6 +95,12 @@
+ w.cancel = cancel
+ w.joined = make(chan struct{})
+ go func() { defer close(w.joined); w.Providers.Regional.Loop(workerCtx, serial, report) }()
++ // The claim agent is running: it refused to start without its
++ // host-actor enrollment and a matching roster row, so this box is
++ // hosting (region HA spec 2 §5, §6). Plan 2b's mode selection owns
++ // this flag from then on: joining mode leaves it false, and only an
++ // active roster entry with an armed guard sets it.
++ lbhealth.Default.SetHosting(true)
+ return nil
+ }
+ func (w *ClaimWiring) close(ctx context.Context) error {
+````
+
+- [ ] **Step 4: Run the tests**
+
+```bash
+nice -n 19 go test ./internal/edge/ -run 'LBHealth|FeedsTheHealth|NoMeshKeyIsNotArmed|CertMirrorLatches|RegistryWatchBeats|HomeMap|Backhaul|Hop|HTTPFallback' -count=1
+nice -n 19 go test ./internal/activator/ -run LastSweep -count=1
+nice -n 19 go test ./internal/host/ -run ClaimWiring -count=1
+```
+Expected: PASS.
+
+- [ ] **Step 5: Boundary list and census**
+
+````diff
+--- a/region_ha_boundary_test.go
++++ b/region_ha_boundary_test.go
+@@ -23,6 +23,7 @@
+ "internal/hostactors/roles.go",
+ "internal/boxchoice/policy.go",
+ "internal/host/boxaddress.go",
++ "internal/edge/lbhealth.go",
+ }
+
+ // regionHAAdapterFiles are spec 2's AWS adapters. Each is named for AWS so
+````
+
+Add this comment line after the overlay's last comment line:
+
+````text
+# Task 9: the load balancer health answer (internal/edge lbhealth.go, answered in httpFallbackHandler; the home map and the certificate mirror latch their first loads, Edge.Watch beats, the activator's sweeper beats through Service.LastSweep, and ClaimWiring.startRegional marks the box hosting). None of the changed units is a root-read unit, and the answer reads only in-memory state. Envelope-only, and the legacy seal stands.
+````
+
+Then set these rows. A `@coverage` row replaces the overlay's row for the same directory, or is appended when there is none; every other row is appended:
+
+````text
+@coverage internal/host 5a5341c760fb8c16bf79ca6fc5cf82e9174157f5fe05d78a74532e237fe40c85
+@coverage internal/edge f05b773955396fd0e54bce32b8a41c095130503005f43871b4bd96c255d3b87d
+@coverage internal/activator c2eb11024375a706d70652dee68d446202a264927f819deab6321b652e82c2bb
+````
+
+Run the census checks and `go test . -run TestRegionHA -count=1`. Expected: PASS.
+
+- [ ] **Step 6: Commit**
+
+```bash
+git add internal/edge internal/activator internal/host/claimwiring.go internal/host/claimwiring_hosting_test.go region_ha_boundary_test.go internal/provision/testdata/plan14-root-reads-region-ha-2a.tsv
+git commit -m "region HA 2a: the load balancer health answer"
+```
+
+---
+
+### Task 10: The role doors, `carlos system box role` and `group role`, and the move destination
+
+**Files:**
+- Create: `internal/console/apiboxroles.go`, `internal/console/apiboxroles_test.go`
+- Modify: `internal/console/authcli.go` (two routes, with host claims on)
+- Modify: `internal/console/boxselect.go` (`selectMoveBoxes` skips excluded destinations)
+- Modify: `internal/console/svcroutes_test.go`, `internal/console/claim_invariant_fixtures_test.go`, `internal/console/claim_invariant_probes_nonapp_test.go`, `internal/console/testdata/claim-routes.txt`
+- Create: `system_roles.go`, `system_roles_test.go`
+- Modify: `system.go` (the `box` and `group` verbs and their usage)
+- Modify: `docs/site/cli.md` (a "Box and group roles" section)
+- Modify: `approved_region_ha_copy_test.go` (wire RHA-C1 to RHA-C16), `region_ha_boundary_test.go`
+- Modify: `internal/provision/plan9_b8_census_test.go`, `internal/provision/plan14_root_reads_test.go` (the seal), the overlay
+
+**Interfaces:**
+- Consumes: Task 3's `ParseBoxRole`, `ParseGroupRole`, `RoleName`, `SetBoxRole`, `SetGroupRole`; Task 4's `boxchoice.ExcludeRoles`; `apiSystemRoller` (ops account owners and `CARLOS_SYSTEM_OPERATORS`).
+- Produces: `POST /api/cli/system/boxes/{box}/role` and `POST /api/cli/system/groups/{region}/role`, body `{"role": "<name>"}`, answering `{"box", "role"}` and `{"region", "group", "role", "boxes"}`; `func (s *Server) boxRoster(ctx) (hostactors.Directory, release.Signer, hostactors.Observed, error)`; `func (s *Server) boxRoles(ctx) (map[string]string, error)`; CLI `setBoxRole`, `setGroupRole`.
+
+The doors are registered only with host claims on (the roster exists only then), closed to every service credential, and answer 404 unless `CARLOS_REGION_HA` is on too. A box is named by its roster reference or its label. With host claims on, the move door's destination skips legacy and platform boxes (the source is never filtered, so an instance on such a box can still move out, A10-3); an unreadable roster refuses the move. The move door then says "no box in `<to>` publishes an edges file yet" when the only eligible boxes are excluded: 2d's rework of the door gives that case its own sentence (open question 3).
+
+**Census (flagged):** `Server.boxRoster` reads the signed roster (`Directory.Read`, kind `file:Read`), a new root-read unit. It is asked of platform-1e as census ruling region-ha-2a, accepted-legacy with main's basis for a console root input (J4). The seal moves.
+
+- [ ] **Step 1: Write the failing tests**
+
+`internal/console/apiboxroles_test.go`:
+
+````go
+package console
+
+import (
+ "bytes"
+ "context"
+ "encoding/json"
+ "errors"
+ "net/http"
+ "net/http/httptest"
+ "reflect"
+ "testing"
+
+ "github.com/carlosframework/platform/internal/hostactors"
+ "github.com/carlosframework/platform/internal/release"
+)
+
+// roleFixture is a console with region HA on and a signed roster: in ie,
+// an operator-enrolled box ie-old and two boxes the group "routers"
+// enrolled; uk has no group.
+func roleFixture(t *testing.T) *consoleFixture {
+ t.Helper()
+ fx := newConsoleFixture(t)
+ fx.srv.cfg.RegionHA = true
+ old := moveRosterBox(ieBox("ie-old"))
+ g1, g2 := moveRosterBox(ieBox("ie-g1")), moveRosterBox(ieBox("ie-g2"))
+ g1.Group, g2.Group = "routers", "routers"
+ putMoveRoster(t, fx, hostactors.Roster{
+ Boxes: map[string]hostactors.Box{"ie-old": old, "ie-g1": g1, "ie-g2": g2},
+ Regions: map[string]hostactors.Region{
+ "ie": {Account: moveTestAccount, AWSRegion: "eu-west-1", GroupName: "routers", GroupARN: "arn:group"},
+ "uk": {Account: moveTestAccount, AWSRegion: "eu-west-2"},
+ },
+ })
+ return fx
+}
+
+// roleCall drives one role door as addr, past cliRoute.
+func roleCall(t *testing.T, h http.HandlerFunc, addr, pathKey, pathValue, body string) (int, map[string]any) {
+ t.Helper()
+ r := httptest.NewRequest(http.MethodPost, "/api/cli/system/x/y/role", bytes.NewBufferString(body))
+ r.Header.Set("Content-Type", "application/json")
+ r.SetPathValue(pathKey, pathValue)
+ r = r.WithContext(context.WithValue(r.Context(), cliRouteKey{}, Principal{Kind: PrincipalUser, Address: addr}))
+ w := httptest.NewRecorder()
+ h(w, r)
+ var out map[string]any
+ _ = json.Unmarshal(w.Body.Bytes(), &out)
+ return w.Code, out
+}
+
+func rosterNow(t *testing.T, fx *consoleFixture) hostactors.Roster {
+ t.Helper()
+ _, _, o, err := fx.srv.boxRoster(context.Background())
+ if err != nil {
+ t.Fatal(err)
+ }
+ return o.Roster
+}
+
+func TestBoxRoleDoor(t *testing.T) {
+ fx := roleFixture(t)
+ h := fx.srv.handleAPISystemBoxRole
+ if st, _ := roleCall(t, h, "someone@keymail.dev", "box", "ie-old", `{"role":"legacy"}`); st != http.StatusNotFound {
+ t.Fatalf("a non-operator = %d, want 404", st)
+ }
+ fx.srv.cfg.RegionHA = false
+ if st, _ := roleCall(t, h, "paul@keymail.dev", "box", "ie-old", `{"role":"legacy"}`); st != http.StatusNotFound {
+ t.Fatalf("region HA off = %d, want 404", st)
+ }
+ fx.srv.cfg.RegionHA = true
+ if st, out := roleCall(t, h, "paul@keymail.dev", "box", "ie-old", `{"role":"draining"}`); st != http.StatusBadRequest || out["error"] != boxRoleRefusal {
+ t.Fatalf("unknown role = %d %v", st, out)
+ }
+ if st, out := roleCall(t, h, "paul@keymail.dev", "box", "nobody", `{"role":"legacy"}`); st != http.StatusNotFound || out["error"] != noEnrolledBox("nobody") {
+ t.Fatalf("unknown box = %d %v", st, out)
+ }
+ // By label, as an operator names it.
+ st, out := roleCall(t, h, "paul@keymail.dev", "box", "ie-old-label", `{"role":"legacy"}`)
+ if st != http.StatusOK || out["box"] != "ie-old" || out["role"] != "legacy" {
+ t.Fatalf("set = %d %v", st, out)
+ }
+ if got := rosterNow(t, fx).Boxes["ie-old"].Role; got != hostactors.RoleLegacy {
+ t.Fatalf("the roster says %q", got)
+ }
+ st, out = roleCall(t, h, "paul@keymail.dev", "box", "ie-old", `{"role":"default"}`)
+ if st != http.StatusOK || out["role"] != "default" || rosterNow(t, fx).Boxes["ie-old"].Role != hostactors.RoleDefault {
+ t.Fatalf("back to default = %d %v", st, out)
+ }
+}
+
+func TestGroupRoleDoor(t *testing.T) {
+ fx := roleFixture(t)
+ h := fx.srv.handleAPISystemGroupRole
+ if st, out := roleCall(t, h, "paul@keymail.dev", "region", "ie", `{"role":"platform"}`); st != http.StatusBadRequest || out["error"] != groupRoleRefusal {
+ t.Fatalf("platform group = %d %v", st, out)
+ }
+ if st, out := roleCall(t, h, "paul@keymail.dev", "region", "uk", `{"role":"legacy"}`); st != http.StatusNotFound || out["error"] != noGroupIn("uk") {
+ t.Fatalf("no group = %d %v", st, out)
+ }
+ st, out := roleCall(t, h, "paul@keymail.dev", "region", "ie", `{"role":"legacy"}`)
+ if st != http.StatusOK || out["group"] != "routers" || out["role"] != "legacy" || !reflect.DeepEqual(out["boxes"], []any{"ie-g1", "ie-g2"}) {
+ t.Fatalf("set = %d %v", st, out)
+ }
+ r := rosterNow(t, fx)
+ if r.Regions["ie"].Role != hostactors.RoleLegacy || r.Boxes["ie-g1"].Role != hostactors.RoleLegacy || r.Boxes["ie-g2"].Role != hostactors.RoleLegacy || r.Boxes["ie-old"].Role != hostactors.RoleDefault {
+ t.Fatalf("roster after the group role: %+v", r.Boxes)
+ }
+}
+
+func TestRoleWriteErrors(t *testing.T) {
+ for _, c := range []struct {
+ err error
+ code int
+ msg string
+ }{
+ {release.ErrCASConflict, http.StatusConflict, boxRosterChanged},
+ {hostactors.ErrRoster, http.StatusConflict, boxRosterChanged},
+ {errors.New("SlowDown"), http.StatusServiceUnavailable, boxRoleWriteFailure},
+ } {
+ w := httptest.NewRecorder()
+ roleWriteError(w, "test", c.err)
+ var out map[string]string
+ _ = json.Unmarshal(w.Body.Bytes(), &out)
+ if w.Code != c.code || out["error"] != c.msg {
+ t.Fatalf("%v = %d %v", c.err, w.Code, out)
+ }
+ }
+}
+
+// The doors exist only on a host-claim console, operator-only and 404
+// with region HA off.
+func TestRoleDoorsAreRegisteredWithHostClaims(t *testing.T) {
+ st := openGatedStore(t)
+ rel := release.NewDirStore(t.TempDir())
+ path := "/api/cli/system/boxes/ie-old/role"
+ off, err := NewServer(Config{Origin: "https://console.example", Keymail: "https://keymail.example", Releases: rel}, st)
+ if err != nil {
+ t.Fatal(err)
+ }
+ ghostStatus, ghostBody := serveHostClaim(t, off, "/unknown-role-route", "")
+ if status, body := serveHostClaim(t, off, path, ""); status != ghostStatus || !bytes.Equal(body, ghostBody) {
+ t.Fatalf("claims off = %d %q, want the unknown route's %d %q", status, body, ghostStatus, ghostBody)
+ }
+ token := mintHostClaimToken(t, st, "paul@keymail.dev")
+ on, err := NewServer(Config{Origin: "https://console.example", Keymail: "https://keymail.example", Releases: rel, HostClaim: true, SystemOperators: []string{"paul@keymail.dev"}}, st)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if status, _ := serveHostClaim(t, on, path, ""); status != http.StatusUnauthorized {
+ t.Fatalf("unsigned = %d", status)
+ }
+ if status, _ := serveHostClaim(t, on, path, token); status != http.StatusNotFound {
+ t.Fatalf("region HA off = %d, want 404", status)
+ }
+}
+
+// Region HA spec 2 §7: a legacy or platform box is never a move's
+// destination, and an instance on one can still move out.
+func TestMoveDestinationSkipsExcludedRoles(t *testing.T) {
+ const host, app = "acme.go.tito.io", "bab/titogo"
+ fx := newConsoleFixture(t)
+ armMoveFleet(t, fx, ieBox("ie-box"), ukBox("uk-box"))
+ ie, uk := moveRosterBox(ieBox("ie-box")), moveRosterBox(ukBox("uk-box"))
+ ie.Role, uk.Role = hostactors.RolePlatform, hostactors.RoleLegacy
+ putMoveRoster(t, fx, hostactors.Roster{Boxes: map[string]hostactors.Box{"ie-box": ie, "uk-box": uk}})
+ // Claims off: no roster roles apply, and main's selection stands.
+ got, err := fx.srv.selectMoveBoxes(context.Background(), host, "uk", "ie", app)
+ if err != nil || got.Target != "uk-box" {
+ t.Fatalf("claims off: %+v %v", got, err)
+ }
+ fx.srv.cfg.HostClaim = true
+ got, err = fx.srv.selectMoveBoxes(context.Background(), host, "uk", "ie", app)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if got.Target != "" || got.Source != "ie-box" {
+ t.Fatalf("selection %+v: want no target (uk-box is legacy) and the platform source kept", got)
+ }
+ // A roster that cannot be read refuses the move rather than guess.
+ fx.srv.cfg.Releases = failingGet{Store: fx.rel, key: hostactors.Key}
+ if _, err := fx.srv.selectMoveBoxes(context.Background(), host, "uk", "ie", app); err == nil {
+ t.Fatal("an unreadable roster chose a destination")
+ }
+}
+````
+
+`system_roles_test.go`:
+
+````go
+package main
+
+import (
+ "context"
+ "errors"
+ "net/http"
+ "reflect"
+ "testing"
+)
+
+func TestSystemVerbKnowsBoxAndGroup(t *testing.T) {
+ for _, verb := range []string{"box", "group"} {
+ got, rest, _ := systemVerb([]string{verb, "role", "x", "legacy"})
+ if got != verb || len(rest) != 3 || rest[0] != "role" {
+ t.Fatalf("systemVerb(%s) = %q %q", verb, got, rest)
+ }
+ }
+}
+
+// fakeRolePost answers one role call with status and body, recording what
+// was asked.
+type fakeRolePost struct {
+ status int
+ body string
+ path string
+ in any
+}
+
+func (f *fakeRolePost) post(_ context.Context, path string, in any) (int, []byte, error) {
+ f.path, f.in = path, in
+ return f.status, []byte(f.body), nil
+}
+
+func TestSetBoxRole(t *testing.T) {
+ f := &fakeRolePost{status: http.StatusOK, body: `{"box":"i-0abc","role":"legacy"}`}
+ line, err := setBoxRole(context.Background(), f.post, "old box", "legacy")
+ if err != nil || line != "i-0abc now has the legacy role" {
+ t.Fatalf("line %q, %v", line, err)
+ }
+ if f.path != "/api/cli/system/boxes/old%20box/role" || !reflect.DeepEqual(f.in, map[string]string{"role": "legacy"}) {
+ t.Fatalf("asked %s %v", f.path, f.in)
+ }
+ f = &fakeRolePost{status: http.StatusBadRequest, body: `{"error":"role must be default, legacy or platform"}`}
+ if _, err := setBoxRole(context.Background(), f.post, "x", "boss"); err == nil || err.Error() != "role must be default, legacy or platform" {
+ t.Fatalf("refusal: %v", err)
+ }
+ failing := func(context.Context, string, any) (int, []byte, error) { return 0, nil, errors.New("offline") }
+ if _, err := setBoxRole(context.Background(), failing, "x", "legacy"); err == nil {
+ t.Fatal("a failed call printed a line")
+ }
+}
+
+func TestSetGroupRole(t *testing.T) {
+ f := &fakeRolePost{status: http.StatusOK, body: `{"region":"ie","group":"routers","role":"legacy","boxes":["i-1","i-2"]}`}
+ lines, err := setGroupRole(context.Background(), f.post, "ie", "legacy")
+ if err != nil {
+ t.Fatal(err)
+ }
+ want := []string{"the group in ie now has the legacy role", "i-1 now has the legacy role", "i-2 now has the legacy role"}
+ if !reflect.DeepEqual(lines, want) || f.path != "/api/cli/system/groups/ie/role" {
+ t.Fatalf("lines %q path %s", lines, f.path)
+ }
+ f = &fakeRolePost{status: http.StatusOK, body: `{"region":"ie","group":"routers","role":"default","boxes":[]}`}
+ if lines, _ := setGroupRole(context.Background(), f.post, "ie", "default"); len(lines) != 1 {
+ t.Fatalf("an empty group: %q", lines)
+ }
+}
+````
+
+The route censuses learn the two doors:
+
+````diff
+--- a/internal/console/svcroutes_test.go
++++ b/internal/console/svcroutes_test.go
+@@ -286,6 +286,10 @@
+ hostClaimDoors := []string{
+ "POST /api/cli/ops/hosts/{host}/resolve/inspect",
+ "POST /api/cli/ops/hosts/{host}/resolve",
++ // Box and group roles (region HA spec 2 §7): operator-only, and 404
++ // unless CARLOS_REGION_HA is on as well.
++ "POST /api/cli/system/boxes/{box}/role",
++ "POST /api/cli/system/groups/{region}/role",
+ }
+ movesDoors := []string{
+ "POST /api/cli/apps/{account}/{app}/instances/{host}/move/resolve",
+````
+
+````diff
+--- a/internal/console/claim_invariant_fixtures_test.go
++++ b/internal/console/claim_invariant_fixtures_test.go
+@@ -1129,6 +1129,8 @@
+ }},
+ "route POST /api/cli/ops/hosts/{host}/resolve/inspect": claimRead("operator host-loss inspection: reads the claim, the record and the roster and signs a challenge; writes nothing"),
+ "route POST /api/cli/system/antivirus": claimNonApp("the deployment scanning switch (system/antivirus.json)"),
++ "route POST /api/cli/system/boxes/{box}/role": claimNonApp("operator box role (region HA spec 2 §7): one signed roster write to control/host-actors.json; names no app"),
++ "route POST /api/cli/system/groups/{region}/role": claimNonApp("operator group role (region HA spec 2 §7): one signed roster write to control/host-actors.json, the group and the boxes it enrolled; names no app"),
+ "route POST /api/cli/system/layout": claimNonApp("the deployment layout intent (control/layout.json)"),
+ "route POST /api/cli/system/activation/cancel": claimNonApp("cancels an unpublished system activation"),
+ "route POST /api/cli/system/pointer-digests": claimNonApp("operator switch: writes or removes each account's root-signed pointer digest marker (releases/<acct>/pointer-digests), classified exactly by the mutation boundary"),
+````
+
+````diff
+--- a/internal/console/claim_invariant_probes_nonapp_test.go
++++ b/internal/console/claim_invariant_probes_nonapp_test.go
+@@ -674,6 +674,54 @@
+ return ""
+ },
+ },
++ // Box and group roles (region HA spec 2 §7): one signed roster write,
++ // only with region HA on.
++ "route POST /api/cli/system/boxes/{box}/role": {
++ setup: func(t *testing.T, cx *claimCase) { cx.srv.cfg.RegionHA = true },
++ do: claimAPIReply(http.MethodPost, "/api/cli/system/boxes/box-ie-default/role", `{"role":"legacy"}`),
++ body: []string{`"role":"legacy"`},
++ effect: func(t *testing.T, cx *claimCase, r claimReply) string {
++ o, err := cx.claimFixtureDirectory().Read(context.Background())
++ if err != nil || o.Roster.Boxes["box-ie-default"].Role != hostactors.RoleLegacy {
++ return "box-ie-default does not have the legacy role in the signed roster"
++ }
++ return ""
++ },
++ },
++ "route POST /api/cli/system/groups/{region}/role": {
++ setup: func(t *testing.T, cx *claimCase) {
++ cx.srv.cfg.RegionHA = true
++ ctx := context.Background()
++ o, err := cx.claimFixtureDirectory().Read(ctx)
++ if err != nil {
++ t.Fatal(err)
++ }
++ next := hostactors.Clone(o.Roster)
++ g := next.Regions["ie"]
++ g.GroupName, g.GroupARN = "routers", "arn:aws:autoscaling:eu-west-1:123456789012:autoScalingGroup:routers"
++ next.Regions["ie"] = g
++ b := next.Boxes["box-ie-default"]
++ b.Group = "routers"
++ next.Boxes["box-ie-default"] = b
++ next.Revision++
++ raw, err := release.SignRootEnvelope(release.RootSigner(cx.priv), next)
++ if err != nil {
++ t.Fatal(err)
++ }
++ if _, err = cx.rel.Put(ctx, hostactors.Key, raw, o.ETag); err != nil {
++ t.Fatal(err)
++ }
++ },
++ do: claimAPIReply(http.MethodPost, "/api/cli/system/groups/ie/role", `{"role":"legacy"}`),
++ body: []string{`"boxes":["box-ie-default"]`},
++ effect: func(t *testing.T, cx *claimCase, r claimReply) string {
++ o, err := cx.claimFixtureDirectory().Read(context.Background())
++ if err != nil || o.Roster.Regions["ie"].Role != hostactors.RoleLegacy || o.Roster.Boxes["box-ie-default"].Role != hostactors.RoleLegacy {
++ return "the group in ie and its box do not have the legacy role in the signed roster"
++ }
++ return ""
++ },
++ },
+ "route POST /api/cli/system/seeds-floor": {
+ do: claimAPIReply(http.MethodPost, "/api/cli/system/seeds-floor", `{"min_release":"v0.27.0"}`),
+ body: []string{`"min_release":"v0.27.0"`},
+````
+
+````diff
+--- a/internal/console/testdata/claim-routes.txt
++++ b/internal/console/testdata/claim-routes.txt
+@@ -237,6 +237,8 @@
+ route POST /api/cli/services/{account}/{name}/rotate
+ route POST /api/cli/system/activation/cancel
+ route POST /api/cli/system/antivirus
++route POST /api/cli/system/boxes/{box}/role
++route POST /api/cli/system/groups/{region}/role
+ route POST /api/cli/system/layout
+ route POST /api/cli/system/legacy-routes
+ route POST /api/cli/system/pointer-digests
+````
+
+- [ ] **Step 2: Run them to see them fail**
+
+Run: `nice -n 19 go test ./internal/console/ -run 'BoxRoleDoor|GroupRoleDoor|RoleWriteErrors|RoleDoorsAreRegistered|MoveDestinationSkips' -count=1`
+Expected: FAIL to build (`undefined: handleAPISystemBoxRole`, `undefined: boxRoster`). `go test . -run 'SystemVerbKnowsBoxAndGroup|SetBoxRole|SetGroupRole'` fails to build on `undefined: setBoxRole`.
+
+- [ ] **Step 3: Write the doors, the move filter and the CLI**
+
+`internal/console/apiboxroles.go`:
+
+````go
+package console
+
+import (
+ "context"
+ "errors"
+ "fmt"
+ "log"
+ "net/http"
+
+ "github.com/carlosframework/platform/internal/hostactors"
+ "github.com/carlosframework/platform/internal/release"
+)
+
+// Box and group roles (region HA spec 2 §7, A11-1): `carlos system box role
+// <box> <role>` and `carlos system group role <region> <role>`. A legacy or
+// platform box takes no new instance, relocation or move; it keeps the
+// ones it has (H25). A group's role is every box it enrolled, now and
+// later. The roles live in the signed roster, so the console, the only
+// holder of the root key, is the only writer.
+//
+// Operator-only (apiSystemRoller, the update doors' authority), registered
+// only with host claims on, and answering 404 unless CARLOS_REGION_HA is on
+// too: with region HA off nothing reads a role.
+
+const (
+ boxRoleRefusal = "role must be default, legacy or platform"
+ groupRoleRefusal = "a group's role must be default or legacy"
+ boxRosterUnread = "could not read the box roster. Try again."
+ boxRosterChanged = "the box roster changed while the role was being saved, and nothing was changed. Try again."
+ boxRoleWriteFailure = "could not save the role. Try again."
+)
+
+func noEnrolledBox(name string) string { return fmt.Sprintf("no enrolled box is named %s", name) }
+
+func noGroupIn(region string) string { return fmt.Sprintf("%s has no group", region) }
+
+func retiredBoxRole(box string) string { return fmt.Sprintf("%s is retired, so it takes no role", box) }
+
+// boxRoster reads the signed roster with the root signer that writes it.
+func (s *Server) boxRoster(ctx context.Context) (hostactors.Directory, release.Signer, hostactors.Observed, error) {
+ if s.cfg.Releases == nil || s.cfg.Bucket == "" || len(s.cfg.Key) == 0 {
+ return hostactors.Directory{}, release.Signer{}, hostactors.Observed{}, errors.New("console: no deployment bucket or root key")
+ }
+ signer := release.RootSigner(s.cfg.Key)
+ dir := hostactors.Directory{Store: s.cfg.Releases, PublicKey: signer.RootPublicKey(), Bucket: s.cfg.Bucket}
+ o, err := dir.Read(ctx)
+ return dir, signer, o, err
+}
+
+// rosterRef finds a box by reference, or by the one label no retired box
+// shares (hostactors.Directory.Resolve's rule, over a roster in hand).
+func rosterRef(r hostactors.Roster, name string) (string, bool) {
+ if _, ok := r.Boxes[name]; ok {
+ return name, true
+ }
+ found := ""
+ for ref, b := range r.Boxes {
+ if b.Label == name && b.State != "retired" {
+ if found != "" {
+ return "", false
+ }
+ found = ref
+ }
+ }
+ return found, found != ""
+}
+
+// roleOperator is the role doors' gate: an operator, with region HA on.
+func (s *Server) roleOperator(w http.ResponseWriter, r *http.Request) (string, bool) {
+ addr, _, ok := s.apiSystemRoller(w, r)
+ if !ok {
+ return "", false
+ }
+ if !s.cfg.RegionHA {
+ http.NotFound(w, r)
+ return "", false
+ }
+ return addr, true
+}
+
+// roleWriteError answers a roster write that did not land.
+func roleWriteError(w http.ResponseWriter, what string, err error) {
+ if errors.Is(err, release.ErrCASConflict) || errors.Is(err, hostactors.ErrRoster) {
+ jsonError(w, http.StatusConflict, boxRosterChanged)
+ return
+ }
+ log.Printf("console: %s: %v", what, err)
+ jsonError(w, http.StatusServiceUnavailable, boxRoleWriteFailure)
+}
+
+func (s *Server) handleAPISystemBoxRole(w http.ResponseWriter, r *http.Request) {
+ by, ok := s.roleOperator(w, r)
+ if !ok {
+ return
+ }
+ var req struct {
+ Role string `json:"role"`
+ }
+ if !readJSON(w, r, &req) {
+ return
+ }
+ role, err := hostactors.ParseBoxRole(req.Role)
+ if err != nil {
+ jsonError(w, http.StatusBadRequest, boxRoleRefusal)
+ return
+ }
+ dir, signer, o, err := s.boxRoster(r.Context())
+ if err != nil {
+ log.Printf("console: box role: roster: %v", err)
+ jsonError(w, http.StatusServiceUnavailable, boxRosterUnread)
+ return
+ }
+ name := r.PathValue("box")
+ ref, ok := rosterRef(o.Roster, name)
+ if !ok {
+ jsonError(w, http.StatusNotFound, noEnrolledBox(name))
+ return
+ }
+ if o.Roster.Boxes[ref].State == "retired" {
+ jsonError(w, http.StatusConflict, retiredBoxRole(ref))
+ return
+ }
+ next, err := dir.SetBoxRole(r.Context(), signer, o, ref, role)
+ if err != nil {
+ roleWriteError(w, "box role "+ref, err)
+ return
+ }
+ log.Printf("console: box %s has the %s role (by %s, roster revision %d)", ref, hostactors.RoleName(role), by, next.Roster.Revision)
+ writeJSON(w, http.StatusOK, map[string]string{"box": ref, "role": hostactors.RoleName(role)})
+}
+
+func (s *Server) handleAPISystemGroupRole(w http.ResponseWriter, r *http.Request) {
+ by, ok := s.roleOperator(w, r)
+ if !ok {
+ return
+ }
+ var req struct {
+ Role string `json:"role"`
+ }
+ if !readJSON(w, r, &req) {
+ return
+ }
+ role, err := hostactors.ParseGroupRole(req.Role)
+ if err != nil {
+ jsonError(w, http.StatusBadRequest, groupRoleRefusal)
+ return
+ }
+ dir, signer, o, err := s.boxRoster(r.Context())
+ if err != nil {
+ log.Printf("console: group role: roster: %v", err)
+ jsonError(w, http.StatusServiceUnavailable, boxRosterUnread)
+ return
+ }
+ region := r.PathValue("region")
+ g, ok := o.Roster.Regions[region]
+ if !ok || g.GroupName == "" {
+ jsonError(w, http.StatusNotFound, noGroupIn(region))
+ return
+ }
+ next, boxes, err := dir.SetGroupRole(r.Context(), signer, o, region, role)
+ if err != nil {
+ roleWriteError(w, "group role "+region, err)
+ return
+ }
+ log.Printf("console: the group in %s (%s) and its boxes %v have the %s role (by %s, roster revision %d)", region, g.GroupName, boxes, hostactors.RoleName(role), by, next.Roster.Revision)
+ if boxes == nil {
+ boxes = []string{}
+ }
+ writeJSON(w, http.StatusOK, map[string]any{"region": region, "group": g.GroupName, "role": hostactors.RoleName(role), "boxes": boxes})
+}
+
+// boxRoles is every enrolled box's role, for a chooser that reads no
+// roster of its own (the move door's destination, selectMoveBoxes).
+func (s *Server) boxRoles(ctx context.Context) (map[string]string, error) {
+ _, _, o, err := s.boxRoster(ctx)
+ if err != nil {
+ return nil, err
+ }
+ roles := make(map[string]string, len(o.Roster.Boxes))
+ for ref, b := range o.Roster.Boxes {
+ roles[ref] = b.Role
+ }
+ return roles, nil
+}
+````
+
+````diff
+--- a/internal/console/authcli.go
++++ b/internal/console/authcli.go
+@@ -512,6 +512,10 @@
+ // ActorDirectory both routes stop at the same inert refusal.
+ s.cliRoute("POST /api/cli/ops/hosts/{host}/resolve/inspect", svcrole.Set{}, s.handleAPIHostResolveInspect)
+ s.cliRoute("POST /api/cli/ops/hosts/{host}/resolve", svcrole.Set{}, s.handleAPIHostResolve)
++ // Box and group roles (region HA spec 2 §7, apiboxroles.go):
++ // operator-only, and 404 unless CARLOS_REGION_HA is on too.
++ s.cliRoute("POST /api/cli/system/boxes/{box}/role", svcrole.Set{}, s.handleAPISystemBoxRole)
++ s.cliRoute("POST /api/cli/system/groups/{region}/role", svcrole.Set{}, s.handleAPISystemGroupRole)
+ }
+ // carlos status --layout (plan 4, task 5): which layout each instance is
+ // on, box by box. Same posture as the door above — an empty role set, the
+````
+
+````diff
+--- a/internal/console/boxselect.go
++++ b/internal/console/boxselect.go
+@@ -7,6 +7,7 @@
+ "log"
+ "os"
+
++ "github.com/carlosframework/platform/internal/boxchoice"
+ "github.com/carlosframework/platform/internal/boxid"
+ "github.com/carlosframework/platform/internal/moves"
+ )
+@@ -89,6 +90,17 @@
+ if err != nil {
+ return moveSelection{}, err
+ }
++ // Region HA spec 2 §7: a legacy or platform box is never a move's
++ // destination. The roles are the signed roster's, which exists only
++ // with host claims on; a roster that cannot be read refuses the move.
++ // The source is never filtered: an instance on such a box can still
++ // move out (A10-3).
++ var roles map[string]string
++ if s.cfg.HostClaim {
++ if roles, err = s.boxRoles(ctx); err != nil {
++ return moveSelection{}, err
++ }
++ }
+ sel := moveSelection{FloorOK: true, ConsoleBelowFloor: floor != nil && moves.ProtocolVersion < floor.Min}
+ seen := map[string]bool{}
+ for _, f := range snap.Files {
+@@ -113,6 +125,9 @@
+ fp := s.expectedSecretsFP()
+ for _, code := range []string{to, from} {
+ cands := eligibleBoxes(snap, roster, code, aws[code], s.cfg.HomeRegion, fp, nil)
++ if code == to {
++ cands = boxchoice.ExcludeRoles(cands, roles)
++ }
+ if len(cands) > 1 {
+ sel.Many, sel.ManyRegion = true, s.homeRegionOut(code)
+ return sel, nil
+````
+
+`system_roles.go`:
+
+````go
+package main
+
+import (
+ "context"
+ "encoding/json"
+ "fmt"
+ "log"
+ "net/http"
+ "net/url"
+ "os"
+)
+
+// carlos system box role <box> <role> and carlos system group role
+// <region> <role> (region HA spec 2 §7, A11-1). A box's role says whether
+// placement may choose it: legacy and platform boxes take no new
+// instances and keep the ones they have. A group's role is every box it
+// has enrolled and every box it enrolls later. Operators only, and only on
+// a console with region HA on.
+
+// rolePost is the one console call a role change makes.
+type rolePost func(ctx context.Context, path string, in any) (int, []byte, error)
+
+// boxRoleLine is what the CLI says once a box has its role.
+func boxRoleLine(box, role string) string { return fmt.Sprintf("%s now has the %s role", box, role) }
+
+// groupRoleLine is what it says once a group has its role.
+func groupRoleLine(region, role string) string {
+ return fmt.Sprintf("the group in %s now has the %s role", region, role)
+}
+
+// setBoxRole asks the console to give box the role, and returns the line
+// to print.
+func setBoxRole(ctx context.Context, post rolePost, box, role string) (string, error) {
+ status, body, err := post(ctx, "/api/cli/system/boxes/"+url.PathEscape(box)+"/role", map[string]string{"role": role})
+ if err != nil {
+ return "", err
+ }
+ if status != http.StatusOK {
+ return "", apiErrorFromBody(status, body)
+ }
+ var out struct {
+ Box string `json:"box"`
+ Role string `json:"role"`
+ }
+ if err := json.Unmarshal(body, &out); err != nil {
+ return "", fmt.Errorf("unreadable response: %w", err)
+ }
+ return boxRoleLine(out.Box, out.Role), nil
+}
+
+// setGroupRole asks the console to give region's group the role, and
+// returns the lines to print: the group's, then one per box it set.
+func setGroupRole(ctx context.Context, post rolePost, region, role string) ([]string, error) {
+ status, body, err := post(ctx, "/api/cli/system/groups/"+url.PathEscape(region)+"/role", map[string]string{"role": role})
+ if err != nil {
+ return nil, err
+ }
+ if status != http.StatusOK {
+ return nil, apiErrorFromBody(status, body)
+ }
+ var out struct {
+ Region string `json:"region"`
+ Role string `json:"role"`
+ Boxes []string `json:"boxes"`
+ }
+ if err := json.Unmarshal(body, &out); err != nil {
+ return nil, fmt.Errorf("unreadable response: %w", err)
+ }
+ lines := []string{groupRoleLine(out.Region, out.Role)}
+ for _, b := range out.Boxes {
+ lines = append(lines, boxRoleLine(b, out.Role))
+ }
+ return lines, nil
+}
+
+// roleArgs reads `role <name> <role>` with an optional --console flag
+// after them, and exits with the usage on anything else.
+func roleArgs(rest []string) (name, role, console string) {
+ if len(rest) < 3 || rest[0] != "role" {
+ systemUsage()
+ os.Exit(2)
+ }
+ name, role = rest[1], rest[2]
+ tail := rest[3:]
+ switch {
+ case len(tail) == 0:
+ case len(tail) == 2 && (tail[0] == "--console" || tail[0] == "-console"):
+ console = tail[1]
+ default:
+ systemUsage()
+ os.Exit(2)
+ }
+ return name, role, console
+}
+
+func runSystemBoxVerb(rest []string) {
+ box, role, consoleFlag := roleArgs(rest)
+ sess := requireDeployAPISession(consoleFlag)
+ printUsing(stdout, sess)
+ line, err := setBoxRole(context.Background(), sess.postJSON, box, role)
+ if err != nil {
+ log.Fatalf("carlos system box role: %v", err)
+ }
+ fmt.Fprintln(stdout, line)
+}
+
+func runSystemGroupVerb(rest []string) {
+ region, role, consoleFlag := roleArgs(rest)
+ sess := requireDeployAPISession(consoleFlag)
+ printUsing(stdout, sess)
+ lines, err := setGroupRole(context.Background(), sess.postJSON, region, role)
+ if err != nil {
+ log.Fatalf("carlos system group role: %v", err)
+ }
+ for _, l := range lines {
+ fmt.Fprintln(stdout, l)
+ }
+}
+````
+
+````diff
+--- a/system.go
++++ b/system.go
+@@ -49,7 +49,7 @@
+ // untouched in rest for runSystemAntivirusVerb to read.
+ // layout is a family too: migrate|revert|status stay in rest for
+ // runSystemLayout.
+- case "guard-arm", "guard-exec", "journal-migrate", "floor-allows-prev", "update", "status", "rollback", "cancel-activation", "apply", "antivirus", "layout", "geoip", "seeds-floor", "legacy-routes", "help", "-h", "--help":
++ case "guard-arm", "guard-exec", "journal-migrate", "floor-allows-prev", "update", "status", "rollback", "cancel-activation", "apply", "antivirus", "layout", "geoip", "seeds-floor", "legacy-routes", "box", "group", "help", "-h", "--help":
+ return args[0], args[1:], component
+ }
+ return "", args, component
+@@ -88,6 +88,10 @@
+ runSystemSeedsFloorVerb(rest)
+ case "legacy-routes":
+ runSystemLegacyRoutesVerb(rest)
++ case "box":
++ runSystemBoxVerb(rest)
++ case "group":
++ runSystemGroupVerb(rest)
+ case "help", "-h", "--help":
+ systemUsage()
+ default:
+@@ -122,6 +126,8 @@
+ carlos system geoip <publish|status|off> [flags]
+ carlos system seeds-floor publish --min-release <release>
+ carlos system legacy-routes assign --name <name> --to <account>/<app>
++ carlos system box role <box> <default|legacy|platform>
++ carlos system group role <region> <default|legacy>
+
+ The platform's own binary (ops account owners and CARLOS_SYSTEM_OPERATORS):
+
+@@ -138,6 +144,12 @@
+ the first release whose boxes can place a seed, such as v0.27.0
+ legacy-routes assign --name <name> --to <account>/<app>
+ `+systemLegacyRoutesVerb+`
++ box role <box> <default|legacy|platform>
++ whether new instances may be placed on a box. A legacy or
++ platform box keeps the instances it has and takes no new ones
++ group role <region> <default|legacy>
++ the same for every box a region's group has enrolled, and for
++ every box it enrolls later
+
+ Deployment-wide settings (CARLOS_STORE_OPERATORS):
+
+````
+
+````diff
+--- a/docs/site/cli.md
++++ b/docs/site/cli.md
+@@ -2200,6 +2200,17 @@
+ carlos system legacy-routes assign --name shop --to bac/shop
+ ```
+
++#### Box and group roles
++
++carlos system box role says whether new instances may be placed on a box. A box with the default role can take them. A legacy or platform box keeps the instances it has and takes no new ones: no new instance, relocation or move lands on it. carlos system group role sets the role of every box a region's group has enrolled, and of every box it enrolls later. Both work only on a console with region HA on.
++
++```sh
++carlos system box role i-0123456789abcdef0 legacy
++carlos system group role ie default
++```
++
++Ops account owners and `CARLOS_SYSTEM_OPERATORS`.
++
+ ### carlos move
+
+ Moves one instance to another region. The instance keeps serving while its
+````
+
+- [ ] **Step 4: Run the tests**
+
+```bash
+nice -n 19 go test ./internal/console/ -run 'BoxRoleDoor|GroupRoleDoor|RoleWriteErrors|RoleDoorsAreRegistered|MoveDestinationSkips|SelectMoveBoxes|EveryCLIDoor|ClaimConsoleRouteCensus|NoRoleReaches' -count=1
+nice -n 19 go test ./internal/console/ -run 'TestClaimConsoleMutationInvariant$/probes/route_POST_/api/cli/system/' -count=1 -v | grep -E -- '--- (PASS|FAIL).*role'
+nice -n 19 go test . -run 'SystemVerb|SetBoxRole|SetGroupRole' -count=1
+nice -n 19 go test ./internal/docsite/ -count=1
+```
+Expected: PASS, and both role probes PASS. The second command runs only the probes, so its parent test reports app-scoped writers "reached by no door's open run": that is the filter, not the change. The whole `TestClaimConsoleMutationInvariant` takes longer than the default ten-minute test timeout; run it once with `-timeout 90m` before handing off (it is part of CI's `make test`).
+
+- [ ] **Step 5: Copy gate, boundary list and census**
+
+````diff
+--- a/approved_region_ha_copy_test.go
++++ b/approved_region_ha_copy_test.go
+@@ -35,7 +35,7 @@
+
+ // regionHACopyWired is every id whose text the code or docs carry. The
+ // task that writes a row's text adds its id here in the same commit.
+-var regionHACopyWired = []string{}
++var regionHACopyWired = []string{"RHA-C1", "RHA-C2", "RHA-C3", "RHA-C4", "RHA-C5", "RHA-C6", "RHA-C7", "RHA-C8", "RHA-C9", "RHA-C10", "RHA-C11", "RHA-C12", "RHA-C13", "RHA-C14", "RHA-C15", "RHA-C16"}
+
+ func mustParseRegionHARecord(t *testing.T) []copyRow {
+ t.Helper()
+````
+
+````diff
+--- a/region_ha_boundary_test.go
++++ b/region_ha_boundary_test.go
+@@ -24,6 +24,8 @@
+ "internal/boxchoice/policy.go",
+ "internal/host/boxaddress.go",
+ "internal/edge/lbhealth.go",
++ "internal/console/apiboxroles.go",
++ "system_roles.go",
+ }
+
+ // regionHAAdapterFiles are spec 2's AWS adapters. Each is named for AWS so
+````
+
+`registerCLIRoutes` is in the record-write census (a key site). Its digest changes:
+
+````diff
+--- a/internal/provision/plan9_b8_census_test.go
++++ b/internal/provision/plan9_b8_census_test.go
+@@ -680,7 +680,7 @@
+ // Registers instance and sidecar HTTP paths; record handlers use the reviewed provision writers.
+ // Re-reviewed at the merge of main into carlos-run (2026-10-13): main's routes (the retry door among them) with the five run doors registered beside them; record writes stay in the reviewed handlers.
+ // Re-reviewed at the merge of carlos-run into seeded-instances (2026-10-13): main's routes and the run doors with the four import doors, the seeds-floor door and the legacy-routes door registered beside them; record writes stay in the reviewed handlers.
+- {"internal/console/authcli.go", "Server.registerCLIRoutes", "key-site", "lit:instances,lit:sidecars", "bb32a85461c3253b2b20f4191ae5fb1f2aa26e5e6c038bc338b6d16a14e7301a"},
++ {"internal/console/authcli.go", "Server.registerCLIRoutes", "key-site", "lit:instances,lit:sidecars", "b55987735c833b952a7ef835302dba728c803379ecb474c3d465bb268cf4c781"},
+ {"internal/console/browserrestart.go", "Server.handleBrowserRestart", "key-site", "lit:instances", "955d71813504f2cfec1e1f64817d5aa5a921aa38e1044c0e903d2cfbe4d073f4"},
+ // Builds the instance page path and picks the confirmation line (a declined box's own line); the only write is the reviewed retry door (requestHostRetry), no record write here.
+ {"internal/console/browserretry.go", "Server.handleInstanceRetry", "key-site", "lit:instances", "4d85d392f3660373f3c282c6391f58d8e25b6df738257ccc067b816e24dfe7ff"},
+````
+
+Ask platform-1e for census ruling region-ha-2a with the `Server.boxRoster` rows below, and write them once it is given (with the ruling's own basis text if it differs; then recompute the fingerprint with `legacyrow.py` and the seal):
+
+Add this comment line after the overlay's last comment line:
+
+````text
+# Task 10: the operator's box and group role doors (internal/console apiboxroles.go, registered with host claims on in registerCLIRoutes), the move door's destination skipping legacy and platform boxes (selectMoveBoxes through boxRoles), and the CLI (system_roles.go). Census ruling region-ha-2a (platform-1e) is asked for one accepted-legacy unit, Server.boxRoster: the console's read of the signed roster (Directory.Read), the read every console roster reader makes, with the console's root signer. Its basis is main's for a console root input (J4). The seal moves.
+````
+
+Then set these rows. A `@coverage` row replaces the overlay's row for the same directory, or is appended when there is none; every other row is appended:
+
+````text
+@coverage internal/console 021626360903282245dfba6d99f505572a4388c4433c39e1aa3880ebb245e368
+@coverage . 011b3548e5615fa1bb046e5a6d0b0e4345206ac7774c571a8320cfd124ba9af3
+@legacy b02b534b6617f57381ba12e9d9a2bf1bd63f737dc68005de926ff18b7db93dd3 internal/console/apiboxroles.go Server.boxRoster legacy accepted-legacy accepted-legacy;rev3=J4;basis=root-input;census-ruling-region-ha-2a file:Read@484 9be9925de14dbbd7d86123bf9faf1535c7b116575cadcb8ba983a665af0513f4
+internal/console/apiboxroles.go Server.boxRoster legacy accepted-legacy accepted-legacy;rev3=J4;basis=root-input;census-ruling-region-ha-2a file:Read@484 9be9925de14dbbd7d86123bf9faf1535c7b116575cadcb8ba983a665af0513f4
+````
+
+````diff
+--- a/internal/provision/plan14_root_reads_test.go
++++ b/internal/provision/plan14_root_reads_test.go
+@@ -769,7 +769,7 @@
+ if err = loadRootReadInventory(filepath.Join(root, "internal/provision/testdata/plan14-root-reads-region-ha-2a.tsv"), &inv, true); err != nil {
+ t.Fatal(err)
+ }
+- seal = "fc38db001ebd8825709a8ca9c41cc0e8f165c1b1967e5967347dc9bd3556afec"
++ seal = "a57250c350c20e4116e5c87b559cfda71f3e0c65d59a2858d83c5f78bea4e9b5"
+ errs := append(compareRootReads(got, inv), checkRootReadLegacy(inv, seal)...)
+ errs = append(errs, checkRootReadDeclarations(got, inv)...)
+ if len(errs) > 0 {
+````
+
+Run the census checks and `go test . -run 'TestRegionHA|RegionHACopy|RegionHAWired' -count=1`. Expected: PASS.
+
+- [ ] **Step 6: Commit**
+
+```bash
+git add internal/console system.go system_roles.go system_roles_test.go docs/site/cli.md approved_region_ha_copy_test.go region_ha_boundary_test.go internal/provision
+git commit -m "region HA 2a: box and group role doors and the carlos system verbs"
+```
+
+---
+
+### Task 11: A create no box can take is refused (H27)
+
+**Files:**
+- Create: `internal/console/nocapacity.go`, `internal/console/nocapacity_test.go`
+- Modify: `internal/console/apiinstances.go` (`handleAPIInstanceCreate` refuses before `writeClaimResponseError`)
+- Modify: `internal/console/server.go` (`Server.noCapacity`)
+- Modify: `internal/hostops/executor_test.go` (the claim is never written)
+- Modify: `approved_region_ha_copy_test.go` (wire RHA-C17 to RHA-C19), `region_ha_boundary_test.go`, `internal/provision/plan9_b8_census_test.go`, the overlay
+
+**Interfaces:**
+- Consumes: `hostops.ErrNoEligibleBox` (from Task 6's `Choose`, through `ProductionAuthority.StartCreate`); `alerting.Page`; `Server.homeRegionOut`.
+- Produces: `func (s *Server) refuseNoCapacity(ctx context.Context, w http.ResponseWriter, err error, region string) bool`; `type noCapacityAlerts` (`due(region string, now time.Time) bool`); `var pageNoCapacity = alerting.Page` (the test seam).
+
+`ProductionAuthority.StartCreate` chooses its executor before any claim write, so a create the chooser refuses writes nothing to the claim: `TestStartCreateWithNoEligibleBoxWritesNothing` pins it against the store. The door answers 503 with `Retry-After: 60` and sentence RHA-C17, and pages at `warning`, at most once per region an hour (in memory: a console restart may page once more). Inert with region HA off.
+
+- [ ] **Step 1: Write the failing tests**
+
+`internal/console/nocapacity_test.go`:
+
+````go
+package console
+
+import (
+ "context"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "net/http"
+ "net/http/httptest"
+ "testing"
+ "time"
+
+ "github.com/carlosframework/platform/internal/hostops"
+)
+
+// Region HA spec 2 §7 (H27): a create no box can take is refused with a
+// 503 to retry, and the operator hears of it once per region an hour.
+func TestCreateWithNoEligibleBoxIsRefusedAndAlertedOncePerHour(t *testing.T) {
+ pages := make(chan string, 8)
+ old := pageNoCapacity
+ pageNoCapacity = func(_ context.Context, level, subject, _ string) error {
+ pages <- level + " " + subject
+ return nil
+ }
+ t.Cleanup(func() { pageNoCapacity = old })
+ s := &Server{cfg: Config{RegionHA: true, HomeRegion: "ie"}}
+ refuse := func(err error, region string) (*httptest.ResponseRecorder, bool) {
+ w := httptest.NewRecorder()
+ return w, s.refuseNoCapacity(context.Background(), w, err, region)
+ }
+ if _, ok := refuse(errors.New("SlowDown"), "uk"); ok {
+ t.Fatal("another error was refused as no capacity")
+ }
+ wrapped := fmt.Errorf("%w: box-a", hostops.ErrNoEligibleBox)
+ w, ok := refuse(wrapped, "uk")
+ if !ok || w.Code != http.StatusServiceUnavailable || w.Header().Get("Retry-After") != "60" {
+ t.Fatalf("refusal: %v %d %v", ok, w.Code, w.Header())
+ }
+ var out map[string]string
+ if err := json.Unmarshal(w.Body.Bytes(), &out); err != nil || out["error"] != noCapacityRefusal("uk") {
+ t.Fatalf("body %s", w.Body.String())
+ }
+ if got := <-pages; got != "warning "+mustSubject("uk") {
+ t.Fatalf("page %q", got)
+ }
+ // A retry loop: refused every time, paged once.
+ for range 5 {
+ if _, ok := refuse(hostops.ErrNoEligibleBox, "uk"); !ok {
+ t.Fatal("a retry was not refused")
+ }
+ }
+ // The home region by its name, and its own alert.
+ if w, _ := refuse(hostops.ErrNoEligibleBox, ""); !jsonHas(w, noCapacityRefusal("ie")) {
+ t.Fatalf("home refusal %s", w.Body.String())
+ }
+ if got := <-pages; got != "warning "+mustSubject("ie") {
+ t.Fatalf("home page %q", got)
+ }
+ select {
+ case got := <-pages:
+ t.Fatalf("paged again within the hour: %q", got)
+ case <-time.After(100 * time.Millisecond):
+ }
+ s.cfg.RegionHA = false
+ if _, ok := refuse(hostops.ErrNoEligibleBox, "uk"); ok {
+ t.Fatal("region HA off: the door's main answer must stand")
+ }
+}
+
+func TestNoCapacityAlertsRepeatAfterAnHour(t *testing.T) {
+ var a noCapacityAlerts
+ t0 := time.Date(2026, 10, 16, 12, 0, 0, 0, time.UTC)
+ if !a.due("uk", t0) || a.due("uk", t0.Add(59*time.Minute)) || !a.due("ie", t0) || !a.due("uk", t0.Add(time.Hour)) {
+ t.Fatal("once per region an hour")
+ }
+}
+
+func mustSubject(region string) string {
+ subject, _ := noCapacityAlert(region)
+ return subject
+}
+
+func jsonHas(w *httptest.ResponseRecorder, msg string) bool {
+ var out map[string]string
+ return json.Unmarshal(w.Body.Bytes(), &out) == nil && out["error"] == msg
+}
+````
+
+````diff
+--- a/internal/hostops/executor_test.go
++++ b/internal/hostops/executor_test.go
+@@ -488,3 +488,30 @@
+ t.Fatalf("unbound: %v", err)
+ }
+ }
++
++// Region HA spec 2 §7 (H27): with every box ineligible a create is
++// refused before anything is written. The executor is chosen first, so no
++// claim without one ever exists.
++func TestStartCreateWithNoEligibleBoxWritesNothing(t *testing.T) {
++ ctx := context.Background()
++ x, s, snap, _ := chooserFixture(t, "box-a", "box-b")
++ for i := range snap.Boxes {
++ snap.Boxes[i].Headroom.MemFreeMiB = 100 // under 15% of 8000
++ }
++ r := chooserRecord()
++ a := ProductionAuthority{Store: s, Directory: x.Directory, Namespace: "platform", Now: x.Now, Executors: &x}
++ before, err := s.List(ctx, "")
++ if err != nil {
++ t.Fatal(err)
++ }
++ if _, err := a.StartCreate(ctx, hostclaim.Observed{Host: r.Holder.Host, Absent: true}, r, "op-1", "paul@keymail.dev"); !errors.Is(err, ErrNoEligibleBox) {
++ t.Fatalf("StartCreate = %v, want ErrNoEligibleBox", err)
++ }
++ if obs, err := hostclaim.Read(ctx, s, r.Holder.Host); err != nil || !obs.Absent {
++ t.Fatalf("claim after the refusal: %+v %v", obs, err)
++ }
++ after, err := s.List(ctx, "")
++ if err != nil || len(after) != len(before) {
++ t.Fatalf("objects before %v, after %v (%v)", before, after, err)
++ }
++}
+````
+
+- [ ] **Step 2: Run them to see them fail**
+
+Run: `nice -n 19 go test ./internal/console/ -run 'NoEligibleBox|NoCapacity' -count=1`
+Expected: FAIL to build (`undefined: pageNoCapacity`, `s.refuseNoCapacity undefined`). The hostops test passes already: it pins what Task 6 left true.
+
+- [ ] **Step 3: Write the refusal**
+
+`internal/console/nocapacity.go`:
+
+````go
+package console
+
+import (
+ "context"
+ "errors"
+ "fmt"
+ "log"
+ "net/http"
+ "sync"
+ "time"
+
+ "github.com/carlosframework/platform/internal/alerting"
+ "github.com/carlosframework/platform/internal/hostops"
+)
+
+// A create no box can take (region HA spec 2 §7, ruling H27). When the
+// placement policy finds no eligible box, the create door refuses the
+// create with a 503 the client may retry, and the claim is never written:
+// ProductionAuthority.StartCreate chooses its executor before any claim
+// write, so no create without an executor exists. An operator alert fires
+// too, at most once per region an hour, so a client retrying in a loop
+// does not flood the operator.
+
+// noCapacityAlertEvery is how often one region's alert may repeat.
+const noCapacityAlertEvery = time.Hour
+
+func noCapacityRefusal(region string) string {
+ return fmt.Sprintf("no box in %s can take a new instance right now. Try again in a few minutes.", region)
+}
+
+func noCapacityAlert(region string) (subject, body string) {
+ return fmt.Sprintf("No room for new instances in %s", region),
+ fmt.Sprintf("A create in %s was refused because no box there can take a new instance. Every box is full, stale or excluded by its role. Add a box, or free memory or disk on one, and check the eligible-capacity alarm for %s.", region, region)
+}
+
+// pageNoCapacity sends the alert; tests replace it.
+var pageNoCapacity = alerting.Page
+
+// noCapacityAlerts remembers when each region's alert was last sent. The
+// zero value is ready.
+type noCapacityAlerts struct {
+ mu sync.Mutex
+ last map[string]time.Time
+}
+
+// due reports whether region's alert should be sent at now, and if so
+// records it as sent.
+func (a *noCapacityAlerts) due(region string, now time.Time) bool {
+ a.mu.Lock()
+ defer a.mu.Unlock()
+ if at, ok := a.last[region]; ok && now.Sub(at) < noCapacityAlertEvery {
+ return false
+ }
+ if a.last == nil {
+ a.last = map[string]time.Time{}
+ }
+ a.last[region] = now
+ return true
+}
+
+// refuseNoCapacity answers a create whose placement found no eligible
+// box, and reports whether it did. region is the record's placement code
+// ("" for the home). Inert with region HA off.
+func (s *Server) refuseNoCapacity(ctx context.Context, w http.ResponseWriter, err error, region string) bool {
+ if !s.cfg.RegionHA || !errors.Is(err, hostops.ErrNoEligibleBox) {
+ return false
+ }
+ name := s.homeRegionOut(region)
+ log.Printf("console: create refused: no eligible box in %s: %v", name, err)
+ if s.noCapacity.due(name, time.Now()) {
+ subject, body := noCapacityAlert(name)
+ go func() {
+ ctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 30*time.Second)
+ defer cancel()
+ if err := pageNoCapacity(ctx, alerting.LevelWarning, subject, body); err != nil {
+ log.Printf("console: no-capacity alert for %s: %v", name, err)
+ }
+ }()
+ }
+ w.Header().Set("Retry-After", "60")
+ jsonError(w, http.StatusServiceUnavailable, noCapacityRefusal(name))
+ return true
+}
+````
+
+````diff
+--- a/internal/console/server.go
++++ b/internal/console/server.go
+@@ -837,6 +837,11 @@
+ type Server struct {
+ claimLossPageMu sync.Mutex
+ claimLossPageDone chan struct{}
++
++ // noCapacity limits the no-capacity alert to once per region an hour
++ // (region HA spec 2 §7, nocapacity.go).
++ noCapacity noCapacityAlerts
++
+ // Enabled claim runtime (ConfigureClaimRuntime). Unset flag-off.
+ claimRuntimeConfigured bool
+ claimOwner hostops.RegistrationOwner
+````
+
+````diff
+--- a/internal/console/apiinstances.go
++++ b/internal/console/apiinstances.go
+@@ -1588,6 +1588,11 @@
+ return prepared, err
+ })
+ if createErr != nil {
++ // No box can take it (region HA spec 2 §7, H27): a 503 to retry,
++ // and nothing was written to the claim.
++ if s.refuseNoCapacity(r.Context(), w, createErr, rec.Region) {
++ return
++ }
+ writeClaimResponseError(w, createErr)
+ return
+ }
+````
+
+- [ ] **Step 4: Run the tests**
+
+Run: `nice -n 19 go test ./internal/console/ -run 'NoEligibleBox|NoCapacity' -count=1 && nice -n 19 go test ./internal/hostops/ -run 'NoEligibleBoxWritesNothing|Executor' -count=1`
+Expected: PASS.
+
+- [ ] **Step 5: Copy gate, boundary list and census**
+
+````diff
+--- a/approved_region_ha_copy_test.go
++++ b/approved_region_ha_copy_test.go
+@@ -35,7 +35,7 @@
+
+ // regionHACopyWired is every id whose text the code or docs carry. The
+ // task that writes a row's text adds its id here in the same commit.
+-var regionHACopyWired = []string{"RHA-C1", "RHA-C2", "RHA-C3", "RHA-C4", "RHA-C5", "RHA-C6", "RHA-C7", "RHA-C8", "RHA-C9", "RHA-C10", "RHA-C11", "RHA-C12", "RHA-C13", "RHA-C14", "RHA-C15", "RHA-C16"}
++var regionHACopyWired = []string{"RHA-C1", "RHA-C2", "RHA-C3", "RHA-C4", "RHA-C5", "RHA-C6", "RHA-C7", "RHA-C8", "RHA-C9", "RHA-C10", "RHA-C11", "RHA-C12", "RHA-C13", "RHA-C14", "RHA-C15", "RHA-C16", "RHA-C17", "RHA-C18", "RHA-C19"}
+
+ func mustParseRegionHARecord(t *testing.T) []copyRow {
+ t.Helper()
+````
+
+````diff
+--- a/region_ha_boundary_test.go
++++ b/region_ha_boundary_test.go
+@@ -26,6 +26,7 @@
+ "internal/edge/lbhealth.go",
+ "internal/console/apiboxroles.go",
+ "system_roles.go",
++ "internal/console/nocapacity.go",
+ }
+
+ // regionHAAdapterFiles are spec 2's AWS adapters. Each is named for AWS so
+````
+
+`handleAPIInstanceCreate` is a record writer in the record-write census (`key:CreateRecord`). Its digest changes:
+
+````diff
+--- a/internal/provision/plan9_b8_census_test.go
++++ b/internal/provision/plan9_b8_census_test.go
+@@ -662,7 +662,7 @@
+ // Region HA stage B: the edges read moved whole into internal/edgesfile (EvidenceComplete); the record is still created through CreateRecord.
+ // Re-reviewed 2026-10-12 (merge of main's home-region work): main's body (claimPlacementRegion) with stage B's one-line EvidenceComplete call; no new write.
+ // Re-reviewed at the merge of carlos-run into seeded-instances (2026-10-13): main's door with the two seed hand-offs: once every ordinary check has passed, a create that names `from` or `import` leaves for createFrom or createImport, which write their own record (beginFrom, beginImport, below) and refuse a claim console first (seedRegionReady); the ordinary create asks bindEnvironmentCreate before its write. The door's own CreateRecord and its claim create are as main wrote them.
+- {"internal/console/apiinstances.go", "Server.handleAPIInstanceCreate", "record-writer", "key:CreateRecord", "6763eab45774b9e624a9b820ce1585ef3910355bc3ce589b82298e1d12469a2e"},
++ {"internal/console/apiinstances.go", "Server.handleAPIInstanceCreate", "record-writer", "key:CreateRecord", "1f2bda930d791d6be625eccb9cb0b68c5ddf1e225e6580d787771187f3766c60"},
+ // Re-reviewed at the merge of carlos-run into seeded-instances (2026-10-13): main's door under the environment-binding lock (lockEnvironmentUpdate); the UpdateRecord call is unchanged, and UpdateRecord refuses any change to the record's seed.
+ {"internal/console/apiinstances.go", "Server.handleAPIInstanceUpdate", "record-writer", "key:UpdateRecord", "43a8bead675a4aa33777c6c68431882d55d79a9c1d27e1a54c8f103dda087de0"},
+ {"internal/console/apiscan.go", "Server.writeScanRecord", "nonrecord", "sel:Put", "c3c092f4b7be9bf6bf09d0fe1e2c9fe52892ee97e57f6c7ab964b85fe36fb5c6"},
+````
+
+Add this comment line after the overlay's last comment line:
+
+````text
+# Task 11: the create door refuses a create no box can take with a 503 and an hourly operator alert (internal/console nocapacity.go, called from handleAPIInstanceCreate; region HA spec 2 §7 as ruled by H27). Neither unit is a root-read unit. Envelope-only, and the seal stands.
+````
+
+Then set these rows. A `@coverage` row replaces the overlay's row for the same directory, or is appended when there is none; every other row is appended:
+
+````text
+@coverage internal/console c6e6f902978fd3edc87cb8bff14d87dd8ddc3f734ba08aebb32c003af379fb9c
+````
+
+Run the census checks and `go test . -run 'TestRegionHA|RegionHACopy|RegionHAWired' -count=1`. Expected: PASS.
+
+- [ ] **Step 6: Commit**
+
+```bash
+git add internal/console/nocapacity.go internal/console/nocapacity_test.go internal/console/server.go internal/console/apiinstances.go internal/hostops/executor_test.go approved_region_ha_copy_test.go region_ha_boundary_test.go internal/provision
+git commit -m "region HA 2a: refuse a create no box can take, and alert once an hour"
+```
+
+---
+
+### Task 12: The eligible-capacity metric
+
+**Files:**
+- Create: `internal/console/regioncapacity.go` (core: the count and the pass)
+- Create: `internal/console/awsregionmetric.go` (the AWS adapter: CloudWatch)
+- Create: `internal/console/regioncapacity_test.go`
+- Modify: `internal/console/hostclaimops.go` (`StartClaimRecoveryPass` starts the pass)
+- Modify: `region_ha_boundary_test.go`, the overlay
+
+**Interfaces:**
+- Consumes: Task 6's `ExecutorChooser.Capacity`; `Server.liveEdges`; `rootaws.LoadDefaultConfig`.
+- Produces: `type regionCapacity struct{ Placement, AWSRegion string; Eligible int }`; `type capacityPublisher interface{ PublishCapacity(ctx, []regionCapacity) error }`; `func regionCapacities(ctx, snap boxchoice.Snapshot, capacity func(context.Context, string) (int, error)) ([]regionCapacity, error)`; `(s *Server) capacityChooser`, `publishCapacity`, `runCapacityPass`; the adapter's `capacityData`, `cloudWatchCapacity`, `capacityPublisherFromEnv(ctx, getenv)`. **For 2c:** the metric is `Carlos/Region` / `EligibleBoxes`, dimension `Region` = the placement code, unit Count, every minute, in the region's own CloudWatch; the console needs `CARLOS_REGION_METRICS=cloudwatch` and the platform box role's `cloudwatch:PutMetricData` on that namespace.
+
+The pass runs beside claim recovery, so only on a host-claim console, and only with region HA on and a publisher armed. Regions are the placements the boxes' edges files name, so a region whose every box has stopped publishing gets no data point, which the alarm reads as breaching. A count that fails stops the whole pass rather than publish a partial set.
+
+- [ ] **Step 1: Write the failing tests**
+
+`internal/console/regioncapacity_test.go`:
+
+````go
+package console
+
+import (
+ "context"
+ "errors"
+ "reflect"
+ "testing"
+ "time"
+
+ "github.com/aws/aws-sdk-go-v2/service/cloudwatch"
+
+ "github.com/carlosframework/platform/internal/boxchoice"
+ "github.com/carlosframework/platform/internal/hostops"
+)
+
+// The count runs without any cloud adapter (§25): it reads only the
+// edges snapshot and the chooser.
+func TestRegionCapacitiesCountsEachPlacement(t *testing.T) {
+ snap := boxchoice.Snapshot{Complete: true, Boxes: []boxchoice.Box{
+ {BoxID: "a", Placement: "uk", Region: "eu-west-2"},
+ {BoxID: "b", Placement: "uk", Region: "eu-west-2"},
+ {BoxID: "c", Placement: "ie", Region: "eu-west-1"},
+ {BoxID: "d", Placement: "fleet/pi", Region: "eu-west-1"},
+ {BoxID: "e", Placement: ""},
+ }}
+ asked := []string{}
+ capacity := func(_ context.Context, p string) (int, error) {
+ asked = append(asked, p)
+ switch p {
+ case "uk":
+ return 2, nil
+ case "ie":
+ return 0, nil
+ }
+ return 0, hostops.ErrContract // no roster region: left out
+ }
+ got, err := regionCapacities(context.Background(), snap, capacity)
+ if err != nil {
+ t.Fatal(err)
+ }
+ want := []regionCapacity{{Placement: "ie", AWSRegion: "eu-west-1", Eligible: 0}, {Placement: "uk", AWSRegion: "eu-west-2", Eligible: 2}}
+ if !reflect.DeepEqual(got, want) || !reflect.DeepEqual(asked, []string{"fleet/pi", "ie", "uk"}) {
+ t.Fatalf("got %+v asked %v", got, asked)
+ }
+ // Any other error stops the count: a partial set would read as no
+ // capacity.
+ failing := func(context.Context, string) (int, error) { return 0, errors.New("SlowDown") }
+ if _, err := regionCapacities(context.Background(), snap, failing); err == nil {
+ t.Fatal("a failed count published")
+ }
+}
+
+func TestCapacityChooserIsScopedLikeCreate(t *testing.T) {
+ s := &Server{}
+ if s.capacityChooser() != nil {
+ t.Fatal("a chooser with no claim runtime")
+ }
+ x := &hostops.ExecutorChooser{Namespace: "stale", Pool: "acme"}
+ s.cfg.ClaimAuthority = &hostops.ProductionAuthority{Namespace: "platform", Executors: x}
+ got := s.capacityChooser()
+ if got == nil || got.Namespace != "platform" || got.Pool != "" || x.Namespace != "stale" {
+ t.Fatalf("chooser %+v, original %+v", got, x)
+ }
+}
+
+func TestCloudWatchCapacityPublishesEachRegionInItsOwnRegion(t *testing.T) {
+ now := time.Date(2026, 10, 16, 12, 0, 0, 0, time.UTC)
+ type call struct {
+ region, ns, metric, dim string
+ value float64
+ }
+ var calls []call
+ c := cloudWatchCapacity{now: func() time.Time { return now }, put: func(_ context.Context, region string, in *cloudwatch.PutMetricDataInput) error {
+ d := in.MetricData[0]
+ calls = append(calls, call{region, *in.Namespace, *d.MetricName, *d.Dimensions[0].Name + "=" + *d.Dimensions[0].Value, *d.Value})
+ if !d.Timestamp.Equal(now) || d.Unit != "Count" {
+ t.Fatalf("datum %+v", d)
+ }
+ return nil
+ }}
+ err := c.PublishCapacity(context.Background(), []regionCapacity{{"ie", "eu-west-1", 2}, {"uk", "eu-west-2", 0}})
+ want := []call{{"eu-west-1", "Carlos/Region", "EligibleBoxes", "Region=ie", 2}, {"eu-west-2", "Carlos/Region", "EligibleBoxes", "Region=uk", 0}}
+ if err != nil || !reflect.DeepEqual(calls, want) {
+ t.Fatalf("calls %+v %v", calls, err)
+ }
+ if capacityPublisherFromEnv(context.Background(), func(string) string { return "" }) != nil {
+ t.Fatal("published with CARLOS_REGION_METRICS unset")
+ }
+}
+````
+
+- [ ] **Step 2: Run them to see them fail**
+
+Run: `nice -n 19 go test ./internal/console/ -run 'RegionCapacities|CapacityChooser|CloudWatchCapacity' -count=1`
+Expected: FAIL to build (`undefined: regionCapacities`, `undefined: cloudWatchCapacity`).
+
+- [ ] **Step 3: Write the pass and the adapter**
+
+`internal/console/regioncapacity.go`:
+
+````go
+package console
+
+import (
+ "context"
+ "errors"
+ "log"
+ "sort"
+ "time"
+
+ "github.com/carlosframework/platform/internal/boxchoice"
+ "github.com/carlosframework/platform/internal/hostops"
+)
+
+// The eligible-capacity metric (region HA spec 2 §4, A1-11): once a
+// minute the console counts, for each region its boxes publish, the boxes
+// that could take a new instance now (hostops.ExecutorChooser.Capacity:
+// enrolled, active, unfenced and placeable), and publishes the counts. An
+// alarm on the count catches what the cloud's own group metrics cannot: a
+// box that is in service but never enrolled, and a launch loop. Where the
+// counts go is an adapter's business (awsregionmetric.go); a console with
+// none publishes nothing.
+
+// capacityInterval is how often the counts are published.
+const capacityInterval = time.Minute
+
+// regionCapacity is one region's count. Placement is the CARLOS placement
+// code the metric is keyed by; AWSRegion is where its boxes run.
+type regionCapacity struct {
+ Placement, AWSRegion string
+ Eligible int
+}
+
+// capacityPublisher sends the counts to a metrics service.
+type capacityPublisher interface {
+ PublishCapacity(ctx context.Context, counts []regionCapacity) error
+}
+
+// regionCapacities counts each placement the snapshot's files name, sorted
+// by placement. A placement the roster does not know (no region entry, so
+// no box there can be chosen) is left out. Any other error stops the
+// count: a partial set would read as regions with no capacity.
+func regionCapacities(ctx context.Context, snap boxchoice.Snapshot, capacity func(context.Context, string) (int, error)) ([]regionCapacity, error) {
+ aws := map[string]string{}
+ for _, b := range snap.Boxes {
+ if b.Placement != "" && b.Region != "" {
+ aws[b.Placement] = b.Region
+ }
+ }
+ placements := make([]string, 0, len(aws))
+ for p := range aws {
+ placements = append(placements, p)
+ }
+ sort.Strings(placements)
+ var out []regionCapacity
+ for _, p := range placements {
+ n, err := capacity(ctx, p)
+ if errors.Is(err, hostops.ErrContract) {
+ continue
+ }
+ if err != nil {
+ return nil, err
+ }
+ out = append(out, regionCapacity{Placement: p, AWSRegion: aws[p], Eligible: n})
+ }
+ return out, nil
+}
+
+// capacityChooser is the create door's chooser scoped as SelectExecutor
+// scopes it for the shared pool, or nil without one (region HA off, or no
+// claim runtime).
+func (s *Server) capacityChooser() *hostops.ExecutorChooser {
+ a := s.cfg.ClaimAuthority
+ if a == nil || a.Executors == nil {
+ return nil
+ }
+ x := *a.Executors
+ x.Namespace, x.Pool = a.Namespace, ""
+ return &x
+}
+
+// publishCapacity counts and publishes once.
+func (s *Server) publishCapacity(ctx context.Context, pub capacityPublisher) error {
+ x := s.capacityChooser()
+ if x == nil {
+ return hostops.ErrContract
+ }
+ snap, err := s.liveEdges(ctx)
+ if err != nil {
+ return err
+ }
+ counts, err := regionCapacities(ctx, boxSnapshot(snap), x.Capacity)
+ if err != nil {
+ return err
+ }
+ return pub.PublishCapacity(ctx, counts)
+}
+
+// runCapacityPass publishes every capacityInterval until ctx ends. A
+// failed pass is logged on its first failure and every tenth; the alarm
+// treats missing data as breaching, so a console that stops publishing is
+// itself an alert.
+func (s *Server) runCapacityPass(ctx context.Context, pub capacityPublisher) {
+ log.Printf("console: eligible-capacity metric armed (every %s)", capacityInterval)
+ t := time.NewTicker(capacityInterval)
+ defer t.Stop()
+ fails := 0
+ for {
+ pass, cancel := context.WithTimeout(ctx, capacityInterval)
+ err := s.publishCapacity(pass, pub)
+ cancel()
+ switch {
+ case err != nil:
+ fails++
+ if fails == 1 || fails%10 == 0 {
+ log.Printf("console: eligible-capacity metric (failure #%d): %v", fails, err)
+ }
+ case fails > 0:
+ log.Printf("console: eligible-capacity metric recovered after %d failure(s)", fails)
+ fails = 0
+ }
+ select {
+ case <-ctx.Done():
+ return
+ case <-t.C:
+ }
+ }
+}
+````
+
+`internal/console/awsregionmetric.go`:
+
+````go
+package console
+
+import (
+ "context"
+ "errors"
+ "log"
+ "strings"
+ "time"
+
+ "github.com/aws/aws-sdk-go-v2/aws"
+ "github.com/aws/aws-sdk-go-v2/service/cloudwatch"
+ "github.com/aws/aws-sdk-go-v2/service/cloudwatch/types"
+
+ "github.com/carlosframework/platform/internal/rootaws"
+)
+
+// The AWS adapter of the eligible-capacity metric (region HA spec 2 §4,
+// §25): CloudWatch, Carlos/Region EligibleBoxes, keyed by the placement
+// code, in each region's own CloudWatch, where the region-host module's
+// alarm reads it. Armed by CARLOS_REGION_METRICS=cloudwatch on the
+// console. The platform box's role needs cloudwatch:PutMetricData on the
+// Carlos/Region namespace: plan 2c grants it.
+
+const (
+ capacityNamespace = "Carlos/Region"
+ capacityMetric = "EligibleBoxes"
+)
+
+// capacityData is the exact PutMetricData input for one region's count.
+func capacityData(c regionCapacity, now time.Time) *cloudwatch.PutMetricDataInput {
+ return &cloudwatch.PutMetricDataInput{Namespace: aws.String(capacityNamespace), MetricData: []types.MetricDatum{{
+ MetricName: aws.String(capacityMetric),
+ Dimensions: []types.Dimension{{Name: aws.String("Region"), Value: aws.String(c.Placement)}},
+ Timestamp: aws.Time(now),
+ Unit: types.StandardUnitCount,
+ Value: aws.Float64(float64(c.Eligible)),
+ }}}
+}
+
+// cloudWatchCapacity publishes each count to its region's CloudWatch.
+type cloudWatchCapacity struct {
+ put func(ctx context.Context, awsRegion string, in *cloudwatch.PutMetricDataInput) error
+ now func() time.Time
+}
+
+func (c cloudWatchCapacity) PublishCapacity(ctx context.Context, counts []regionCapacity) error {
+ var errs []error
+ for _, n := range counts {
+ errs = append(errs, c.put(ctx, n.AWSRegion, capacityData(n, c.now())))
+ }
+ return errors.Join(errs...)
+}
+
+// capacityPublisherFromEnv is the console's publisher: CloudWatch when
+// CARLOS_REGION_METRICS is "cloudwatch", and nil otherwise, or when the
+// AWS configuration cannot be loaded (logged).
+func capacityPublisherFromEnv(ctx context.Context, getenv func(string) string) capacityPublisher {
+ if strings.TrimSpace(getenv("CARLOS_REGION_METRICS")) != "cloudwatch" {
+ return nil
+ }
+ cfg, err := rootaws.LoadDefaultConfig(ctx)
+ if err != nil {
+ log.Printf("console: eligible-capacity metric disabled: load aws config: %v", err)
+ return nil
+ }
+ return cloudWatchCapacity{now: time.Now, put: func(ctx context.Context, awsRegion string, in *cloudwatch.PutMetricDataInput) error {
+ _, err := cloudwatch.NewFromConfig(cfg, func(o *cloudwatch.Options) { o.Region = awsRegion }).PutMetricData(ctx, in)
+ return err
+ }}
+}
+````
+
+````diff
+--- a/internal/console/hostclaimops.go
++++ b/internal/console/hostclaimops.go
+@@ -4,6 +4,7 @@
+ "context"
+ "errors"
+ "log"
++ "os"
+ "time"
+
+ "github.com/carlosframework/platform/internal/hostclaim"
+@@ -43,6 +44,14 @@
+ log.Printf("console: claim recovery: %v", hostops.ErrContract)
+ return
+ }
++ // The eligible-capacity metric (region HA spec 2 §4, regioncapacity.go)
++ // runs beside claim recovery, on a host-claim console with region HA
++ // on and a metrics adapter armed.
++ if s.cfg.RegionHA {
++ if pub := capacityPublisherFromEnv(ctx, os.Getenv); pub != nil {
++ go s.runCapacityPass(ctx, pub)
++ }
++ }
+ t := time.NewTicker(claimRecoveryInterval)
+ defer t.Stop()
+ for {
+````
+
+- [ ] **Step 4: Run the tests**
+
+Run: `nice -n 19 go test ./internal/console/ -run 'RegionCapacities|CapacityChooser|CloudWatchCapacity' -count=1`
+Expected: PASS.
+
+- [ ] **Step 5: Boundary list and census**
+
+````diff
+--- a/region_ha_boundary_test.go
++++ b/region_ha_boundary_test.go
+@@ -27,12 +27,14 @@
+ "internal/console/apiboxroles.go",
+ "system_roles.go",
+ "internal/console/nocapacity.go",
++ "internal/console/regioncapacity.go",
+ }
+
+ // regionHAAdapterFiles are spec 2's AWS adapters. Each is named for AWS so
+ // a reader can see where the provider lives.
+ var regionHAAdapterFiles = []string{
+ "internal/host/awsaddr.go",
++ "internal/console/awsregionmetric.go",
+ }
+
+ // regionHAPackages are the packages spec 2 touches. Every S3 client built
+````
+
+Add this comment line after the overlay's last comment line:
+
+````text
+# Task 12: the eligible-capacity metric (internal/console regioncapacity.go, counted through hostops.ExecutorChooser.Capacity; awsregionmetric.go, the CloudWatch adapter), started beside claim recovery (StartClaimRecoveryPass). No root-read unit changed or was added. Envelope-only, and the seal stands.
+````
+
+Then set these rows. A `@coverage` row replaces the overlay's row for the same directory, or is appended when there is none; every other row is appended:
+
+````text
+@coverage internal/console 415c03036eab35637459b27ccf0078e96d5ad57f5c10689df3d79074765a8221
+````
+
+Run the census checks and `go test . -run TestRegionHA -count=1`. Expected: PASS: `regioncapacity.go` imports no AWS package, and `awsregionmetric.go` is named for AWS.
+
+- [ ] **Step 6: Commit**
+
+```bash
+git add internal/console/regioncapacity.go internal/console/regioncapacity_test.go internal/console/awsregionmetric.go internal/console/hostclaimops.go region_ha_boundary_test.go internal/provision/testdata/plan14-root-reads-region-ha-2a.tsv
+git commit -m "region HA 2a: publish each region's eligible capacity"
+```
+
+---
+
+## Before handing off
+
+- [ ] `gofmt -l .` is empty, and `go build ./...` passes with `GOOS` linux, darwin and windows.
+- [ ] `GOOS=linux go vet ./...` and `GOOS=darwin go vet ./...` pass.
+- [ ] Every task's targeted tests pass again, then once: `nice -n 19 go test ./internal/console/ -run 'TestClaimConsoleMutationInvariant$' -count=1 -timeout 90m` and the census checks.
+- [ ] Push the branch and read `amadan ci status`: CI's `make test` is the full run, `internal/host` included.
+- [ ] The rollout checklist for region HA gains: the copy record approved before any production console turns `CARLOS_REGION_HA` on; consoles before boxes (the console is the roster's only writer, and an older console's roster write would drop the roles); 2c's IAM grant and `CARLOS_REGION_METRICS` before the alarm exists.
+
+## Open questions
+
+1. **"The default certificate available" (§6).** The edge has no single default certificate: every certificate is per host. Task 9 reads the condition as "the certificate mirror has landed `control/certs/` in the local cache once" (latched like forwarding). A deployment with no published certificate passes it. If a specific certificate is meant (the apps-domain wildcard?), the condition needs naming.
+2. **H27's refusal covers the instance create door only.** The sidecar door and the first-route path also create through the claim; with no eligible box they keep today's 409-class claim refusal, without the alert. Should they share the 503 and the alert?
+3. **The move door's destination (brief item 1).** Task 10 filters legacy and platform boxes out of the destination, which leaves the door's existing "no box in `<to>` publishes an edges file yet" when only excluded boxes remain. A sentence of its own means changing `handleAPIInstanceMoveRequest`, an accepted-legacy unit that 2d rewrites anyway. Is the interim wording acceptable until 2d?
+4. **Hosting before 2b.** Task 9 marks a box hosting when the claim agent's wiring starts, which proves enrollment and a matching roster row but does not itself check that the floor guard is armed for that enrollment. 2b's mode selection makes it exact. Is that acceptable for 2a, given no load balancer reads the answer until 2c?
+5. **Census rulings.** Task 7's authority re-digest and Task 10's accepted-legacy `Server.boxRoster` need platform-1e (census ruling region-ha-2a). On a base newer than `70e57449` every census value here must be recomputed (see Census tooling).