amadan
Plan 2a rev 3: Astra round 2 and census ruling region-ha-2a rev 2
Answers Astra's round 2 (three P2s, all accepted): with region HA on, the move door's earlier placement checks (no edges file, blank public addresses) and instance create's armed-placement gate answer a region with no room as H27's 503 and alert (roomIn); the instance create door chooses before its first write, the environment binding, and beginClaimCreate keeps the carried choice; the capacity metric takes each region's AWS region from the signed roster through ExecutorChooser.Capacity. Meets ruling rev 2's conditions: chooseExecutorFirst takes no lease, writes nothing and never chooses again (tested); the move room check only refuses and never narrows the candidates (tested both ways); no record-write site is added or changed. One new census item: Server.chooseCreateFirst. Fixes the census tooling's unitdigest build line. 13 tasks, every code block dry-run and replayed onto a clean export.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+717
−158
docs/superpowers/plans/2026-10-16-region-ha-2a-core.md+717 −158
diff --git a/docs/superpowers/plans/2026-10-16-region-ha-2a-core.md b/docs/superpowers/plans/2026-10-16-region-ha-2a-core.md| index 86857d493..4e9e74137 100644 |
| --- a/docs/superpowers/plans/2026-10-16-region-ha-2a-core.md |
| +++ b/docs/superpowers/plans/2026-10-16-region-ha-2a-core.md |
| @@ -1,10 +1,10 @@ |
| -# Region HA 2a: The Provider-Neutral Core Implementation Plan (rev 2) |
| +# Region HA 2a: The Provider-Neutral Core Implementation Plan (rev 3) |
| > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| **Goal:** Build region HA spec 2's provider-neutral core, dormant behind `CARLOS_REGION_HA`: box and group roles, the placement policy, H27's no-room refusal on every door that chooses a box, the load balancer health answer, private-address forwarding with a validated address source, and the eligible-capacity metric. |
| -**Architecture:** Every new rule sits on spec 1's code without changing what it proves. The placement policy runs over `boxchoice.Eligible`'s identity-filtered candidates inside `hostops.ExecutorChooser`, which creates, relocation targets and (with region HA on) move destinations share; roles live in the signed roster. A create door chooses its box before it registers anything, so a refusal leaves nothing behind. The health answer is an in-memory state (`internal/lbhealth`) that the agent's own loops feed and the edge reads on `:80`. Forwarding changes only the address an edge dials. Each AWS part is an adapter in a file named for AWS (`internal/host/awsaddr.go`, `internal/console/awsregionmetric.go`), and a test holds the core to that. |
| +**Architecture:** Every new rule sits on spec 1's code without changing what it proves. The placement policy runs over `boxchoice.Eligible`'s identity-filtered candidates inside `hostops.ExecutorChooser`, which creates and relocation targets share, and which the create and move doors ask whether a region has room; roles live in the signed roster. A create door chooses its box before its first write, so a refusal leaves nothing behind. The health answer is an in-memory state (`internal/lbhealth`) that the agent's own loops feed and the edge reads on `:80`. Forwarding changes only the address an edge dials. Each AWS part is an adapter in a file named for AWS (`internal/host/awsaddr.go`, `internal/console/awsregionmetric.go`), and a test holds the core to that. |
| **Tech Stack:** Go 1.26, module `github.com/carlosframework/platform`; `internal/hostactors`, `internal/boxchoice`, `internal/edgesfile`, `internal/hostops`, `internal/host`, `internal/edge`, `internal/activator`, `internal/console`, the `carlos` CLI (repo root); AWS SDK v2 CloudWatch in one adapter. |
| @@ -29,6 +29,21 @@ Rev 2 answers Astra's review of rev 1 (`docs/superpowers/plans/CODEX-REVIEW-REGI |
| Task numbers moved: the move door is its own task (12), and the metric is now Task 13. |
| +## Rev 3: Astra round 2 and census ruling region-ha-2a rev 2 |
| + |
| +Rev 3 answers Astra's round 2 (`docs/superpowers/plans/CODEX-REVIEW-REGION-HA-2A-2.md`: rev 2's findings closed but one, and two new P2s, all accepted) and platform-1e's ruling region-ha-2a rev 2, which approved items 3 and 4 below with conditions. |
| + |
| +| # | Finding or condition | Rev 3 | Where | |
| +|---|---|---|---| |
| +| R1 | The move door's earlier placement checks (a destination with no edges file, or only blank public addresses) still answered 400 with no alert | With region HA on, `roomIn` asks the chooser before those checks: a destination the roster places boxes in, with no box that has room, is H27's refusal (503, RHA-C17, the alert). The instance create door's armed-placement gate gets the same treatment, the same defect. HTTP tests through the whole move door (both shapes, region HA on and off, and with room) and the create door | Tasks 11, 12 | |
| +| R2 | `handleAPIInstanceCreate` bound the environment (`bindEnvironmentCreate`, which can clear a seed's mark on a store) before `beginClaimCreate`'s early choice | The create door chooses before its first write (`chooseCreateFirst`, before the binding) and passes the choice through; `beginClaimCreate` keeps a choice the context carries and does not choose again. Tests: a refused create leaves the store's bytes unchanged and writes nothing, both when every box is stale and when the last box goes between the region check and the choice; with room, the same create takes the store | Task 11 | |
| +| R3 | The capacity metric took each region's AWS region from edges files | `ExecutorChooser.Capacity` also answers the AWS region the signed roster names for the placement, the mapping the chooser reads, and the metric goes there. Tests: a conflicting file (another AWS region, sorting last) does not move the count, unit and end to end | Task 13 | |
| +| C3 | Ruling item 3: `chooseExecutorFirst` takes no lease and makes no write; its executor is passed through and never chosen again; a refusal leaves nothing behind | `TestChooseFirstWritesNothingAndChoosesOnce`: no bucket or backend event while choosing, the choice is the authority's own choice for the same record, a carried choice is kept without calling the build, and `SelectExecutor` returns the carried choice. Every refusal test also checks that nothing was written (`wroteNothing`) | Task 11 | |
| +| C4 | Ruling item 4: the move room check only refuses, never picks or changes the destination | `moveRoomCheck` replaces rev 2's `placeableMoveTargets`: it refuses when no destination candidate is placeable and otherwise leaves the candidates as main's. `TestMoveRoomCheckOnlyRefuses`: with room the selection equals region HA off's; with none, the refusal; a legacy box beside a default one gives main's several-boxes refusal, not a pick | Task 12 | |
| +| C5 | Record-write re-digests stand only if no write site changes; a new write site needs its own line | No write site changes and none is added: the record-write census reports digest changes only (`handleAPIInstanceCreate`, `handleAPIInstanceMoveRequest`, `registerCLIRoutes`, `Directory.Replace`) | Census | |
| + |
| +One new census item: `Server.chooseCreateFirst` (Task 11), a new accepted-legacy unit whose one root read is the claim read `beginClaimCreate` makes, made before the binding. The census tooling's `unitdigest` build line is corrected (rev 2 built it over its own source directory). |
| + |
| ## Scope |
| Spec 2 is four plans. This is **2a**. The others, and the seams 2a leaves them: |
| @@ -36,20 +51,20 @@ Spec 2 is four plans. This is **2a**. The others, and the seams 2a leaves them: |
| | Plan | Builds | 2a's seam | |
| |---|---|---| |
| | 2b | Joining mode, the membership-confirmation worker, the console's enrolment pass | `lbhealth.Default.SetHosting` (Task 9): 2b's mode selection owns it, and joining mode leaves it false. `hostactors.Directory.EnrollAutomatic` already gives an admitted box its group's role in the enrolling write (Task 3) | |
| -| 2c | The `region-host` module's group mode, the start-up script, SSM | The health path `/.carlos/lb-health` on port 80 (Task 9). The metric `Carlos/Region EligibleBoxes`, dimension `Region` = the placement code, in each region's own CloudWatch (Task 13): 2c grants the platform box's role `cloudwatch:PutMetricData` conditioned on `cloudwatch:namespace = Carlos/Region`, and sets `CARLOS_REGION_METRICS=cloudwatch` on the console. The box role's IMDS access must cover `network/interfaces/macs/*` (Task 7), and the security group opens 443 from the region's VPC CIDR | |
| -| 2d | The §8 single-box fixes | The move door's destination already takes §7's placeable set with region HA on (Task 12); 2d's rework of the door (its multi-box regions, the source from the claim) builds on it | |
| +| 2c | The `region-host` module's group mode, the start-up script, SSM | The health path `/.carlos/lb-health` on port 80 (Task 9). The metric `Carlos/Region EligibleBoxes`, dimension `Region` = the placement code, in each region's own CloudWatch, the AWS region the signed roster names for it (Task 13): 2c grants the platform box's role `cloudwatch:PutMetricData` conditioned on `cloudwatch:namespace = Carlos/Region`, and sets `CARLOS_REGION_METRICS=cloudwatch` on the console. The box role's IMDS access must cover `network/interfaces/macs/*` (Task 7), and the security group opens 443 from the region's VPC CIDR | |
| +| 2d | The §8 single-box fixes | With region HA on the move door already refuses a destination with no placeable box (Task 12), without narrowing its candidates; 2d's rework of the door (its multi-box regions, the source from the claim) builds on it | |
| **Not in 2a:** forwarding of platform routes from a group box to the platform box (§6 "Platform routes"), the region ingress list in the console (§6), and every row of §8; those are 2d's or 2c's. |
| ## Global Constraints |
| -- Everything new is dormant with `CARLOS_REGION_HA` off: with it off, every door, file, edge and box behaves exactly as on main. The health answer is not served, the role doors answer 404, the edges file publishes no `private_ip` and keeps its EIP hold, the home map dials public addresses only, the create and move doors choose as on main (no early choice, no placeable filter, no roster read), and the capacity pass does not start. |
| +- Everything new is dormant with `CARLOS_REGION_HA` off: with it off, every door, file, edge and box behaves exactly as on main. The health answer is not served, the role doors answer 404, the edges file publishes no `private_ip` and keeps its EIP hold, the home map dials public addresses only, the create and move doors choose as on main (no early choice, no room check, no roster read), and the capacity pass does not start. |
| - Multi-box regions stay AWS-only until #299 (H26). Spec 2's new core files import no `github.com/aws/...` package; AWS parts are adapters in files named for AWS (§25). `region_ha_boundary_test.go` holds both, plus `s3compat.Options()` on every S3 client in the packages 2a touches. |
| - Wire shapes are additive: `private_ip`, `private_cidr`, `mem_total_mib`, `disk_total_mib` (edges file), `group` and `role` (roster box), `role` (roster region) are all `omitempty`, and a reader that predates them is unaffected. |
| - `private_ip` is routing data only: an edge dials it, and nothing grants anything on it. It comes only from IMDS or root-owned `host.env`, never from a file the agent can write, and it is published and dialled only when `edgesfile.PrivateAddress` passes it. |
| - The roster's default role is stored as `""`. Role values: box `""`, `legacy`, `platform`; group `""`, `legacy`. Roles are read by the placement chooser alone and never feed root's claim authority. |
| - §7's thresholds: free memory at least 15% of total (`boxchoice.MinMemPercent`), free disk at least 10% of the volume (`MinDiskPercent`), an edges file no older than 15 minutes (`MaxEdgesAge`). Choice: spread, then most `mem_free_mib`, then the P5-D2 hash. |
| -- H27: when no box qualifies, every door that chooses a box (instance create, domain attach's first route, standalone sidecar create, the move door's destination) answers 503 with RHA-C17, chooses before it registers anything, and alerts at most once per region an hour per console process. |
| +- H27: when no box qualifies, every door that chooses a box (instance create, domain attach's first route, standalone sidecar create, the move door's destination) answers 503 with RHA-C17, and alerts at most once per region an hour per console process. A door's own earlier placement checks answer a region with no room as H27 too. A create door chooses before its first write (registration, environment binding or any other) and passes that choice through; nothing chooses again. |
| - §6's health answer: `GET` or `HEAD` `/.carlos/lb-health`, any host, port 80. 200 only when hosting, forwarding armed (map and key each loaded once, latched), the certificate mirror loaded once (latched), the edge's and the activator's beats within 60 seconds, and a home map refresh attempt completed within 5 minutes. |
| - User-facing copy is in `docs/superpowers/plans/2026-10-16-region-ha-2a-copy.md`, gated by `approved_region_ha_copy_test.go`. No pending row reaches code, CLI help or docs. No em dashes. |
| - Go commands: `export GOFLAGS=-buildvcs=false GOCACHE=$TMPDIR/gocache`. Run tests with `nice -n 19` and a `-run` filter; never the whole `internal/host` suite (CI's `make test` is the full run). Tests that bind unix sockets fail inside the sandbox ("socket: operation not permitted"); that is the sandbox, not the change. |
| @@ -63,10 +78,11 @@ Approved in principle with conditions (amadan #236). The overlay asks for exactl |
| |---|---|---|---|---| |
| | 7 | `internal/host/edgesfile.go:edgesFileWriteClaims` | authority, F,J1 (kept) | Re-digest, class, task, basis and sites unchanged: with region HA on it reads the address source before every publication | stands | |
| | 10 | `internal/console/apiboxroles.go:Server.boxRoster` | accepted-legacy, J4 root-input | New: the role doors' read of the signed roster (`Directory.Read`), operator-only, read-only | moves | |
| -| 11 | `internal/console/claim_create.go:Server.beginClaimCreate` | accepted-legacy, J4 root-input | Re-freeze: one call (`chooseExecutorFirst`) before `RegisterStart`; its one read site unchanged | moves | |
| -| 12 | `internal/console/apiinstancemove.go:Server.handleAPIInstanceMoveRequest` | accepted-legacy | Re-freeze: one more refusal before any write (no room in the destination); read sites unchanged, offsets moved | moves | |
| +| 11 | `internal/console/claim_create.go:Server.beginClaimCreate` | accepted-legacy, J4 root-input | Re-freeze: one call (`chooseExecutorFirst`) before `RegisterStart`, which keeps a choice the context carries; its one read site unchanged (approved, ruling rev 2 item 3) | moves | |
| +| 11 | `internal/console/nocapacity.go:Server.chooseCreateFirst` | accepted-legacy, J4 root-input | **New in rev 3:** the instance create door's choice before its first write. Its one root read is the host's claim (`hostclaim.Read`), the read `beginClaimCreate` makes, feeding only the placement choice; it takes no lease and writes nothing | moves | |
| +| 12 | `internal/console/apiinstancemove.go:Server.handleAPIInstanceMoveRequest` | accepted-legacy | Re-freeze: two refusals added before any write, both no room in the destination (`roomIn` before the placement checks, `moveRoomCheck` through `selectMoveBoxes`); both only refuse; read sites unchanged, offsets moved (approved, ruling rev 2 item 4; the basis text names both refusals) | moves | |
| -`Server.attachClaimFirstRoute` is not changed: the first route's refusal comes through `beginClaimCreate` and `claimResponseRefusal`, which is not a root-read unit. Record-write census (B8) digests change for `Directory.Replace` (Task 3), `registerCLIRoutes` (Task 10), `handleAPIInstanceCreate` (Task 11) and `handleAPIInstanceMoveRequest` (Task 12). |
| +`Server.attachClaimFirstRoute` is not changed: the first route's refusal comes through `beginClaimCreate` and `claimResponseRefusal`, which is not a root-read unit. Record-write census (B8) digests change for `Directory.Replace` (Task 3), `registerCLIRoutes` (Task 10), `handleAPIInstanceCreate` (Task 11) and `handleAPIInstanceMoveRequest` (Task 12). **No write site changes and none is added** (ruling rev 2): each re-digest is a change of the unit's other lines; the census reports no new or changed site. |
| **Build base (condition 3).** `plan14-root-reads-region-ha-2a.tsv` loads last at build time. The values in this plan were computed at this branch's base, main `70e57449`; the build base will be later (after Plan 14 Parts 2 and 3 land), so recompute every value there with the tooling below: each `@coverage` row, each B8 digest, each unit's kinds and digest, each `@legacy` fingerprint, and each seal. The rows to write stay the same. |
| @@ -94,7 +110,7 @@ for dir in "$@"; do |
| done |
| ```` |
| -`unitdigest/main.go` prints a top-level unit's digest (the sha256 of its declaration's source), the value both censuses pin. Build it with `cd $TMPDIR/census/unitdigest && GOFLAGS= go build -o ../unitdigest main.go`, then run `$TMPDIR/census/unitdigest <file> <unit>` from the repo root: |
| +`unitdigest/main.go` prints a top-level unit's digest (the sha256 of its declaration's source), the value both censuses pin. Build it with `cd $TMPDIR/census/unitdigest && GOFLAGS= go build -o ../unitdigest.bin main.go` (not `-o ../unitdigest`, which is the source directory: Go would write the binary inside it), then run `$TMPDIR/census/unitdigest.bin <file> <unit>` from the repo root: |
| ````go |
| // unitdigest prints the census digest (sha256 of the declaration's source) |
| @@ -267,10 +283,12 @@ nice -n 19 go test ./internal/provision/ -run 'TestPlan9B8RecordWriteCensus|Test |
| 3. **An edges file stamped by a box whose clock runs ahead** is believed for a minute of skew and no more; a stale or unreadable generation is never trusted. Task 4's `TestPlaceableAppliesEachRule` (the future and unreadable cases). |
| 4. **A bucket outage after a box has armed** must leave its health answer at 200 (no termination for a shared outage), while a refresh stuck inside one attempt turns it 503. Task 2's `TestBucketOutageAfterArmingStaysHealthy` and Task 9's `TestHomeMapFeedsTheHealthState`. |
| 5. **A client retrying a refused create in a loop** must not page the operator on every retry, and must leave nothing for recovery each time. Task 11's `TestCreateDoorsRefuseWithNoRoomBeforeRegistering` (three refusals, one page, no registration, no claim after a recovery pass). |
| +6. **A create refused for want of room must change nothing, even when the fleet changes under it.** The choice comes before the environment binding, and the create names the box that was chosen. Task 11's `TestRefusedCreateLeavesTheEnvironmentsStoreAlone` (the last box goes between the region check and the choice) and `TestChooseFirstWritesNothingAndChoosesOnce`. |
| +7. **The move door's room check only refuses.** A regression that narrows the candidates would quietly pick a destination main's selection refuses (a region with several boxes). Task 12's `TestMoveRoomCheckOnlyRefuses`. |
| ## Dry run |
| -Every code block below was written into this branch at `aa5b6b07` (main `70e57449`), task by task in order, built, vetted with `GOOS=linux` and `GOOS=darwin` (and built with `GOOS=windows`), and its tests run as each task's steps say; the census values are those runs'. The copy gate was checked without approving anything: with every row pending, `TestRegionHABuiltCopyIsApplied`, `TestRegionHACopyRecordIsComplete` and `TestRegionHACopyHasNoEmDash` pass, and `TestRegionHABuiltCopyIsApproved` fails naming exactly the built ids (RHA-C1 to RHA-C16 after Task 10, RHA-C1 to RHA-C19 after Task 11). That failure is what an implementer sees until Paul approves the rows. The whole `TestClaimConsoleMutationInvariant` ran once on the final tree. The plan text was then replayed onto a clean export, task by task, and every intermediate state built and passed both censuses. The code was then removed. |
| +Every code block below was written into this branch on main `70e57449` (Tasks 1 to 10 at `aa5b6b07` for rev 2, replayed unchanged for rev 3; Tasks 11 to 13 rewritten at `0a31a455`), task by task in order, built, vetted with `GOOS=linux` and `GOOS=darwin` (and built with `GOOS=windows`), and its tests run as each task's steps say; the census values are those runs'. The copy gate was checked without approving anything: with every row pending, `TestRegionHABuiltCopyIsApplied`, `TestRegionHACopyRecordIsComplete` and `TestRegionHACopyHasNoEmDash` pass, and `TestRegionHABuiltCopyIsApproved` fails naming exactly the built ids (RHA-C1 to RHA-C16 after Task 10, RHA-C1 to RHA-C19 after Task 11). That failure is what an implementer sees until Paul approves the rows. Each rev 3 regression test was first run against rev 2's code and failed (the store test's refused create wrote the store; the move door answered 400; a legacy box beside a default one was picked; the metric went to the edges file's region). The whole `TestClaimConsoleMutationInvariant` ran once on the final tree. The plan text was then replayed onto a clean export, task by task, and every intermediate state built and passed both censuses. The code was then removed. |
| --- |
| @@ -2234,7 +2252,7 @@ git commit -m "region HA 2a: edges files carry total memory and disk" |
| **Interfaces:** |
| - Consumes: `boxchoice.Placeable`, `boxchoice.ChooseRoomiest` (Task 4); `hostactors.Box.Role` (Task 3). |
| -- Produces: `func (x ExecutorChooser) PlaceableBoxes(ctx context.Context, placement string) ([]string, error)` (Tasks 11 and 12 read it) and `func (x ExecutorChooser) Capacity(ctx context.Context, placement string) (int, error)` (Task 13 publishes it); `Choose` and `Permits` now apply the roles and thresholds. `ExecutorChooser` exists only with `CARLOS_REGION_HA` on (the console binds it only then, and the box only in its region HA relocation runtime), so the policy is dormant with it off. |
| +- Produces: `func (x ExecutorChooser) PlaceableBoxes(ctx context.Context, placement string) ([]string, error)` (Tasks 11 and 12 read it) and `func (x ExecutorChooser) Capacity(ctx context.Context, placement string) (int, error)` (Task 13 publishes it, and extends it to answer the roster's AWS region); `Choose` and `Permits` now apply the roles and thresholds. `ExecutorChooser` exists only with `CARLOS_REGION_HA` on (the console binds it only then, and the box only in its region HA relocation runtime), so the policy is dormant with it off. |
| `ExecutorChooser.eligible` is an accepted-legacy unit (merge ruling 2) and does not change: the policy runs over its result. Relocation's `RegionChooser` uses `Choose` and `Permits`, so relocation targets follow the same policy. |
| @@ -4964,31 +4982,32 @@ git commit -m "region HA 2a: box and group role doors and the carlos system verb |
| ### Task 11: The create doors choose first, and refuse when no box has room (H27) |
| **Files:** |
| -- Create: `internal/hostops/chosen.go` (`WithChosenExecutor`, `chosenFor`) |
| +- Create: `internal/hostops/chosen.go` (`WithChosenExecutor`, `ChosenExecutor`) |
| - Modify: `internal/hostops/backfill.go` (`ProductionAuthority.SelectExecutor` honours the door's choice) |
| - Modify: `internal/hostops/executor_test.go` |
| - Create: `internal/console/nocapacity.go`, `internal/console/nocapacity_test.go` |
| - Modify: `internal/console/claim_create.go` (`beginClaimCreate` calls `chooseExecutorFirst` before `RegisterStart`) |
| - Modify: `internal/console/claim_response.go` (`claimResponseRefusal` answers `noCapacityError` with a 503) |
| -- Modify: `internal/console/apiinstances.go` (`handleAPIInstanceCreate`'s app-derived region check uses the chooser with region HA on) |
| +- Modify: `internal/console/apiinstances.go` (`handleAPIInstanceCreate`: the app-derived region check uses the chooser, the armed-placement gate asks `roomIn` first, and `chooseCreateFirst` runs before the environment binding) |
| - Modify: `internal/console/server.go` (`Server.noCapacity`) |
| - Modify: `approved_region_ha_copy_test.go` (wire RHA-C17 to RHA-C19), `region_ha_boundary_test.go`, `internal/provision/plan9_b8_census_test.go`, `internal/provision/plan14_root_reads_test.go` (the seal), the overlay |
| **Interfaces:** |
| - Consumes: Task 6's `ExecutorChooser.PlaceableBoxes` and `Choose` (through `ProductionAuthority.SelectExecutor`); `hostops.ErrNoEligibleBox`; `alerting.Page`; `Server.homeRegionOut`. |
| -- Produces: `func hostops.WithChosenExecutor(ctx, host string, sel hostclaim.ExecutorSelection) context.Context`; `type noCapacityError struct{ Region string }` (`errors.Is(err, hostops.ErrNoEligibleBox)`); `type noCapacityAlerts` (`due(region string, now time.Time) bool`); `func (s *Server) noRoom(ctx, region string) error`; `func (s *Server) placementChooser(pool string) *hostops.ExecutorChooser` (Tasks 12 and 13 use it); `func (s *Server) chooseExecutorFirst(ctx, before hostclaim.Observed, build func(time.Time, string) (hostops.Record, error)) (context.Context, error)`; `var pageNoCapacity = alerting.Page` (the test seam). |
| +- Produces: `func hostops.WithChosenExecutor(ctx, host string, sel hostclaim.ExecutorSelection) context.Context`; `func hostops.ChosenExecutor(ctx, host string) (hostclaim.ExecutorSelection, bool)`; `type noCapacityError struct{ Region string }` (`errors.Is(err, hostops.ErrNoEligibleBox)`); `type noCapacityAlerts` (`due(region string, now time.Time) bool`); `func (s *Server) noRoom(ctx, region string) error`; `func (s *Server) placementChooser(pool string) *hostops.ExecutorChooser` (Tasks 12 and 13 use it); `func (s *Server) roomIn(ctx, region, pool string) error` (Task 12 uses it); `func (s *Server) chooseExecutorFirst(ctx, host string, before hostclaim.Observed, build func(time.Time, string) (hostops.Record, error)) (context.Context, error)`; `func (s *Server) chooseCreateFirst(ctx, host string, build ...) (context.Context, error)`; `var pageNoCapacity = alerting.Page` (the test seam). |
| **Wait for the copy.** RHA-C17 to RHA-C19 are the refusal and the alert. Start once Paul has approved them, and use his text. |
| -H27 (spec rev 15), as Astra's findings 1 to 3 and 6 ask: |
| -- **Choose before registering (finding 1).** Every create through the claim starts in `beginClaimCreate`. With region HA on, `chooseExecutorFirst` builds the record the create would write, asks `SelectExecutor` for its executor, and returns before `RegisterStart` when there is none. Otherwise it puts the choice in the context, and `SelectExecutor` returns that choice to `StartCreate` (claim Plan 5's checks still run on it), so the create names exactly the box the door checked. A refused request leaves no registration, no claim and nothing for a recovery pass. |
| -- **Every choosing door (finding 2).** One error, `noCapacityError`, answered in one place, `claimResponseRefusal`: instance create, domain attach's first route (the API's `handleAPIDomainAttach` and the browser's `handleAppDomainAttach`, both through `attachClaimFirstRoute`, which is unchanged), and standalone sidecar create. The move door's destination is Task 12. A sidecar attached to an instance chooses nothing (it lives on its instance's box) and is unchanged. |
| -- **The earlier region check (finding 3).** `handleAPIInstanceCreate` derives an app's region from `regionCandidates` before the claim. With region HA on, that region is judged by the chooser's eligible set instead: no placeable box is the no-room refusal, not a 400, and a misconfigured box beside an eligible one no longer blocks the create. |
| -- **The alert limit (finding 6).** At most one alert per region an hour, remembered in the console process; a restart may send one more, which H27 accepts. |
| +H27 (spec rev 15), as Astra's findings ask (round 1: 1, 2, 3 and 6; round 2: 1 and 2): |
| +- **Choose before the first write.** Every create through the claim starts in `beginClaimCreate`. With region HA on, `chooseExecutorFirst` builds the record the create would write, asks `SelectExecutor` for its executor, and returns before `RegisterStart` when there is none. Otherwise it puts the choice in the context, and `SelectExecutor` returns that choice to `StartCreate` (claim Plan 5's checks still run on it), so the create names exactly the box the door checked. The instance create door has a write before `beginClaimCreate`: `bindEnvironmentCreate` can clear a seed's mark on the environment's store (`takeSeedStore`). So that door calls `chooseCreateFirst` before the binding; it reads the host's claim and chooses the same way, and `beginClaimCreate` finds the choice in the context and keeps it. The domain attach and standalone sidecar doors write nothing before `beginClaimCreate`, and the seed doors refuse on a claim console (`seedRegionReady`). |
| +- **No lease, no write, chosen once (ruling rev 2, item 3).** The choice is `SelectExecutor`'s read of the roster, the fences and the edges files. It takes no lease and writes nothing; a choice the context carries for the host is never chosen again; a refusal leaves no registration, no claim, no store change and nothing for a recovery pass. |
| +- **Every choosing door.** One error, `noCapacityError`, answered in one place, `claimResponseRefusal`: instance create, domain attach's first route (the API's `handleAPIDomainAttach` and the browser's `handleAppDomainAttach`, both through `attachClaimFirstRoute`, which is unchanged), and standalone sidecar create. The move door's destination is Task 12. A sidecar attached to an instance chooses nothing (it lives on its instance's box) and is unchanged. |
| +- **The earlier region checks.** `handleAPIInstanceCreate` judges an app-derived region (`regionCandidates`) by the chooser's eligible set instead, and asks `roomIn` before the armed-placement gate for an explicit region. `roomIn` only refuses: a region the roster places boxes in, with no placeable box (no edges file, no public address, stale, full or excluded), is the no-room refusal rather than a 400; a region the roster does not know is left to the gate. |
| +- **The alert limit.** At most one alert per region an hour, remembered in the console process; a restart may send one more, which H27 accepts. |
| -With region HA off nothing here runs: `chooseExecutorFirst` returns at once and the create is main's. |
| +With region HA off nothing here runs: `roomIn`, `chooseCreateFirst` and `chooseExecutorFirst` return at once, and every create is main's. |
| -**Census (platform-1e):** `Server.beginClaimCreate` is an accepted-legacy unit (J4 root input). It gains one call before `RegisterStart`; its one read site is unchanged. The overlay re-freezes it with the ruling's tag, and the seal moves. `Server.attachClaimFirstRoute` does not change. The record-write census digest of `handleAPIInstanceCreate` changes. |
| +**Census (platform-1e):** `Server.beginClaimCreate` is an accepted-legacy unit (J4 root input). It gains one call before `RegisterStart`; its one read site is unchanged. The overlay re-freezes it with the ruling's tag. `Server.chooseCreateFirst` is new: its one root read is the host's claim, the read `beginClaimCreate` makes, made before the binding and feeding only the placement choice. The overlay asks for it as accepted-legacy (J4 root input). The seal moves. `Server.attachClaimFirstRoute` does not change. The record-write census digest of `handleAPIInstanceCreate` changes; no write site is added or changed. |
| - [ ] **Step 1: Write the failing tests** |
| @@ -5007,8 +5026,10 @@ import ( |
| "testing" |
| "time" |
| + "github.com/carlosframework/platform/internal/boxchoice" |
| "github.com/carlosframework/platform/internal/hostclaim" |
| "github.com/carlosframework/platform/internal/hostops" |
| + "github.com/carlosframework/platform/internal/objstore" |
| "github.com/carlosframework/platform/internal/provision" |
| "github.com/carlosframework/platform/internal/status" |
| ) |
| @@ -5082,8 +5103,9 @@ func haClaimCase(t *testing.T, home string) *claimCase { |
| return cx |
| } |
| -// publishHABox publishes a roster box's edges file and status snapshot: placeable |
| -// in ie unless mut changes it, and stale when stale is set. |
| +// publishHABox publishes (or republishes) a roster box's edges file and |
| +// status snapshot: placeable in ie unless mut changes it, and stale when |
| +// stale is set. |
| func publishHABox(t *testing.T, cx *claimCase, box string, stale bool, mut func(map[string]any)) { |
| t.Helper() |
| ctx := context.Background() |
| @@ -5101,7 +5123,12 @@ func publishHABox(t *testing.T, cx *claimCase, box string, stale bool, mut func( |
| if err != nil { |
| t.Fatal(err) |
| } |
| - if _, err := cx.rel.Put(ctx, "control/edges/"+box+".json", body, ""); err != nil { |
| + key := "control/edges/" + box + ".json" |
| + _, tag, err := cx.rel.Get(ctx, key) |
| + if err != nil { |
| + tag = "" |
| + } |
| + if _, err := cx.rel.Put(ctx, key, body, tag); err != nil { |
| t.Fatal(err) |
| } |
| at := time.Now() |
| @@ -5145,6 +5172,15 @@ func refusedBeforeRegistering(t *testing.T, cx *claimCase, host string, register |
| } |
| } |
| +// wroteNothing checks that nothing reached the bucket or a backend since |
| +// mark: no lease, no registration, no record and no store write. |
| +func wroteNothing(t *testing.T, cx *claimCase, mark int) { |
| + t.Helper() |
| + if events := cx.rec.since(mark); len(events) != 0 { |
| + t.Fatalf("a refused request wrote %v", events) |
| + } |
| +} |
| + |
| // Region HA spec 2 §7 (H27): every create door that chooses a box refuses |
| // with a 503 when none is eligible, before it registers anything, and the |
| // operator hears of it once per region an hour. Every box here is stale. |
| @@ -5155,12 +5191,14 @@ func TestCreateDoorsRefuseWithNoRoomBeforeRegistering(t *testing.T) { |
| cx.instanceWorld(t) |
| publishHABox(t, cx, "box-ie-tito", true, nil) |
| before := registrations(t, cx, "bab", "titogo") |
| + mark := cx.rec.mark() |
| for range 3 { // a retry loop: refused every time, paged once |
| r := cx.apiReply(t, http.MethodPost, "/api/cli/apps/bab/titogo/instances", `{"host":"acme.go.tito.io","region":"ie"}`) |
| if r.Status != http.StatusServiceUnavailable || !strings.Contains(r.Body, noCapacityRefusal("ie")) { |
| t.Fatalf("create = %d %s", r.Status, r.Body) |
| } |
| } |
| + wroteNothing(t, cx, mark) |
| refusedBeforeRegistering(t, cx, "acme.go.tito.io", before) |
| expectPages(t, pages, noRoomSubject("ie")) |
| }) |
| @@ -5169,10 +5207,12 @@ func TestCreateDoorsRefuseWithNoRoomBeforeRegistering(t *testing.T) { |
| cx := haClaimCase(t, "") |
| cx.sidecarWorld(t) |
| before := registrations(t, cx, "bab", "titogo") |
| + mark := cx.rec.mark() |
| r := cx.apiReply(t, http.MethodPost, "/api/cli/apps/bab/titogo/sidecars", `{"name":"bot","region":"ie"}`) |
| if r.Status != http.StatusServiceUnavailable || !strings.Contains(r.Body, noCapacityRefusal("ie")) { |
| t.Fatalf("sidecar create = %d %s", r.Status, r.Body) |
| } |
| + wroteNothing(t, cx, mark) |
| refusedBeforeRegistering(t, cx, provision.SidecarHost("bot", "titogo", "bab"), before) |
| expectPages(t, pages, noRoomSubject("ie")) |
| }) |
| @@ -5182,10 +5222,12 @@ func TestCreateDoorsRefuseWithNoRoomBeforeRegistering(t *testing.T) { |
| shipVersion(t, cx.consoleFixture, "titogo", "v1") |
| seedChannel(t, cx.consoleFixture, "titogo", "edge", "v1") |
| before := registrations(t, cx, "bab", "titogo") |
| + mark := cx.rec.mark() |
| r := cx.apiReply(t, http.MethodPost, "/api/cli/apps/bab/titogo/domains", `{"host":"first.titogo-claim.example","kind":"static"}`) |
| if r.Status != http.StatusServiceUnavailable || !strings.Contains(r.Body, noCapacityRefusal("ie")) { |
| t.Fatalf("attach = %d %s", r.Status, r.Body) |
| } |
| + wroteNothing(t, cx, mark) |
| refusedBeforeRegistering(t, cx, "first.titogo-claim.example", before) |
| expectPages(t, pages, noRoomSubject("ie")) |
| }) |
| @@ -5195,10 +5237,12 @@ func TestCreateDoorsRefuseWithNoRoomBeforeRegistering(t *testing.T) { |
| shipVersion(t, cx.consoleFixture, "titogo", "v1") |
| seedChannel(t, cx.consoleFixture, "titogo", "edge", "v1") |
| before := registrations(t, cx, "bab", "titogo") |
| + mark := cx.rec.mark() |
| r := cx.formReply(t, "/a/bab/apps/titogo/domains/attach", "host", "first.titogo-claim.example", "kind", "static") |
| if r.Status != http.StatusServiceUnavailable || !strings.Contains(r.Body, noCapacityRefusal("ie")) { |
| t.Fatalf("browser attach = %d %s", r.Status, r.Body) |
| } |
| + wroteNothing(t, cx, mark) |
| refusedBeforeRegistering(t, cx, "first.titogo-claim.example", before) |
| expectPages(t, pages, noRoomSubject("ie")) |
| }) |
| @@ -5264,6 +5308,161 @@ func TestAppRegionWithAMisconfiguredSiblingStillCreates(t *testing.T) { |
| t.Fatalf("executor %q / %q, want box-ie-default", exec, chosen) |
| } |
| } |
| + |
| +// putSeedStore gives titogo's canary environment a store a seed made, and |
| +// returns the store record's bytes. |
| +func putSeedStore(t *testing.T, cx *claimCase) (key string, body []byte) { |
| + t.Helper() |
| + ctx := context.Background() |
| + at := time.Now().UTC() |
| + if _, err := objstore.PutIn(ctx, cx.rel, "bab", "titogo", "canary", &objstore.Record{ |
| + Declared: &objstore.Declared{EnvPrefix: "S3", At: at, By: "bob@keymail.dev", Seed: newSeedID(t)}, |
| + Grant: &objstore.GrantBlock{Backend: "mem", Bucket: "carlos-bab-titogo-canary", Source: objstore.GrantSourceAccountApproval, At: at, By: "operator@keymail.dev"}, |
| + }, ""); err != nil { |
| + t.Fatal(err) |
| + } |
| + key = objstore.KeyIn("bab", "titogo", "canary") |
| + body, _, err := cx.rel.Get(ctx, key) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + return key, body |
| +} |
| + |
| +// Astra 2a round 2 finding 2: the instance create door chooses before its |
| +// first write. A create naming an environment whose store a seed made would |
| +// take that store (bindEnvironmentCreate). Refused for want of room, it |
| +// leaves the store's bytes as they were and writes nothing at all: when |
| +// every box is stale (the region check refuses), and when the region's last |
| +// placeable box goes between the region check and the choice (the early |
| +// choice refuses, before the binding). With room, the same create takes the |
| +// store, so the binding is on the path. |
| +func TestRefusedCreateLeavesTheEnvironmentsStoreAlone(t *testing.T) { |
| + const body = `{"host":"acme.go.tito.io","region":"ie","environment":"canary"}` |
| + refused := func(t *testing.T, cx *claimCase, pages chan string) { |
| + t.Helper() |
| + key, stored := putSeedStore(t, cx) |
| + before := registrations(t, cx, "bab", "titogo") |
| + mark := cx.rec.mark() |
| + r := cx.apiReply(t, http.MethodPost, "/api/cli/apps/bab/titogo/instances", body) |
| + if r.Status != http.StatusServiceUnavailable || !strings.Contains(r.Body, noCapacityRefusal("ie")) { |
| + t.Fatalf("create = %d %s", r.Status, r.Body) |
| + } |
| + wroteNothing(t, cx, mark) |
| + if now, _, err := cx.rel.Get(context.Background(), key); err != nil || string(now) != string(stored) { |
| + t.Fatalf("the store after the refusal:\n%s\nwant\n%s (%v)", now, stored, err) |
| + } |
| + refusedBeforeRegistering(t, cx, "acme.go.tito.io", before) |
| + expectPages(t, pages, noRoomSubject("ie")) |
| + } |
| + t.Run("every box stale", func(t *testing.T) { |
| + pages := pagesTo(t) |
| + cx := haClaimCase(t, "") |
| + cx.instanceWorld(t) |
| + publishHABox(t, cx, "box-ie-tito", true, nil) |
| + refused(t, cx, pages) |
| + |
| + publishHABox(t, cx, "box-ie-tito", false, nil) |
| + r := cx.apiReply(t, http.MethodPost, "/api/cli/apps/bab/titogo/instances", body) |
| + if r.Status != http.StatusCreated && r.Status != http.StatusAccepted { |
| + t.Fatalf("create with room = %d %s", r.Status, r.Body) |
| + } |
| + store, _, err := objstore.GetIn(context.Background(), cx.rel, "bab", "titogo", "canary") |
| + if err != nil || store == nil || store.Declared == nil || store.Declared.Seed != "" { |
| + t.Fatalf("the store after a create with room: %+v %v, want the seed's mark cleared", store, err) |
| + } |
| + }) |
| + t.Run("the last box goes before the choice", func(t *testing.T) { |
| + pages := pagesTo(t) |
| + cx := haClaimCase(t, "") |
| + cx.instanceWorld(t) |
| + publishHABox(t, cx, "box-ie-tito", false, nil) |
| + x := cx.srv.cfg.ClaimAuthority.Executors |
| + edges, reads := x.Edges, 0 |
| + x.Edges = func(ctx context.Context) (boxchoice.Snapshot, error) { |
| + if reads++; reads > 1 { |
| + return boxchoice.Snapshot{Complete: true}, nil |
| + } |
| + return edges(ctx) |
| + } |
| + refused(t, cx, pages) |
| + }) |
| +} |
| + |
| +// Astra 2a round 2 finding 1, at the create door: with region HA on, a |
| +// region whose boxes publish no public address (or no edges file) is a |
| +// region with no room, not the armed-placement gate's 400. With region HA |
| +// off the gate answers, as on main. |
| +func TestCreateRegionWithNoReachableBoxIsNoRoom(t *testing.T) { |
| + const body = `{"host":"acme.go.tito.io","region":"ie"}` |
| + pages := pagesTo(t) |
| + cx := haClaimCase(t, "") |
| + cx.instanceWorld(t) |
| + publishHABox(t, cx, "box-ie-tito", false, func(m map[string]any) { m["eip"] = "" }) |
| + cx.srv.cfg.RegionHA = false |
| + if r := cx.apiReply(t, http.MethodPost, "/api/cli/apps/bab/titogo/instances", body); r.Status != http.StatusBadRequest { |
| + t.Fatalf("region HA off: create = %d %s, want main's 400", r.Status, r.Body) |
| + } |
| + cx.srv.cfg.RegionHA = true |
| + before := registrations(t, cx, "bab", "titogo") |
| + mark := cx.rec.mark() |
| + r := cx.apiReply(t, http.MethodPost, "/api/cli/apps/bab/titogo/instances", body) |
| + if r.Status != http.StatusServiceUnavailable || !strings.Contains(r.Body, noCapacityRefusal("ie")) { |
| + t.Fatalf("create = %d %s", r.Status, r.Body) |
| + } |
| + wroteNothing(t, cx, mark) |
| + refusedBeforeRegistering(t, cx, "acme.go.tito.io", before) |
| + expectPages(t, pages, noRoomSubject("ie")) |
| +} |
| + |
| +// Census ruling region-ha-2a rev 2, item 3: the early choice takes no lease |
| +// and writes nothing, and its executor is the one the authority chooses on |
| +// its own (the create's choice before the early choice existed). A choice |
| +// the context already carries is kept: nothing is chosen again. |
| +func TestChooseFirstWritesNothingAndChoosesOnce(t *testing.T) { |
| + cx := haClaimCase(t, "") |
| + appDerivedWorld(t, cx) |
| + publishHABox(t, cx, "box-ie-default", false, nil) |
| + ctx := context.Background() |
| + host := "acme.go.tito.io" |
| + build := func(now time.Time, acquisition string) (hostops.Record, error) { |
| + rec := provision.Record{Schema: 1, Host: host, App: "titogo", Account: "bab", Channel: "stable", Kind: "static", CreatedBy: "bob@keymail.dev", Acquisition: acquisition} |
| + return hostops.PrepareInstance(rec, now, cx.srv.cfg.claimPlacementRegion("ie")) |
| + } |
| + probe, err := build(time.Now().UTC(), "") |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + before, err := hostclaim.Read(ctx, cx.rel, host) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + want, chosen, err := cx.srv.cfg.ClaimAuthority.SelectExecutor(ctx, before, probe) |
| + if err != nil || !chosen || want.Box != "box-ie-default" { |
| + t.Fatalf("the authority's own choice: %+v %v %v", want, chosen, err) |
| + } |
| + |
| + mark := cx.rec.mark() |
| + got, err := cx.srv.chooseCreateFirst(ctx, host, build) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + wroteNothing(t, cx, mark) |
| + if sel, ok := hostops.ChosenExecutor(got, host); !ok || sel != want { |
| + t.Fatalf("early choice %+v (%v), want the authority's %+v", sel, ok, want) |
| + } |
| + |
| + again, err := cx.srv.chooseExecutorFirst(got, host, before, func(time.Time, string) (hostops.Record, error) { |
| + t.Fatal("chose again") |
| + return hostops.Record{}, nil |
| + }) |
| + if err != nil || again != got { |
| + t.Fatalf("a carried choice was not kept: %v", err) |
| + } |
| + if sel, chosen, err := cx.srv.cfg.ClaimAuthority.SelectExecutor(got, before, probe); err != nil || !chosen || sel != want { |
| + t.Fatalf("the create's selection %+v %v %v, want the door's %+v", sel, chosen, err, want) |
| + } |
| +} |
| ```` |
| ````diff |
| @@ -5305,7 +5504,7 @@ func TestAppRegionWithAMisconfiguredSiblingStillCreates(t *testing.T) { |
| - [ ] **Step 2: Run them to see them fail** |
| -Run: `nice -n 19 go test ./internal/console/ -run 'NoCapacity|ClaimRefusalAnswersNoRoom|CreateDoorsRefuse|AppRegionWith' -count=1` |
| +Run: `nice -n 19 go test ./internal/console/ -run 'NoCapacity|ClaimRefusalAnswersNoRoom|CreateDoorsRefuse|AppRegionWith|RefusedCreateLeaves|ChooseFirstWrites|CreateRegionWithNoReachable' -count=1` |
| Expected: FAIL to build (`undefined: pageNoCapacity`, `undefined: noCapacityError`). The hostops test fails to build on `undefined: WithChosenExecutor`. |
| - [ ] **Step 3: Write the early choice and the refusal** |
| @@ -5341,8 +5540,9 @@ func WithChosenExecutor(ctx context.Context, host string, sel hostclaim.Executor |
| return context.WithValue(ctx, chosenExecutorKey{}, chosenExecutor{host: host, sel: sel}) |
| } |
| -// chosenFor is the executor ctx carries for host, if any. |
| -func chosenFor(ctx context.Context, host string) (hostclaim.ExecutorSelection, bool) { |
| +// ChosenExecutor is the executor ctx carries for host, if any: a door that |
| +// already chose finds its choice here and does not choose again. |
| +func ChosenExecutor(ctx context.Context, host string) (hostclaim.ExecutorSelection, bool) { |
| c, ok := ctx.Value(chosenExecutorKey{}).(chosenExecutor) |
| if !ok || c.host == "" || c.host != host || c.sel.Box == "" || c.sel.Enrollment == "" { |
| return hostclaim.ExecutorSelection{}, false |
| @@ -5365,7 +5565,7 @@ func chosenFor(ctx context.Context, host string) (hostclaim.ExecutorSelection, b |
| if a.Executors == nil { |
| return hostclaim.ExecutorSelection{}, false, nil |
| + } |
| -+ if sel, ok := chosenFor(ctx, r.Holder.Host); ok { |
| ++ if sel, ok := ChosenExecutor(ctx, r.Holder.Host); ok { |
| + return sel, true, nil |
| } |
| var shape struct { |
| @@ -5480,16 +5680,47 @@ func (s *Server) placementChooser(pool string) *hostops.ExecutorChooser { |
| return &x |
| } |
| +// roomIn is H27's question about region (record vocabulary) for pool, asked |
| +// with region HA on before a door's own placement checks (Astra 2a round 2 |
| +// finding 1): a region the roster places boxes in, whose placeable set is |
| +// empty (no edges file, no public address, stale, full or excluded by |
| +// role), is the no-room refusal, never a 400. It only refuses: it chooses |
| +// nothing and changes nothing the door goes on to do. It answers nil with |
| +// region HA off, for a region the roster does not place (the door's own |
| +// checks answer that), and when a box has room. |
| +func (s *Server) roomIn(ctx context.Context, region, pool string) error { |
| + x := s.placementChooser(pool) |
| + if x == nil { |
| + return nil |
| + } |
| + boxes, err := x.PlaceableBoxes(ctx, s.cfg.claimPlacementRegion(region)) |
| + switch { |
| + case errors.Is(err, hostops.ErrContract): |
| + return nil |
| + case err != nil: |
| + return err |
| + case len(boxes) == 0: |
| + return s.noRoom(ctx, region) |
| + } |
| + return nil |
| +} |
| + |
| // chooseExecutorFirst is beginClaimCreate's first step with region HA on: |
| // it chooses the executor for the record build would make, before anything |
| // is registered, and carries the choice to the create |
| -// (hostops.WithChosenExecutor). No eligible box is noRoom's refusal. With |
| -// region HA off it returns ctx unchanged, and the create is main's. |
| -func (s *Server) chooseExecutorFirst(ctx context.Context, before hostclaim.Observed, build func(time.Time, string) (hostops.Record, error)) (context.Context, error) { |
| +// (hostops.WithChosenExecutor). It takes no lease and writes nothing: the |
| +// choice is SelectExecutor's read of the roster and the edges files. No |
| +// eligible box is noRoom's refusal. A choice ctx already carries for host |
| +// (chooseCreateFirst's) is kept, and nothing is chosen again. With region |
| +// HA off it returns ctx unchanged, and the create is main's. |
| +func (s *Server) chooseExecutorFirst(ctx context.Context, host string, before hostclaim.Observed, build func(time.Time, string) (hostops.Record, error)) (context.Context, error) { |
| a := s.cfg.ClaimAuthority |
| if !s.cfg.RegionHA || a == nil || a.Executors == nil { |
| return ctx, nil |
| } |
| + if _, ok := hostops.ChosenExecutor(ctx, host); ok { |
| + return ctx, nil |
| + } |
| probe, err := build(time.Now().UTC(), "") |
| if err != nil { |
| return ctx, err |
| @@ -5501,7 +5732,26 @@ func (s *Server) chooseExecutorFirst(ctx context.Context, before hostclaim.Obser |
| if err != nil || !chosen { |
| return ctx, err |
| } |
| - return hostops.WithChosenExecutor(ctx, probe.Holder.Host, sel), nil |
| + return hostops.WithChosenExecutor(ctx, host, sel), nil |
| +} |
| + |
| +// chooseCreateFirst is the instance create door's choice, made before the |
| +// door's first write (Astra 2a round 2 finding 2): the environment binding |
| +// (bindEnvironmentCreate) can move a seed's store to the new instance, so |
| +// a create refused for want of room must be refused before it. It reads |
| +// host's claim and chooses as chooseExecutorFirst does; beginClaimCreate |
| +// then finds the choice in ctx and keeps it. With region HA off it |
| +// returns ctx unchanged and reads nothing. |
| +func (s *Server) chooseCreateFirst(ctx context.Context, host string, build func(time.Time, string) (hostops.Record, error)) (context.Context, error) { |
| + a := s.cfg.ClaimAuthority |
| + if !s.cfg.RegionHA || a == nil || a.Executors == nil { |
| + return ctx, nil |
| + } |
| + before, err := hostclaim.Read(ctx, s.cfg.Releases, host) |
| + if err != nil { |
| + return ctx, err |
| + } |
| + return s.chooseExecutorFirst(ctx, host, before, build) |
| } |
| ```` |
| @@ -5531,7 +5781,7 @@ func (s *Server) chooseExecutorFirst(ctx context.Context, before hostclaim.Obser |
| + return nil, err |
| + } |
| + // Region HA (spec 2 §7, H27): choose before registering anything. |
| -+ if ctx, err = s.chooseExecutorFirst(ctx, before, build); err != nil { |
| ++ if ctx, err = s.chooseExecutorFirst(ctx, host, before, build); err != nil { |
| return nil, err |
| } |
| appID, err := s.store.ClaimAppID(acct.ID, app) |
| @@ -5583,12 +5833,73 @@ func (s *Server) chooseExecutorFirst(ctx context.Context, before hostclaim.Obser |
| // Each list names only what an explicit request for this same |
| // instance would pass (codex plan-R4 2): instanceChoices. |
| if reason != "" { |
| +@@ -1331,6 +1350,12 @@ |
| + // provision.Record's own (DerivesPaths), asked of a probe record rather |
| + // than restated here, so it cannot drift from what reconcile will mint. |
| + if placed == "" && region != "" { |
| ++ // Region HA (spec 2 §7, H27): a region with no box that has room is |
| ++ // the no-room refusal, before this gate answers it as a 400. |
| ++ if err := s.roomIn(ctx, region, m.Pool); err != nil { |
| ++ writeClaimResponseError(w, err) |
| ++ return |
| ++ } |
| + probe := provision.Record{Kind: strings.TrimSpace(req.Kind), Addrs: req.Addrs, Role: probeRec.Role} |
| + caps, published := placements(ctx, s.cfg.Releases) |
| + // The replica placement rides the same gate as the record's own |
| +@@ -1568,25 +1593,36 @@ |
| + cancel() |
| + ctx, cancel = context.WithTimeout(r.Context(), instanceStoreTimeout) |
| + defer cancel() |
| ++ buildClaim := func(now time.Time, acquisition string) (hostops.Record, error) { |
| ++ rec.CreatedAt = time.Time{} |
| ++ rec.Acquisition = acquisition |
| ++ region := s.cfg.claimPlacementRegion(rec.Region) |
| ++ prepared, err := hostops.PrepareInstance(rec, now, region) |
| ++ if err == nil { |
| ++ rec.CreatedAt = now |
| ++ } |
| ++ return prepared, err |
| ++ } |
| ++ // Region HA (spec 2 §7, H27): the box is chosen before the door's first |
| ++ // write, the environment binding below, so a create refused for want of |
| ++ // room changes nothing. beginClaimCreate keeps this choice. |
| ++ createContext := r.Context() |
| ++ if s.cfg.HostClaim { |
| ++ var chooseErr error |
| ++ if createContext, chooseErr = s.chooseCreateFirst(createContext, host, buildClaim); chooseErr != nil { |
| ++ writeClaimResponseError(w, chooseErr) |
| ++ return |
| ++ } |
| ++ } |
| + // An environment named without a store walk of its own takes the store |
| + // from a seed that made it (final review F3, seedenvbind.go). |
| + if !m.PrivateStorage && !s.bindEnvironmentCreate(ctx, w, rec) { |
| + return |
| + } |
| + var claimRequest *claimCreateRequest |
| +- createContext := r.Context() |
| + if s.cfg.HostClaim { |
| + var createErr error |
| +- claimRequest, createErr = s.beginClaimCreate(r.Context(), acct, app, host, principal, func(now time.Time, acquisition string) (hostops.Record, error) { |
| +- rec.CreatedAt = time.Time{} |
| +- rec.Acquisition = acquisition |
| +- region := s.cfg.claimPlacementRegion(rec.Region) |
| +- prepared, err := hostops.PrepareInstance(rec, now, region) |
| +- if err == nil { |
| +- rec.CreatedAt = now |
| +- } |
| +- return prepared, err |
| +- }) |
| ++ claimRequest, createErr = s.beginClaimCreate(createContext, acct, app, host, principal, buildClaim) |
| + if createErr != nil { |
| + writeClaimResponseError(w, createErr) |
| + return |
| ```` |
| - [ ] **Step 4: Run the tests** |
| ```bash |
| -nice -n 19 go test ./internal/console/ -run 'NoCapacity|ClaimRefusalAnswersNoRoom|CreateDoorsRefuse|AppRegionWith|ClaimCreate|InstanceCreate|DomainAttach|Sidecar' -count=1 |
| +nice -n 19 go test ./internal/console/ -run 'NoCapacity|ClaimRefusalAnswersNoRoom|CreateDoorsRefuse|AppRegionWith|RefusedCreateLeaves|ChooseFirstWrites|CreateRegionWithNoReachable|ClaimCreate|InstanceCreate|DomainAttach|Sidecar|EnvironmentCreate|EnvironmentBinding' -count=1 |
| nice -n 19 go test ./internal/hostops/ -run 'Executor|Chooser|SelectExecutor|StartCreate|ProductionCreate' -count=1 |
| ``` |
| Expected: PASS. |
| @@ -5626,7 +5937,7 @@ Expected: PASS once RHA-C17 to RHA-C19 are approved. |
| // regionHAAdapterFiles are spec 2's AWS adapters. Each is named for AWS so |
| ```` |
| -`handleAPIInstanceCreate` is a record writer in the record-write census (`key:CreateRecord`). Its digest changes: |
| +`handleAPIInstanceCreate` is a record writer in the record-write census (`key:CreateRecord`). Its digest changes; its write site does not: |
| ````diff |
| --- a/internal/provision/plan9_b8_census_test.go |
| @@ -5636,25 +5947,29 @@ Expected: PASS once RHA-C17 to RHA-C19 are approved. |
| // Re-reviewed 2026-10-12 (merge of main's home-region work): main's body (claimPlacementRegion) with stage B's one-line EvidenceComplete call; no new write. |
| // Re-reviewed at the merge of carlos-run into seeded-instances (2026-10-13): main's door with the two seed hand-offs: once every ordinary check has passed, a create that names `from` or `import` leaves for createFrom or createImport, which write their own record (beginFrom, beginImport, below) and refuse a claim console first (seedRegionReady); the ordinary create asks bindEnvironmentCreate before its write. The door's own CreateRecord and its claim create are as main wrote them. |
| - {"internal/console/apiinstances.go", "Server.handleAPIInstanceCreate", "record-writer", "key:CreateRecord", "6763eab45774b9e624a9b820ce1585ef3910355bc3ce589b82298e1d12469a2e"}, |
| -+ {"internal/console/apiinstances.go", "Server.handleAPIInstanceCreate", "record-writer", "key:CreateRecord", "385e85091d7ee5c89e3218bc55cd15dd4cac753df71b6ebbad5a67c15a3c14de"}, |
| ++ {"internal/console/apiinstances.go", "Server.handleAPIInstanceCreate", "record-writer", "key:CreateRecord", "55b802e0cdbae1a970cc8c8a8d1f79b137f82b60cd2ce76af8a2f408f10380d5"}, |
| // Re-reviewed at the merge of carlos-run into seeded-instances (2026-10-13): main's door under the environment-binding lock (lockEnvironmentUpdate); the UpdateRecord call is unchanged, and UpdateRecord refuses any change to the record's seed. |
| {"internal/console/apiinstances.go", "Server.handleAPIInstanceUpdate", "record-writer", "key:UpdateRecord", "43a8bead675a4aa33777c6c68431882d55d79a9c1d27e1a54c8f103dda087de0"}, |
| {"internal/console/apiscan.go", "Server.writeScanRecord", "nonrecord", "sel:Put", "c3c092f4b7be9bf6bf09d0fe1e2c9fe52892ee97e57f6c7ab964b85fe36fb5c6"}, |
| ```` |
| +The `Server.chooseCreateFirst` rows are census ruling region-ha-2a's (platform-1e). If the ruling's own basis text differs from the row's, use it, then recompute the fingerprint with `legacyrow.py` and the seal. |
| + |
| Add this comment line after the overlay's last comment line: |
| ````text |
| -# Task 11: H27 on the create doors (internal/console nocapacity.go; hostops chosen.go): beginClaimCreate chooses the executor before it registers anything (chooseExecutorFirst) and carries the choice to the create (hostops.WithChosenExecutor, honoured by ProductionAuthority.SelectExecutor), claimResponseRefusal answers no room with a 503 on instance create, domain attach's first route (attachClaimFirstRoute, unchanged) and standalone sidecar create, and handleAPIInstanceCreate judges an app-derived region by the chooser's eligible set. Census ruling region-ha-2a is asked to re-freeze one accepted-legacy unit, Server.beginClaimCreate: one call added before RegisterStart, its one root-read site (the claim read, file:Read@470) unchanged. The seal moves. |
| +# Task 11: H27 on the create doors (internal/console nocapacity.go; hostops chosen.go): beginClaimCreate chooses the executor before it registers anything (chooseExecutorFirst) and carries the choice to the create (hostops.WithChosenExecutor, honoured by ProductionAuthority.SelectExecutor), claimResponseRefusal answers no room with a 503 on instance create, domain attach's first route (attachClaimFirstRoute, unchanged) and standalone sidecar create, and handleAPIInstanceCreate judges an app-derived region by the chooser's eligible set. handleAPIInstanceCreate chooses before its first write, the environment binding (chooseCreateFirst), and beginClaimCreate keeps that choice; with region HA on, an explicit region with no box that has room is the no-room refusal before the armed-placement gate (roomIn). Census ruling region-ha-2a covers two accepted-legacy units: Server.beginClaimCreate re-frozen (one call added before RegisterStart, its one root-read site, the claim read file:Read@470, unchanged), and Server.chooseCreateFirst, new: the same claim read, made before the binding, feeding only the placement choice. The seal moves. |
| ```` |
| Then set these rows. A `@coverage` row replaces the overlay's row for the same directory, or is appended when there is none; every other row is appended: |
| ````text |
| -@coverage internal/hostops 7c8e88d80099105a0cf3da88141ec6b054bbaaac756a6d441ec035bcd89db1ab |
| -@coverage internal/console 6e39bd88c50d06c7c1a67098d2aa0ca00efe03e35c3aaf082d4e57df965bede6 |
| -@legacy 1d712288cb56c8e2391ca40dc4c9bd2cfb02d1cd8455c64726807464106c1439 internal/console/claim_create.go Server.beginClaimCreate legacy accepted-legacy accepted-legacy;rev3=J4;basis=root-input;census-ruling-region-ha-2a file:Read@470 97ac0e0ead81eb2c72de7534a19f31724ecae42f074db6388d90c05a5f2747f7 |
| -internal/console/claim_create.go Server.beginClaimCreate legacy accepted-legacy accepted-legacy;rev3=J4;basis=root-input;census-ruling-region-ha-2a file:Read@470 97ac0e0ead81eb2c72de7534a19f31724ecae42f074db6388d90c05a5f2747f7 |
| +@coverage internal/hostops 6e494cc9e53bf4fa4433fad798ecd3493c972f65083b0b2153b253f0bd50dcba |
| +@coverage internal/console 2cbd830ba2753a8a6e844ce413220e89b2c85fecd0c0f891d4afd44c68cfeb7f |
| +@legacy d3df5cf0504d03e83d6b4d9ac0d5f7f96ce83f2be9edde6fa45f6b53aac0a28d internal/console/claim_create.go Server.beginClaimCreate legacy accepted-legacy accepted-legacy;rev3=J4;basis=root-input;census-ruling-region-ha-2a file:Read@470 c1f1d4ad231ad74c975cd99178f8c0856b38690e0b25c9ea38a1888324953499 |
| +internal/console/claim_create.go Server.beginClaimCreate legacy accepted-legacy accepted-legacy;rev3=J4;basis=root-input;census-ruling-region-ha-2a file:Read@470 c1f1d4ad231ad74c975cd99178f8c0856b38690e0b25c9ea38a1888324953499 |
| +@legacy 6e1630474813d7844c18d06016a19d55081a63e5b60223fa2641b9c198439bb8 internal/console/nocapacity.go Server.chooseCreateFirst legacy accepted-legacy accepted-legacy;rev3=J4;basis=root-input;census-ruling-region-ha-2a file:Read@271 a867392e56072cf16b712a56d7ea613c2a36e0380c304aee33f6ab8b4939d135 |
| +internal/console/nocapacity.go Server.chooseCreateFirst legacy accepted-legacy accepted-legacy;rev3=J4;basis=root-input;census-ruling-region-ha-2a file:Read@271 a867392e56072cf16b712a56d7ea613c2a36e0380c304aee33f6ab8b4939d135 |
| ```` |
| ````diff |
| @@ -5665,7 +5980,7 @@ internal/console/claim_create.go Server.beginClaimCreate legacy accepted-legacy |
| t.Fatal(err) |
| } |
| - seal = "a57250c350c20e4116e5c87b559cfda71f3e0c65d59a2858d83c5f78bea4e9b5" |
| -+ seal = "2a0c03ae8d2c0ba3712151a98e1533e6be8512271dc98968bc5435b014bd6bab" |
| ++ seal = "5013c9ebddabb1d84a57cc2199991fd0f75b97bc5a955c13d80ccf48efb1c235" |
| errs := append(compareRootReads(got, inv), checkRootReadLegacy(inv, seal)...) |
| errs = append(errs, checkRootReadDeclarations(got, inv)...) |
| if len(errs) > 0 { |
| @@ -5682,23 +5997,25 @@ git commit -m "region HA 2a: create doors choose first and refuse when no box ha |
| --- |
| -### Task 12: The move door's destination is the placeable set |
| +### Task 12: The move door refuses a destination with no room |
| **Files:** |
| -- Modify: `internal/console/boxselect.go` (`placeableMoveTargets`; `selectMoveBoxes` filters the destination) |
| -- Modify: `internal/console/apiinstancemove.go` (`handleAPIInstanceMoveRequest` answers the no-room refusal) |
| +- Modify: `internal/console/apiinstancemove.go` (`handleAPIInstanceMoveRequest` asks `roomIn` before its placement checks, and answers the no-room refusal from `selectMoveBoxes`) |
| +- Modify: `internal/console/boxselect.go` (`moveRoomCheck`; `selectMoveBoxes` asks it of the destination) |
| - Create: `internal/console/boxselect_ha_test.go` |
| - Modify: `internal/provision/plan9_b8_census_test.go`, `internal/provision/plan14_root_reads_test.go` (the seal), the overlay |
| **Interfaces:** |
| -- Consumes: Task 11's `placementChooser`, `noRoom`, `noCapacityError`, `writeClaimResponseError`; Task 6's `PlaceableBoxes`. |
| -- Produces: `func (s *Server) placeableMoveTargets(ctx, to string, cands []boxCandidate) ([]boxCandidate, error)`. |
| +- Consumes: Task 11's `placementChooser`, `roomIn`, `noRoom`, `noCapacityError`, `writeClaimResponseError`; Task 6's `PlaceableBoxes`. |
| +- Produces: `func (s *Server) moveRoomCheck(ctx, to string, cands []boxCandidate) error`. |
| -The move door chooses a box in the destination, so it is H27's fourth door. With region HA on, the destination's candidates are `eligibleBoxes`' (main's identity and floor checks) narrowed to the chooser's placeable set (roles, headroom, fresh edges files); an empty result is the no-room refusal, answered and alerted as Task 11's. The source is never filtered, so an instance on an excluded or full box can still move out (A10-3). |
| +The move door chooses a box in the destination, so it is H27's fourth door. With region HA on, two checks ask the chooser, and both only refuse (ruling rev 2, item 4): |
| +- **Before the placement checks** (Astra round 2 finding 1), `roomIn(ctx, to, "")`: a destination the roster places boxes in, with no placeable box (no edges file, only blank public addresses, stale, full or excluded by role), is the no-room refusal, alert included, instead of the placement checks' 400. |
| +- **In `selectMoveBoxes`**, `moveRoomCheck`: when none of the destination's candidates (main's `eligibleBoxes`) is placeable, the no-room refusal. Otherwise the candidates stay exactly main's, so the destination is the box main's selection picks, and a region with several candidates still gets main's several-boxes refusal: the check never narrows the set to pick one. |
| -With region HA off (Astra finding 4), `placementChooser` is nil, so the destination is main's and no roster is read: host claims on and region HA off behave exactly as main, an unreadable roster included. |
| +The source is never checked, so an instance on an excluded or full box can still move out (A10-3). With region HA off (Astra round 1 finding 4), `placementChooser` is nil, so both checks answer nil and no roster is read: host claims on and region HA off behave exactly as main, an unreadable roster included. |
| -**Census (platform-1e):** `Server.handleAPIInstanceMoveRequest` is an accepted-legacy unit. It gains one refusal before any write; its read sites are unchanged and their offsets move. The overlay re-freezes it with the ruling's basis, and the seal moves. Its record-write census digest changes too. |
| +**Census (platform-1e):** `Server.handleAPIInstanceMoveRequest` is an accepted-legacy unit. It gains two refusals before any write; its read sites are unchanged and their offsets move. The overlay re-freezes it, its basis naming both refusals, and the seal moves. Its record-write census digest changes; its write sites do not. |
| - [ ] **Step 1: Write the failing tests** |
| @@ -5711,22 +6028,23 @@ import ( |
| "context" |
| "encoding/json" |
| "errors" |
| + "net/http" |
| + "strings" |
| "testing" |
| "time" |
| "github.com/carlosframework/platform/internal/hostactors" |
| + "github.com/carlosframework/platform/internal/hostclaim" |
| "github.com/carlosframework/platform/internal/hostops" |
| + "github.com/carlosframework/platform/internal/provision" |
| ) |
| -// haMoveFleet is armMoveFleet's ie and uk boxes, each with fresh, roomy |
| -// headroom, a signed roster naming both regions, and, with ha, a region |
| -// HA chooser bound as the create door binds it. ukRole is uk-box's role. |
| -func haMoveFleet(t *testing.T, ha bool, ukRole string) *consoleFixture { |
| +// roomyEdges gives each box's edges file fresh, roomy headroom, then edit |
| +// (when set) changes uk-box's. |
| +func roomyEdges(t *testing.T, fx *consoleFixture, edit func(map[string]any), boxes ...string) { |
| t.Helper() |
| ctx := context.Background() |
| - fx := newConsoleFixture(t) |
| - armMoveFleet(t, fx, ieBox("ie-box"), ukBox("uk-box")) |
| - for _, box := range []string{"ie-box", "uk-box"} { |
| + for _, box := range boxes { |
| key := edgesFilePrefix + box + ".json" |
| raw, etag, err := fx.rel.Get(ctx, key) |
| if err != nil { |
| @@ -5738,64 +6056,108 @@ func haMoveFleet(t *testing.T, ha bool, ukRole string) *consoleFixture { |
| } |
| m["headroom"] = map[string]any{"mem_free_mib": 4000, "mem_total_mib": 8000, "disk_free_mib": 50000, "disk_total_mib": 100000, "awake": 0} |
| m["generation"] = time.Now().UTC().Format(time.RFC3339) |
| + if box == "uk-box" && edit != nil { |
| + edit(m) |
| + } |
| body, _ := json.Marshal(m) |
| if _, err := fx.rel.Put(ctx, key, body, etag); err != nil { |
| t.Fatal(err) |
| } |
| } |
| - ie, uk := moveRosterBox(ieBox("ie-box")), moveRosterBox(ukBox("uk-box")) |
| - uk.Role = ukRole |
| - putMoveRoster(t, fx, hostactors.Roster{Boxes: map[string]hostactors.Box{"ie-box": ie, "uk-box": uk}, |
| - Regions: map[string]hostactors.Region{ |
| - "ie": {Account: moveTestAccount, AWSRegion: "eu-west-1"}, |
| - "uk": {Account: moveTestAccount, AWSRegion: "eu-west-2"}, |
| - }}) |
| +} |
| + |
| +// haRoster signs a roster naming both regions, every box of fleet, and |
| +// each box's role from roles. |
| +func haRoster(t *testing.T, fx *consoleFixture, roles map[string]string, fleet ...fleetBox) { |
| + t.Helper() |
| + boxes := map[string]hostactors.Box{} |
| + for _, b := range fleet { |
| + rb := moveRosterBox(b) |
| + rb.Role = roles[b.Box] |
| + boxes[b.Box] = rb |
| + } |
| + putMoveRoster(t, fx, hostactors.Roster{Boxes: boxes, Regions: map[string]hostactors.Region{ |
| + "ie": {Account: moveTestAccount, AWSRegion: "eu-west-1"}, |
| + "uk": {Account: moveTestAccount, AWSRegion: "eu-west-2"}, |
| + }}) |
| +} |
| + |
| +// bindRegionHA turns region HA on for fx's claim runtime, with the region |
| +// HA chooser bound as the create door binds it. |
| +func bindRegionHA(fx *consoleFixture) { |
| + fx.srv.cfg.RegionHA = true |
| + a := fx.srv.cfg.ClaimAuthority |
| + a.Namespace = "platform" |
| + a.Executors = fx.srv.executorChooser(a.Directory) |
| +} |
| + |
| +// haMoveFleet is armMoveFleet's ie-box and uk-box (and any more boxes), |
| +// each with fresh, roomy headroom, a signed roster naming both regions |
| +// with each box's role from roles, and, with ha, a region HA chooser. |
| +func haMoveFleet(t *testing.T, ha bool, roles map[string]string, more ...fleetBox) *consoleFixture { |
| + t.Helper() |
| + fx := newConsoleFixture(t) |
| + fleet := append([]fleetBox{ieBox("ie-box"), ukBox("uk-box")}, more...) |
| + armMoveFleet(t, fx, fleet...) |
| + names := []string{} |
| + for _, b := range fleet { |
| + names = append(names, b.Box) |
| + } |
| + roomyEdges(t, fx, nil, names...) |
| + haRoster(t, fx, roles, fleet...) |
| fx.srv.cfg.HostClaim = true |
| - fx.srv.cfg.RegionHA = ha |
| if ha { |
| d := hostactors.Directory{Store: fx.rel, PublicKey: fx.pub, Bucket: moveTestBucket} |
| - fx.srv.cfg.ClaimAuthority = &hostops.ProductionAuthority{Namespace: "platform", Directory: d, Executors: fx.srv.executorChooser(d)} |
| + fx.srv.cfg.ClaimAuthority = &hostops.ProductionAuthority{Directory: d} |
| + bindRegionHA(fx) |
| } |
| return fx |
| } |
| -// Region HA spec 2 §7 (H27): a move's destination is a door that chooses |
| -// a box. With region HA on, only a box the chooser would place on is a |
| -// destination, and none is the no-room refusal with its alert. |
| -func TestMoveDestinationIsTheChoosersPlaceableSet(t *testing.T) { |
| +// Region HA spec 2 §7 (H27), census ruling region-ha-2a rev 2 item 4: the |
| +// move door's room check only refuses. With room, the selection is the |
| +// one region HA off makes; with none, it is the no-room refusal with its |
| +// alert; and it never narrows the candidates to pick a destination main's |
| +// selection would not. |
| +func TestMoveRoomCheckOnlyRefuses(t *testing.T) { |
| const host, app = "acme.go.tito.io", "bab/titogo" |
| - t.Run("placeable", func(t *testing.T) { |
| - fx := haMoveFleet(t, true, hostactors.RoleDefault) |
| - got, err := fx.srv.selectMoveBoxes(context.Background(), host, "uk", "ie", app) |
| - if err != nil || got.Target != "uk-box" || got.Source != "ie-box" { |
| - t.Fatalf("selection %+v %v", got, err) |
| + ctx := context.Background() |
| + t.Run("room: the selection region HA off makes", func(t *testing.T) { |
| + off, err := haMoveFleet(t, false, nil).srv.selectMoveBoxes(ctx, host, "uk", "ie", app) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + on, err := haMoveFleet(t, true, nil).srv.selectMoveBoxes(ctx, host, "uk", "ie", app) |
| + if err != nil || on != off || on.Target != "uk-box" || on.Source != "ie-box" { |
| + t.Fatalf("region HA on %+v %v, off %+v", on, err, off) |
| } |
| }) |
| - t.Run("the only destination is legacy", func(t *testing.T) { |
| + t.Run("no room: the only destination is legacy", func(t *testing.T) { |
| pages := pagesTo(t) |
| - fx := haMoveFleet(t, true, hostactors.RoleLegacy) |
| - _, err := fx.srv.selectMoveBoxes(context.Background(), host, "uk", "ie", app) |
| + fx := haMoveFleet(t, true, map[string]string{"uk-box": hostactors.RoleLegacy}) |
| + _, err := fx.srv.selectMoveBoxes(ctx, host, "uk", "ie", app) |
| var full *noCapacityError |
| if !errors.As(err, &full) || full.Region != "uk" { |
| t.Fatalf("err = %v, want no room in uk", err) |
| } |
| - if ref := claimResponseRefusal(err); ref.Status != 503 || ref.Msg != noCapacityRefusal("uk") { |
| + if ref := claimResponseRefusal(err); ref.Status != http.StatusServiceUnavailable || ref.Msg != noCapacityRefusal("uk") { |
| t.Fatalf("refusal %+v", ref) |
| } |
| expectPages(t, pages, noRoomSubject("uk")) |
| }) |
| - t.Run("a platform source still moves out", func(t *testing.T) { |
| - fx := haMoveFleet(t, true, hostactors.RoleDefault) |
| - ctx := context.Background() |
| - o, err := hostactors.Directory{Store: fx.rel, PublicKey: fx.pub, Bucket: moveTestBucket}.Read(ctx) |
| - if err != nil { |
| - t.Fatal(err) |
| + t.Run("a legacy box beside a default one is not filtered out", func(t *testing.T) { |
| + roles := map[string]string{"uk-box": hostactors.RoleLegacy} |
| + off, err := haMoveFleet(t, false, roles, ukBox("uk-box-2")).srv.selectMoveBoxes(ctx, host, "uk", "ie", app) |
| + if err != nil || !off.Many { |
| + t.Fatalf("region HA off: %+v %v, want main's several-boxes refusal", off, err) |
| + } |
| + on, err := haMoveFleet(t, true, roles, ukBox("uk-box-2")).srv.selectMoveBoxes(ctx, host, "uk", "ie", app) |
| + if err != nil || on != off { |
| + t.Fatalf("region HA on %+v %v, want region HA off's %+v", on, err, off) |
| } |
| - r := hostactors.Clone(o.Roster) |
| - b := r.Boxes["ie-box"] |
| - b.Role = hostactors.RolePlatform |
| - r.Boxes["ie-box"] = b |
| - putMoveRoster(t, fx, r) |
| + }) |
| + t.Run("a platform source still moves out", func(t *testing.T) { |
| + fx := haMoveFleet(t, true, map[string]string{"ie-box": hostactors.RolePlatform}) |
| got, err := fx.srv.selectMoveBoxes(ctx, host, "uk", "ie", app) |
| if err != nil || got.Source != "ie-box" || got.Target != "uk-box" { |
| t.Fatalf("selection %+v %v", got, err) |
| @@ -5808,7 +6170,7 @@ func TestMoveDestinationIsTheChoosersPlaceableSet(t *testing.T) { |
| // reads no roster, so an unreadable one cannot refuse the move. |
| func TestMoveDestinationWithRegionHAOffIsMains(t *testing.T) { |
| const host, app = "acme.go.tito.io", "bab/titogo" |
| - fx := haMoveFleet(t, false, hostactors.RoleLegacy) |
| + fx := haMoveFleet(t, false, map[string]string{"uk-box": hostactors.RoleLegacy}) |
| got, err := fx.srv.selectMoveBoxes(context.Background(), host, "uk", "ie", app) |
| if err != nil || got.Target != "uk-box" { |
| t.Fatalf("selection %+v %v", got, err) |
| @@ -5818,80 +6180,166 @@ func TestMoveDestinationWithRegionHAOffIsMains(t *testing.T) { |
| t.Fatalf("with the roster unreadable: %+v %v", got, err) |
| } |
| } |
| + |
| +// haMoveDoor is moveDoorWithClaim's world on a host-claim console with the |
| +// claim runtime armed: roomy edges files (uk-box's changed by edit), a |
| +// roster naming both regions and, with ha, region HA on. |
| +func haMoveDoor(t *testing.T, code string, ha bool, edit func(map[string]any)) (*consoleFixture, string, string) { |
| + t.Helper() |
| + fx, token, moveURL := moveDoorWithClaim(t, code) |
| + roomyEdges(t, fx, edit, "ie-box", "uk-box") |
| + haRoster(t, fx, nil, ieBox("ie-box"), ukBox("uk-box")) |
| + armClaimMoves(t, fx, true, true, nil) |
| + if ha { |
| + bindRegionHA(fx) |
| + } |
| + return fx, token, moveURL |
| +} |
| + |
| +// Astra 2a round 2 finding 1: through the whole HTTP door, a destination |
| +// with no box that has room (no edges file, or only blank public |
| +// addresses) is the no-room refusal, alert included, and writes nothing: |
| +// not the placement checks' 400. With region HA off it is main's 400; with |
| +// room the move is admitted to uk-box either way. |
| +func TestMoveDoorWithNoRoomInTheDestinationIsNoRoom(t *testing.T) { |
| + for _, c := range []struct { |
| + name string |
| + edit func(map[string]any) |
| + }{ |
| + {"only a blank public address", func(m map[string]any) { m["eip"] = "" }}, |
| + {"no edges file", nil}, |
| + } { |
| + t.Run(c.name, func(t *testing.T) { |
| + noFile := c.edit == nil |
| + world := func(t *testing.T, code string, ha bool) (*consoleFixture, string, string) { |
| + fx, token, moveURL := haMoveDoor(t, code, ha, c.edit) |
| + if noFile { |
| + if err := fx.rel.Delete(context.Background(), edgesFilePrefix+"uk-box.json"); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + return fx, token, moveURL |
| + } |
| + _, token, moveURL := world(t, "code-ha-move-off", false) |
| + if code, _, out := rawReq(t, http.MethodPost, moveURL, token, `{"region":"uk"}`); code != http.StatusBadRequest { |
| + t.Fatalf("region HA off: move = %d %s, want main's 400", code, out) |
| + } |
| + pages := pagesTo(t) |
| + fx, token, moveURL := world(t, "code-ha-move-on", true) |
| + claim := claimBytes(t, fx, "acme.go.tito.io") |
| + code, _, out := rawReq(t, http.MethodPost, moveURL, token, `{"region":"uk"}`) |
| + if code != http.StatusServiceUnavailable || !strings.Contains(out, noCapacityRefusal("uk")) { |
| + t.Fatalf("move = %d %s", code, out) |
| + } |
| + expectPages(t, pages, noRoomSubject("uk")) |
| + if claimBytes(t, fx, "acme.go.tito.io") != claim { |
| + t.Fatal("a refused move wrote the claim") |
| + } |
| + a, err := hostops.ReadAdmission(context.Background(), fx.rel, "bab", "titogo", claimMoveAppID) |
| + if err != nil || len(a.Value.Registrations) != 0 { |
| + t.Fatalf("a refused move registered: %+v %v", a.Value.Registrations, err) |
| + } |
| + if rec, err := provision.GetRecord(context.Background(), fx.rel, "bab", "titogo", "acme.go.tito.io"); err != nil || rec.Move != nil { |
| + t.Fatalf("a refused move journaled: %+v %v", rec.Move, err) |
| + } |
| + }) |
| + } |
| + t.Run("room", func(t *testing.T) { |
| + fx, token, moveURL := haMoveDoor(t, "code-ha-move-room", true, nil) |
| + if code, _, out := rawReq(t, http.MethodPost, moveURL, token, `{"region":"uk"}`); code != http.StatusAccepted { |
| + t.Fatalf("move = %d %s", code, out) |
| + } |
| + o, err := hostclaim.Read(context.Background(), fx.rel, "acme.go.tito.io") |
| + if err != nil || o.Claim == nil || o.Claim.Op.Intent["target"] != "uk-box" { |
| + t.Fatalf("claim %+v %v, want a move to uk-box", o.Claim, err) |
| + } |
| + }) |
| +} |
| ```` |
| - [ ] **Step 2: Run them to see them fail** |
| -Run: `nice -n 19 go test ./internal/console/ -run 'MoveDestination' -count=1` |
| -Expected: FAIL: `TestMoveDestinationIsTheChoosersPlaceableSet` reports `err = <nil>, want no room in uk` (with region HA on, a destination whose only box is legacy is still chosen). `TestMoveDestinationWithRegionHAOffIsMains` passes already: it pins main's behaviour. |
| +Run: `nice -n 19 go test ./internal/console/ -run 'MoveRoomCheckOnlyRefuses|MoveDestinationWithRegionHAOff|MoveDoorWithNoRoom' -count=1` |
| +Expected: FAIL: the legacy-only destination is chosen (`err = <nil>, want no room in uk`), and both no-room shapes answer the placement checks' `400` (`region uk cannot take a move in this deployment...`). `TestMoveDestinationWithRegionHAOffIsMains` and the region HA off halves pass already: they pin main's behaviour. |
| -- [ ] **Step 3: Filter the destination and answer the refusal** |
| +- [ ] **Step 3: Ask about room, and answer the refusal** |
| ````diff |
| --- a/internal/console/boxselect.go |
| +++ b/internal/console/boxselect.go |
| -@@ -8,6 +8,7 @@ |
| +@@ -6,8 +6,10 @@ |
| + "fmt" |
| + "log" |
| "os" |
| ++ "slices" |
| "github.com/carlosframework/platform/internal/boxid" |
| + "github.com/carlosframework/platform/internal/hostops" |
| "github.com/carlosframework/platform/internal/moves" |
| ) |
| -@@ -113,6 +114,11 @@ |
| +@@ -113,6 +115,11 @@ |
| fp := s.expectedSecretsFP() |
| for _, code := range []string{to, from} { |
| cands := eligibleBoxes(snap, roster, code, aws[code], s.cfg.HomeRegion, fp, nil) |
| + if code == to { |
| -+ if cands, err = s.placeableMoveTargets(ctx, to, cands); err != nil { |
| ++ if err := s.moveRoomCheck(ctx, to, cands); err != nil { |
| + return moveSelection{}, err |
| + } |
| + } |
| if len(cands) > 1 { |
| sel.Many, sel.ManyRegion = true, s.homeRegionOut(code) |
| return sel, nil |
| -@@ -162,3 +168,35 @@ |
| +@@ -162,3 +169,29 @@ |
| } |
| return out, nil |
| } |
| + |
| -+// placeableMoveTargets is a move's destination under region HA (spec 2 |
| -+// §7, H27): the destination is a door that chooses a box, so only boxes |
| -+// the create door's chooser would place on (roles, headroom, fresh edges |
| -+// files) are candidates, and none is the no-room refusal. The source is |
| -+// never filtered, so an instance on an excluded box can still move out |
| -+// (A10-3). With region HA off, or no claim runtime, cands are main's and |
| -+// no roster is read. |
| -+func (s *Server) placeableMoveTargets(ctx context.Context, to string, cands []boxCandidate) ([]boxCandidate, error) { |
| ++// moveRoomCheck is a move's destination under region HA (spec 2 §7, |
| ++// H27): the destination is a door that chooses a box, so when none of its |
| ++// candidates is a box the create door's chooser would place on (roles, |
| ++// headroom, fresh edges files), the move is the no-room refusal. It only |
| ++// refuses (census ruling region-ha-2a rev 2): the candidates, and so the |
| ++// destination main's selection picks, are never narrowed or changed. The |
| ++// source is never checked, so an instance on an excluded box can still |
| ++// move out (A10-3). With region HA off, or no claim runtime, it answers nil |
| ++// and no roster is read. |
| ++func (s *Server) moveRoomCheck(ctx context.Context, to string, cands []boxCandidate) error { |
| + x := s.placementChooser("") |
| + if x == nil { |
| -+ return cands, nil |
| ++ return nil |
| + } |
| + boxes, err := x.PlaceableBoxes(ctx, s.cfg.claimPlacementRegion(to)) |
| + if err != nil && !errors.Is(err, hostops.ErrContract) { |
| -+ return nil, err |
| -+ } |
| -+ placeable := map[string]bool{} |
| -+ for _, b := range boxes { |
| -+ placeable[b] = true |
| ++ return err |
| + } |
| -+ var out []boxCandidate |
| + for _, c := range cands { |
| -+ if placeable[c.BoxID] { |
| -+ out = append(out, c) |
| ++ if slices.Contains(boxes, c.BoxID) { |
| ++ return nil |
| + } |
| + } |
| -+ if len(out) == 0 { |
| -+ return nil, s.noRoom(ctx, to) |
| -+ } |
| -+ return out, nil |
| ++ return s.noRoom(ctx, to) |
| +} |
| ```` |
| ````diff |
| --- a/internal/console/apiinstancemove.go |
| +++ b/internal/console/apiinstancemove.go |
| -@@ -602,6 +602,12 @@ |
| +@@ -591,6 +591,13 @@ |
| + return |
| + } |
| + } |
| ++ // Region HA (spec 2 §7, H27): a destination with no box that has room |
| ++ // is the no-room refusal, before the placement checks below answer it |
| ++ // as a 400. |
| ++ if err := s.roomIn(ctx, to, ""); err != nil { |
| ++ writeClaimResponseError(w, err) |
| ++ return |
| ++ } |
| + caps, published := placements(ctx, s.cfg.Releases) |
| + if !published { |
| + jsonError(w, http.StatusBadRequest, "this deployment publishes no armed regions. nothing to move to") |
| +@@ -602,6 +609,12 @@ |
| } |
| from := s.homeRegionOut(rec.Region) |
| sel, err := s.selectMoveBoxes(ctx, host, to, rec.Region, acct.Sqid+"/"+app) |
| @@ -5908,7 +6356,7 @@ Expected: FAIL: `TestMoveDestinationIsTheChoosersPlaceableSet` reports `err = <n |
| - [ ] **Step 4: Run the tests** |
| -Run: `nice -n 19 go test ./internal/console/ -run 'MoveDestination|SelectMoveBoxes|InstanceMove|MoveDoor' -count=1` |
| +Run: `nice -n 19 go test ./internal/console/ -run 'MoveRoom|MoveDestination|MoveDoor|SelectMoveBoxes|InstanceMove' -count=1` |
| Expected: PASS. |
| - [ ] **Step 5: Census** |
| @@ -5921,7 +6369,7 @@ Expected: PASS. |
| {"internal/console/apiinstancemove.go", "Server.moveTransitionSaying", "record-writer", "key:SetRecordMove", "d1f7162eecdc234607c708a83e9633cdf77cc6931bac6754fb51042dd8409377"}, |
| // Re-reviewed at the merge of carlos-run into seeded-instances (2026-10-13): main's door with the seed rule (refuseSeedMove) after its gate and before the transition; the transition is unchanged and refuses the same seed again under its CAS. |
| - {"internal/console/apiinstancemove.go", "Server.handleAPIInstanceMoveRequest", "record-writer", "key:SetRecordMove,lit:instances", "d4350f8b160076e4e69a0e6fdb5da9a042f4445a7c768dee49be0f6281fc20f3"}, |
| -+ {"internal/console/apiinstancemove.go", "Server.handleAPIInstanceMoveRequest", "record-writer", "key:SetRecordMove,lit:instances", "d9b96ed9652f5879ada1700f94473347d652dcfae1e024e35256770952384783"}, |
| ++ {"internal/console/apiinstancemove.go", "Server.handleAPIInstanceMoveRequest", "record-writer", "key:SetRecordMove,lit:instances", "867ed0feabca3d5c72990474887d5f88820b9b0e7ef7f72e48703ac4dfee10a8"}, |
| {"internal/console/apiinstancemove.go", "Server.registerMovesPhase1Routes", "key-site", "lit:instances", "5af589d288ba22856f631481c40cc001aac342ea74cee6ee0464ec77d0320bbc"}, |
| {"internal/console/apiinstances.go", "Server.instanceManifest", "key-site", "lit:instances", "e6160ffe84abda9163527ccb33e2ffff2109b35908adb2f3a754534325658fdf"}, |
| {"internal/console/apiinstances.go", "Server.handleAPIInstanceList", "key-site", "lit:instances", "558d0b3e7b0f06faa8e5067ff6f8c131eb899f13d08b71218dfb2454c3e319e1"}, |
| @@ -5930,15 +6378,15 @@ Expected: PASS. |
| Add this comment line after the overlay's last comment line: |
| ````text |
| -# Task 12: H27 on the move door's destination (internal/console boxselect.go placeableMoveTargets, through hostops.ExecutorChooser.PlaceableBoxes and only with region HA on). Census ruling region-ha-2a is asked to re-freeze one accepted-legacy unit, Server.handleAPIInstanceMoveRequest: one more refusal before any write, no box in the destination has room; its read sites unchanged, offsets moved. The seal moves. |
| +# Task 12: H27 on the move door's destination, through hostops.ExecutorChooser.PlaceableBoxes and only with region HA on: roomIn refuses a destination with no box that has room before the placement checks (no edges file, blank public addresses), and moveRoomCheck (internal/console boxselect.go) refuses when none of selectMoveBoxes' destination candidates is placeable. Both only refuse: the candidates and the destination are main's. Census ruling region-ha-2a is asked to re-freeze one accepted-legacy unit, Server.handleAPIInstanceMoveRequest: two refusals added before any write; its read sites unchanged, offsets moved. The seal moves. |
| ```` |
| Then set these rows. A `@coverage` row replaces the overlay's row for the same directory, or is appended when there is none; every other row is appended: |
| ````text |
| -@coverage internal/console 90eea3fdd16b35a1f43058b1082480206f6e0815f41c8863f3b3952578eaac1a |
| -@legacy 831306bc42c758ff8d82c4eb1e1860b35ebadd24ca2f8e1a45177ec7bb70019b internal/console/apiinstancemove.go Server.handleAPIInstanceMoveRequest legacy accepted-legacy accepted-legacy;rev3=census-ruling-region-ha-2a;basis=the console (cmd/console serve under console.service, user carlos-console): main's body with one more refusal before any write: no box in the destination has room (H27); read sites unchanged, offsets moved file:Exec@7187,file:Exec@7275,file:Exec@8266,file:Exec@8443,file:Exec@8594,file:Exec@9594,file:Read@5673,file:Read@7985 d9b96ed9652f5879ada1700f94473347d652dcfae1e024e35256770952384783 |
| -internal/console/apiinstancemove.go Server.handleAPIInstanceMoveRequest legacy accepted-legacy accepted-legacy;rev3=census-ruling-region-ha-2a;basis=the console (cmd/console serve under console.service, user carlos-console): main's body with one more refusal before any write: no box in the destination has room (H27); read sites unchanged, offsets moved file:Exec@7187,file:Exec@7275,file:Exec@8266,file:Exec@8443,file:Exec@8594,file:Exec@9594,file:Read@5673,file:Read@7985 d9b96ed9652f5879ada1700f94473347d652dcfae1e024e35256770952384783 |
| +@coverage internal/console 2a3739453eafdeda8cc9bb91fa2ae37b11ae6e83508e9210d93d1d370b324732 |
| +@legacy 88952543956a1b090e6b819c05be0504a899945d5c039dee8373fbaa805881a9 internal/console/apiinstancemove.go Server.handleAPIInstanceMoveRequest legacy accepted-legacy accepted-legacy;rev3=census-ruling-region-ha-2a;basis=the console (cmd/console serve under console.service, user carlos-console): main's body with two refusals added before any write, both no room in the destination (H27): roomIn before the placement checks and moveRoomCheck's through selectMoveBoxes; read sites unchanged, offsets moved file:Exec@7439,file:Exec@7527,file:Exec@8518,file:Exec@8695,file:Exec@8846,file:Exec@9846,file:Read@5925,file:Read@8237 867ed0feabca3d5c72990474887d5f88820b9b0e7ef7f72e48703ac4dfee10a8 |
| +internal/console/apiinstancemove.go Server.handleAPIInstanceMoveRequest legacy accepted-legacy accepted-legacy;rev3=census-ruling-region-ha-2a;basis=the console (cmd/console serve under console.service, user carlos-console): main's body with two refusals added before any write, both no room in the destination (H27): roomIn before the placement checks and moveRoomCheck's through selectMoveBoxes; read sites unchanged, offsets moved file:Exec@7439,file:Exec@7527,file:Exec@8518,file:Exec@8695,file:Exec@8846,file:Exec@9846,file:Read@5925,file:Read@8237 867ed0feabca3d5c72990474887d5f88820b9b0e7ef7f72e48703ac4dfee10a8 |
| ```` |
| ````diff |
| @@ -5948,8 +6396,8 @@ internal/console/apiinstancemove.go Server.handleAPIInstanceMoveRequest legacy a |
| if err = loadRootReadInventory(filepath.Join(root, "internal/provision/testdata/plan14-root-reads-region-ha-2a.tsv"), &inv, true); err != nil { |
| t.Fatal(err) |
| } |
| -- seal = "2a0c03ae8d2c0ba3712151a98e1533e6be8512271dc98968bc5435b014bd6bab" |
| -+ seal = "a7efc5beddc7793ca2647df7812ebd48120fb148f79649ddcca110131d393da4" |
| +- seal = "5013c9ebddabb1d84a57cc2199991fd0f75b97bc5a955c13d80ccf48efb1c235" |
| ++ seal = "9581ac17ef7967d92e3a97fa873792e1da3367f517b62e2bdbdf2603aca5212b" |
| errs := append(compareRootReads(got, inv), checkRootReadLegacy(inv, seal)...) |
| errs = append(errs, checkRootReadDeclarations(got, inv)...) |
| if len(errs) > 0 { |
| @@ -5961,7 +6409,7 @@ Run the census checks. Expected: PASS. |
| ```bash |
| git add internal/console/boxselect.go internal/console/boxselect_ha_test.go internal/console/apiinstancemove.go internal/provision |
| -git commit -m "region HA 2a: the move door's destination is the placeable set" |
| +git commit -m "region HA 2a: the move door refuses a destination with no room" |
| ``` |
| --- |
| @@ -5969,6 +6417,7 @@ git commit -m "region HA 2a: the move door's destination is the placeable set" |
| ### Task 13: The eligible-capacity metric |
| **Files:** |
| +- Modify: `internal/hostops/executor.go` (`Capacity` also answers the roster's AWS region; `PlaceableBoxes`' comment), `internal/hostops/executor_test.go` |
| - Create: `internal/console/regioncapacity.go` (core: the count and the pass) |
| - Create: `internal/console/awsregionmetric.go` (the AWS adapter: CloudWatch) |
| - Create: `internal/console/regioncapacity_test.go` |
| @@ -5976,13 +6425,44 @@ git commit -m "region HA 2a: the move door's destination is the placeable set" |
| - Modify: `region_ha_boundary_test.go`, the overlay |
| **Interfaces:** |
| -- Consumes: Task 6's `ExecutorChooser.Capacity`; Task 11's `placementChooser`; `Server.liveEdges`; `rootaws.LoadDefaultConfig`. |
| -- Produces: `type regionCapacity struct{ Placement, AWSRegion string; Eligible int }`; `type capacityPublisher interface{ PublishCapacity(ctx, []regionCapacity) error }`; `func regionCapacities(ctx, snap boxchoice.Snapshot, capacity func(context.Context, string) (int, error)) ([]regionCapacity, error)`; `(s *Server) publishCapacity`, `runCapacityPass`; the adapter's `capacityData`, `cloudWatchCapacity`, `capacityPublisherFromEnv(ctx, getenv)`. **For 2c:** the metric is `Carlos/Region` / `EligibleBoxes`, dimension `Region` = the placement code, unit Count, every minute, in the region's own CloudWatch; the console needs `CARLOS_REGION_METRICS=cloudwatch` and the platform box role's `cloudwatch:PutMetricData` on that namespace. |
| +- Consumes: Task 6's `ExecutorChooser`; Task 11's `placementChooser`; `Server.liveEdges`; `rootaws.LoadDefaultConfig`. |
| +- Produces: `func (x ExecutorChooser) Capacity(ctx context.Context, placement string) (int, string, error)` (the count and the AWS region the signed roster names for the placement); `type regionCapacity struct{ Placement, AWSRegion string; Eligible int }`; `type capacityPublisher interface{ PublishCapacity(ctx, []regionCapacity) error }`; `func regionCapacities(ctx, snap boxchoice.Snapshot, capacity func(context.Context, string) (int, string, error)) ([]regionCapacity, error)`; `(s *Server) publishCapacity`, `runCapacityPass`; the adapter's `capacityData`, `cloudWatchCapacity`, `capacityPublisherFromEnv(ctx, getenv)`. **For 2c:** the metric is `Carlos/Region` / `EligibleBoxes`, dimension `Region` = the placement code, unit Count, every minute, in the region's own CloudWatch; the console needs `CARLOS_REGION_METRICS=cloudwatch` and the platform box role's `cloudwatch:PutMetricData` on that namespace. |
| -The pass runs beside claim recovery, so only on a host-claim console, and only with region HA on and a publisher armed. It counts through the same chooser the create doors use (`placementChooser("")`), so the alarm and the refusal agree. Regions are the placements the boxes' edges files name, so a region whose every box has stopped publishing gets no data point, which the alarm reads as breaching. A count that fails stops the whole pass rather than publish a partial set. |
| +The pass runs beside claim recovery, so only on a host-claim console, and only with region HA on and a publisher armed. It counts through the same chooser the create doors use (`placementChooser("")`), so the alarm and the refusal agree. Regions are the placements the boxes' edges files name, so a region whose every box has stopped publishing gets no data point, which the alarm reads as breaching. Each region's count goes to the AWS region **the signed roster** names for it (Astra round 2 finding 3): `Capacity` reads it from the roster `eligible` has just read, with `eligible`'s own home mapping, so a stale or misconfigured edges file naming another AWS region cannot send the count elsewhere. `ExecutorChooser.eligible` is unchanged. A count that fails stops the whole pass rather than publish a partial set. |
| - [ ] **Step 1: Write the failing tests** |
| +````diff |
| +--- a/internal/hostops/executor_test.go |
| ++++ b/internal/hostops/executor_test.go |
| +@@ -150,8 +150,8 @@ |
| + if err := x.Permits(ctx, absent, chooserRecord(), "box-b"); err != nil { |
| + t.Fatalf("Permits(box-b) = %v", err) |
| + } |
| +- if n, err := x.Capacity(ctx, "ie"); err != nil || n != 1 { |
| +- t.Fatalf("Capacity = %d, %v; want 1", n, err) |
| ++ if n, aws, err := x.Capacity(ctx, "ie"); err != nil || n != 1 || aws != "eu-west-1" { |
| ++ t.Fatalf("Capacity = %d in %q, %v; want 1 in eu-west-1", n, aws, err) |
| + } |
| + if boxes, err := x.PlaceableBoxes(ctx, "ie"); err != nil || len(boxes) != 1 || boxes[0] != "box-b" { |
| + t.Fatalf("PlaceableBoxes = %v, %v; want [box-b]", boxes, err) |
| +@@ -159,10 +159,10 @@ |
| + }) |
| + } |
| + x, _, _, _ := chooserFixture(t, "box-a", "box-b") |
| +- if n, err := x.Capacity(ctx, "ie"); err != nil || n != 2 { |
| +- t.Fatalf("Capacity with both placeable = %d, %v; want 2", n, err) |
| +- } |
| +- if _, err := x.Capacity(ctx, "us"); !errors.Is(err, ErrContract) { |
| ++ if n, aws, err := x.Capacity(ctx, "ie"); err != nil || n != 2 || aws != "eu-west-1" { |
| ++ t.Fatalf("Capacity with both placeable = %d in %q, %v; want 2 in eu-west-1", n, aws, err) |
| ++ } |
| ++ if _, _, err := x.Capacity(ctx, "us"); !errors.Is(err, ErrContract) { |
| + t.Fatalf("Capacity of a region the roster lacks = %v, want ErrContract", err) |
| + } |
| + } |
| +```` |
| + |
| `internal/console/regioncapacity_test.go`: |
| ````go |
| @@ -6002,7 +6482,9 @@ import ( |
| ) |
| // The count runs without any cloud adapter (§25): it reads only the |
| -// edges snapshot and the chooser. |
| +// edges snapshot and the chooser. Each region's AWS region is the |
| +// chooser's (the signed roster's), whatever a file claims: box z's file |
| +// names uk in eu-west-1. |
| func TestRegionCapacitiesCountsEachPlacement(t *testing.T) { |
| snap := boxchoice.Snapshot{Complete: true, Boxes: []boxchoice.Box{ |
| {BoxID: "a", Placement: "uk", Region: "eu-west-2"}, |
| @@ -6010,17 +6492,18 @@ func TestRegionCapacitiesCountsEachPlacement(t *testing.T) { |
| {BoxID: "c", Placement: "ie", Region: "eu-west-1"}, |
| {BoxID: "d", Placement: "fleet/pi", Region: "eu-west-1"}, |
| {BoxID: "e", Placement: ""}, |
| + {BoxID: "z", Placement: "uk", Region: "eu-west-1"}, |
| }} |
| asked := []string{} |
| - capacity := func(_ context.Context, p string) (int, error) { |
| + capacity := func(_ context.Context, p string) (int, string, error) { |
| asked = append(asked, p) |
| switch p { |
| case "uk": |
| - return 2, nil |
| + return 2, "eu-west-2", nil |
| case "ie": |
| - return 0, nil |
| + return 0, "eu-west-1", nil |
| } |
| - return 0, hostops.ErrContract // no roster region: left out |
| + return 0, "", hostops.ErrContract // no roster region: left out |
| } |
| got, err := regionCapacities(context.Background(), snap, capacity) |
| if err != nil { |
| @@ -6032,12 +6515,37 @@ func TestRegionCapacitiesCountsEachPlacement(t *testing.T) { |
| } |
| // Any other error stops the count: a partial set would read as no |
| // capacity. |
| - failing := func(context.Context, string) (int, error) { return 0, errors.New("SlowDown") } |
| + failing := func(context.Context, string) (int, string, error) { return 0, "", errors.New("SlowDown") } |
| if _, err := regionCapacities(context.Background(), snap, failing); err == nil { |
| t.Fatal("a failed count published") |
| } |
| } |
| +// capacityRecorder keeps what one pass would publish. |
| +type capacityRecorder struct{ got []regionCapacity } |
| + |
| +func (c *capacityRecorder) PublishCapacity(_ context.Context, counts []regionCapacity) error { |
| + c.got = counts |
| + return nil |
| +} |
| + |
| +// Astra 2a round 2 finding 3, end to end: a stray box's edges file naming |
| +// ie in another AWS region (and sorting after the real box) does not move |
| +// ie's count out of the roster's eu-west-1. |
| +func TestPublishCapacityTakesTheRegionFromTheRoster(t *testing.T) { |
| + cx := haClaimCase(t, "") |
| + publishHABox(t, cx, "box-ie-default", false, nil) |
| + publishHABox(t, cx, "box-ie-stray", false, func(m map[string]any) { m["region"] = "eu-west-2" }) |
| + var pub capacityRecorder |
| + if err := cx.srv.publishCapacity(context.Background(), &pub); err != nil { |
| + t.Fatal(err) |
| + } |
| + want := []regionCapacity{{Placement: "ie", AWSRegion: "eu-west-1", Eligible: 1}} |
| + if !reflect.DeepEqual(pub.got, want) { |
| + t.Fatalf("published %+v, want %+v", pub.got, want) |
| + } |
| +} |
| + |
| func TestPlacementChooserIsScopedLikeCreate(t *testing.T) { |
| s := &Server{cfg: Config{RegionHA: true}} |
| if s.placementChooser("") != nil { |
| @@ -6083,10 +6591,55 @@ func TestCloudWatchCapacityPublishesEachRegionInItsOwnRegion(t *testing.T) { |
| - [ ] **Step 2: Run them to see them fail** |
| -Run: `nice -n 19 go test ./internal/console/ -run 'RegionCapacities|PlacementChooserIsScoped|CloudWatchCapacity' -count=1` |
| -Expected: FAIL to build (`undefined: regionCapacities`, `undefined: cloudWatchCapacity`). |
| +Run: `nice -n 19 go test ./internal/hostops/ -run 'Executor|Chooser' -count=1; nice -n 19 go test ./internal/console/ -run 'RegionCapacities|PlacementChooserIsScoped|CloudWatchCapacity|PublishCapacityTakes' -count=1` |
| +Expected: FAIL to build (`assignment mismatch: 3 variables but x.Capacity returns 2 values`; `undefined: regionCapacities`, `undefined: cloudWatchCapacity`). |
| -- [ ] **Step 3: Write the pass and the adapter** |
| +- [ ] **Step 3: Write the count, the pass and the adapter** |
| + |
| +````diff |
| +--- a/internal/hostops/executor.go |
| ++++ b/internal/hostops/executor.go |
| +@@ -89,8 +89,8 @@ |
| + |
| + // PlaceableBoxes is every box in placement that could take a new |
| + // instance in the chooser's namespace and pool now, sorted: Choose's set, |
| +-// before the choice. The console's move door takes a move's destination |
| +-// from it (region HA spec 2 §7, H27). |
| ++// before the choice. The console's create and move doors ask it whether |
| ++// a region has room before they choose (region HA spec 2 §7, H27). |
| + func (x ExecutorChooser) PlaceableBoxes(ctx context.Context, placement string) ([]string, error) { |
| + r := Record{Holder: hostclaim.NewHolder{Region: placement}} |
| + o, cands, err := x.eligible(ctx, hostclaim.Observed{Absent: true}, r) |
| +@@ -104,12 +104,22 @@ |
| + return boxes, nil |
| + } |
| + |
| +-// Capacity is how many boxes PlaceableBoxes names. The console publishes |
| +-// it per region as the eligible-capacity metric (region HA spec 2 §4, |
| +-// A1-11). |
| +-func (x ExecutorChooser) Capacity(ctx context.Context, placement string) (int, error) { |
| +- boxes, err := x.PlaceableBoxes(ctx, placement) |
| +- return len(boxes), err |
| ++// Capacity is how many boxes PlaceableBoxes names, and the AWS region the |
| ++// signed roster names for placement. The console publishes the count per |
| ++// region as the eligible-capacity metric (region HA spec 2 §4, A1-11), in |
| ++// that AWS region: the roster's mapping, the one eligible reads, never an |
| ++// edges file's own claim (Astra 2a round 2 finding 3). |
| ++func (x ExecutorChooser) Capacity(ctx context.Context, placement string) (int, string, error) { |
| ++ r := Record{Holder: hostclaim.NewHolder{Region: placement}} |
| ++ o, cands, err := x.eligible(ctx, hostclaim.Observed{Absent: true}, r) |
| ++ if err != nil { |
| ++ return 0, "", err |
| ++ } |
| ++ // eligible's own home mapping, so the region is the one it read. |
| ++ if placement == park.HomeRegion && x.HomeName != "" && park.PlacementRegion(x.HomeName) == x.HomeName { |
| ++ placement = x.HomeName |
| ++ } |
| ++ return len(x.placeable(o, cands)), o.Roster.Regions[placement].AWSRegion, nil |
| + } |
| + |
| + // placeable is region HA spec 2's policy over the eligible set (§7): |
| +```` |
| `internal/console/regioncapacity.go`: |
| @@ -6129,31 +6682,35 @@ type capacityPublisher interface { |
| } |
| // regionCapacities counts each placement the snapshot's files name, sorted |
| -// by placement. A placement the roster does not know (no region entry, so |
| -// no box there can be chosen) is left out. Any other error stops the |
| -// count: a partial set would read as regions with no capacity. |
| -func regionCapacities(ctx context.Context, snap boxchoice.Snapshot, capacity func(context.Context, string) (int, error)) ([]regionCapacity, error) { |
| - aws := map[string]string{} |
| +// by placement, and takes each one's AWS region from capacity's answer: |
| +// the signed roster's, never the files' own claim, so a stale or |
| +// misconfigured file naming another AWS region cannot send a region's |
| +// count elsewhere (Astra 2a round 2 finding 3). A placement the roster |
| +// does not know (no region entry, so no box there can be chosen) is left |
| +// out. Any other error stops the count: a partial set would read as |
| +// regions with no capacity. |
| +func regionCapacities(ctx context.Context, snap boxchoice.Snapshot, capacity func(context.Context, string) (int, string, error)) ([]regionCapacity, error) { |
| + named := map[string]bool{} |
| for _, b := range snap.Boxes { |
| - if b.Placement != "" && b.Region != "" { |
| - aws[b.Placement] = b.Region |
| + if b.Placement != "" { |
| + named[b.Placement] = true |
| } |
| } |
| - placements := make([]string, 0, len(aws)) |
| - for p := range aws { |
| + placements := make([]string, 0, len(named)) |
| + for p := range named { |
| placements = append(placements, p) |
| } |
| sort.Strings(placements) |
| var out []regionCapacity |
| for _, p := range placements { |
| - n, err := capacity(ctx, p) |
| + n, aws, err := capacity(ctx, p) |
| if errors.Is(err, hostops.ErrContract) { |
| continue |
| } |
| if err != nil { |
| return nil, err |
| } |
| - out = append(out, regionCapacity{Placement: p, AWSRegion: aws[p], Eligible: n}) |
| + out = append(out, regionCapacity{Placement: p, AWSRegion: aws, Eligible: n}) |
| } |
| return out, nil |
| } |
| @@ -6312,7 +6869,7 @@ func capacityPublisherFromEnv(ctx context.Context, getenv func(string) string) c |
| - [ ] **Step 4: Run the tests** |
| -Run: `nice -n 19 go test ./internal/console/ -run 'RegionCapacities|PlacementChooserIsScoped|CloudWatchCapacity|ClaimRecovery' -count=1` |
| +Run: `nice -n 19 go test ./internal/hostops/ -run 'Executor|Chooser' -count=1 && nice -n 19 go test ./internal/console/ -run 'RegionCapacities|PlacementChooserIsScoped|CloudWatchCapacity|PublishCapacityTakes|ClaimRecovery' -count=1` |
| Expected: PASS. |
| - [ ] **Step 5: Boundary list and census** |
| @@ -6340,13 +6897,14 @@ Expected: PASS. |
| Add this comment line after the overlay's last comment line: |
| ````text |
| -# Task 13: the eligible-capacity metric (internal/console regioncapacity.go, counted through hostops.ExecutorChooser.Capacity; awsregionmetric.go, the CloudWatch adapter), started beside claim recovery (StartClaimRecoveryPass). No root-read unit changed or was added. Envelope-only, and the seal stands. |
| +# Task 13: the eligible-capacity metric (internal/console regioncapacity.go, counted through hostops.ExecutorChooser.Capacity, which now also answers the AWS region the signed roster names for the placement; awsregionmetric.go, the CloudWatch adapter), started beside claim recovery (StartClaimRecoveryPass). No root-read unit changed or was added: Capacity reads through eligible, unchanged. Envelope-only, and the seal stands. |
| ```` |
| Then set these rows. A `@coverage` row replaces the overlay's row for the same directory, or is appended when there is none; every other row is appended: |
| ````text |
| -@coverage internal/console cddef46a81289d748b6b50a1f2bf9e2aa2c95144bdd1cea4095d3fb476058cbd |
| +@coverage internal/hostops a86df6341b8b1d44676da9ff1310a204f2b34380cb438cc05fe71ae1834834f3 |
| +@coverage internal/console 745590987511f5758f0af07f63851275c30334825379af7f0dde62fa05c991e0 |
| ```` |
| Run the census checks and `go test . -run 'TestRegionHA(Core|Adapters|S3)' -count=1`. Expected: PASS: `regioncapacity.go` imports no AWS package, and `awsregionmetric.go` is named for AWS. |
| @@ -6354,7 +6912,7 @@ Run the census checks and `go test . -run 'TestRegionHA(Core|Adapters|S3)' -coun |
| - [ ] **Step 6: Commit** |
| ```bash |
| -git add internal/console/regioncapacity.go internal/console/regioncapacity_test.go internal/console/awsregionmetric.go internal/console/hostclaimops.go region_ha_boundary_test.go internal/provision/testdata/plan14-root-reads-region-ha-2a.tsv |
| +git add internal/hostops/executor.go internal/hostops/executor_test.go internal/console/regioncapacity.go internal/console/regioncapacity_test.go internal/console/awsregionmetric.go internal/console/hostclaimops.go region_ha_boundary_test.go internal/provision/testdata/plan14-root-reads-region-ha-2a.tsv |
| git commit -m "region HA 2a: publish each region's eligible capacity" |
| ``` |
| @@ -6372,4 +6930,5 @@ git commit -m "region HA 2a: publish each region's eligible capacity" |
| ## Open questions |
| 1. **Hosting before 2b.** Task 9 marks a box hosting when the claim agent's wiring starts, which proves enrolment and a matching roster row but does not itself check that the floor guard is armed for that enrolment. 2b's mode selection makes it exact. Is that acceptable for 2a, given no load balancer reads the answer until 2c? |
| -2. **The census values (condition 3).** Every value here was computed at main `70e57449`. Plan 14 Parts 2 and 3 will move the base; the implementer recomputes every value there with the tooling above, and platform-1e's ruling covers the rows, not these hashes. |
| +2. **`Server.chooseCreateFirst` (new in rev 3).** It is asked of platform-1e as accepted-legacy, with the basis `beginClaimCreate` has. If the ruling prefers no second claim read, the alternative is a hook inside `beginClaimCreate` that runs the binding after its choice and before `RegisterStart`; that changes the signature of a re-frozen unit and its three callers, so rev 3 does not take it. |
| +3. **The census values (condition 3).** Every value here was computed at main `70e57449`. Plan 14 Parts 2 and 3 will move the base; the implementer recomputes every value there with the tooling above, and platform-1e's ruling covers the rows, not these hashes. |