amadan
docs: region HA plan 1b-next rev 11 tests the console chooser's readers against real evidence
Astra round 10 P2, accepted. The console chooser test now fills the bucket with two live boxes' edges and status files and three serving claims. It asserts box identities and load counts through the constructed chooser, then changes the evidence and checks the next read reflects it. A scan of every stage A test for stub-satisfiable assertions found two more. ProductionAuthority.SelectExecutor now uses a two-box fixture where the chooser must pick box-b. regionHAEnabled gains its own test. Stage B now requires each stubbed provider to be tested with its production value. Dry-run in task order on 3407d114; 4 guards removed in turn, each failing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+161
−14
docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md+161 −14
diff --git a/docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md b/docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md| index c7db808d2..d7f253e74 100644 |
| --- a/docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md |
| +++ b/docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md |
| @@ -1,15 +1,45 @@ |
| -# Region HA 1b-next: Relocation, Row Cases, Create Choice and Fences Implementation Plan (rev 10) |
| +# Region HA 1b-next: Relocation, Row Cases, Create Choice and Fences Implementation Plan (rev 11) |
| > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| **Goal:** Build what region HA spec 1 had to leave until the host lifecycle claim was on main: the `relocating` operation within one region, released by the source's stop evidence (drain) or by its termination fence (failover); reconcile's row cases for it; create's once-only choice of executor box with `boxchoice.Choose`; and every claim CAS that consumes a termination fence. All of it dormant behind `CARLOS_REGION_HA`, and none of it mergeable until the joint two-box simulation round passes (L18). |
| -**Architecture:** Two stages (ruling 24). **Stage A, build now:** Tasks 1 to 7 and 9a, buildable on main today and dry-run on every revision: the claim library's relocation transitions and `released` coverage (`internal/hostclaim`), create's executor selection (`internal/hostclaim`, `internal/hostops`), the row and wake decisions (`internal/hostbox`), the writers-stopped observation (`internal/stopevidence`), the regional worker and the participant driver (new `internal/hostrelocate`), and the box-side release adapter with its receipt verifier (`internal/host`). **Stage B, gated:** Tasks 8a, 8b, 8c, 9 and 10 bind to claim Plan 6b and moves Plans 1 and 2, none merged. They keep their design and contracts here; their code is rewritten against the real 6b code after it merges and re-reviewed then (the "Stage B re-plan" step before Task 8a lists every obligation rounds 1 to 9 placed on them). What those need and no plan declares is a numbered request, never an assumption. The last task is the merge gate. The worker only reads termination fences: `hostfence.TerminationWorker` writes them, and `hostactors.Directory` is the only roster reader. |
| +**Architecture:** Two stages (ruling 24). **Stage A, build now:** Tasks 1 to 7 and 9a, buildable on main today and dry-run on every revision: the claim library's relocation transitions and `released` coverage (`internal/hostclaim`), create's executor selection (`internal/hostclaim`, `internal/hostops`), the row and wake decisions (`internal/hostbox`), the writers-stopped observation (`internal/stopevidence`), the regional worker and the participant driver (new `internal/hostrelocate`), and the box-side release adapter with its receipt verifier (`internal/host`). **Stage B, gated:** Tasks 8a, 8b, 8c, 9 and 10 bind to claim Plan 6b and moves Plans 1 and 2, none merged. They keep their design and contracts here; their code is rewritten against the real 6b code after it merges and re-reviewed then (the "Stage B re-plan" step before Task 8a lists every obligation rounds 1 to 10 placed on them). What those need and no plan declares is a numbered request, never an assumption. The last task is the merge gate. The worker only reads termination fences: `hostfence.TerminationWorker` writes them, and `hostactors.Directory` is the only roster reader. |
| **Tech Stack:** Go 1.26, module `github.com/carlosframework/platform`; `release.Store` (create-only and ETag compare-and-swap puts); the claim library (`hostclaim`), roster (`hostactors`), fences (`hostfence`, `termfence`), box choice (`boxchoice`), files (`hostfiles`), generation selection (`genselect`), stop evidence (`stopevidence`, `internal/host` stop receipts), execution controller (`hostexec`). |
| **Spec:** `docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md`, rev 5.4 with §24 (rulings H1 to H19 final). Read §0, §2, §5.2, §5.3, §5.8, §5.9, §5.10, §9 (L1 to L20, accepted) and §24 first. Then the claim spec rev 6.30 on `origin/design-replica-ownership` (`docs/superpowers/specs/2026-10-02-host-lifecycle-claim-design.md`): §1.1's activation gate, §4 ("Same-region relocation (L1)" and the relocation exception to CC11), §5.1 and P5-D2/P5-D6 (the create intent), §6.1 ("Release before full stop (L10)"), §6.4a item 2 (`released`), §7.3 D2 ("Automatic termination (L2, L4)", "The fence's journal reference") and D4 ("Automatic loss persistence"). Then moves rev 7.22 on `origin/design-moves-any-region` §7.6 ("One install routine, two sources", L16 and L17) and §7.10 (the pre-flip conversion). The contracts note `docs/superpowers/notes/2026-10-07-region-ha-contracts.md` gives `hostfiles` v2, `CopyManifest`, `Publish` and `boxchoice`. Plan 1b (`docs/superpowers/plans/2026-10-06-region-ha-1b-claim-readers.md`) lists in "Waits for the claim" what this plan now builds. |
| +## Rev 11: round 10 dispositions |
| + |
| +Astra's round 10 (`CODEX-PLAN-REGION-HA-1BN-R10.md`) resolved both findings from round 9 and raised one P2: a test that would still pass with its production wiring disconnected. It is accepted; none is declined. |
| + |
| +I then checked every stage A test for the same weakness. For each test of production wiring, the question was whether its assertions still hold with the wiring replaced by a stub. Two more tests failed that check. Both are fixed here, and the table lists them as S1 and S2. |
| + |
| +The changes were dry-run on a clean 3407d114, applying stage A in task order: build, vet for darwin and windows, cross-builds, and the targeted `hostops`, `console` and `cmd/console` tests. Four new guards were removed in turn, and each removal failed its test (listed in the Self-review). |
| + |
| +| # | Stage | Evidence checked | Disposition | What changed | Where | |
| +|---|---|---|---|---|---| |
| +| 1 (P2) | A | `TestCreateExecutorChooserReadsTheLiveEdgesAndTheClaims` read an empty bucket. Both readers returning nothing satisfied every assertion. | accepted | The bucket now holds evidence for two boxes in `ie`: their edges files (box-b advertising `region-ha-1`) and fresh status snapshots. It also holds three serving claims, two on box-a for bab/jam and one on box-b for bab/tea. The test reads through the constructed chooser's `Edges` and `Loads`. It asserts each box's identity, freshness, placement and capabilities, and each box's host and per-app counts. Then the evidence changes: box-a starts draining and goes stale, and a fourth host serves on box-b. The next read must show both changes. | Task 2 (`TestCreateExecutorChooserReadsTheLiveEdgesAndTheClaims`) | |
| +| S1 | A | Found in the scan. `TestProductionAuthoritySelectsOnlyWithAChooser` used a one-box fixture, so a `SelectExecutor` that ignored the chooser and returned a fixed box-a still passed. | fixed | Two boxes, with box-a draining. The selection must be box-b with its roster enrollment. With both draining it must be `ErrNoEligibleBox` and no selection. | Task 2 | |
| +| S2 | A | Found in the scan. `regionHAEnabled` sets `Config.RegionHA` from `CARLOS_REGION_HA`, and no test covered it. | fixed | `cmd/console/regionha_test.go`: on only for `strconv.ParseBool` true values, with whitespace trimmed. Unset, `0`, `false`, `no` and garbage are off. | Task 2 | |
| + |
| +The scan found nothing else in stage A. These tests already run against real wiring: |
| +- hostclaim's transitions, on a real store. |
| +- `ClaimLoads`, on real claims. |
| +- `Guard.AdmissionFenced` and Task 9a's fence observer, against a real `BeginFence`. |
| +- `RelocationPolicy`, through `Policy.Check`. |
| +- The writers-stopped scan, against the real kernel and a fixture `/proc`. |
| +- The worker, over a signed roster, `hostactors.Directory` and `hostfence.ClaimFenceAuthority`. |
| +- `StopReceiptReleases`, against stored receipts. |
| +- `ReleaseFlushFiles`, against a real store and registry. |
| +- `ReleasePositionsFile` and `RelocationControllers`, on disk. |
| +- `OpenRelocationController`, with a real hostexec controller. |
| + |
| +The remaining stubs are providers that Task 10 (stage B) supplies: the worker's `Choose`, `Eligible`, `Leases` and `Prepared`; `StorePinner.SelectDB`, whose production value is `genselect.Select` over the versioned bucket; and `RelocationEffects`' providers. The stage B re-plan has a row that requires each one to be tested with its production value. |
| + |
| +Astra's readings of rulings 1 to 24 stand. |
| + |
| ## Rev 10: round 9 dispositions |
| Astra's round 9 (`CODEX-PLAN-REGION-HA-1BN-R9.md`) resolved both round 8 findings and raised two new ones. Both are accepted; none is declined. |
| @@ -221,7 +251,7 @@ Left out, with where it goes: |
| - **Go environment.** `GOFLAGS=-buildvcs=false`. Leave `GOCACHE` at its default; if the sandbox refuses it, `GOCACHE=$TMPDIR/gocache` for that command. Socket tests need `TMPDIR=/tmp/k` (`mkdir -p /tmp/k`). Run tests niced (`nice -n 19`), targeted with `-run`; never the full `-race` suite, and no simulation sweeps. Scratch goes under `$TMPDIR`. |
| - **Gate before each commit:** `go build ./... && go vet ./<pkgs>/ && gofmt -l <dirs>` (empty), the task's targeted tests, then `GOOS=darwin go build ./... && GOOS=windows go build ./...`. (`GOOS=windows go vet ./internal/...` fails on main's own test files, `release/versions_test.go`, `appconfig/syncexec_test.go` and `activator/activator_test.go`; vet the task's packages instead.) |
| - Commit messages: lowercase `area:` prefix and a sentence stating the behaviour, then a blank line and `Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>`. Work on a branch cut from `origin/main` (suggested `region-ha-1b-next`). Merging is Paul's call, through `amadan branch merge`, and only after Task 11. |
| -- **Dry run.** Tasks 1 to 7 were applied in order to `origin/main` at 3407d114, built, vetted, cross-built for darwin, windows and linux/arm64, and their tests run; nineteen guards were each removed in turn and their test watched to fail (rev 1), then rev 2's twenty-five, rev 3's eleven, rev 4's eight, rev 5's seven, rev 6's eleven, rev 7's three, rev 8's two, rev 9's two and rev 10's one (listed in the Self-review); from rev 10 on, stage A is also applied one task at a time in plan order, building and testing after each. Then the code was removed; only this document is committed. |
| +- **Dry run.** Tasks 1 to 7 were applied in order to `origin/main` at 3407d114, built, vetted, cross-built for darwin, windows and linux/arm64, and their tests run; nineteen guards were each removed in turn and their test watched to fail (rev 1), then rev 2's twenty-five, rev 3's eleven, rev 4's eight, rev 5's seven, rev 6's eleven, rev 7's three, rev 8's two, rev 9's two, rev 10's one and rev 11's four (listed in the Self-review); from rev 10 on, stage A is also applied one task at a time in plan order, building and testing after each. Then the code was removed; only this document is committed. |
| ## Review Focus |
| @@ -2551,6 +2581,7 @@ Spec §5.2. The executor is `boxchoice.Choose(boxchoice.Eligible(...))` over the |
| - Modify: `internal/hostops/operations.go` (`Authority.SelectExecutor`, `Attempt`), `internal/hostops/backfill_controller.go`, `internal/hostops/backfill.go` (`ProductionAuthority.Executors`, `SelectExecutor`), `internal/hostops/operations_test.go` (the fake authority) |
| - Create: `internal/console/hostclaimexecutor.go`, `internal/console/hostclaimexecutor_test.go` (the console's chooser; the create door uses it once claim Plan 5's door lands, Task 10) |
| - Modify: `internal/console/server.go` (`Config.RegionHA`), `cmd/console/main.go` (sets it from `CARLOS_REGION_HA` with `strconv.ParseBool`) |
| +- Test: `cmd/console/regionha_test.go` (rev 11: the switch itself) |
| **Interfaces:** |
| - Consumes: `boxchoice.Eligible`, `boxchoice.Choose`, `boxchoice.Snapshot`, `boxchoice.Load`, `hostactors.Directory.Read`, `Directory.TerminationFence`, `hostactors.Roster.Regions`; Task 1's `hostclaim.LocalChoreBoxes` (the eligible set excludes a box owing a `park-local`, rev 6). |
| @@ -2817,11 +2848,20 @@ func TestProductionAuthoritySelectsOnlyWithAChooser(t *testing.T) { |
| if err != nil || selected || sel != (hostclaim.ExecutorSelection{}) { |
| t.Fatalf("no chooser: %+v %v %v", sel, selected, err) |
| } |
| - x, _, _, _ := chooserFixture(t, "box-a") |
| - sel, selected, err = (ProductionAuthority{Executors: &x}).SelectExecutor(context.Background(), hostclaim.Observed{Host: "shop.bab.oncarlos.com", Absent: true}, chooserRecord()) |
| - if err != nil || !selected || sel.Box != "box-a" { |
| + // With a chooser the selection is the chooser's, read from its |
| + // evidence: box-a is draining, so box-b with its roster enrollment; |
| + // with both draining, no box and an error, never a selection. |
| + x, _, snap, _ := chooserFixture(t, "box-a", "box-b") |
| + snap.Boxes[0].Draining = true |
| + absent := hostclaim.Observed{Host: "shop.bab.oncarlos.com", Absent: true} |
| + sel, selected, err = (ProductionAuthority{Executors: &x}).SelectExecutor(context.Background(), absent, chooserRecord()) |
| + if err != nil || !selected || sel.Box != "box-b" || sel.Enrollment != strings.Repeat("b", 32) { |
| t.Fatalf("with a chooser: %+v %v %v", sel, selected, err) |
| } |
| + snap.Boxes[1].Draining = true |
| + if sel, selected, err = (ProductionAuthority{Executors: &x}).SelectExecutor(context.Background(), absent, chooserRecord()); !errors.Is(err, ErrNoEligibleBox) || selected { |
| + t.Fatalf("no eligible box: %+v %v %v", sel, selected, err) |
| + } |
| } |
| ``` |
| @@ -2832,13 +2872,24 @@ package console |
| import ( |
| "context" |
| + "slices" |
| "testing" |
| + "time" |
| + "github.com/carlosframework/platform/internal/boxchoice" |
| "github.com/carlosframework/platform/internal/hostactors" |
| + "github.com/carlosframework/platform/internal/hostclaim" |
| "github.com/carlosframework/platform/internal/release" |
| ) |
| +// The console's chooser reads the bucket through its two production |
| +// readers: the live edges (each box's edges file joined with its status |
| +// snapshot) and the serving claims' loads. The bucket holds real evidence, |
| +// the assertions name boxes and counts, and a change to the stored |
| +// evidence shows in the next read, so neither reader can be replaced by |
| +// one returning nothing (round 10). |
| func TestCreateExecutorChooserReadsTheLiveEdgesAndTheClaims(t *testing.T) { |
| + ctx := context.Background() |
| store := release.NewDirStore(t.TempDir()) |
| off := &Server{cfg: Config{Releases: store, HomeRegion: "ie"}} |
| if off.executorChooser(hostactors.Directory{Store: store, Bucket: "bucket"}) != nil { |
| @@ -2849,13 +2900,107 @@ func TestCreateExecutorChooserReadsTheLiveEdgesAndTheClaims(t *testing.T) { |
| if x == nil || x.Edges == nil || x.Loads == nil || x.Now == nil || x.HomeName != "ie" || x.SecretsKeyFP != s.expectedSecretsFP() { |
| t.Fatalf("chooser: %+v", x) |
| } |
| - snap, err := x.Edges(context.Background()) |
| - if err != nil || len(snap.Boxes) != 0 { |
| - t.Fatalf("an empty bucket: %+v %v", snap, err) |
| + |
| + // Two live boxes in ie, each with its edges file and a fresh status |
| + // snapshot; box-b advertises the region-ha-1 capability. |
| + edges := func(box, extra string) string { |
| + return `{"v":1,"box":"` + box + `","box_id":"` + box + `","eip":"203.0.113.1","region":"eu-west-1","placement":"ie","hibernation_configured":true` + extra + `,"hosts":{}}` |
| + } |
| + for box, body := range map[string]string{"box-a": edges("box-a", ""), "box-b": edges("box-b", `,"capabilities":["`+boxchoice.Capability+`"]`)} { |
| + setBoxSnapshot(t, store, box, time.Now()) |
| + if _, err := store.Put(ctx, "control/edges/"+box+".json", []byte(body), ""); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + // Two serving claims on box-a for bab/jam, one on box-b for bab/tea. |
| + created := time.Date(2026, 10, 4, 1, 0, 0, 0, time.UTC) |
| + serve := func(host, app, box string) { |
| + t.Helper() |
| + nh := hostclaim.NewHolder{Family: "instance", Key: "apps/bab/" + app + "/instances/" + host + ".json", Account: "bab", App: app, Host: host, Region: "ie", RecordRegion: "ie", CreatedAt: created} |
| + body := []byte(`{"host":"` + host + `","account":"bab","app":"` + app + `","region":"ie","created_at":"2026-10-04T01:00:00Z"}`) |
| + o, err := hostclaim.Read(ctx, store, host) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if o, err = hostclaim.Update(ctx, store, o, hostclaim.BeginCreate(nh, body, "c", "console", created)); err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, err = hostclaim.Update(ctx, store, o, hostclaim.CompleteCreate("c", box)); err != nil { |
| + t.Fatal(err) |
| + } |
| } |
| - loads, err := x.Loads(context.Background()) |
| - if err != nil || len(loads) != 0 { |
| - t.Fatalf("no claims: %+v %v", loads, err) |
| + serve("a.bab.oncarlos.com", "jam", "box-a") |
| + serve("b.bab.oncarlos.com", "jam", "box-a") |
| + serve("c.bab.oncarlos.com", "tea", "box-b") |
| + |
| + read := func() (map[string]boxchoice.Box, map[string]boxchoice.Load) { |
| + t.Helper() |
| + snap, err := x.Edges(ctx) |
| + if err != nil || !snap.Complete { |
| + t.Fatalf("edges: %+v %v", snap, err) |
| + } |
| + boxes := map[string]boxchoice.Box{} |
| + for _, b := range snap.Boxes { |
| + boxes[b.BoxID] = b |
| + } |
| + loads, err := x.Loads(ctx) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + return boxes, loads |
| + } |
| + boxes, loads := read() |
| + if len(boxes) != 2 || !boxes["box-a"].Fresh || !boxes["box-b"].Fresh || boxes["box-a"].Placement != "ie" || |
| + slices.Contains(boxes["box-a"].Capabilities, boxchoice.Capability) || !slices.Contains(boxes["box-b"].Capabilities, boxchoice.Capability) { |
| + t.Fatalf("live edges: %+v", boxes) |
| + } |
| + if loads["box-a"].Hosts != 2 || loads["box-a"].Apps["bab/jam"] != 2 || loads["box-b"].Hosts != 1 || loads["box-b"].Apps["bab/tea"] != 1 { |
| + t.Fatalf("loads: %+v", loads) |
| + } |
| + |
| + // The evidence changes: box-a starts draining and stops reporting |
| + // (stale), and a fourth host serves on box-b. The next read says so. |
| + _, etag, err := store.Get(ctx, "control/edges/box-a.json") |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, err := store.Put(ctx, "control/edges/box-a.json", []byte(edges("box-a", `,"draining":true`)), etag); err != nil { |
| + t.Fatal(err) |
| + } |
| + setBoxSnapshot(t, store, "box-a", time.Now().Add(-time.Hour)) |
| + serve("d.bab.oncarlos.com", "tea", "box-b") |
| + boxes, loads = read() |
| + if !boxes["box-a"].Draining || boxes["box-a"].Fresh || !boxes["box-b"].Fresh { |
| + t.Fatalf("live edges after the change: %+v", boxes) |
| + } |
| + if loads["box-b"].Hosts != 2 || loads["box-b"].Apps["bab/tea"] != 2 || loads["box-a"].Hosts != 2 { |
| + t.Fatalf("loads after the change: %+v", loads) |
| + } |
| +} |
| +``` |
| + |
| +`cmd/console/regionha_test.go` (the switch that sets `Config.RegionHA`): |
| + |
| +```go |
| +package main |
| + |
| +import "testing" |
| + |
| +// CARLOS_REGION_HA reaches Config.RegionHA through regionHAEnabled |
| +// (strconv.ParseBool): only a true value turns the console's create-time |
| +// chooser on; unset, false and garbage leave create as today. |
| +func TestRegionHAEnabledParsesTheSwitch(t *testing.T) { |
| + for v, want := range map[string]bool{"": false, "0": false, "false": false, "no": false, "garbage": false, |
| + "1": true, "t": true, "true": true, "TRUE": true, " true ": true} { |
| + got := regionHAEnabled(func(k string) string { |
| + if k != "CARLOS_REGION_HA" { |
| + t.Fatalf("read %s", k) |
| + } |
| + return v |
| + }) |
| + if got != want { |
| + t.Errorf("CARLOS_REGION_HA=%q: %v, want %v", v, got, want) |
| + } |
| } |
| } |
| ``` |
| @@ -3378,13 +3523,13 @@ func (s *Server) executorChooser(d hostactors.Directory) *hostops.ExecutorChoose |
| - [ ] **Step 6: Run the tests** |
| -Run: `nice -n 19 go test ./internal/hostclaim/ ./internal/hostops/ -count=1 && nice -n 19 go test ./internal/console/ -run 'CreateExecutorChooser|Claim|BoxChoice' -count=1 && go vet ./internal/console/ ./internal/host/` |
| +Run: `nice -n 19 go test ./internal/hostclaim/ ./internal/hostops/ -count=1 && nice -n 19 go test ./internal/console/ -run 'CreateExecutorChooser|Claim|BoxChoice' -count=1 && nice -n 19 go test ./cmd/console/ -run RegionHA -count=1 && go vet ./internal/console/ ./internal/host/ ./cmd/console/` |
| Expected: PASS; vet clean (no other `hostops.Authority` implementation exists on main). |
| - [ ] **Step 7: Commit** |
| ```bash |
| -git add internal/hostclaim/create.go internal/hostclaim/create_test.go internal/hostops/ internal/console/hostclaimexecutor.go internal/console/hostclaimexecutor_test.go internal/console/server.go cmd/console/main.go |
| +git add internal/hostclaim/create.go internal/hostclaim/create_test.go internal/hostops/ internal/console/hostclaimexecutor.go internal/console/hostclaimexecutor_test.go internal/console/server.go cmd/console/main.go cmd/console/regionha_test.go |
| git commit -m "hostops: with region HA on, create chooses its executor once with boxchoice over the roster and records it in the create intent |
| Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| @@ -9260,6 +9405,7 @@ When claim Plan 6b and moves Plans 1 and 2 are on main, rewrite Tasks 8a, 8b, 8c |
| | Obligation | Round | Covered by | Proved by | |
| |---|---|---|---| |
| | Release providers bound to the real attempt: identity, producers, collect, writers-only stop, descriptors recorded at open | 1 (1, 8), 2 (N3) | 6b's `localStopIdentity`, `Wake.Producers`, `CollectClaimStop`; R2; R3 (agent half) | `TestRelocationStopWritersLeavesTheReplicatorOpen`, `TestRelocationEffectsAreBound` | |
| +| Every stage A provider a test stubs is tested with its production value once wired: the worker's `Choose` and `Eligible` (`ExecutorChooser` adapters), `Leases` (`*lease.Held`), `Prepared` (`PreparedFromInstallManifest`), `StorePinner.SelectDB` (`genselect.Select` over the versioned data bucket), and `RelocationEffects`' providers; replacing any with a stub that returns nothing must fail a test | 10 (scan) | Task 10's adapters; Tasks 8a to 8c | a test per provider over real evidence (genselect's versioned-bucket fixture for the pinner) | |
| | The producers' drain is idempotent: a second `DrainClaim` for the same stop and fence, after the producers have closed, acknowledges again with the same digest (Task 7's resumed drain relies on it after root fails between the drain and its receipt) | 7 (1) | 6b's `Wake.Producers` / `ClaimProducers.DrainClaim` | over the assembly: drain, root fails before `CompleteFence`, retry; one flush, two acknowledgements, a receipt | |
| | The complete writers-stopped scan is root's, authenticated, over the control socket | 2 (N3) | R3 (root half), `CollectWritersStopped` | the root endpoint under the production uid | |
| | `Region` on every stop request; `Recover` for the recorded owner after a restart | 2 (9), 3 (R1-9), 4 (R1-9) | 6b's `StopRequest.Region`, `Recover` | `…ReplaysOnlyRecordedPositions/after_a_restart…` against 6b's coordinator | |
| @@ -11076,3 +11222,4 @@ Expected: exit 0. Exit 2 is pending: wait and read again. Exit 1: read the log a |
| - **Rev 8 guards**, each removed in turn and its test watched to fail: `Release` marking a same-controller retry with recorded positions as resumed; the resumed drain skipping straight to the producers' drain. Two in all. |
| - **Rev 9 guards**, each removed in turn and its test watched to fail: `FenceExhaustedTarget` keeping a live source's `op.exec` and phase (the `starting` and `releasing` cases revoke it); `Release` keeping the recorded directory revision whenever positions are recorded (the same-controller retry asks at the new revision). Two in all. |
| - **Rev 10 guard**, removed and its test watched to fail: `Attempt` creating with the selection it made (`TestCreateIsTodaysWithoutASelection`: the completion substitutes box-a). The in-order dry run is the check for finding 1. One in all. |
| +- **Rev 11 guards**, each removed in turn and its test watched to fail: the console chooser's live-edges reader (an empty snapshot fails on the box identities); its claim-loads reader (an empty map fails on the counts); `ProductionAuthority.SelectExecutor` consulting its chooser (a fixed box-a fails); `regionHAEnabled` honouring the parsed value (any parseable value as on fails on `0`). Four in all. |