amadan
docs: nearest-region plan rev 6, after Astra's plan round 5, proved in scratch
canary --region carries the record its direct read returned into the create/rechannel/ready decision and the --environment bind, so a same-region canary past the capped list is reused, never created twice. Command tests cover the matching and a different channel, and the bind. Rollout step 2 marks CARLOS_SECRETS_KEY staged on all four flagship boxes (2026-10-03).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+206
−26
docs/superpowers/plans/2026-10-02-nearest-region-default.md+206 −26
diff --git a/docs/superpowers/plans/2026-10-02-nearest-region-default.md b/docs/superpowers/plans/2026-10-02-nearest-region-default.md| index 8d4bebf9e..893277964 100644 |
| --- a/docs/superpowers/plans/2026-10-02-nearest-region-default.md |
| +++ b/docs/superpowers/plans/2026-10-02-nearest-region-default.md |
| @@ -2,7 +2,7 @@ |
| > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| -**Revision:** 5. |
| +**Revision:** 6. |
| - Rev 1 (9513d52c) was reviewed by Astra |
| (`CODEX-PLAN-NEAREST-REGION-R1.md`, beside this file): not ready, with |
| @@ -14,10 +14,12 @@ |
| carries spec rev 6.1's R9a note (`region_known`, `placement_known`). |
| - Rev 5 answers Astra's round 4 (`CODEX-PLAN-NEAREST-REGION-R4.md`) and |
| carries spec rev 6.2 (`instance_domain`, approved by Paul 2026-10-03). |
| +- Rev 6 answers Astra's round 5 (`CODEX-PLAN-NEAREST-REGION-R5.md`), and |
| + marks the flagship's secrets-key staging done (Rollout step 2). |
| - The code blocks the findings touched were **proved**: extracted verbatim |
| into a throwaway worktree at the branch base, built and vetted, and each |
| prescribed test was run before and after its implementation step. |
| -- The four dispositions tables, with the proof results, are at the end. |
| +- The five dispositions tables, with the proof results, are at the end. |
| **Goal:** Give every new app a region at creation, preselected from a timezone-then-IP guess, and have new instances default to that region. Never place an instance automatically on a box that cannot open the deployment's secrets. Existing apps, older CLIs and DNS stay unchanged. |
| @@ -6600,7 +6602,9 @@ func (r apiRegionsResult) nameOf(code string) string // a place name, else the c |
| func deployRegionPreflight(res apiRegionsResult, readErr error, app, asked string) error |
| ``` |
| -`regionE2EConsole` is a root-package test helper, reused by Task 22. |
| +`regionE2EConsole` is a root-package test helper, reused by Task 22. It |
| +counts writes, and separately the instance-create requests (`creates`), so |
| +Task 22 can tell a duplicate create from the other writes. |
| - [ ] **Step 1: Write the failing unit test** |
| @@ -6805,7 +6809,10 @@ type regionE2E struct { |
| store release.Store |
| mode atomic.Value // "", "fail", "legacy", "region-unknown", "placement-unknown" |
| writes atomic.Int64 |
| - cst *console.Store |
| + // creates counts instance-create requests (POST …/instances), so a |
| + // test can tell a duplicate create from the other writes (Task 22). |
| + creates atomic.Int64 |
| + cst *console.Store |
| // failGet is a store key whose reads fail while the listing succeeds, |
| // the way one bucket GET can (Task 22, codex plan-R3 1). |
| failGet atomic.Value |
| @@ -6856,6 +6863,9 @@ func regionE2EConsole(t *testing.T, appRegion string) *regionE2E { |
| if r.Method != http.MethodGet && r.Method != http.MethodHead && !strings.Contains(r.URL.Path, "/auth/") { |
| e.writes.Add(1) |
| } |
| + if r.Method == http.MethodPost && strings.HasSuffix(r.URL.Path, "/instances") { |
| + e.creates.Add(1) |
| + } |
| if strings.HasSuffix(r.URL.Path, "/regions") { |
| switch e.mode.Load().(string) { |
| case "fail": |
| @@ -7037,7 +7047,9 @@ Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| with its `failGet` store; |
| - Task 15: `instance_domain` on `app_default`. |
| - Produces: |
| - - `func canaryRegionCheck(res apiRegionsResult, readErr error, app, name, asked string, get func(host string) (apiInstanceRecord, bool, error)) (string, error)`; |
| + - `func canaryRegionCheck(res apiRegionsResult, readErr error, app, name, asked string, get func(host string) (apiInstanceRecord, bool, error)) (canaryPreflight, error)`; |
| + - `type canaryPreflight struct{ Host string; Rec apiInstanceRecord; Found bool }`; |
| + - `func canaryPlanFor(pre canaryPreflight, recs []apiInstanceRecord, host, channel string) canaryPlan`; |
| - `func (s *apiSession) instanceGet(ctx context.Context, a apiAcct, app, host string) (apiInstanceRecord, bool, error)`; |
| - `func instanceCreatedLine(rec apiInstanceRecord, envSuffix string) string`. |
| @@ -7061,6 +7073,15 @@ false `region_known` refuses with C30b like a false `placement_known` |
| (codex plan-R4 1), and the e2e has a fleet-placed case with the region |
| unread that must refuse with zero writes. |
| +The record that one GET returns, and whether it exists, also decide what |
| +happens next (codex plan-R5 1). `canaryRegionCheck` returns them in a |
| +`canaryPreflight`, and `canaryPlanFor` turns that into create, rechannel or |
| +ready. With `--region` the direct read decides, so a same-region canary the |
| +capped list never showed is reused or rechannelled, never created twice |
| +(which the door would refuse). The `--environment` bind uses the same |
| +decision. Without `--region` nothing is read and the list decides, exactly |
| +as today. |
| + |
| After the PUT, the host built from the PUT's domain must equal the host the |
| check read. If it does not (an operator re-homed the manifest between the |
| two), the command stops with C30c before any record write. |
| @@ -7145,7 +7166,7 @@ func TestCanaryRegionCheck(t *testing.T) { |
| } |
| for _, c := range cases { |
| get, asked := records(c.readErr, c.recs...) |
| - checked, err := canaryRegionCheck(c.res, c.err, "blog", "c", c.asked, get) |
| + pre, err := canaryRegionCheck(c.res, c.err, "blog", "c", c.asked, get) |
| got := "" |
| if err != nil { |
| got = err.Error() |
| @@ -7156,8 +7177,35 @@ func TestCanaryRegionCheck(t *testing.T) { |
| if len(*asked) > 1 || (len(*asked) == 1 && (*asked)[0] != host) { |
| t.Errorf("%s: read %v, want at most one read of %s", c.name, *asked, host) |
| } |
| - if c.want == "" && c.asked != "" && checked != host { |
| - t.Errorf("%s: checked %q, want %q", c.name, checked, host) |
| + if c.want == "" && c.asked != "" && pre.Host != host { |
| + t.Errorf("%s: checked %q, want %q", c.name, pre.Host, host) |
| + } |
| + if c.want == "" && c.asked != "" && pre.Found != (len(c.recs) > 0 && c.recs[0].Host == host) { |
| + t.Errorf("%s: found = %v, want the read's own answer", c.name, pre.Found) |
| + } |
| + } |
| +} |
| + |
| +// codex plan-R5 1: with --region, the direct read decides the plan, even |
| +// when the capped list does not show the record at all. |
| +func TestCanaryPlanFor(t *testing.T) { |
| + const host, ch = "c.blog.go.tito.io", "canary/c" |
| + listed := []apiInstanceRecord{{Host: host, Channel: ch}} |
| + cases := []struct { |
| + name string |
| + pre canaryPreflight |
| + recs []apiInstanceRecord |
| + want canaryPlan |
| + }{ |
| + {"no --region: the list decides", canaryPreflight{}, listed, canaryPlanReady}, |
| + {"no --region, not listed", canaryPreflight{}, nil, canaryPlanCreate}, |
| + {"read found it, list did not, same channel", canaryPreflight{Host: host, Found: true, Rec: apiInstanceRecord{Host: host, Channel: ch}}, nil, canaryPlanReady}, |
| + {"read found it, list did not, other channel", canaryPreflight{Host: host, Found: true, Rec: apiInstanceRecord{Host: host, Channel: "canary/old"}}, nil, canaryPlanRechannel}, |
| + {"read found none", canaryPreflight{Host: host}, listed, canaryPlanCreate}, |
| + } |
| + for _, c := range cases { |
| + if got := canaryPlanFor(c.pre, c.recs, host, ch); got != c.want { |
| + t.Errorf("%s: %v, want %v", c.name, got, c.want) |
| } |
| } |
| } |
| @@ -7180,7 +7228,7 @@ func TestInstanceCreatedLine(t *testing.T) { |
| - [ ] **Step 2: Run them to verify they fail** |
| -Run: `go test . -run 'TestCanaryRegionCheck|TestInstanceCreatedLine' -count=1` |
| +Run: `go test . -run 'TestCanaryRegionCheck|TestCanaryPlanFor|TestInstanceCreatedLine' -count=1` |
| Expected: FAIL. |
| - [ ] **Step 3: Implement the helpers** |
| @@ -7238,27 +7286,28 @@ In `canary.go`: |
| // instance_domain) or a read that fails refuses (C30c); a record in |
| // another region refuses (C25), named by place. |
| // |
| -// It returns the host it checked, which the caller holds the later |
| -// instancesEnable answer to. |
| -func canaryRegionCheck(res apiRegionsResult, readErr error, app, name, asked string, get func(host string) (apiInstanceRecord, bool, error)) (string, error) { |
| +// It returns what it read (codex plan-R5 1): the host, which the caller |
| +// holds the later instancesEnable answer to, and the record and whether it |
| +// exists, which decide create, rechannel or ready (canaryPlanFor). |
| +func canaryRegionCheck(res apiRegionsResult, readErr error, app, name, asked string, get func(host string) (apiInstanceRecord, bool, error)) (canaryPreflight, error) { |
| if asked == "" { |
| - return "", nil |
| + return canaryPreflight{}, nil |
| } |
| // Both flags: effective is only stated when the stored region was read |
| // too (R9a), and the fleet check below reads effective (codex plan-R4 1). |
| if readErr != nil || res.AppDefault == nil || !res.AppDefault.PlacementKnown || !res.AppDefault.RegionKnown { |
| - return "", errors.New(canaryFleetUnknown(app)) |
| + return canaryPreflight{}, errors.New(canaryFleetUnknown(app)) |
| } |
| if strings.HasPrefix(res.AppDefault.Effective, "fleet/") { |
| - return "", errors.New(canaryRegionOnFleet(app, strings.TrimPrefix(res.AppDefault.Effective, "fleet/"))) |
| + return canaryPreflight{}, errors.New(canaryRegionOnFleet(app, strings.TrimPrefix(res.AppDefault.Effective, "fleet/"))) |
| } |
| if res.AppDefault.InstanceDomain == "" { |
| - return "", errors.New(canaryRegionUnconfirmed(name)) |
| + return canaryPreflight{}, errors.New(canaryRegionUnconfirmed(name)) |
| } |
| host := canaryHost(name, app, res.AppDefault.InstanceDomain) |
| rec, found, err := get(host) |
| if err != nil { |
| - return "", errors.New(canaryRegionUnconfirmed(name)) |
| + return canaryPreflight{}, errors.New(canaryRegionUnconfirmed(name)) |
| } |
| if found { |
| have := rec.Region |
| @@ -7266,10 +7315,33 @@ func canaryRegionCheck(res apiRegionsResult, readErr error, app, name, asked str |
| have = res.Default |
| } |
| if have != asked { |
| - return "", errors.New(canaryRegionMismatch(host, res.nameOf(have))) |
| + return canaryPreflight{}, errors.New(canaryRegionMismatch(host, res.nameOf(have))) |
| } |
| } |
| - return host, nil |
| + return canaryPreflight{Host: host, Rec: rec, Found: found}, nil |
| +} |
| + |
| +// canaryPreflight is what canaryRegionCheck read. Host is "" when --region |
| +// was not given; then nothing was read. |
| +type canaryPreflight struct { |
| + Host string |
| + Rec apiInstanceRecord |
| + Found bool |
| +} |
| + |
| +// canaryPlanFor decides create, rechannel or ready. With --region the |
| +// direct read decides (codex plan-R5 1): the list is capped at 500 and |
| +// skips records it cannot read, so it can miss a canary that exists, and |
| +// creating it again would fail. Without --region the list decides, as |
| +// before. |
| +func canaryPlanFor(pre canaryPreflight, recs []apiInstanceRecord, host, channel string) canaryPlan { |
| + if pre.Host == "" { |
| + return canaryRecordPlan(recs, host, channel) |
| + } |
| + if !pre.Found { |
| + return canaryPlanCreate |
| + } |
| + return canaryRecordPlan([]apiInstanceRecord{pre.Rec}, host, channel) |
| } |
| ``` |
| @@ -7301,12 +7373,12 @@ After `ctx := context.Background()` and `channel := canaryChannel(canary)` |
| ```go |
| // --region is checked with reads only, before the first write (the |
| // instancesEnable PUT below): nothing changes when it is refused. |
| - checked := "" |
| + var pre canaryPreflight |
| if asked := strings.TrimSpace(*regionFlag); asked != "" { |
| res, rerr := sess.appRegions(ctx, acct, app2) |
| get := func(h string) (apiInstanceRecord, bool, error) { return sess.instanceGet(ctx, acct, app2, h) } |
| var err error |
| - if checked, err = canaryRegionCheck(res, rerr, app2, canary, asked, get); err != nil { |
| + if pre, err = canaryRegionCheck(res, rerr, app2, canary, asked, get); err != nil { |
| log.Fatalf("canary: %v", err) |
| } |
| } |
| @@ -7315,10 +7387,25 @@ After `ctx := context.Background()` and `channel := canaryChannel(canary)` |
| `ctx` is declared at `canary.go:375`, so the block must come after it (codex |
| plan-R2 5). |
| +Then the one plan decision, used by the create/rechannel/ready switch and by |
| +the `--environment` bind. Replace `switch canaryRecordPlan(recs, host, channel) {` |
| +(`canary.go:405`) with: |
| + |
| +```go |
| + // One decision, used twice below: with --region it comes from the |
| + // direct read of the canary's own record (codex plan-R5 1). |
| + plan := canaryPlanFor(pre, recs, host, channel) |
| + switch plan { |
| +``` |
| + |
| +and, in the `--environment` block, replace |
| +`canaryRecordPlan(recs, host, channel) != canaryPlanCreate` with |
| +`plan != canaryPlanCreate`. |
| + |
| Right after `host := canaryHost(canary, app2, m.Domain)`: |
| ```go |
| - if checked != "" && host != checked { |
| + if pre.Host != "" && host != pre.Host { |
| // The PUT named a different domain from the one the check read |
| // under: the canary checked is not the one about to be written. |
| log.Fatalf("canary: %s", canaryRegionUnconfirmed(canary)) |
| @@ -7448,6 +7535,57 @@ func TestCanaryRegionRefusesBeforeAnyWrite(t *testing.T) { |
| } |
| } |
| +// codex plan-R5 1: a same-region canary past the list's 500-record cap is |
| +// the record the command reuses, found by the direct read, never re-created. |
| +// On its own channel it is left as it is; on another it is rechannelled. |
| +// Either way there is no create request and the run goes on to ship. |
| +func TestCanaryRegionReusesACanaryPastTheListCap(t *testing.T) { |
| + const existing = "probe.titogo.bab.oncarlos.test" |
| + for _, c := range []struct{ name, channel, env string }{ |
| + {"same channel", "canary/probe", ""}, |
| + {"different channel", "canary/old", ""}, |
| + // The environment bind follows the same decision: an existing |
| + // record gets its own bind write, never a create carrying it. |
| + {"same channel, with --environment", "canary/probe", "beta"}, |
| + } { |
| + t.Run(c.name, func(t *testing.T) { |
| + e := regionE2EConsole(t, "") |
| + ctx := context.Background() |
| + for i := 0; i < 520; i++ { |
| + rec := provision.Record{Schema: 1, Host: fmt.Sprintf("a%03d.titogo.bab.oncarlos.test", i), App: "titogo", |
| + Account: release.OpsAccount, Channel: "stable", CreatedAt: time.Now().UTC()} |
| + if err := provision.CreateRecord(ctx, e.store, rec); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + rec := provision.Record{Schema: 1, Host: existing, App: "titogo", Account: release.OpsAccount, |
| + Region: "uk", Channel: c.channel, CreatedAt: time.Now().UTC()} |
| + if err := provision.CreateRecord(ctx, e.store, rec); err != nil { |
| + t.Fatal(err) |
| + } |
| + args := []string{"canary", "deploy", "--app", "titogo", "--name", "probe", "--region", "uk", |
| + "--console", e.url, "--version", "abc1234", "--timeout", "1s"} |
| + if c.env != "" { |
| + args = append(args, "--environment", c.env) |
| + } |
| + out, _ := e.run(t, append(args, artifact(t))...) |
| + if n := e.creates.Load(); n != 0 { |
| + t.Fatalf("%d create request(s) for a canary that exists:\n%s", n, out) |
| + } |
| + if e.releases(t) == 0 { |
| + t.Fatalf("the run never shipped:\n%s", out) |
| + } |
| + got, err := provision.GetRecord(ctx, e.store, release.OpsAccount, "titogo", existing) |
| + if err != nil || got.Region != "uk" || got.Channel != "canary/probe" || got.Environment != c.env { |
| + t.Fatalf("the canary = %+v %v, want it in uk on canary/probe, environment %q", got, err, c.env) |
| + } |
| + if recs, _, err := provision.ListRecords(ctx, e.store, release.OpsAccount, "titogo", 1000); err != nil || len(recs) != 521 { |
| + t.Fatalf("records = %d %v, want the 521 there were", len(recs), err) |
| + } |
| + }) |
| + } |
| +} |
| + |
| // codex plan-R3 6: a successful replica create prints C14 with its place, |
| // not an older console's "(channel X)". This console has no secrets key, |
| // so no box is eligible and the place falls back to its code. |
| @@ -7480,6 +7618,12 @@ canary sort before `probe.` and fill the list's 500-record page. The |
| unreadable case fails only the store GET of the canary's record key, so the |
| listing still names it and `ListRecords` skips it. |
| +`TestCanaryRegionReusesACanaryPastTheListCap` is codex plan-R5 1: a `uk` |
| +canary behind 520 records, run with `--region uk`. On its own channel it is |
| +left alone, on another it is rechannelled, and with `--environment` it gets |
| +its bind write. Each case sends no create request (the harness counts |
| +`POST …/instances`), keeps 521 records and goes on to ship. |
| + |
| `TestInstancesCreateReplicaPrintsItsPlace` is codex plan-R3 6's CLI half: |
| a successful replica create prints C14 (`channel beta, uk`), where an older |
| console's response gives today's `(channel beta)`. The harness console has |
| @@ -7496,6 +7640,13 @@ Then the ordering check: move the `--region` block from step 4 to after the |
| `go test . -run TestCanaryRegionRefusesBeforeAnyWrite -count=1`. It must |
| FAIL on writes. Move it back, and re-run: PASS. |
| +Then the reuse checks (codex plan-R5 1). Replace `plan := canaryPlanFor(…)` |
| +with `plan := canaryRecordPlan(recs, host, channel)` (rev 5's wiring): all |
| +three `ReusesACanaryPastTheListCap` cases must FAIL with "1 create |
| +request(s)". Restore. Then make only the `--environment` condition read the |
| +list again: the `with --environment` case must FAIL. Restore, and re-run: |
| +PASS. |
| + |
| Then the authoritative-read check (codex plan-R3 1): replace the |
| `canaryRegionCheck` call with a scan of `instanceList`'s records for the |
| canary's host (rev 3's design). The `past the list cap` and `unreadable` |
| @@ -8021,9 +8172,14 @@ operators. |
| notes and the carloku-infrastructure roll recipe. |
| 2. **Stage the secrets key (R9).** Put the deployment's `CARLOS_SECRETS_KEY` |
| in `/etc/carlos/host.env` on every box in every region that should take |
| - placed instances. On the flagship that means UK, US and AU, which today |
| - lack it. This is infrastructure work in carloku-infrastructure, under |
| - AGENTS.md's scoped-approval rule. |
| + placed instances. This is infrastructure work in carloku-infrastructure, |
| + under AGENTS.md's scoped-approval rule. |
| + - **Flagship: done (2026-10-03).** `CARLOS_SECRETS_KEY` is on all four |
| + flagship boxes (Ireland, UK, US, AU), from SSM `/carlos/secrets-key` in |
| + every region, and config sync and generation are armed everywhere. |
| + Step 3's check (every box's `secrets_key_fp` equals the console's) is |
| + still to run once the box release publishes it. |
| + - Any other deployment still needs this step. |
| 3. **Box release** (Tasks 1 and 1b) to every box. Confirm each region's |
| `control/edges/*.json` carries `serves_shared_pool`, `accepts_inbound` and |
| `secrets_key_fp`, and that every `secrets_key_fp` equals the console's. |
| @@ -8094,7 +8250,8 @@ to the follow-on spec. |
| and every console call site uses `s.regionCands(snap)`. |
| - These are defined once and used consistently: `regionCandidate` |
| (`KeyOK`, `NoKeyProven`), `regionReq`, `appDefaultView`/`apiAppDefault`, |
| - `apiRegionsResult.nameOf`, `canaryRegionCheck(res, readErr, app, name, asked, get)`, |
| + `apiRegionsResult.nameOf`, `canaryRegionCheck(res, readErr, app, name, asked, get)` |
| + returning a `canaryPreflight`, `canaryPlanFor`, |
| `instanceGet`, `platformInstanceDomain`, `explicitGate`, |
| `instanceChoices`, |
| `deployRegionPreflight(res, readErr, …)`, |
| @@ -8296,3 +8453,26 @@ For Paul, outside the findings: |
| `region_known` as well as `placement_known`. It follows from R9a, which |
| leaves `effective` empty without the stored region, rather than being a |
| new ruling. |
| + |
| +## Review dispositions (Astra, plan round 5, on 7db8869f with spec 56756f09) |
| + |
| +Astra found every earlier finding resolved and accepted the `/tmp/g` |
| +operator rule. One new P1. It was proved as before: a throwaway worktree at |
| +the branch base (e8406ce6), rev 5's blocks replayed and committed there as |
| +the baseline, the new tests run failing against rev 5's wiring and passing |
| +after the fix, and the changed blocks then compared with the scratch files |
| +byte for byte (all matched). The scratch worktrees and branch were deleted |
| +afterwards. |
| + |
| +| # | sev | finding | disposition | proof | |
| +| --- | --- | --- | --- | --- | |
| +| R5-1 | P1 | `canary --region` discarded the record it read, so a same-region canary past the list's cap took the create path | **accepted.** `canaryRegionCheck` returns a `canaryPreflight` (host, record, found). `canaryPlanFor` makes the one create/rechannel/ready decision from it when `--region` was given, and the list decides only without `--region`, as today. The `--environment` bind uses the same decision. The Task 21 harness now also counts `POST …/instances`. | `TestCanaryRegionReusesACanaryPastTheListCap` (a `uk` canary behind 520 records, `--region uk`): against rev 5's wiring all three cases FAIL with "1 create request(s)" (the console refuses the duplicate); after, same channel, different channel (rechannelled to `canary/probe`) and with `--environment beta` (bound) all PASS with no create request, 521 records and the release shipped. Reading the list again for the `--environment` decision alone FAILs the third case. `TestCanaryPlanFor` covers the decision table. The other 7 canary e2e cases, the replica test and the deploy tests still PASS | |
| + |
| +Whole-tree results on the rev 6 scratch tree: `go build ./...`, |
| +`go vet ./...` and `gofmt -l` clean, and all five cross-compile targets |
| +built. The 17 existing tests in `canary_test.go` were run one at a time on |
| +the rev 5 and rev 6 trees: 17 PASS on each. |
| + |
| +Rollout: `CARLOS_SECRETS_KEY` is on all four flagship boxes as of |
| +2026-10-03 (SSM `/carlos/secrets-key` in every region; config sync and |
| +generation armed everywhere), so step 2 is marked done for the flagship. |