amadan
plan: multi-region plan 3 of 3 — steering (spec Part 4)
Eight tasks, dormant until CARLOS_STEERING_ZONE_ID arms: the two stage-5 gates from plan 2's final review land first (seeded edges gate + region field + stuck-state visibility); then the steering field (additive, 4-place, pool-railed at the door), the pure desired-state fold (steered hosts x armed edges + the always-on edge.<apps-domain> custom-domain target), a Route53 converger bounded to its managed record set with per-EIP health checks, the housekeeper duty (mintMeshKeyLoop's shape), IAM surface, and rollout notes. Spec amendment recorded: the DNS writer is the housekeeper, not the console (writer-map consistency; "never tofu-per-app" unchanged).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 file changed,
+336
−0
docs/superpowers/plans/2026-08-23-multi-region-steering.md+336 −0
diff --git a/docs/superpowers/plans/2026-08-23-multi-region-steering.md b/docs/superpowers/plans/2026-08-23-multi-region-steering.md| new file mode 100644 |
| index 000000000..2bb55264a |
| --- /dev/null |
| +++ b/docs/superpowers/plans/2026-08-23-multi-region-steering.md |
| @@ -0,0 +1,336 @@ |
| +# Multi-region ingress, plan 3 of 3: steering (spec Part 4) |
| + |
| +> **For agentic workers:** REQUIRED SUB-SKILL: Use |
| +> superpowers:subagent-driven-development (recommended) or |
| +> superpowers:executing-plans to implement this plan task-by-task. Steps |
| +> use checkbox (`- [ ]`) syntax for tracking. |
| + |
| +**Goal:** Ship spec Part 4 — the per-route `steering` opt-in, the Route53 |
| +latency-record + health-check converger, and the custom-domain steering |
| +hostname — dormant until a deployment arms `CARLOS_STEERING_ZONE_ID`. |
| + |
| +**Architecture:** `steering` is an additive record/registry field |
| +(`"latency"` or absent), member-settable through the console door and |
| +CLI, refused on non-shared-pool routes (v1 guardrail from the plan-2 |
| +ledger). A housekeeper duty (the one election) converges Route53: one |
| +latency A record per steered host per armed edge (armed = present in |
| +`control/edges/` with an EIP; SetIdentifier = box label; Region = the |
| +edge's AWS region, an additive edges-file field), each bound to a |
| +per-EIP HTTPS health check; withdrawal and health-check GC are part of |
| +the same convergence. `edge.<apps-domain>` is the always-steered CNAME |
| +target for custom domains. Task 1 first lands the two |
| +MUST-BEFORE-STAGE-5 gates from plan 2's final review. |
| + |
| +**Tech Stack:** Go; aws-sdk-go-v2 route53 service package (new direct |
| +dep beside the existing SDK family); the edge-side `route53dns.go` |
| +provider shows the house auth/client idioms; `release.Store` for state |
| +reads. No DNSimple path for steering (Route53-only; both production |
| +zones are Route53 — dated spec note in Task 8). |
| + |
| +## Global Constraints |
| + |
| +- Spec: `docs/superpowers/specs/2026-08-20-multi-region-ingress-design.md` |
| + Part 4 (as amended through main 920d599). Steering field values: |
| + exactly `""`(absent) and `"latency"` — a string, never a bool. |
| +- **Writer-map rules**: nothing writes `control/serial` from a box; the |
| + DNS converger is a HOUSEKEEPER duty (Task 8 amends the spec's |
| + "console-driven" phrasing, dated — the console SETS the field; the |
| + elected housekeeper CONVERGES DNS; "never tofu-per-app" unchanged). |
| +- **Registry column rule**: adding `steering` touches FOUR places |
| + (ALTER list, routeCols, scanRoute, Upsert) — currently 35 columns, |
| + becomes 36; `TestUpsertRoundTripsAllColumns` catches a miscount. |
| + `Fingerprint` and `SnapshotFingerprint` gain it automatically via |
| + routeCols — pin with a test, don't assume. |
| +- **Wire-shape changes are additive**: `steering` and the edges-file |
| + `region` field are omitempty; records written before this plan parse |
| + unchanged (round-trip test on a pre-plan fixture). |
| +- **Pool guardrail (v1, from plan-2 ledger)**: the door and CLI REFUSE |
| + `steering:"latency"` on a route whose Pool != "" — a steered |
| + non-shared-pool route would get latency records pointing at edges |
| + that structurally cannot serve it. Refusal text names the constraint; |
| + the per-pool-record-scoping alternative is a recorded future, not v1. |
| +- **Dormancy**: with `CARLOS_STEERING_ZONE_ID` unset the converger duty |
| + returns immediately (log once at arm, then silent); setting the field |
| + on a route changes NOTHING until a deployment arms the zone. Every |
| + task states its inertness. |
| +- Gate before every commit: `go build ./... && go vet ./... && |
| + go test ./... -count=1`, `GOOS=darwin go build ./...`, `gofmt -l` |
| + clean on touched files, `-race` on packages with concurrency tests, |
| + env `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off` |
| + (if /tmp fills: `go clean -cache`). |
| +- Production channel names (`internal/release/laddercheck_test.go`) |
| + untouched. carlosTestEnv + scratch cmd.Dir for CLI-shelling tests. |
| + |
| +## File Structure |
| + |
| +- `internal/host/edgesfile.go` — Task 1 (seed `last` from bucket; fix |
| + the :124 heal-cycle comment; additive `region` field) |
| +- `internal/edge/homemap.go`, `internal/edge/edge.go` — Task 1 |
| + (stuck-state visibility) |
| +- `internal/provision/provision.go`, `internal/registry/registry.go`, |
| + `internal/provision/reconcile.go` — Task 2 (the field, 4-place) |
| +- `internal/console/apiinstances.go`, CLI `instances_cli.go`/`cli.go` — |
| + Task 3 (door + CLI + pool rail) |
| +- `internal/steering/steering.go` + `route53.go` + tests — Tasks 4-6 |
| + (desired-state builder, converger, fake-API tests) |
| +- `internal/host/agent.go`, `internal/host/housekeeper.go` — Task 4 |
| + (the duty loop, mintMeshKeyLoop's shape) |
| +- `infra/modules/region-host/` — Task 7 (IAM) |
| +- Spec + this plan — Task 8 |
| + |
| +--- |
| + |
| +### Task 1: The stage-5 gates (plan-2 final-review debts) |
| + |
| +**Files:** |
| +- Modify: `internal/host/edgesfile.go` (loop at ~:122-153; the :124 |
| + comment; the pass's schema struct) |
| +- Modify: `internal/host/edgesfile_test.go` |
| +- Modify: `internal/edge/homemap.go` (`staleHopRefusal` ~:545-565) |
| +- Modify: `internal/edge/edge.go` (the backhaul 502 ErrorHandler ~:1962) |
| + |
| +**Interfaces:** |
| +- Produces: the edges-file schema gains `"region":"<aws-region>"` |
| + (omitempty; value = `CARLOS_REGION` env, empty when unset) — Task 4's |
| + converger requires it for latency records. `edgesFileLoop` seeds its |
| + change-gate from the bucket at arm. |
| + |
| +- [ ] **Step 1: failing tests** — (a) seed: pre-populate the fake store |
| + with a prior edges file, run one pass with identical registry state, |
| + assert ZERO writes (today it always writes on first tick); a pass with |
| + a transient blank EIP right after arm keeps the bucket's EIP (the |
| + restart-blip case from the final review). (b) region: the raw wire |
| + body carries `"region":"eu-west-1"` when the env is set, omits it |
| + when unset (extend the existing raw-body pin). (c) visibility: |
| + `staleHopRefusal` logs first-and-every-tenth per (box,host) like the |
| + file's partial-poll pattern; the backhaul ErrorHandler goes through |
| + `logThrottled` keyed `"backhaul-error "+box`. |
| +- [ ] **Step 2: verify failures. Step 3: implement.** Seeding: at loop |
| + arm, Get `edgesFileKey(box)`; if present, unmarshal, recover the EIP |
| + for the hold AND recompute the change-gate body (marshal the fetched |
| + file with Generation cleared) into `last`. Fix the :124 comment (the |
| + heal cycle is this loop + the 30s mirror — the same correction the |
| + docs wave made in homemap.go). Region: read `CARLOS_REGION` once at |
| + arm (the domains loop's read-once precedent), thread into the pass. |
| +- [ ] **Step 4:** `go test ./internal/host/ ./internal/edge/ -count=1` |
| + + `-race`. **Step 5: Commit** |
| + `host+edge: stage-5 gates — seeded edges gate, region field, stuck-state visibility`. |
| + |
| +--- |
| + |
| +### Task 2: The steering field, storage to converge |
| + |
| +**Files:** |
| +- Modify: `internal/provision/provision.go` (Record struct + validation) |
| +- Modify: `internal/registry/registry.go` (4-place column add + |
| + `SetSteering(host, steering string)` narrow UPDATE) |
| +- Modify: `internal/provision/reconcile.go` (`routeFor`, |
| + `shapeMismatch`, `convergeOwned` carry it) |
| +- Test: each package's existing suites + a pre-plan-fixture round-trip |
| + |
| +**Interfaces:** |
| +- Produces: `Record.Steering string json:"steering,omitempty"`; |
| + `ValidSteering(s string) bool` (`""` or `"latency"`); |
| + `Route.Steering` registry column (36th); `SetSteering`. Task 3's door |
| + and Task 4's desired-state builder consume all three. |
| + |
| +- [ ] **Step 1: failing tests** — validation table (`""` ok, `"latency"` |
| + ok, `"Latency"`/`"geo"`/garbage refused); registry round-trip via the |
| + existing `TestUpsertRoundTripsAllColumns` extension; `Fingerprint` |
| + AND `SnapshotFingerprint` change when steering changes (pin, don't |
| + assume routeCols coverage); a record JSON written WITHOUT the field |
| + (pre-plan fixture, verbatim bytes from a current production-shaped |
| + record) parses and round-trips with `Steering == ""`. |
| +- [ ] **Step 2-4: red, implement, green** (4-place rule; shapeMismatch |
| + gains the Steering case so a record edit converges the row; |
| + convergeOwned updates it via SetSteering). **Step 5: Commit** |
| + `provision+registry: the steering field — additive, 4-place, converge-owned`. |
| + |
| +--- |
| + |
| +### Task 3: Setting it — console door + CLI, with the pool rail |
| + |
| +**Files:** |
| +- Modify: `internal/console/apiinstances.go` (PATCH handler — the |
| + existing CAS/`DerivesPaths`-rail shape at ~:649-768 is the idiom; |
| + steering joins the PATCHable set with its own rails) |
| +- Modify: CLI (`instances_cli.go` verbs; follow the set-upstreams verb's |
| + shape for a `steering` verb: `carlos steering -app X -host H latency` |
| + and `... off`) |
| +- Test: console API tests + CLI tests (carlosTestEnv rule) |
| + |
| +**Interfaces:** |
| +- Consumes: `ValidSteering`, `provision.UpdateRecord` CAS, |
| + `s.bumpSerial(ctx)` on success (console-mediated write — serial is |
| + CORRECT here; it is the box-side that never bumps). |
| +- Produces: PATCH `{"steering":"latency"}` / `{"steering":""}`; |
| + refusals: invalid value 400; **Pool != "" 409** with body naming the |
| + v1 constraint ("steering requires the shared pool — pool-scoped |
| + steering is a recorded future"); route-kind refusals NONE (any served |
| + kind may steer — the spec's any-edge-any-app; home-boxed kinds steer |
| + via backhaul). |
| + |
| +- [ ] **Step 1: failing tests** — door: set/clear round-trip + serial |
| + bump; invalid 400; pool 409 (fixture route with Pool set); CAS 409 on |
| + stale etag (existing idiom). CLI: verb round-trip against a fake |
| + console; refusal text surfaced. |
| +- [ ] **Step 2-4: red, implement, green.** **Step 5: Commit** |
| + `console+cli: steering opt-in — member-settable, pool-railed`. |
| + |
| +--- |
| + |
| +### Task 4: The desired state — steered hosts × armed edges |
| + |
| +**Files:** |
| +- Create: `internal/steering/steering.go` + `steering_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: records (steered routes) via `release.Store` listing the |
| + records prefix (the audit doc's enumeration recipe); edges files |
| + (armed set: box, EIP non-empty, region non-empty — an edge without a |
| + region CANNOT receive latency records, log-once and skip); the |
| + deployment's apps domain (`PLATFORM_APPS_DOMAIN` — already box env). |
| +- Produces (pure functions, no AWS here): |
| + |
| +```go |
| +// A desiredRecord is one latency A record: name -> eip, keyed by the |
| +// edge that serves it. SetIdentifier is the box label; Region is the |
| +// edge's AWS region; HealthCheckRef ties it to the edge's check. |
| +type desiredRecord struct { |
| + Name string // fqdn, trailing-dot normalized |
| + EIP string |
| + Box string // SetIdentifier |
| + Region string // AWS region for latency routing |
| +} |
| + |
| +// desiredState folds steered routes × armed edges. The steering |
| +// hostname edge.<apps-domain> is ALWAYS present when any edge is |
| +// armed (custom-domain CNAME target). Non-shared-pool routes are |
| +// excluded defensively (the door refuses them; the bucket may not |
| +// have) with a loud log. |
| +func desiredState(routes []registry.Route, edges []edgesFileView, appsDomain string) []desiredRecord |
| +``` |
| + |
| +- [ ] **Step 1: failing tests** — table-driven: one steered host × two |
| + armed edges = two records with distinct SetIdentifiers/regions; |
| + unsteered host absent; edge with empty EIP or region skipped (logged); |
| + `edge.<apps-domain>` present whenever ≥1 edge armed, absent at zero; |
| + pool-carrying steered route excluded + logged; wildcard registry rows |
| + NEVER produce records (steering is per-concrete-host — a `*.parent` |
| + row with steering set is skipped + logged, spec's per-host DNS rule). |
| +- [ ] **Step 2-4: red, implement, green (pure, no mocks needed).** |
| + **Step 5: Commit** `steering: desired state — steered hosts × armed edges`. |
| + |
| +--- |
| + |
| +### Task 5: The Route53 converger |
| + |
| +**Files:** |
| +- Create: `internal/steering/route53.go` + `route53_test.go` |
| +- Modify: `go.mod` (aws-sdk-go-v2/service/route53) |
| + |
| +**Interfaces:** |
| +- Consumes: `desiredState`; env `CARLOS_STEERING_ZONE_ID` (bare Z-id, |
| + validated like the region-host module's pattern; unset = dormant); |
| + AWS creds via the default chain (instance profile — same as the |
| + edge's route53dns.go, read its client construction). |
| +- Produces: `Converge(ctx, client, zoneID string, want []desiredRecord) error`: |
| + - Lists current carlos-managed records in the zone (managed = name |
| + matches a steered set member OR carries the SetIdentifier scheme; |
| + NEVER touches records without a SetIdentifier — the zone's static |
| + records are out of bounds, stated in code). |
| + - Health checks: one per distinct EIP, HTTPS:443, |
| + FullyQualifiedDomainName = the steering hostname (SNI a cert every |
| + edge holds), IPAddress = the EIP, tagged |
| + `carlos-steering=<deployment-zone-id>`; GC checks no desired record |
| + references. |
| + - Upserts/deletes via ChangeResourceRecordSets batches (latency A |
| + records, TTL 60, HealthCheckId bound); deletes only records whose |
| + (name, SetIdentifier) is managed-but-unwanted. |
| + - Idempotent: a second Converge with identical want = zero API writes |
| + (the test asserts call counts on the fake). |
| +- [ ] **Step 1: failing tests against a fake Route53 API** (an |
| + httptest-backed or interface-faked client — the SDK client is an |
| + interface-shaped dependency; define `route53API` with the four calls |
| + used and fake it; NO network): create-from-empty; add-an-edge; |
| + remove-an-edge (record + orphaned check GC'd); remove-steering |
| + (records gone, shared check retained while another host uses it); |
| + idempotency zero-writes; static-record in zone untouched. |
| +- [ ] **Step 2-4: red, implement, green.** **Step 5: Commit** |
| + `steering: route53 converger — latency records + health checks, bounded to managed set`. |
| + |
| +--- |
| + |
| +### Task 6: The duty — wiring the converger into the housekeeper |
| + |
| +**Files:** |
| +- Modify: `internal/host/agent.go` (a `steeringLoop` beside |
| + `mintMeshKeyLoop` — read that loop first; IDENTICAL shape: fleet-skip, |
| + openDeployStore, 5min+jitter ticks of |
| + `runHousekeeperDuties(housekeeperGate, convergeSteering)`) |
| +- Test: `internal/host/` loop-arming tests (mirror the mesh-key loop's) |
| + |
| +**Interfaces:** |
| +- Consumes: Tasks 4-5; `CARLOS_STEERING_ZONE_ID` (unset → the loop arms, |
| + logs once "steering: zone unset — dormant", duty returns nil |
| + immediately); the registry for routes; the store for edges files. |
| +- Dormancy statement: on today's boxes the env is unset → one log line |
| + at arm, zero API calls, zero writes. With the env set but one box: the |
| + steering hostname + any steered hosts get single-edge latency records |
| + (harmless — same answers as today's static DNS, and the health check |
| + adds rung-2 withdrawal for steered hosts even single-box: state this |
| + in the report as the first REAL behavior). |
| +- [ ] **Step 1: failing tests** — arming independence (no pubkey needed, |
| + fleet-skip, dormant-when-unset logged once); duty invokes Converge |
| + only when held (the housekeeper contract tests' shape). |
| +- [ ] **Step 2-4: red, implement, green + -race.** **Step 5: Commit** |
| + `host: steering converger rides the housekeeper`. |
| + |
| +--- |
| + |
| +### Task 7: IAM + attach guidance |
| + |
| +**Files:** |
| +- Modify: `infra/modules/region-host/variables.tf` + `main.tf` (new |
| + `steering_zone_ids` list var, same Z-id validation as |
| + `route53_zone_ids`; policy: `route53:ChangeResourceRecordSets` + |
| + `route53:ListResourceRecordSets` scoped to those zones, plus |
| + `route53:CreateHealthCheck`/`DeleteHealthCheck`/`UpdateHealthCheck`/ |
| + `ListHealthChecks`/`ChangeTagsForResource`/`ListTagsForResource` |
| + (health checks are not zone-scoped — condition on the |
| + `carlos-steering` tag where the API supports it, and say plainly in a |
| + comment where it does not)) |
| +- Modify: `internal/console/apidomains.go` (attach guidance string: |
| + custom domains gain the CNAME-to-`edge.<apps-domain>` steering option |
| + beside the A-record guidance; apexes keep A-records — the spec's |
| + honesty rule, quoted) |
| +- Test: guidance string test; `tofu validate`-style check is manual |
| + (module has no test harness — state it) |
| + |
| +- [ ] **Step 1-3: implement + guidance test + gate.** **Step 4: Commit** |
| + `infra+console: steering IAM surface + custom-domain guidance`. |
| + |
| +--- |
| + |
| +### Task 8: Bookkeeping + rollout notes |
| + |
| +**Files:** |
| +- Modify: spec (Part 4 rows → implemented-dormant with commits; the |
| + dated housekeeper-not-console DNS-writer amendment; Route53-only |
| + note; the pool-rail v1 note) + this plan's rollout notes. |
| + |
| +- [ ] Fill rollout notes from the ledger + these knowns: arming order |
| + per deployment = tofu apply (steering_zone_ids) → set |
| + `CARLOS_STEERING_ZONE_ID` on the box → verify the housekeeper's |
| + converge log → steer one guinea-pig host → dig from two vantage |
| + points; the stage-5 checklist from plan 2 consumed here (edges files |
| + present per armed box FIRST); the egress ruling request stays open |
| + (#-ref); Paul's applies rule (sessions blocked on tofu apply — |
| + commands ready-to-run); edges-file `role` over-claim note (steering |
| + reads the ARMED-EDGE set, not roles — unaffected, but read-local |
| + still wants the replica role first, restated). |
| +- [ ] Full gate; commit `docs: steering rollout notes`. |
| + |
| +## Rollout notes (filled by Task 8 at execution) |
| + |
| +- (filled at execution) |