amadan
docs: nearest-region spec rev 4, after Astra's round 3
All six round-3 findings verified and accepted, marked "(codex R3-N)". Three are faults in the existing steering converger, and they would bite on arming #169 with or without this spec, because static platform hosts already carry steering "latency": an empty non-terminal shadows the single *.oncarlos.com wildcard for unsteered siblings; one latency record per box is invalid when two boxes share an AWS region; over-limit withdrawals refuse every pass. Three rounds in a row found new faults in DNS steering, so rev 4 stops patching examples. Phase L moves to a follow-on spec with six stated invariants (continuity, validity grouped by AWS region, serviceability, withdrawal, dormancy, authority) and a generated-fleet simulation against an RFC 4592 resolver model. This spec writes no stamp, and arming #169 is gated on that spec. R6 stays binding: sequenced, not dropped. Also: edges-file eligibility is computed from the reconciler's own parsed config (CARLOS_POOLS=,tito publishes false); every non-null guess, including the home fallback, passes the normalized policy predicate; the section-4 summary now says app-derived values need positive eligibility. One sequencing question remains for Paul.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+271
−251
docs/superpowers/specs/2026-10-01-nearest-region-default-design.md+271 −251
diff --git a/docs/superpowers/specs/2026-10-01-nearest-region-default-design.md b/docs/superpowers/specs/2026-10-01-nearest-region-default-design.md| index 9dcda01c6..c29e1ed06 100644 |
| --- a/docs/superpowers/specs/2026-10-01-nearest-region-default-design.md |
| +++ b/docs/superpowers/specs/2026-10-01-nearest-region-default-design.md |
| @@ -1,17 +1,21 @@ |
| # Nearest region by default: each app picks its region at creation |
| **Date:** 2026-10-01 |
| -**Revision:** 3. |
| +**Revision:** 4. |
| - Rev 1 (7fbc9479) went to Astra (GPT-6, via codex), and the findings are in |
| `CODEX-FINDINGS-NEAREST-REGION-R1.md`. |
| - Rev 2 (bd4ed0c4) went back, and the findings are in |
| `CODEX-FINDINGS-NEAREST-REGION-R2.md`. |
| -- Every finding of both rounds was checked against the code and accepted. |
| - Changes are marked "(codex N)" for round 1 and "(codex R2-N)" for round 2, |
| - and §21 records each disposition. |
| -- Rev 3 also folds in Paul's answers to rev 2's questions as rulings R5 to R7. |
| - R6 brings latency routing into scope (§13). |
| +- Rev 3 (8e59f598) went back again, and the findings are in |
| + `CODEX-FINDINGS-NEAREST-REGION-R3.md`. |
| +- Every finding of all three rounds was checked against the code and |
| + accepted. Changes are marked "(codex N)", "(codex R2-N)" and |
| + "(codex R3-N)", and §21 records each disposition. |
| +- Rev 3 folded in Paul's answers to rev 2's questions as rulings R5 to R7. |
| +- **Rev 4 moves latency routing (R6) into a follow-on spec**, because the |
| + existing steering converger is not safe to arm (§13). That spec gets |
| + invariants to meet, and placement ships first. |
| **Status:** DRAFT for review. The rulings in §3 are Paul's and are not |
| reopened here. Everything marked *(decision)* is this spec's and open to |
| @@ -57,7 +61,10 @@ Every create path ends at one door, and that door's default is the home box: |
| And nothing is steered. Latency routing exists (`internal/steering`), but it |
| is armed nowhere, because no deployment sets `CARLOS_STEERING_ZONE_ID` |
| (amadan #169). Every instance host therefore resolves through the |
| -deployment's static DNS. |
| +deployment's static DNS. On the flagship that is one `*.oncarlos.com` record |
| +pointing at the Ireland EIP, served by DNSimple. The Route53 copy of the zone |
| +is staged and is not yet authoritative (carloku-infrastructure `dns.tf` and |
| +`dns53.tf`). |
| What exists to build on: |
| @@ -85,20 +92,29 @@ Goals: |
| - A new app gets a region at creation, preselected from a guess of where its |
| creator is. Its new instances start there unless told otherwise. |
| - A new instance placed outside the home is reached through the nearest edge, |
| - not through the home's (R6, §13). |
| + not through the home's (R6). The follow-on spec for Phase L delivers this |
| + goal, against the invariants in §13. |
| - The guess is generic: any deployment, any set of AWS regions. |
| - The guess can never name a region the create door would refuse, or one no |
| box would mint (codex 1, codex R2-1). |
| - Existing apps, older CLIs and external callers that send nothing behave |
| exactly as today. |
| -**Done** means that new apps get regions, and new non-home instances get |
| -latency routing once #169 is armed (R5). Existing apps keep their region. |
| +**Done** has two stages: |
| + |
| +- **This spec** is done when new apps get regions and new instances follow |
| + them. |
| +- **The work as a whole** (R5, R6) is done when the follow-on spec has made |
| + latency routing safe to arm, and #169 is armed. |
| + |
| +Existing apps keep their region. |
| Relieving Ireland of the instances it already holds is the instance-move |
| work (#175/#331), not this spec. |
| Non-goals, each with a follow-up in §20: |
| +- Arming DNS steering and changing the converger. This is Phase L, the |
| + follow-on spec (§13). |
| - Moving existing instances. |
| - Changing an existing app's region (R5). |
| - A browser instance-create form. |
| @@ -116,7 +132,7 @@ Non-goals, each with a follow-up in §20: |
| | R3 | Guess **timezone first**. Fall back to IP geolocation when the timezone is UTC, `Etc/*` or otherwise unmappable. | §6 | |
| | R4 | The CLI guesses too. `carlos deploy` creating an app sends its timezone, the console resolves it, and the CLI prints one line ("Region: UK · London (from your timezone). Use --region to choose another."). No prompt. `--region` overrides everywhere. `carlos instances create` defaults to the app's region. | §11 | |
| | R5 | **Done = new apps only, plus moves.** Existing apps keep their region. Relieving Ireland comes from instance moves. An owner-only verb to change an existing app's region is out of scope (a follow-up). | §2, §20 | |
| -| R6 | **Latency routing is in scope** for non-home instances, not placement alone. | §13 | |
| +| R6 | **Latency routing is in scope** for non-home instances, not placement alone. | §13: its invariants and gates, delivered by the follow-on spec | |
| | R7 | **The rollback floor is a runbook rule.** It is documented for release notes and the roll recipe, with no outside guard. | §5.7, §17 | |
| | D1 | The app region is stored console-side and set once at creation. The create door fills an empty `req.Region` from it **before** normalization, the allow-list and the armed checks. Existing apps are backfilled to the home, so behaviour does not change. | §5, §7 | |
| | D2 | The guess works in coordinates: timezone to a point, IP to a point through an operator-supplied MaxMind-format `.mmdb` (`CONSOLE_GEOIP_DB`), then the great-circle nearest eligible region. Home when everything fails. | §6 | |
| @@ -137,12 +153,13 @@ finding the great-circle nearest region whose boxes have positively said |
| they can mint an ordinary instance. |
| From then on, an instance create that names no region gets the app's region, |
| -and that value meets every check an explicit one meets. A non-app-derived |
| -value additionally needs that positive eligibility. |
| +and that value meets every check an explicit one meets. An **app-derived** |
| +value additionally needs that positive eligibility, while an explicit one |
| +keeps today's gate (codex R3-6). |
| -A new socket-backed instance outside the home is stamped for latency routing |
| -at mint, exactly as static platform hosts already are. That stamp is inert |
| -until the deployment arms steering (#169). |
| +Latency routing for non-home instances (R6) is specified as invariants |
| +(§13) and delivered by a follow-on spec. Until that lands, this spec writes |
| +nothing to DNS and stamps nothing. |
| Existing rows read `""`, which is what they have always meant. Nothing writes |
| a non-home app region until the operator turns the feature on, and the |
| @@ -338,6 +355,14 @@ additive fields** in `edgesFile` (`internal/host/edgesfile.go:155-176`): |
| - `serves_shared_pool`: `poolServed(cfg.Pools, "")`; |
| - `accepts_inbound`: `cfg.Inbound != InboundNone`. |
| +**Both are computed from the very `provision.ReconcileConfig` the reconcile |
| +pass uses**, which `provisionTickConfig` builds (`internal/host/agent.go:2030`). |
| +They are never re-parsed from the environment (codex R3-3). That parser |
| +drops empty elements, so `CARLOS_POOLS=,tito` is `["tito"]` and does **not** |
| +serve the shared pool. The box publishes `false` for it. A box that should |
| +serve both the shared pool and a named one needs a parser change with its |
| +own tenancy tests, which is not part of this spec. |
| + |
| They follow `hibernation_configured`'s precedent: not `omitempty`, so absent |
| means "unknown" and false means "no". This is the one box-side change in this |
| spec. It is purely additive, and old readers ignore it. |
| @@ -374,8 +399,8 @@ Candidate rules: |
| - **Read failure** (codex R2-2): the listing failing is distinct from "no |
| file publishes a placement". `readEdgesFiles` (`instancesplacement.go:83`) |
| swallows the first today. A new `readEdgesFilesStrict` returns the error. |
| - Candidates built after a failed read are the home alone, and the door |
| - treats the failure as unconfirmed (§7). |
| + Candidates built after a failed read are the home alone, if the policy |
| + admits it (§6.4), and the door treats the failure as unconfirmed (§7). |
| ### 6.2 Points |
| @@ -468,11 +493,20 @@ else IP gives a point → nearest → "ip" |
| else home → "default" |
| ``` |
| -If the home has no point, the result is the home with `default`, whatever the |
| -inputs. There is no guess (`"guess": null`) when: |
| +**The guess invariant** (codex R3-4): every non-null guess, including every |
| +fallback to the home, passes the same normalized policy predicate the door |
| +applies (`instanceRegionAllowed` on the `homeRegionIn` form). The steps run in |
| +this order: |
| -- the policy is pooled or unreadable, or |
| -- the allow-list excludes the home and no nearest exists. |
| +1. Filter candidates by policy (§6.1). |
| +2. Pick the nearest among the filtered candidates that have a point. |
| +3. If the home has no point, or nothing maps, the fallback is the home, **but |
| + only if the home is itself a filtered candidate**. |
| +4. Otherwise the answer is `"guess": null`. |
| + |
| +So a manifest that allows only `uk`, with the home's coordinates missing, |
| +gives `uk` when the hint maps and `null` when it does not, never the home. |
| +There is also no guess when the policy is pooled or unreadable. |
| ### 6.5 Logged, and not logged (codex 8) |
| @@ -554,9 +588,6 @@ refusal before anyone tries (codex 7). |
| *(decision)* **A defaulted home whose region is in its own `replicas` set is |
| refused** (C26). Explicit requests keep today's behaviour. |
| -**The latency stamp** (R6) is set beside the static default at `:936`; §13 |
| -gives the full rule. |
| - |
| **Response.** `apiRecordView` (`apiinstances.go:535`) gains two fields that |
| sit outside the record, like `PrivateStoreBucket`: |
| @@ -593,7 +624,7 @@ and both are stored (§12). |
| | path | file:line | today | change | |
| | --- | --- | --- | --- | |
| -| The door | `handleAPIInstanceCreate`, `apiinstances.go:608`; region at `:772` | `""` means the home | the fill and eligibility (§7); the latency stamp (§13) | |
| +| The door | `handleAPIInstanceCreate`, `apiinstances.go:608`; region at `:772` | `""` means the home | the fill and eligibility (§7) | |
| | `carlos instances create` | `instances_cli.go:322` | sends `--region` or `""` | help text; prints `region_name` (§11) | |
| | `carlos canary` | `canary.go:407` | sends `""` | the door fills the app's region; a new `--region`, with the existing-canary rule (§11) | |
| | `carlos deploy` static host | `mintPlatformHost`, `deploy.go:858` | static, `""` | none | |
| @@ -765,17 +796,18 @@ command. |
| is explicit: **fleet placement, then the request's region, then the app's |
| region, then the home.** A placed app's region is stored, stays dormant, |
| and is reported separately from its placement (§9.3). |
| -- **Pools** (`Manifest.Pool`). There is no default fill, no guess and no |
| - latency stamp, and `app_default` reports `pooled`. An explicit `--region` |
| +- **Pools** (`Manifest.Pool`). There is no default fill and no guess, and |
| + `app_default` reports `pooled`. An explicit `--region` |
| behaves as today. |
| - **Pool-restricted and no-inbound boxes** (codex R2-1) are never eligible |
| for an automatic choice (§6.1). Explicit requests keep today's gate. |
| -- **Replicas (#162).** A `role: replica` create is never filled or stamped. A |
| +- **Replicas (#162).** A `role: replica` create is never filled. A |
| defaulted home inside its own `replicas` set is refused (§7). |
| - **Moves (#175/#331).** A move rewrites one record's `region` at its cut. |
| - **It never touches the app's region**, and it keeps the record's |
| - `Steering` field (§13). |
| -- **Static.** Not filled; it already carries its own latency default. |
| + **It never touches the app's region**. Steering for moved instances is the |
| + follow-on spec's, derived from region (§13.1). |
| +- **Static.** Not filled. Its existing latency default is subject to the |
| + §13.3 gate like everything else steered. |
| - **Private storage (#412).** Unaffected. |
| - **Sidecars.** Unchanged. Their door has no armed gate, so following the app |
| region needs its own design (§20). |
| @@ -784,189 +816,143 @@ command. |
| the home today. This change keeps that and reinterprets no manifest. The |
| README documents it, and rollout audits production manifests first (§17). |
| -## 13. Latency routing (R6) |
| - |
| -### 13.1 What already exists, and why no new subsystem |
| - |
| -The steering converger is built and tested. |
| - |
| -- **Where it runs.** It runs on whichever platform box holds the housekeeper |
| - lease (`internal/host/steering_loop.go:204`). |
| -- **What it reads.** It reads every bucket record with a non-empty |
| - `Steering` field (`steeredRoutesFromRecords`, `steering_loop.go:99`), never |
| - a box's registry. |
| -- **What it builds.** It folds those hosts × armed edges into Route53 latency |
| - A records (`desiredState`, `internal/steering/steering.go:349`). An edge is |
| - armed when it publishes an EIP and an AWS-shaped region (`armedEdges`). |
| -- **What it writes.** It converges the zone to that set (`converge`, |
| - `internal/steering/route53.go:152`), with one tagged health check per EIP. |
| -- **Its guards.** An empty-set refusal, a shrink limit, conflict skipping, and |
| - fail-closed edges reads. |
| - |
| -Steering is **any-edge**. Every armed edge gets a record for every steered |
| -host, and a visitor lands on the nearest healthy edge, which serves locally |
| -or backhauls to the record's home box over the existing hop |
| -(`internal/edge/homemap.go`). The only exclusion is #162's replica mark. |
| - |
| -So latency routing for a non-home instance needs two things, and neither is |
| -a subsystem: |
| - |
| -1. a stamp on its record at mint; |
| -2. the deployment arming steering, which is #169. |
| - |
| -Phase L below is that pair. It is shippable separately and has its own gate. |
| -Placement (everything else in this spec) does not wait for it. |
| - |
| -### 13.2 Which hosts |
| - |
| -**(decision)** At mint, the door sets `rec.Steering = "latency"` when **all** |
| -of these hold: |
| - |
| -| condition | why | |
| -| --- | --- | |
| -| `rec.Region != ""` after normalization | R6 is for non-home instances. A home instance keeps today's DNS. | |
| -| `placed == ""` | Fleet boxes are not steering edges. | |
| -| `rec.Role == ""` | A replica is served under the home's steered host (#162), never its own. | |
| -| `rec.DerivesPaths()` | A socket-backed instance, the class this spec places. | |
| -| `platformNamespace` | `steeredHosts` only writes names under the apps domain (`steering.go` `inAppsZone`). A custom domain steers through its owner's CNAME to `edge.<apps-domain>`. | |
| -| `rec.Pool == ""` | The pool rail: the PATCH door refuses steering on a pooled record (`apiinstances.go:1396`), and the fold excludes it again. | |
| -| the request did not name `steering` | The create body has no steering field today, so this is always true at the door. `carlos steering` is the later override. | |
| - |
| -The rule applies to every new non-home instance, whether the region was |
| -explicit or app-derived, because the record's region is what decides where |
| -the process lives. |
| - |
| -The host steered is **the instance's own host**, the record's `Host` |
| -(`<label>.<app>.<sqid>.<apps-domain>`). There is no separate account host: |
| -under #162 the "account host" is the home record's own host, which this rule |
| -covers when that home is non-home. Its replicas stay excluded by the existing |
| -replica mark. Custom-domain aliases of the instance reach the same edges |
| -through `edge.<apps-domain>`, which the converger mints once armed. |
| - |
| -This makes, on purpose, the trade the static default's comment declined for |
| -home-boxed routes (`apiinstances.go:919-921`): "a home-boxed route reaches a |
| -far edge by backhaul". R6 rules it in for non-home instances. A visitor near |
| -the instance's region lands on that region's edge and is served locally. A |
| -visitor near the home lands on the home's edge and is backhauled to the |
| -instance, which is the path every visitor takes today. |
| - |
| -**A value at mint, never a converge**, exactly like the static default: |
| -nothing re-stamps an existing record. `carlos steering --app x --host y off` |
| -is the escape hatch. |
| - |
| -### 13.3 Written where, and by whom |
| - |
| -| what | written by | where | |
| -| --- | --- | --- | |
| -| `steering: "latency"` on the record | the console's create door, at mint | `apps/<sqid>/<app>/instances/<host>.json` in the deployment bucket | |
| -| one latency A record per armed edge for the host, `SetIdentifier` = box label, `Region` = the edge's AWS region | the housekeeper box's steering duty | the zone `CARLOS_STEERING_ZONE_ID` | |
| -| one health check per EIP, tagged `carlos-steering=<zone>` | the same duty | Route53, shared by every steered name | |
| -| `edge.<apps-domain>` latency records | the same duty, once armed | the same zone | |
| - |
| -Records appear on the next pass (5 minutes plus jitter) and then reach |
| -resolvers on the record TTL. Until then the host resolves exactly as it does |
| -today, through the deployment's static DNS, and is served by backhaul. Being |
| -late is never an outage. |
| - |
| -### 13.4 Removal |
| - |
| -- **Delete.** The record leaves the bucket, so the host leaves the desired |
| - set, and the next pass DELETEs its managed records. The shrink limit |
| - (`route53.go:288`) may spread a mass delete over passes. Health checks are |
| - per EIP and are collected only when nothing references them. |
| -- **Move.** A move changes `region` and keeps `Steering`. Because steering is |
| - any-edge, the record set does not change, and visitors keep landing on the |
| - nearest edge, which now backhauls to the new home. A move *to* the home |
| - leaves the stamp, which is harmless. A move *from* the home does not add |
| - one: moves are #175's code, and the follow-up is in §20. Until then the |
| - move runbook ends with `carlos steering --app <a> --host <h> latency`. |
| -- **Opt-out.** `carlos steering ... off` clears the field, and the next pass |
| - removes the records. |
| - |
| -### 13.5 When steering is not armed |
| - |
| -The stamp is inert, as it is today for every static platform host: |
| - |
| -- the duty logs that the zone is unset and it is dormant, reads nothing, and |
| - calls no AWS API |
| - (`steering_loop.go:219`); |
| -- no request fails; |
| -- placement works. |
| - |
| -The deployment is then **placement-only**: a UK instance is reached through |
| -whatever the static DNS names, and backhauled. Nothing in this spec returns |
| -an error because steering is unarmed. |
| - |
| -### 13.6 The infrastructure prerequisite (#169) |
| - |
| -Arming steering is production infrastructure work. It needs scoped approval |
| -under AGENTS.md before anyone applies it. The plan to approve: |
| - |
| -1. **Read first.** |
| - - Which hosted zone holds the flagship's apps domain. |
| - - What the apps-domain wildcard resolves to today. |
| - - How many bucket records already carry `steering: "latency"`. Every |
| - static platform host minted since the static-HA default does, and |
| - arming steers them all. |
| - - Whether any steered name collides with an existing record in the zone. |
| - The converger skips and logs conflicted names, so check before, not |
| - after. |
| -2. **`carlosframework/carloku-infrastructure`, root module.** Pass the zone |
| - to every platform box's `region-host` module: |
| - - `steering_zone_ids = [<apps zone id>]`; |
| - - `steering_record_names = ["*.<apps-domain>"]`. |
| - |
| - This grants the IAM policy at `infra/modules/region-host/main.tf:281`. |
| - Without the name bound, the role can write any A record in the zone, the |
| - apex included (`variables.tf:153`). |
| -3. **Each platform box's `/etc/carlos/host.env`** (user data in the same |
| - repo). Set `CARLOS_STEERING_ZONE_ID=<zone id>` and confirm |
| - `PLATFORM_APPS_DOMAIN`. Restart the agent, because the loop reads the env |
| - once at arm. |
| -4. **Cost.** One Route53 health check per EIP, plus latency-record queries. |
| - The approval request states both. |
| -5. **Cleanup and disarm.** Unsetting the env makes the duty dormant but |
| - leaves the records it wrote. A clean disarm deletes the managed latency |
| - records and the health checks tagged `carlos-steering=<zone>` by hand. |
| - That is the cleanup step in the approval request. |
| -6. **Verification**, which is §13.7. |
| - |
| -### 13.7 Measuring latency, before and after |
| - |
| -**Test instances.** A test app with one instance in each armed region, plus |
| -one at the home, created through the API. Creating instances is ordinary API |
| -use, not infrastructure. |
| - |
| -**DNS answers**, read-only, from anywhere. Route53 honours EDNS client |
| -subnet, so this works without being in each region: |
| - |
| -``` |
| -dig +short +subnet=<client prefix>/24 @<zone nameserver> <instance host> A |
| -``` |
| - |
| -Run it for a UK, an Irish and an Australian client prefix. Before arming, the |
| -answer is the static DNS answer for every prefix. After, each prefix gets its |
| -own region's edge EIP. |
| - |
| -**The request path**, from a vantage in each region: each platform box, run |
| -read-only as platform work, and the operator's own machine. |
| - |
| -``` |
| -curl -s -o /dev/null -w '%{time_connect} %{time_appconnect} %{time_starttransfer}\n' https://<instance host>/ |
| -``` |
| - |
| -Run 30 requests per host per vantage and record the median and the p90 of |
| -connect and TTFB. Take a baseline after Phase P and before arming, and |
| -measure again after arming, with the same hosts and vantages. |
| - |
| -**The result to record** in #169 and in this spec's discussion, as numbers |
| -rather than a claim: |
| - |
| -- A UK vantage's connect time to a UK instance should fall to roughly the |
| - in-region round trip. |
| -- Its TTFB should lose the home backhaul. |
| -- A home vantage's numbers to the same instance should be unchanged within |
| - noise. |
| +## 13. Latency routing (R6): the requirements Phase L must meet |
| + |
| +### 13.1 Why this is now a separate spec (codex R3-1, R3-2, R3-5) |
| + |
| +Rev 3 treated latency routing as two small things: a stamp on each non-home |
| +record at mint, and arming the existing converger (#169). Round 3 showed that |
| +the converger itself is not ready to arm. Each of these three faults was |
| +checked against the code: |
| + |
| +- **Arming can break DNS for names nobody steered** (codex R3-1). The flagship |
| + answers every tenant name with one `*.oncarlos.com` record pointing at the |
| + Ireland EIP (carloku-infrastructure `dns53.tf` and `dns.tf`; the platform |
| + README documents the same single wildcard at `README.md:442`). The converger |
| + writes A records at exact names (`internal/steering/steering.go:369`, `:386`). |
| + One latency record at `new.bap.oncarlos.com` creates the empty non-terminal |
| + `bap.oncarlos.com`. From then on, `old.bap.oncarlos.com` finds `bap` as its |
| + closest encloser, looks for `*.bap.oncarlos.com`, which does not exist, and |
| + no longer matches `*.oncarlos.com` (RFC 4592 §3.3.1). Every unsteered |
| + sibling under that account stops resolving. Static platform hosts have been |
| + stamped `latency` at mint for weeks (`apiinstances.go:936`), so **arming #169 |
| + as the code stands would do this to every account with a static site.** It |
| + would happen with or without this spec. |
| +- **Two edges in one AWS region produce an invalid record set** (codex R3-2). |
| + `desiredRRSet` keys each record by box (`SetIdentifier`) but stamps it with |
| + the edge's AWS region (`internal/steering/route53.go:597`). Route53 allows |
| + one latency record per region within a name and type. A second London box |
| + makes every batch fail, on every pass. This spec allows several placement |
| + codes in one AWS region (§6.1), so the topology is reachable. |
| +- **Large withdrawals never complete** (codex R3-5). When a pass would delete |
| + more than the shrink limit, it refuses **every** change and returns |
| + (`route53.go:207`). The loop's own notes say this repeats forever until an |
| + operator steps in (`internal/host/steering_loop.go:154`). Rev 3's claim that |
| + deletes "spread over passes" was wrong. |
| + |
| +Behind these sits a fourth fact. The flagship's Route53 zone for |
| +`oncarlos.com` is marked "STAGED, NOT YET AUTHORITATIVE" in `dns53.tf`, and |
| +DNSimple still serves the live answers (`dns.tf`). A converger writing to a |
| +zone nobody consults is inert at best. |
| + |
| +Three rounds have each found a new fault in this one area. Per the lesson this |
| +repo has already paid for, the answer is not another patch to an example. It |
| +is an invariant, a design built to it, and a simulation that checks it. |
| +That is a spec of its own, about the converger. It is not a section of a |
| +spec about choosing regions. |
| + |
| +**(decision) Phase L moves to a follow-on spec, and placement ships without |
| +waiting for it.** R6 is unchanged: latency routing for non-home instances is |
| +still in scope for this work and still binding. It is sequenced, not dropped. |
| +This spec now does three things for R6: |
| + |
| +1. It **writes no stamp.** Rev 3's mint-time `Steering: "latency"` is |
| + withdrawn. Every stamped record would join the set of names whose DNS |
| + changes when the zone is armed, and that set is exactly the danger above. |
| + The follow-on spec should derive "steer this host" from the record's |
| + region, not from a stamp: any socket-backed home record with a non-home |
| + region, in the shared pool and the platform namespace. That covers |
| + instances created before it lands, and instances moved out of the home, |
| + with no backfill and no change to the move code. |
| +2. It **states the invariants** that spec must meet (§13.2). |
| +3. It **gates arming #169** on that spec (§13.3). |
| + |
| +### 13.2 Invariants for the follow-on spec |
| + |
| +The DNS model is the authoritative zone for the apps domain, the set of armed |
| +edges with their AWS regions and health, and the bucket records. Over that |
| +model, these must hold after every converge pass, armed or not: |
| + |
| +- **I1, continuity.** Arming, and every later pass, leaves the answer |
| + unchanged for every name that has no steered record set of its own. This |
| + includes names that sit beneath an empty non-terminal a steered name |
| + creates, nested canaries, names whose steering was turned off, and names |
| + whose record was deleted. "The answer" is the result of RFC 4592 |
| + resolution over the whole zone, not an exact-name lookup. |
| +- **I2, validity.** For every steered name, the zone holds at most one latency |
| + record per AWS region. The record set is a function of the set of eligible |
| + edges serving that name, **grouped by AWS region**. Adding, removing or |
| + renaming a box changes the answers inside a region's group. It never |
| + changes how many latency records the name has for that region. One way to |
| + get this is one latency record per region, aliasing a per-region, |
| + health-checked set of that region's edges. The spec chooses. |
| + |
| + This groups by region rather than "keyed by edge" because Route53's rule is |
| + per region. |
| +- **I3, serviceability.** Every answer for a steered name is an edge that can |
| + serve it: healthy, holding the account's certificate, and able to reach the |
| + name's home through the home map. The far edge backhauls; it never |
| + answers 404. |
| +- **I4, withdrawal.** A withdrawal justified by a complete read (a record |
| + deleted, steering turned off, a box retired) converges within a bounded |
| + number of passes. If it cannot, it raises a named operator alert with a |
| + documented procedure. The short-read guard still refuses deletions that a |
| + complete read does not justify, but a refusal of deletions never blocks |
| + unrelated additions in the same pass. |
| +- **I5, dormancy.** With no zone configured, nothing is read and nothing is |
| + written. This is already true (`steering_loop.go:219`). |
| +- **I6, authority.** The converger only writes to the zone the apps domain is |
| + actually delegated to. It verifies the delegation before its first write, |
| + and stays dormant, loudly, when it cannot. |
| + |
| +**The proof the follow-on spec must ship:** a property test over generated |
| +fleets. It covers random edges, random AWS regions including duplicates, |
| +random steered and unsteered names including nested and sibling names, and |
| +random sequences of creates, deletes, opt-outs, moves and box retirements. |
| + |
| +Each scenario runs through `desiredState` and `converge` against a Route53 |
| +fake. The fake enforces the per-region latency rule, batch atomicity, and an |
| +RFC 4592 resolver. The test asserts I1 to I5 after every step, at high seed |
| +counts, and each invariant's assertion is checked by mutation: break the |
| +code, and watch the test fail. Examples alone have now missed three faults |
| +in a row. |
| + |
| +### 13.3 Gates |
| + |
| +- **Do not arm #169** (`CARLOS_STEERING_ZONE_ID`) on any deployment until the |
| + follow-on spec has landed and its simulation passes. This holds for the |
| + static-site steering that already exists, independently of this spec. |
| +- The follow-on spec carries the infrastructure plan for scoped approval |
| + (AGENTS.md): |
| + - the zone and its delegation; |
| + - `steering_zone_ids` and `steering_record_names = ["*.<apps-domain>"]` on |
| + every platform box's `region-host` module |
| + (`infra/modules/region-host/main.tf:281`, `variables.tf:153`); |
| + - `CARLOS_STEERING_ZONE_ID` in each box's `host.env`; |
| + - cost, cleanup and disarm. |
| +- It also carries the latency measurement. Rev 3's method stands and moves |
| + there: `dig +subnet` against the zone's nameservers, then |
| + `curl -w '%{time_connect} %{time_appconnect} %{time_starttransfer}'` from a |
| + vantage in each region, 30 runs per host, with medians and p90. Take a |
| + baseline after this spec's placement ships and before arming, measure again |
| + after, and record the numbers in #169. |
| + |
| +Until then, a deployment that runs this spec is **placement-only**. A UK |
| +instance is reached through the deployment's static DNS (on the flagship, the |
| +Ireland EIP) and backhauled to the UK box. That costs the visitor about one |
| +Ireland–London round trip compared with landing in London. It is no worse |
| +than today for anyone, and it relieves Ireland's disk for every new non-home |
| +instance. |
| ## 14. Place names |
| @@ -1005,9 +991,8 @@ than the picker, but it must agree geographically, and the README says so. |
| | manifest unreadable | The guess endpoint returns 502 (C27). `/regions` answers `policy-unreadable`. | |
| | switch on, home unnamed | The switch is treated as off; logged once. | |
| | console binary rolled back below the floor | R7: no-region creates land at the home; roll-forward lists possible discrepancies. | |
| -| steering unarmed | Placement-only, with no error (§13.5). | |
| -| steering armed, one edge's region not AWS-shaped | That edge gets no records; logged by `armedEdges`. | |
| -| a steered name collides with an existing zone record | The converger skips the name and logs it; the host keeps its static DNS. | |
| +| steering unarmed | Placement-only, with no error. A non-home instance is reached through the static DNS and backhauled (§13.3). | |
| +| a manifest's allow-list excludes the home, and no hint maps | `"guess": null`, never the home (§6.4). | |
| | owner adopts an existing app | `""`/`default`, unless `region` is explicit. | |
| ## 16. Test plan |
| @@ -1040,8 +1025,13 @@ than the picker, but it must agree geographically, and the README says so. |
| `CARLOS_POOLS` and `CARLOS_INBOUND`, for: |
| - unset; |
| - `CARLOS_POOLS=tito`; |
| - - `CARLOS_POOLS=,tito`, which includes the shared pool; |
| + - `CARLOS_POOLS=,tito`, which the parser reads as `["tito"]`, so it |
| + publishes `serves_shared_pool: false` (codex R3-3); |
| - `CARLOS_INBOUND=none`. |
| +- The published fields are asserted equal to `poolServed` and |
| + `cfg.Inbound != InboundNone` over the same `provisionTickConfig` output, for |
| + a table of `CARLOS_POOLS` and `CARLOS_INBOUND` spellings. That makes |
| + publication and reconciliation one predicate. |
| - The reader treats absent as unknown and false as no. |
| - Eligibility needs all three fields true on one reachable box. |
| - The guess against a named-pool-only box returns no guess for that code. |
| @@ -1076,21 +1066,27 @@ than the picker, but it must agree geographically, and the README says so. |
| explicit create is unchanged. |
| - The switch off with an existing `uk` row: the fill still applies. |
| -**Latency stamp** (R6): |
| +**No stamp** (§13.1). A non-home instance created by this spec carries no |
| +`Steering` field. The record is byte-identical to one created by |
| +`instances create --region` before this spec. |
| + |
| +**The guess invariant** (codex R3-4). A property test over generated |
| +policies, candidate sets, missing coordinates, unmappable hints and |
| +edges-read failures asserts two things: |
| + |
| +- every non-null guess passes `instanceRegionAllowed(m.Regions, |
| + homeRegionIn(code))`; |
| +- a non-null guess exists whenever some filtered candidate is reachable by |
| + the order in §6.4. |
| -- A non-home, socket-backed, platform-namespace, shared-pool home record is |
| - stamped `latency`. This holds whether its region was explicit or |
| - app-derived. |
| -- Not stamped: a home-region record, a fleet-placed app, a replica, a pooled |
| - manifest, a custom-domain manifest, and an addr-backed or static record. |
| - Static keeps its own default. |
| -- `steeredRoutesFromRecords` picks the stamped record up. |
| -- `desiredState` builds one record per armed edge for its host. |
| -- Deleting the record removes the host from the desired set. |
| -- A move keeps the field. |
| +Named cases: |
| -**Pools, end to end.** A pooled manifest gives `guess: null`, |
| -`app_default.reason: pooled`, and no stamp. |
| +- an allow-list of only `uk`, with the home's coordinates missing; |
| +- an unmappable hint against the same list; |
| +- an edges-read failure against the same list, which gives `null`. |
| + |
| +**Pools, end to end.** A pooled manifest gives `guess: null` and |
| +`app_default.reason: pooled`. |
| **App create.** |
| @@ -1168,13 +1164,11 @@ and `ip` appear in none of: |
| to every box, then check every region's files report all three fields. |
| Until then, no coded region is eligible (§6.1). |
| 3. **Console release, switch off.** It carries the migration, `internal/geo`, |
| - the fill, the stamp, the endpoints and the UI. Behaviour is unchanged |
| - (the stamp only fires on non-home records, which only explicit `--region` |
| - creates today, and it is inert while steering is unarmed). It can be |
| + the fill, the endpoints and the UI. Behaviour is unchanged, and it can be |
| rolled back freely. |
| 4. **Verify on dev.** Turn the switch on with two eligible regions. Then: |
| - create an app in the browser, and read its row; |
| - - create an instance with no region, and read its record and its stamp; |
| + - create an instance with no region, and read its record; |
| - diff `/regions` for an old app against the pre-roll answer. |
| 5. **Flagship.** Roll the console, then turn the switch on. **From here the |
| binary rollback floor applies (R7).** That release's notes and the roll |
| @@ -1183,21 +1177,26 @@ and `ip` appear in none of: |
| 6. **CLI release.** It is safe in either order (§9.6). |
| 7. **titogo**, in its own repo, adopts `/regions/guess` and the place names. |
| -**Phase L: latency (R6)**, gated on Phase P step 3 and on scoped approval: |
| - |
| -1. Take the §13.7 baseline on the flagship. |
| -2. Bring the §13.6 plan for approval, then apply it in carloku-infrastructure. |
| -3. Watch one converge pass: the records it wrote, and the conflicts it |
| - logged. |
| -4. Measure again (§13.7), and record the numbers in #169. If the numbers |
| - regress, disarm per §13.6 step 5. Placement is unaffected either way. |
| +**Phase L: latency (R6)** is the follow-on spec's rollout, not this one's |
| +(§13.3). Its gate is that spec landing with its simulation passing, plus |
| +scoped approval for the infrastructure. Until then, **no deployment arms |
| +`CARLOS_STEERING_ZONE_ID`**, including for the static-site steering that |
| +already exists. Take the latency baseline once Phase P step 5 has run. |
| Copy review (§19) happens before Phase P step 3 ships any string. |
| ## 18. Questions for Paul |
| -None. R5 to R7 settled rev 2's three questions, and every other open item has |
| -a decision in the body or a follow-up in §20. |
| +1. **Sequencing R6.** Do you agree that placement (Phase P) ships on this |
| + spec, and latency routing (Phase L) follows on a spec of its own, built to |
| + §13.2's invariants? |
| + - The alternative is to hold placement until the converger is fixed. |
| + Ireland's disk is the incident, and placement relieves it for new |
| + instances without touching DNS. |
| + - **Recommendation: split.** R6 stays binding on the follow-on spec. It is |
| + sequenced, not dropped. |
| + |
| + This is a sequencing call under R6, which is why it is yours. |
| ## 19. Copy (pending copy review) |
| @@ -1268,9 +1267,11 @@ The `docs/site/cli.md`, README and release-note additions (§5.7, §11, §12, |
| - **Changing an existing app's region** (R5). An owner-only verb that affects |
| future creates only and leaves existing instances alone. |
| -- **Moves stamping steering.** A cut from the home into a non-home region |
| - should set `Steering: latency` under §13.2's rule. Until then, the move |
| - runbook adds `carlos steering … latency`. |
| +- **Phase L: latency routing (R6)**, on its own spec, meeting §13.2's |
| + invariants with a simulation over generated fleets. It derives "steer this |
| + host" from the record's region, which also covers moved instances. It |
| + carries #169's infrastructure plan, the delegation check for the flagship's |
| + `oncarlos.com` zone, and the latency measurement. |
| - **Pool-aware candidates.** Pool eligibility proven by the target box itself. |
| - **Sidecars following the app region**, with an armed gate on their door. |
| - **A browser instance-create form.** |
| @@ -1307,7 +1308,7 @@ The `docs/site/cli.md`, README and release-note additions (§5.7, §11, §12, |
| | rev 1 Q | disposition | |
| | --- | --- | |
| | Q1 titogo adoption in "done" | superseded by R5: done is new apps plus moves | |
| -| Q2 placement-only, measure first | superseded by R6: latency is in scope. Measurement is kept (§13.7). | |
| +| Q2 placement-only, measure first | superseded by R6: latency is in scope. Measurement moves to the follow-on spec (§13.3). | |
| | Q3 refuse an unavailable default | adopted (§7) | |
| | Q4 creation-only region | adopted; the change verb is a follow-up (R5, §20) | |
| | Q5 defer the browser instance form | adopted (§20) | |
| @@ -1328,5 +1329,24 @@ The `docs/site/cli.md`, README and release-note additions (§5.7, §11, §12, |
| | rev 2 Q | disposition | |
| | --- | --- | |
| | Q1 (b): titogo in "done" | superseded by R5 | |
| -| Q2 (a): placement-only, measure first | superseded by R6. Measurement is kept as Phase L's gate (§13.7, §17). | |
| +| Q2 (a): placement-only, measure first | superseded by R6. Measurement moves with Phase L to the follow-on spec (§13.3). | |
| | Q3 (a): accept the floor | matches R7 | |
| + |
| +### Round 3 (rev 3, 8e59f598) |
| + |
| +| # | sev | finding | disposition | |
| +| --- | --- | --- | --- | |
| +| R3-1 | P1 | Arming per-host steering breaks DNS for unsteered siblings, through empty non-terminals shadowing `*.<apps-domain>` | **accepted.** Verified against `steering.go:369`, `:386` and carloku's single wildcard (`dns53.tf`, `dns.tf`). It already affects static-site steering. Resolved by invariant I1, the gate on arming #169, and the move of Phase L to a follow-on spec (§13). | |
| +| R3-2 | P1 | One latency record per edge is invalid when two edges share an AWS region | **accepted.** Verified at `route53.go:597`. Resolved by invariant I2, which groups by AWS region, and a Route53 fake that enforces the rule (§13.2). This departs from the coordinator's example wording "keyed by edge, not region", because Route53's constraint is per region. | |
| +| R3-3 | P1 | The `CARLOS_POOLS=,tito` test would publish shared eligibility the reconciler denies | **accepted.** Verified at `agent.go:2033`. Publication is computed from the same `ReconcileConfig` and predicate, `,tito` publishes false, and a test pins the equality (§6.1, §16). | |
| +| R3-4 | P2 | Home fallback is not constrained by the allow-list | **accepted.** The guess invariant, with the home fallback only when the home passes the policy, and a property test (§6.4, §16). | |
| +| R3-5 | P2 | Large DNS withdrawals are refused indefinitely, not spread over passes | **accepted.** Verified at `route53.go:207` and `steering_loop.go:154`. The wrong claim is removed, and invariant I4 makes it the follow-on spec's requirement (§13). | |
| +| R3-6 | P3 | §4 reverses which requests need positive eligibility | **accepted.** Corrected to app-derived (§4). | |
| + |
| +### Astra's round-3 recommendations |
| + |
| +| former question | disposition | |
| +| --- | --- | |
| +| Q1: apply R5, titogo separate | adopted (§2, §17 step 7) | |
| +| Q2: implement R6 with the DNS corrections, keep the separate approved Phase L rollout and measurements | adopted in substance. The corrections are larger than a patch, so they move to the follow-on spec with invariants (§13); the sequencing is §18 Q1. | |
| +| Q3: name the exact floor version before activation; keep region-aware reads when disabling | adopted (§5.6, §5.7, §17 step 5) | |