amadan
docs: nearest-region spec rev 6, R9: placed instances need the secrets key
Production finding (Paul approved the amendment 2026-10-02): only the Ireland box holds CARLOS_SECRETS_KEY, so instances placed on the UK, US and AU boxes ran with no config, secrets or store binding (part of the kpmg outage). Boxes publish secrets_key_fp, the move-any-app spec's field, in a format fixed here for both (sha256 of the public key's SEC1 bytes, one appconfig helper). The console expects the fingerprint of its own CARLOS_SECRETS_PUBKEY. Automatic placement needs a match; explicit socket-backed and replica creates are refused (C31) once a current writer proves the mismatch, and keep today's behaviour while unknown. Rollout gains a key-staging step. Changes marked "(R9)".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+139
−28
docs/superpowers/specs/2026-10-01-nearest-region-default-design.md+139 −28
diff --git a/docs/superpowers/specs/2026-10-01-nearest-region-default-design.md b/docs/superpowers/specs/2026-10-01-nearest-region-default-design.md| index 3ba95f956..30ed55727 100644 |
| --- a/docs/superpowers/specs/2026-10-01-nearest-region-default-design.md |
| +++ b/docs/superpowers/specs/2026-10-01-nearest-region-default-design.md |
| @@ -1,7 +1,7 @@ |
| # Nearest region by default: each app picks its region at creation |
| **Date:** 2026-10-01 |
| -**Revision:** 5. |
| +**Revision:** 6. |
| - Rev 1 (7fbc9479) went to Astra (GPT-6, via codex), and the findings are in |
| `CODEX-FINDINGS-NEAREST-REGION-R1.md`. |
| @@ -18,11 +18,19 @@ |
| invariants to meet, and placement ships first. |
| - Rev 4.1 records Paul's ruling on that split as R8 and removes the question. |
| - Rev 5 answers Astra's round 4 (`CODEX-FINDINGS-NEAREST-REGION-R4.md`): three |
| - P2s, marked "(codex R4-N)". |
| - |
| -**Status:** DRAFT for review. The rulings in §3 are Paul's and are not |
| -reopened here. Everything marked *(decision)* is this spec's and open to |
| -review. |
| + P2s, marked "(codex R4-N)". Astra's round 5 found nothing new; Paul |
| + approved rev 5 on 2026-10-02. |
| +- **Rev 6 is an amendment Paul approved on 2026-10-02, ruling R9**, after a |
| + production finding: only the Ireland box holds `CARLOS_SECRETS_KEY`. On the |
| + UK, US and AU boxes the config loops log "config sync loop disabled: |
| + CARLOS_SECRETS_KEY not set" (`internal/host/agent.go:881-886`), and a wake |
| + treats a missing `<host>.config.env` as empty |
| + (`internal/activator/backend_exec.go:1146-1149`). An instance placed there |
| + runs with no config, no secrets and no store binding. That was part of the |
| + kpmg outage. Changes are marked "(R9)". |
| + |
| +**Status:** Approved by Paul on 2026-10-02: rev 5, plus the R9 amendment in |
| +this revision. The rulings in §3 are Paul's and are not reopened here. |
| **Discussion:** not yet filed on amadan. Latency routing depends on amadan |
| #169. |
| **Builds on:** |
| @@ -138,6 +146,7 @@ Non-goals, each with a follow-up in §20: |
| | R6 | **Latency routing is in scope** for non-home instances, not placement alone. | §13: its invariants and gates, delivered by the follow-on spec | |
| | R7 | **The rollback floor is a runbook rule.** It is documented for release notes and the roll recipe, with no outside guard. | §5.7, §17 | |
| | R8 | **Split** (Paul, 2026-10-01). Placement ships on this spec. Latency routing (R6) is sequenced to its own follow-on spec, built to the §13 invariants. | §13, §17 | |
| +| R9 | **A box must hold the deployment's secrets key to receive placed instances** (Paul, 2026-10-02). | §6.1, §7, §8.1, §17 | |
| | D1 | The app region is stored console-side and set once at creation. The create door fills an empty `req.Region` from it **before** normalization, the allow-list and the armed checks. Existing apps are backfilled to the home, so behaviour does not change. | §5, §7 | |
| | D2 | The guess works in coordinates: timezone to a point, IP to a point through an operator-supplied MaxMind-format `.mmdb` (`CONSOLE_GEOIP_DB`), then the great-circle nearest eligible region. Home when everything fails. | §6 | |
| | D3 | Surfaces: a new-app picker; names and the guess for external pickers; the CLI as R4. | §9, §10, §11 | |
| @@ -381,6 +390,46 @@ an automatic one. |
| Until boxes roll with the fields, no coded region is eligible. The guess then |
| picks only the home, so the feature cannot outrun the box roll. |
| +**The secrets key (R9).** An exec instance's config, secrets and store |
| +binding reach it only through `<host>.config.env`, which the agent renders |
| +only when the box holds `CARLOS_SECRETS_KEY` (`agent.go:881-886`, `:1812`). |
| +A wake treats a missing file as empty (`backend_exec.go:1146-1149`), so a box |
| +without the key runs a placed instance with nothing, silently. Eligibility |
| +therefore gains a fourth fact. |
| + |
| +- **What a box publishes.** One more edges-file field, `secrets_key_fp`: the |
| + fingerprint of the public half of the box's `CARLOS_SECRETS_KEY`, never the |
| + key. The name, its place beside `placement` and `data_bucket`, and its |
| + shape (additive, `omitempty`, absent when the box holds no key or a key |
| + that does not parse) are the move-any-app spec's |
| + (`2026-10-02-move-any-app-design.md` §6, on `design-moves-any-region` |
| + at e41c4fef). Both features read one field. |
| +- **The format**, which that spec names but does not fix, is fixed here for |
| + both: `"sha256:"` followed by the lowercase hex SHA-256 of the public key's |
| + uncompressed SEC1 bytes (`(*ecdh.PublicKey).Bytes()`, 65 bytes for P-256). |
| + Those are exactly the bytes `CARLOS_SECRETS_PUBKEY` base64-decodes to, so a |
| + box (from its private key) and the console (from the public key) compute the |
| + same string. One helper, `appconfig.KeyFingerprint`, is the only code that |
| + renders it. The move-any-app spec should cite it rather than define a |
| + second one. |
| +- **What the console expects.** The console learns the deployment's |
| + fingerprint from its own `CARLOS_SECRETS_PUBKEY` (`Config.SecretsPub`, |
| + `cmd/console/main.go:467-490`). That is the key the CLI seals config to, |
| + so it is by definition the one a box must be able to open. The move-any-app |
| + spec's console makes the same choice. The console never learns the |
| + fingerprint from a box: a "majority of boxes" or "the home box" rule would |
| + certify whatever key happened to be widespread, including a wrong one. |
| + With `CARLOS_SECRETS_PUBKEY` unset, the console cannot judge. It logs that |
| + once at startup, and no coded region is eligible for automatic placement. |
| +- **The rule.** A code is eligible only if at least one box that meets the |
| + other three facts also publishes a `secrets_key_fp` equal to the |
| + console's. The home stays exempt, as it is from every eligibility fact: |
| + it is where every legacy create already lands, and on the flagship it is |
| + the one box that holds the key. |
| +- **The reason word.** A code that fails only this fact reports |
| + `no-secrets-key`, between `not-hosting` and `not-eligible` in the order of |
| + checks (§9.3, C16). |
| + |
| Candidate rules: |
| - **Coded regions**: boxes whose `Placement` passes `provision.ValidRegion`. |
| @@ -585,8 +634,8 @@ strictly (§6.1) and refuses in three cases: |
| - **The read failed**: a 503 with C28. The region could not be confirmed, |
| which is different from "absent". |
| -- **No eligible box publishes the code**: C16, naming `absent`, `not-hosting` |
| - or `not-eligible`. This covers the last box in a region being retired, and |
| +- **No eligible box publishes the code**: C16, naming `absent`, `not-hosting`, |
| + `no-secrets-key` (R9) or `not-eligible`. This covers the last box in a region being retired, and |
| every box losing its placement code. |
| - **The allow-list excludes it**: C17. |
| @@ -600,6 +649,28 @@ refusal before anyone tries (codex 7). |
| *(decision)* **A defaulted home whose region is in its own `replicas` set is |
| refused** (C26). Explicit requests keep today's behaviour. |
| +*(decision, R9)* **An explicit request into a region whose boxes provably |
| +lack the deployment's secrets key is refused** (C31, 400), for a |
| +socket-backed instance (`DerivesPaths`) and for a replica. Static and |
| +addr-backed records need no config on the box and are unaffected. |
| + |
| +- **What "provably" means.** The console's `CARLOS_SECRETS_PUBKEY` is set, |
| + and every reachable, hosting box publishing the code is a **current |
| + writer** (it publishes `accepts_inbound`, which the same box release adds |
| + and which is never omitted). Among those boxes, none publishes a matching |
| + `secrets_key_fp`. A current writer that publishes no fingerprint holds no |
| + key, because the field is omitted exactly then. |
| +- **Unknown keeps today's behaviour.** An old writer, or a console with no |
| + `CARLOS_SECRETS_PUBKEY`, cannot prove a mismatch, so the request proceeds |
| + as it would today. Roll order therefore never refuses a working region. |
| +- **Why refuse, and not keep today's behaviour.** The record this would |
| + write is an instance that starts, answers health checks, and runs with no |
| + config, no secrets and no store, which is the kpmg outage. It belongs with |
| + the existing "reachable but cannot host" refusal (`apiinstances.go:827-852`, |
| + #110). That one also refuses an explicit request the box would accept but |
| + could never serve. A person who named the region learns why at the door, |
| + not from a broken app. |
| + |
| **Response.** `apiRecordView` (`apiinstances.go:535`) gains two fields that |
| sit outside the record, like `PrivateStoreBucket`: |
| @@ -626,9 +697,11 @@ instance door validates one: |
| 1. `ValidRegion`. |
| 2. Armed and hosting, unless the gate is dormant. |
| -3. The home's name is accepted through `homeRegionIn`. |
| +3. Not provably missing the secrets key (R9, the same rule as §7), refused |
| + with C31. |
| +4. The home's name is accepted through `homeRegionIn`. |
| -Refusals list what would pass (C18, C19). There is no allow-list yet, because |
| +Refusals list what would pass (C18, C19, C31). There is no allow-list yet, because |
| there is no manifest yet. A fleet `place` and a `region` may both be sent, |
| and both are stored (§12). |
| @@ -699,6 +772,7 @@ The response gains one block: |
| - `absent`: the code is reported even though `regions` does not list it |
| (codex 7); |
| - `not-hosting`; |
| + - `no-secrets-key` (R9, §6.1); |
| - `not-eligible` (§6.1); |
| - `not-allowed`; |
| - `pooled`; |
| @@ -1002,6 +1076,8 @@ than the picker, but it must agree geographically, and the README says so. |
| | mmdb lookup error | A miss; rate-limited log. | |
| | timezone UTC, `Etc/*`, unknown or malformed | IP step, then home. | |
| | boxes not yet publishing the eligibility fields | No coded region is eligible. The guess is the home, and app-derived defaults are refused as `not-eligible`. | |
| +| a region's boxes hold no secrets key, or a different one (R9) | Not eligible for any automatic choice. App-derived defaults are refused with C16 `no-secrets-key`. Explicit socket-backed or replica creates are refused with C31 once a current writer proves the mismatch. | |
| +| console has no `CARLOS_SECRETS_PUBKEY` (R9) | Logged once. No coded region is eligible for automatic placement. Explicit creates keep today's behaviour. | |
| | edges listing fails | The guess is the home. App-derived creates get 503 C28. Explicit creates are unchanged. | |
| | last box of the app's region retired | App-derived creates are refused (C16 `absent`), and `app_default` and C8 say why. | |
| | home publishes no AWS region | Every guess is `default`. Fix it with the box's `CARLOS_AWS_REGION`. | |
| @@ -1039,6 +1115,23 @@ than the picker, but it must agree geographically, and the README says so. |
| - City and Country-only records; |
| - private, loopback and CGNAT addresses never reach the reader. |
| +**Secrets key (R9):** |
| + |
| +- `appconfig.KeyFingerprint` of a key's public half equals the fingerprint |
| + computed from the base64 `CARLOS_SECRETS_PUBKEY` of the same key. |
| +- A box with the key publishes `secrets_key_fp`. With no key, or a key that |
| + does not parse, it omits the field. The body never contains the private key |
| + or the public key itself. |
| +- Eligibility: |
| + - a box with a matching fingerprint is eligible; |
| + - a box with a different fingerprint, or none, is `no-secrets-key`; |
| + - a console with no `CARLOS_SECRETS_PUBKEY` makes no coded region |
| + eligible. |
| +- An explicit socket-backed create into a region where only current writers |
| + publish a mismatch is refused with C31. Under an old writer, or with no |
| + console key, it is accepted as today. Static and addr-backed creates are |
| + never refused for it. |
| + |
| **Edges file and eligibility** (codex R2-1): |
| - The writer publishes `serves_shared_pool` and `accepts_inbound` from |
| @@ -1199,18 +1292,25 @@ and `ip` appear in none of: |
| flagship's Ireland box publishes `region: eu-west-1` and no `placement`. |
| Use `CARLOS_AWS_REGION` where it does not. |
| - Set `CONSOLE_HOME_REGION` on every console that will turn the switch on. |
| -2. **Box release.** It carries the two edges-file eligibility fields. Roll it |
| - to every box, then check every region's files report all three fields. |
| - Until then, no coded region is eligible (§6.1). |
| -3. **Console release, switch off.** It carries the migration, `internal/geo`, |
| +2. **Stage the secrets key (R9).** Every box in a region that should take |
| + placed instances gets the deployment's `CARLOS_SECRETS_KEY` in |
| + `/etc/carlos/host.env`, from the deployment's secret store. On the flagship |
| + today that means the UK, US and AU boxes. This is infrastructure work in |
| + carloku-infrastructure, under the scoped-approval rule (AGENTS.md). Set |
| + `CARLOS_SECRETS_PUBKEY` on the console if it is not already set. |
| +3. **Box release.** It carries the edges-file eligibility fields and |
| + `secrets_key_fp`. Roll it to every box. Then check that every region's |
| + files carry all of them, and that each `secrets_key_fp` equals the |
| + console's. Until then, no coded region is eligible (§6.1). |
| +4. **Console release, switch off.** It carries the migration, `internal/geo`, |
| the fill, the endpoints and the UI. Behaviour is unchanged, and it can be |
| rolled back freely. |
| -4. **Verify on dev.** Turn the switch on with two eligible regions. Then: |
| +5. **Verify on dev.** Turn the switch on with two eligible regions. Then: |
| - create an app in the browser, and read its row; |
| - create an instance with no region, and read its record; |
| - diff `/regions` for an old app against the pre-roll answer; |
| - run the service gate below. |
| -5. **Service gate on the flagship, with the switch still off** (codex R4-3). |
| +6. **Service gate on the flagship, with the switch still off** (codex R4-3). |
| Eligibility proves configuration, not a working request path. |
| Reachability is only a non-empty EIP (`instancesplacement.go:185`), and |
| @@ -1240,12 +1340,12 @@ and `ip` appear in none of: |
| The probe instances are deleted afterwards. The same gate is re-run before |
| the switch is turned on again after any period off, and whenever a new |
| region becomes eligible. |
| -6. **Flagship.** Roll the console, then turn the switch on. **From here the |
| +7. **Flagship.** Roll the console, then turn the switch on. **From here the |
| binary rollback floor applies (R7).** That release's notes and the roll |
| recipe in carloku-infrastructure name the floor version. Optionally, set |
| `CONSOLE_GEOIP_DB` to a DB-IP Lite file. |
| -7. **CLI release.** It is safe in either order (§9.6). |
| -8. **titogo**, in its own repo, adopts `/regions/guess` and the place names. |
| +8. **CLI release.** It is safe in either order (§9.6). |
| +9. **titogo**, in its own repo, adopts `/regions/guess` and the place names. |
| No step of Phase P depends on Phase L. Phase P writes nothing to DNS, stamps |
| no `Steering` field, and needs neither `CARLOS_STEERING_ZONE_ID` nor any |
| @@ -1255,10 +1355,10 @@ change to the zone. |
| (§13.3). Its gate is that spec landing with its simulation passing, plus |
| scoped approval for the infrastructure. Until then, **no deployment arms |
| `CARLOS_STEERING_ZONE_ID`**, including for the static-site steering that |
| -already exists. The latency baseline is taken once Phase P step 6 has run. |
| +already exists. The latency baseline is taken once Phase P step 7 has run. |
| That makes Phase L depend on Phase P, never the other way round. |
| -Copy review (§19) happens before Phase P step 3 ships any string. |
| +Copy review (§19) happens before Phase P step 4 ships any string. |
| ## 18. Questions for Paul |
| @@ -1294,16 +1394,18 @@ review. There are no em dashes. `{…}` is filled at runtime. |
| | C22 | flag help | apps create: "region this app's instances start in (default: the region nearest you)". deploy: "region for the app this deploy creates (default: the region nearest you)". canary: "region for the canary instance (default: the app's region)". instances create: "region for this instance (default: the app's region, subject to the manifest's allow-list)". | |
| | C24 | new-app page, IP step configured | IP location by DB-IP. (Linked to db-ip.com; DB-IP's required wording to be confirmed against its licence page.) | |
| | C30 | CLI, canary --region on a fleet-placed app | {app} runs on your fleet {fleet}, which decides where every instance runs, so --region does not apply. Run the command without --region. | |
| +| C31 | API, explicit create into a region without the secrets key (R9) | region {code} can't run instances right now: its boxes don't hold this deployment's secrets key, so an instance there would start without its config or secrets. Choose another region. | |
| | C25 | CLI, canary --region on an existing canary | canary {host} already runs in {place}. --region applies only when canary creates the host. Run carlos canary remove first to start it somewhere else. | |
| | C26 | API, defaulted home inside its replica set | this app's region {code} is also one of this instance's replica placements. Pass a region for the instance, or drop {code} from replicas. | |
| | C27 | API, guess with unreadable policy | could not read this app's instance policy, so there is no region suggestion. Try again. | |
| | C28 | API, app region unconfirmed | could not confirm that this app's region {code} can take new instances. Try again, or pass a region. | |
| | C29 | roll-forward log, read by operators | possible placement discrepancy: {host} of {app} has no region, but {app}'s region is {code}. It may be a deliberate home override; review it. | |
| -The `{reason}` words in C16 are `absent`, `not-hosting` and `not-eligible`. |
| -Shown to a person, they read: "no box runs there now", "boxes there can't |
| -start instances", and "boxes there don't take ordinary instances". Those are |
| -copy too. |
| +The `{reason}` words in C16 are `absent`, `not-hosting`, `no-secrets-key` |
| +(R9) and `not-eligible`. Shown to a person, they read: "no box runs there |
| +now", "boxes there can't start instances", "boxes there can't open this |
| +deployment's secrets" (R9), and "boxes there don't take ordinary instances". |
| +Those are copy too. |
| C23, place names: |
| @@ -1414,9 +1516,9 @@ The `docs/site/cli.md`, README and release-note additions (§5.7, §11, §12, |
| | former question | disposition | |
| | --- | --- | |
| -| Q1: apply R5, titogo separate | adopted (§2, §17 step 8) | |
| +| Q1: apply R5, titogo separate | adopted (§2, §17 step 9) | |
| | Q2: implement R6 with the DNS corrections, keep the separate approved Phase L rollout and measurements | adopted in substance. The corrections are larger than a patch, so they move to the follow-on spec with invariants (§13); the sequencing is ruled by R8. | |
| -| Q3: name the exact floor version before activation; keep region-aware reads when disabling | adopted (§5.6, §5.7, §17 step 6) | |
| +| Q3: name the exact floor version before activation; keep region-aware reads when disabling | adopted (§5.6, §5.7, §17 step 7) | |
| ### Round 4 (rev 4.1, 4ad64f7c) |
| @@ -1424,6 +1526,15 @@ The `docs/site/cli.md`, README and release-note additions (§5.7, §11, §12, |
| | --- | --- | --- | --- | |
| | R4-1 | P2 | Fleet-placed apps still get instance guesses the door will override | **accepted.** Verified at `fleets.go:479` and `apiinstances.go:798`. The guess endpoint resolves fleet placement first and returns `null`; app-creation inference is unchanged; end-to-end test (§6.1, §9.4, §12, §16). | |
| | R4-2 | P2 | `canary --region` on a fleet behaves differently before and after creation | **accepted.** Verified at `canary.go:405` and `apiinstances.go:798`. Fleet precedence is resolved before either branch, and `--region` on a fleet-placed app refuses identically every time with C30, never suggesting a remove (§11, §12, §16). | |
| -| R4-3 | P2 | Phase P verification can pass while non-home instances are unreachable | **accepted.** Verified at `instancesplacement.go:185`, `edge.go:2699` and `:1428`. A service gate before the production switch: a fresh probe per eligible region, reached from outside through the platform host and through every other edge with `--resolve`, checking 2xx and `X-Carlos-Version`. All regions must pass. Plus an integration test for the missing-mesh-key case (§16, §17 step 5). | |
| +| R4-3 | P2 | Phase P verification can pass while non-home instances are unreachable | **accepted.** Verified at `instancesplacement.go:185`, `edge.go:2699` and `:1428`. A service gate before the production switch: a fresh probe per eligible region, reached from outside through the platform host and through every other edge with `--resolve`, checking 2xx and `X-Carlos-Version`. All regions must pass. Plus an integration test for the missing-mesh-key case (§16, §17 step 6). | |
| Astra raised no open questions in round 4. §18 stands at none. |
| + |
| +### Amendment R9 (rev 6, Paul 2026-10-02) |
| + |
| +Not a review finding: a production finding, ruled by Paul. Recorded here so |
| +the dispositions list every change since rev 5. The changes are §3 (R9), |
| +§6.1 (`secrets_key_fp`, its format, the console's expected fingerprint, the |
| +fourth eligibility fact), §7 and §8.1 (the explicit refusal, C31), §9.3 and |
| +C16 (the `no-secrets-key` reason), §15, §16, §17 (staging the key), and §19 |
| +(C31). |