amadan
#162: design review brief for the replica-placements design round
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 file changed,
+199
−0
FABLE-REVIEW-BRIEF-162-design.md+199 −0
CI passed — run details
diff --git a/FABLE-REVIEW-BRIEF-162-design.md b/FABLE-REVIEW-BRIEF-162-design.md| new file mode 100644 |
| index 000000000..46f1f7e66 |
| --- /dev/null |
| +++ b/FABLE-REVIEW-BRIEF-162-design.md |
| @@ -0,0 +1,199 @@ |
| +# Review brief: replica placements on a route (carlos/platform #162), design round |
| + |
| +You are the second reader on a design that has not been written up as a spec |
| +yet. Review the design sketch below against the code as it exists at the |
| +commit this file is committed in. Find what is wrong, what is missing, and |
| +what will not survive contact with the fleet — not what could be worded |
| +better. |
| + |
| +## What the ask is |
| + |
| +amadan discussion carlos/platform #162, "A route can carry replica |
| +placements, served locally instead of hairpinned (edge checkout)", filed by |
| +titogo. Its three asks, in titogo's words: |
| + |
| +1. **Replica placements on a route.** A route carries, alongside its `Home`, |
| + a set of replica placements. A router whose own placement is in that set |
| + serves the request from a LOCAL socket (the replica instance) instead of |
| + hairpinning. Two constraints titogo names: the hairpin decision is made |
| + only in root-domain path routing and the host-routed fork deliberately |
| + never consults `remote()` (structural loop protection), so the replica |
| + decision must cover host-routed requests too and a replica must never |
| + hairpin; and for a replica to be reachable, the account host and custom |
| + domains need latency routing to the replica placements as well. |
| +2. **Provisioning a replica instance.** The sandbox verb with a different |
| + role and a chosen placement: same app, same account, `role=replica`, |
| + placed on box X, with the instance API exposing role and placement so the |
| + home instance can list its replicas and address them. |
| +3. **A signed instance-to-instance channel via the router.** Home POSTs to |
| + its replicas (issue/renew/reclaim a lease); a replica POSTs to home |
| + (drain a batch). titogo asks whether the relay's signed-delivery path |
| + (`TITO_RELAY_SIGNING_KEY` -> `TITO_RELAY_PUBKEY`, forwarded through the |
| + router's `/__internal/*` exception) may carry an app-defined internal |
| + call, or what should. |
| + |
| +**Residence.** A replica in placement P holds order data for an account |
| +homed in H until the drain confirms. Replica placement must be per-account |
| +opt-in and constrained by the account's residence policy — never silently |
| +on. |
| + |
| +Not in scope: an AU box, and the lease/drain protocol itself (titogo's). |
| + |
| +Titogo's design record is `docs/superpowers/specs/2026-09-21-edge-checkout-design.md` |
| +in the titogo repo (`/home/paulca/github.com/tito/titogo`, commit c6ed556e0). |
| + |
| +## The correction this design makes to the ask |
| + |
| +#162 is written against **titogo's router** (`internal/carlos/router.go` |
| +in the titogo repo): `remote()`, the hairpin, `tables.slugs`. On CARLOS that |
| +is not the mechanism that carries account hosts: |
| + |
| +- `carlos-edge` owns host routing. A request for a host this box does not |
| + serve is **backhauled** to the box that does — one HMAC-signed hop, mesh |
| + key, `internal/edge/edge.go` `backhaul` (~:2507), home map from the fleet's |
| + edges files (`internal/edge/homemap.go`). |
| +- The edge tries the local upstream first and only calls `backhaul` on a |
| + miss (`ServeHTTP`, ~:1285). |
| +- titogo's `Route.Home` is never written at HEAD, so the hairpin is dormant |
| + everywhere. |
| + |
| +So the platform's answer to Ask 1 is an EDGE change, not a router change, |
| +and the "never hairpins" property becomes structural rather than a rule: |
| +a box with a local row for a host never reaches the backhaul call at all. |
| + |
| +## The design under review |
| + |
| +### 1. Record shape (`internal/provision/provision.go`) |
| + |
| +Three additive fields on `provision.Record`, every one empty on records |
| +written before it, so an empty set is byte-identical to today at every |
| +layer: |
| + |
| +- `Role string` — `""` (the serving home) or `"replica"`. Closed |
| + vocabulary validated by a `ValidRole` in `ValidSteering`'s shape. |
| + `ValidateShape` refuses `role=replica` on anything but `kind=instance`. |
| +- `Replicas []string` — on the HOME record only: the placement codes the |
| + account opts into. Validated against `ValidRegion`, deduplicated, sorted, |
| + capped. Never defaulted and never inferred from `Region`. This field IS |
| + the residence opt-in. |
| +- `Home string` — on the REPLICA record only: the public host it mirrors. |
| + Written at create, never converged; repointing a replica is |
| + delete-and-recreate. |
| + |
| +Provisioning a replica is the existing instance-create verb with a role and |
| +a placement: same app, same account, an unguessable host, `Region` = the |
| +replica's box placement. Because it carries the same app and channel, the |
| +ordinary deploy pass gives it the same version as home with no new code. |
| + |
| +### 2. Routing (Ask 1) |
| + |
| +On the replica's box, the reconcile pass mints, for a replica record: |
| + |
| +- the replica's own route row (host = the unguessable replica host -> |
| + the replica instance's socket), exactly as any instance record does; and |
| +- a second row for the account's PUBLIC host (`Home`) pointing at the same |
| + socket, plus rows for `Home`'s aliases whose `dns_status` is `pointing`. |
| + |
| +The edge then serves the account host locally because it has a local row, |
| +and `backhaul` — which only runs on an upstream miss — never fires there. |
| +Host-routed coverage is automatic: the edge has no other kind of routing. |
| + |
| +The replica box publishes those entries in its **edges file** under role |
| +`"replica"`, not `"home"`. `internal/edge/homemap.go` ignores roles it does |
| +not recognise, so every other box keeps backhauling the account host to the |
| +real home and the home map stays single-valued. The replica's own host is |
| +published as `"home"` (it genuinely is homed there), so home and siblings |
| +can reach the replica by its own name through ordinary backhaul. |
| + |
| +### 3. DNS / steering — no change |
| + |
| +`internal/steering/steering.go` `desiredState` already emits one latency A |
| +record per ARMED EDGE for every steered host, and custom domains CNAME to |
| +`edge.<apps-domain>`. Buyers therefore already land on the nearest edge |
| +today; the only question was what that edge does when the request arrives, |
| +which is §2. The design changes no steering code. titogo's second bullet |
| +under Ask 1 rests on the router model, where the set of answers is per-home; |
| +on CARLOS every edge already answers. |
| + |
| +### 4. Instance-to-instance calls (Ask 3) |
| + |
| +No new channel and no relay change. An instance dials a sibling at the |
| +sibling box's EIP with the real hostname as SNI — which is what `backhaul` |
| +itself does (`backhaulTransport`, dial the EIP, authority names the host) — |
| +and that bypasses the local edge, so a replica can reach home through the |
| +account host without landing on its own local row. |
| + |
| +The platform supplies only what an instance cannot know: |
| + |
| +- its own role and, for a replica, its home's host and dial address; |
| +- for a home, the list of its replicas with their hosts and dial addresses; |
| +- one per-account replica key shared by home and its replicas, derived from |
| + the box mesh key (`internal/host/mesh.go`) as |
| + `HMAC(meshKey, "replica/" + account)`, published with a `_NEXT` companion |
| + so a mesh rotation does not break a call in flight. |
| + |
| +Delivered as unit environment through the named-unit config path (amadan |
| +discussion #114, `worktree-unit-route-config`). titogo signs and shapes the |
| +lease and drain calls itself; the answer to "may the relay carry an |
| +app-defined internal call" is "do not — use this instead". |
| + |
| +### 5. API and CLI |
| + |
| +`internal/console/apiinstances.go`: create accepts `role` and `home`; PATCH |
| +accepts `replicas` on a home record and refuses a placement the deployment |
| +does not have armed; list and get expose `role`, `home` and `replicas` |
| +beside the existing `served_by` (box + region). `instances_cli.go` grows a |
| +role column and a `carlos instances replicas` verb. A new `carlos` verb is |
| +not done until `docs/site/cli.md` names it (AGENTS.md). |
| + |
| +### 6. Testing |
| + |
| +Reconcile mints both rows on the replica's box and nothing on any other; |
| +the edge serves the account host from the local replica socket with no |
| +backhaul attempt; a third box's home map still resolves the account host to |
| +the real home; an empty `Replicas` set is byte-identical to today at every |
| +layer. |
| + |
| +## What to review for |
| + |
| +1. **Does §2 actually work in this repo's edge and reconcile code?** Read |
| + `internal/provision/reconcile.go`, `internal/edge/edge.go`, |
| + `internal/edge/homemap.go`, `internal/host/edgesfile.go`. Two route rows |
| + pointing at one socket, a row for a host whose record lives on another |
| + box, aliases, certificates (does the replica box obtain a certificate for |
| + the account host and its custom domains? by what mechanism?), the |
| + activator and hibernation, `Fingerprint`/`SnapshotFingerprint`, the |
| + registry's uniqueness assumptions. Name the specific code that breaks. |
| +2. **Is publishing role `"replica"` in the edges file really ignored by |
| + every reader**, including `internal/edgesfile` and anything in titogo |
| + that parses the same file? What does a reader that treats an unknown role |
| + as home do to the home map? |
| +3. **The certificate question is the one I am least sure of.** If the |
| + replica box cannot obtain or mirror a certificate for the account host, |
| + §2 does not serve a single request. Say what the real mechanism has to |
| + be. |
| +4. **Is the mesh-derived per-account key sound**, and is the mesh key |
| + something an instance process may hold a derivative of? Rotation |
| + semantics, blast radius if one replica is compromised, whether the |
| + platform should instead sign per call. |
| +5. **Residence**: is `Replicas` on the home record enough of a gate, given |
| + the policy itself lives in titogo? What must the platform refuse on its |
| + own? |
| +6. **What is missing entirely** — deploys to replicas, hibernation and |
| + sleep/wake on a replica, the status/health surface, moves |
| + (`docs/superpowers/specs/2026-08-28-instance-move-design.md`), deletion |
| + ordering (a replica holding undrained orders), object storage and per |
| + instance buckets, backups, the console's own views. |
| +7. **Scope**: anything here that should be cut to make the first landing |
| + smaller, and anything cut that must not be. |
| + |
| +## How to answer |
| + |
| +A numbered list of findings, most severe first. For each: a severity |
| +(P1 blocks the design, P2 must be answered in the spec, P3 worth knowing), |
| +the file and line of the code it rests on, what breaks concretely, and what |
| +to do instead. Verify every code citation before you make it — cite what |
| +you read, not what the brief claims. Say plainly if a section is sound; do |
| +not invent findings to fill the list. |
| + |
| +Do not edit any file. Your output is the findings document. |