amadan
#162: replica-placements design spec, and the Fable findings behind it
Summary
the design round's findings (3 P1) committed unedited, and the spec that answers them — replica placements ride a route_aliases row the domains pass derives, the registry carries a replica role for the edges file, and a home record with replicas must be latency-steered or the feature silently serves nothing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2 files changed,
+785
−0
FABLE-FINDINGS-162-design.md+398 −0docs/superpowers/specs/2026-09-21-replica-placements-design.md+387 −0
CI passed — run details
diff --git a/FABLE-FINDINGS-162-design.md b/FABLE-FINDINGS-162-design.md| new file mode 100644 |
| index 000000000..3e7824591 |
| --- /dev/null |
| +++ b/FABLE-FINDINGS-162-design.md |
| @@ -0,0 +1,398 @@ |
| +# Findings: replica placements on a route (#162), design round |
| + |
| +Second reader's findings against commit d6810d99 in this worktree, read |
| +against FABLE-REVIEW-BRIEF-162-design.md. Every citation below was read in |
| +this checkout; where the brief's account of the code differs from what the |
| +code says, that is called out inline. The titogo side was read at |
| +c6ed556e0 (`docs/superpowers/specs/2026-09-21-edge-checkout-design.md`, |
| +`internal/carlos/router.go`, `internal/instanceapi/instanceapi.go`). |
| + |
| +Severity: P1 blocks the design as written, P2 must be answered in the |
| +spec, P3 worth knowing. |
| + |
| +Summary: the core idea (a local row on the replica box so the edge never |
| +reaches `backhaul`) is right and the certificate story is sound. Three |
| +things block it as sketched: the "second route row" cannot survive the |
| +passes that own the routing tables, nothing on this repo steers an |
| +instance-kind host to every edge today, and the edges-file writer has no |
| +way to publish role `"replica"`. All three have clear fixes. |
| + |
| +--- |
| + |
| +## 1. P1. A second ROUTE row for the public host cannot exist on the replica box. Use `route_aliases`, and teach the pass that owns that table. |
| + |
| +§2 mints "a second row for the account's PUBLIC host (`Home`) pointing at |
| +the same socket". Read as a `routes` row, four things break, each on its |
| +own: |
| + |
| +- **The instance reconcile pass deletes it on the next tick.** The removal |
| + phase retires every `Provisioned` row in scope whose host is not in the |
| + record set: `internal/provision/reconcile.go:1075-1092` (`why := |
| + "record gone"` when `recs[rt.Host]` misses, then `reg.Delete`). The |
| + public host has no record on the replica box (its record has |
| + `Region == home`), so a Provisioned row for it is "record gone" and goes. |
| + Minting it un-Provisioned instead means nothing ever retires it when the |
| + replica is deleted. |
| +- **The adopt pass treats it as an instance to deploy.** `Adoptable` is |
| + `Kind == "instance" && App != "" && !Platform && ready` |
| + (`internal/release/adopt.go:816-819`). The row would be staged, pointed |
| + live and restarted under its OWN host name (`adoptHost` keys everything |
| + on `rt.Host`), for a process that does not exist under that name. |
| +- **The activator keys children by `rt.Host` and execs `live/<host>`.** |
| + `internal/activator/activator.go:701` (`a.entries[rt.Host]`) and |
| + `internal/activator/backend_exec.go:804` (`bin := b.liveLink(host)`), |
| + with `--socket rt.Socket --db dbPath` at `:913`. A hibernating row for the |
| + public host would try to spawn a second process named after the public |
| + host; a non-hibernating one is simply an unwakeable socket path. |
| +- **`routeFor` derives Socket and DBPath from the record host** |
| + (`internal/provision/reconcile.go:348-372`), so a record-driven mint |
| + cannot produce "same socket, different host" at all. |
| + |
| +The primitive that does exactly what §2 wants already exists: an alias. |
| +`Edge.Reload` folds `route_aliases` so the alias name resolves to the |
| +PRIMARY's own map entry, same proxy, same route row, same activator |
| +keying, same version header (`internal/edge/edge.go:751-756`, `up.alias = |
| +true; m[a.Alias] = up`). A request for the public host on the replica box |
| +then wakes and bills the REPLICA route. Aliases are in the edge's 2s reload |
| +fingerprint (`internal/registry/registry.go:2147-2148`), and routes always |
| +win an alias collision (`edge.go:747-749`), which is the right answer if a |
| +home ever lands on the same box. |
| + |
| +But `route_aliases` is wholly owned by `domains.ReconcileAliases`, which |
| +rebuilds the desired set from claims every pass and deletes every alias |
| +not in it: `internal/domains/aliases.go:363-369`. A replica alias inserted |
| +by any other writer is swept within seconds. So the spec must either |
| + |
| +- extend `ReconcileAliases` to derive replica aliases (public host and its |
| + `pointing` custom domains) from replica records on this box, so they are |
| + in `desired`; or |
| +- give the table an ownership mark (the `Provisioned` precedent on routes) |
| + and have `ReconcileAliases` leave marked rows alone, with the replica's |
| + own pass minting and retiring them. |
| + |
| +The first is smaller and keeps one writer. Either way the custom-domain |
| +half also needs it: today an alias for a custom domain is only derived |
| +when `byHost[c.Route]` is a LOCAL route (`aliases.go:95-100`), and on the |
| +replica box the public host is not a route, so the claim never |
| +materialises there without new code. |
| + |
| +## 2. P1. §3 "DNS / steering, no change" is wrong for every host this feature is for. Nothing in this repo sends an instance-kind host to every edge today. |
| + |
| +The brief says buyers "already land on the nearest edge today". In this |
| +checkout that is only true for a host that meets ALL of: |
| + |
| +- its record carries `steering: "latency"`. `steeredHosts` skips `""` |
| + silently (`internal/steering/steering.go:157-162`); |
| +- it lies under the apps domain (`steering.go:177-184`); a custom domain |
| + or a tenant-owned instance domain gets no direct record at all, only the |
| + advice to CNAME to `edge.<apps-domain>`; |
| +- the deployment has set `CARLOS_STEERING_ZONE_ID`; unset, the duty |
| + "reads nothing and calls no AWS API" |
| + (`internal/host/steering_loop.go:219`). No root module in `infra/` sets |
| + `steering_zone_ids`; only the `region-host` module declares the variable |
| + (`infra/modules/region-host/variables.tf:153`). |
| + |
| +And the create door defaults `latency` ONLY for `kind: static` in the |
| +platform namespace (`internal/console/apiinstances.go:827`). titogo's |
| +instances are `kind: instance`, so every account host is unsteered unless |
| +someone runs `carlos steering ... latency` per host. |
| + |
| +Consequence: the replica box mints its local row and never receives a |
| +buyer, because the account host resolves to one edge (whichever the |
| +wildcard or the tenant's DNS names). The design does not fail loudly; it |
| +serves nothing locally and looks correct from every box. |
| + |
| +What to do: make it a rule of the record, not of the deployment. A home |
| +record with a non-empty `Replicas` MUST carry `steering: "latency"` (refuse |
| +the PATCH otherwise, or set it in the same write), and the spec must state |
| +the tenant-DNS half: the account host under a tenant-owned domain and every |
| +custom domain must CNAME to `edge.<apps-domain>`, or the replica placement |
| +is inert. Say which hosts on the flagship this actually applies to today. |
| + |
| +## 3. P1. The edges-file writer cannot publish role `"replica"`. The registry has no role, and publishing the rows as `"home"` splits the fold. |
| + |
| +`edgesFilePass` derives every host entry from `reg.List()` and hard-codes |
| +`Role: "home"` (`internal/host/edgesfile.go:219`); aliases go out as |
| +`Role: "home", Alias: true` (`:269`). There is no registry column and no |
| +row shape from which the writer could learn that a row is a replica's |
| +projection of another box's host. The brief's §2 assumes a writer that |
| +does not exist. |
| + |
| +If the projection goes out as a route entry (`home`, not alias), the fold |
| +on every OTHER box gets two dial targets for the public host and picks the |
| +first box in sorted order (`internal/edge/homemap.go:566-584`, "first box |
| +in sorted order wins the dial"). A third box would then backhaul the |
| +account host to the replica, contradicting §2's own claim that "every |
| +other box keeps backhauling the account host to the real home". If it goes |
| +out as an alias entry, the public host is safe (route beats alias, |
| +`:581`), but the custom DOMAINS are alias entries on BOTH boxes, so they |
| +too fall to sort order. |
| + |
| +Verified reader behaviour, for the record: |
| + |
| +- `foldHomes` drops any role other than `home`/`wildcard` outright |
| + (`homemap.go:563-564`), so a `"replica"` role is invisible to routing, |
| + `KnownHost`, `Hosts()` and `ServedHosts()`. The brief's claim about this |
| + reader is correct. |
| +- `internal/edgesfile.DeploymentRoutes` counts `Role == RoleHome && |
| + !Alias` only (`internal/edgesfile/fold.go:163`), so a `"replica"` |
| + role does not reserve the hostname against claims. Correct and desirable. |
| +- `internal/steering` reads no `Hosts` at all |
| + (`internal/steering/steering.go:60-65`). |
| +- The console's `served_by` join takes EVERY role |
| + (`internal/console/instancesplacement.go:68-70`), so the public host |
| + would list the replica box beside its home. That is arguably the answer |
| + titogo asked for in Ask 2; state it rather than let it happen. |
| +- titogo parses no edges file (no reference to `control/edges` or the |
| + `role` key in `internal/`). |
| + |
| +What to do: add the mark at the registry, since that is the only thing the |
| +writer reads. The least invasive shape is a column on `route_aliases` |
| +(`replica_of` or a `role` text) that `edgesFilePass` maps to |
| +`Role: "replica"` and that `ReconcileAliases` derives (finding 1). Then |
| +`foldHomes` needs no change and the map stays single-valued by |
| +construction. Add the `"replica"` case to `foldHomes` explicitly (a no-op |
| +`continue` with a comment) so the ignore is a decision, not an accident. |
| + |
| +## 4. P2. Residence: `Replicas` on the home record is the opt-in, but every cross-record invariant is unstated. The platform must refuse these on its own. |
| + |
| +The doors today validate a record against itself and its manifest, never |
| +against a sibling record (`handleAPIInstanceCreate`, |
| +`apiinstances.go:550-905`; `handleAPIInstanceUpdate`, `:1041`). A replica |
| +introduces cross-record facts the spec has to pin, because nothing in |
| +titogo can enforce them and nothing in the reconcile pass will: |
| + |
| +- create `role=replica`: `Home` must name an existing record in the SAME |
| + account and app whose own `Role` is `""` (no replica of a replica); |
| + `Region` must be non-empty, must differ from the home's `Region`, must be |
| + in `home.Replicas`, must pass the manifest allow-list |
| + (`instanceRegionAllowed`, `:1573`) and the armed-and-hosting gate |
| + (`placements`, `internal/console/instancesplacement.go:173-202`). |
| + `ValidRegion` accepts `""` (`internal/provision/provision.go:549-551`), so |
| + `Replicas` needs its own rail refusing `""` and the home's own region. |
| +- PATCH `replicas` on a home: removing a placement while a replica record |
| + exists there must be refused (or the spec says the replica is retired, |
| + and by whom). Otherwise the opt-in and the data disagree, which is the |
| + residence failure the field exists to prevent. |
| +- moves: `MoveRequest` (`provision.go:797`) on a replica record must be |
| + refused (repointing is delete-and-recreate by the design's own rule); a |
| + move of the HOME to a placement in its `Replicas` must be refused, or |
| + the home and the replica projection collide on one box (finding 1's |
| + route-beats-alias covers the edge, but the replica is then pointless |
| + and `Home` on the replica record names a host on its own box). |
| +- `everywhereClass`/`inScope` are unaffected: a socket-backed replica |
| + record scopes to its own region only (`reconcile.go:202-236, 319-336`). |
| + Good; say so. |
| + |
| +Also: `ValidateShape` is re-run on every reconcile pass and a failing |
| +record is `protect`ed, not removed (`reconcile.go:711-719`). Add the |
| +`Role`/`Home`/`Replicas` shape rules there too, so a hand-written bucket |
| +record cannot mint a replica the door would have refused. |
| + |
| +## 5. P2. The per-account mesh-derived key: derivation is sound, but rotation and delivery are not worked out, and per-instance keys are strictly better. |
| + |
| +Sound parts, verified: the agent process already holds the mesh key pair |
| +in memory (`homeMap.refreshKeys`, `internal/edge/homemap.go:675-683`; |
| +`loadMeshKeys`, `internal/edge/mesh.go:184-199`), and the edge runs inside |
| +`carlos agent`, so no new key read is needed. `HMAC(meshKey, label)` is a |
| +PRF, so a compromised replica holding the derivative cannot recover the |
| +mesh key; blast radius is one account's channel. The mesh key is never |
| +served through the fleet store door (`mesh.go:92-95`), so a fleet box |
| +cannot obtain it. |
| + |
| +Not worked out: |
| + |
| +- **Delivery seam.** The brief says "the named-unit config path (#114)". |
| + None of #114's commits are on this branch (`d5a8fa1c`, `5a4f100f`, |
| + `fcd941af` are not ancestors of HEAD), and titogo's instances are |
| + activator exec children, not named units (titogo `router.go:504-513`, |
| + `carlosManaged` = socket under the activator's dir). The seam that |
| + exists today is `ExecBackend.agentOwnedVars` |
| + (`internal/activator/backend_exec.go:977-994`), which already injects a |
| + per-box `CARLOS_REGION` at spawn, and `appconfig.applyAgentVars` for |
| + unit tenants (`internal/appconfig/agentvars.go:36-51`). The spec should |
| + name `agentOwnedVars` and add the variables there; the activator will |
| + need the key pair handed down from the edge (a `func() (cur, next |
| + []byte)` on the backend, like `b.Region`). |
| +- **Rotation.** Env is composed at spawn (`backend_exec.go:820`, |
| + `instanceEnv`). A running instance keeps the derived key it was born |
| + with. `.next` exists only while a rotation is in flight (`mesh.go:101`), |
| + and `signingKey` switches signers the instant it appears (`:202-207`). |
| + So the runbook step "promote .next" changes the derivative on every |
| + future spawn while awake instances still hold the old one. The spec |
| + must say: the platform publishes `_NEXT` from the moment `.next` is |
| + mirrored; titogo verifies under either key (the `verifyHopEitherKey` |
| + rule, `mesh.go:258-268`); and promotion is followed by a restart of |
| + every home and replica instance of every account with replicas, or |
| + the platform bounces them itself. |
| +- **Scope.** Derive per INSTANCE, not per account: `HMAC(meshKey, |
| + "replica/"+account+"/"+replicaHost)`. Home can derive every replica's |
| + key from the list it already receives; a compromised replica can then |
| + only speak as itself, never as a sibling. Cost is nil. |
| +- **Per-call platform signing** is not available: the outbound call |
| + deliberately bypasses the local edge (dial the EIP, SNI the host), so |
| + no platform component is on the sending side. Keep app-signed calls. |
| + |
| +## 6. P2. Deletion ordering and what is left behind. |
| + |
| +`handleAPIInstanceDelete` deletes the record unconditionally |
| +(`apiinstances.go:1288-1345`); the reconcile pass then `reg.Delete`s the |
| +row (`reconcile.go:1090`), the activator retires the child, and the DB |
| +file, the hibernate replica in the box's bucket and the env file stay on |
| +disk (the only DB removal in the activator is failed-wake cleanup, |
| +`backend_exec.go:315-327`). For a replica holding undrained orders that |
| +is stranded data with no route. The spec must: |
| + |
| +- refuse deleting a HOME whose `Replicas` is non-empty or for which a |
| + replica record exists (the platform can see both); |
| +- document that a replica delete is titogo's to sequence after its drain |
| + confirms (the platform cannot see drain state), and state what the |
| + platform does with the replica's DB and hibernate prefix afterwards |
| + (today: nothing, forever); |
| +- say what happens to the replica's alias rows when its record goes |
| + (finding 1 answers this if `ReconcileAliases` derives them). |
| + |
| +## 7. P2. Home and replica must follow one channel and will not be on one version. |
| + |
| +The brief: "because it carries the same app and channel, the ordinary |
| +deploy pass gives it the same version as home with no new code". Two |
| +gaps: |
| + |
| +- Nothing keeps the channels equal. `Channel` is per record, converged |
| + onto the row every pass (`convergeOwned`, `reconcile.go:572-580`). A |
| + channel PATCH or `carlos canary` on the home does not touch the |
| + replica's record. Either the replica record has no channel of its own |
| + and the pass reads the home's, or the PATCH door fans out. Say which. |
| +- Adoption runs per box on its own cadence (`adopt.go:303-360`), so home |
| + and replica WILL differ by one release for minutes, in either |
| + direction. titogo's lease/drain protocol must be version-tolerant. That |
| + belongs in the titogo design record, but the platform spec should state |
| + the guarantee it does not give. |
| + |
| +## 8. P2. The app token a replica holds is app-wide; a replica can delete its home. |
| + |
| +`handleAPIInstanceToken` mints via `NewAppToken(acct.ID, app, ...)` |
| +(`apiinstances.go:1392`), scoped to the app, not the instance (amadan |
| +#155). A replica process that holds it to list its siblings can also |
| +`DELETE` the home record. With finding 6's refusal that becomes a 409, but |
| +the spec should say whether the replica needs a token at all: the design |
| +already hands it its home's host and dial address as environment, so the |
| +replica may not need the console API at all. Only the HOME needs to list |
| +replicas. Give the token to home only. |
| + |
| +## 9. P3. Certificates: sound. Write the mechanism down. |
| + |
| +The brief is least sure of this; the code says it works, and it is worth |
| +stating in the spec so nobody re-derives it: |
| + |
| +- Every edge mirrors the WHOLE `control/certs/` prefix, not the names it |
| + owns: `certMirror.Refresh` lists the prefix and fetches every entry |
| + (`internal/edge/certsync.go:621-650`); `ownsCertName` gates only the |
| + backfill writer (`:1105-1121`), not what is mirrored or served. |
| +- The TLS hello is answered from the mirror for ANY name the mirror |
| + holds, gated or not (`wrapGetCertificate`, `:1346-1395`; `certificate`, |
| + `:1425-1437`), and from the local cache as a last resort (`:1518`). |
| +- Wildcard PEMs are mirrored the same way and loaded for parents from |
| + three sources: env static parents (tito's domain), this box's |
| + platform-class rows (`currentHosts`, `internal/edge/edge.go:888-899`, |
| + which excludes alias entries), and other boxes' platform-class rows via |
| + the home map (`deploymentHosts`, `:906-925`). The replica's OWN host is |
| + a platform-class row two labels under the apps domain, so it derives |
| + the same `<sqid>.<apps-domain>` parent the public host needs; the |
| + public-host alias derives nothing and needs to derive nothing. |
| +- Issuance and renewal stay with the housekeeper, whose order gate is |
| + `deploymentHost` = `Allowed || KnownHost` (`certsync.go:191-201`, |
| + `edge.go:999-1001`); the home still publishes the host, so nothing |
| + moves. The CA's validation lands on any edge and is answered by the |
| + read-through, as today. |
| + |
| +One nuance to state: on the replica box `Allowed(publichost)` becomes true |
| +(it is in the exact map, `edge.go:968`), so if that box is ever the |
| +housekeeper it may order for the public host itself. That is the same |
| +issuer, same account, same rate bucket. Harmless, but say it. |
| + |
| +## 10. P3. Hibernation and wake on a replica work through the alias, and lease traffic wakes sleeping replicas. |
| + |
| +With the alias primitive (finding 1), a request for the public host on the |
| +replica box resolves to the replica route's entry, and `up.hibernating` is |
| +the replica's (`edge.go:668`, `:751-756`), so the activator wakes the |
| +right child and the idle clock is the replica's. Home's periodic lease |
| +renewals will wake a sleeping replica on every renewal; titogo should |
| +size the renewal interval against `CARLOS_HIBERNATE_IDLE` or accept that |
| +replicas never sleep while a lease is live. Say it in the spec. |
| + |
| +## 11. P3. "Unguessable host" is not a secret and should not be described as one. |
| + |
| +The replica host is published in the box's edges file as `role: "home"` |
| +(by design, so home can reach it), is reachable from every edge by |
| +backhaul, and is under a wildcard the DNS already answers. Anyone who |
| +learns it gets the same checkout the public host serves. Fine, but the |
| +spec should call it "a name of its own" and not lean on unguessability |
| +for anything. |
| + |
| +## 12. P3. Console and CLI surface. |
| + |
| +`instanceListRecord` has no role column; `served_by` already lists every |
| +box whose edges file names the host, any role |
| +(`instancesplacement.go:68-70`), so with finding 3 the public host will |
| +show home and replica boxes together. Add `role` and `home`/`replicas` to |
| +the list and get views, a role column to `carlos instances`, and the new |
| +verb to `docs/site/cli.md` per AGENTS.md. Nothing else in the console |
| +needs to change for the first landing. |
| + |
| +## 13. P3. What to cut for the first landing, and what not to. |
| + |
| +Cut, safely: |
| + |
| +- custom-domain coverage on the replica box (the second half of finding |
| + 1 and the tenant-DNS half of finding 2). Land the account host first; |
| + custom domains are a follow-on that needs `ReconcileAliases` and a |
| + tenant DNS change anyway. |
| +- the `_NEXT` companion, IF the spec records that a mesh rotation is a |
| + restart-everything event until it lands. Rotation has never happened on |
| + this deployment; the promotion step is a runbook, not code |
| + (`mesh.go:97-101`). |
| +- the console's PATCH armed-placement check on `Replicas`: the create |
| + door already refuses a replica in an unarmed placement, which is where |
| + it matters. |
| + |
| +Do not cut: |
| + |
| +- the registry-level replica mark (finding 3). Without it the edges file |
| + cannot be honest and the fold is not single-valued. |
| +- the home-delete refusal (finding 6). It is one `ListRecords` at the |
| + door. |
| +- `steering: "latency"` on the home record (finding 2). Without it the |
| + feature serves nothing and nobody can tell. |
| + |
| +--- |
| + |
| +## Where the design is sound |
| + |
| +- Ask 1 as an EDGE change rather than a router change is right. The edge |
| + tries the local upstream first and only calls `backhaul` on a miss |
| + (`internal/edge/edge.go:1280-1301`; the call is at `:1297`, the brief's |
| + "~1285" is the surrounding branch), and `backhaul` itself refuses to |
| + forward a verified hop (`:2507-2512`). A local entry for the public host |
| + is structural loop protection. The brief's account of titogo's router is |
| + also accurate: the hairpin is decided only in `tables.slugs` |
| + (`router.go:656-663`) and never in the host-routed map (`:640-649`), |
| + and `remote()` is dormant with `TITO_REGION` unset (`:528-538`). |
| +- Ask 3 by direct dial (EIP + real SNI) is exactly what `backhaulProxy` |
| + does (`edge.go:2664-2700`, `backhaulTransport` `:2589-2660`), and the |
| + home edge presents the mirrored public cert, so no private CA and no |
| + new channel. Refusing to carry an app-defined call over the relay's |
| + `/__internal/connect/` exception is right: that path forwards exactly |
| + one shape and fails closed (`router.go:955-1011`). |
| +- The record shape (three additive fields, empty means today) matches the |
| + additivity rule every other field on `Record` follows |
| + (`provision.go:38-146`) and the anticipated `replica_regions` slot the |
| + move spec left open (`docs/superpowers/specs/2026-08-28-instance-move-design.md:5-6`). |
| +- The reconcile scope for the replica's OWN record is correct with no |
| + change: it is a socket-backed instance record with `Region` = the |
| + replica box, so it mints there and nowhere else (`reconcile.go:319-336`, |
| + `:735-738`). |
| +- Testing (§6) is the right set once finding 1's primitive is fixed; add |
| + "a replica alias survives a `ReconcileAliases` pass" and "the fold on a |
| + third box has exactly one dial target for the public host and for each |
| + custom domain". |
diff --git a/docs/superpowers/specs/2026-09-21-replica-placements-design.md b/docs/superpowers/specs/2026-09-21-replica-placements-design.md| new file mode 100644 |
| index 000000000..1ae75ce35 |
| --- /dev/null |
| +++ b/docs/superpowers/specs/2026-09-21-replica-placements-design.md |
| @@ -0,0 +1,387 @@ |
| +# Replica placements: an account served locally in a second placement |
| + |
| +Design record, 2026-09-21. The platform half of amadan discussion |
| +carlos/platform #162, *A route can carry replica placements, served locally |
| +instead of hairpinned*, filed by titogo for its edge-checkout work (titogo |
| +`docs/superpowers/specs/2026-09-21-edge-checkout-design.md`, commit |
| +c6ed556e0). |
| + |
| +Reviewed once before writing (`FABLE-REVIEW-BRIEF-162-design.md` -> |
| +`FABLE-FINDINGS-162-design.md`, commit d6810d99); accepted findings are |
| +marked **(fable N)** where they changed the design, and the declined ones |
| +are listed at the end. |
| + |
| +## 1. What is being asked for, and what it is on CARLOS |
| + |
| +titogo wants an account to be **served** in more than one placement: a |
| +second instance of the same account, near the buyer, holding a one-way |
| +mirror, selling against a lease from home and draining orders back. It |
| +asked for three things: replica placements on a route so a router in one of |
| +those placements serves locally instead of forwarding; provisioning that |
| +replica; and a signed instance-to-instance channel. |
| + |
| +The ask is written against titogo's own router (`internal/carlos/router.go` |
| +— `remote()`, the hairpin, `tables.slugs`). **That is not the mechanism |
| +that carries account hosts on CARLOS.** Here `carlos-edge` owns host |
| +routing: the edge resolves the host in its own map, and only if there is no |
| +local upstream does it `backhaul` the request to the box that homes it — |
| +one HMAC-signed hop over the mesh key (`internal/edge/edge.go:1280-1301`, |
| +the call at `:1297`; `backhaul` at `:2507`; the home map in |
| +`internal/edge/homemap.go`). titogo's hairpin is dormant everywhere, |
| +because nothing writes `Route.Home` at HEAD. |
| + |
| +So the platform's answer to Ask 1 is an **edge** change, and "a replica |
| +never forwards" stops being a rule anybody has to honour: a box that |
| +resolves the host locally never reaches the backhaul call at all. The loop |
| +protection is structural, which is what titogo asked for and stronger than |
| +what it asked for. |
| + |
| +Everything below is additive. An account with no replicas is byte-identical |
| +to today at every layer: no new column has a non-empty value, no pass |
| +derives a new row, no request takes a new branch. |
| + |
| +## 2. The record |
| + |
| +Three additive fields on `provision.Record` |
| +(`internal/provision/provision.go`), each empty on every record written |
| +before it. |
| + |
| + Role string // "" (the serving home) or "replica" |
| + Replicas []string // HOME records only: the placements the account opts into |
| + Home string // REPLICA records only: the public host it mirrors |
| + |
| +**`Role`** is a closed vocabulary in `ValidSteering`'s shape: a `ValidRole` |
| +helper, checked at the door and again in `ValidateShape`, so a hand-written |
| +bucket record cannot mint a replica the console would have refused |
| +(`reconcile.go:711-719` re-runs `ValidateShape` every pass and protects a |
| +failing record rather than removing it) **(fable 4)**. |
| + |
| +**`Replicas`** is the account's opt-in and *is* the residence gate: replica |
| +placement is never inferred, never defaulted, and never derived from the |
| +record's own `Region`. Values are validated against `ValidRegion`, refused |
| +when empty or equal to the record's own `Region`, deduplicated, sorted, and |
| +capped at `MaxReplicas = 4` — each entry is a decision about where order |
| +data lives until a drain confirms, not a cache hint. |
| + |
| +**`Home`** is written when the replica record is created and never |
| +converged. Repointing a replica at a different home is delete-and-recreate, |
| +because what it serves is a mirror *of* something. |
| + |
| +`ValidateShape` rails, all of them refusals: |
| + |
| +- `Role` outside the vocabulary. |
| +- `Role == "replica"` with `Kind != "instance"`: a replica is a process |
| + holding a mirror, never a file tree and never a foreign upstream. |
| +- `Role == "replica"` with an empty `Home` or an empty `Region`. |
| +- `Role == "replica"` with a non-empty `Replicas`: no replica of a replica. |
| +- A `Replicas` entry that is empty, not a placement code, or the record's |
| + own `Region`; more than `MaxReplicas` of them. |
| +- A non-empty `Replicas` with `Steering != "latency"` — see §5, without |
| + which the feature silently serves nothing **(fable 2)**. |
| + |
| +The replica's own record is an ordinary socket-backed instance record whose |
| +`Region` is the replica's box, so the existing scope test mints it there |
| +and nowhere else, with no change to `inScope` or `everywhereClass` |
| +(`reconcile.go:319-336`, `:735-738`). |
| + |
| +**Channel.** A replica record carries no `Channel` of its own. `routeFor` |
| +reads the *home* record's channel when minting the replica's route, so a |
| +`carlos canary` or a channel PATCH on the home cannot leave the two |
| +instances on different channels with nothing to reconcile them |
| +**(fable 7)**. Version equality is a different question and is not |
| +promised: see §9. |
| + |
| +## 3. Routing: the replica serves the account's public host |
| + |
| +The replica instance has a route of its own — its own host, its own socket, |
| +its own database — provisioned exactly as any instance record is. What |
| +makes it *serve the account* is one extra row: |
| + |
| + route_aliases: alias = <the home record's host> |
| + host = <the replica's own host> |
| + role = "replica" |
| + |
| +on the replica's box only. |
| + |
| +`Edge.Reload` already folds `route_aliases` so an alias name resolves to |
| +the primary's own map entry — same proxy, same route row, same activator |
| +keying, same version header (`edge.go:751-756`). A request for the account |
| +host arriving on the replica's box therefore wakes and bills the *replica*, |
| +and never reaches `backhaul`. Routes beat aliases on a collision |
| +(`edge.go:747-749`), which is the right answer if the home ever lands on |
| +the same box. Aliases are inside the edge's 2-second reload fingerprint |
| +(`registry.go:2147-2148`), so the row takes effect on the next reload. |
| + |
| +**Why an alias and not a second `routes` row.** A second `routes` row for |
| +the public host cannot survive the passes that own that table, four |
| +separate ways **(fable 1)**: the instance reconcile pass retires it as |
| +"record gone" on the next tick (`reconcile.go:1075-1092`), the deploy pass |
| +finds it `Adoptable` and restarts it under a name no process answers to |
| +(`internal/release/adopt.go:816-819`), the activator keys children by |
| +`rt.Host` and would exec `live/<public host>` |
| +(`internal/activator/activator.go:701`, `backend_exec.go:804`), and |
| +`routeFor` derives the socket and DB path from the record's host, so a |
| +record-driven mint cannot produce "same socket, different host" |
| +(`reconcile.go:348-372`). |
| + |
| +**Who writes the alias.** `route_aliases` is wholly owned by |
| +`domains.ReconcileAliases`, which rebuilds the desired set from claims |
| +every pass and deletes every alias it did not derive |
| +(`internal/domains/aliases.go:363-369`). So that pass learns to derive |
| +replica aliases too: for each replica record whose `Region` is this box, it |
| +adds `home.Host -> replica.Host` with `role = "replica"` to `desired`. One |
| +writer keeps one sweep, and a replica record that disappears takes its |
| +alias with it on the next pass with no new retirement code **(fable 1)**. |
| + |
| +**The registry mark.** `route_aliases` gains a `role TEXT NOT NULL DEFAULT |
| +''` column (additive, the table's established migration shape). It exists |
| +for one reader: the edges-file writer, which today derives every entry from |
| +`reg.List()` and hard-codes `Role: "home"` (`internal/host/edgesfile.go:219`, |
| +aliases at `:269`) and so has no way to say what this row is **(fable 3)**. |
| + |
| +## 4. What the fleet is told |
| + |
| +The replica box publishes, in its edges file: |
| + |
| +- its replica instance's **own host** as `role: "home"` — it genuinely is |
| + homed there, and that is how the home instance and every other box reach |
| + it by ordinary backhaul; |
| +- the **account's public host** as `role: "replica"`. |
| + |
| +`foldHomes` drops any role other than `home` and `wildcard` |
| +(`homemap.go:563-564`), so the account host stays single-valued in every |
| +other box's home map: a third box still backhauls it to the real home, and |
| +the replica placement cannot capture the dial. `internal/edgesfile`'s |
| +`DeploymentRoutes` counts only `RoleHome && !Alias` (`fold.go:163`), so the |
| +mark also does not reserve the hostname against domain claims. Both are |
| +already true; the spec adds an explicit `case "replica": continue` with a |
| +comment in `foldHomes`, so the ignore is a decision rather than an accident |
| +**(fable 3)**. |
| + |
| +Publishing the public host as `role: "home"` on the replica box is |
| +specifically what must not happen: the fold would then have two dial |
| +targets and settle it by sort order (`homemap.go:566-584`), and a third box |
| +would backhaul the account's traffic to the replica. |
| + |
| +**Console side effect, stated rather than discovered:** `served_by` joins |
| +every role (`internal/console/instancesplacement.go:68-70`), so the account |
| +host will list the replica's box beside its home's. That is the answer Ask |
| +2 wanted — the home instance can see where its replicas are — and it is |
| +correct; it is recorded here so nobody reads it as a leak **(fable 3)**. |
| + |
| +## 5. DNS: the placement is inert without it |
| + |
| +My first draft claimed buyers already land on the nearest edge and no |
| +steering change was needed. That is wrong for exactly the hosts this |
| +feature is for **(fable 2)**. A host gets a per-edge latency record only |
| +if its record carries `steering: "latency"` (`steering.go:157-162`), it |
| +lies under the apps domain (`:177-184`), and the deployment has armed |
| +`CARLOS_STEERING_ZONE_ID` (`internal/host/steering_loop.go:219`). The |
| +create door defaults `latency` only for `kind: static` in the platform |
| +namespace (`apiinstances.go:827`), and titogo's instances are |
| +`kind: instance`. |
| + |
| +So: |
| + |
| +- A home record with a non-empty `Replicas` MUST carry |
| + `steering: "latency"`. `ValidateShape` refuses the combination and the |
| + PATCH door refuses to set `replicas` on a record that is not steered, |
| + with a message that says to set steering first. Without this the replica |
| + box mints its alias, receives no buyer, and looks correct from every box. |
| +- For an account reached through a **tenant-owned** domain or a custom |
| + domain, the tenant's DNS must CNAME to `edge.<apps-domain>`. The platform |
| + cannot do this and does not pretend to; the spec states it, and §11 keeps |
| + custom domains out of the first landing. |
| +- **Deployment prerequisite:** `CARLOS_STEERING_ZONE_ID` is not armed by |
| + any root module in `infra/` today (only `modules/region-host` declares |
| + the variable, `variables.tf:153`). Until it is, replica placements are |
| + inert on the flagship. That is an infrastructure change and needs its own |
| + scoped approval (AGENTS.md, "Production infrastructure needs scoped |
| + approval"); it is filed as its own discussion and is not part of this |
| + branch. |
| + |
| +Steering itself needs no code change: once the record is steered, every |
| +armed edge already answers for it, and §3 decides what the nearest edge |
| +does when the request lands. |
| + |
| +## 6. Doors: what the console refuses |
| + |
| +Every rule here is a cross-record invariant, which is new for this API — |
| +the create and PATCH doors validate a record against itself and its |
| +manifest today (`apiinstances.go:550-905`, `:1041`). Nothing in titogo can |
| +enforce them and no reconcile pass will **(fable 4)**. |
| + |
| +**Create `role=replica`** requires: a `home` naming an existing record in |
| +the same account and app whose own `Role` is `""`; a `region` that is |
| +non-empty, differs from the home's, is listed in the home's `Replicas`, |
| +passes the manifest allow-list (`instanceRegionAllowed`, `:1573`) and the |
| +armed-and-hosting gate (`instancesplacement.go:173-202`); and a home |
| +already carrying `steering: "latency"`. The host is minted by the platform |
| +as a name of its own under the app's instance domain — not a secret, and |
| +the spec does not lean on unguessability for anything, since the name is |
| +published in the box's edges file by design **(fable 11)**. |
| + |
| +**PATCH `replicas`** on a home record: validated as in §2, and a placement |
| +may not be removed while a replica record still exists there. The opt-in |
| +and the data must not disagree — that is the residence failure the field |
| +exists to prevent. |
| + |
| +**Delete** a home whose `Replicas` is non-empty, or for which any replica |
| +record exists, is refused with a 409 naming them **(fable 6)**. Deleting |
| +the *replica* is titogo's to sequence once its drain confirms; the platform |
| +cannot see drain state. What the platform leaves behind when a replica |
| +record goes — the instance's database file, its hibernate prefix in the |
| +box's bucket, its env file — is what it leaves behind for any deleted |
| +instance today, which is everything, forever (the only DB removal is |
| +failed-wake cleanup, `backend_exec.go:315-327`). Stated, not fixed, here. |
| + |
| +**Move** (`MoveRequest`, `provision.go:797`) is refused on a replica record |
| +— repointing is delete-and-recreate by §2's own rule — and a move of a home |
| +into one of its own replica placements is refused too, or the home's route |
| +and the replica's alias collide on one box **(fable 4)**. |
| + |
| +**Tokens.** `handleAPIInstanceToken` mints an app-wide token |
| +(`apiinstances.go:1392`, amadan #155), so a replica holding one could |
| +delete its own home. A replica needs no console API — §7 hands it |
| +everything it needs as data — so the door refuses to mint a token for a |
| +replica record, and only the home lists its replicas **(fable 8)**. |
| + |
| +## 7. Sibling addressing (Ask 3): no new channel |
| + |
| +An instance reaches a sibling by dialling **that box's EIP with the real |
| +hostname as SNI**, which is exactly what backhaul does already |
| +(`backhaulProxy`, `edge.go:2664-2700`; `backhaulTransport`, `:2589-2660`), |
| +and which bypasses the local edge — so a replica can reach home through the |
| +account host without landing on its own alias. The home edge presents the |
| +mirrored public certificate. No private CA, no new listener, no relay |
| +change. Refusing to carry an app-defined call over titogo's |
| +`/__internal/connect/` exception is deliberate: that path forwards exactly |
| +one shape and fails closed, and widening it would make a narrow, verified |
| +door general. |
| + |
| +The platform supplies only what an instance cannot know. A **siblings |
| +file**, written by the agent beside the instance's own data and refreshed |
| +every reconcile pass, with its path handed to the child as |
| +`CARLOS_SIBLINGS_FILE` in `ExecBackend.agentOwnedVars` |
| +(`backend_exec.go:977-994`) — the seam that already injects `CARLOS_REGION` |
| +**(fable 5)**. A file rather than more environment variables because the |
| +child's environment is composed at spawn (`instanceEnv`, `:1135`): adding a |
| +replica would otherwise require bouncing a running home instance to tell it. |
| + |
| + {"role":"home","host":"acme.apps.example", |
| + "replicas":[{"host":"r7x2.apps.example","placement":"uk", |
| + "addr":"203.0.113.9:443","key":"<base64>"}]} |
| + |
| + {"role":"replica","host":"r7x2.apps.example","placement":"uk", |
| + "home":{"host":"acme.apps.example","addr":"198.51.100.4:443", |
| + "key":"<base64>"}} |
| + |
| +Written 0600, owned by the instance user, atomically replaced. |
| + |
| +**The key** is per *pair*, not per account: `HMAC(meshKey, |
| +"replica/" + account + "/" + homeHost + "/" + replicaHost)` **(fable 5, |
| +adapted)**. Fable proposed per-instance keys the home derives itself; the |
| +home cannot, because it must never hold the mesh key — so the platform |
| +derives each pair key and writes it into both sides' files. A compromised |
| +replica can then speak only as itself, never as a sibling. The mesh key |
| +never leaves the agent (`internal/edge/mesh.go:92-95`), HMAC is a PRF so |
| +the derivative does not expose it, and the agent already holds the pair in |
| +memory (`homemap.go:675-683`). |
| + |
| +**Rotation.** A mesh rotation changes every derived key. The platform |
| +rewrites both siblings files on the next pass after `.next` is promoted, |
| +and because the file is re-readable the app can pick the new key up without |
| +a restart — provided titogo re-reads it on a verification failure. The spec |
| +requires that of titogo and says plainly that a rotation is otherwise a |
| +restart-everything event. No `_NEXT` companion in the first landing |
| +**(fable 13)**. |
| + |
| +titogo signs and shapes the lease and drain calls itself over that address |
| +and that key. The protocol stays entirely theirs. |
| + |
| +## 8. Certificates: nothing to build, worth writing down |
| + |
| +The replica box serves a name it does not home, which looked like the |
| +design's weakest point and is not **(fable 9)**. Every edge mirrors the |
| +*whole* `control/certs/` prefix rather than the names it owns |
| +(`internal/edge/certsync.go:621-650`; `ownsCertName` at `:1105-1121` gates |
| +only the backfill writer), and the TLS hello is answered for any name the |
| +mirror holds (`wrapGetCertificate`, `:1346-1395`). Issuance and renewal |
| +stay with the housekeeper, whose order gate is `Allowed || KnownHost` |
| +(`certsync.go:191-201`), and the home still publishes the host, so nothing |
| +moves. One nuance: on the replica box `Allowed(<public host>)` becomes true |
| +once the alias exists (`edge.go:968`), so that box may order for the public |
| +host if it is ever the housekeeper — same issuer, same account, same rate |
| +bucket, harmless. |
| + |
| +## 9. What the platform does not promise |
| + |
| +- **Version equality.** Adoption runs per box on its own cadence |
| + (`adopt.go:303-360`), so a home and its replica will differ by one |
| + release for minutes after any deploy, in either direction. titogo's lease |
| + and drain protocol must be version-tolerant; the platform will not |
| + serialise a fleet deploy to make two instances land together **(fable 7)**. |
| +- **Wake-free leases.** A request through the alias wakes the replica and |
| + bills it, and the idle clock is the replica's (`edge.go:668`, |
| + `:751-756`). Home's lease renewals will wake a sleeping replica every |
| + time, so titogo sizes its renewal interval against |
| + `CARLOS_HIBERNATE_IDLE` or accepts that a replica never sleeps while a |
| + lease is live **(fable 10)**. |
| +- **Drain safety.** The platform cannot see whether a replica holds |
| + undrained orders; §6's refusals protect the home, not the replica. |
| + |
| +## 10. Tests |
| + |
| +Each of these fails before its change and passes after: |
| + |
| +1. `ValidateShape` refuses each rail in §2, and accepts an empty-everything |
| + record unchanged. |
| +2. The console door refuses: a replica of a replica, a placement not in the |
| + home's `Replicas`, an unarmed placement, a home that is not steered, |
| + removing a placement under a live replica, deleting a home with |
| + replicas, moving a replica, and minting a token for one. |
| +3. `ReconcileAliases` derives the replica alias on the replica's box, does |
| + not derive it anywhere else, and — the regression this design most |
| + needs — **a replica alias survives a full pass** rather than being swept |
| + as underived **(fable 13)**. |
| +4. The edges-file writer publishes the public host as `role: "replica"` and |
| + the replica's own host as `role: "home"`. |
| +5. `foldHomes` on a third box has exactly **one** dial target for the |
| + account host, with the replica box publishing its entry. |
| +6. The edge resolves the account host on the replica box to the replica's |
| + own upstream, with no backhaul attempt (the existing backhaul test's |
| + inverse). |
| +7. The siblings file is written for both sides with the same pair key, 0600, |
| + and refreshed when the replica set changes. |
| +8. An account with no replicas produces byte-identical edges-file, |
| + registry-row and route-table output to before the change. |
| + |
| +## 11. Scope of the first landing |
| + |
| +**In:** the record fields and their rails; the console doors; the replica |
| +alias and its derivation; the `route_aliases.role` column; the edges-file |
| +mark and the explicit `foldHomes` case; the siblings file and its keys; the |
| +list/get/CLI surface (`role`, `home`, `replicas`, a role column, and |
| +`docs/site/cli.md` for any new verb, per AGENTS.md). |
| + |
| +**Out, deliberately:** custom domains on the replica box — an alias for a |
| +custom domain is only derived when its primary is a local route |
| +(`aliases.go:95-100`), so it needs `ReconcileAliases` work *and* a tenant |
| +DNS change, and the account host is enough to prove the feature |
| +**(fable 13)**. Also out: `_NEXT` keys, a platform-side drain or mirror |
| +(titogo reuses its sandbox sync), an AU box, and any change to the lease |
| +protocol. |
| + |
| +**Prerequisite, not in this branch:** arming `CARLOS_STEERING_ZONE_ID` in |
| +the deployment (§5). |
| + |
| +## 12. Declined findings |
| + |
| +- **fable 5's per-instance key derived by home.** Declined as written: it |
| + requires the home instance to hold the mesh key. Replaced by per-pair |
| + keys the platform derives for both sides, which has the same blast |
| + radius. |
| +- **fable 13's suggestion to cut the PATCH armed-placement check.** |
| + Declined: the check is three lines and the failure it prevents (an |
| + opt-in naming a placement nothing can serve) is invisible until a buyer |
| + hits it. |