amadan
#162: spec revision 2 — the mark moves to the route row, and the channel rule learns to mint
Summary
round 2's findings (2 P1) committed unedited, and the revision that answers them. ReconcileAliases cannot read bucket records and cannot import the package that can, so the replica mark becomes routes.replica_of, written by routeFor and read by both the alias derivation and the edges file. The channel rule now survives Reconcile: an 'all' map keeps the home record the scope filter drops, and a replica whose home is missing is protected rather than minted on a guess.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2 files changed,
+684
−128
FABLE-FINDINGS-162-spec.md+384 −0docs/superpowers/specs/2026-09-21-replica-placements-design.md+300 −128
CI passed — run details
diff --git a/FABLE-FINDINGS-162-spec.md b/FABLE-FINDINGS-162-spec.md| new file mode 100644 |
| index 000000000..27b7b7f38 |
| --- /dev/null |
| +++ b/FABLE-FINDINGS-162-spec.md |
| @@ -0,0 +1,384 @@ |
| +# Findings: replica placements spec (#162), round 2 |
| + |
| +Second reader's findings against commit 5dab29d4 in this worktree, read |
| +against `docs/superpowers/specs/2026-09-21-replica-placements-design.md`, |
| +`FABLE-REVIEW-BRIEF-162-design.md` and `FABLE-FINDINGS-162-design.md`. The |
| +code is identical to round 1's d6810d99 (the only commit between them adds |
| +the spec and the findings), so round 1's line numbers still hold. Every |
| +citation below was read in this checkout. titogo's record was read at |
| +c6ed556e0. |
| + |
| +Severity: P1 blocks implementation, P2 must be answered before the plan, |
| +P3 worth knowing. |
| + |
| +Summary: the spec is a faithful and mostly correct answer to round 1. The |
| +routing primitive (a `route_aliases` row on the replica box, folded by |
| +`Edge.Reload`) does what §3 claims. What does not survive contact with the |
| +code is HOW the row gets there and HOW the replica's route gets its channel: |
| +`ReconcileAliases` cannot see records at all (and cannot import the package |
| +that can), and the reconcile pass discards out-of-scope records before |
| +`routeFor` runs and protects any record with an empty channel. Both are |
| +fixable with one registry column on `routes`, which also makes the |
| +`route_aliases.role` column unnecessary. Beyond that: the edges-file mark is |
| +gated on a DNS stamp the spec never mentions, the token refusal names a door |
| +that has no record to refuse on, the "not the record's own region" rail is |
| +inert for a home in the deployment's home region, and the siblings-file |
| +writer is named ("the agent") without any of its three inputs. |
| + |
| +--- |
| + |
| +## Part A. Disposition of round 1 |
| + |
| +| # | Round 1 finding | Spec's disposition | Holds against the code? | |
| +|---|---|---|---| |
| +| 1 | Second `routes` row cannot survive; use `route_aliases`, teach `ReconcileAliases` | Accepted (§3) | Alias primitive: yes (`edge.go:746-755`). Derivation: **no**. `ReconcileAliases` has claims, `reg.List()` and `reg.ListAliases()` only (`aliases.go:68-81`); it has no records and cannot import `provision` (`reconcile.go:26` imports `domains`; cycle noted at `aliases.go:29-33`). See B1. | |
| +| 2 | Steering is not "no change"; require `steering: latency`, state tenant DNS, zone prerequisite | Accepted (§5) | Yes. `steering.go:157-184`, `steering_loop.go:219`, `variables.tf:153` all as cited. The sub-ask "say which flagship hosts this applies to today" was dropped. | |
| +| 3 | Edges-file writer cannot publish role `replica`; add a registry mark | Accepted (§3 role column, §4 mark, explicit `foldHomes` case) | Fold and `DeploymentRoutes` halves: yes (`homemap.go:562-584`, `fold.go:163`). Publication half: **only partly**. `edgesFilePass` publishes an alias only when `dns_status == "pointing"` (`edgesfile.go:249-258`), stamped by a different pass. §4 says "publishes" unconditionally. See B3. | |
| +| 4 | Cross-record invariants unstated; add rails to `ValidateShape` and doors | Accepted (§2, §6) | Yes, with one inert rail: "not the record's own Region" cannot fire for a home in the deployment's home region, whose `Region` is `""`. See B5. | |
| +| 5 | Mesh-derived key: delivery seam, rotation, scope | Accepted-adapted (§7: siblings file, per-pair keys, no `_NEXT`) | Seam: yes, `agentOwnedVars` (`backend_exec.go:977-994`) reaches both spawn paths (`:895`, `:1055`) and wins over file env. Writer: **not worked out**; the spec names "the agent" and "every reconcile pass" but the reconcile pass holds none of the three inputs the file needs. See B6. Rotation: partly, see B11. | |
| +| 6 | Deletion ordering | Accepted (§6) | Yes. `handleAPIInstanceDelete` reads nothing today (`apiinstances.go:1288-1345`); adding a `ListRecords` is one call. Two edges unstated, see B13. | |
| +| 7 | Home and replica must share a channel; versions will differ | Accepted (§2 channel rule, §9) | **No.** `reconcile.go:726` protects any record with an empty `Channel` (never minted); `routeFor(rec, dataDir, seed)` (`:348`) has no second record; `recs` is in-scope only (`:654`, `:734-736`) so the home record is not retained; `convergeOwned` converges the row back to `rec.Channel` every pass (`:573-580`). See B2. | |
| +| 8 | App token is app-wide; give it to home only | Accepted (§6 "the door refuses to mint a token for a replica record") | **No.** The token door is `POST /api/cli/apps/{account}/{app}/instance-token` (`authcli.go:636`), gated on `apiAccountApp` (`apiinstances.go:1366-1371`). It takes no host and reads no record; there is nothing at that door to refuse on. See B4. | |
| +| 9 | Certificates sound; write it down | Accepted (§8) | Yes. `certsync.go:621`, `:1105`, `:1346`, `:191-200`, `edge.go:968` as cited. | |
| +| 10 | Hibernation through the alias; leases wake replicas | Accepted (§9) | Yes. `edge.go:668` is the managed-hibernation decision, close enough. | |
| +| 11 | "Unguessable" is not a secret | Accepted (§6) | Yes as prose; the mint itself is unspecified. See B8. | |
| +| 12 | Console and CLI surface | Accepted (§11) | Yes. | |
| +| 13 | Scope cuts | Accepted-adapted (custom domains and `_NEXT` cut; PATCH armed check kept) | Yes. The three "do not cut" items are all in §11. | |
| + |
| +**The two explicit declines (§12).** Both are right. Round 1's "home can |
| +derive every replica's key from the list it already receives" was round 1's |
| +error: the home holds no mesh key, so it cannot derive anything; the spec's |
| +per-pair keys derived by the platform and written to both sides are the |
| +correct adaptation. Keeping the PATCH armed-placement check is a judgement |
| +call and the spec's reasoning is sound. |
| + |
| +**Silently dropped.** Only round 1's request to name which flagship hosts |
| +are affected today (finding 2), and finding 5's note that whoever writes the |
| +key material needs the pair handed down from the edge (the spec moved the |
| +writer and lost the seam; B6). |
| + |
| +--- |
| + |
| +## Part B. New findings |
| + |
| +### B1. P1. `ReconcileAliases` cannot derive the replica alias from records. The spec's "one writer" needs a mark on the ROUTE row, not on `route_aliases`. |
| + |
| +§3: "for each replica record whose `Region` is this box, [`ReconcileAliases`] |
| +adds `home.Host -> replica.Host` with `role = "replica"` to `desired`." |
| + |
| +`ReconcileAliases(ctx, s, reg, now, deploymentRoutes, cache)` reads claims |
| +through the cache, local routes and existing aliases (`aliases.go:68-81`). |
| +It never lists `apps/`. It cannot be taught to: `internal/provision` |
| +imports `internal/domains` (`reconcile.go:26`), and `aliases.go:29-33` and |
| +`claims.go:240` both record that importing back would be a cycle. Its two |
| +callers (`agent.go:1800`, `:2336`) have no record list in scope either; the |
| +records were read and discarded inside `provision.Reconcile` a few lines |
| +earlier. |
| + |
| +Two implementers would build different things here: one re-lists the bucket |
| +in `agent.go` and threads a `[]ReplicaPair` argument in (a second full |
| +`apps/` listing on the seconds-cadence fast leg); the other adds a routes |
| +column. Only the second keeps every property the spec claims. |
| + |
| +What to do: give `routes` an additive column, say `replica_of TEXT NOT NULL |
| +DEFAULT ''` (the `ALTER TABLE routes` shape at `registry.go:443-452`), that |
| +`routeFor` writes from `rec.Home` for a `role=replica` record and |
| +`convergeOwned` holds steady. Then: |
| + |
| +- `ReconcileAliases` derives `rt.ReplicaOf -> rt.Host` from `reg.List()` |
| + with no new input and no cycle, and the alias is swept the pass after the |
| + instance pass retires the route (`reconcile.go:1075-1092`), which is the |
| + retirement §3 promises. |
| +- `edgesFilePass` already has the routes in hand (`edgesfile.go:174`) and |
| + can mark an alias `role: "replica"` when `byHost[a.Host].ReplicaOf == |
| + a.Alias`, so the `route_aliases.role` column of §3 is not needed at all, |
| + and `UpsertAlias` (`registry.go:1841-1848`) needs no change. |
| +- `everywhereRoute`/`inScope` read the row, not the record, so the removal |
| + phase is unaffected. |
| + |
| +The spec should replace §3's "registry mark" paragraph with this, or state |
| +the `[]ReplicaPair` argument and accept the second listing. It should not |
| +leave the choice to the plan: it decides the schema. |
| + |
| +### B2. P1. The channel rule cannot mint a replica route as written. |
| + |
| +§2: "A replica record carries no `Channel` of its own. `routeFor` reads the |
| +home record's channel when minting the replica's route." |
| + |
| +Three things in `Reconcile` stop that: |
| + |
| +- `reconcile.go:726-730`: a record with `Channel == ""` is `protect`ed and |
| + skipped before scope is even asked. A channel-less replica record never |
| + mints, on any box, forever, with one log line per pass. |
| +- `reconcile.go:654` (`recs` is "host -> record, in-scope only") and |
| + `:734-736` (`continue` before `recs[rec.Host] = rec`): the home record, |
| + whose `Region` is the home box, is decoded on the replica box (`s.Get` at |
| + `:668`) and then dropped. By the time `routeFor` runs (`:348`, signature |
| + `(rec, dataDir, seed)`), there is no home record to read. |
| +- `convergeOwned` (`:573-580`) converges `existing.Channel` to `rec.Channel` |
| + whenever the record's is non-empty. If the door writes a channel onto the |
| + replica record (as it does today, `apiinstances.go:770-779`), a mint that |
| + copied home's channel is converged back to the record's own on the next |
| + pass. |
| + |
| +What to do, all three together: keep a second map of every decoded record |
| +(`all`, keyed by host, filled before the scope check) and hand |
| +`all[rec.Home]` to both `routeFor` and `convergeOwned` for a `role=replica` |
| +record; exempt `role=replica` from the `:726` guard, or have the door write |
| +the home's channel at create as a fallback and let the pass override it; |
| +and state what the replica box does when the home record is absent, |
| +malformed or protected this pass (protect the replica too; never mint on a |
| +guess). Say which of the two the record carries in the bucket, because |
| +`Channel` is not `omitempty` (`provision.go:49`) and the list/get views show |
| +it. |
| + |
| +### B3. P2. The edges-file mark is gated on a DNS stamp the spec never mentions. |
| + |
| +§4: the replica box "publishes ... the account's public host as |
| +`role: "replica"`". `edgesFilePass` publishes an alias only when |
| +`a.DNSStatus == "pointing"` (`edgesfile.go:249-258`, with a once-per-name |
| +"not published" log). That stamp is written by the domains sweep |
| +(`domains/reconcile.go:465-472`) from `dnsStatus` (`:512-527`): resolve the |
| +alias, "pointing" if any A record is in `cfg.EdgeIPs()` (PLATFORM_EDGE_IPS), |
| +"elsewhere" otherwise, and the sweep runs only with `cfg.Resolve` and |
| +`cfg.EdgeIPs` set. |
| + |
| +So the mark reaches the file on the sweep's cadence plus the edges loop's |
| +5-minute tick, and only if the public host resolves to an address in |
| +PLATFORM_EDGE_IPS on the replica box. On a box whose list is stale, or on a |
| +deployment where the host resolves elsewhere (which, per §5, is every host |
| +until the zone is armed), the alias is withheld, `served_by` never shows the |
| +replica box, and §10 test 4 fails for a reason unrelated to the feature. |
| + |
| +The gate's stated rationale (an unbounded tenant-chosen name set the shared |
| +issuer would order for) does not apply here: the public host is a |
| +platform-provisioned route on the home box. Say that a replica alias is |
| +published regardless of `dns_status` (the derivation in B1 knows which rows |
| +those are), or accept the gate and state the lag and the PLATFORM_EDGE_IPS |
| +dependency in §4. |
| + |
| +### B4. P2. The token refusal names a door that has no record to refuse on. |
| + |
| +§6 "Tokens": "the door refuses to mint a token for a replica record." |
| +`handleAPIInstanceToken` is registered at `POST |
| +/api/cli/apps/{account}/{app}/instance-token` (`authcli.go:636`), gated on |
| +`apiPerson` and `apiAccountApp` (`apiinstances.go:1366-1371`), and mints |
| +`NewAppToken(acct.ID, app, ...)` (`:1392`). No host in the path, no record |
| +read, no way to know which process will hold the result. |
| + |
| +Restate the answer as what it can be: the platform mints app-wide tokens |
| +only (amadan #155 is the fix, not this branch); a replica must not be |
| +configured with one, which is titogo's to honour; and the home-delete 409 |
| +of §6 is the platform-side guard that actually holds if it is. Drop |
| +"refuses to mint a token for a replica record" from §6 and from §10 test 2. |
| + |
| +### B5. P2. "Not the record's own Region" is inert for a home in the deployment's home region, and the create door's placed-app override is unaddressed. |
| + |
| +The console normalises the home region's code to `""` before it reaches a |
| +record (`homeRegionIn`, `apiinstancemove.go:136-141`; used at |
| +`apiinstances.go:710`). So a home in the deployment's home region has |
| +`Region == ""`, and §2's `ValidateShape` rail "a `Replicas` entry ... [that |
| +is] the record's own `Region`" compares a placement code against `""` and |
| +never fires. A member can list the home region itself as a replica |
| +placement; the create door's "differs from the home's" check has the same |
| +hole if written against `rec.Region`. |
| + |
| +`ValidateShape` cannot know the deployment's home-region code; only the |
| +console can (`s.cfg.HomeRegion`). Say that this rail lives at the door, |
| +compared through `homeRegionOut`, and that `ValidateShape` refuses only the |
| +literal own-region case it can see. |
| + |
| +Same door, one line up: `if placed != "" { region = placed }` |
| +(`apiinstances.go:711-712`) overrides the caller's region for a fleet-placed |
| +app. §6 must refuse `role=replica` on a placed app, or say what a replica of |
| +a placed app means. |
| + |
| +### B6. P2. The siblings-file writer is named without any of its inputs, its path, or its write primitive. |
| + |
| +§7 says "written by the agent beside the instance's own data and refreshed |
| +every reconcile pass." `provision.Reconcile` has none of what the file |
| +needs: |
| + |
| +- **Records on both sides.** The home's file lists replica records whose |
| + `Region` is another box; the replica's file needs the home record. Both |
| + are out of scope on the box that writes the file (B2's `all` map fixes |
| + this too). |
| +- **The sibling's dial address.** That is the sibling box's EIP, which |
| + lives in the edge's home map (`homeMap.Home`, `homemap.go:611-625`, |
| + returns `eip, box`) or in the edges-file fold (`internal/edgesfile` |
| + exports `EdgeIPs`, `DeploymentRoutes`, `Parse`, `Key`; no per-box EIP |
| + lookup today). `Home()` refuses its own box (`m.self`), which is fine |
| + because §6 refuses co-location. |
| +- **The mesh key pair.** `homeMap.meshKeys()` is unexported |
| + (`homemap.go:693-696`) and edge-owned. The agent reaches the edge only |
| + through the edge-ready callback (`agent.go:510`; the `upstreamSource` |
| + pattern at `:2490-2510` is the precedent for publishing an edge method to |
| + a loop that started before the edge existed). |
| + |
| +And the path: "beside the instance's own data" is `pathsFor(rt).dataDir` |
| +(`activator/paths.go:38-42`), which is `/data/instances/<id>` for a nested |
| +row and `/data` for a flat one (`provision.InstanceDir`, `provision.go:1084`; |
| +`internal/host/schedule_wiring.go:193-203` already computes both for the |
| +schedule wiring and is the precedent to cite). For an isolated instance |
| +that is the ONLY writable `/data` path inside its namespace |
| +(`carlos-instance@.service`: `BindPaths=/data/instances/%I`, |
| +`TemporaryFileSystem=/data:ro`), so a file written anywhere else in `/data` |
| +is invisible to the process that must read it. The instance directory is |
| +tenant-writable ground, so the write must go through |
| +`instancefs.WriteAtomic` (`prepare.go:154` is the precedent), never a plain |
| +path open. Ownership: both the agent (`carlos-edge.service:28`, |
| +`User=carlos`) and the instance (`carlos-instance@.service`, `User=carlos`) |
| +run as `carlos`; there is no per-instance uid, isolation is the mount |
| +namespace. "Owned by the instance user" should say that, and 0600 then |
| +works. |
| + |
| +`CARLOS_SIBLINGS_FILE` through `agentOwnedVars` is the right seam: |
| +`withVar` overrides the file env (`backend_exec.go:990`), `childEnv` strips |
| +every `CARLOS_*` from the agent's own environment (`:1177-1180`), and both |
| +`startChild` (`:895`) and `startUnit` (`:1055`) go through it. Say the |
| +variable is stripped (`withoutVars`) when no file was written, like |
| +`CARLOS_REGION`. |
| + |
| +Unstated and needed by the plan: what the entry looks like for a replica |
| +whose box has not yet published an edges file (no `addr`; omit the entry or |
| +write `"addr":""`), and that the file is written for the home even when |
| +`replicas` is empty (so the app can tell "no replicas" from "no file"). |
| + |
| +### B7. P2. A console build that predates the fields strips them on PATCH. |
| + |
| +`UpdateRecord` marshals the `Record` struct it was handed |
| +(`provision.go:928-933`); `DecodeRecord` (`maintenance.go:38-78`) walks the |
| +raw keys only to police `maintenance`, then `json.Unmarshal`s into the |
| +struct. Only `maintenance` is protected (`protectedFieldsEqual`). A PATCH |
| +door on a console built before `Role`/`Home`/`Replicas` exist reads the |
| +record without them and writes it back without them: `replicas` on a home |
| +vanishes (bypassing §6's "may not be removed while a replica exists"), |
| +and `role`/`home` on a replica vanish, which under B1 clears `replica_of` |
| +and sweeps the alias on the next pass, silently. |
| + |
| +Tito's console is pinned on an older build today (a deliberate pin, see the |
| +store-boundary outage). State a rollout rule in §11: no replica record may |
| +be created for an app until every console that can PATCH that app's records |
| +is on a build carrying the fields. Or add the three fields to |
| +`protectedFieldsEqual` so an old writer's PATCH is refused with |
| +`ErrMaintenanceProjection` rather than accepted. |
| + |
| +### B8. P2. How the replica's host is minted is unspecified. |
| + |
| +The create door requires `req.Host` and refuses anything but one label under |
| +`m.Domain` (`apiinstances.go:781-786`); `Slug` is derived from it (`:777`). |
| +§6 says the host "is minted by the platform as a name of its own under the |
| +app's instance domain." Neither the generator (length, alphabet, source of |
| +randomness), the collision rule (`CreateRecord` already 409s on a race, |
| +`provision.go:31-33`; retry or refuse?), nor what happens when the caller |
| +sends `host` with `role=replica` (refuse, ignore, honour?) is stated. The |
| +CLI shape for `carlos instances create --role replica --home <host> |
| +--region <code>` and the response body (the caller needs the minted name) |
| +belong in §11. |
| + |
| +### B9. P3. `role` is not in the edge's fingerprint, and `UpsertAlias` would not write it. |
| + |
| +If the spec keeps `route_aliases.role` (B1 argues it should not), |
| +`Fingerprint` hashes `alias, host, account, app` only |
| +(`registry.go:2147-2148`) and `UpsertAlias`'s `ON CONFLICT` sets |
| +`host, account, app` only (`:1841-1848`). The first is fine and should be |
| +said (the edge does not read `role`; the edges file carries it and rewrites |
| +on its own change gate, `edgesfile.go:340-348`). The second means a row that |
| +existed before the column, or was ever upserted without it, stays at `''`. |
| + |
| +### B10. P3. Turning steering off on a home with replicas: say what happens. |
| + |
| +§5 says the PATCH door refuses to set `replicas` on an unsteered record. The |
| +reverse case is the common one: `carlos steering ... off` on a home that |
| +already has replicas. The PATCH door runs `ValidateShape` (`apiinstances.go: |
| +1025-1027` describes the contract), so with §2's rail it is a 400; the door |
| +comment "Turning steering OFF is never blocked" (`:1206-1207`) becomes |
| +false. State the refusal and its message ("remove the replicas first"), so |
| +an implementer does not instead clear `Replicas` to make the request |
| +succeed. |
| + |
| +### B11. P3. Rotation: the two siblings files move on two cadences. |
| + |
| +§7: "the platform rewrites both siblings files on the next pass after |
| +`.next` is promoted." Each box's `refreshKeys` sees `.next` and the |
| +promotion on its own poll (`homemap.go:675-683`); each box's pass rewrites |
| +its own file on its own cadence (seconds on a serial bump, 5 minutes |
| +otherwise). So for up to one cadence the home's file and the replica's file |
| +hold different pair keys, and "re-read on verification failure" cannot help |
| +the side whose file has not been rewritten yet. Also say which mesh key the |
| +derivation uses during the window when both are mirrored: `signingKey` |
| +prefers `.next` the instant it exists (`mesh.go:202-207`), so deriving from |
| +`signingKey(current, next)` moves the pair key at mirror time, not at |
| +promotion, and the spec's sentence is wrong by one step either way. State |
| +the window as the guarantee (bounded by the pass cadence, calls fail closed |
| +inside it), or write both derivatives to the file, which is the `_NEXT` |
| +companion §11 cut. |
| + |
| +### B12. P3. "No bounce to add a replica" only holds if the home re-reads the file on a timer. |
| + |
| +§7 justifies the file over env with "adding a replica would otherwise |
| +require bouncing a running home instance." The only re-read the spec asks of |
| +titogo is "on a verification failure." A home that never fails a |
| +verification never learns of a new replica. Say the app re-reads the file |
| +on every use or on a short timer, and put it in the contract §7 hands |
| +titogo. |
| + |
| +### B13. P3. The delete and PATCH refusals: truncation and read failure. |
| + |
| +Both new refusals need `provision.ListRecords(ctx, s, sqid, app, |
| +instanceListMax)` with `instanceListMax = 500` (`apiinstances.go:57`) and a |
| +`truncated` return (`:416`). An app with more than 500 records could hide a |
| +replica from the check. And the delete door reads nothing from the store |
| +today (`:1288-1345`, "SHAPE ONLY, always"); a list that fails must either |
| +refuse the delete (fail closed, the residence-safe answer) or allow it (the |
| +"deletion must outlive every policy" answer the door's own comment argues |
| +for). Say which, and say the truncated case refuses. |
| + |
| +### B14. P3. Two console surfaces will show the alias as something it is not. |
| + |
| +`instancepage.go:151-159` lists every `route_aliases` row whose `Host` is |
| +the instance as an attached domain of that instance, from the console box's |
| +own registry; on a replica homed on the console's box the public host will |
| +appear as a "domain" of the replica. `agent.go:2779-2784` carries alias |
| +DNS/cert stamps into the box status snapshot. Neither breaks anything; both |
| +should be listed beside §4's `served_by` note as known cosmetic effects, or |
| +filtered by the B1 mark. |
| + |
| +### B15. P3. titogo's record names no platform contract. |
| + |
| +titogo's design record (c6ed556e0 §"What the platform already has") asks for |
| +"the sandbox-provisioning verb with a different role" and nothing else; it |
| +names no environment variable and no file. `CARLOS_SIBLINGS_FILE`, the two |
| +JSON shapes, the pair-key label, the "re-read on failure" and B12's re-read |
| +rule, and §9's three non-promises are the contract. §7 should say that the |
| +titogo record is to be updated to name them, and that their lease and drain |
| +signing is done with the `key` field of that file and nothing else. |
| + |
| +--- |
| + |
| +## Citations checked |
| + |
| +Every file:line in the spec was read. None is wrong in a way that misleads; |
| +the ones that are approximate: `edge.go:668` is the managed-hibernation |
| +decision (`hib := ...`) rather than `up.hibernating`; `homemap.go:563-564` |
| +is the `default: continue` at `:562-563`; `edge.go:751-756` is the alias |
| +fold at `:746-755`; `instancesplacement.go:68-70` is `:69-71`. The claim in |
| +§7 that "the agent already holds the pair in memory (`homemap.go:675-683`)" |
| +is true of the process (the edge runs inside `carlos agent`, `agent.go:510`) |
| +but not of any code path the reconcile pass can reach; that is B6. |
| + |
| +## Where the spec is sound and needs no further word |
| + |
| +- §1 and the structural loop protection (`edge.go:1280-1300`, `:2507`). |
| +- §3's choice of the alias primitive over a second route row, and all four |
| + reasons given for it. |
| +- §4's fold argument and the explicit `foldHomes` case. |
| +- §5 in full, including the zone prerequisite being filed separately. |
| +- §6's create rails other than B5, the delete 409, and the move refusals. |
| +- §7's dial-the-EIP-with-real-SNI answer and the refusal to widen titogo's |
| + `/__internal/connect/` path; the per-pair key derivation and its blast |
| + radius. |
| +- §8 in full. |
| +- §9's three non-promises. |
| +- §10's test list, once B1 to B4 change what tests 3, 4 and 2 assert. |
| +- §11's cuts. |
diff --git a/docs/superpowers/specs/2026-09-21-replica-placements-design.md b/docs/superpowers/specs/2026-09-21-replica-placements-design.md| index 1ae75ce35..c859b4a5e 100644 |
| --- a/docs/superpowers/specs/2026-09-21-replica-placements-design.md |
| +++ b/docs/superpowers/specs/2026-09-21-replica-placements-design.md |
| @@ -1,15 +1,17 @@ |
| # Replica placements: an account served locally in a second placement |
| -Design record, 2026-09-21. The platform half of amadan discussion |
| -carlos/platform #162, *A route can carry replica placements, served locally |
| -instead of hairpinned*, filed by titogo for its edge-checkout work (titogo |
| -`docs/superpowers/specs/2026-09-21-edge-checkout-design.md`, commit |
| -c6ed556e0). |
| - |
| -Reviewed once before writing (`FABLE-REVIEW-BRIEF-162-design.md` -> |
| -`FABLE-FINDINGS-162-design.md`, commit d6810d99); accepted findings are |
| -marked **(fable N)** where they changed the design, and the declined ones |
| -are listed at the end. |
| +Design record, 2026-09-21, revision 2. The platform half of amadan |
| +discussion carlos/platform #162, *A route can carry replica placements, |
| +served locally instead of hairpinned*, filed by titogo for its edge-checkout |
| +work (titogo `docs/superpowers/specs/2026-09-21-edge-checkout-design.md`, |
| +commit c6ed556e0). |
| + |
| +Two review rounds precede this revision, both committed beside it: the |
| +design sketch (`FABLE-REVIEW-BRIEF-162-design.md` -> |
| +`FABLE-FINDINGS-162-design.md`, 13 findings) and revision 1 of this spec |
| +(`FABLE-FINDINGS-162-spec.md`, 15 findings). Accepted findings are marked |
| +**(fable N)** and **(fable B-N)** respectively where they changed the |
| +design; declines are listed in §13. |
| ## 1. What is being asked for, and what it is on CARLOS |
| @@ -59,9 +61,9 @@ failing record rather than removing it) **(fable 4)**. |
| **`Replicas`** is the account's opt-in and *is* the residence gate: replica |
| placement is never inferred, never defaulted, and never derived from the |
| record's own `Region`. Values are validated against `ValidRegion`, refused |
| -when empty or equal to the record's own `Region`, deduplicated, sorted, and |
| -capped at `MaxReplicas = 4` — each entry is a decision about where order |
| -data lives until a drain confirms, not a cache hint. |
| +when empty, deduplicated, sorted, and capped at `MaxReplicas = 4` — each |
| +entry is a decision about where order data lives until a drain confirms, |
| +not a cache hint. |
| **`Home`** is written when the replica record is created and never |
| converged. Repointing a replica at a different home is delete-and-recreate, |
| @@ -72,24 +74,48 @@ because what it serves is a mirror *of* something. |
| - `Role` outside the vocabulary. |
| - `Role == "replica"` with `Kind != "instance"`: a replica is a process |
| holding a mirror, never a file tree and never a foreign upstream. |
| -- `Role == "replica"` with an empty `Home` or an empty `Region`. |
| +- `Role == "replica"` with an empty `Home` or an empty `Region` — a replica |
| + is always in a named placement, and `ValidRegion("")` is true |
| + (`provision.go:549-551`), so this rail is written explicitly. |
| - `Role == "replica"` with a non-empty `Replicas`: no replica of a replica. |
| -- A `Replicas` entry that is empty, not a placement code, or the record's |
| - own `Region`; more than `MaxReplicas` of them. |
| +- A `Replicas` entry that is empty, not a placement code, or literally |
| + equal to the record's own `Region`; more than `MaxReplicas` of them. |
| - A non-empty `Replicas` with `Steering != "latency"` — see §5, without |
| which the feature silently serves nothing **(fable 2)**. |
| +The "not the home's own placement" rail is **only partly enforceable |
| +here** **(fable B-5)**. The console normalises the deployment's home region |
| +to `""` before it reaches a record (`homeRegionIn`, |
| +`apiinstancemove.go:136-141`, used at `apiinstances.go:710`), so a home in |
| +the home region has `Region == ""` and a literal comparison never fires. |
| +`ValidateShape` refuses only the literal case it can see; the real rail |
| +lives at the door, which knows `s.cfg.HomeRegion` and compares through |
| +`homeRegionOut` (§6). |
| + |
| The replica's own record is an ordinary socket-backed instance record whose |
| `Region` is the replica's box, so the existing scope test mints it there |
| and nowhere else, with no change to `inScope` or `everywhereClass` |
| (`reconcile.go:319-336`, `:735-738`). |
| -**Channel.** A replica record carries no `Channel` of its own. `routeFor` |
| -reads the *home* record's channel when minting the replica's route, so a |
| -`carlos canary` or a channel PATCH on the home cannot leave the two |
| -instances on different channels with nothing to reconcile them |
| -**(fable 7)**. Version equality is a different question and is not |
| -promised: see §9. |
| +**Channel.** Home and replica must run one channel, and the record cannot |
| +express that by leaving the field empty: `Reconcile` protects any record |
| +with an empty `Channel` before scope is even asked (`reconcile.go:726-730`), |
| +so a channel-less replica record would never mint on any box |
| +**(fable B-2)**. So: |
| + |
| +- the create door writes the **home record's channel** onto the replica |
| + record, and refuses a caller-supplied channel on a `role=replica` create; |
| +- `Reconcile` keeps a second map of **every** decoded record (`all`, keyed |
| + by host, filled before the scope check — `recs` at `:654` is in-scope |
| + only and the home record is dropped at `:734-736`), and for a |
| + `role=replica` record hands `all[rec.Home]` to both `routeFor` and |
| + `convergeOwned`, so the row's channel follows the home's even after a |
| + `carlos canary` on the home; |
| +- if the home record is **absent, malformed or protected** this pass, the |
| + replica is protected too and nothing is minted or converged on a guess. |
| + A replica whose home has gone is a mirror of nothing. |
| + |
| +Version equality is a different question and is not promised: see §10. |
| ## 3. Routing: the replica serves the account's public host |
| @@ -99,7 +125,6 @@ makes it *serve the account* is one extra row: |
| route_aliases: alias = <the home record's host> |
| host = <the replica's own host> |
| - role = "replica" |
| on the replica's box only. |
| @@ -124,20 +149,35 @@ finds it `Adoptable` and restarts it under a name no process answers to |
| record-driven mint cannot produce "same socket, different host" |
| (`reconcile.go:348-372`). |
| -**Who writes the alias.** `route_aliases` is wholly owned by |
| +**The mark, and who writes the alias.** `route_aliases` is wholly owned by |
| `domains.ReconcileAliases`, which rebuilds the desired set from claims |
| every pass and deletes every alias it did not derive |
| -(`internal/domains/aliases.go:363-369`). So that pass learns to derive |
| -replica aliases too: for each replica record whose `Region` is this box, it |
| -adds `home.Host -> replica.Host` with `role = "replica"` to `desired`. One |
| -writer keeps one sweep, and a replica record that disappears takes its |
| -alias with it on the next pass with no new retirement code **(fable 1)**. |
| - |
| -**The registry mark.** `route_aliases` gains a `role TEXT NOT NULL DEFAULT |
| -''` column (additive, the table's established migration shape). It exists |
| -for one reader: the edges-file writer, which today derives every entry from |
| -`reg.List()` and hard-codes `Role: "home"` (`internal/host/edgesfile.go:219`, |
| -aliases at `:269`) and so has no way to say what this row is **(fable 3)**. |
| +(`aliases.go:363-369`). That pass **cannot read records**: it takes claims, |
| +`reg.List()` and `reg.ListAliases()` (`aliases.go:68-81`), it never lists |
| +`apps/`, and it cannot be taught to — `internal/provision` imports |
| +`internal/domains` (`reconcile.go:26`), so the reverse is a cycle |
| +(`aliases.go:29-33`). Revision 1's "derive it from the replica records" had |
| +no data path **(fable B-1)**. |
| + |
| +So the mark goes on the **route row**, which that pass already reads: |
| + |
| + routes: replica_of TEXT NOT NULL DEFAULT '' |
| + |
| +additive, in the established `ALTER TABLE routes` migration shape |
| +(`registry.go:443-452`), written by `routeFor` from `rec.Home` for a |
| +`role=replica` record and held steady by `convergeOwned`. Then: |
| + |
| +- `ReconcileAliases` derives `rt.ReplicaOf -> rt.Host` from `reg.List()` |
| + with no new input and no cycle, and adds it to `desired`, so the sweep |
| + keeps it; |
| +- when the replica record goes, the instance pass retires its route |
| + (`reconcile.go:1075-1092`) and the alias is swept on the next domains |
| + pass, with no new retirement code; |
| +- `edgesFilePass` already holds the routes (`edgesfile.go:174`) and marks |
| + an alias `role: "replica"` when `byHost[a.Host].ReplicaOf == a.Alias`, so |
| + `route_aliases` needs no new column, `UpsertAlias` needs no change |
| + (`registry.go:1841-1848`), and the alias half of the edge's fingerprint |
| + is untouched **(fable B-1, B-9)**. |
| ## 4. What the fleet is told |
| @@ -163,19 +203,33 @@ specifically what must not happen: the fold would then have two dial |
| targets and settle it by sort order (`homemap.go:566-584`), and a third box |
| would backhaul the account's traffic to the replica. |
| +**A replica alias is published regardless of `dns_status`** **(fable B-3)**. |
| +`edgesFilePass` withholds an ordinary alias until the domains sweep has |
| +stamped it `pointing` (`edgesfile.go:249-258`, stamped from a resolve |
| +against `PLATFORM_EDGE_IPS` in `domains/reconcile.go:465-472`, `:512-527`). |
| +That gate exists for tenant-chosen names the shared issuer would otherwise |
| +order for; a replica alias is a platform-provisioned route on the home box, |
| +derived from a row this box wrote, and its publication must not wait on a |
| +resolver — least of all when §5's whole point is that the name does not yet |
| +resolve to every edge. The writer therefore publishes replica aliases |
| +unconditionally and ordinary aliases exactly as before. |
| + |
| **Console side effect, stated rather than discovered:** `served_by` joins |
| every role (`internal/console/instancesplacement.go:68-70`), so the account |
| host will list the replica's box beside its home's. That is the answer Ask |
| 2 wanted — the home instance can see where its replicas are — and it is |
| correct; it is recorded here so nobody reads it as a leak **(fable 3)**. |
| +The console shows an alias as a plain extra hostname, so a replica alias |
| +will read as one; cosmetics are §12's business, not this landing's |
| +**(fable B-14)**. |
| ## 5. DNS: the placement is inert without it |
| -My first draft claimed buyers already land on the nearest edge and no |
| -steering change was needed. That is wrong for exactly the hosts this |
| -feature is for **(fable 2)**. A host gets a per-edge latency record only |
| -if its record carries `steering: "latency"` (`steering.go:157-162`), it |
| -lies under the apps domain (`:177-184`), and the deployment has armed |
| +Revision 1 of this design claimed buyers already land on the nearest edge |
| +and no steering change was needed. That is wrong for exactly the hosts this |
| +feature is for **(fable 2)**. A host gets a per-edge latency record only if |
| +its record carries `steering: "latency"` (`steering.go:157-162`), it lies |
| +under the apps domain (`:177-184`), and the deployment has armed |
| `CARLOS_STEERING_ZONE_ID` (`internal/host/steering_loop.go:219`). The |
| create door defaults `latency` only for `kind: static` in the platform |
| namespace (`apiinstances.go:827`), and titogo's instances are |
| @@ -184,21 +238,25 @@ namespace (`apiinstances.go:827`), and titogo's instances are |
| So: |
| - A home record with a non-empty `Replicas` MUST carry |
| - `steering: "latency"`. `ValidateShape` refuses the combination and the |
| - PATCH door refuses to set `replicas` on a record that is not steered, |
| - with a message that says to set steering first. Without this the replica |
| - box mints its alias, receives no buyer, and looks correct from every box. |
| + `steering: "latency"`. `ValidateShape` refuses the combination, the PATCH |
| + door refuses to set `replicas` on a record that is not steered, and it |
| + refuses to **turn steering off** while `Replicas` is non-empty — the same |
| + invariant from the other side, which revision 1 left open |
| + **(fable B-10)**. Without this the replica box mints its alias, receives |
| + no buyer, and looks correct from every box. |
| - For an account reached through a **tenant-owned** domain or a custom |
| domain, the tenant's DNS must CNAME to `edge.<apps-domain>`. The platform |
| - cannot do this and does not pretend to; the spec states it, and §11 keeps |
| - custom domains out of the first landing. |
| + cannot do this and does not pretend to; §12 keeps custom domains out of |
| + the first landing. |
| - **Deployment prerequisite:** `CARLOS_STEERING_ZONE_ID` is not armed by |
| any root module in `infra/` today (only `modules/region-host` declares |
| the variable, `variables.tf:153`). Until it is, replica placements are |
| - inert on the flagship. That is an infrastructure change and needs its own |
| + inert on the flagship. That is an infrastructure change needing its own |
| scoped approval (AGENTS.md, "Production infrastructure needs scoped |
| - approval"); it is filed as its own discussion and is not part of this |
| - branch. |
| + approval"); it is amadan discussion #169 and is not part of this branch. |
| + Which hosts on the flagship are steered today is a question for that |
| + thread, answerable with `carlos instances list` against the deployment; |
| + the expected answer is none. |
| Steering itself needs no code change: once the record is steered, every |
| armed edge already answers for it, and §3 decides what the nearest edge |
| @@ -211,20 +269,44 @@ the create and PATCH doors validate a record against itself and its |
| manifest today (`apiinstances.go:550-905`, `:1041`). Nothing in titogo can |
| enforce them and no reconcile pass will **(fable 4)**. |
| -**Create `role=replica`** requires: a `home` naming an existing record in |
| -the same account and app whose own `Role` is `""`; a `region` that is |
| -non-empty, differs from the home's, is listed in the home's `Replicas`, |
| -passes the manifest allow-list (`instanceRegionAllowed`, `:1573`) and the |
| -armed-and-hosting gate (`instancesplacement.go:173-202`); and a home |
| -already carrying `steering: "latency"`. The host is minted by the platform |
| -as a name of its own under the app's instance domain — not a secret, and |
| -the spec does not lean on unguessability for anything, since the name is |
| -published in the box's edges file by design **(fable 11)**. |
| +**Create `role=replica`** requires: |
| + |
| +- a `home` naming an existing record in the same account and app whose own |
| + `Role` is `""`; |
| +- a `region` that is non-empty, differs from the home's placement compared |
| + through `homeRegionOut` (so the home region's `""` is compared as its |
| + real code, **fable B-5**), is listed in the home's `Replicas`, passes the |
| + manifest allow-list (`instanceRegionAllowed`, `:1573`) and the |
| + armed-and-hosting gate (`instancesplacement.go:173-202`); |
| +- a home already carrying `steering: "latency"`; |
| +- an app that is **not fleet-placed**. The door overrides the caller's |
| + region for a placed app (`if placed != "" { region = placed }`, |
| + `apiinstances.go:711-712`), which would silently move the replica to the |
| + fleet's placement; a replica of a placed app is refused until someone |
| + decides what it means **(fable B-5)**. |
| + |
| +The channel is taken from the home record (§2) and a caller-supplied |
| +channel is refused. |
| + |
| +**The host is minted by the platform** and a caller-supplied `host` is |
| +refused on a `role=replica` create **(fable B-8)**. The name is one label |
| +under the app's instance domain, `r` followed by 10 characters of |
| +`crypto/rand` base32 (Crockford, lower case), which keeps the existing |
| +one-label rail (`apiinstances.go:781-786`) and the slug derivation |
| +(`:777`). `CreateRecord` is create-only and 409s on a race |
| +(`provision.go:31-33`); the door retries a fresh name twice, then fails. |
| +The minted host is returned in the create response and printed by the CLI, |
| +because the caller cannot otherwise learn it. It is not a secret and |
| +nothing depends on it being unguessable — it is published in the box's |
| +edges file by design **(fable 11)**. |
| **PATCH `replicas`** on a home record: validated as in §2, and a placement |
| may not be removed while a replica record still exists there. The opt-in |
| and the data must not disagree — that is the residence failure the field |
| -exists to prevent. |
| +exists to prevent. Both this refusal and the delete refusal below read the |
| +app's records with `ListRecords`, which is capped: a truncated or failed |
| +list **refuses the mutation** rather than proceeding on a partial view |
| +**(fable B-13)**. |
| **Delete** a home whose `Replicas` is non-empty, or for which any replica |
| record exists, is refused with a 409 naming them **(fable 6)**. Deleting |
| @@ -240,11 +322,26 @@ failed-wake cleanup, `backend_exec.go:315-327`). Stated, not fixed, here. |
| into one of its own replica placements is refused too, or the home's route |
| and the replica's alias collide on one box **(fable 4)**. |
| -**Tokens.** `handleAPIInstanceToken` mints an app-wide token |
| -(`apiinstances.go:1392`, amadan #155), so a replica holding one could |
| -delete its own home. A replica needs no console API — §7 hands it |
| -everything it needs as data — so the door refuses to mint a token for a |
| -replica record, and only the home lists its replicas **(fable 8)**. |
| +**Tokens.** There is nothing to refuse at the token door: it is |
| +`POST /api/cli/apps/{account}/{app}/instance-token` (`authcli.go:636`), |
| +takes no host, reads no record, and mints an app-wide token |
| +(`NewAppToken`, `apiinstances.go:1392`) **(fable B-4)**. So the honest |
| +statement is: the platform mints app-wide tokens only, a token handed to a |
| +replica can act on its home, amadan #155 is the fix and is not this branch, |
| +and a replica should be configured with no token at all — §7 hands it |
| +everything it needs as data. The platform-side guard that does hold is the |
| +home-delete 409 above. |
| + |
| +**An old console strips the new fields.** `UpdateRecord` marshals the |
| +struct it was handed (`provision.go:928-933`) and `DecodeRecord` polices |
| +only `maintenance` (`maintenance.go:38-78`), so a console build that |
| +predates these fields reads a record without them and writes it back |
| +without them — `replicas` vanishes from a home (bypassing the refusal |
| +above), `role`/`home` vanish from a replica, which clears `replica_of` and |
| +sweeps the alias on the next pass, silently. Tito's console is pinned on an |
| +older build today. So `Role`, `Home` and `Replicas` join |
| +`protectedFieldsEqual`: an old writer's PATCH is refused with the |
| +maintenance-projection error rather than accepted **(fable B-7)**. |
| ## 7. Sibling addressing (Ask 3): no new channel |
| @@ -259,45 +356,83 @@ change. Refusing to carry an app-defined call over titogo's |
| one shape and fails closed, and widening it would make a narrow, verified |
| door general. |
| -The platform supplies only what an instance cannot know. A **siblings |
| -file**, written by the agent beside the instance's own data and refreshed |
| -every reconcile pass, with its path handed to the child as |
| -`CARLOS_SIBLINGS_FILE` in `ExecBackend.agentOwnedVars` |
| -(`backend_exec.go:977-994`) — the seam that already injects `CARLOS_REGION` |
| -**(fable 5)**. A file rather than more environment variables because the |
| -child's environment is composed at spawn (`instanceEnv`, `:1135`): adding a |
| -replica would otherwise require bouncing a running home instance to tell it. |
| +The platform supplies only what an instance cannot know: a **siblings |
| +file**, with its path handed to the child as `CARLOS_SIBLINGS_FILE` in |
| +`ExecBackend.agentOwnedVars` (`backend_exec.go:977-994`) — the seam that |
| +already injects `CARLOS_REGION`, overrides file env (`withVar`, `:990`), |
| +survives the `CARLOS_*` strip (`childEnv`, `:1177-1180`) and reaches both |
| +spawn paths (`startChild` `:895`, `startUnit` `:1055`). The variable is |
| +stripped with `withoutVars` when no file was written, exactly as |
| +`CARLOS_REGION` is **(fable 5, fable B-6)**. |
| - {"role":"home","host":"acme.apps.example", |
| - "replicas":[{"host":"r7x2.apps.example","placement":"uk", |
| - "addr":"203.0.113.9:443","key":"<base64>"}]} |
| + {"role":"home","host":"acme.apps.example","placement":"ie", |
| + "replicas":[{"host":"r7x2k9m4qa.apps.example","placement":"uk", |
| + "addr":"203.0.113.9:443","keys":["<b64>","<b64 prev>"]}]} |
| - {"role":"replica","host":"r7x2.apps.example","placement":"uk", |
| + {"role":"replica","host":"r7x2k9m4qa.apps.example","placement":"uk", |
| "home":{"host":"acme.apps.example","addr":"198.51.100.4:443", |
| - "key":"<base64>"}} |
| - |
| -Written 0600, owned by the instance user, atomically replaced. |
| - |
| -**The key** is per *pair*, not per account: `HMAC(meshKey, |
| -"replica/" + account + "/" + homeHost + "/" + replicaHost)` **(fable 5, |
| -adapted)**. Fable proposed per-instance keys the home derives itself; the |
| -home cannot, because it must never hold the mesh key — so the platform |
| -derives each pair key and writes it into both sides' files. A compromised |
| -replica can then speak only as itself, never as a sibling. The mesh key |
| -never leaves the agent (`internal/edge/mesh.go:92-95`), HMAC is a PRF so |
| -the derivative does not expose it, and the agent already holds the pair in |
| -memory (`homemap.go:675-683`). |
| - |
| -**Rotation.** A mesh rotation changes every derived key. The platform |
| -rewrites both siblings files on the next pass after `.next` is promoted, |
| -and because the file is re-readable the app can pick the new key up without |
| -a restart — provided titogo re-reads it on a verification failure. The spec |
| -requires that of titogo and says plainly that a rotation is otherwise a |
| -restart-everything event. No `_NEXT` companion in the first landing |
| -**(fable 13)**. |
| + "keys":["<b64>","<b64 prev>"]}} |
| + |
| +**Where it is written.** `pathsFor(rt).dataDir` (`activator/paths.go:38-42`), |
| +which is `/data/instances/<id>` for a nested row and `/data` for a flat one |
| +(`provision.InstanceDir`, `provision.go:1084`; |
| +`internal/host/schedule_wiring.go:193-203` computes both already and is the |
| +precedent to follow). For an isolated instance that is the only writable |
| +`/data` path inside its mount namespace (`carlos-instance@.service`: |
| +`BindPaths=/data/instances/%I`, `TemporaryFileSystem=/data:ro`), so a file |
| +written anywhere else in `/data` is invisible to the process that must read |
| +it. The directory is tenant-writable ground, so the write goes through |
| +`instancefs.WriteAtomic` (`prepare.go:154` is the precedent), never a plain |
| +open. Mode 0600; both the agent (`carlos-edge.service`, `User=carlos`) and |
| +the instance (`carlos-instance@.service`, `User=carlos`) run as `carlos` — |
| +isolation here is the mount namespace, not a per-instance uid |
| +**(fable B-6)**. |
| + |
| +**What the writer needs, and where it lives.** The file needs three inputs |
| +the reconcile pass does not hold today, so the writer is a small pass of |
| +its own in `internal/host` beside the other agent loops, fed by: |
| + |
| +- **records on both sides** — the home's file lists replica records homed |
| + on other boxes and the replica's file needs the home record, both out of |
| + scope on the box that writes the file. §2's `all` map is the same need |
| + and is the single source for both; |
| +- **the sibling's dial address** — the sibling box's EIP, which the edge's |
| + home map holds (`homeMap.Home`, `homemap.go:611-625`, returns `eip, box`; |
| + it refuses this box's own, which is fine because §6 refuses |
| + co-location). Published to the loop through the edge-ready callback, the |
| + `upstreamSource` pattern at `agent.go:2490-2510`; |
| +- **the mesh key pair** — `homeMap.meshKeys()` is unexported and |
| + edge-owned (`homemap.go:693-696`); the same callback publishes an |
| + accessor. |
| + |
| +A sibling whose box has published no edges file yet has no EIP: its entry |
| +is **omitted** rather than written with an empty `addr`, and the file is |
| +written for a home even when its `replicas` list is empty, so the app can |
| +tell "no replicas" from "no file" **(fable B-6)**. |
| + |
| +**The keys** are per *pair*: `HMAC(meshKey, "replica/" + account + "/" + |
| +homeHost + "/" + replicaHost)` **(fable 5, adapted)**. Round 1 proposed |
| +per-instance keys the home derives itself; the home cannot, because it must |
| +never hold the mesh key — so the platform derives each pair key and writes |
| +it into both sides' files. A compromised replica can then speak only as |
| +itself, never as a sibling. The mesh key never leaves the agent |
| +(`mesh.go:92-95`), HMAC is a PRF so the derivative does not expose it, and |
| +the agent already holds the pair in memory (`homemap.go:675-683`). |
| + |
| +**Rotation.** The two files sit on two boxes and are rewritten by two |
| +passes, so a mesh rotation moves them at different moments; a key that is |
| +current on one side and not yet on the other would fail every call in |
| +between **(fable B-11)**. So `keys` is a list, current first, previous |
| +second: the caller signs with the first and the receiver accepts any entry. |
| +The previous key is dropped on the pass after the rotation is complete. No |
| +`_NEXT` variable and no restart **(fable 13)**. |
| titogo signs and shapes the lease and drain calls itself over that address |
| -and that key. The protocol stays entirely theirs. |
| +and those keys. The protocol stays entirely theirs. Two things the platform |
| +asks of it in return: **re-read the file** rather than caching the key at |
| +boot (that is what makes adding a replica and rotating a key restart-free, |
| +**fable B-12**), and treat a signature that verifies under the second key |
| +as valid. |
| ## 8. Certificates: nothing to build, worth writing down |
| @@ -314,7 +449,26 @@ once the alias exists (`edge.go:968`), so that box may order for the public |
| host if it is ever the housekeeper — same issuer, same account, same rate |
| bucket, harmless. |
| -## 9. What the platform does not promise |
| +## 9. Order of operations |
| + |
| +A replica is inert until every step is in place, and each step is safe on |
| +its own: |
| + |
| +1. The home record is latency-steered (§5) and its `Replicas` names the |
| + placement. |
| +2. The replica record is created; the instance pass on that box mints its |
| + route, `replica_of` and all. |
| +3. The domains pass derives the alias; the edges file publishes the two |
| + entries; the siblings files appear on both boxes. |
| +4. The steering pass gives the account host a latency record per armed edge |
| + — the deployment prerequisite in §5, without which buyers still reach |
| + one edge and the replica serves nothing. |
| + |
| +Removing a replica runs backwards: delete the replica record, the route is |
| +retired, the alias is swept, the aliases and siblings files follow, and |
| +only then may the placement leave `Replicas`. |
| + |
| +## 10. What the platform does not promise |
| - **Version equality.** Adoption runs per box on its own cadence |
| (`adopt.go:303-360`), so a home and its replica will differ by one |
| @@ -330,7 +484,7 @@ bucket, harmless. |
| - **Drain safety.** The platform cannot see whether a replica holds |
| undrained orders; §6's refusals protect the home, not the replica. |
| -## 10. Tests |
| +## 11. Tests |
| Each of these fails before its change and passes after: |
| @@ -338,44 +492,54 @@ Each of these fails before its change and passes after: |
| record unchanged. |
| 2. The console door refuses: a replica of a replica, a placement not in the |
| home's `Replicas`, an unarmed placement, a home that is not steered, |
| + turning steering off under a live `Replicas`, a replica of a |
| + fleet-placed app, a caller-supplied host or channel on a replica create, |
| removing a placement under a live replica, deleting a home with |
| - replicas, moving a replica, and minting a token for one. |
| -3. `ReconcileAliases` derives the replica alias on the replica's box, does |
| - not derive it anywhere else, and — the regression this design most |
| - needs — **a replica alias survives a full pass** rather than being swept |
| - as underived **(fable 13)**. |
| -4. The edges-file writer publishes the public host as `role: "replica"` and |
| - the replica's own host as `role: "home"`. |
| -5. `foldHomes` on a third box has exactly **one** dial target for the |
| + replicas, moving a replica, and a PATCH from a writer that drops the new |
| + fields. |
| +3. `routeFor` writes `replica_of`; `convergeOwned` holds it steady; a |
| + replica record whose home record is missing this pass is protected |
| + rather than minted. |
| +4. The replica's route carries the **home's** channel, including after the |
| + home's channel changes. |
| +5. `ReconcileAliases` derives the replica alias on the replica's box, does |
| + not derive it anywhere else, and — the regression this design most needs |
| + — **a replica alias survives a full pass** rather than being swept as |
| + underived **(fable 13)**. |
| +6. The edges-file writer publishes the public host as `role: "replica"` |
| + with no `dns_status` stamp present, and the replica's own host as |
| + `role: "home"`. |
| +7. `foldHomes` on a third box has exactly **one** dial target for the |
| account host, with the replica box publishing its entry. |
| -6. The edge resolves the account host on the replica box to the replica's |
| - own upstream, with no backhaul attempt (the existing backhaul test's |
| - inverse). |
| -7. The siblings file is written for both sides with the same pair key, 0600, |
| - and refreshed when the replica set changes. |
| -8. An account with no replicas produces byte-identical edges-file, |
| - registry-row and route-table output to before the change. |
| - |
| -## 11. Scope of the first landing |
| - |
| -**In:** the record fields and their rails; the console doors; the replica |
| -alias and its derivation; the `route_aliases.role` column; the edges-file |
| -mark and the explicit `foldHomes` case; the siblings file and its keys; the |
| -list/get/CLI surface (`role`, `home`, `replicas`, a role column, and |
| -`docs/site/cli.md` for any new verb, per AGENTS.md). |
| +8. The edge resolves the account host on the replica box to the replica's |
| + own upstream, with no backhaul attempt. |
| +9. The siblings file is written for both sides with the same pair key, 0600, |
| + in the right directory for a nested and a flat row, refreshed when the |
| + replica set changes, with an EIP-less sibling omitted and a home with no |
| + replicas still getting a file. |
| +10. An account with no replicas produces byte-identical edges-file, |
| + registry-row and route-table output to before the change. |
| + |
| +## 12. Scope of the first landing |
| + |
| +**In:** the record fields and their rails; the console doors; the |
| +`routes.replica_of` column, its derivation in `ReconcileAliases` and the |
| +edges-file mark; the explicit `foldHomes` case; the siblings-file pass and |
| +its keys; the list/get/CLI surface (`role`, `home`, `replicas`, a role |
| +column, and `docs/site/cli.md` for any new verb, per AGENTS.md). |
| **Out, deliberately:** custom domains on the replica box — an alias for a |
| custom domain is only derived when its primary is a local route |
| (`aliases.go:95-100`), so it needs `ReconcileAliases` work *and* a tenant |
| DNS change, and the account host is enough to prove the feature |
| -**(fable 13)**. Also out: `_NEXT` keys, a platform-side drain or mirror |
| -(titogo reuses its sandbox sync), an AU box, and any change to the lease |
| -protocol. |
| +**(fable 13)**. Also out: a platform-side drain or mirror (titogo reuses |
| +its sandbox sync), an AU box, console cosmetics for replica aliases, and |
| +any change to the lease protocol. |
| **Prerequisite, not in this branch:** arming `CARLOS_STEERING_ZONE_ID` in |
| -the deployment (§5). |
| +the deployment (§5, amadan #169). |
| -## 12. Declined findings |
| +## 13. Declined findings |
| - **fable 5's per-instance key derived by home.** Declined as written: it |
| requires the home instance to hold the mesh key. Replaced by per-pair |
| @@ -385,3 +549,11 @@ the deployment (§5). |
| Declined: the check is three lines and the failure it prevents (an |
| opt-in naming a placement nothing can serve) is invisible until a buyer |
| hits it. |
| +- **fable B-1's alternative** (re-list the bucket in `agent.go` and thread |
| + a `[]ReplicaPair` argument into `ReconcileAliases`). Declined in favour |
| + of the `replica_of` column it recommends: a second full `apps/` listing |
| + on the seconds-cadence leg is a real cost for a fact the route row can |
| + carry. |
| +- **fable B-15** (titogo's design record names no platform contract). |
| + Answered on amadan #162 rather than in this spec: the contract is the |
| + siblings file, the dial shape, and the three non-promises in §10. |