amadan
spec: relay revision 4, codex round 3; roadmap passages reconciled
Eight findings taken: the reconcile converges config and aggregate from every record every five minutes under a per-app lock, so stale writes and interrupted deletions are repaired; binding cleanup belongs to the wrapped socket since pion's deletion event carries no relay address; the OOM ordering claim is narrowed and PSI-driven shedding added; rolling budgets rebuild from both hourly objects; config acks are per route and name the rendered kids; a staged master issues nothing until control/relay-issuer.json is moved by rotate. The roadmap's custody, placement, peer policy, authority lifecycle and accounting passages now match the spec.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
3 files changed,
+330
−101
docs/superpowers/specs/2026-09-07-media-platform-roadmap-design.md+32 −20docs/superpowers/specs/2026-09-07-relay-stun-turn-design.md+142 −81docs/superpowers/specs/CODEX-FINDINGS-RELAY-R3.md+156 −0
CI passed — run details
diff --git a/docs/superpowers/specs/2026-09-07-media-platform-roadmap-design.md b/docs/superpowers/specs/2026-09-07-media-platform-roadmap-design.md| index 512dca9bd..d8b39f56d 100644 |
| --- a/docs/superpowers/specs/2026-09-07-media-platform-roadmap-design.md |
| +++ b/docs/superpowers/specs/2026-09-07-media-platform-roadmap-design.md |
| @@ -629,15 +629,17 @@ box, built on `pion/turn`, pure Go, in the same binary as the edge, a |
| would end any longer call). The exact wire shape, a base64-wrapped |
| JSON `CARLOS_RELAY` with a platform-supplied username suffix and a |
| key id, is the stage 4a spec's §2. The relay holds a **relay |
| - authority record** it syncs from the route table the way the edge |
| - syncs routes: for each enabled app, its allowed regions and current |
| - incarnation. On a request it parses the username, refuses it unless |
| - the app is enabled and the incarnation matches the record, re-derives |
| - the key from its own region master and the username's fields, and |
| - verifies. Verification is local and per-app metering is real. Disable |
| - bumps the incarnation in the record, which ends existing allocations |
| - and makes every outstanding credential fail, and the incarnation is |
| - never reused across deletion or name reuse. A relay whose authority |
| + authority record**, `control/relay.json`, rebuilt by the console from |
| + per-app relay records and read by every relay every minute: for each |
| + enabled app, its regions and current incarnation. On a request it |
| + parses the username, refuses it unless the app is enabled and the |
| + incarnation matches the record, re-derives the key from its own |
| + region master and the username's fields, and verifies. Verification |
| + is local and per-app metering is real. Disable removes the app from |
| + the record, which ends existing allocations and makes every |
| + outstanding credential fail; re-enable draws a new random |
| + incarnation, so nothing minted before a disable ever works again and |
| + nothing is reused across deletion or name reuse. A relay whose authority |
| record is older than fifteen minutes refuses new allocations **and |
| terminates the ones it is carrying**, the same lease rule the edge |
| applies to media: a disable the relay never heard about must not |
| @@ -647,20 +649,29 @@ box, built on `pion/turn`, pure Go, in the same binary as the edge, a |
| - **What that does and does not protect.** Per-region masters mean a |
| relay box holds no other region's master. They do not mean a |
| compromised host learns nothing about other regions: sealed config is |
| - decryptable on every host (§1), so a compromised host can read every |
| - enabled app's derived keys for every region, exactly as it can read |
| - every app's store keys today. Relay keys are no worse than the secrets |
| - the platform already delivers this way, and no better. A short-lived |
| - issuance API would narrow that and is the alternative if it is ever |
| - wanted. **(codex R2-2)** |
| + decryptable on every host (§1), and the console must hold every |
| + region's master as sealed config to derive app keys, so **a |
| + compromised host can read every region's master** and mint for any |
| + app in any region, present and future, until the operator rotates |
| + (Paul, 2026-09-07, stage 4a spec ruling 3). That is wider than the |
| + derived-key exposure alone, and it is stated in those words in the |
| + operator docs. Rotation of one region recovers that region. A design |
| + where only a region's boxes ever hold its master, the relay deriving |
| + and writing app keys itself, is filed as the tightening. **(codex |
| + R2-2; amended by the stage 4a spec)** |
| - **Policy and limits.** Peer permissions refuse loopback, link-local, |
| - RFC 1918, the VPC's ranges and the relay's own addresses. Allocations, |
| + RFC 1918, the VPC's ranges and the box's private addresses; the box's |
| + own Elastic IP is reachable, but only on ports holding a live relay |
| + allocation, so two clients on one relay can reach each other and |
| + nothing else on the box is exposed. Allocations, |
| allocation rate, bandwidth and lifetimes are bounded per app, per |
| account and per box. Established allocations end when their |
| credential expires or the app is disabled. These are operational |
| limits in 4a; commercial quotas wait for measurement. **(codex 14)** |
| -- **Placement.** `CARLOS_RELAY_URLS` lists the allowed regions with the |
| - instance's own first. The browser gathers candidates from all of them |
| +- **Placement.** The app's `CARLOS_RELAY` lists its regions in the |
| + console's order, one list for the whole app, since placement names |
| + are not regions and an app may have several placements (stage 4a |
| + spec). The browser gathers candidates from all of them |
| and ICE nominates a working pair by its own procedures; that is not a |
| lowest-latency guarantee and the roadmap does not claim one. |
| Client-aware relay selection is a separate later decision. |
| @@ -977,8 +988,9 @@ Every finding from revisions 2 to 5 was accepted. |
| for apps; the platform's own relay and SFU units are the exception and |
| are specified in stage 4. |
| - `2026-08-11-unit-economics-design.md`: a media section, at stage 1 |
| - for served bytes, at 4a for the three relay meters, at 4b for SFU |
| - egress. |
| + for served bytes, at 4a for the two relay meters (bytes forwarded in |
| + each direction), at stage 5 for the inter-region meter and the usage |
| + line, at 4b for SFU egress. |
| - `infra/modules/region-host/main.tf:6` header: residency by |
| construction now has a named exception, the per-region media bucket |
| policy, and the exception is itself region-scoped. |
diff --git a/docs/superpowers/specs/2026-09-07-relay-stun-turn-design.md b/docs/superpowers/specs/2026-09-07-relay-stun-turn-design.md| index a2fd9983f..c0fe3f470 100644 |
| --- a/docs/superpowers/specs/2026-09-07-relay-stun-turn-design.md |
| +++ b/docs/superpowers/specs/2026-09-07-relay-stun-turn-design.md |
| @@ -1,13 +1,14 @@ |
| # Stage 4a: a platform STUN and TURN relay |
| -**Date:** 2026-09-07, revision 3 |
| +**Date:** 2026-09-07, revision 4 |
| **Status:** rulings taken. Paul ruled on the six decisions in §9 on |
| 2026-09-07, each marked **(ruling)**; the roadmap's stage 4a text and |
| -gate are amended in the same commit so the two documents agree. |
| -Reviewed twice by GPT-6 via Codex: revision 1's 20 findings are in |
| +gate are amended so the two documents agree. Reviewed three times by |
| +GPT-6 via Codex: revision 1's 20 findings are in |
| `CODEX-FINDINGS-RELAY.md`, revision 2's 12 in |
| -`CODEX-FINDINGS-RELAY-R2.md`, unedited; every one is taken, marked |
| -**(codex R1-N)** or **(codex R2-N)**. Stage 4a of the media roadmap |
| +`CODEX-FINDINGS-RELAY-R2.md`, revision 3's 8 in |
| +`CODEX-FINDINGS-RELAY-R3.md`, unedited; every one is taken, marked |
| +**(codex R1-N)**, **(codex R2-N)** or **(codex R3-N)**. Stage 4a of the media roadmap |
| (`2026-09-07-media-platform-roadmap-design.md`), built under rulings 8, |
| 10 and 11 there. |
| @@ -107,12 +108,15 @@ restart and a master rotation all keep it. **(codex R1-13)** |
| **What changes the key.** A master rotation in a region changes that |
| region's key and `kid`; a new incarnation changes every region's key. |
| -Either re-renders the app's config, which for unit-backed routes, and |
| -exec-backed routes whose bundle opts into file delivery, restarts the |
| -instance within about a minute, as any config change does today; a |
| -TCP-address route without file delivery receives nothing and the app |
| -must fetch its config another way (`internal/appconfig/syncexec.go:11`). |
| -**(codex R1-10, R1-14, R2-6)** |
| +Either re-renders the app's config within about a minute, as any config |
| +change does today, and what happens next depends on the route's shape |
| +(`internal/appconfig/sync.go:30`, `syncexec.go:1`): a unit-backed route |
| +and a socket-backed exec route are restarted onto the new file; a |
| +TCP-address exec route whose bundle opts into file delivery receives |
| +the file and is **not** restarted, so the app must re-read it; a |
| +TCP-address route without file delivery receives nothing. `carlos relay |
| +status` names which of these each route is. **(codex R1-10, R1-14, |
| +R2-6, R3-4)** |
| **A credential keeps working for its whole life** provided the master |
| that signed its key is not force-retired before its expiry. The |
| @@ -159,16 +163,21 @@ network-facing platform unit like the edge, not a codec helper. |
| `User=carlos`, `ProtectSystem=strict`, `ReadWritePaths=/data/relay |
| /run/carlos`, `RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX`. |
| Memory: `MemoryMax=256M`, `MemoryHigh=160M`, `OOMScoreAdjust=1000`. |
| - The box's ceilings oversubscribe it (edge 80% of RAM, tenants' tier |
| - 1 GiB, relay 256 MiB on a 2 GiB `t4g.small`), and no setting makes |
| - the kernel's choice strict, so the relay is made the cheapest victim |
| - by arithmetic and sheds itself first: the kernel's score is RSS as a |
| - fraction of RAM in thousandths plus the adjustment, so the relay |
| - scores at least 1000 and at most 1125, a tenant at most 1000 (1 GiB |
| - cap, adjustment 500), the edge at most 800; and above `MemoryHigh` |
| - the relay refuses new allocations and ends its oldest until it is |
| - under, before the kernel is involved. `Nice=5` and `CPUWeight=50` for |
| - CPU. **(codex R1-19, R2-8)** |
| + What that guarantees, stated exactly: the relay can never take more |
| + than 256 MiB of the box; above 160 MiB it refuses new allocations and |
| + ends its oldest until it is under; and it is the kernel's preferred |
| + victim among processes of similar footprint, since 1000 is the |
| + largest adjustment there is. What it does not guarantee: that the |
| + relay dies before a tenant under every host-wide pressure. The |
| + kernel's score counts swap and page tables as well as RSS, tenants |
| + may swap up to their memory cap (`internal/cgroup/cgroup.go:313`), |
| + and a kill inside the edge's own 80% cgroup considers only the edge's |
| + tree. So the relay does not wait for the kernel: it watches the |
| + box's memory pressure (`/proc/pressure/memory`, the `some` ten-second |
| + average) and sheds allocations, oldest first, whenever it exceeds |
| + 20%, whatever caused the pressure, resuming admission when it falls |
| + under 10%. The gate in §7 tests that shedding, not a kill order. |
| + `Nice=5` and `CPUWeight=50` for CPU. **(codex R1-19, R2-8, R3-7)** |
| - **Listeners.** UDP 3478 and TCP 3478 on the box's primary address. |
| No TLS listener **(ruling 4)**. The relay allocation range is UDP |
| 49152 to 65535, bound through the relay's own address generator |
| @@ -208,8 +217,15 @@ network-facing platform unit like the edge, not a codec helper. |
| `OnAllocationCreated` it records that key to the identity fields |
| (account, app, incarnation); its `AuthHandler` refuses a username |
| whose identity differs from the one bound to a live allocation from |
| - that key; `OnAllocationDeleted` removes only the binding whose relay |
| - address matches the event's. **(codex R1-3, R2-3)** |
| + that key. Cleanup belongs to the wrapped relay socket, not to |
| + `OnAllocationDeleted`, which carries no relay address |
| + (`internal/allocation/event_handler.go:20`): each wrapper knows the |
| + binding it belongs to, and on `Close`, whether pion closed it or the |
| + relay did, it removes that binding only if the map still points at |
| + itself, so a replacement allocation bound while the old one was |
| + closing (pion removes from its manager before closing, |
| + `allocation_manager.go:150`) is never erased. **(codex R1-3, R2-3, |
| + R3-3)** |
| - **Termination.** An allocation is ended, by closing its relay socket, |
| at the earliest of: its credential's expiry (a timer set at creation |
| from the username's expiry field); the app being disabled, its |
| @@ -241,8 +257,9 @@ network-facing platform unit like the edge, not a codec helper. |
| Refusals and drops are counted per app. **(codex R1-8, R2-9)** |
| - **Limits (ruling 6)**, all admitted under one mutex across every |
| listener so counts cannot race, with counters reset on process start |
| - and the rolling-hour byte windows rebuilt from the relay's own |
| - metrics object on start: |
| + and the rolling-hour byte windows rebuilt on start from every hourly |
| + metrics object that intersects the window, normally the previous |
| + hour's and the current one **(codex R3-6)**: |
| - per app per box: 64 live allocations, 30 new a minute, 20 GB a |
| rolling hour; |
| - per account per box: 256 live, 60 new a minute, 60 GB a rolling |
| @@ -313,8 +330,10 @@ ignored for new requests and its live allocations are ended (§3). |
| The console must hold every region's masters to derive app keys at |
| enable. It holds them as console secrets, `CARLOS_RELAY_MASTERS_<REGION>` |
| (region name uppercased with hyphens as underscores, since variable |
| -names refuse hyphens, `internal/appconfig/appconfig.go:103`), and |
| -console secrets are sealed config like any app's. Sealed config is |
| +names refuse hyphens, `internal/appconfig/appconfig.go:103`), one or |
| +more `kid:key` pairs per region, of which the one named by the issuer |
| +pointer `control/relay-issuer.json` is the one keys are derived from |
| +(§5, rotation); and console secrets are sealed config like any app's. Sealed config is |
| decryptable on every host (roadmap §1). So the honest statement is: **a |
| compromised host can read every region's master**, not only the derived |
| keys the roadmap already conceded, and with a master it can mint |
| @@ -386,21 +405,29 @@ ours. **(codex R1-20)** |
| so relays notice within seconds; the bump is a latency hint, not the |
| lease. |
| -**Every operation is a walk the reconcile can finish.** An operation |
| -writes its intent to `relay.json` first (`op` bumped, `phase` set), |
| -then does the config write, then the aggregate publish, advancing |
| -`phase` after each step with a CAS that carries the `op`, so a writer |
| -whose `op` has been superseded finds its CAS refused and stops. Before |
| -the config write it re-reads the record and refuses if the `op` is not |
| -its own. The **reconcile** lists every record every five minutes and |
| -at console start, and for any record not at `phase: done` performs the |
| -remaining steps of that record's operation idempotently (the config |
| -value is a pure function of the record and the masters, so re-applying |
| -it is harmless; the aggregate publish is always a rebuild). So an |
| -interrupted enable delivers its config within five minutes, an |
| -interrupted disable removes it within five minutes, and a repeated |
| -`enable` while a previous one is unfinished resumes it rather than |
| -skipping the config write. **(codex R2-2)** |
| +**Every operation is a walk the reconcile can finish, and the |
| +reconcile converges everything.** An operation writes its intent to |
| +`relay.json` first (`op` bumped, `phase` set), then does the config |
| +write, then the aggregate publish, advancing `phase` after each step |
| +with a CAS that carries the `op`. Two fences against a superseded |
| +writer: first, the console serialises every relay operation for one |
| +app, handler or reconcile, behind a per-app lock in its process, so |
| +two operations on one app never interleave their steps in the common |
| +case; second, because the config write is a separate object the lock |
| +cannot cover across processes, the **reconcile** does not only finish |
| +unfinished records, it **converges both derived artefacts from every |
| +record, every five minutes and at console start**: for each enabled |
| +record it derives the expected `CARLOS_RELAY` value from the record and |
| +the active masters and re-applies it if the config bundle holds |
| +anything else; for each disabled record it removes the entry if |
| +present; and it always rebuilds and publishes the aggregate whether or |
| +not any record is unfinished. So a stale config write that lands after |
| +a disable is undone within five minutes, an interrupted enable |
| +delivers its config within five minutes, an interrupted disable |
| +removes it within five minutes, a repeated `enable` while a previous |
| +one is unfinished resumes it rather than skipping the config write, and |
| +a deletion that removed the record but died before publishing is |
| +repaired by the next unconditional rebuild. **(codex R2-2, R3-1, R3-2)** |
| **When a disable is complete.** The app is out of every relay's |
| authority when the aggregate has been published without it **and** |
| @@ -452,27 +479,38 @@ ruling 2): |
| "config generation <g> written; delivery is reported by `carlos |
| relay status`", never "delivered". |
| -**Config adoption, observed.** The host's config sync, after it renders |
| -an app's env file and restarts the route (or writes the file for an |
| -exec route that opted into file delivery), writes an additive |
| -acknowledgement `control/config-ack/<box>/<acct>/<app>.json` carrying |
| -the generation string it rendered, the time, and whether a restart |
| -happened. Nothing reads it today but `carlos relay status`; it is the |
| -first per-instance statement of "this config is what the process runs" |
| +**Config adoption, observed per route.** The host's config sync, after |
| +it renders a route's env file and does whatever that route's shape |
| +calls for, writes an additive acknowledgement |
| +`control/config-ack/<box>/<route host>.json`: the route's shape (unit, |
| +socket exec, TCP exec with file delivery), the time, whether a restart |
| +happened and succeeded, and **the relay key identities it rendered**: |
| +the `kid` per region parsed out of the `CARLOS_RELAY` value it just |
| +wrote, or none. The ack is keyed on the route, not the app, because an |
| +app can have several routes on one box and one can adopt while another |
| +does not; and it names the kids rather than a generation string, |
| +because the app-wide generation object is a change notification that |
| +can coalesce or fail after the manifest write |
| +(`internal/appconfig/generation.go:28`) and is not an identity of what |
| +was rendered. Nothing reads the ack today but `carlos relay status`; |
| +it is the first per-route statement of "this is what the process has" |
| the platform has, and the plan adds it to `internal/appconfig/sync.go` |
| -as its own small step. **(codex R1-14, R2-6)** |
| +as its own small step. **(codex R1-14, R2-6, R3-4)** |
| **`carlos relay status`** reports, separately: desired state from |
| -`relay.json` with its `op` and `phase`; the config generation written; |
| -delivery per box, "adopted at <time>" when a config-ack for that |
| -generation exists, "written, awaiting delivery" when it does not, |
| -"written, not restarted by the platform" for an exec route with file |
| -delivery, and "not delivered: route has no config delivery" for a |
| -TCP-address route without it; disable progress as above; and per |
| -region, from `control/relay-ack/<box>.json`, the heartbeat age, whether |
| -the record it holds is current, whether its lease is live, which |
| -listeners are up, and which `kid`s it has loaded. A heartbeat older than |
| -five minutes is shown as "stale", so a dead relay cannot look current. |
| +`relay.json` with its `op` and `phase`; the kids the console last |
| +issued per region; delivery per route, "adopted at <time>, restarted" |
| +when a config-ack names the issued kids and a successful restart, |
| +"written, awaiting delivery" when no ack names them yet, "file |
| +delivered, not restarted by the platform" for a TCP exec route with |
| +file delivery (the app must re-read), "restart failed" when the ack |
| +says so, "hibernating, delivered on wake" when the route is asleep, |
| +and "not delivered: route has no config delivery" for a TCP-address |
| +route without it; disable progress as above; and per region, from |
| +`control/relay-ack/<box>.json`, the heartbeat age, whether the record |
| +it holds is current, whether its lease is live, which listeners are |
| +up, and which `kid`s it has loaded. A heartbeat older than five minutes |
| +is shown as "stale", so a dead relay cannot look current. |
| **`carlos relay disable`** applies the transition (`phase: config`), |
| removes the `CARLOS_RELAY` entry through `appconfig.Apply`, publishes |
| @@ -486,26 +524,40 @@ R1-10, R2-4)** |
| region's boxes with `retire_at` `-`. The relay re-reads the file; |
| no restart, no allocation touched. Its next heartbeat lists the new |
| `kid`. |
| -3. Operator updates the console secret for the region to the new |
| - `kid:key`. |
| +3. Operator **adds** the new `kid:key` to the console secret for the |
| + region, which holds every master the console may need, |
| + `CARLOS_RELAY_MASTERS_<REGION>="<kid>:<key>[,<kid>:<key>]"`. Adding |
| + it issues nothing: **which master the console issues from is a |
| + separate, unsealed pointer**, `control/relay-issuer.json` |
| + `{"<region>": "<kid>"}`, and every path that derives a key for an |
| + app (enable, region change, re-enable, reconcile) reads that |
| + pointer and uses the master it names. A staged master is inert |
| + until the pointer moves. **(codex R3-5)** |
| 4. `carlos relay rotate --region r` **refuses until every armed relay |
| - in the region has heartbeated the new `kid`**, then re-derives and |
| - re-delivers keys for every app enabled in that region as an |
| - operation on each record, and prints the earliest permitted |
| - retirement time for the old `kid`: 28 hours from now. |
| -5. `carlos relay status --region r` shows, per app, whether the new |
| - generation is adopted. The old master's `retire_at` may be set to |
| - the later of: 28 hours after rotation, and four hours after the last |
| - app's adoption of the new generation. Setting it earlier is refused |
| - by nothing but the docs, and ends any allocation still on the old |
| - `kid`, which is the force-retirement case §2 excludes from its |
| - guarantee. |
| -6. After `retire_at` the operator removes the line at leisure. |
| + in the region has heartbeated the new `kid`**, then moves the |
| + issuer pointer to it, re-derives and re-delivers keys for every app |
| + enabled in that region as an operation on each record, and prints |
| + the earliest permitted retirement time for the old `kid`: 28 hours |
| + from now. |
| +5. `carlos relay status --region r` shows, per route of every enabled |
| + app, which kid its config-ack names, and lists the routes that |
| + cannot be observed (TCP-address exec routes with file delivery, |
| + which must be restarted by hand, and routes with no delivery). The |
| + old master's `retire_at` may be set to the later of: 28 hours after |
| + rotation, and four hours after the last observed route acked the |
| + new kid with every unobservable route restarted by hand. Setting it |
| + earlier is refused by nothing but the docs, and ends any allocation |
| + still on the old `kid`, which is the force-retirement case §2 |
| + excludes from its guarantee. **(codex R3-4)** |
| +6. After `retire_at` the operator removes the line from the boxes and |
| + the console secret at leisure. |
| **App deletion** removes `relay.json` with the rest of the app prefix, |
| and the deletion walk publishes the aggregate so the app is gone from |
| -authority at once; the reconcile would catch it within five minutes |
| -otherwise. |
| +authority at once. If the walk dies between the two, the reconcile's |
| +unconditional rebuild, which lists records rather than looking for |
| +unfinished ones, publishes without the app within five minutes. |
| +**(codex R3-1)** |
| ## 6. Infrastructure |
| @@ -538,7 +590,10 @@ otherwise. |
| a token bucket dropping a burst; the meter against wrapped in-memory |
| sockets, checking that rejected peer ingress is not counted as `in` |
| and that both forwarding legs are counted; the hourly object surviving |
| - a restart. |
| + a restart, including a restart across an hour boundary that rebuilds |
| + the rolling window from both objects; a wrapper's close removing only |
| + its own binding while a replacement binding for the same typed |
| + source survives; shedding triggered by a fake pressure reading. |
| - End to end, on loopback with a fixture master: a pion TURN client |
| allocates, refreshes, and keeps relaying until the credential's |
| expiry, at which point traffic stops with no further client request; |
| @@ -557,8 +612,13 @@ otherwise. |
| the transition table, the config entry written and removed, the |
| aggregate after concurrent enable and disable of two apps with the |
| etag-first ordering, an enable interrupted after `phase: config` |
| - finished by the reconcile, a superseded writer refused by its CAS, |
| - and `rotate` refusing until the fixture heartbeats list the kid. |
| + finished by the reconcile, a superseded writer refused by its CAS, a |
| + stale config write landing after a disable and undone by the next |
| + reconcile, a deletion that removed the record without publishing |
| + repaired by the next rebuild, an enable during a staged-but-unmoved |
| + rotation issuing from the old kid, and `rotate` refusing until the |
| + fixture heartbeats list the kid; the config sync writing a per-route |
| + ack naming the kids it rendered. |
| - `docs/site/cli.md` gains `### carlos relay` with a fenced example, or |
| the docsite gate fails. |
| - Flagship gate, as amended in the roadmap: a Meet call between two |
| @@ -572,8 +632,9 @@ otherwise. |
| lease older than fifteen minutes refusing new allocations and ending |
| carried ones; a permission to a private address refused; the relay |
| meter object showing the call's bytes on both legs; a rotation |
| - during a call with the call unaffected; the relay shedding and then |
| - being killed first under memory pressure with the edge still serving. |
| + during a call with the call unaffected; the relay shedding |
| + allocations under host memory pressure caused by a tenant, with the |
| + edge still serving. |
| ## 8. Not built |
diff --git a/docs/superpowers/specs/CODEX-FINDINGS-RELAY-R3.md b/docs/superpowers/specs/CODEX-FINDINGS-RELAY-R3.md| new file mode 100644 |
| index 000000000..aa9a6d447 |
| --- /dev/null |
| +++ b/docs/superpowers/specs/CODEX-FINDINGS-RELAY-R3.md |
| @@ -0,0 +1,156 @@ |
| +# Findings on the relay spec, revision 3 |
| + |
| +Reviewed by **GPT-6 (Codex), 2026-09-07**, third pass, against revision 3 at `6b64d75`, with Paul's rulings treated as fixed. Unedited apart from this note. Revision 4 records how each finding was taken. |
| + |
| +| Revision-2 finding | Disposition | Reason | |
| +|---|---|---| |
| +| R2-1 — Aggregate publication | Partial | ETag-first ordering fixes stale publication, but interrupted deletion escapes the specified reconciler. | |
| +| R2-2 — Interrupted operations | Partial | Phases recover interrupted enable; checking `op` before a separate config write does not fence superseded writers. | |
| +| R2-3 — Owner binding | Partial | Typed addresses distinguish transports, but the deletion callback lacks the relay address required by the text. | |
| +| R2-4 — Rotation | Partial | Retirement and loaded-key heartbeats are specified; adoption evidence remains insufficient and other issuance paths bypass the gate. | |
| +| R2-5 — Metering | Partial | Forwarding directions and rejected ingress are corrected; restart recovery loses the preceding-hour portion of the budget. | |
| +| R2-6 — Config adoption | Partial | An acknowledgement is specified, but its identity and granularity cannot establish per-instance adoption. | |
| +| R2-7 — Operational limits | Closed | Account and box admission limits and token buckets supply the bounds chosen by ruling 6. | |
| +| R2-8 — Memory priority | Partial | The new arithmetic omits swap and page tables; local shedding does not establish the stated host-pressure behavior. | |
| +| R2-9 — EIP isolation | Closed | Forwarding is restricted to live relay ports, with a real-EIP acceptance test. | |
| +| R2-10 — Authoritative stage boundary | Partial | Paul’s decisions are recorded, but several contradictory roadmap passages remain. | |
| +| R2-11 — Region ordering | Closed | Console order removes the placement-conversion ambiguity; the stale parent promise is covered below. | |
| +| R2-12 — Incorrect references | Closed | The nonexistent section reference is removed and `writeEnv` now points to its definition. | |
| + |
| +Reviewed commit `6b64d753c9e62d5a4b8550fa5d2cd317ec98c0f4`. Paul’s six rulings are treated as fixed. All three cryptographic vector values reproduce. No files were changed and no live TURN or EC2 tests were run. |
| + |
| +## R3-1 — P1 — Interrupted deletion can leave authority enabled indefinitely |
| + |
| +**Section:** §5, reconciliation and app deletion. |
| + |
| +The reconciler processes existing records whose phase is not `done`. App deletion removes the record itself before publishing the aggregate. If deletion stops between those steps, no unfinished record remains for reconciliation to find. |
| + |
| +The old aggregate can therefore continue authorizing the deleted app. Successful sixty-second reads renew the relay’s lease against that stale aggregate, so the fifteen-minute lease does not bound the failure. A holder of the derived key can keep minting credentials while that authority remains. |
| + |
| +**Evidence:** The [reconciliation algorithm][reconcile] only resumes unfinished records, whereas [app deletion][deletion] claims that reconciliation catches a missing record within five minutes. |
| + |
| +**Correction:** Require an unconditional aggregate rebuild at console start and every five minutes, including when no unfinished records exist, or preserve a deletion tombstone until publication completes. Specify recovery after deletion removes the record but before aggregate publication. |
| + |
| +## R3-2 — P2 — The operation check still permits stale config publication |
| + |
| +**Section:** §5, operation walk. |
| + |
| +The following sequence satisfies the text: |
| + |
| +1. Enable operation A re-reads `relay.json` and confirms its `op`. |
| +2. A pauses before calling `appconfig.Apply`. |
| +3. Disable operation B removes the credentials, publishes the aggregate and reaches `done`. |
| +4. A resumes. `Apply` reads the current config bundle and successfully writes A’s credentials back. |
| +5. A’s subsequent phase CAS fails, but its config write has already landed. |
| + |
| +Reconciliation skips B because it is `done`. The wrong config can persist indefinitely. Re-enable or rotation can similarly leave an enabled app holding superseded credentials. |
| + |
| +**Evidence:** The [operation check][reconcile] precedes the config write. [`applyAt`][apply] obtains its own config ETag and conditionally writes that object; it does not condition publication on the relay operation. |
| + |
| +**Correction:** Define a fence enforced at config publication, covering API handlers and reconciliation. The config write must reject a superseded operation, rather than discovering supersession afterward. Merely adding another record read leaves the same race. |
| + |
| +## R3-3 — P2 — Allocation deletion requires an argument Pion does not provide |
| + |
| +**Section:** §3, owner binding. |
| + |
| +The spec requires `OnAllocationDeleted` to remove only the binding whose relay address matches the event’s. Pion v4.1.4 supplies no relay address in that callback. It supplies source and destination addresses, protocol, username and realm. The creation callback does supply the relay address. |
| + |
| +This matters because Pion removes the allocation from its manager before closing it and issuing the deletion event. Cleanup must not erase a replacement binding created while the old allocation is closing. |
| + |
| +**Evidence:** [Owner-binding requirement][binding]; Pion’s [callback signatures][events] and [deletion sequence][pion-delete]. |
| + |
| +**Correction:** Associate cleanup with the registered relay-socket wrapper, which knows its allocation identity, or specify another available discriminator that cannot remove a replacement binding. Correct the callback claim. |
| + |
| +## R3-4 — P2 — The config acknowledgement cannot prove the rotation cutoff |
| + |
| +**Section:** §2, config delivery and credential lifetime; §5, config adoption and rotation. |
| + |
| +The proposed object is keyed by box, account and app, but adoption happens per route. One app can have multiple routes and environments on one box. If one route restarts successfully while another remains on old config, the successful acknowledgement can make the box appear adopted without identifying the stale instance. |
| + |
| +The generation string also lacks the required relationship to rendered content. The existing `config-gen.json` is a separate change notification across all bundles. Its update can fail after the manifest write, and concurrent touches can coalesce. It is not an immutable identifier embedded in the config being rendered. |
| + |
| +There is also a direct delivery contradiction: §2 says exec routes opting into file delivery restart, while §5 says they are not restarted. Existing socket-exec routes receive files and are bounced; opted-in TCP-address routes receive files but are never bounced. |
| + |
| +These distinctions determine whether the last issuer actually stopped using the old key. They cannot be collapsed into “a config-ack for that generation exists.” |
| + |
| +**Evidence:** [Acknowledgement schema and status rules][ack]; [delivery and lifetime promises][delivery]; existing [per-route/environment synchronization][sync], [generation semantics][generation], and [exec delivery contract][syncexec]. |
| + |
| +**Correction:** Define acknowledgements per route, including the effective relay key identity actually adopted and the result of application/restart. Distinguish running adoption, hibernating delivery, failed restart and externally managed processes. Bind the acknowledgement to the rendered config identity rather than relying on the app-wide notification alone. Retirement must wait for evidence covering every issuer; file delivery alone must not satisfy that condition. |
| + |
| +## R3-5 — P2 — Enable and reconciliation can issue the new key before the rotation gate passes |
| + |
| +**Section:** §5, enable, reconciliation and rotation. |
| + |
| +Rotation updates the console’s master secret before invoking the command that checks relay heartbeats. During that interval, a first enable, region addition or resumed config operation derives credentials from the region’s current console master. |
| + |
| +Those paths have no heartbeat gate. They can distribute the new `kid` while a relay has failed to load it—even while `rotate` correctly refuses to proceed. A member then receives credentials that an advertised relay rejects. |
| + |
| +**Evidence:** [Enable selects the current master][enable]; [reconciliation derives from the record and masters][reconcile]; [rotation changes the console secret before checking acknowledgements][rotation]. |
| + |
| +**Correction:** Separate staged masters from the active issuance master, and switch issuance only after the heartbeat gate succeeds. Every credential-writing path must use that gated selection, including reconciliation and region additions. |
| + |
| +## R3-6 — P2 — Restart recovery drops part of the rolling-hour budget |
| + |
| +**Section:** §3, limits and metering. |
| + |
| +The spec explicitly recovers only the current hour’s metrics object. At 12:01, the rolling-hour budget still includes traffic from 11:01 onward, most of which resides in the preceding object. |
| + |
| +A restart therefore forgets almost the entire window and can immediately readmit an app or account whose budget remains exhausted. This contradicts the stated recovery behavior. |
| + |
| +**Evidence:** [Rolling-hour limits and current-hour recovery][meters]. |
| + |
| +**Correction:** Load every hourly object intersecting the rolling window—normally the previous and current hour—and rebuild budgets from the relevant samples. Include a restart across an hour boundary in the acceptance tests. |
| + |
| +## R3-7 — P2 — The OOM arithmetic does not establish the claimed ordering |
| + |
| +**Section:** §3, hardening and memory; §7, flagship gate. |
| + |
| +The stated tenant maximum of 1000 follows from RSS alone. Linux’s score also includes swap entries and page tables, and global scoring uses RAM plus swap. This repository permits a tenant to use swap equal to its memory cap, so a 1 GiB memory ceiling does not establish the claimed scoring bound. [Linux scoring implementation](https://raw.githubusercontent.com/torvalds/linux/v6.12/mm/oom_kill.c); [tenant swap allowance][swap]. |
| + |
| +There is a separate eligibility problem: exhausting the edge’s 80% cgroup ceiling invokes a kill within that tree. The relay, in its own unit, is not a candidate. Shedding only when the relay exceeds its own 160 MiB threshold does not respond to pressure created elsewhere while the relay remains small. |
| + |
| +**Evidence:** [The arithmetic and shedding trigger][memory]; [the edge’s cgroup boundary][edge-memory]; [the relay-first gate][relay-gate]. |
| + |
| +**Correction:** State which pressure conditions the guarantee covers and correct the scoring envelope to include swap and page tables. If relay-first shedding is required under host-wide pressure, specify a host-pressure trigger. Align the gate with that scope; the fixed defaults alone do not establish it. |
| + |
| +## R3-8 — P2 — The roadmap still contradicts the approved relay contract |
| + |
| +**Sections:** Roadmap stage 4a and §11; relay §§2–5 and §9. |
| + |
| +The amendments resolve TLS, credential duration and the main metering gate, but leave these conflicting instructions: |
| + |
| +| Topic | Remaining roadmap text | Revision-3 contract | |
| +|---|---|---| |
| +| Master exposure | The custody paragraph describes exposure of enabled apps’ derived keys and calls the boundary no worse than existing secrets. | Ruling 3 explicitly exposes every region’s master through console sealed config, including authority to derive future app keys. | |
| +| Region order | Placement puts the instance’s own region first. | Every instance receives console order. | |
| +| Peer policy | Permissions refuse the relay’s own addresses. | The EIP is permitted, with forwarding restricted to live relay ports. | |
| +| Authority lifecycle | Authority is synced from the route table; disable bumps incarnation. | Authority derives from per-app relay records; disable removes authority and re-enable creates the new incarnation. | |
| +| Accounting amendment | §11 assigns three relay meters to stage 4a. | Ruling 5 assigns two meters to 4a and moves the inter-region meter to stage 5. | |
| + |
| +**Evidence:** Roadmap [authority and custody][roadmap-authority], [peer policy and placement][roadmap-policy], and [unit-economics amendment][roadmap-accounting]. |
| + |
| +**Correction:** Update these passages to reflect the fixed rulings and revision-3 mechanisms. No further policy decision is needed; the parent must stop giving the implementation plan conflicting instructions. |
| + |
| +**Verdict: request changes.** One P1 and seven P2 findings remain. |
| + |
| +[reconcile]: /home/paulca/amadan.net/carlos/platform/.claude/worktrees/media-roadmap-spec/docs/superpowers/specs/2026-09-07-relay-stun-turn-design.md:389 |
| +[deletion]: /home/paulca/amadan.net/carlos/platform/.claude/worktrees/media-roadmap-spec/docs/superpowers/specs/2026-09-07-relay-stun-turn-design.md:505 |
| +[apply]: /home/paulca/amadan.net/carlos/platform/.claude/worktrees/media-roadmap-spec/internal/appconfig/store.go:146 |
| +[binding]: /home/paulca/amadan.net/carlos/platform/.claude/worktrees/media-roadmap-spec/docs/superpowers/specs/2026-09-07-relay-stun-turn-design.md:199 |
| +[events]: /home/paulca/.claude/jobs/f1f6099c/tmp/gomodcache/github.com/pion/turn/v4@v4.1.4/internal/allocation/event_handler.go:20 |
| +[pion-delete]: /home/paulca/.claude/jobs/f1f6099c/tmp/gomodcache/github.com/pion/turn/v4@v4.1.4/internal/allocation/allocation_manager.go:150 |
| +[ack]: /home/paulca/amadan.net/carlos/platform/.claude/worktrees/media-roadmap-spec/docs/superpowers/specs/2026-09-07-relay-stun-turn-design.md:455 |
| +[delivery]: /home/paulca/amadan.net/carlos/platform/.claude/worktrees/media-roadmap-spec/docs/superpowers/specs/2026-09-07-relay-stun-turn-design.md:108 |
| +[sync]: /home/paulca/amadan.net/carlos/platform/.claude/worktrees/media-roadmap-spec/internal/appconfig/sync.go:30 |
| +[generation]: /home/paulca/amadan.net/carlos/platform/.claude/worktrees/media-roadmap-spec/internal/appconfig/generation.go:28 |
| +[syncexec]: /home/paulca/amadan.net/carlos/platform/.claude/worktrees/media-roadmap-spec/internal/appconfig/syncexec.go:1 |
| +[enable]: /home/paulca/amadan.net/carlos/platform/.claude/worktrees/media-roadmap-spec/docs/superpowers/specs/2026-09-07-relay-stun-turn-design.md:442 |
| +[rotation]: /home/paulca/amadan.net/carlos/platform/.claude/worktrees/media-roadmap-spec/docs/superpowers/specs/2026-09-07-relay-stun-turn-design.md:481 |
| +[meters]: /home/paulca/amadan.net/carlos/platform/.claude/worktrees/media-roadmap-spec/docs/superpowers/specs/2026-09-07-relay-stun-turn-design.md:242 |
| +[swap]: /home/paulca/amadan.net/carlos/platform/.claude/worktrees/media-roadmap-spec/internal/cgroup/cgroup.go:313 |
| +[memory]: /home/paulca/amadan.net/carlos/platform/.claude/worktrees/media-roadmap-spec/docs/superpowers/specs/2026-09-07-relay-stun-turn-design.md:157 |
| +[edge-memory]: /home/paulca/amadan.net/carlos/platform/.claude/worktrees/media-roadmap-spec/internal/host/units/carlos-edge.service:57 |
| +[relay-gate]: /home/paulca/amadan.net/carlos/platform/.claude/worktrees/media-roadmap-spec/docs/superpowers/specs/2026-09-07-relay-stun-turn-design.md:560 |
| +[roadmap-authority]: /home/paulca/amadan.net/carlos/platform/.claude/worktrees/media-roadmap-spec/docs/superpowers/specs/2026-09-07-media-platform-roadmap-design.md:620 |
| +[roadmap-policy]: /home/paulca/amadan.net/carlos/platform/.claude/worktrees/media-roadmap-spec/docs/superpowers/specs/2026-09-07-media-platform-roadmap-design.md:656 |
| +[roadmap-accounting]: /home/paulca/amadan.net/carlos/platform/.claude/worktrees/media-roadmap-spec/docs/superpowers/specs/2026-09-07-media-platform-roadmap-design.md:979 |
| \ No newline at end of file |