| new file mode 100644 |
| index 000000000..595edafad |
| --- /dev/null |
| +++ b/docs/superpowers/specs/2026-09-07-relay-stun-turn-design.md |
| @@ -0,0 +1,348 @@ |
| +# Stage 4a: a platform STUN and TURN relay |
| + |
| +**Date:** 2026-09-07 |
| +**Status:** draft for codex review, then Paul. Stage 4a of the media |
| +roadmap (`2026-09-07-media-platform-roadmap-design.md`), built under |
| +rulings 8, 10 and 11 there. One parameter differs from the roadmap's |
| +text and is flagged in §9 for Paul: credential life is four hours, not |
| +one. |
| + |
| +**Why now.** Oficina's Meet is peer-to-peer WebRTC and cannot name a |
| +third-party STUN server, because suite ruling 6 forbids egress that is a |
| +property of the product. Without STUN, browsers gather only LAN |
| +candidates and calls connect on the same network only. A platform STUN |
| +listener is the smallest thing that makes Meet work across the |
| +internet; TURN on the same listener is what makes the one call in ten |
| +that cannot hole-punch connect anyway. Meet is blocked on nothing else |
| +(Meet session, 2026-09-07), and it has reviewed the app-facing contract |
| +in §2. |
| + |
| +## 1. What is built |
| + |
| +One new platform unit, `carlos-relay.service`, on every region box that |
| +the operator enables, running `carlos relay serve` from the platform |
| +binary, built on `pion/turn` v4, pure Go. It answers STUN binding |
| +requests to anyone and TURN allocations to holders of a credential an |
| +enabled app minted. It learns which apps are enabled from a record the |
| +console writes, meters what it relays per app, and refuses peers on |
| +private networks. |
| + |
| +Three console doors and a CLI verb, `carlos relay enable|status|disable`, |
| +that turn the relay on for an app and deliver its keys as sealed config. |
| +An operator-side key per region, staged in `host.env` like the other |
| +platform secrets. A security-group change opening the relay's ports. A |
| +DNS name per region relay. |
| + |
| +Not built: an SFU, recording, an issuance API, client-side relay |
| +selection, TURN over TLS on 443, commercial quotas. §8 lists them. |
| + |
| +## 2. The app-facing contract |
| + |
| +This is what Meet codes against. It is fixed by this spec; the rest of |
| +the design may change without touching it. |
| + |
| +**Environment.** Two variables reach the instance through the sealed |
| +`config.json` path, the way store credentials do: |
| + |
| +- `CARLOS_RELAY`: a JSON array, one element per region the app may use, |
| + nearest first (the instance's own region first, then the rest of the |
| + app's allowed list in the console's order). Absent or empty means no |
| + relay: the app shows its floor. |
| + |
| + [{"region":"eu-west-1", |
| + "urls":["stun:relay.eu-west-1.oncarlos.com:3478", |
| + "turn:relay.eu-west-1.oncarlos.com:3478?transport=udp", |
| + "turn:relay.eu-west-1.oncarlos.com:3478?transport=tcp", |
| + "turns:relay.eu-west-1.oncarlos.com:5349"], |
| + "key":"<base64, 32 bytes>"}] |
| + |
| +- `CARLOS_RELAY_INCARNATION`: sixteen lowercase hex characters. It is |
| + not per region. |
| + |
| +**Minting, per call.** The app computes, for the region it wants: |
| + |
| + username = "<expiry>:<account>:<app>:<incarnation>" |
| + credential = base64std( HMAC-SHA256( key, username ) ) |
| + |
| +where `expiry` is a Unix time in seconds no more than four hours ahead, |
| +`account` is the account's sqid (alphabet `[a-z0-9]`), `app` is the app |
| +name (`^[a-z][a-z0-9-]{0,31}$`), and `incarnation` is the value above. |
| +None of the fields can contain a colon, so the relay parses by splitting |
| +on `:` into exactly four fields. The app hands the browser an |
| +`RTCIceServer` list: `stun:` entries bare, `turn:` and `turns:` entries |
| +carrying `username` and `credential`. STUN needs no credential. |
| + |
| +**Expiry** is checked by the relay on every authenticated request: |
| +allocate and every refresh, permission and channel bind. A credential |
| +that expires mid-call ends that call's relay at the next refresh, which |
| +pion sends every few minutes. Four hours covers any meeting; an app |
| +that wants longer re-mints and performs an ICE restart before the hour |
| +is up, and Meet does not need to. **(differs from the roadmap's one |
| +hour; §9)** |
| + |
| +**What changes the incarnation.** Only `carlos relay disable` followed by |
| +`carlos relay enable`. Deploys, restarts, key rotation and region changes |
| +do not. An in-flight credential survives everything except disable. |
| + |
| +**What changes the key.** Region master rotation (§5) and the |
| +incarnation. Either re-renders the app's config, which restarts the |
| +instance within about a minute, as any config change does today. The |
| +relay accepts the previous master for twenty-four hours after rotation, |
| +so a credential minted before the restart keeps working for its life. |
| + |
| +**Test vector.** Master is `SHA-256("carlos-relay-test-master")`, |
| +account `abc`, app `meet`, incarnation `0123456789abcdef`, expiry |
| +`1900000000`: |
| + |
| + master 841d374df509340d6ab18a139a27ed0d9c5d61d0bdf306e4124ff913a7a92fb5 (hex) |
| + key uMhwW4P/9O5dQUIs91t2CCHGhwJbeiM0xQMQ2G0CJLU= |
| + username 1900000000:abc:meet:0123456789abcdef |
| + credential khBx1BoommS1C69pSAgv0r0TM2H9IKDDE0M9OVr7VXQ= |
| + |
| +The key derivation is in §4; an app never needs it, only the key it is |
| +given. The vector is checked into `internal/relay/vector_test.go` and |
| +Meet's minting test uses the same numbers. |
| + |
| +**Logging.** The app logs which region it used and nothing else about |
| +the credential. The relay logs the account, app and outcome of an |
| +authentication, never the credential or the derived key. |
| + |
| +## 3. The relay unit |
| + |
| +`carlos-relay.service` is a §3 media service in the roadmap's terms: a |
| +network-facing platform unit like the edge, not a codec helper. |
| + |
| +- **Process.** `carlos relay serve`, from the platform binary, so it |
| + rides the same release, promote and adopt path as the edge. A new |
| + file in `internal/host/units/`, added to `enabledUnits` in |
| + `bootstrap.go` under a guard: the unit is written on every box but |
| + armed only where `host.env` carries `CARLOS_RELAY_MASTERS`, so a box |
| + without a master never listens. Bootstrap is idempotent and converges |
| + an existing box the way it does for every unit. |
| +- **Hardening.** Copied from `carlos-edge.service` and then cut down: no |
| + `CAP_NET_BIND_SERVICE` (3478 and 5349 are unprivileged), `User=carlos`, |
| + `ProtectSystem=strict`, `ReadWritePaths=/data/relay /run/carlos`, |
| + `RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX`, |
| + `MemoryHigh=20%`, `MemoryMax=25%`, `TasksMax=512`, `Restart=always`. |
| + It shares the box with the edge and must lose first: `Nice=5` and |
| + `CPUWeight=50` against the edge's default. |
| +- **Listeners.** UDP 3478 and TCP 3478 on the box's primary address; TLS |
| + 5349 when a certificate for the relay hostname is present. The relay |
| + allocation range is UDP 49152 to 65535, configured as |
| + `RelayAddressGeneratorPortRange` with `RelayAddress` set to the box's |
| + Elastic IP, which the process learns the way the edge does, from |
| + IMDS (`internal/host/agent.go:1761`), with `CARLOS_RELAY_PUBLIC_IP` as |
| + the override for a box without IMDS. |
| +- **Certificate for 5349.** The relay hostname is a platform host the |
| + edge already knows how to mint a certificate for; the edge adds |
| + `relay.<aws region>.<platform domain>` to its allowed set on boxes |
| + with a master, and the relay reads the certificate from the edge's |
| + autocert cache read-only, reloading when the file changes. If no |
| + certificate is present yet, the TLS listener is not started and the |
| + relay logs once; `turns:` is still advertised, because the browser |
| + falls through to the other URLs. |
| +- **STUN.** Binding requests are answered on every listener without |
| + authentication, which is pion's default and the whole point of §0. |
| +- **Authentication.** A custom `AuthHandler` (§4). pion calls it on every |
| + authenticated request, which is where expiry is enforced. |
| +- **Peer policy.** A `PermissionHandler` that refuses `CreatePermission` |
| + and `ChannelBind` to loopback, link-local, RFC 1918 and unique-local |
| + ranges, the box's own VPC CIDR (from `CARLOS_RELAY_DENY_CIDRS`, set by |
| + the operator from tofu output), and the relay's own addresses. |
| + Refusals are counted per app. |
| +- **Limits.** Per app, per box, in the relay process: at most 64 live |
| + allocations, at most 30 new allocations a minute, allocation lifetime |
| + capped at ten minutes so a refresh is required (pion's default), and |
| + a byte budget of 20 GB per hour after which new allocations for that |
| + app are refused with 486 (allocation quota reached) until the hour |
| + rolls. Per box: at most 2048 allocations. These are operational |
| + limits, tuned by env with those defaults; commercial quotas are out |
| + of scope. |
| +- **Metering.** The relay wraps its UDP and TCP connections to count |
| + bytes in and bytes out per allocation, attributes them to the |
| + allocation's account and app, and once a minute appends a sample to |
| + the box's metrics chunks alongside the per-unit samples the metrics |
| + loop already flushes (`internal/host/metrics_loop.go`). The sample |
| + carries `relay_in`, `relay_out` and `relay_allocs` per app. Reading |
| + those samples into an app page is stage 5 work and not in this spec; |
| + the numbers exist from the first byte relayed. |
| +- **Authority record.** The relay reads `control/relay.json` (§5) from |
| + the deployment bucket on start and whenever `control/serial` changes, |
| + through the same fast-converge poll the host agent already runs |
| + (`internal/host/agent.go:1304`). If the record it holds is older than |
| + fifteen minutes it refuses new allocations and ends the ones it is |
| + carrying, per the roadmap's serving lease. A relay with no record at |
| + all serves STUN only. |
| + |
| +## 4. Keys and verification |
| + |
| +**Region master.** Thirty-two random bytes per region, generated by the |
| +operator with `carlos relay keygen` and staged as |
| +`CARLOS_RELAY_MASTERS="<kid>:<base64>[,<kid>:<base64>]"` in that |
| +region's `host.env`, the way `CARLOS_SECRETS_KEY` is staged today. The |
| +console holds every region's masters as sealed console secrets, |
| +`CARLOS_RELAY_MASTERS_<REGION>`, so it can derive app keys at enable. |
| +Two copies, both staged by hand, both rotated by hand; §5 gives the |
| +procedure. There is no bucket copy of a master, sealed or otherwise, |
| +because sealed config is decryptable on every host (roadmap §1). |
| + |
| +**Derived key.** For an app, region and incarnation: |
| + |
| + key = HKDF-SHA256( ikm = master, |
| + salt = "carlos-relay-v1", |
| + info = account || 0x00 || app || 0x00 || incarnation, |
| + length = 32 ) |
| + |
| +**Verification**, in the relay's `AuthHandler(username, realm, src)`: |
| + |
| +1. Split the username on `:` into exactly four fields; anything else is |
| + refused. |
| +2. Parse `expiry` as a decimal Unix time; refuse if it is in the past or |
| + more than four hours ahead. |
| +3. Look up `account/app` in the authority record; refuse if absent, if |
| + the app's incarnation differs, or if this relay's region is not in |
| + the app's allowed list. |
| +4. For the current master, then the previous one if within its |
| + twenty-four-hour grace: derive the key, compute the expected |
| + credential, and return `turn.GenerateAuthKey(username, realm, |
| + credential)`. pion verifies `MESSAGE-INTEGRITY` against that key, so |
| + a wrong credential fails inside pion; the handler itself never sees |
| + the credential the client sent. |
| +5. Refuse with the reason counted per app: bad shape, expired, unknown |
| + app, wrong incarnation, wrong region. |
| + |
| +The MD5 inside `GenerateAuthKey` is the long-term credential mechanism |
| +RFC 8489 mandates and is not ours to choose; the HMAC-SHA256 above is. |
| + |
| +## 5. Enable, disable, rotate |
| + |
| +**Records.** Two, both additive: |
| + |
| +- `apps/<acct>/<app>/relay.json`, CAS-guarded like `store.json`, holding |
| + `{enabled, incarnation, regions, enabled_at, disabled_at}`. The |
| + incarnation is sixteen hex characters drawn at random at each enable, |
| + never a counter, so reuse across delete and name reuse is not a |
| + question. |
| +- `control/relay.json`, the authority record the relays read: `{apps: |
| + [{account, app, incarnation, regions}], written_at}`, rewritten by |
| + the console from the per-app records on every enable and disable, |
| + followed by a `control/serial` bump so every box notices within |
| + seconds. |
| + |
| +**`carlos relay enable [--regions r1,r2]`** (member, API-mode only, on |
| +the store model in `store.go` and `internal/console/apistore.go`): |
| + |
| +1. `apiAccountApp` gate; unknown and foreign are one byte-identical 404. |
| +2. Refuse if the deployment has no relay regions (`CARLOS_RELAY_REGIONS` |
| + in console config is empty) with a message naming the operator. |
| +3. Regions default to every region in `CARLOS_RELAY_REGIONS` ordered |
| + with the app's placement region first; `--regions` narrows. |
| +4. Write `relay.json` with a fresh incarnation and the region list. |
| +5. Derive one key per region from that region's current master and |
| + write `CARLOS_RELAY` (as `kind:"secret"`, since it carries keys) and |
| + `CARLOS_RELAY_INCARNATION` (as `kind:"env"`) through |
| + `appconfig.Apply`, the `writeEnv` shape in `internal/objstore/walk.go`. |
| + That bumps the app's config generation; the box re-renders and |
| + restarts the instance within about a minute. |
| +6. Rewrite `control/relay.json`, bump `control/serial`. |
| +7. Answer with the record; the CLI prints regions, incarnation and |
| + "config delivered, instance restarting". |
| + |
| +**`carlos relay status`** prints the record and, per region, whether |
| +that region's relay has acknowledged the current `control/relay.json` |
| +(each relay writes `control/relay-ack/<box>.json` with the |
| +`written_at` it holds, every minute), so a member can see "eu-west-1: |
| +current, ap-southeast-2: 3 minutes behind" without a box command. |
| + |
| +**`carlos relay disable`** clears the two env entries, marks the record |
| +disabled, rewrites `control/relay.json` without the app, bumps the |
| +serial. Every relay ends the app's allocations on its next record read. |
| + |
| +**Rotation** (`carlos relay rotate --region r`, operator): |
| + |
| +1. Operator runs `carlos relay keygen`, adds the new `kid:key` in front |
| + of the old one in that region's `host.env`, restarts |
| + `carlos-relay.service`; the relay now accepts both, new first. |
| +2. Operator updates the console secret for that region the same way. |
| +3. `carlos relay rotate` re-derives and re-delivers keys for every app |
| + enabled in that region; each instance restarts within a minute. |
| +4. After twenty-four hours the operator removes the old entry from |
| + `host.env` and the console secret. Any credential minted from the old |
| + key has expired by then. |
| + |
| +**App deletion** removes `relay.json` with the rest of the app prefix |
| +and the console's deletion walk drops the app from `control/relay.json`. |
| + |
| +## 6. Infrastructure |
| + |
| +- `infra/modules/region-host/main.tf`: a `relay` variable, default off, |
| + that adds ingress for UDP 3478, TCP 3478, TCP 5349 and UDP 49152 to |
| + 65535 from `0.0.0.0/0`, and outputs the VPC CIDR for |
| + `CARLOS_RELAY_DENY_CIDRS`. Flagship turns it on per region. |
| +- DNS: an A record `relay.<aws region>.<platform domain>` to the region's |
| + Elastic IP, next to the existing EIP resource. One line per region. |
| +- `host.env` on each enabled region box gains `CARLOS_RELAY_MASTERS` |
| + and `CARLOS_RELAY_DENY_CIDRS`. |
| +- Console config gains `CARLOS_RELAY_REGIONS` and one |
| + `CARLOS_RELAY_MASTERS_<REGION>` secret per region. |
| +- Egress cost: every relayed byte leaves the box once. The |
| + unit-economics spec gets a line for relay egress per region, and the |
| + meters in §3 are what fill it. |
| + |
| +## 7. Testing |
| + |
| +- `internal/relay`: the key derivation and credential functions against |
| + the §2 test vector; the `AuthHandler` against a fixture authority |
| + record covering every refusal reason; the permission handler against |
| + each denied range; the limits against a fake clock; the meter against |
| + a wrapped in-memory conn. |
| +- An end-to-end test that starts the relay on loopback with a fixture |
| + master, mints a credential with the app-side code, and drives a pion |
| + TURN client through allocate, refresh past expiry, and a permission |
| + to a denied peer. |
| +- `internal/host`: the unit is embedded and written by bootstrap |
| + (`TestUnitsEmbedComplete`, `TestBootstrapWritesUnitsToStagingDir`), |
| + and armed only when the staging `host.env` carries a master. |
| +- `internal/console`: the three doors through `rawReq` and a |
| + `relayURL` fixture, asserting the byte-identical 404, the no-regions |
| + refusal, the config entries written, and `control/relay.json` after |
| + enable and disable. |
| +- `docs/site/cli.md` gains `### carlos relay` with a fenced example, or |
| + the docsite gate fails. |
| +- Flagship gate, from the roadmap: a Meet call between two symmetric-NAT |
| + networks connecting through the Ireland relay; a credential minted for |
| + one app refused for another; a credential from before a disable |
| + refused after re-enable and its allocation ended; a relay with a |
| + record older than fifteen minutes refusing new allocations and ending |
| + carried ones; a permission to a private address refused; the meters |
| + moving; a 443-only network failing as documented. |
| + |
| +## 8. Not built |
| + |
| +- TURN over TLS on 443: needs a second address per box. |
| +- An issuance API: apps mint offline from delivered keys; the alternative |
| + is named in the roadmap if the trade ever changes. |
| +- Client-aware relay selection: the list is ordered by the instance's |
| + region; ICE picks. |
| +- Commercial quotas and an app-page usage view: the meters exist, the |
| + page is stage 5. |
| +- An SFU: stage 4b, after this ships. |
| +- Any change to how apps signal: SSE plus POST stays Meet's. |
| + |
| +## 9. Decisions for Paul |
| + |
| +1. **Credential life four hours, not one.** The roadmap's recommendation |
| + said one hour. pion enforces expiry on every refresh, so a one-hour |
| + credential drops the relay from any call longer than an hour unless |
| + the app re-mints and restarts ICE. Four hours makes that Meet's |
| + problem never. Recommended. |
| +2. **Enable is member-level, no operator grant.** The store model gates |
| + `grant` on operators because it spends the deployment's cloud |
| + account. Enabling a relay spends bandwidth, bounded by the §3 limits, |
| + and the operator's consent is the region list in console config. |
| + Recommended over a per-app operator step that Meet would wait on. |
| +3. **Limits' defaults** (64 allocations, 30 a minute, 20 GB an hour per |
| + app per box). Numbers to tune; the mechanism is the decision. |
| +4. **`turns:` advertised before its certificate exists.** Harmless, since |
| + browsers fall through, and it means no config re-render when the |
| + certificate lands. Recommended. |
amadan