| index 595edafad..ceee6269d 100644 |
| --- a/docs/superpowers/specs/2026-09-07-relay-stun-turn-design.md |
| +++ b/docs/superpowers/specs/2026-09-07-relay-stun-turn-design.md |
| @@ -1,11 +1,12 @@ |
| # Stage 4a: a platform STUN and TURN relay |
| |
| -**Date:** 2026-09-07 |
| -**Status:** draft for codex review, then Paul. Stage 4a of the media |
| -roadmap (`2026-09-07-media-platform-roadmap-design.md`), built under |
| -rulings 8, 10 and 11 there. One parameter differs from the roadmap's |
| -text and is flagged in §9 for Paul: credential life is four hours, not |
| -one. |
| +**Date:** 2026-09-07, revision 2 |
| +**Status:** draft. Revision 1 was reviewed by GPT-6 via Codex; its 20 |
| +findings are in `CODEX-FINDINGS-RELAY.md`, unedited, and every one is |
| +taken here, marked **(codex R1-N)**. Stage 4a of the media roadmap |
| +(`2026-09-07-media-platform-roadmap-design.md`), built under rulings 8, |
| +10 and 11 there. §9 lists what Paul is asked to decide, including three |
| +places this spec departs from the roadmap's text. |
| |
| **Why now.** Oficina's Meet is peer-to-peer WebRTC and cannot name a |
| third-party STUN server, because suite ruling 6 forbids egress that is a |
| @@ -19,172 +20,258 @@ in §2. |
| |
| ## 1. What is built |
| |
| -One new platform unit, `carlos-relay.service`, on every region box that |
| -the operator enables, running `carlos relay serve` from the platform |
| -binary, built on `pion/turn` v4, pure Go. It answers STUN binding |
| +One new platform unit, `carlos-relay.service`, on every region box the |
| +operator arms, running `carlos relay serve` from the platform binary, |
| +built on `pion/turn` v4.1.4, pure Go, which fetches and cross-compiles |
| +clean for the CI shards (checked 2026-09-07). It answers STUN binding |
| requests to anyone and TURN allocations to holders of a credential an |
| enabled app minted. It learns which apps are enabled from a record the |
| -console writes, meters what it relays per app, and refuses peers on |
| +console writes, binds every allocation to the identity that made it, |
| +ends allocations when their credential, app, region or its own |
| +authority lapses, meters what it relays per app, and refuses peers on |
| private networks. |
| |
| Three console doors and a CLI verb, `carlos relay enable|status|disable`, |
| that turn the relay on for an app and deliver its keys as sealed config. |
| -An operator-side key per region, staged in `host.env` like the other |
| -platform secrets. A security-group change opening the relay's ports. A |
| -DNS name per region relay. |
| +An operator-side master key per region, staged in a file on the region's |
| +boxes and, as a console secret, in the console. A security-group change |
| +opening the relay's ports. A DNS name per region relay. |
| |
| -Not built: an SFU, recording, an issuance API, client-side relay |
| -selection, TURN over TLS on 443, commercial quotas. §8 lists them. |
| +Not built: TURN over TLS on 5349 or 443, an SFU, recording, an issuance |
| +API, client-side relay selection, an inter-region byte meter, commercial |
| +quotas. §8 lists them with reasons. |
| |
| ## 2. The app-facing contract |
| |
| This is what Meet codes against. It is fixed by this spec; the rest of |
| the design may change without touching it. |
| |
| -**Environment.** Two variables reach the instance through the sealed |
| -`config.json` path, the way store credentials do: |
| +**Environment.** One variable reaches the instance through the sealed |
| +`config.json` path, the way store credentials do. Because the config |
| +renderer refuses quotes, backslashes and newlines in any value |
| +(`internal/appconfig/materialize.go:23`), the value is the **standard |
| +base64 encoding, with padding, of a JSON document**; the app decodes |
| +the base64, then parses the JSON. **(codex R1-1)** |
| |
| -- `CARLOS_RELAY`: a JSON array, one element per region the app may use, |
| - nearest first (the instance's own region first, then the rest of the |
| - app's allowed list in the console's order). Absent or empty means no |
| +- `CARLOS_RELAY`: base64 of a JSON array, one element per region the |
| + app may use, in the console's order (§5). Absent or empty means no |
| relay: the app shows its floor. |
| |
| [{"region":"eu-west-1", |
| "urls":["stun:relay.eu-west-1.oncarlos.com:3478", |
| "turn:relay.eu-west-1.oncarlos.com:3478?transport=udp", |
| - "turn:relay.eu-west-1.oncarlos.com:3478?transport=tcp", |
| - "turns:relay.eu-west-1.oncarlos.com:5349"], |
| - "key":"<base64, 32 bytes>"}] |
| - |
| -- `CARLOS_RELAY_INCARNATION`: sixteen lowercase hex characters. It is |
| - not per region. |
| - |
| -**Minting, per call.** The app computes, for the region it wants: |
| - |
| - username = "<expiry>:<account>:<app>:<incarnation>" |
| - credential = base64std( HMAC-SHA256( key, username ) ) |
| - |
| -where `expiry` is a Unix time in seconds no more than four hours ahead, |
| -`account` is the account's sqid (alphabet `[a-z0-9]`), `app` is the app |
| -name (`^[a-z][a-z0-9-]{0,31}$`), and `incarnation` is the value above. |
| -None of the fields can contain a colon, so the relay parses by splitting |
| -on `:` into exactly four fields. The app hands the browser an |
| -`RTCIceServer` list: `stun:` entries bare, `turn:` and `turns:` entries |
| + "turn:relay.eu-west-1.oncarlos.com:3478?transport=tcp"], |
| + "key":"<standard base64, with padding, of 32 raw bytes>", |
| + "username_suffix":"abc:meet:0123456789abcdef:k1"}] |
| + |
| + `region` is the AWS region name, which is also the vocabulary of the |
| + relay hostnames. `username_suffix` is `<account>:<app>:<incarnation>: |
| + <kid>`, supplied by the platform so the app never has to know its own |
| + canonical account sqid or app name. **(codex R1-11, R1-12)** There is |
| + no separate incarnation variable; the suffix carries it. |
| + |
| +**Minting, per call.** For the region it wants, the app computes: |
| + |
| + key = base64_decode( element.key ) // 32 raw bytes |
| + username = "<expiry>:" + element.username_suffix // ASCII |
| + credential = base64_std( HMAC-SHA256( key, username ) ) |
| + |
| +where `expiry` is a decimal Unix time in seconds, at most four hours |
| +ahead. The username has exactly five colon-separated fields: |
| +`expiry` (`[0-9]+`), `account` (`[a-z0-9]+`), `app` |
| +(`^[a-z][a-z0-9-]{0,31}$`), `incarnation` (`[0-9a-f]{16}`) and `kid` |
| +(`[a-z0-9]{1,8}`). No field can contain a colon. The app hands the |
| +browser an `RTCIceServer` list: `stun:` entries bare, `turn:` entries |
| carrying `username` and `credential`. STUN needs no credential. |
| |
| -**Expiry** is checked by the relay on every authenticated request: |
| -allocate and every refresh, permission and channel bind. A credential |
| -that expires mid-call ends that call's relay at the next refresh, which |
| -pion sends every few minutes. Four hours covers any meeting; an app |
| -that wants longer re-mints and performs an ICE restart before the hour |
| -is up, and Meet does not need to. **(differs from the roadmap's one |
| -hour; §9)** |
| +**Expiry.** The relay checks the expiry on every authenticated request |
| +the browser's ICE agent sends (allocate, refresh, permission, channel |
| +bind) **and** ends the allocation at the expiry regardless of what the |
| +client sends (§3, "termination"), so a credential's deadline is a hard |
| +deadline on relayed traffic. An app that needs a call to outlive its |
| +credential re-mints and performs an ICE restart before the deadline. |
| +Four hours is the maximum, not a promise about meeting lengths. |
| +**(codex R1-4, R1-20; differs from the roadmap's one hour, §9)** |
| |
| **What changes the incarnation.** Only `carlos relay disable` followed by |
| -`carlos relay enable`. Deploys, restarts, key rotation and region changes |
| -do not. An in-flight credential survives everything except disable. |
| - |
| -**What changes the key.** Region master rotation (§5) and the |
| -incarnation. Either re-renders the app's config, which restarts the |
| -instance within about a minute, as any config change does today. The |
| -relay accepts the previous master for twenty-four hours after rotation, |
| -so a credential minted before the restart keeps working for its life. |
| +`carlos relay enable`. A repeated enable, a region change, a deploy, a |
| +restart and a master rotation all keep it. **(codex R1-13)** |
| + |
| +**What changes the key.** A master rotation in a region changes that |
| +region's key and `kid`; a new incarnation changes every region's key. |
| +Either re-renders the app's config, which for unit-backed and |
| +exec-backed routes restarts the instance within about a minute, as any |
| +config change does today; a TCP-address route receives the file and is |
| +not restarted (`internal/appconfig/syncexec.go:11`), and must re-read it |
| +itself. The relay accepts a rotated-out master until its retirement |
| +time, at least 28 hours after rotation (§5), so a credential minted from |
| +the old key keeps working for its whole life. **(codex R1-10, R1-14)** |
| |
| **Test vector.** Master is `SHA-256("carlos-relay-test-master")`, |
| -account `abc`, app `meet`, incarnation `0123456789abcdef`, expiry |
| -`1900000000`: |
| +account `abc`, app `meet`, incarnation `0123456789abcdef`, kid `k1`, |
| +expiry `1900000000`: |
| |
| master 841d374df509340d6ab18a139a27ed0d9c5d61d0bdf306e4124ff913a7a92fb5 (hex) |
| key uMhwW4P/9O5dQUIs91t2CCHGhwJbeiM0xQMQ2G0CJLU= |
| - username 1900000000:abc:meet:0123456789abcdef |
| - credential khBx1BoommS1C69pSAgv0r0TM2H9IKDDE0M9OVr7VXQ= |
| + username 1900000000:abc:meet:0123456789abcdef:k1 |
| + credential jRU479Y3GO45nrRHqxBf3BMeBHE6Fip1cwAccNv28MQ= |
| |
| The key derivation is in §4; an app never needs it, only the key it is |
| -given. The vector is checked into `internal/relay/vector_test.go` and |
| -Meet's minting test uses the same numbers. |
| +given. The vector will be checked into `internal/relay/vector_test.go` |
| +by the plan, and Meet's minting test uses the same numbers. |
| +**(codex R1-20)** |
| |
| **Logging.** The app logs which region it used and nothing else about |
| the credential. The relay logs the account, app and outcome of an |
| -authentication, never the credential or the derived key. |
| +authentication, never the credential or a key. |
| |
| ## 3. The relay unit |
| |
| -`carlos-relay.service` is a §3 media service in the roadmap's terms: a |
| +`carlos-relay.service` is a media service in the roadmap's §3 terms: a |
| network-facing platform unit like the edge, not a codec helper. |
| |
| - **Process.** `carlos relay serve`, from the platform binary, so it |
| - rides the same release, promote and adopt path as the edge. A new |
| - file in `internal/host/units/`, added to `enabledUnits` in |
| - `bootstrap.go` under a guard: the unit is written on every box but |
| - armed only where `host.env` carries `CARLOS_RELAY_MASTERS`, so a box |
| - without a master never listens. Bootstrap is idempotent and converges |
| - an existing box the way it does for every unit. |
| -- **Hardening.** Copied from `carlos-edge.service` and then cut down: no |
| - `CAP_NET_BIND_SERVICE` (3478 and 5349 are unprivileged), `User=carlos`, |
| + rides the same release and promote path as the edge. A new file in |
| + `internal/host/units/`, embedded and written by bootstrap like every |
| + unit, with `ConditionPathExists=/etc/carlos/relay-masters` so a box |
| + without a master file never listens. Added to `enabledUnits` and to |
| + the list of units the systemctl plan **starts**, not only enables, |
| + and to the adopt restart path beside `restartEdge` |
| + (`internal/host/systemapply.go:568`), so a platform update restarts |
| + the relay onto the new binary and a fresh box runs it without a |
| + reboot. The staging bootstrap plan prints the same steps without |
| + running them, as it does for the edge. **(codex R1-16)** |
| +- **Hardening.** From `carlos-edge.service` and then cut down: no |
| + `CAP_NET_BIND_SERVICE` (3478 is unprivileged), `User=carlos`, |
| `ProtectSystem=strict`, `ReadWritePaths=/data/relay /run/carlos`, |
| - `RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX`, |
| - `MemoryHigh=20%`, `MemoryMax=25%`, `TasksMax=512`, `Restart=always`. |
| - It shares the box with the edge and must lose first: `Nice=5` and |
| - `CPUWeight=50` against the edge's default. |
| -- **Listeners.** UDP 3478 and TCP 3478 on the box's primary address; TLS |
| - 5349 when a certificate for the relay hostname is present. The relay |
| - allocation range is UDP 49152 to 65535, configured as |
| - `RelayAddressGeneratorPortRange` with `RelayAddress` set to the box's |
| - Elastic IP, which the process learns the way the edge does, from |
| - IMDS (`internal/host/agent.go:1761`), with `CARLOS_RELAY_PUBLIC_IP` as |
| - the override for a box without IMDS. |
| -- **Certificate for 5349.** The relay hostname is a platform host the |
| - edge already knows how to mint a certificate for; the edge adds |
| - `relay.<aws region>.<platform domain>` to its allowed set on boxes |
| - with a master, and the relay reads the certificate from the edge's |
| - autocert cache read-only, reloading when the file changes. If no |
| - certificate is present yet, the TLS listener is not started and the |
| - relay logs once; `turns:` is still advertised, because the browser |
| - falls through to the other URLs. |
| + `RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX`. Memory is an |
| + absolute budget, not a share: `MemoryMax=256M`, `MemoryHigh=192M`, |
| + and `OOMScoreAdjust=800` so that under host memory pressure the relay |
| + is the first process the kernel kills, ahead of the edge and tenants. |
| + `Nice=5` and `CPUWeight=50` for CPU. The plan's flagship gate loads |
| + edge, a tenant and the relay together on a `t4g.small` and confirms |
| + the relay dies first. **(codex R1-19)** |
| +- **Listeners.** UDP 3478 and TCP 3478 on the box's primary address. |
| + No TLS listener in version one (§8). The relay allocation range is |
| + UDP 49152 to 65535, bound through the relay's own |
| + `RelayAddressGenerator` (below) with the box's Elastic IP as the |
| + advertised relay address, learned as the edge learns it, from IMDS |
| + (`readPublicIP`, `internal/host/agent.go:1808`), with |
| + `CARLOS_RELAY_PUBLIC_IP` as the override for a box without IMDS. |
| + **(codex R1-20)** |
| - **STUN.** Binding requests are answered on every listener without |
| authentication, which is pion's default and the whole point of §0. |
| -- **Authentication.** A custom `AuthHandler` (§4). pion calls it on every |
| - authenticated request, which is where expiry is enforced. |
| -- **Peer policy.** A `PermissionHandler` that refuses `CreatePermission` |
| - and `ChannelBind` to loopback, link-local, RFC 1918 and unique-local |
| - ranges, the box's own VPC CIDR (from `CARLOS_RELAY_DENY_CIDRS`, set by |
| - the operator from tofu output), and the relay's own addresses. |
| - Refusals are counted per app. |
| -- **Limits.** Per app, per box, in the relay process: at most 64 live |
| - allocations, at most 30 new allocations a minute, allocation lifetime |
| - capped at ten minutes so a refresh is required (pion's default), and |
| - a byte budget of 20 GB per hour after which new allocations for that |
| - app are refused with 486 (allocation quota reached) until the hour |
| - rolls. Per box: at most 2048 allocations. These are operational |
| - limits, tuned by env with those defaults; commercial quotas are out |
| - of scope. |
| -- **Metering.** The relay wraps its UDP and TCP connections to count |
| - bytes in and bytes out per allocation, attributes them to the |
| - allocation's account and app, and once a minute appends a sample to |
| - the box's metrics chunks alongside the per-unit samples the metrics |
| - loop already flushes (`internal/host/metrics_loop.go`). The sample |
| - carries `relay_in`, `relay_out` and `relay_allocs` per app. Reading |
| - those samples into an app page is stage 5 work and not in this spec; |
| - the numbers exist from the first byte relayed. |
| -- **Authority record.** The relay reads `control/relay.json` (§5) from |
| - the deployment bucket on start and whenever `control/serial` changes, |
| - through the same fast-converge poll the host agent already runs |
| - (`internal/host/agent.go:1304`). If the record it holds is older than |
| - fifteen minutes it refuses new allocations and ends the ones it is |
| - carrying, per the roadmap's serving lease. A relay with no record at |
| - all serves STUN only. |
| +- **Our own address generator, for termination and metering.** pion has |
| + no public call to end an allocation; its allocation manager is |
| + internal and `Server` exposes only `AllocationCount` and `Close`. What |
| + it does expose is the `RelayAddressGenerator` that creates each |
| + allocation's relay socket, and its read loop deletes the allocation |
| + when that socket returns an error |
| + (`internal/allocation/allocation.go:301`, v4.1.4). So the relay |
| + implements the generator itself: every relay socket it hands pion is |
| + wrapped, recorded against the client's five-tuple, and can be closed |
| + by the relay at any time, which ends that allocation inside pion |
| + within one read. The wrapper also counts bytes in and out. That one |
| + mechanism is how every deadline below is enforced and how metering is |
| + attributed. **(codex R1-4, R1-17)** |
| +- **Owner binding.** pion locates a refresh, permission or channel bind |
| + by the client's five-tuple and does not compare the authenticating |
| + username to the one that created the allocation |
| + (`internal/server/turn.go`, v4.1.4). The relay does: on |
| + `OnAllocationCreated` it records five-tuple to identity, and its |
| + `AuthHandler`, which receives the source address, refuses a username |
| + whose identity differs from the one bound to a live allocation from |
| + that source. The binding is cleared on `OnAllocationDeleted`. |
| + **(codex R1-3)** |
| +- **Termination.** An allocation is ended, by closing its relay socket, |
| + at the earliest of: its credential's expiry (a timer set at creation |
| + from the username's expiry field); the app being disabled, its |
| + incarnation changing, or this region leaving its list (on the next |
| + authority read); the relay's authority lease expiring (below); the |
| + app crossing its byte budget (below). Ending one allocation ends its |
| + permissions and channels with it. pion's own maximum requested |
| + lifetime is one hour; the relay does not rely on it. **(codex R1-4)** |
| +- **Authentication.** A custom `AuthHandler` (§4), called by pion on |
| + every authenticated request, which is where the username's expiry and |
| + the authority record are checked; pion verifies the message integrity |
| + against the key the handler returns. |
| +- **Peer policy.** A `PermissionHandler` refuses `CreatePermission` and |
| + `ChannelBind` to loopback, link-local (including the IMDS address), |
| + RFC 1918 and unique-local ranges, the box's own private address and |
| + VPC CIDR (from `CARLOS_RELAY_DENY_CIDRS`, set by the operator from |
| + tofu output), and any other relay's private address. **The box's own |
| + Elastic IP is allowed**: two clients that both fall back to this relay |
| + hold relay candidates on that address at different ports and must |
| + reach each other, and a packet from the box to its own EIP hairpins |
| + through the internet gateway under the same security group as any |
| + outside sender, so it can reach nothing the public cannot. Refusals |
| + are counted per app. **(codex R1-8)** |
| +- **Limits**, all admitted under one mutex across every listener so |
| + counts cannot race, with counters reset on process start: |
| + - per app per box: 64 live allocations, 30 new allocations a minute; |
| + - per account per box: 256 live allocations, whatever the app count; |
| + - per box: 2048 live allocations; |
| + - per app per box: a byte budget of 20 GB per rolling hour; crossing |
| + it refuses new allocations with 486 and **ends the app's live |
| + allocations**, because a budget that only stops new ones does not |
| + bound anything. Failed allocations count against the per-minute |
| + rate. |
| + These are operational limits with those defaults, tunable by env; |
| + commercial quotas are out of scope. **(codex R1-9)** |
| +- **Metering.** From the wrapped sockets, per allocation, attributed to |
| + the identity bound at creation: bytes in from the client, bytes out |
| + to peers, and allocation count. Once a minute the relay writes its |
| + own chunk, `metrics/relay/<box>/<hour>.json`, a new additive key |
| + with one writer, rather than appending to the host sampler's objects, |
| + which that sampler rewrites from its own buffer |
| + (`internal/host/metrics_loop.go`, `writeHourChunk`). The schema is |
| + `{apps: {"<account>/<app>": {in, out, allocs}}, written_at}`, and the |
| + raw readers and rollups learn the new key in the plan. An |
| + inter-region meter is not built (§8). **(codex R1-17, R1-18)** |
| +- **Authority record and lease.** The relay reads `control/relay.json` |
| + (§5) from the deployment bucket **every sixty seconds**, and |
| + additionally whenever `control/serial` changes as seen by its own |
| + poll of that object (the host agent's `fastConvergeLoop`, |
| + `internal/host/agent.go:1437`, is a separate process; the relay runs |
| + the same shape of poll itself). The lease clock is the time of the |
| + last **successful** read; a failed read renews nothing. If that time |
| + is more than fifteen minutes ago the relay refuses new allocations |
| + and ends the ones it carries, per the roadmap's serving lease, and |
| + resumes on the next successful read. A relay with no record yet |
| + serves STUN only. Each read also writes a heartbeat, |
| + `control/relay-ack/<box>.json`, carrying the record's `written_at`, |
| + the read time, and which listeners are up, so status (§5) can tell a |
| + live relay from a dead one. **(codex R1-5, R1-14, R1-20)** |
| |
| ## 4. Keys and verification |
| |
| **Region master.** Thirty-two random bytes per region, generated by the |
| -operator with `carlos relay keygen` and staged as |
| -`CARLOS_RELAY_MASTERS="<kid>:<base64>[,<kid>:<base64>]"` in that |
| -region's `host.env`, the way `CARLOS_SECRETS_KEY` is staged today. The |
| -console holds every region's masters as sealed console secrets, |
| -`CARLOS_RELAY_MASTERS_<REGION>`, so it can derive app keys at enable. |
| -Two copies, both staged by hand, both rotated by hand; §5 gives the |
| -procedure. There is no bucket copy of a master, sealed or otherwise, |
| -because sealed config is decryptable on every host (roadmap §1). |
| +operator with `carlos relay keygen`, which prints a `kid` and the key. |
| +On each of the region's boxes they live in `/etc/carlos/relay-masters`, |
| +root-owned, mode 0640, group `carlos`, one per line: |
| + |
| + <kid> <base64 key> <retire_at RFC 3339 or "-"> |
| + |
| +The relay reads the file at start and re-reads it when it changes, so |
| +a rotation never restarts the process. A master past its `retire_at` is |
| +ignored. **(codex R1-10)** |
| + |
| +**Where the console's copy lives, and what that exposes.** The console |
| +must hold every region's masters to derive app keys at enable. It holds |
| +them as console secrets, `CARLOS_RELAY_MASTERS_<REGION>` (region name |
| +uppercased with hyphens as underscores, since variable names refuse |
| +hyphens, `internal/appconfig/appconfig.go:103`), and console secrets |
| +are sealed config like any app's. Sealed config is decryptable on every |
| +host (roadmap §1). So the honest statement is: **a compromised host can |
| +read every region's master**, not only the derived keys the roadmap |
| +already conceded, and with a master it can mint credentials for any |
| +app in that region, present and future, until the operator rotates. |
| +Per-region masters still mean a relay box itself holds only its own |
| +region's, and rotation of one region recovers that region. A design |
| +where only a region's boxes ever hold its master (the region's relay |
| +deriving app keys and writing them into config itself) is possible and |
| +is not version one, because it makes the relay a writer of app config |
| +with all the CAS and ordering that implies. **(codex R1-7; §9)** |
| |
| **Derived key.** For an app, region and incarnation: |
| |
| @@ -193,41 +280,64 @@ because sealed config is decryptable on every host (roadmap §1). |
| info = account || 0x00 || app || 0x00 || incarnation, |
| length = 32 ) |
| |
| -**Verification**, in the relay's `AuthHandler(username, realm, src)`: |
| +**Verification**, in the relay's `AuthHandler(username, realm, src)`, |
| +which pion calls once per authenticated request and which returns one |
| +key, so the master is chosen from the username, not by trial: |
| +**(codex R1-2)** |
| |
| -1. Split the username on `:` into exactly four fields; anything else is |
| +1. Split the username on `:` into exactly five fields; anything else is |
| refused. |
| -2. Parse `expiry` as a decimal Unix time; refuse if it is in the past or |
| - more than four hours ahead. |
| -3. Look up `account/app` in the authority record; refuse if absent, if |
| - the app's incarnation differs, or if this relay's region is not in |
| - the app's allowed list. |
| -4. For the current master, then the previous one if within its |
| - twenty-four-hour grace: derive the key, compute the expected |
| - credential, and return `turn.GenerateAuthKey(username, realm, |
| - credential)`. pion verifies `MESSAGE-INTEGRITY` against that key, so |
| - a wrong credential fails inside pion; the handler itself never sees |
| - the credential the client sent. |
| -5. Refuse with the reason counted per app: bad shape, expired, unknown |
| - app, wrong incarnation, wrong region. |
| - |
| -The MD5 inside `GenerateAuthKey` is the long-term credential mechanism |
| -RFC 8489 mandates and is not ours to choose; the HMAC-SHA256 above is. |
| +2. Parse `expiry`; refuse if in the past or more than four hours ahead. |
| +3. If `src` has a live allocation bound to a different identity (§3, |
| + owner binding), refuse. |
| +4. Look up `account/app` in the authority record; refuse if absent, if |
| + the incarnation differs, or if this relay's region is not in the |
| + app's list. |
| +5. Look up `kid` in the masters file; refuse if absent or retired. |
| +6. Derive the key, compute the expected credential, and return |
| + `turn.GenerateAuthKey(username, realm, credential)`. pion verifies |
| + `MESSAGE-INTEGRITY` against that key, so a wrong credential fails |
| + inside pion; the handler never sees what the client sent. |
| +7. Count the refusal reason per app: bad shape, expired, bound |
| + elsewhere, unknown app, wrong incarnation, wrong region, unknown |
| + kid. |
| + |
| +The MD5 inside `GenerateAuthKey` is the long-term credential key of RFC |
| +8489's compatibility mechanism, which pion implements; RFC 8489 also |
| +defines a SHA-256 variant that pion does not. The HMAC-SHA256 above is |
| +ours. **(codex R1-20)** |
| |
| ## 5. Enable, disable, rotate |
| |
| **Records.** Two, both additive: |
| |
| -- `apps/<acct>/<app>/relay.json`, CAS-guarded like `store.json`, holding |
| - `{enabled, incarnation, regions, enabled_at, disabled_at}`. The |
| - incarnation is sixteen hex characters drawn at random at each enable, |
| - never a counter, so reuse across delete and name reuse is not a |
| - question. |
| -- `control/relay.json`, the authority record the relays read: `{apps: |
| - [{account, app, incarnation, regions}], written_at}`, rewritten by |
| - the console from the per-app records on every enable and disable, |
| - followed by a `control/serial` bump so every box notices within |
| - seconds. |
| +- `apps/<acct>/<app>/relay.json`, CAS-guarded like `store.json`: |
| + `{enabled, incarnation, regions, config_generation, enabled_at, |
| + disabled_at}`. The incarnation is sixteen hex characters drawn at |
| + random at first enable and at each re-enable after a disable, never a |
| + counter, so reuse across delete and name reuse is not a question. |
| +- `control/relay.json`, the authority record: `{apps: [{account, app, |
| + incarnation, regions}], written_at}`. It is **always rebuilt from a |
| + listing of every per-app record**, never patched from a snapshot, and |
| + written with a CAS on its own etag; a conflict re-lists and retries. |
| + A reconcile that does exactly that rebuild runs at console start and |
| + hourly, so an operation that died between the per-app write and the |
| + aggregate write is repaired within the hour and the aggregate can |
| + never carry an app whose record says disabled. Followed by a |
| + `control/serial` bump so relays notice within seconds; the bump is a |
| + latency hint, not the lease. **(codex R1-5, R1-6)** |
| + |
| +**State transitions** for `relay.json`: **(codex R1-13)** |
| + |
| +| From | Operation | Incarnation | Keys re-derived | Allocations | |
| +|---|---|---|---|---| |
| +| none | enable | new | yes | none | |
| +| enabled | enable (repeat, same regions) | kept | no, no config write | untouched | |
| +| enabled | enable `--regions` (change) | kept | only for added regions; removed regions' entries dropped | ended in removed regions on next authority read | |
| +| enabled | disable | kept in record, app removed from aggregate | entries removed | ended everywhere on next authority read | |
| +| disabled | enable | **new** | yes, all regions | none | |
| +| any | app deleted | record removed with the app prefix; reconcile drops it from the aggregate | | ended on next authority read | |
| +| any | rotate `--region r` | kept | region `r` only, new kid | untouched; old kid accepted until retirement | |
| |
| **`carlos relay enable [--regions r1,r2]`** (member, API-mode only, on |
| the store model in `store.go` and `internal/console/apistore.go`): |
| @@ -235,95 +345,135 @@ the store model in `store.go` and `internal/console/apistore.go`): |
| 1. `apiAccountApp` gate; unknown and foreign are one byte-identical 404. |
| 2. Refuse if the deployment has no relay regions (`CARLOS_RELAY_REGIONS` |
| in console config is empty) with a message naming the operator. |
| -3. Regions default to every region in `CARLOS_RELAY_REGIONS` ordered |
| - with the app's placement region first; `--regions` narrows. |
| -4. Write `relay.json` with a fresh incarnation and the region list. |
| -5. Derive one key per region from that region's current master and |
| - write `CARLOS_RELAY` (as `kind:"secret"`, since it carries keys) and |
| - `CARLOS_RELAY_INCARNATION` (as `kind:"env"`) through |
| - `appconfig.Apply`, the `writeEnv` shape in `internal/objstore/walk.go`. |
| - That bumps the app's config generation; the box re-renders and |
| - restarts the instance within about a minute. |
| -6. Rewrite `control/relay.json`, bump `control/serial`. |
| -7. Answer with the record; the CLI prints regions, incarnation and |
| - "config delivered, instance restarting". |
| - |
| -**`carlos relay status`** prints the record and, per region, whether |
| -that region's relay has acknowledged the current `control/relay.json` |
| -(each relay writes `control/relay-ack/<box>.json` with the |
| -`written_at` it holds, every minute), so a member can see "eu-west-1: |
| -current, ap-southeast-2: 3 minutes behind" without a box command. |
| - |
| -**`carlos relay disable`** clears the two env entries, marks the record |
| -disabled, rewrites `control/relay.json` without the app, bumps the |
| -serial. Every relay ends the app's allocations on its next record read. |
| - |
| -**Rotation** (`carlos relay rotate --region r`, operator): |
| - |
| -1. Operator runs `carlos relay keygen`, adds the new `kid:key` in front |
| - of the old one in that region's `host.env`, restarts |
| - `carlos-relay.service`; the relay now accepts both, new first. |
| -2. Operator updates the console secret for that region the same way. |
| -3. `carlos relay rotate` re-derives and re-delivers keys for every app |
| - enabled in that region; each instance restarts within a minute. |
| -4. After twenty-four hours the operator removes the old entry from |
| - `host.env` and the console secret. Any credential minted from the old |
| - key has expired by then. |
| - |
| -**App deletion** removes `relay.json` with the rest of the app prefix |
| -and the console's deletion walk drops the app from `control/relay.json`. |
| +3. Regions default to every region in `CARLOS_RELAY_REGIONS`, ordered |
| + with the app's placement region first when the app has one, then |
| + the console's order; `--regions` narrows. The order is fixed at |
| + enable for the whole app; an app placed in two regions gets one |
| + order, and its browsers try every listed relay anyway. **(codex R1-12)** |
| +4. Apply the transition above to `relay.json` (CAS). |
| +5. For each region needing keys, derive from that region's current |
| + master and its kid, and write `CARLOS_RELAY` as one `kind:"secret"` |
| + entry through `appconfig.Apply` (the `writeEnv` shape in |
| + `internal/objstore/walk.go:554`). Record the resulting config |
| + generation in `relay.json`. That bumps the app's config generation; |
| + the box re-renders within about a minute. |
| +6. Rebuild `control/relay.json`, bump `control/serial`. |
| +7. Answer with the record and the config generation written. The CLI |
| + prints regions, incarnation, and "config generation N written; the |
| + instance restarts on delivery", never "delivered". **(codex R1-14)** |
| + |
| +**`carlos relay status`** reports, separately: desired state from |
| +`relay.json`; the config generation written; delivery, as "adopted" when |
| +the instances API shows the instance restarted after that generation, |
| +"written, awaiting delivery" otherwise, and "written, not restarted by |
| +the platform" for a TCP-address route; and per region, from |
| +`control/relay-ack/<box>.json`, the heartbeat age, whether the record it |
| +holds is the current one, whether its lease is live (heartbeat under |
| +fifteen minutes) and which listeners are up. A heartbeat older than |
| +five minutes is shown as "stale", so a dead relay cannot look current. |
| +**(codex R1-14)** |
| + |
| +**`carlos relay disable`** applies the transition, removes the |
| +`CARLOS_RELAY` entry through `appconfig.Apply`, rebuilds the aggregate, |
| +bumps the serial. |
| + |
| +**Rotation** (`carlos relay rotate --region r`, operator): **(codex R1-10)** |
| + |
| +1. Operator runs `carlos relay keygen`, which prints a new `kid` and key. |
| +2. Operator adds the line to `/etc/carlos/relay-masters` on each of the |
| + region's boxes with `retire_at` `-`, and sets `retire_at` on the old |
| + line to at least 28 hours ahead (24 hours for config to reach every |
| + instance plus a four-hour credential life). The relay re-reads the |
| + file; no restart, no allocation touched. |
| +3. Operator updates the console secret for the region to the new |
| + `kid:key`. |
| +4. `carlos relay rotate --region r` re-derives and re-delivers keys for |
| + every app enabled in that region, recording each app's new config |
| + generation; `carlos relay status` for each shows when it is adopted. |
| +5. After every enabled app shows adopted, or after the retirement time |
| + passes, the old line's `retire_at` does its work; the operator |
| + removes the line at leisure. An app still minting from the old key |
| + past retirement gets refused with "unknown kid" and re-reads its |
| + config on restart. |
| + |
| +**App deletion** removes `relay.json` with the rest of the app prefix, |
| +and the reconcile drops the app from the aggregate within the hour, or |
| +at once if the deletion walk calls the rebuild, which the plan has it do. |
| |
| ## 6. Infrastructure |
| |
| - `infra/modules/region-host/main.tf`: a `relay` variable, default off, |
| - that adds ingress for UDP 3478, TCP 3478, TCP 5349 and UDP 49152 to |
| - 65535 from `0.0.0.0/0`, and outputs the VPC CIDR for |
| - `CARLOS_RELAY_DENY_CIDRS`. Flagship turns it on per region. |
| + that adds ingress for UDP 3478, TCP 3478 and UDP 49152 to 65535 from |
| + `0.0.0.0/0`, and outputs the VPC CIDR for `CARLOS_RELAY_DENY_CIDRS`. |
| + Flagship turns it on per region. |
| - DNS: an A record `relay.<aws region>.<platform domain>` to the region's |
| - Elastic IP, next to the existing EIP resource. One line per region. |
| -- `host.env` on each enabled region box gains `CARLOS_RELAY_MASTERS` |
| - and `CARLOS_RELAY_DENY_CIDRS`. |
| + Elastic IP, next to the existing `aws_eip.host` resource. |
| +- Each armed box gains `/etc/carlos/relay-masters` (staged by the |
| + operator, like `host.env`) and `CARLOS_RELAY_DENY_CIDRS` and |
| + optionally `CARLOS_RELAY_PUBLIC_IP` in `host.env`. |
| - Console config gains `CARLOS_RELAY_REGIONS` and one |
| `CARLOS_RELAY_MASTERS_<REGION>` secret per region. |
| - Egress cost: every relayed byte leaves the box once. The |
| unit-economics spec gets a line for relay egress per region, and the |
| - meters in §3 are what fill it. |
| + `out` meter in §3 is what fills it. |
| |
| ## 7. Testing |
| |
| -- `internal/relay`: the key derivation and credential functions against |
| - the §2 test vector; the `AuthHandler` against a fixture authority |
| - record covering every refusal reason; the permission handler against |
| - each denied range; the limits against a fake clock; the meter against |
| - a wrapped in-memory conn. |
| -- An end-to-end test that starts the relay on loopback with a fixture |
| - master, mints a credential with the app-side code, and drives a pion |
| - TURN client through allocate, refresh past expiry, and a permission |
| - to a denied peer. |
| -- `internal/host`: the unit is embedded and written by bootstrap |
| - (`TestUnitsEmbedComplete`, `TestBootstrapWritesUnitsToStagingDir`), |
| - and armed only when the staging `host.env` carries a master. |
| -- `internal/console`: the three doors through `rawReq` and a |
| - `relayURL` fixture, asserting the byte-identical 404, the no-regions |
| - refusal, the config entries written, and `control/relay.json` after |
| - enable and disable. |
| +- `internal/relay`: derivation and credential against the §2 vector; |
| + the `AuthHandler` against a fixture authority record and masters file |
| + covering every refusal reason in §4, including a valid credential for |
| + app B presented from a source bound to app A; the permission handler |
| + against each denied range and the allowed EIP; limits against a fake |
| + clock, including an account spreading allocations across apps and an |
| + allocation ended by the byte budget; the meter against wrapped |
| + in-memory sockets. |
| +- End to end, on loopback with a fixture master: a pion TURN client |
| + allocates, refreshes, and keeps relaying until the credential's |
| + expiry, at which point traffic stops with no further client request; |
| + the same after a disable written to the fixture authority record; |
| + two clients on one relay reaching each other's relay candidates; a |
| + rotation, by rewriting the masters file, during which an allocation |
| + on the old kid carries traffic uninterrupted; a lease lapse ending |
| + allocations and a successful read restoring service. |
| +- `internal/host`: the unit is embedded, written and started by the |
| + bootstrap plan when the staging root carries a masters file, and not |
| + started when it does not; the adopt restart path names the relay. |
| +- `internal/console`: the three doors through `rawReq` and a `relayURL` |
| + fixture: the byte-identical 404, the no-regions refusal, each row of |
| + the transition table, the config entry written and removed, the |
| + aggregate after concurrent enable and disable of two apps, and the |
| + reconcile repairing an aggregate that disagrees with the records. |
| - `docs/site/cli.md` gains `### carlos relay` with a fenced example, or |
| the docsite gate fails. |
| -- Flagship gate, from the roadmap: a Meet call between two symmetric-NAT |
| - networks connecting through the Ireland relay; a credential minted for |
| - one app refused for another; a credential from before a disable |
| - refused after re-enable and its allocation ended; a relay with a |
| - record older than fifteen minutes refusing new allocations and ending |
| - carried ones; a permission to a private address refused; the meters |
| - moving; a 443-only network failing as documented. |
| +- Flagship gate, from the roadmap with the corrections above: a Meet |
| + call between two symmetric-NAT networks connecting through the |
| + Ireland relay; both browsers forced to relay and connecting through |
| + it; a credential minted for one app refused for another, including on |
| + an existing allocation's tuple; a credential from before a disable |
| + refused after re-enable and its allocation ended; traffic stopping at |
| + a credential's expiry with the browser doing nothing; a relay with a |
| + lease older than fifteen minutes refusing new allocations and ending |
| + carried ones; a permission to a private address refused; the relay |
| + meter chunk showing the call's bytes; a rotation during a call with |
| + the call unaffected; the relay dying first under memory pressure. |
| |
| ## 8. Not built |
| |
| -- TURN over TLS on 443: needs a second address per box. |
| -- An issuance API: apps mint offline from delivered keys; the alternative |
| - is named in the roadmap if the trade ever changes. |
| -- Client-aware relay selection: the list is ordered by the instance's |
| - region; ICE picks. |
| +- **TURN over TLS (5349 or 443).** Certificate issuance for a non-HTTP |
| + listener needs the relay hostname registered with the deployment's |
| + ACME issuer, which is elected and may be another box |
| + (`internal/edge/certsync.go`), and its warmer excludes such names |
| + (`internal/edge/certwarm.go`); a listener that cannot start until a |
| + certificate exists cannot trigger issuance by being visited. That is |
| + its own small design. UDP and TCP on 3478 cover Meet's floor; the |
| + networks that block 3478 mostly block 5349 too, and 443 needs a second |
| + address per box. **(codex R1-15; §9)** |
| +- **An inter-region byte meter.** Classifying a peer address by AWS |
| + region needs the published IP ranges on the box and a rule for |
| + non-AWS peers; version one meters bytes in and out per app, which is |
| + what the bill is made of. **(codex R1-18; §9)** |
| +- An issuance API: apps mint offline from delivered keys. |
| +- Client-aware relay selection: the list is ordered at enable; ICE picks. |
| - Commercial quotas and an app-page usage view: the meters exist, the |
| page is stage 5. |
| - An SFU: stage 4b, after this ships. |
| @@ -331,18 +481,24 @@ and the console's deletion walk drops the app from `control/relay.json`. |
| |
| ## 9. Decisions for Paul |
| |
| -1. **Credential life four hours, not one.** The roadmap's recommendation |
| - said one hour. pion enforces expiry on every refresh, so a one-hour |
| - credential drops the relay from any call longer than an hour unless |
| - the app re-mints and restarts ICE. Four hours makes that Meet's |
| - problem never. Recommended. |
| -2. **Enable is member-level, no operator grant.** The store model gates |
| - `grant` on operators because it spends the deployment's cloud |
| - account. Enabling a relay spends bandwidth, bounded by the §3 limits, |
| - and the operator's consent is the region list in console config. |
| - Recommended over a per-app operator step that Meet would wait on. |
| -3. **Limits' defaults** (64 allocations, 30 a minute, 20 GB an hour per |
| - app per box). Numbers to tune; the mechanism is the decision. |
| -4. **`turns:` advertised before its certificate exists.** Harmless, since |
| - browsers fall through, and it means no config re-render when the |
| - certificate lands. Recommended. |
| +1. **Credential life four hours, not one.** The roadmap's text said one |
| + hour. Expiry is now a hard deadline the relay enforces by ending the |
| + allocation, so one hour would end any call longer than that unless |
| + the app re-mints and restarts ICE. Recommended: four. |
| +2. **Enable is member-level, no operator grant.** The operator's consent |
| + is the region list in console config and the limits in §3. |
| + Recommended. |
| +3. **The console holds every region's master as sealed config**, with |
| + the exposure stated in §4: a compromised host can read them all. The |
| + alternative, relays deriving and writing app keys themselves, is |
| + named there and deferred. Recommended for version one, with the |
| + exposure written into the operator docs. |
| +4. **No TURN over TLS in version one.** Recommended, for the reasons in |
| + §8; it returns as its own small spec when a network that needs it is |
| + reported. |
| +5. **No inter-region meter in version one.** The roadmap named three |
| + meters; this ships two. Recommended, with the third filed for stage |
| + 5. |
| +6. **Limits' defaults** (64 and 30 a minute per app, 256 per account, |
| + 2048 per box, 20 GB an hour per app). Numbers to tune; the |
| + mechanisms are the decision. |