| 1 | # Stage 4a: a platform STUN and TURN relay |
| 2 | |
| 3 | **Date:** 2026-09-07, revision 6 |
| 4 | **Status:** rulings taken. Paul ruled on the six decisions in §9 on |
| 5 | 2026-09-07, each marked **(ruling)**; the roadmap's stage 4a text and |
| 6 | gate are amended so the two documents agree. Reviewed five times by |
| 7 | GPT-6 via Codex: revision 1's 20 findings are in |
| 8 | `CODEX-FINDINGS-RELAY.md`, revision 2's 12 in |
| 9 | `CODEX-FINDINGS-RELAY-R2.md`, revision 3's 8 in |
| 10 | `CODEX-FINDINGS-RELAY-R3.md`, revision 4's 3 in |
| 11 | `CODEX-FINDINGS-RELAY-R4.md`, revision 5's 1 in |
| 12 | `CODEX-FINDINGS-RELAY-R5.md`, unedited; every one is taken, marked |
| 13 | **(codex R1-N)** through **(codex R5-N)**. **Round 6 returned accept** |
| 14 | with no findings (`CODEX-FINDINGS-RELAY-R6.md`). Stage 4a of the media roadmap |
| 15 | (`2026-09-07-media-platform-roadmap-design.md`), built under rulings 8, |
| 16 | 10 and 11 there. |
| 17 | |
| 18 | **Why now.** Oficina's Meet is peer-to-peer WebRTC and cannot name a |
| 19 | third-party STUN server, because suite ruling 6 forbids egress that is a |
| 20 | property of the product. Without STUN, browsers gather only LAN |
| 21 | candidates and calls connect on the same network only. A platform STUN |
| 22 | listener is the smallest thing that makes Meet work across the |
| 23 | internet; TURN on the same listener is what makes the one call in ten |
| 24 | that cannot hole-punch connect anyway. Meet is blocked on nothing else |
| 25 | (Meet session, 2026-09-07), and it has reviewed the app-facing contract |
| 26 | in §2. |
| 27 | |
| 28 | ## 1. What is built |
| 29 | |
| 30 | One new platform unit, `carlos-relay.service`, on every region box the |
| 31 | operator arms, running `carlos relay serve` from the platform binary, |
| 32 | built on `pion/turn` v4.1.4, pure Go, which fetches and cross-compiles |
| 33 | clean for the CI shards (checked 2026-09-07). It answers STUN binding |
| 34 | requests to anyone and TURN allocations to holders of a credential an |
| 35 | enabled app minted. It learns which apps are enabled from a record the |
| 36 | console writes, binds every allocation to the identity that made it, |
| 37 | ends allocations when their credential, app, region, master or its own |
| 38 | authority lapses, meters what it forwards per app, and refuses peers on |
| 39 | private networks. |
| 40 | |
| 41 | Three console doors and a CLI verb, `carlos relay enable|status|disable`, |
| 42 | that turn the relay on for an app and deliver its keys as sealed config, |
| 43 | plus an operator verb `carlos relay rotate`. An operator-side master key |
| 44 | per region, staged in a file on the region's boxes and, as a console |
| 45 | secret, in the console. A config-adoption acknowledgement the host's |
| 46 | config sync writes, so status can say "adopted" truthfully. A |
| 47 | security-group change opening the relay's ports. A DNS name per region |
| 48 | relay. |
| 49 | |
| 50 | Not built, by ruling: TURN over TLS, an inter-region byte meter. Not |
| 51 | built, by scope: an SFU, recording, an issuance API, client-side relay |
| 52 | selection, commercial quotas. §8 lists them with reasons. |
| 53 | |
| 54 | ## 2. The app-facing contract |
| 55 | |
| 56 | This is what Meet codes against. It is fixed by this spec; the rest of |
| 57 | the design may change without touching it. |
| 58 | |
| 59 | **Environment.** One variable reaches the instance through the sealed |
| 60 | `config.json` path, the way store credentials do. Because the config |
| 61 | renderer refuses quotes, backslashes and newlines in any value |
| 62 | (`internal/appconfig/materialize.go:23`), the value is the **standard |
| 63 | base64 encoding, with padding, of a JSON document**; the app decodes |
| 64 | the base64, then parses the JSON. **(codex R1-1)** |
| 65 | |
| 66 | - `CARLOS_RELAY`: base64 of a JSON array, one element per region the |
| 67 | app may use, in the console's order (§5). Absent or empty means no |
| 68 | relay: the app shows its floor. |
| 69 | |
| 70 | [{"region":"eu-west-1", |
| 71 | "urls":["stun:relay.eu-west-1.oncarlos.com:3478", |
| 72 | "turn:relay.eu-west-1.oncarlos.com:3478?transport=udp", |
| 73 | "turn:relay.eu-west-1.oncarlos.com:3478?transport=tcp"], |
| 74 | "key":"<standard base64, with padding, of 32 raw bytes>", |
| 75 | "username_suffix":"abc:meet:0123456789abcdef:k1"}] |
| 76 | |
| 77 | `region` is the AWS region name, which is also the vocabulary of the |
| 78 | relay hostnames. `username_suffix` is `<account>:<app>:<incarnation>: |
| 79 | <kid>`, supplied by the platform so the app never has to know its own |
| 80 | canonical account sqid or app name. **(codex R1-11, R1-12)** There is |
| 81 | no separate incarnation variable; the suffix carries it. The app |
| 82 | builds its ICE server list from whatever `urls` are present and |
| 83 | assumes nothing about their number or schemes. |
| 84 | |
| 85 | **Minting, per call.** For the region it wants, the app computes: |
| 86 | |
| 87 | key = base64_decode( element.key ) // 32 raw bytes |
| 88 | username = "<expiry>:" + element.username_suffix // ASCII |
| 89 | credential = base64_std( HMAC-SHA256( key, username ) ) |
| 90 | |
| 91 | where `expiry` is a decimal Unix time in seconds, at most four hours |
| 92 | ahead **(ruling 1)**. The username has exactly five colon-separated |
| 93 | fields: `expiry` (`[0-9]+`), `account` (`[a-z0-9]+`), `app` |
| 94 | (`^[a-z][a-z0-9-]{0,31}$`), `incarnation` (`[0-9a-f]{16}`) and `kid` |
| 95 | (`[a-z0-9]{1,8}`). No field can contain a colon. The app hands the |
| 96 | browser an `RTCIceServer` list: `stun:` entries bare, `turn:` entries |
| 97 | carrying `username` and `credential`. STUN needs no credential. |
| 98 | |
| 99 | **Expiry.** The relay checks the expiry on every authenticated request |
| 100 | the browser's ICE agent sends (allocate, refresh, permission, channel |
| 101 | bind) **and** ends the allocation at the expiry regardless of what the |
| 102 | client sends (§3, "termination"), so a credential's deadline is a hard |
| 103 | deadline on relayed traffic. An app that needs a call to outlive its |
| 104 | credential re-mints and performs an ICE restart before the deadline. |
| 105 | Four hours is the maximum, not a promise about meeting lengths. |
| 106 | **(codex R1-4, R1-20)** |
| 107 | |
| 108 | **What changes the incarnation.** Only `carlos relay disable` followed by |
| 109 | `carlos relay enable`. A repeated enable, a region change, a deploy, a |
| 110 | restart and a master rotation all keep it. **(codex R1-13)** |
| 111 | |
| 112 | **What changes the key.** A master rotation in a region changes that |
| 113 | region's key and `kid`; a new incarnation changes every region's key. |
| 114 | Either re-renders the app's config within about a minute, as any config |
| 115 | change does today, and what happens next depends on the route's shape |
| 116 | (`internal/appconfig/sync.go:30`, `syncexec.go:1`): a unit-backed route |
| 117 | and a socket-backed exec route are restarted onto the new file; a |
| 118 | TCP-address exec route whose bundle opts into file delivery receives |
| 119 | the file and is **not** restarted, because its process belongs to an |
| 120 | external supervisor CARLOS cannot bounce, so the app must re-read it |
| 121 | and then tell the platform it has (`carlos relay adopted`, §5); a |
| 122 | TCP-address route without file delivery receives nothing, and **an app |
| 123 | with such a route cannot enable the relay** until it opts that route |
| 124 | into file delivery or moves it to a unit. `carlos relay status` names |
| 125 | which of these each route is. **(codex R1-10, R1-14, R2-6, R3-4, R5-1)** |
| 126 | |
| 127 | **A credential keeps working for its whole life** provided the master |
| 128 | that signed its key is not force-retired before its expiry. The |
| 129 | rotation procedure in §5 guarantees that when followed: the old master |
| 130 | is retired no earlier than four hours after the last instance stopped |
| 131 | minting from it, as observed by the console, and no earlier than 28 |
| 132 | hours after rotation regardless. **(codex R2-4)** |
| 133 | |
| 134 | **Test vector.** Master is `SHA-256("carlos-relay-test-master")`, |
| 135 | account `abc`, app `meet`, incarnation `0123456789abcdef`, kid `k1`, |
| 136 | expiry `1900000000`: |
| 137 | |
| 138 | master 841d374df509340d6ab18a139a27ed0d9c5d61d0bdf306e4124ff913a7a92fb5 (hex) |
| 139 | key uMhwW4P/9O5dQUIs91t2CCHGhwJbeiM0xQMQ2G0CJLU= |
| 140 | username 1900000000:abc:meet:0123456789abcdef:k1 |
| 141 | credential jRU479Y3GO45nrRHqxBf3BMeBHE6Fip1cwAccNv28MQ= |
| 142 | |
| 143 | The key derivation is in §4; an app never needs it, only the key it is |
| 144 | given. The vector will be checked into `internal/relay/vector_test.go` |
| 145 | by the plan, and Meet's minting test uses the same numbers. |
| 146 | |
| 147 | **Logging.** The app logs which region it used and nothing else about |
| 148 | the credential. The relay logs the account, app and outcome of an |
| 149 | authentication, never the credential or a key. |
| 150 | |
| 151 | ## 3. The relay unit |
| 152 | |
| 153 | `carlos-relay.service` is a media service in the roadmap's §3 terms: a |
| 154 | network-facing platform unit like the edge, not a codec helper. |
| 155 | |
| 156 | - **Process.** `carlos relay serve`, from the platform binary, so it |
| 157 | rides the same release and promote path as the edge. A new file in |
| 158 | `internal/host/units/`, embedded and written by bootstrap like every |
| 159 | unit, with `ConditionPathExists=/etc/carlos/relay-masters` so a box |
| 160 | without a master file never listens. Added to `enabledUnits` and to |
| 161 | the list of units the systemctl plan **starts**, not only enables, |
| 162 | and to the adopt restart path beside `restartEdge` |
| 163 | (`internal/host/systemapply.go:568`), so a platform update restarts |
| 164 | the relay onto the new binary and a fresh box runs it without a |
| 165 | reboot. The staging bootstrap plan prints the same steps without |
| 166 | running them, as it does for the edge. **(codex R1-16)** |
| 167 | - **Hardening and memory.** From `carlos-edge.service` and then cut |
| 168 | down: no `CAP_NET_BIND_SERVICE` (3478 is unprivileged), |
| 169 | `User=carlos`, `ProtectSystem=strict`, `ReadWritePaths=/data/relay |
| 170 | /run/carlos`, `RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX`. |
| 171 | Memory: `MemoryMax=256M`, `MemoryHigh=160M`, `OOMScoreAdjust=1000`. |
| 172 | What that guarantees, stated exactly: the relay can never take more |
| 173 | than 256 MiB of the box; above 160 MiB it refuses new allocations and |
| 174 | ends its oldest until it is under; and it is the kernel's preferred |
| 175 | victim among processes of similar footprint, since 1000 is the |
| 176 | largest adjustment there is. What it does not guarantee: that the |
| 177 | relay dies before a tenant under every host-wide pressure. The |
| 178 | kernel's score counts swap and page tables as well as RSS, tenants |
| 179 | may swap up to their memory cap (`internal/cgroup/cgroup.go:313`), |
| 180 | and a kill inside the edge's own 80% cgroup considers only the edge's |
| 181 | tree. So the relay does not wait for the kernel: it watches the |
| 182 | box's memory pressure (`/proc/pressure/memory`, the `some` ten-second |
| 183 | average) and sheds allocations, oldest first, whenever it exceeds |
| 184 | 20%, whatever caused the pressure, resuming admission when it falls |
| 185 | under 10%. The gate in §7 tests that shedding, not a kill order. |
| 186 | `Nice=5` and `CPUWeight=50` for CPU. **(codex R1-19, R2-8, R3-7)** |
| 187 | - **Listeners.** UDP 3478 and TCP 3478 on the box's primary address. |
| 188 | No TLS listener **(ruling 4)**. The relay allocation range is UDP |
| 189 | 49152 to 65535, bound through the relay's own address generator |
| 190 | (below) with the box's Elastic IP as the advertised relay address, |
| 191 | learned as the edge learns it, from IMDS (`readPublicIP`, |
| 192 | `internal/host/agent.go:1808`), with `CARLOS_RELAY_PUBLIC_IP` as the |
| 193 | override for a box without IMDS. |
| 194 | - **STUN.** Binding requests are answered on every listener without |
| 195 | authentication, which is pion's default and the reason this stage |
| 196 | exists. |
| 197 | - **The relay owns every socket pion uses.** pion has no public call to |
| 198 | end an allocation; its allocation manager is internal and `Server` |
| 199 | exposes only `AllocationCount` and `Close`. What it does expose is the |
| 200 | `RelayAddressGenerator` that creates each allocation's relay socket |
| 201 | (`server_config.go:20`, returning a `net.PacketConn`), the listener |
| 202 | sockets the relay passes in, and an `OnAllocationCreated` callback |
| 203 | that supplies the relay address before pion starts reading from it |
| 204 | (`internal/allocation/allocation_manager.go:140`). pion's read loop |
| 205 | deletes the allocation when the relay socket returns an error |
| 206 | (`internal/allocation/allocation.go:301`). So the relay implements |
| 207 | the generator and wraps both kinds of socket: every relay socket is |
| 208 | recorded against its allocation at `OnAllocationCreated` and can be |
| 209 | closed by the relay at any time, which ends that allocation inside |
| 210 | pion within one read; and both relay and listener sockets count what |
| 211 | they forward and apply the token buckets below. That one mechanism is |
| 212 | how every deadline is enforced, how metering is attributed, and how |
| 213 | bandwidth is bounded. **(codex R1-4, R1-17)** |
| 214 | - **Owner binding.** pion locates a refresh, permission or channel bind |
| 215 | by the client's five-tuple and does not compare the authenticating |
| 216 | username to the one that created the allocation |
| 217 | (`internal/server/turn.go:34`); it also tags every allocation's tuple |
| 218 | as UDP even for TCP clients, so the event's protocol string cannot |
| 219 | distinguish transports. The relay keys its binding on the **typed |
| 220 | source address**: the Go type of the `net.Addr` it receives |
| 221 | (`*net.UDPAddr` or `*net.TCPAddr`) plus its string, which pion passes |
| 222 | unchanged to `AuthHandler` and to the events. On |
| 223 | `OnAllocationCreated` it records that key to the identity fields |
| 224 | (account, app, incarnation); its `AuthHandler` refuses a username |
| 225 | whose identity differs from the one bound to a live allocation from |
| 226 | that key. Cleanup belongs to the wrapped relay socket, not to |
| 227 | `OnAllocationDeleted`, which carries no relay address |
| 228 | (`internal/allocation/event_handler.go:20`): each wrapper knows the |
| 229 | binding it belongs to, and on `Close`, whether pion closed it or the |
| 230 | relay did, it removes that binding only if the map still points at |
| 231 | itself, so a replacement allocation bound while the old one was |
| 232 | closing (pion removes from its manager before closing, |
| 233 | `allocation_manager.go:150`) is never erased. **(codex R1-3, R2-3, |
| 234 | R3-3)** |
| 235 | - **Termination.** An allocation is ended, by closing its relay socket, |
| 236 | at the earliest of: its credential's expiry (a timer set at creation |
| 237 | from the username's expiry field); the app being disabled, its |
| 238 | incarnation changing, or this region leaving its list (on the next |
| 239 | authority read); the master named by its `kid` reaching `retire_at` |
| 240 | (on the next masters-file read or timer); the relay's authority lease |
| 241 | expiring (below); the app or account crossing a byte budget (below); |
| 242 | the relay shedding under memory pressure. Ending one allocation ends |
| 243 | its permissions and channels with it. pion's own maximum requested |
| 244 | lifetime is one hour; the relay does not rely on it. **(codex R1-4, |
| 245 | R2-4)** |
| 246 | - **Authentication.** A custom `AuthHandler` (§4), called by pion on |
| 247 | every authenticated request, which is where the username's expiry and |
| 248 | the authority record are checked; pion verifies the message integrity |
| 249 | against the key the handler returns. |
| 250 | - **Peer policy.** A `PermissionHandler` refuses `CreatePermission` and |
| 251 | `ChannelBind` to loopback, link-local (including the IMDS address), |
| 252 | RFC 1918 and unique-local ranges, the box's own private address and |
| 253 | VPC CIDR (from `CARLOS_RELAY_DENY_CIDRS`, set by the operator from |
| 254 | tofu output), and any other relay's private address. The box's own |
| 255 | Elastic IP is allowed by the permission handler, because two clients |
| 256 | that both fall back to this relay hold relay candidates on that |
| 257 | address and must reach each other, **but only live relay allocation |
| 258 | ports are reachable there**: the permission handler sees only an IP, |
| 259 | so the wrapped relay socket enforces the port on `WriteTo`, dropping |
| 260 | any packet to the EIP whose destination port is not a currently |
| 261 | allocated relay port. Nothing else on the box is reachable through |
| 262 | the relay, whatever the security group says about source addresses. |
| 263 | Refusals and drops are counted per app. **(codex R1-8, R2-9)** |
| 264 | - **Limits (ruling 6)**, all admitted under one mutex across every |
| 265 | listener so counts cannot race, with counters reset on process start |
| 266 | and the rolling-hour byte windows rebuilt on start from every hourly |
| 267 | metrics object that intersects the window, normally the previous |
| 268 | hour's and the current one **(codex R3-6)**: |
| 269 | - per app per box: 64 live allocations, 30 new a minute, 20 GB a |
| 270 | rolling hour; |
| 271 | - per account per box: 256 live, 60 new a minute, 60 GB a rolling |
| 272 | hour, whatever the app count; |
| 273 | - per box: 2048 live, 500 new a minute; |
| 274 | - instantaneous: a token bucket of 8 Mbps per allocation and 200 Mbps |
| 275 | per box applied in the wrapped sockets on both forwarding legs, |
| 276 | dropping packets beyond the bucket, so a burst is bounded before it |
| 277 | reaches the edge's share of the NIC; |
| 278 | - crossing a byte budget refuses new allocations with 486 for the |
| 279 | app or account and **ends its live allocations**, because a budget |
| 280 | that only stops new ones bounds nothing. Failed allocations count |
| 281 | against the per-minute rates. |
| 282 | These are operational limits with those defaults, tunable by env; |
| 283 | commercial quotas are out of scope. **(codex R1-9, R2-7)** |
| 284 | - **Metering.** Two meters per app and one gauge, measured where pion |
| 285 | forwards and only for what it forwards: **(codex R1-17, R1-18, R2-5; |
| 286 | ruling 5)** |
| 287 | - `in`: payload bytes the relay accepted from the client and forwarded |
| 288 | to a peer, counted at the relay socket's `WriteTo`; |
| 289 | - `out`: payload bytes the relay forwarded from a peer to the client, |
| 290 | counted at the listener socket's `WriteTo` to the client's address, |
| 291 | attributed through the owner binding; |
| 292 | - `allocs`: live allocations at sample time. |
| 293 | Both meters are host egress and are what the bill is made of. |
| 294 | Peer bytes that arrive at a relay socket and are dropped for lack of |
| 295 | permission are counted separately as `rejected` and never against a |
| 296 | budget. Every sixty seconds the relay appends one sample |
| 297 | `{at, apps: {"<account>/<app>": {in, out, allocs, rejected}}}`, with |
| 298 | `in`, `out` and `rejected` as **deltas since the previous sample**, |
| 299 | to its own hourly object `metrics/relay/<box>/<hour>.json`, which it |
| 300 | alone writes: read, append, CAS-write, retry on conflict. On restart |
| 301 | it reads the current hour's object back, so earlier samples survive |
| 302 | and its rolling-hour budgets resume from them. The raw readers and |
| 303 | rollups learn the new key in the plan. An inter-region meter is not |
| 304 | built (§8). |
| 305 | - **Authority record and lease.** The relay reads `control/relay.json` |
| 306 | (§5) from the deployment bucket **every sixty seconds**, and |
| 307 | additionally whenever `control/serial` changes as seen by its own |
| 308 | poll of that object (the host agent's `fastConvergeLoop`, |
| 309 | `internal/host/agent.go:1437`, is a separate process; the relay runs |
| 310 | the same shape of poll itself). The lease clock is the time of the |
| 311 | last **successful** read; a failed read renews nothing. If that time |
| 312 | is more than fifteen minutes ago the relay refuses new allocations |
| 313 | and ends the ones it carries, per the roadmap's serving lease, and |
| 314 | resumes on the next successful read. A relay with no record yet |
| 315 | serves STUN only. Each read also writes a heartbeat, |
| 316 | `control/relay-ack/<box>.json`, carrying the record's `written_at`, |
| 317 | the read time, which listeners are up, the `kid`s currently loaded |
| 318 | from the masters file, and the last masters-file reload error if |
| 319 | any. **(codex R1-5, R1-14, R2-4)** |
| 320 | |
| 321 | ## 4. Keys and verification |
| 322 | |
| 323 | **Region master.** Thirty-two random bytes per region, generated by the |
| 324 | operator with `carlos relay keygen`, which prints a `kid` and the key. |
| 325 | On each of the region's boxes they live in `/etc/carlos/relay-masters`, |
| 326 | root-owned, mode 0640, group `carlos`, one per line: |
| 327 | |
| 328 | <kid> <base64 key> <retire_at RFC 3339 or "-"> |
| 329 | |
| 330 | The relay reads the file at start and re-reads it when it changes, so |
| 331 | a rotation never restarts the process. A master past its `retire_at` is |
| 332 | ignored for new requests and its live allocations are ended (§3). |
| 333 | **(codex R1-10, R2-4)** |
| 334 | |
| 335 | **Where the console's copy lives, and what that exposes (ruling 3).** |
| 336 | The console must hold every region's masters to derive app keys at |
| 337 | enable. It holds them as console secrets, `CARLOS_RELAY_MASTERS_<REGION>` |
| 338 | (region name uppercased with hyphens as underscores, since variable |
| 339 | names refuse hyphens, `internal/appconfig/appconfig.go:103`), one or |
| 340 | more `kid:key` pairs per region, of which the one named by the issuer |
| 341 | pointer `control/relay-issuer.json` is the one keys are derived from |
| 342 | (§5, rotation); and console secrets are sealed config like any app's. Sealed config is |
| 343 | decryptable on every host (roadmap §1). So the honest statement is: **a |
| 344 | compromised host can read every region's master**, not only the derived |
| 345 | keys the roadmap already conceded, and with a master it can mint |
| 346 | credentials for any app in that region, present and future, until the |
| 347 | operator rotates. Per-region masters still mean a relay box itself |
| 348 | holds only its own region's, and rotation of one region recovers that |
| 349 | region. The operator docs say this in those words. A design where only |
| 350 | a region's boxes ever hold its master, the region's relay deriving app |
| 351 | keys and writing them into config itself, is filed as the tightening |
| 352 | and is not version one, because it makes the relay a writer of app |
| 353 | config with all the CAS and ordering that implies. **(codex R1-7, |
| 354 | R2-10)** |
| 355 | |
| 356 | **Derived key.** For an app, region and incarnation: |
| 357 | |
| 358 | key = HKDF-SHA256( ikm = master, |
| 359 | salt = "carlos-relay-v1", |
| 360 | info = account || 0x00 || app || 0x00 || incarnation, |
| 361 | length = 32 ) |
| 362 | |
| 363 | **Verification**, in the relay's `AuthHandler(username, realm, src)`, |
| 364 | which pion calls once per authenticated request and which returns one |
| 365 | key, so the master is chosen from the username, not by trial: |
| 366 | **(codex R1-2)** |
| 367 | |
| 368 | 1. Split the username on `:` into exactly five fields; anything else is |
| 369 | refused. |
| 370 | 2. Parse `expiry`; refuse if in the past or more than four hours ahead. |
| 371 | 3. If the typed source address has a live allocation bound to a |
| 372 | different identity (§3, owner binding), refuse. |
| 373 | 4. Look up `account/app` in the authority record; refuse if absent, if |
| 374 | the incarnation differs, or if this relay's region is not in the |
| 375 | app's list. |
| 376 | 5. Look up `kid` in the masters file; refuse if absent or retired. |
| 377 | 6. Derive the key, compute the expected credential, and return |
| 378 | `turn.GenerateAuthKey(username, realm, credential)`. pion verifies |
| 379 | `MESSAGE-INTEGRITY` against that key, so a wrong credential fails |
| 380 | inside pion; the handler never sees what the client sent. |
| 381 | 7. Count the refusal reason per app: bad shape, expired, bound |
| 382 | elsewhere, unknown app, wrong incarnation, wrong region, unknown |
| 383 | kid. |
| 384 | |
| 385 | The MD5 inside `GenerateAuthKey` is the long-term credential key of RFC |
| 386 | 8489's compatibility mechanism, which pion implements; RFC 8489 also |
| 387 | defines a SHA-256 variant that pion does not. The HMAC-SHA256 above is |
| 388 | ours. **(codex R1-20)** |
| 389 | |
| 390 | ## 5. Enable, disable, rotate |
| 391 | |
| 392 | **Records.** Two, both additive. **(codex R1-6, R2-1, R2-2)** |
| 393 | |
| 394 | - `apps/<acct>/<app>/relay.json`, CAS-guarded like `store.json`, is the |
| 395 | **source of truth** for the app: `{enabled, incarnation, regions, |
| 396 | op, phase, issued_digest, issued_suffixes, cutovers, enabled_at, |
| 397 | disabled_at}`. `issued_digest` is the sha256 of the sealed envelope |
| 398 | that was delivered, which is what `Record.IssuedDigest` records and |
| 399 | what reconcile compares; an earlier draft of this spec called it |
| 400 | `config_generation` and no implementation ever carried that name. |
| 401 | `issued_suffixes` (region to the username suffix last issued) and |
| 402 | `cutovers` (the per-route member acknowledgements) are the other two |
| 403 | fields `relay.Record` has always carried and this listing has always |
| 404 | omitted. **(discussion #120)** `op` is a |
| 405 | counter bumped by every operation; `phase` is one of `config`, |
| 406 | `aggregate`, `done` and records how far the operation got. The |
| 407 | incarnation is sixteen hex characters drawn at random at first |
| 408 | enable and at each re-enable after a disable, never a counter, so |
| 409 | reuse across delete and name reuse is not a question. |
| 410 | - `control/relay.json`, the authority record the relays read: |
| 411 | `{apps: [{account, app, incarnation, regions}], written_at}`. It is |
| 412 | **derived**, never patched. Publishing it is one fixed sequence: read |
| 413 | the aggregate's current etag; list every `relay.json`; build the |
| 414 | aggregate from records whose `enabled` is true, whatever their |
| 415 | `phase`; write with a CAS on the etag read **first**; on conflict |
| 416 | start the whole sequence again. Because the etag is read before the |
| 417 | listing, a disable that lands during the listing makes the write |
| 418 | fail rather than restore the app. Followed by a `control/serial` bump |
| 419 | so relays notice within seconds; the bump is a latency hint, not the |
| 420 | lease. |
| 421 | |
| 422 | **Every operation is a walk the reconcile can finish, and the |
| 423 | reconcile converges everything.** An operation writes its intent to |
| 424 | `relay.json` first (`op` bumped, `phase` set), then does the config |
| 425 | write, then the aggregate publish, advancing `phase` after each step |
| 426 | with a CAS that carries the `op`. **One writer, enforced.** Every relay |
| 427 | operation, handler or reconcile, runs only in a console process that |
| 428 | holds the relay-writer lease, `control/relay-writer.json`: a CAS-written |
| 429 | record naming the holder with a sixty-second expiry, renewed every |
| 430 | twenty seconds, taken over by another process only after it lapses. A |
| 431 | process that fails a renewal stops every relay write in flight before |
| 432 | its next bucket write and abandons the operation, which the next |
| 433 | holder's reconcile finishes. Inside the holder, operations on one app |
| 434 | are serialised behind a per-app lock. So two console processes, |
| 435 | during a deploy overlap or otherwise, never both write relay config, |
| 436 | and a superseded key selection cannot reach config after a migration: |
| 437 | the writer that selected it either holds the lease, in which case it |
| 438 | is the only writer and no migration happened under it, or lost it and |
| 439 | stopped. **(codex R3-2, R4-1)** On top of that fence, the **reconcile** |
| 440 | does not only finish unfinished records, it **converges both derived |
| 441 | artefacts from every record, every five minutes and at console |
| 442 | start**: for each enabled |
| 443 | record it derives the expected `CARLOS_RELAY` value from the record and |
| 444 | the active masters and re-applies it if the config bundle holds |
| 445 | anything else; for each disabled record it removes the entry if |
| 446 | present; and it always rebuilds and publishes the aggregate whether or |
| 447 | not any record is unfinished. So a stale config write that lands after |
| 448 | a disable is undone within five minutes, an interrupted enable |
| 449 | delivers its config within five minutes, an interrupted disable |
| 450 | removes it within five minutes, a repeated `enable` while a previous |
| 451 | one is unfinished resumes it rather than skipping the config write, and |
| 452 | a deletion that removed the record but died before publishing is |
| 453 | repaired by the next unconditional rebuild. **(codex R2-2, R3-1, R3-2)** |
| 454 | |
| 455 | **When a disable is complete.** The app is out of every relay's |
| 456 | authority when the aggregate has been published without it **and** |
| 457 | every armed relay's heartbeat carries a `written_at` at or after that |
| 458 | publish. `carlos relay status` shows "disable pending on <regions>" |
| 459 | until then, and "revoked" after. Bounds: the publish is immediate on the |
| 460 | happy path and within five minutes after a crash; each relay reads |
| 461 | within sixty seconds of the publish; a relay that cannot read stops |
| 462 | serving on its own fifteen-minute lease. **(codex R2-1)** |
| 463 | |
| 464 | **State transitions** for `relay.json`: **(codex R1-13)** |
| 465 | |
| 466 | | From | Operation | Incarnation | Keys re-derived | Allocations | |
| 467 | |---|---|---|---|---| |
| 468 | | none | enable | new | yes | none | |
| 469 | | enabled, done | enable (same regions) | kept | no, no config write | untouched | |
| 470 | | enabled, not done | enable | kept | resumes the unfinished operation | untouched | |
| 471 | | enabled | enable `--regions` (change) | kept | only for added regions; removed regions' entries dropped | ended in removed regions on next authority read | |
| 472 | | enabled | disable | kept in record, app removed from aggregate | entry removed | ended everywhere on next authority read | |
| 473 | | disabled | enable | **new** | yes, all regions | none | |
| 474 | | any | app deleted | record removed with the app prefix; the deletion walk publishes the aggregate | | ended on next authority read | |
| 475 | | any | rotate `--region r` | kept | region `r` only, new kid | untouched; old kid accepted until retirement | |
| 476 | |
| 477 | **`carlos relay enable [--regions r1,r2]`** (member, API-mode only, on |
| 478 | the store model in `store.go` and `internal/console/apistore.go`; |
| 479 | ruling 2): |
| 480 | |
| 481 | 1. `apiAccountApp` gate; unknown and foreign are one byte-identical 404. |
| 482 | 2. Refuse if the deployment has no relay regions (`CARLOS_RELAY_REGIONS` |
| 483 | in console config is empty) with a message naming the operator. |
| 484 | Refuse if any of the app's routes has no config delivery (a |
| 485 | TCP-address route whose bundle does not opt into file delivery), |
| 486 | naming the route, because such a route could never adopt a rotated |
| 487 | key and would block retirement forever. **(codex R5-1)** |
| 488 | 3. Regions are every region in `CARLOS_RELAY_REGIONS`, **in the |
| 489 | console's order**, or `--regions` narrows them keeping that order. |
| 490 | There is no placement-first promise: placement names (`ie`, |
| 491 | `fleet/x`) are not AWS regions and an app may have several, so the |
| 492 | order is one fixed list for the app and every browser tries every |
| 493 | listed relay. **(codex R1-12, R2-11)** |
| 494 | 4. Apply the transition above to `relay.json` (CAS, `op` bumped, |
| 495 | `phase: config`). |
| 496 | 5. For each region needing keys, derive from that region's current |
| 497 | master and its kid, and write `CARLOS_RELAY` as one `kind:"secret"` |
| 498 | entry through `appconfig.Apply` (the `writeEnv` shape in |
| 499 | `internal/objstore/walk.go:602`). `Apply` touches the app's config |
| 500 | generation, a nonce plus timestamp (`internal/appconfig/ |
| 501 | generation.go:40`); record that generation string in `relay.json` |
| 502 | and advance to `phase: aggregate`. **(codex R2-6, R2-12)** |
| 503 | 6. Publish `control/relay.json`, bump `control/serial`, advance to |
| 504 | `phase: done`. |
| 505 | 7. Answer with the record. The CLI prints regions, incarnation, and |
| 506 | "config generation <g> written; delivery is reported by `carlos |
| 507 | relay status`", never "delivered". |
| 508 | |
| 509 | **Config adoption, observed per route.** The host's config sync, after |
| 510 | it renders a route's env file and does whatever that route's shape |
| 511 | calls for, writes an additive acknowledgement |
| 512 | `control/config-ack/<box>/<route host>.json`: the route's shape (unit, |
| 513 | socket exec, TCP exec with file delivery), the time, whether a restart |
| 514 | happened and succeeded, and **the relay key identities it rendered**: |
| 515 | the `username_suffix` per region parsed out of the `CARLOS_RELAY` value |
| 516 | it just wrote (account, app, incarnation and kid together), or none. A |
| 517 | kid alone would not do: a disable and re-enable between two host syncs |
| 518 | changes the incarnation and the key while keeping the kid, and an ack |
| 519 | naming only kids would report the old credentials as adopted. |
| 520 | **(codex R4-2)** The ack is keyed on the route, not the app, because an |
| 521 | app can have several routes on one box and one can adopt while another |
| 522 | does not; and it names identities rather than a generation string, |
| 523 | because the app-wide generation object is a change notification that |
| 524 | can coalesce or fail after the manifest write |
| 525 | (`internal/appconfig/generation.go:28`) and is not an identity of what |
| 526 | was rendered. Nothing reads the ack today but `carlos relay status`; |
| 527 | it is the first per-route statement of "this is what the process has" |
| 528 | the platform has, and the plan adds it to `internal/appconfig/sync.go` |
| 529 | as its own small step. **(codex R1-14, R2-6, R3-4)** |
| 530 | |
| 531 | **`carlos relay status`** reports, separately: desired state from |
| 532 | `relay.json` with its `op` and `phase`; the identities (username |
| 533 | suffixes) the console last issued per region; delivery per route, |
| 534 | "adopted at <time>, restarted" when a config-ack names exactly the |
| 535 | issued identities and a successful restart, |
| 536 | "written, awaiting delivery" when no ack names them yet, "file |
| 537 | delivered, awaiting `carlos relay adopted`" for a TCP exec route with |
| 538 | file delivery until a member acknowledges it, "adopted by member at |
| 539 | <time>" after, "restart failed" when the ack says so, and |
| 540 | "hibernating, delivered on wake" when the route is asleep (a route |
| 541 | added without delivery after enable shows "no config delivery: relay |
| 542 | will not reach this route" and blocks retirement until fixed); disable |
| 543 | progress as above; and per region, from |
| 544 | `control/relay-ack/<box>.json`, the heartbeat age, whether the record |
| 545 | it holds is current, whether its lease is live, which listeners are |
| 546 | up, and which `kid`s it has loaded. A heartbeat older than five minutes |
| 547 | is shown as "stale", so a dead relay cannot look current. |
| 548 | |
| 549 | **`carlos relay disable`** applies the transition (`phase: config`), |
| 550 | removes the `CARLOS_RELAY` entry through `appconfig.Apply`, publishes |
| 551 | the aggregate, marks `done`. |
| 552 | |
| 553 | **Rotation** (`carlos relay rotate --region r`, operator): **(codex |
| 554 | R1-10, R2-4)** |
| 555 | |
| 556 | 1. Operator runs `carlos relay keygen`, which prints a new `kid` and key. |
| 557 | 2. Operator adds the line to `/etc/carlos/relay-masters` on each of the |
| 558 | region's boxes with `retire_at` `-`. The relay re-reads the file; |
| 559 | no restart, no allocation touched. Its next heartbeat lists the new |
| 560 | `kid`. |
| 561 | 3. Operator **adds** the new `kid:key` to the console secret for the |
| 562 | region, which holds every master the console may need, |
| 563 | `CARLOS_RELAY_MASTERS_<REGION>="<kid>:<key>[,<kid>:<key>]"`. Adding |
| 564 | it issues nothing: **which master the console issues from is a |
| 565 | separate, unsealed pointer**, `control/relay-issuer.json` |
| 566 | `{"<region>": "<kid>"}`, and every path that derives a key for an |
| 567 | app (enable, region change, re-enable, reconcile) reads that |
| 568 | pointer and uses the master it names. A staged master is inert |
| 569 | until the pointer moves. **(codex R3-5)** |
| 570 | 4. `carlos relay rotate --region r` **refuses until every armed relay |
| 571 | in the region has heartbeated the new `kid`**, then moves the |
| 572 | issuer pointer to it, re-derives and re-delivers keys for every app |
| 573 | enabled in that region as an operation on each record, and prints |
| 574 | the earliest permitted retirement time for the old `kid`: 28 hours |
| 575 | from now. |
| 576 | 5. `carlos relay status --region r` shows, per route of every enabled |
| 577 | app, which identities its config-ack names and when. **Each route's |
| 578 | cutover** is established one of two ways, by route shape. A |
| 579 | unit-backed or socket-exec route: the config sync's ack naming the |
| 580 | issued identities, since the platform restarted it. A TCP-address |
| 581 | exec route with file delivery: its process belongs to an external |
| 582 | supervisor the platform cannot restart, so the cutover is a |
| 583 | **member acknowledgement**, `carlos relay adopted --host <route>`, |
| 584 | given after the member has restarted the process; the console |
| 585 | records it in `relay.json` against that route together with the |
| 586 | identities issued at that moment, so an acknowledgement given |
| 587 | before a later rotation does not count for it. A route in neither |
| 588 | category cannot exist under an enabled relay (enable refuses it), |
| 589 | and a route added later without delivery blocks retirement until |
| 590 | it is fixed. The old master's `retire_at` may be set no earlier |
| 591 | than `max(rotation + 28h, latest cutover + 4h)` across every route |
| 592 | of every enabled app in the region, and `status` prints that time |
| 593 | once every route has a cutover, or names the routes still without |
| 594 | one. Setting it earlier is refused by nothing but the docs, and ends |
| 595 | any allocation still on the old `kid`, which is the force-retirement |
| 596 | case §2 excludes from its guarantee. **(codex R3-4, R4-3, R5-1)** |
| 597 | 6. After `retire_at` the operator removes the line from the boxes and |
| 598 | the console secret at leisure. |
| 599 | |
| 600 | **App deletion** removes `relay.json` with the rest of the app prefix, |
| 601 | and the deletion walk publishes the aggregate so the app is gone from |
| 602 | authority at once. If the walk dies between the two, the reconcile's |
| 603 | unconditional rebuild, which lists records rather than looking for |
| 604 | unfinished ones, publishes without the app within five minutes. |
| 605 | **(codex R3-1)** |
| 606 | |
| 607 | ## 6. Infrastructure |
| 608 | |
| 609 | - `infra/modules/region-host/main.tf`: a `relay` variable, default off, |
| 610 | that adds ingress for UDP 3478, TCP 3478 and UDP 49152 to 65535 from |
| 611 | `0.0.0.0/0`, and outputs the VPC CIDR for `CARLOS_RELAY_DENY_CIDRS`. |
| 612 | Flagship turns it on per region. |
| 613 | - DNS: an A record `relay.<aws region>.<platform domain>` to the region's |
| 614 | Elastic IP, next to the existing `aws_eip.host` resource. |
| 615 | - Each armed box gains `/etc/carlos/relay-masters` (staged by the |
| 616 | operator, like `host.env`) and `CARLOS_RELAY_DENY_CIDRS` and |
| 617 | optionally `CARLOS_RELAY_PUBLIC_IP` in `host.env`. |
| 618 | - Console config gains `CARLOS_RELAY_REGIONS` and one |
| 619 | `CARLOS_RELAY_MASTERS_<REGION>` secret per region. |
| 620 | - Egress cost: every forwarded byte leaves the box once. The |
| 621 | unit-economics spec gets a line for relay egress per region, and the |
| 622 | two meters in §3 are what fill it. |
| 623 | |
| 624 | ## 7. Testing |
| 625 | |
| 626 | - `internal/relay`: derivation and credential against the §2 vector; |
| 627 | the `AuthHandler` against a fixture authority record and masters file |
| 628 | covering every refusal reason in §4, including a valid credential for |
| 629 | app B presented from a typed source bound to app A, and a UDP and a |
| 630 | TCP client with the same address string holding separate bindings; |
| 631 | the permission handler against each denied range and the allowed |
| 632 | EIP; the relay socket dropping a packet to the EIP on a non-allocated |
| 633 | port; limits against a fake clock, including an account spreading |
| 634 | allocations across apps, an allocation ended by the byte budget, and |
| 635 | a token bucket dropping a burst; the meter against wrapped in-memory |
| 636 | sockets, checking that rejected peer ingress is not counted as `in` |
| 637 | and that both forwarding legs are counted; the hourly object surviving |
| 638 | a restart, including a restart across an hour boundary that rebuilds |
| 639 | the rolling window from both objects; a wrapper's close removing only |
| 640 | its own binding while a replacement binding for the same typed |
| 641 | source survives; shedding triggered by a fake pressure reading. |
| 642 | - End to end, on loopback with a fixture master: a pion TURN client |
| 643 | allocates, refreshes, and keeps relaying until the credential's |
| 644 | expiry, at which point traffic stops with no further client request; |
| 645 | the same after a disable written to the fixture authority record; |
| 646 | two clients on one relay reaching each other's relay candidates; a |
| 647 | rotation, by rewriting the masters file, during which an allocation |
| 648 | on the old kid carries traffic uninterrupted, then a `retire_at` |
| 649 | ending it; a lease lapse ending allocations and a successful read |
| 650 | restoring service. |
| 651 | - `internal/host`: the unit is embedded, written and started by the |
| 652 | bootstrap plan when the staging root carries a masters file, and not |
| 653 | started when it does not; the adopt restart path names the relay; the |
| 654 | config sync writes a config-ack. |
| 655 | - `internal/console`: the doors through `rawReq` and a `relayURL` |
| 656 | fixture: the byte-identical 404, the no-regions refusal, each row of |
| 657 | the transition table, the config entry written and removed, the |
| 658 | aggregate after concurrent enable and disable of two apps with the |
| 659 | etag-first ordering, an enable interrupted after `phase: config` |
| 660 | finished by the reconcile, a superseded writer refused by its CAS, a |
| 661 | stale config write landing after a disable and undone by the next |
| 662 | reconcile, a deletion that removed the record without publishing |
| 663 | repaired by the next rebuild, an enable during a staged-but-unmoved |
| 664 | rotation issuing from the old kid, a process that lost the writer |
| 665 | lease making no further relay write, a delayed old-key publisher |
| 666 | refused after a migration, a disable and re-enable between two host |
| 667 | syncs not reported as adopted by the old ack, and `rotate` refusing |
| 668 | until the fixture heartbeats list the kid; the config sync writing a |
| 669 | per-route ack naming the identities it rendered; the retirement time |
| 670 | computed with a manual cutover after hour 28; enable refused for an |
| 671 | app with a route lacking config delivery; a `carlos relay adopted` |
| 672 | given before a later rotation not counting as that rotation's |
| 673 | cutover; an externally supervised TCP route with file delivery |
| 674 | blocking retirement until acknowledged. |
| 675 | - `docs/site/cli.md` gains `### carlos relay` with a fenced example, or |
| 676 | the docsite gate fails. |
| 677 | - Flagship gate, as amended in the roadmap: a Meet call between two |
| 678 | symmetric-NAT networks connecting through the Ireland relay; both |
| 679 | browsers forced to relay and connecting through it **on the real |
| 680 | EIP**, and a probe to a non-allocated port on the EIP dropped; a |
| 681 | credential minted for one app refused for another, including on an |
| 682 | existing allocation's tuple; a credential from before a disable |
| 683 | refused after re-enable and its allocation ended; traffic stopping at |
| 684 | a credential's expiry with the browser doing nothing; a relay with a |
| 685 | lease older than fifteen minutes refusing new allocations and ending |
| 686 | carried ones; a permission to a private address refused; the relay |
| 687 | meter object showing the call's bytes on both legs; a rotation |
| 688 | during a call with the call unaffected; the relay shedding |
| 689 | allocations under host memory pressure caused by a tenant, with the |
| 690 | edge still serving. |
| 691 | |
| 692 | ## 8. Not built |
| 693 | |
| 694 | - **TURN over TLS (5349 or 443)** **(ruling 4)**. Certificate issuance |
| 695 | for a non-HTTP listener needs the relay hostname registered with the |
| 696 | deployment's ACME issuer, which is elected and may be another box |
| 697 | (`internal/edge/certsync.go`), and its warmer excludes such names |
| 698 | (`internal/edge/certwarm.go`); a listener that cannot start until a |
| 699 | certificate exists cannot trigger issuance by being visited. That is |
| 700 | its own small design, to be written when a network that needs it is |
| 701 | reported. 443 additionally needs a second address per box. The |
| 702 | roadmap's stage 4a text is amended to match. **(codex R1-15, R2-10)** |
| 703 | - **An inter-region byte meter** **(ruling 5)**. Classifying a peer |
| 704 | address by AWS region needs the published IP ranges on the box and a |
| 705 | rule for non-AWS peers; version one meters what is forwarded in each |
| 706 | direction, which is what the bill is made of. Filed for stage 5 with |
| 707 | the usage view; the roadmap's meters and gate are amended. |
| 708 | **(codex R1-18, R2-10)** |
| 709 | - An issuance API: apps mint offline from delivered keys. |
| 710 | - Client-aware relay selection: the list is in console order; ICE picks. |
| 711 | - Commercial quotas and an app-page usage view: the meters exist, the |
| 712 | page is stage 5. |
| 713 | - An SFU: stage 4b, after this ships. |
| 714 | - Any change to how apps signal: SSE plus POST stays Meet's. |
| 715 | |
| 716 | ## 9. Rulings |
| 717 | |
| 718 | Paul ruled on each of these on 2026-09-07. **(codex R2-10)** |
| 719 | |
| 720 | 1. **(ruling)** Maximum credential life is four hours. The roadmap's |
| 721 | one-hour figure is amended. |
| 722 | 2. **(ruling)** Enable is member-level; the operator's consent is the |
| 723 | region list and the limits. |
| 724 | 3. **(ruling)** The console holds every region's master as sealed |
| 725 | config; the exposure is stated in §4 and the operator docs; the |
| 726 | relay-derives-and-writes design is filed as the tightening. |
| 727 | 4. **(ruling)** No TURN over TLS in version one; the roadmap's stage 4a |
| 728 | ingress text is amended. |
| 729 | 5. **(ruling)** Two meters in version one, in and out; the inter-region |
| 730 | meter and the usage line move to stage 5; the roadmap's metering |
| 731 | text and gate are amended. |
| 732 | 6. **(ruling)** The limits in §3 with their stated defaults, including |
| 733 | the token buckets. |
| 734 | |
amadan